Regulating AI: What's the Definition?

[phinds]

It doesn't really matter if we classify the decision as an AI decision, what matters is that the event is not explainable. You can't point to any flaw in the design that resulted in the event, and you can't find anybody who made any decision which can be blamed for the event. The best you could do is blame the company for not knowing what the risks were and for deploying a product with unknown risks, but are there regulations requiring them to? Or you can chalk it up to bad luck.

You should watch "Airplane Disasters" on TV. The number of plane accidents that remain unexplained after the investigators finish is somewhere between few and none. The is no reason why the same could not be applied to AI things, so I seriously doubt your contention that an accident is "not explainable".

 

[Jarvis323]

You should watch "Airplane Disasters" on TV. The number of plane accidents that remain unexplained after the investigators finish is somewhere between few and none. The is no reason why the same could not be applied to AI things, so I seriously doubt your contention that an accident is "not explainable".

Deep learning models can have billions of interdependent parameters. Those parameters are set by the back-propagation algorithm as it learns from data. The intractability of understanding a multi-billion parameter non-linear model aside, the back-propegation process which parameterizes it is a proven chaotic process. Say such a system makes a decision that results in an accident. How do you propose to try and explain it? What kind of human choices do you suppose can be made accountable? Would you subpoena all of the training data? Then would you comb through the data, and compare it with the multi-billion parameter model and come to a conclusion that something in the data was the problem, and someone should have known that the emergent model would have made a particular decision under a particular set of conditions after the training process?

 

[Vanadium 50]

What if it is legal for the AI to drive your car, and the AI is a statistically better/safer driver than a human being?

We'll burn that bridge when we come to it.

 

[anorlunda]

The article linked in #1 about the EU proposal says, "Make sure that AI is appropriately transparent and explainable"

I think the requirement is there because of potential race bias, not because of accidents.

White person A is approved for credit, but black person B is denied. Why?
A test on 10000 people shows that whites are approved more often. Transparent and explainable are needed to prove absence of racism. (Think of the bank red lining scandals.)

It is also my understanding that no neural net is explainable in that sense. That makes an explainability requirement a very big deal. Scrubbing training data to exclude racial bias still would not provide explanations of the results.

Consider the facial authorization application. A nxn pixel image is input, it is compared to a nxn reference image. There are two outputs, 1) Yes the two images are of the same person, or 2) No, not the same person. How could it be modified to provide an explanation of the result? Why no for person X? Why are there different success rates for different races or genders or ages? Neural networks are unable to answer those why questions.

 

[Jarvis323]

The article linked in #1 about the EU proposal says, "Make sure that AI is appropriately transparent and explainable"

I think the requirement is there because of potential race bias, not because of accidents.

White person A is approved for credit, but black person B is denied. Why?
A test on 10000 people shows that whites are approved more often. Transparent and explainable are needed to prove absence of racism. (Think of the bank red lining scandals.)

It is also my understanding that no neural net is explainable in that sense. That makes an explainability requirement a very big deal. Scrubbing training data to exclude racial bias still would not provide explanations of the results.

Consider the facial authorization application. A nxn pixel image is input, it is compared to a nxn reference image. There are two outputs, 1) Yes the two images are of the same person, or 2) No, not the same person. How could it be modified to provide an explanation of the result? Why no for person X? Why are there different success rates for different races or genders or ages? Neural networks are unable to answer those why questions.

I agree this is a big issue. I don't think explainability applies only to this kind of case though. AI systems are going to control our systems in every sector. In all cases where something could go wrong, too many to list, explainability matters.

 

[Dale]

It doesn't really matter if we classify the decision as an AI decision, what matters is that the event is not explainable. You can't point to any flaw in the design that resulted in the event, and you can't find anybody who made any decision which can be blamed for the event.

In traditional products it is generally not sufficient for a company to not know that a product is dangerous. They need to know that it is safe.

The best you could do is blame the company for not knowing what the risks were and for deploying a product with unknown risks, but are there regulations requiring them to?

There are not just regulations, there are laws. Product safety issues are not just regulatory issues, they are also criminal negligence and liability issues. That the risks were unknown is generally taken to mean that the company’s product safety testing protocols were insufficient. In consumer safety ignorance is not typically a sound defense.

Although there are criminal statutes, by far the bigger legal threat to traditional engineering companies is civil litigation. Such litigation depends much more on the the harm received than on any knowledge by the company of the risk.

Or you can chalk it up to bad luck.

Humans generally prefer to fix blame, including humans serving on juries.

 

[Oldman too]

AI systems can evolve constantly, even on the fly.

I honestly, myself, believe your point there is one of the biggest factors to consider, especially in the long term. Rather than a "fix" for the problem, there will need to be some form of constant monitoring and "upgrading" of the fix. By that, I'm thinking that regulating AI, especially the legal aspects will have to be a dynamic, always adapting to new situations process. (It's unfortunate that the best candidate for that job is AI, how's that for irony?) After chasing through the links cited in the openai piece, its obvious this is a can of worms without a gordian knot solution. As an afterthought, it does guarantee this thread a long life though.

Would an approval process need to be redone with each update to a deep learning system? Does it depend on context? For medical diagnosis? For a lawn mower? For a self driving car? For a search engine? For a digital assistant? For a weapon? For a stock market trading system?

Very good questions, this is another facet of the can of worms issue. I'm more qualified as a spectator in this thread than anything so I'm better off following the conversation than giving serious answers.

 

[Dale]

Would an approval process need to be redone with each update to a deep learning system?

I hadn’t seen this comment before.

Just so that you can understand my perspective, let me explain my relevant background. I currently work for a global medical device manufacturer, one that has one of the largest number of patents in applying AI technologies to medical products. I am not currently in one of our R&D groups, but I was for 14 years and I worked on more than one AI-related project.

Indeed, the regulatory process needs to be redone with each update to any of our AI systems. Our AI technologies are regulated the same as all of our other technologies. As a result our development is slow, incremental, careful, and thoroughly tested, as it should be, for AI just like for any other technology we incorporate into our medical devices.

 

[Jarvis323]

I hadn’t seen this comment before.

Just so that you can understand my perspective, let me explain my relevant background. I currently work for a global medical device manufacturer, one that has one of the largest number of patents in applying AI technologies to medical products. I am not currently in one of our R&D groups, but I was for 14 years and I worked on more than one AI-related project.

Indeed, the regulatory process needs to be redone with each update to any of our AI systems. Our AI technologies are regulated the same as all of our other technologies. As a result our development is slow, incremental, careful, and thoroughly tested, as it should be, for AI just like for any other technology we incorporate into our medical devices.

I think that in the case of medical devices, and also self driving cars, the issue is somewhat straightforward because the risks are taken seriously, and there are existing regulatory frameworks which can be mostly relied on already. But other sectors can be much fuzzier.

 

[russ_watters]

It doesn't really matter if we classify the decision as an AI decision, what matters is that the event is not explainable. You can't point to any flaw in the design that resulted in the event, and you can't find anybody who made any decision which can be blamed for the event. The best you could do is blame the company for not knowing what the risks were and for deploying a product with unknown risks, but are there regulations requiring them to? Or you can chalk it up to bad luck.

Yes that's how liability works/no it can't be chalked-up to bad luck. And yes, the flaw in the design is obvious: the AI made a bad decision therefore the AI is faulty. That's the entire point of AI: it makes its own decisions.

I agree with Dale here, which is why I think this entire discussion is much ado about nothing. At some point maybe AI will be granted personhood. Until then there is no implication for product liability law in AI.

 

[Vanadium 50]

think the requirement is there because of potential race bias

That's kind of the point.

Say you are a mortgage lender. You can't legally use race in the decision process - but from a purely dollar basis, you'd like to. Race and default rates are correlated, but one can't legally use this correlation. So, if you a zillion dollar bank, what do you do?

You build a model where the only inputs are perfectly legal, and you don't try to infer race directly - but if the model comes up with an output that has a race-based correlation, well, what's a banker to do? He has plausible deniablity.

In our privacy-free world, getting this information is easier than it should be: phone records, groceries, magazine subscriptions, other purchases...

 

[Jarvis323]

Yes that's how liability works/no it can't be chalked-up to bad luck. And yes, the flaw in the design is obvious: the AI made a bad decision therefore the AI is faulty. That's the entire point of AI: it makes its own decisions.

I agree with Dale here, which is why I think this entire discussion is much ado about nothing. At some point maybe AI will be granted personhood. Until then there is no implication for product liability law in AI.

At the least, AI being unexplainable makes product liability law very messy and complex. Because, people will try to shift blame, and to get to the bottom of who is to blame, people will try to determine what went wrong. Then you have the consequence that such cases could be very costly and drawn out. And that can strain the legal system, as well as make for situations where one side is forced to forfeit because they can't afford to proceed.

 

[russ_watters]

That's kind of the point.

Say you are a mortgage lender. You can't legally use race in the decision process - but from a purely dollar basis, you'd like to.

What if the mortgage lender doesn't want to use race in the decision process but the AI decides it's a good metric anyway?

 

[russ_watters]

At the least, AI being unexplainable makes product liability law very messy and complex. Because, people will try to shift blame, and to get to the bottom of who is to blame, people will try to determine what went wrong.

I don't think that's true. You don't have to unravel every bit of the decision making process in order to judge whether the decision was faulty. Why the AI made its wrong decision doesn't matter. Liability doesn't hinge on whether the decision was accidental or on purpose, it just depends on whether the decision was bad/good.

 

[Jarvis323]

• Ensure that AI is used safely
• Ensure that AI is technically secure and functions as designed
• Make sure that AI is appropriately transparent and explainable
• Consider fairness
• Identify a legal person to be responsible for AI
• Clarify routes to redress or contestability

Above are the principles that they are guiding their regulation proposal.

https://www.gov.uk/government/news/...tion-and-boost-public-trust-in-the-technology

The first 2 principles require testing/approval/certification. Sure this is already done in some cases where a self learning AI replaces a person or operates a device which used to be operated by an explicit algorithm. But not all.

The 3rd is arguably impossible in an absolute sense, but can be strived for. Testing requirements can help here as well.

The 4th, fairness, we have discussed. It can also be helped by testing.

The 5th and 6th (identify legal person responsible and clarify routes to redress contestability) are crucial if you want to avoid messy and costly legal battles.

 

[Dale]

people will try to shift blame

Of course they will try. They would have some pretty terrible lawyers if they didn’t even try to shift blame.

But it is very difficult for a manufacturer who has produced a product that did serious harm to actually convince a jury that they are not to blame. It can happen on occasion, but that is very much the exception rather than the rule. This is why companies are usually the ones that offer a settlement and plaintiffs are usually the ones that threaten to go to trial.

 

[Vanadium 50]

What if the mortgage lender doesn't want to use race in the decision process but the AI decides it's a good metric anyway?

The AI doesn't know about race. Ironically, if you want a race-neutral process, you need to feed that into the AI training: you can't be race-blind and race-neutral.

 

[Vanadium 50]

There is a lot about "unexplainable". In many - possibly most - cases, the actual determination is deterministic. The probability of a loan default is a_1 * income + a_2 * average number of hamburgers eaten in a year * a_3 * number of letters of your favorite color + ... The AI training process produces the a's, and that's a black box, but the actual calculation in use is transparent.

There are many good reasons to do this.

 

[Jarvis323]

You build a model where the only inputs are perfectly legal, and you don't try to infer race directly - but if the model comes up with an output that has a race-based correlation, well, what's a banker to do? He has plausible deniablity.

It doesn't just apply to race. You could be a person with a 23 character name, who's birthday is on April 1st, who watches basketball, and who has a green fence. And the AI could determine this makes you more likely to default, because people with those features happen by chance to have defaulted more often than normal, and the AI just sees correlation, not causality.

A solution involves having sufficiently balanced and stratified training data. If you have stratified over all features well enough, then race shouldn't be predictable from the features for the training data. You could verify that. However, the combinatorial explosion of feature combinations makes this difficult to do perfectly if you use a lot of features. The more rare your set of features, the more prone you are to be the victom of a selection bias that leaks into the model.

If you could train a model with data that is properly stratified, and you can verify that the feature set is not a predictor of race within the training data, then the next step is to eliminate race as a feature in the model and theoretically it should be unlikely to have a racial bias. The next thing you do is test it.

 

[Rive]

All we need to do is hold them to the same product safety standards as every other engineer and engineering company.

I don't think that that would ever work. Validating a complex software is such a nightmare just as is, even without any 'learning' involved (which is supposed to be kind of a programmer-independent process, exactly to correct/prevent errors in judgement in the programming process) that on industrial level it's just - won't do.

"I let my five-year-old* drive my car" - how does that differ from "I let an AI, who is less smart than a five-year old."

Involving 'smart' is a trap here. I don't need a 'smart car'. I need one which can drive better than I do. I don't care if it's dumb as a flatworm.

The human brain is an amazing learning machine that we don’t yet understand and cannot yet artificially replicate or exceed.

The human brain has a specific advantage, which is: choice. I won't drive out in a snowstorm, for example (I prefer to imagine that not much people would do). That alone prevents plenty of accidents to happen.
In this regards AIs will be in quite a disadvantage, I think

Why the AI made its wrong decision doesn't matter. Liability doesn't hinge on whether the decision was accidental or on purpose, it just depends on whether the decision was bad/good.

It's just the AI in discussion is about interacting with unclear factors (human beings).
Kind of like omitting the fact from the decision of the jury about you shooting somebody that whether you were at gunpoint or not?

 

[Swamp Thing]

It seems to me that one needs a highly specific definition of what AI is before enforcing such a law. I prefer a very broad definition. I would include James Watt's flyball governor from 1788 as an AI. It figured out by itself how to move the throttle, and it displaced human workers who could have done the same thing manually.

View attachment 304739

On an abstract level, if we have a black box with which we communicate, what is the test we can use to prove that the content of the box is an AI?

And then there was Cornelis Drebbel and his temperature-controlled oven, "one of the first manmade feedback mechanisms in history". (This was around 1620).

 

[Borg]

Kind of like omitting the fact from the decision of the jury about you shooting somebody that whether you were at gunpoint or not?

The gun in this case is the data and how the AI was trained. There are a host of questions that could be asked when investigating an algorithm. Just a few that come to mind at 5am:

• What data was used? Was the data biased? If you have an AI that tells you where to send police cars to deter crime and it's basing its decision on where the most arrests occur, it will be a self-reinforcing algorithm since people tend to get arrested where the police are. Similarly, an algorithm will tell you that people default on mortgages more often in poorer neighborhoods. If your only data for black people is from a ghetto, it will decide that black people are a risk. But what about adding a zip code and leaving out race? You could still have problems with this.
• What algorithm was used? What other algorithms were tried and what were their results? This could be helpful for the defense if they could show that they tried multiple paths to build the best possible algorithm.
• Additionally, there are two major types of algorithms in machine learning - supervised and unsupervised. In supervised learning, you have data that is labeled with the correct classification (i.e. this is a picture of a cat). In unsupervised learning, that isn't the case and the algorithms are learning on their own. Picture a video game where you click randomly on the screen and your only reward is whether you rescue the princess or die. Algorithms trained in this manner can learn how to play video games very well. However, the data is not only the game inputs that were fed to the algorithm but also its randomly selected initial attempts and statistical tracking of results while attempting to win over millions of games. There are then parameters for randomly choosing the statistically correct actions or retrying what it currently believes to be statistically wrong ones in order to explore. How do you reproduce the 'data' in this case?
• What are the acceptable operating inputs for the data? In the case of a mortgage, a defendant could argue that their AI rejected an applicant because of missing or incorrect information on the application. Also, consider this case with respect to self-driving cars - Hacking Stop Signs. Who is to blame here, if someone defaces a stop sign so that a self-driving car doesn't recognize it. Humans would still recognize stop signs in the examples that are given but there could just as easily be cases where the AI would still recognize the sign when humans wouldn't. The company would surely have a map with all of the stop signs in it but that is a maintainence task that never finishes. There will always be real world information that isn't up to date in the database.

 

[anorlunda]

If your only data for black people is from a ghetto, it will decide that black people are a risk.

The problem is that if your training data set is big enough to cover 100% of the population, and the result still turns out to correlate to race.

The critics and regulators will never be satisfied with unbiased inputs (the training data), they will judge compliance by the outcome (the AI's results).

 

[Dale]

I don't think that that would ever work.

It is already working in my company

 

[Rive]

It is already working in my company

Based on the approach you wrote about I have a feeling that either the AI you are talking about and the AI which is 'in the air' is slightly different things or yours is working with some really constrained datasets.

 

[Borg]

The problem is that if your training data set is big enough to cover 100% of the population, and the result still turns out to correlate to race.

Yes. Data bias can take many forms. While you could try to remove race to avoid some of this, the algorithms are very good at finding other ways of creating this situation. After all, their purpose is to learn to classify on their own. In this case, the AI could potentially pick up societal biases through other inputs that we might not even dream are related to a particular race or group of people. If there is systemic racism or bias, the algorithms will tend to pick up on it.

 

[Vanadium 50]

The problem is that if your training data set is big enough to cover 100% of the population, and the result still turns out to correlate to race.

This is a certainty.

It's true trivially - a correlation of exactly zero is a set of measure zero.

But it's also true because of correlations we already know about. Mortgage rate default is correlated with income (or debt-to-income, if you prefer), unemploym,ent rate etc. These variables are also correlated with race. So of course race will show up as a factor.

The question is "what are we going to do about it?". The answer seems to be "blame the mathematics" because AI (or really this is more ML) is producing an answer we don't like.

One answer that our society has come up with is "well, let's just feed in all the information we have, except race." But of course the AI finds proxies for race elsewhere in the data - in the books we read, the food we eat, the movies we watch, and as mentioned earlier, our ZIP codes (and an even better proxy is our parents' ZIP codes). Of course it does. That's its job.

More generally, if we plot correlations of a zillion variables against a zillion other variables, we are going to find some we don't like. What are we going to do about that?

 

[russ_watters]

I don't think that that would ever work. Validating a complex software is such a nightmare just as is, even without any 'learning' involved (which is supposed to be kind of a programmer-independent process, exactly to correct/prevent errors in judgement in the programming process) that on industrial level it's just - won't do.

I don't see a problem there. Can you clarify/elaborate?

It's just the AI in discussion is about interacting with unclear factors (human beings).
Kind of like omitting the fact from the decision of the jury about you shooting somebody that whether you were at gunpoint or not?

I don't understand what point you are trying to make. Can you clarify/elaborate on what I'm omitting/why it matters?

 

[Dale]

Based on the approach you wrote about I have a feeling that either the AI you are talking about and the AI which is 'in the air' is slightly different things or yours is working with some really constrained datasets.

AI is a very broad class of technologies, and we use many different ones for different purposes. My company’s experience using AI is highly relevant to a discussion regarding actual companies using actual AI technology in actual safety-critical products within actual regulatory environments. Our current real-world experience directly contradicts your claim that holding AI products to the same product safety standards as every other engineer and engineering company would never work. AI products should not get a pass on safety and in our industry they do not. They are held to the same standards as our non-AI software, and so far it works.

 

[Rive]

Our current real-world experience directly contradicts your claim that holding AI products to the same product safety standards as every other engineer and engineering company would never work.

Sorry, but the only thing I will answer with is exactly those lines you quoted from me.

Based on the approach you wrote about I have a feeling that either the AI you are talking about and the AI which is 'in the air' is slightly different things or yours is working with some really constrained datasets.

I have great respect for that kind of mindset by the way: but it has some really severe limitations and impacts, quite incompatible with the current hustle and bustle around AI.

 

[russ_watters]

I have great respect for that kind of mindset by the way: but it has some really severe limitations and impacts, quite incompatible with the current hustle and bustle around AI.

Are you trying to say that the current definition/status of AI is too broad/soft, so current experiences are not relevant? And under a narrower/harder definition things might be different? If so, sure, but there will be a clear-cut marker even if not a clear-cut threshold for when the shift in liability happens: legal personhood.

 

[Dale]

the only thing I will answer with is exactly those lines you quoted from me

Not exactly a productive conversational approach, but whatever. You are just hoping to exclude a clear counterexample to your argument with no sound basis for doing so.

 

[anorlunda]

They are held to the same standards as our non-AI software, and so far it works.

I agree, but the topic in this thread is special regulations that treat AI differently than non-AI stuff. AI is irrelevant in negligence tort claims.

 

[Dale]

I agree, but the topic in this thread is special regulations that treat AI differently than non-AI stuff. AI is irrelevant in negligence tort claims.

Right, my position is that such regulations seem unnecessary to me. At least in my field, the existing regulations seem sufficient.

 

[Jarvis323]

One answer that our society has come up with is "well, let's just feed in all the information we have, except race." But of course the AI finds proxies for race elsewhere in the data - in the books we read, the food we eat, the movies we watch, and as mentioned earlier, our ZIP codes (and an even better proxy is our parents' ZIP codes). Of course it does. That's its job.

You can also choose your data+features so that the model becomes biased against a group intentionally.

But you can also choose your data+features so that the model is fair as I explained in post 54. The only problem is you need enough data, and you would have to restrict the number of features you use so that you are able to sufficiently stratify.

So the question is, should we regulate how companies choose their training data? Should they be required to be transparent about their training data, training pipeline and validation? Should their data+features need to pass a test that proves fairness?