What Insights Does Scott Aaronson Offer on AI Safety?

[Jarvis323]

Aaronson went on leave to join OpenAI and study AI safety, and wrote a blog post on the topic.

https://scottaaronson.blog/?m=202211

Maybe this could be a good jumping off point to discuss the topic in general.

 

[anorlunda]

Here's the flaw from that blog post:

AI ethics and AI alignment, might both feel like the other is completely missing the point! On top of that, AI ethics people are almost all on the political left, while AI alignment people are often centrists or libertarians or whatever, so that surely feeds into it as well.

It has already started to be politically polarized.

But a bigger flaw is the most basic. Find a definition of AI that everyone agrees to. Unless people agree as to the subject of a debate, the debate is doomed.

 

[Drakkith]

The usual story here is that someone puts an AI in charge of a paperclip factory, they tell it to figure out how to make as many paperclips as possible, and the AI (being superhumanly intelligent) realizes that it can invent some molecular nanotechnology that will convert the whole solar system into paperclips.

If you have an AI that can invent this kind of nanotechnology, then you don't have paperclip factories anymore to put AI's in. Besides, paperclip factories only have the facilities to make paperclips. They don't have the ability to invent super advanced nanotechnology. This is true for ALL manufacturing facilities. The supreme law of, "I don't want to spend money on stuff I don't need to make my product" pretty much prevents factories that make ball bearings or aluminum cans from making weapons of mass destruction or magic nanotech.

It also prevents people from spending money on AI's that they don't need. If we get to the point where a factory is paying for a general purpose AI that can do millions of tasks that the factory doesn't need it to do, including advanced research and development, then we're to the point where everyone has similar AI that can probably be used to prevent other AI's from doing bad things.

I'm sorry but destroying the world is hard. It takes a lot of effort. Of the potential world-ending weapons I can think of off the top of my head:

• Nuclear weapons have huge infrastructure chains supporting their research, manufacture, deployment, maintenance, and use. No one, not even an AI, is just going to take them over and wipe us out in a day. Even the launching of existing nukes is easily avoidable by not giving control over them to an AI or any single source (which is what we already do).
• Bioweapons are potentially easy for other advanced AI's to reverse engineer a cure for and are subject to many potential issues with surviving and spreading once released. The recent Covid pandemic shows a (admittedly flawed) way of handling such things. I suspect that by the time we reach the point that we understand how to artificially create super-pathogens, we will also understand how to quickly analyze and stop them.
• 'Nanotech' is a magic word used to describe fantasy technology that will probably never exist. Going from raw materials to even a simple finished product takes a huge amount of work and, often, other raw materials and finished products. Going from raw ore to a refined metal by itself is a rather delicate process that requires various chemicals, high-temperatures, LOTS of energy, and the right conditions to do it correctly. Squirting some grey goo on a rock in Alaska is not going to make a bunch of paperclips.
• Armed robots are countered by other armed robots controlled by other AI's working for humans and not against them. And there's probably going to be a lot more on our side.

But a bigger flaw is the most basic. Find a definition of AI that everyone agrees to. Unless people agree as to the subject of a debate, the debate is doomed.

I think the biggest flaw is the complete and utter lack of understanding of how the real world works that most people debating this issue have.

"Oh no! Scary AI will kill us all if we let it!"
"Ok. So don't let it. "
"But it's so much smarter than us!"
"But we're the ones who invented it, created and run the infrastructure that runs it, and maintains its systems."
"But what if it gets control of everything?! Then we're doomed!!"
"Ok. So don't let it do that."
"But it's so much smarter than us!"
...
...

 

[Jarvis323]

I'm sorry but destroying the world is hard. It takes a lot of effort.

Human effort? Or how do we measure effort for a hypothetical AI? Maybe in resources and energy?

 

[Jarvis323]

Here's the flaw from that blog post:It has already started to be politically polarized.

I think Scott is just pointing this out, and then making a suggestion to broaden and depoliticize it. I see it more as (what Scott sees as) a flaw in our collective effort so far, rather than a flaw in his blog post.

But a bigger flaw is the most basic. Find a definition of AI that everyone agrees to. Unless people agree as to the subject of a debate, the debate is doomed.

I don't think a definition is needed. We don't have such a definition for human intelligence. For AI or human intelligence, we may never have one. The complexity is generally beyond our ability to pin down through analytical thinking.

I prefer a more functional basis. What can it do? This depends on the specific instance of technology. And what is possible? In general it is unknown, or not agreed on.

Also, I don't see it as a topic to debate, so much as a topic to study.

 

[Drakkith]

Human effort? Or how do we measure effort for a hypothetical AI? Maybe in resources and energy?

As in a vigorous or determined attempt. It takes time, energy, resources, etc. It's not trivial. An AI isn't going to build the entire infrastructure needed to manufacture and deploy nuclear weapons overnight or in a single warehouse someone forgot about. The same is true for any kind of advanced weapon or non-primitive technology.

 

[anorlunda]

The supreme law of, "I don't want to spend money on stuff I don't need to make my product" pretty much prevents factories that make ball bearings or aluminum cans from making weapons of mass destruction or magic nanotech.

Don't forget Mickey in Fantasia. Those brooms were the paper clip factory.

 

[Jarvis323]

As in a vigorous or determined attempt. It takes time, energy, resources, etc. It's not trivial. An AI isn't going to build the entire infrastructure needed to manufacture and deploy nuclear weapons overnight or in a single warehouse someone forgot about. The same is true for any kind of advanced weapon or non-primitive technology.

I see your point. It might be worth reading Scott's other blog post.

https://scottaaronson.blog/?p=6821

 

[Drakkith]

I see your point. It might be worth reading Scott's other blog post.

https://scottaaronson.blog/?p=6821

Yes, this sort of goes with what I was saying. His "Reformer AI-Riskers" consider AI danger to be incremental and long-term, as opposed to a single 'incident' screwing everything up. I would broadly agree with this, though I think the overall risk of AI destroying us is extremely overblown. I think we'll run into some number of limited cases where AI screws something up, costing companies lots of money, and fueling safety-oriented research.