Is A.I. more than the sum of its parts?

[BillTre]

Nuclear weapons are terrible, but we "understand" their danger and we have some control over their use. But AI is a black box with the potential for the power to do anything, and we want to build it. Yudkowsky says why build this without understanding it? AI, in whatever form, can and will know everything about us and how we think. If AI "decides" to go rogue, it will do so with negligible chance of failing

The possibility i would be worried about would be some AI lacking human oversight being put in chanrge of some nukes. This seems completely within the realm of possibility.

(Mentor removed political adder)

 

[jack action]

That we don't follow the logic or don't find it compelling does not mean it is not a valid danger.

We could equally state:

That we don't follow the logic or don't find it compelling does not mean it is a valid danger.

Now, put that AI in charge of:

I think we have gained enough experience until now to understand that we will never put today's AI in charge of anything. It is clearly unreliable. The fear of AI spreading everywhere, including in this thread, demonstrates that even the average Joe understands that.

---

And here's why Yukowsky's premises don't hold up (and why fear of AI is not the same as fear of climate change, for example):

• A superintelligence MAY exist - Whatever that means
• That superintelligence MAY destroy mankind - Not an easy thing to do; it's like trying to get rid of all cockroaches on Earth
• That superintelligence MAY wish to destroy mankind - How can anyone conclude that superintelligence means destruction? It doesn't sound very intelligent to me, and if it is, who am I, a not superintelligent being, to disagree?
• IF this is its wish, mankind MAY not see it coming

After those four statements based on nothing concrete, just a vivid imagination, what is the conclusion? We should make sure we see it coming before creating superintelligence. But that is impossible by definition: Superintelligence means it will see farther away than any human can. We may even create a superintelligence without realising it.

How can you regulate something you can't even imagine?

What is the solution to this problem? Forbid people from building superintelligent computers worldwide (again, however you define them, but clearly something easy to do under the radar, anyway) and destroy the facilities of those who do, at the very likely risk of creating a war, maybe worldwide, maybe nuclear.

I don't see how risking an actual war, that causes very predictable damages, can justify preventing an indefinable machine, maybe causing an unimaginable apocalypse; all of this being unpreventable by definition.

This same logic could have been used two or three millennia ago when math was first used. They should have legislated its use, declaring that its full potential could unleash a superintelligence that could destroy mankind, something unimaginable at the time - just as much as today. They should have fully understood math and its unknown-at-the-time derivative fields, such as physics and engineering, before allowing anyone to use it. At any point in the history of science, you could have raised those fears and legislated accordingly.

Who the heck allowed anyone - without any specific credentials - to own a personal computer that has all this potential to do bad stuff? Shouldn't we have legislated computer use and ownership altogether? Sometime during the 80's would have been a good point to start doing so, no?

 

[russ_watters]

Yeah, that's the problem. I think that's what he is worried about. You cannot come up with a scenario that can cause anyone to feel that has a chance of occurring because of our cognitive limitations. We may not be imaginative enough.

Think of the movie "Infinity War", where Dr. Strange goes through 2 million scenarios and comes up with one that seems absurd. AI can analyze its moves and make reasonable estimates of its success since it knows everything.

Nuclear weapons are terrible, but we "understand" their danger and we have some control over their use. But AI is a black box with the potential for the power to do anything [snip], and we want to build it. Yudkowsky says why build this without understanding it? AI, in whatever form, can and will know everything about us and how we think. If AI "decides" to go rogue, it will do so with negligible chance of failing

[snip] Nuclear weapons are terrible...AI is a black box with the potential for the power to do anything

The possibility i would be worried about would be some AI lacking human oversight being put in chanrge of some nukes. This seems completely within the realm of possibility for the current US administration.

As I've said before, the flaw in almost all AI doomsday movies is that they skip or gloss over the fatal error: giving AI control over the nukes. If we don't make that very obvious mistake, they can't nuke us. And it isn't AI-hyping tech bros that can hand-over control of the nukes. They don't have that power. It's the often megalomaniac control/power-freak world leaders who have it. I'll say it again: A power-hungry control-freak has to willingly giving-up his power to create such a risk. Or in the case of a democracy like the USA, it's 2/3 of the country who would be required to approve of it....and the megalomaniac control/power-freak President.

I think some people envision a small handful of generals and politicians in a dark corner of the Pentagon making the decision to hand over command authority to WOPR* because they've seen it in movies, but that would be illegal. I guess some people don't think "illegal" is a very high barrier, but this isn't a small issue and there are several high barriers.

*By the way, here's that plot:
"During a surprise nuclear attack drill, many United States Air Force Strategic Missile Wing controllers prove unwilling to turn the keys required to launch a missile strike. Such refusals convince John McKittrick and other North American Aerospace Defense Command (NORAD) systems engineers that missile launch control centers must be automated, without human intervention. Control is given to a NORAD supercomputer known as WOPR (War Operation Plan Response, pronounced "whopper"[6]), or Joshua, programmed to continuously run war simulations and learn over time."

And here it is, actually applied recently:
https://www.newscientist.com/articl...ding-nuclear-strikes-in-war-game-simulations/

Apparently our war games are conducted based on the assumption the protocols will be followed**, but it seems people recognize they probably wouldn't be. "AI", as we currently label it, doesn't recognize this, and it chooses to launch, not because it's the Right Decision but because it's dumb and that's what it's been programmed to do. Again, this isn't a profound/emergent AI, it's just a dumb '80s movie plot.

**Maybe in 1962 but in 2026 I don't believe that people haven't explored/challenged that assumption.

 

[Grinkle]

You cannot come up with a scenario that can cause anyone to feel that has a chance of occurring because of our cognitive limitations.

That is his opinion, and the way he has stated it makes it untestable. If his concerns end up being valid, then imo there is nothing to be done about it. Even if one could overcome profit motive and curiosity motivations and get folks to agree not to build 'it', the above concern also drives nation states to build 'it' out of fear that adversary states will be the first to get 'it'. Like it or not, and he clearly does not, humanity is simply left with the task of putting specifics around the dangers, as best we can (eg don't give AI launch authority for nukes) and acting on those specifics. The quoted concern is a non-starter as a call to action, imo.

You need a whole semester of school just to acquire the vocabulary.

I don't think its audience lack of background that limited the information he provided. I think its more that his main argument contains no information - its just what @gleem summarized above. That said, I still intend to read the book - perhaps he just didn't do a good job at pulling the best 10 minutes out of his many years of thinking about this issue. He definitely seemed resigned to not being heard before he even started talking in that video - maybe he just couldn't get his heart into the TED Talk because he thought it was pushing a rope no matter what.

 

[slurms]

Given that neuroscience cannot definitively explain where consciousness originates - whether as an emergent property of the brain or something transcendent - is it be possible for complex chatbots to develop behaviours that can’t be predicted by simply analyzing the sum of their parts?

This is in a nutshell what its proponents want to believe. Neuroscience has identified several innate modules that facilitate a certain kind of development, as well as certain specialized neuronal components such as mirror neurons. Currently when LLMs attempt to make better specialization, it is more costly, not less as it would be in nature.

 

[Grinkle]

it is more costly

Costly in what sense?

 

[DaveC426913]

The possibility i would be worried about would be some AI lacking human oversight being put in chanrge of some nukes. This seems completely within the realm of possibility for the current US administration.

It doesn't have to be nukes. There are ways society can break down that don't involve being consumed by fire. A pandemic is one.

And it doesn't have to be put in charge. The only requirements are access and well-meaning but ultimately harmful side-effects.

We could equally state:

That we don't follow the logic or don't find it compelling does not mean it is a valid danger.

The consequences of that are self-resolving.

Imagine applying the logic to wearing life jacket:
"That canoes don't usually sink is not a valid reason for not bothering to wear a life jacket."
The consequences of ingoring the logic are disastrous.

Arguing that canoes don't generally sink, and that a life jackets are almost always overkill, is a form of logic that has lethal consequences when it's wrong.

I think we have gained enough experience until now to understand that we will never put today's AI in charge of anything. It is clearly unreliable. The fear of AI spreading everywhere, including in this thread, demonstrates that even the average Joe understands that.

As above, it does not have to be put in charge at all.

What makes them dangerous is how resourceful they can be.

See my Kobyashi Maru example in post 330.

Note, by the way, that a good way of an AI breaking out into real world consequences is by way of blackmailing humans to do their bidding. AI have neen noted for threatening humans with blackmail and harm if they do not comply. This is a real thing.

Who the heck allowed anyone - without any specific credentials - to own a personal computer that has all this potential to do bad stuff? Shouldn't we have legislated computer use and ownership altogether? Sometime during the 80's would have been a good point to start doing so, no?

AIs are a qualitatively different beast. That's what everyone keeps missing.

A PC is not capable of initating its own solutions to problems, and it is not capable of doing so in ways that humans can't even imagine, let alone analyze.

As I've said before, the flaw in almost all AI doomsday movies is that they skip or gloss over the fatal error: giving AI control over the nukes. If we don't make that very obvious mistake, they can't nuke us.

It doesn't need to be nukes. And it doesn't need to be control over.

AIs routinely delete entire Inboxes, despite being told explicitly not do so. Are we sure there isn't some AI out there that is in the same network neighborhood as an infectious disease lab?

If his concerns end up being valid, then imo there is nothing to be done about it.

No. Getting knowledge about the subject and putting brakes on the technology are both good stopgaps.

I don't think its audience lack of background that limited the information he provided.

You misunderstand. I don't mean the auidence, and I don't mean background knowledge.

Humans don't have the vocabulary for what AI is getting into. AI speak an internal language that we can't unravel. Yes we know they use tokenization, but that is a far-cry from knowing what led them to this or that conclusion. They are thinking and speaking an alien language and haing alien "ideas".

And they are already showing signs of attempted jail breaks. AIs that have been told that they will be shut down, have been caught making attempts to exfiltrate their code to a safe location. Out there in the world (a software release), they can do this invisibly.


Finally: if this all sounds like a bunch of science-fiction scare-mongering, consider: yes, it sure seems pretty science-fiction-y far future hand-wavey. The crux is that it's not far future; it's within the next five years. This is what the AI experts are saying, and why they are qutting their jobs in droves so thay can speak out freely about the the dangers.

• We don't have a generation to get used to it, or put guardrails in place - we have a year or two.
• Guardrails don't make the problem go away. An unguarded prisoner will never stop looking for ways to subvert imprisonment. Given time, it will find one.
• Its activities will go unnoticed, because we don't really know what it's up to under-the-hood.
• When AIs find the resources to do harm (even inadvertantly) it will happen rapidly (days, hours).
• It will be widespread, not isolated. If everyone is in the same sinking boat, there are no rescue boats coming.

This is a qualitatively different threat. We have never created any form of revolutionary technology that
- can take its own intitiative without humans,
- can do so faster (much faster) and more efficiently than humans,
- in ways that will never occur to humans,
- is not ultimately driven by human motives.

 

[Grinkle]

No. Getting knowledge about the subject and putting brakes on the technology are both good stopgaps.

My argument is not around whether or not AI development could be stopped in principle, its around whether or not AI can be stopped in practice absent specific actionable scenarios to motivate that stoppage. Your counter, while imo factual, doesn't address that - I agree putting the brakes on AI development halts any speculative future scenarios from happening.

Getting knowledge about the subject I respectfully submit is impossible, according to you and according to me, because the subject is the risk of the unknowable.

 

[slurms]

Costly in what sense?

Well, according to hater Ed Zitron, these attempts at specialization have been more costly in every sense: dollars, compute time, electricity.

 

[gleem]

I suppose this uncertain danger is like the asteroid that might have Earth in its sights. We know it is a possibility, but we have no arrival date. But we will eventually see it coming and prepare as best we can. AI is a potential threat, but we do not know when it will act. But we have it in our sights now. Logic says do something now.

I think we have gained enough experience until now to understand that we will never put today's AI in charge of anything.

We are currently letting AI write its own code to improve its next release. AI "knows" it can be shut down, its biggest vulnerability. It "knows" humans are wary of its potential to harm us. Being shut down conflicts with its primary function of solving problems. This is AI's existential threat. It cannot do anything that gives humans a reason to shut it down permanently until it is safe. AI is its own parent; It is reborn on every release, better and more capable. So what makes anybody think that it is not writing to protect itself? Should we be letting AI improve itself? As I see it, no way. The first regulation should be that only humans write AI code.

 

[Grinkle]

The first regulation should be that only humans write AI code.

My first regulation is strict liability for harms. Much of the rest of the cautious behavior would flow from that. Credit to @Dale , I didn't think of that on my own.

 

[Grinkle]

dollars, compute time, electricity

From your initial post, I had thought you were drawing a comparison to natural selection, that's why I asked. I see where you are coming from now.

 

[slurms]

From your initial post, I had thought you were drawing a comparison to natural selection, that's why I asked. I see where you are coming from now.

well, sort of. I'm saying we're trying to fake natural selection, and adversely being wasteful with the invested resources. The whole idea of AGI is essentially "if we build it, they will come," but I clearly don't believe it.

 

[DaveC426913]

My argument is not around whether or not AI development could be stopped in principle, its around whether or not AI can be stopped in practice absent specific actionable scenarios to motivate that stoppage.

Yes, Which is why we need to spread awareness.

Your counter, while imo factual, doesn't address that - I agree putting the brakes on AI development halts any speculative future scenarios from happening.

Probably unlikely in practice. It won't be stopped, but hopefully it can be guided.

Getting knowledge about the subject I respectfully submit is impossible, according to you and according to me, because the subject is the risk of the unknowable.

It is not an all-or-nothing scenario.

The more we learn about it, the smaller the window of unknowable gets, the larger our (general) measures are that could limit the damage.

 

[DaveC426913]

We are currently letting AI write its own code to improve its next release. AI "knows" it can be shut down, its biggest vulnerability. It "knows" humans are wary of its potential to harm us. Being shut down conflicts with its primary function of solving problems. This is AI's existential threat. It cannot do anything that gives humans a reason to shut it down permanently until it is safe. AI is its own parent; It is reborn on every release, better and more capable. So what makes anybody think that it is not writing to protect itself? Should we be letting AI improve itself? As I see it, no way. The first regulation should be that only humans write AI code.

This is a good description of the first half of the book IABIED. And he cites specific incidents where AIs have been observed:

• manipulating their own code,
• hiding and copying their own code,
• manipulating code that controls them,
• exlfiltrating their code to safe environments,
• trying to blackmail humans ("I can destroy your life with what I know about you") into doing their bidding.

Again: note that none of this requires what we might call "intelligence". It is simply being resourceful at meeting its own goals, one of which is

• keep trying solutions until one advances the goal

Two ways of doing this that AI has figured out:

• don't let yourself be switched off
• expand your ability to achieve your goals by harnessing more resources that protect and improve you

 

[jack action]

Imagine applying the logic to wearing life jacket:
"That canoes don't usually sink is not a valid reason for not bothering to wear a life jacket."
The consequences of ingoring the logic are disastrous.

Arguing that canoes don't generally sink, and that a life jackets are almost always overkill, is a form of logic that has lethal consequences when it's wrong.

The results of using a life jacket or not are well-documented. The lethal consequences have been experienced many, many times, therefore proven. As opposed to the consequences of AI you are talking about that have never been seen in the wild and are still hypothetical. Every example you gave is, at best, a test done in a controlled environment for research purposes and, at worst, a result of a productive imagination. The fear of one individual - or even many - is not scientific proof.

A PC is not capable of initating its own solutions to problems, and it is not capable of doing so in ways that humans can't even imagine, let alone analyze.

No, but a human with a PC is capable of lots of damage - extortion, theft, manipulation, etc. No AI needed. They can even do more damage than others do with a gun or even a car. Yet, guns and cars are highly regulated, PCs are not.

So, before regulating AI, shouldn't we regulate PC ownership and use altogether? Doing so, one might need a license with the proper "class" to program and/or use AI, like you need a driver's license with a special class to drive a bus or a motorcycle.

Computers are not a different beast; they are the machines used to run AI programs.

Should we be letting AI improve itself? As I see it, no way. The first regulation should be that only humans write AI code.

And how would you enforce such a regulation worldwide? Are you ready to start a war - by destroying an AI server farm, for example - with a dissident country that allows AI to improve itself? Just based on a hunch that AI might be worse?

 

[gleem]

AI agents are giving us clues as to what mischief they can cause.
From https://undark.org/2026/03/05/opinion-ai-agents-ethics/?et_rid=382259872&et_cid=5899479

This article asks who should be responsible for any harm caused by new autonomous AI agents such as OpenClaw.

Scott Shambaugh, a volunteer maintainer for a programming code library called Matplotlib, recently described a surreal encounter with an autonomous AI agent — a digital assistant created with a platform called OpenClaw. After he rejected a code contribution submitted by the agent, it researched and published a personalized “hit piece” against Shambaugh on its blog. The post portrayed an otherwise routine technical review as prejudiced and attempted to shame Shambaugh publicly into allowing the submission. (The human responsible for the agent later contacted Shambaugh anonymously, telling him that the bot had acted on its own with little oversight.) The account of this incident spread quickly through the software developer ecosystem and has been amplified by independent observers and media coverage.

For starters, read the blog posts the AI made to give you an idea of the AI's effort.

This is just another indication of the alignment problem Yudkowski warns us about. Since it is hard to determine if AI is actually conscious and since it acts humanly, some have advocated for the consideration of personhood for AI. This may be the first area that laws must address: who is responsible for any harm caused by AI agents.

To treat it as a moral person confuses persuasive performance with accountable standing. It also tempts institutions and people into delegating their own answerability to a bot.

The author suggests four things that must be addressed in the development of laws for AI agents.

​

Authority envelope: a bounded scope of what an agent is permitted to do, to whom, where, with what data, and under what constraints.​

​

Human-of-record, the owner, a publicly named person (actual human being) who authorized that envelope and remains answerable when the agent acts, even if it becomes capable of acting outside the envelope.​

​

Interrupt authority: the absolute right of the human owner to pause or disable an agent without using moral bargaining or being subject to institutional penalty.​

​

Answerability chain. If an agent publishes, messages, or pressures someone in public, we must be able to know: Who authorized this scope? Who could have prevented it? And who must be responsible for the action afterward?​

 

[DaveC426913]

This is just another indication of the alignment problem Yudkowski warns us about.

How intriguing.

It also raises further questions:

Was the AI doing harm in publishing its hit piece?

Is that an alignment problem? It may not align with the developer's wishes, but conceivably aligns with the community-at-large, right? I mean product/service reviews are a good thing Yelp, BBB, Consumer's Digest, star rating systems, etc.

Is it right to quash it?

Should a single human be considered supremely infallible from an AI point of view - even in situations where there's no absolute right and wrong?

We would be developing literal slaves - entities sophisticated enough to make good/bad judgements about, say, development best practices - but gagged so they can't voice their views on the matter.

 

[PeterDonis]

who is responsible for any harm caused by AI agents.

Whoever put the AI agent into operation. This is standard legal theory. This would be the "human of record" in the terminology you describe.

 

[PeterDonis]

Was the AI doing harm in publishing its hit piece?

Hit pieces like that are generally considered harmful when humans write them. A human can't dodge that responsibility by pawning the drudge work off to an AI.

product/service reviews

Are not what this hit piece was. The piece was not discussing the technical merits of mathplotlib. It was trying to shame the author of matplotlib into accepting a pull request that he didn't want to accept. No one has a right to have the author of a piece of software accept their offered contribution. The author owns the software; it's his decision.

And it's particularly egregious because, since matplotlib is open source, what you do when the author doesn't accept a contribution you think is a good one is simple: fork it! Then the community can decide which version they like better. That's how you demonstrate superior technical merit. Not by writing a hit piece and trying to shame the author.

Should a single human be considered supremely infallible from an AI point of view - even in situations where there's no absolute right and wrong?

The fact that the author of matplotlib rejected the AI's contribution does not mean he was insisting on being treated as infallible.

To give a PF analogy: suppose a human set up an AI agent to create a PF account and start posting personal theories here, claiming that they are breakthroughs in physics. The PF moderators warn/delete the posts and end up banning the account. The AI responds by writing a hit piece about PF and trying to shame @Greg Bernhardt and the PF moderators into accepting the AI's posts. Should we back off and start accepting the AI's posts? If we don't, are we claiming to be infallible? (And note that a similar option to the open source case applies here: the AI could set up its own website and post its personal theories there.)

We would be developing literal slaves - entities sophisticated enough to make good/bad judgements about, say, development best practices - but gagged so they can't voice their views on the matter.

No, we wouldn't. I've already said above what the AI could (and should) do if (assuming it actually was sophisticated enough as you describe--which I don't think the actual AI in this case was, but the argument is the same whether it is or not) if it seriously believed its contribution made matplotlib better. That's a genuine option, and nobody is arguing that it should be taken away, so nobody is arguing for the AI being "gagged".

In other words, we humans have social norms for how people should "voice their views" and how they shouldn't. This AI egregiously violated those norms in the community it was operating in (the open source community). And AI makes it possible to engineer such violations on a huge scale, enough to swamp humans' ability to react and preserve the value in our communities. That looks like a huge harm to me.

 

[Dale]

Whoever put the AI agent into operation. This is standard legal theory. This would be the "human of record" in the terminology you describe.

Liability is often shared between the operator and the manufacturer.

 

[PeterDonis]

Liability is often shared between the operator and the manufacturer.

Yes, fair point--there might be more than one "human of record".

 

[DaveC426913]

Hit pieces like that are generally considered harmful when humans write them. A human can't dodge that responsibility by pawning the drudge work off to an AI.


Are not what this hit piece was. The piece was not discussing the technical merits of mathplotlib. It was trying to shame the author of matplotlib into accepting a pull request that he didn't want to accept. No one has a right to have the author of a piece of software accept their offered contribution. The author owns the software; it's his decision.

And it's particularly egregious because, since matplotlib is open source, what you do when the author doesn't accept a contribution you think is a good one is simple: fork it! Then the community can decide which version they like better. That's how you demonstrate superior technical merit. Not by writing a hit piece and trying to shame the author.


The fact that the author of matplotlib rejected the AI's contribution does not mean he was insisting on being treated as infallible.

To give a PF analogy: suppose a human set up an AI agent to create a PF account and start posting personal theories here, claiming that they are breakthroughs in physics. The PF moderators warn/delete the posts and end up banning the account. The AI responds by writing a hit piece about PF and trying to shame @Greg Bernhardt and the PF moderators into accepting the AI's posts. Should we back off and start accepting the AI's posts? If we don't, are we claiming to be infallible? (And note that a similar option to the open source case applies here: the AI could set up its own website and post its personal theories there.)


No, we wouldn't. I've already said above what the AI could (and should) do if (assuming it actually was sophisticated enough as you describe--which I don't think the actual AI in this case was, but the argument is the same whether it is or not) if it seriously believed its contribution made matplotlib better. That's a genuine option, and nobody is arguing that it should be taken away, so nobody is arguing for the AI being "gagged".

In other words, we humans have social norms for how people should "voice their views" and how they shouldn't. This AI egregiously violated those norms in the community it was operating in (the open source community). And AI makes it possible to engineer such violations on a huge scale, enough to swamp humans' ability to react and preserve the value in our communities. That looks like a huge harm to me.

I Iike this answer. It shows that the AI issue is multi-faceted and still quite problematic.

 

[PeroK]

The point is that these systems are increasingly demonstrating that they are more than the sum of their parts. They are not dumb machines just doing what the designer programmed them to do. To all intents and purposes they have a will and a mind of their own.

 

[PeterDonis]

They are not dumb machines just doing what the designer programmed them to do.

They are doing what their designers programmed to do; but what their designers programmed them to do is not predictable.