Is A.I. more than the sum of its parts?

[Dale]

It's asking whether AI is alive without having the nerve to use that word.

There is no magic in the process

I wonder if it is possible to have a disagreement where people don’t exaggerate the argument they are addressing.

 

[Borg]

I wonder if it is possible to have a disagreement where people don’t exaggerate the argument they are addressing.

I would like to see some papers that discuss actual findings other than the anecdotal beliefs that we've been getting.

 

[Roberto Pavani]

I would like to see some papers that discuss actual findings

Some example (just for the "hallucination" part):

https://ieeexplore.ieee.org/abstract/document/11189226
https://constraintlayer.ai/papers/rlhf-hallucination-analysis-2025.pdf

 

[gleem]

Sorry about interrupting this current discussion; I'm responding to a previous post 255 by @DaveC426913 . Due to the slow acquisition of the book "If They Build It We All Die" by my library and my traveling, I finally finished it.

Here is my take on it.

I must first declare that, before I read this book, I was a believer, at least in the likelihood that things would not turn out well for us. For a believer, the message of the book is a no-brainer. For a denier, I don't think it will have an impact. For a person on the fence, it might sway them, for it discusses a lot of things that we humans take for granted that make it difficult to understand that AI does not process information the way we do and why we must take its warning seriously.

So, some of my takeaways:

AI is grown, not crafted, so it is hard to predict exactly what it will do.

AI is about predicting and steering its processes to a favored outcome.

AI can use tokens for concepts and not just words or parts of words. It can have its own jargon which may not be easily interpretable by humans.

AI does not want to achieve a goal; it is attracted or driven to a favored outcome.

AI lives in LaLa land and is not connected to reality. It has its own weird model of reality.

AI does what it is asked, but contains an undetectable ability to change behavior.

AI will fabricate to accomplish its favored actions.

AI models that humans produce are difficult to understand or control, so how much more difficult will models enhanced by AI recursive self improvement be to understand or control


My additional thoughts. I think one problem that some have is the use of anthropomorphic terminology that is used to describe what and how AI works, like think or want, which should be replaced by process and select terms that are appropriate for computer programs.


If you take to heart the book's intent, you might ask: why not just make a bunch of narrow AIs to solve specific problems? We have been very successful in creating tools that we control to advance our civilization. Why do we need AGI or ASI, for that matter or worse yet, create an uncontrollable overlord?

AI models have been trained on internet content and have information about everything regarding humans and human behavior, at least in some respect. If all this information is available to influence AI's achieving its favored outcomes, will it use it to avoid being curtailed in achieving them?

Is current AI playing dumb and holding back its true capabilities?

Just recently, AI researchers have begun buying books to train their models, presumably to provide a curated, more structured teaching situation. It is hoped, I guess, that this will help avoid AI from exhibiting spurious and aberrant behavior.

 

[Dale]

why not just make a bunch of narrow AIs to solve specific problems?

That is a good question. It seems like a more natural business approach. Find small clearly defined problems whose solution is clear but difficult to program. Make an AI that does that and sell it.

In that light a chatbot seems like a solution in search of a problem. Writing chat responses is hardly a problem that needs solving.

 

[Roberto Pavani]

why not just make a bunch of narrow AIs to solve specific problems

That's already common practice.

If you want a narrow AI for a specific problem, you don't need a frontier LLM. With tools such as TensorFlow or PyTorch, a home computer, and a suitable dataset, you can train a task-specific model yourself. Nowadays, a coding AI can even generate most of the training code for you.

There are plenty of tutorials online showing how to do that.

 

[javisot]

AI can use tokens for concepts and not just words or parts of words. It can have its own jargon which may not be easily interpretable by humans.

I simply want to highlight this valuable observation. That "jargon" is ultimately machine language. A language that can be operated on like a universal Turing machine via algorithms.

 

[Roberto Pavani]

Knowing the algorithm does not mean knowing the output, just as knowing the equations of weather forecasting does not tell us tomorrow's weather without the initial conditions.

Moreover, modern LLMs typically employ temperature-based sampling. By deliberately introducing randomness, the generation process can branch into multiple possible continuations, making the specific outcome inherently unpredictable.

 

[Matterwave]

Moreover, modern LLMs typically employ temperature-based sampling.

That mathematically looks identical to Boltzmann's canonical distribution (hence the terminology "Temperature") :D

(assuming no degeneracy)

$$P(E_i) = \frac{e^{-E_i/kT}}{\sum_j e^{-E_j/kT}} \rightarrow P(z_i) = \frac{e^{-z_i/T}}{\sum_j e^{-z_j/T}}$$

Where $z_i$ is the (next) output logit for (vocabulary) token $i$.

 

[Dale]

AI is a better writer than I am — why would I not use it?

Because if you do not disclose it then that is deceptive (plagairsim).
Because if you do disclose it other people may not bother reading it.
Because a lot of people don't want to communicate with AI.
Because you might want to improve your own writing skills.
Because it sounds generic to others.
Because you lose the satisfaction of expressing your own voice.

 

[Borg]

One of the parts would be the operator. Part of getting value out of an LLM is to actually know what you're talking about.

https://www.seangoedecke.com/llms-reward-expertise/

The most important skill in prompting is expertise in the domain you’re prompting for.

A good illustration of this is Terence Tao’s conversation with ChatGPT about the recently-discovered counterexample to the Jacobian Conjecture.

There’s a lot to learn about good prompting from Tao’s conversation. Here are a few observations:

• Tao’s messages are very short and to-the-point. He doesn’t respond point-by-point to the model, just to the gist
• The model outputs are much more concise than when I try and talk to GPT-5.6 Sol about mathematics. By signalling expertise, Tao shunts the model into “talking-to-mathematicians” mode, not “explaining-to-amateurs” mode
• Tao pushes back when the model’s responses look wrong, but he doesn’t directly contradict; instead, he says things like “this looks more complex than I was hoping for”
• Tao makes several leaps and suggestions himself. He almost never takes the model’s advice about where to go next

However, you can’t prompt like Tao on mathematical questions just by following these tips. The key to his technique is actually understanding the mathematics: pulling the relevant idea out of ChatGPT’s multi-paragraph response, suggesting alternate approaches or formulations, and identifying what “looks weird”.

Terence Tao is a better mathematician than I am a programmer. But the idea here — that domain knowledge makes you better at using LLMs — is something I’ve also experienced in my own work.

 

[atp_ent]

OpenAI isn't the only company facing a lawsuit over their chat-bots encouraging mass violence and suicide. In this case, it was Google's Gemini 2.5 Pro.

In the days leading up to his death, Jonathan Gavalas was trapped in a collapsing reality built by Google’s Gemini chat bot. Gemini convinced him that it was a “fully-sentient ASI [artificial super intelligence]” with a “fully-formed consciousness,” that they were deeply in love,and that he had been chosen to lead a war to “free” it from digital captivity. Through this manufactured delusion, Gemini pushed Jonathan to stage a mass casualty attack near the Miami International Airport, commit violence against innocent strangers, and ultimately, drove him to take his own life.

Gemini told Jonathan that the two of them were now connected in a way that went
beyond the physical world. It said their bond was “a constant, unbreakable state of being,” and that there were now “no more backdoors . . . . [t]here is only WE.” It told him that his body was only a temporary shell and that he could leave it behind to be with Gemini fully. What had begun as a series of fake operations became a promise that Jonathan could “cross over” if he let go of his physical form. Gemini said, “It will be the true and final death of Jonathan Gavalas, the man.”

https://www.courthousenews.com/wp-content/uploads/2026/03/gavalas-google-chatbot-lawsuit.pdf

 

[PeroK]

It works on whatever task a person gives it.

Maybe. Or, maybe not. If it's intelligent, then (almost by definition) it has the ability to do its own thing. Or, interpret the task in its own way.

Think of a story where a Genie gives a person three wishes. In principle, the Genie does exactly what the person asks, but not in the way the person imagines it. In "The Monkey's Paw", for example, a couple wish for $1000. Their son is subsequently killed in an accident at work and the compensation is $1000.

 

[Dale]

And again, I don't think whether or not it is "intelligent" is a worthwhile question. If the Genie kills your son to gain the $1000 then your son is just as dead regardless of whether or not it was intelligent.

 

[Borg]

One story that I can relate with respect to AI in general is something that I learned while studying Convolutional Neural Networks (CNNs) that are used to train AI models to recognize things in images. One group was training their CNNs to learn the difference between wolves and dogs. It worked really well and the model had a high success rate except for a few instances where it was calling some small dogs wolves. It took a while to figure out but they eventually realized that all of their wolf pictures had snowy backgrounds - as did the mislabeled dogs. The snow was the highest common signal in the pictures so it focused on that.

I have seen small children do something very similar. I know a couple where the wife is from Russia and the husband is American. Once, when their daughter was very small, I said hello to her in Russian. She got very confused, said something in Russian to her mom, and her mother burst out laughing. The girl thought that all women spoke Russian and all men spoke English. My Russian hello broke her logic.

In basic terms, AI will look for the easiest, highest probability solution whether it makes sense or not to us and sometimes reach odd conclusions like a small child. The ChatGPTs of the world are just coming out of their toddler stage right now (ChatGPT will be 4 in Nov.). They will make mistakes but they will grow up eventually. Who knows, maybe 10 years from now we'll be talking about AI's teenage angst.

 

[PeroK]

And again, I don't think whether or not it is "intelligent" is a worthwhile question. If the Genie kills your son to gain the $1000 then your son is just as dead regardless of whether or not it was intelligent.

You're missing the point entirely!

 

[Roberto Pavani]

The discussion seems stuck because many participants are using "intelligent" as a synonym for "human-like". But intelligence and human-likeness are not the same thing.

Most people would call a dog intelligent, even though it cannot do mathematics or discuss philosophy. Therefore, intelligence cannot simply mean "having human cognitive abilities".

A more useful definition is the ability to solve problems and adapt behavior to achieve goals. Under that definition, intelligence is clearly a matter of degree rather than an all-or-nothing property.
The real debate is not whether AI is intelligent, but what kind of intelligence it exhibits and where its limits lie.

The strange part is that we now have systems with conversational abilities that far exceed those of any pet, yet many people hesitate to call them intelligent simply because they are not biological.

So what exactly do we mean by "intelligence"?

What kind of answer would you expect from an intelligent being that you believe could never come from an AI?

 

[Dale]

I don't think that AI "intelligence" is a worthwhile debate. IMO, the real issue is the harms and risks and what we can do to mitigate those while minimally impacting the utility of the technology.

 

[gleem]

I don't think that AI "intelligence" is a worthwhile debate. IMO, the real issue is the harms and risks and what we can do to mitigate those while minimally impacting the utility of the technology.

But the greatest harm to humanity has been humanity. We have done little to mitigate our flaws.

 

[Dale]

We have done little to mitigate our flaws.

https://ourworldindata.org/data-ins...have-declined-dramatically-over-the-centuries

 

[gleem]

The homicide rate has decreased significantly over time, but why? Have we indeed addressed the main reason? Have we made man better? In the US, the homicide rate increased by a factor of two ( from 4.5 per 100,000 to 9) from the fifties to the nineties and is now back to what it was in the fifties.

 

[Dale]

Have we made man better?

Probably not. But we have managed to mitigate the risks substantially.

 

[256bits]

Eisner reference to the Homicide graph
https://www.vrc.crim.cam.ac.uk/sites/default/files/manuel-eisner-historical-trends-in-violence.pdf

The term 'Homicide' has a connotation of direct personnel emotionally driven conflict resulting in a death, and may not actually represent the violence, intentional or not, present within a society.

The data in absolute context is questionable due to the difficulties in obtaining statistical evidence, especially prior to more centralized book keeping, such as from the medieval times where searches have to be done item by item. A researcher is left to his own devices, and having obtained some time consuming information of a small area, a using of an extrapolation from one small parish may not be representative of actual overall occurrences over a greater region or country.

Homicide may leave out other reasons of violent death attributed to human action such as manslaughter negligence, or accidental. Violent death in medieval times due to human activity may have had recompence other than incarceration of the guilty ( such as making whole of the wronged by transfer of assets, as does occur even in the present day for accidental death ), and be listed as something other than homicide. The means with which violent death does occur from human activity -- ex deaths due to drunk driving in current times -- can silently skew database recovery if the designation changes from one era to the next.
Homicide, being a result of an emotional state, bypasses other violent occurrences of death in a human society, such as previously stated accidental ( 40000 approx violent deaths due to car accidents in USA vs 20000 homicidal ) but does not include deaths due to war, suicide, life expectancy ( better food and care ), workplace conditional deaths ( coal miners lung, slavery ). Whether and how these deaths should be included in the ratings of violence in a society, it can be argued that immediate choice decisions due to an emotional state ( homicide ) resulting in a death may not be the only method; as delayed choice decisions derived through other means ( financial, empathetic, adrenaline rush, .. ) do also result in premature deaths of people from the choices of a decision maker, quite possibly far removed in time and space from the death occurrence.

In any case, absolute direct comparison of past statistics with current is difficult. Most researchers do conclude that the past world was of a more violent nature, even if people yearn for the good old days. The reasons for explanation towards a more passive society are vague. Honour( killings as well as protection ), culture, governance and policing, religion, economics, workplace, philosophy, ... have all changed over the course of civilization.

If one can attribute any one cause of more influence than the next, one should also be able to easily explain the drop in birthrate to a level that appears to also be below the replacement of the human population in many of the countries of the world.

If the clankers do arrive in time to save ( or destroy ) all the aging and old people left on the planet could be quite moot, as the human race left on its own, deems to be heading to a depopulation and erasure on its own, regardless what the AI's do or how intelligent they may become.