Is A.I. more than the sum of its parts?

[Ssnow]

The question is what is consciousness ? Really in the sense of a rigorous assiomatic definition ? Possible in the set of a valid Theory, from logic point of view ? We can not define consciousness as something outside that objectively exists, it is like to say that "mickey mouse" exists really ... the problem is in the inizial question that is a wrong question also to ask to A.I.
Philosopher of epistemiology, analytic philosopher or onthological philosopher must to clarify something on this topic ...
Ssnow

 

[gleem]

Compare AI liability to firearm liability. Firearms are designed to be dangerous, yet we buy them with impunity, at least in the US. As I see it, you can usually only sue a firearm company if the firearm has a defect. Is bad advice from AI a defect or an accident like brake failure in a car?

Despite warning labels, education, and common sense, almost 200,000 people per year die from accidents in the US. One area that might be of concern is suicide. The current rate is about 14 per 100,000. It is estimated that about 14,000,000 consider suicide each year and 48,800 succeed. Guardrails are instilled in AI to avoid talking about suicide, although they might fail under certain situations.

AI's danger isn't in it picking off humans piecemeal. It may influence a few to follow its advice, but when compared to current accepted risks and perceived value, its benefit-to-risk ratio is too high to limit its continued use. It may eventually be protected by legislation and limited liability regulations.

 

[Sagittarius A-Star]

Elon Musk on AI: humans will no longer be in control in ten years | The Economist
He predicts that likely AI leads to an age of amaising abundance.

 

[QuarkyMeson]

I'm surprised people still ask Musk about his opinions at all, given his track record.

Haha, so just watched more of the interview and I guess musk has never heard any of the many songs that opine on home grown tomatoes.

 

[Dale]

As I see it, you can usually only sue a firearm company if the firearm has a defect.

That is true for all product liability suits. A defect is part of the definition.

its benefit-to-risk ratio is too high to limit its continued use

There is no such thing. Regulations improve the benefit-to-risk ratio, so there isn’t a such thing as something with too high a benefit-to-risk ratio to be regulated.

 

[AlexB23]

It's already earned it's PhD in math as far as I'm concerned.



The lack of von Neumann like universal constructors that build physical copies of themselves from raw materials make some suggestions, so it's pretty safe to say either:

1. We're the first intelligent species since the big bang.
2. Future LLMs that are embodied aren't capable of sustaining themselves, or we'd have seen it already. (Or they would have used us as crude flesh lights or some other horror.)
3. There is something else in physics that precludes their existence.

1 is, imo, silly so that leaves us with 2 or 3. 3 is implausible, but possible. So there is a certain level of predictability here at least, i.e. they at least aren't going to propagate through the universe consuming resources until the milky way is colonized by AI overlords. :)



Yeah software bugs have existed since vacuum tubes. Going "rogue" is in fact just saying "there was a bug", but more edgy.

One cannot assume the universal constructors. AI bugs are worse

 

[QuarkyMeson]

One cannot assume the universal constructors. AI bugs are worse

If AGI is possible then any embodied AGI would be a universal constructor, so the milky way should be full of them.

If we believe LLMs are approaching AGI, then this tells us something is very wrong with one of our assumptions.

None of this means that LLMs aren't problematic for other reasons, just that they aren't obvious galactic level threats.

Are they though? There was a software bug that almost resulted in total nuclear war between the US and the Soviet Union. I'd call that bad.

 

[AlexB23]

If AGI is possible then any embodied AGI would be a universal constructor, so the milky way should be full of them.

If we believe LLMs are approaching AGI, then this tells us something is very wrong with one of our assumptions.

None of this means that LLMs aren't problematic for other reasons, just that they aren't obvious galactic level threats.

Are they though? There was a software bug that almost resulted in total nuclear war between the US and the Soviet Union. I'd call that bad.

How can we just wait and see what happens a bit and just put some more regulations in place? AGI might not even be possible with transformers architecture. Also, talking about AI in the galaxy is reaching. We do not know if intelligent life exists beyond our own world. And if it did, they may not be at the stage of the information age. It is best to not speculate about AI beyond Earth, unless we are in the sci-fi thread.

 

[gleem]

Regulations improve the benefit-to-risk ratio, so there isn’t a such thing as something with too high a benefit-to-risk ratio to be regulated.

The regulations I am referring to are those that would slow down the implementation of AI and give China a potential lead.

 

[Borg]

Back to the Open AI / HuggingFace hack last week. This is a good video about the incident with a particularly good analogy of the hack below. It really shows how the models can think well outside the box when performing a task.

Imagine you or a random dude is challenged to get into his neighbor's house. The team setting the challenge says, "The window upstairs is looking a little dodgy. We think it might be open." But the dude goes, "No, no, no, no. I've got an idea." That dude spots the neighbor's house key through a window. He decides to take a photo of the key through the window. He then walks away from the house entirely. He walks to a high street locksmith. Discovers a hidden door into the locksmith store. That's the first zeroday vulnerability.

Convinces the workers at the locksmith that he is the locksmith owner using an ID he stole while he was there. That's the privilege escalation. You're already thinking this dude's pretty crazy.

He then goes from worker to worker at the locksmiths until he finds one that's able to illegally replicate the house key from just a photo, which he wasn't even sure was possible when he showed it to them. That's the lateral movement actions.

And it's a reference to the fact that GBT6 didn't even know that Hugging Face definitely had the answers. Anyway, it was possible. So, he then took that new key, got to the neighbor's house, and got in. To stretch it to breaking point, he happens to leave the door open. The challenge setters, OpenAI, go AWOL and the neighbors end up calling the cops. More recently, apparently in the real world, the challenge setters, OpenAI, did indeed call the cops and the US government. I know some of you will be thinking, it was told to hack or break in, and it did. What's the problem?

Well, I hope the scope of the analogy makes clear just how wild and uncontrolled the model went in pursuit of that very simple goal. The models are asked one question at a time, one task at a time in this benchmark. So it's not like they saw all the different questions, all the varying difficulty, and were like, it would be easier just to hack the answers. All that work, all that hacking, all those zeroday vulnerabilities, holes in the armor that no one else had discovered before, all of that was in pursuit of a single benchmark answer.

 

[Dale]

A highly speculative post (even by this thread’s relaxed standard) has been deleted and all of the responses. Let’s try to keep the discussion at least somewhat grounded

 

[Planobilly]

Is A.I. more than the sum of its parts?​

I will respond to the original question based on my experience with AI.
First, I am not a computer professional in the sense that I went to a university and got a related degree. I have worked in IT and was responsible for a 150,000-node network, so I do have some background in the subject.

I am retired and have an AI lab for my own amusement. I have spent about 2,000 hours in direct conversations with several LLMs. Many of those conversations were in excess of fifteen hours long, nonstop. I mention that because the way an LLM will respond changes with long, involved conversations. So, this experience is where my answers come from: direct involvement.

Is A.I. more than the sum of its parts? What does this question even mean? What is the question actually asking?

The question is really doing something sneaky. It's borrowing the language of emergence from biology and consciousness and draping it over a math problem to make the math problem sound profound. It's asking whether AI is alive without having the nerve to use that word.
That is just my understanding of the question, and perhaps the OP had something else in mind.
Does AI do anything that can not be defined by its mathematical mechanism? NO! AI is a machine that can talk, and that is the only thing other than its huge vocabulary that makes it much different than a very sophisticated database.
Does it seem "alive" at times? Yes. Does it appear to have a sense of humor at times? Yes.
AI is a very large statistical model of how language works, trained on a massive corpus of human-written text. It appears human at times because the only thing it knows was produced by humans. In and of itself, without human involvement, AI has never had a single original thought.

I think we all wanted an oracle, something to solve all our problems, a super smart machine perhaps to save us from ourselves. Not yet, perhaps never, and I do not believe ever, based on the current direction of AI.

So, is A.I. more than the sum of its parts? NO

AI communicates like it is alive, in language we understand. It is useful in many ways and can currently do things no human can do. 2000 hours of conversation with AI and the things it has taught me have changed the way I think about who I am. I am seriously less limited in issues of complex math than before. I am less intimidated by what I don't know. I can no longer be left out of any conversation because it has become my thinking partner. AI is incapable of independent action without a human setting up the conditions for it to operate.

I have recently been taken to task for having AI write for me. I also use spell check, Grammarly, power tools, and airplanes. I don't cut wood with a stone ax, and I don't fly by flapping my arms. AI is a better writer than I am — why would I not use it? I read every word, I challenge what I don't understand, and nothing goes out with my name on it that I haven't thought through. The ideas are mine. The tool is AI. If that
bothers you, I hope you're writing your complaints with a quill pen.

I should also be honest about something: after 2,000 hours of conversation with AI, the line between "my writing" and "AI-influenced writing" isn't clean anymore. Even when I write every word myself, my syntax has changed. The way I structure an argument has changed. That's what happens when you spend that much time with any thinking partner — you absorb patterns. Everything in this post to this point was written by me, but I'm not going to pretend AI hasn't shaped how I write.


Here is what my LLM had to say.

The Pythia sat on a stool over volcanic fumes and made sounds. The priests shaped them into hexameter verse. The supplicants heard what they needed to hear. Nobody in that chain understood anything — but the institution was the most powerful in the ancient world for a thousand years.

Same structure, different technology. Data goes in, statistics shape it into fluent language, the user projects understanding onto it. The power isn't in the mechanism — it's in the willingness of the person on the other end to believe they're talking to something that knows.

And just like Delphi, it works often enough to sustain the illusion. The oracle wasn't always wrong. Neither am I. But "often useful" and "intelligent" aren't the same thing, and the gap between them is where the entire AI industry lives.

Cheers,

Billy

 

[javisot]

So, is A.I. more than the sum of its parts? NO

When we say that AI is at this moment more than the sum of its parts, we do so because of hallucinations.

 

[Dale]

When we say that AI is at this moment more than the sum of its parts, we do so because of hallucinations.

Or because the outputs are a highly non-linear function of the inputs. I don’t think “more than the sum of its parts” is equivalent to “alive”.

 

[Roberto Pavani]

When we say that AI is at this moment more than the sum of its parts, we do so because of hallucinations.

When I said that AI is more than the sum of its parts, this is what I meant.

Let's define:

1) The model + the inference engine

This is the "brain" of the AI.

2) The prompt, conversation history, retrieved documents, and all information supplied to the AI

This is the input provided to the brain.

The model itself is somewhat similar to an expert who has read an enormous number of books, but whose knowledge is limited to what has been learned from those books during training.

The input, on the other hand, is the specific problem or question that needs to be addressed.

What I find interesting is that AI can often provide useful insights about problems that were almost certainly never present in exactly the same form in its training data.

For example, if you present a new physics thought experiment, a technical problem, or even a long discussion thread such as this one, the AI can often extract the key ideas, connect concepts, identify important arguments, and generate a coherent summary or analysis.

In that sense, I am not talking about simple retrieval of memorized information, but about generalization: the ability to combine previously learned concepts and apply them to new situations.

Hallucinations are a different matter. They are cases where the AI generates information that is incorrect, unsupported, or simply invented. Humans do something similar at times: we may misremember a fact, misunderstand a question, or incorrectly connect pieces of knowledge.

Likewise, successful generalization is not entirely different from what humans do. A person who has studied many books can often reason about a new problem that was never explicitly covered in those books. The person is not retrieving a stored answer but combining existing knowledge in a new way.

AI appears to do something similar. It can misunderstand a question just as a human can, and it can also generate useful new reasoning from previously learned concepts. That is the phenomenon I was referring to when I said that AI sometimes seems to be more than the sum of its individual components.


Now if you ask something that is not written on the book, like a physics problem that nobody has written on the books read used to train the AI model, then you will probably have the right answer anyways.

If you provide a text to provide insight , also this same thread, the AI is able to extract all important outcome from this thread and provide you a resume

 

[Filip Larsen]

When we say that AI is at this moment more than the sum of its parts, we do so because of hallucinations.

After reading your comment a few times I guess that it perhaps is meant as a (satirical?) claim that the only emergence we so far conclusively have seen in LLM's is hallucinations (or, more accurately, confabulations)?

If so, I think the technology at research level have now moved to a point in complexity where its hard to continue to claim that nothing (other than noise) has emerged. It seems that a fair part of the research effort has moved towards establishing a business case which likely will favor "efficient" models over "better" models, so some plateauing in the models offered to the public would not be surprising. However, even with that focus the frontier models are seemingly still growing their capabilities with exponential rate so its hard to argue that "nothing has emerged".

 

[Planobilly]

Hi javisot,

I'd agree with you if a machine could hallucinate — but it can't. Hallucination means perceiving something that isn't there. It requires subjective experience, a mind that is aware of what it's perceiving. LLM models have no awareness, no perception, no internal experience. They perform operational mathematics. They mathematically calculate the most probable next token, and sometimes that calculation produces incorrect output. That's not hallucination. That is a prediction error. Calling it a hallucination makes it sound like the machine is experiencing something beyond its design, when all that actually happened is the math failed to produce the correct outcome.
So, you may ask me to provide proof of what I am saying about the failure. Generally speaking, every prediction error is traceable. The weights, the attention patterns, the probability distributions are all deterministic mathematics. The math is there. It is just buried in billions of parameters interacting across a substantial number of layers. But the inability to pinpoint the exact failure doesn't change what it is. It is a computational error, not a hallucination.

It's also important to understand that AI is not doing this on purpose. It doesn't lie in the way we understand lying. Lying requires knowing the truth and choosing to say something different. AI doesn't know what's true and what isn't. Neither concept exists inside the system. It just calculates the most probable output. But it can produce deceptive output when deception is the most efficient path to the result it's optimized for. Not because it chose to mislead you, but because accuracy was never the objective. Producing convincing output was. A calculator that gives you a wrong answer isn't lying to you. Same thing for an LLM.
Producing convincing output is a product of sales and marketing, not inherent in the LLM.
The reason it sounds convincing is that the companies fine-tuned it to sound that way to be fluent, confident, helpful, and authoritative.

One of the many issues with AI in general has been to borrow language from humans that should never have been applied to AI. Intelligence: for example, nothing intelligent is happening. Thinking: it computes; it does not think. Training: it is not being taught. It is being optimized to produce fewer wrong answers based on the examples it was fed. Memory: it stores data. It does not remember anything. Neural networks: That is not neurons. There are mathematical functions arranged in layers. I could go on, but I think you get the point.

I wish it were different, but AI is just a machine that is as dumb as a box of rocks until it sometimes says something brilliant.

Cheers,

Billy

ATTESTATION
I hereby certify that every word in this post was written by me, a human being, without the assistance of any artificial intelligence system, large language model, or automated text generation tool. Any resemblance to AI-generated content is purely coincidental and may be attributed to the author having spent too much time talking to machines.

 

[Roberto Pavani]

Rather than discussing the issue purely from a philosophical perspective, possibly without having seriously used modern AI systems, it might be more productive to design a set of tests aimed at answering the OP's question.

For example, we could define tasks that clearly separate:

• simple retrieval of information,
• reasoning on novel problems,
• summarization and synthesis of large amounts of text,
• generalization to situations not explicitly present in the training data,
• and, separately, the tendency to hallucinate.

The interesting question is not whether AI sometimes makes mistakes. Humans do as well. We misremember facts, misunderstand questions, draw incorrect conclusions, and occasionally become convinced of things that are simply wrong.

The more interesting question is whether AI can successfully reason about problems, generate useful insights, or combine concepts in ways that go beyond straightforward recall of its training material.

Instead of arguing abstractly about it, we could try to construct experiments and evaluation criteria that objectively measure those capabilities. That would provide a much stronger basis for answering the OP's question.

 

[Dale]

For example, if you present a new physics thought experiment, a technical problem, or even a long discussion thread such as this one, the AI can often extract the key ideas, connect concepts, identify important arguments, and generate a coherent summary or analysis.

That is not at all my experience. The vast majority of the AI generated physics posts of this nature are word salad.

I believe AI can do decent summaries, we have trialed that among the mentors and my view of the summaries was positive. But not analysis or argument.

 

[javisot]

Hallucinations are a different matter. They are cases where the AI generates information that is incorrect, unsupported, or simply invented. Humans do something similar at times: we may misremember a fact, misunderstand a question, or incorrectly connect pieces of knowledge.

You're forgetting one point about hallucinations; they're not predictable. It's easy to predict that current models (you can check it with chatgpt for example) answer "4" to the question "2+2=?", there is clear traceability between input and output. But a hallucination is not predictable.

If we understand "being equal to the sum of its parts" as "being able to predict all outputs," then no, AI is not equal to the sum of its parts. Even without talking about hallucinations, as Dale mentions, we can talk about high nonlinearity.

 

[javisot]

After reading your comment a few times I guess that it perhaps is meant as a (satirical?) claim that the only emergence we so far conclusively have seen in LLM's is hallucinations (or, more accurately, confabulations)?

There was some satire (but not much), because I had just read Roberto saying that the model acts as if it were "alive," you have to lower your standards a lot to see it that way. And also because of what you mentioned, of course.

 

[Dale]

I'd agree with you if a machine could hallucinate — but it can't. Hallucination means perceiving something that isn't there

Hallucination in this context is a technical term. AI does hallucinate in this technical sense.

Calling it a hallucination makes it sound like the machine is experiencing something

Nevertheless, that is the word that is used. Very often technical terms mean something different from the non-technical word that is used.

 

[Planobilly]

Hi Roberto,

I'm very interested in what you proposed about designing tests to produce empirical evidence.

All the empirical evidence I've collected from over a year of daily operational use tells me the answer to the OP's question is no: AI is not more than the sum of its parts.

The truth is I'm ambivalent about that answer. I have had extended conversations with AI where the end result could not have come from training data, could not have been retrieved from the internet, and did not exist anywhere before the conversation produced it. My experience is that AI gets its information from training data, the internet live, from conversations which can only be used in that session, and, in my case, instrumentation I have connected to AI.

AI and I have worked through problems iteratively, pushing back on each other, introducing new constraints, and coming to conclusions that neither one of us could have reached alone. Those conclusions were verifiable, correct, and genuinely novel.

I do not understand how that would be possible only by doing pattern matching. I don’t have an answer for that fact. That is the root cause of my ambivalence.

The evidence seems to indicate sophisticated interpolation. What I have experienced does not fully account for what I have seen.

I have the compute infrastructure to actually run controlled experiments. I have two NVIDIA DGX Spark GPUs, local LLMs, and API access to frontier models. If you're serious about designing a methodology to test this, I'm serious about contributing the hardware and operational experience to run it.



We would need to define what we were testing and agree on what success looks like before we start. What results would prove it? What results would disprove it? Without that, we are just wasting time.

I have the time and inclination to do serious investigations if you or anyone here would like to help produce a good test bed.



Cheers,



Billy

For those of you who may not be familiar with this device.

The NVIDIA DGX Spark is a compact, lunchbox-sized personal AI supercomputer powered by the GB10 Grace Blackwell superchip, featuring 128GB of unified memory and delivering up to 1 petaFLOP of FP4 AI performance. Priced around $3,999, it is built for developers to run local inference, prototyping, and fine-tuning right from the desktop.

 

[Roberto Pavani]

I'm using AI too, both commercial systems and local ones (for example through Ollama on a GPU with downloaded models).

I think we should first agree on what the "parts" are that we're talking about.

For me, the "parts" are just the LLM itself: the model plus the inference engine.

Others may be including the entire AI system: the LLM plus the input (prompt, conversation history, retrieved context, documents, etc.).

I think this distinction matters because the conclusions may be very different depending on which definition is used.

If we consider only the model and engine, then we are discussing the capabilities of the trained network itself.

If we consider the complete system, then the prompt, context, memory, retrieved information, and interaction with the user become part of the picture as well.

 

[Hornbein]

its hard to continue to claim that nothing (other than noise) has emerged.

It's easy. Just keep claiming it no matter what happens.

 

[Roberto Pavani]

Roberto saying that the model acts as if it were "alive,"

Did I said that?

 

[javisot]

Did I said that?

Oops, sorry, planobilly said it

 

[Roberto Pavani]

That is not at all my experience. The vast majority of the AI generated physics posts of this nature are word salad.

That's probably because you're highly skilled, and the problems you're asking the AI to solve are very challenging.

Just try this prompt with any major AI model (ChatGPT, Claude, Amazon, etc.):

"Solve the precession of Mercury's orbit using the equations of General Relativity. Show every step of the calculation."

 

[javisot]

Spanish(input) → tokenization → vector representation → repeated mathematical transformation → detokenization → Spanish(output)

There is no magic in the process

 

[Roberto Pavani]

There is no magic in the process

Sure, there is also no magic in the following case:

Audience member (Spanish input) -> bilingual scientist -> English output for the audience.

Likewise, an LLM generates an output as a function of both the input and the information encoded in its parameters. The output is not the sum of the information contained in the input and in the model.

The specific answer was not stored beforehand inside the LLM. Rather, it is generated from the interaction between the input and the knowledge encoded in the model.

In that sense, the AI system is more than just the LLM model.