Is A.I. more than the sum of its parts?

[PeroK]

No, he is saying something even weaker. He is simply categorizing every possible AI harm as a human harm and every possible way of dealing with it as dealing with a human.

It also misses the point that AI is, by definition, an attempt to create something that is on a level with human intelligence. This makes it potentially different because if AI is successful, then we have two types of high intelligence. And, one of the risks is that we end up in conflict with AI. And, at that stage, it is by definition no longer a human problem. That is one of the long-term risks. That humans are superseded by the AI that we have created.

 

[danieltanfh95]

It also misses the point that AI is, by definition, an attempt to create something that is on a level with human intelligence. This makes it potentially different because if AI is successful, then we have two types of high intelligence. And, one of the risks is that we end up in conflict with AI. And, at that stage, it is by definition no longer a human problem. That is one of the long-term risks. That humans are superseded by the AI that we have created.

I think that is largely a misconception, I believe, perhaps on the concept of intelligence, which reveals how little research we have on intelligence. I don't think AI is as intelligent as we think it is (otherwise why are humans involved in every major leap in capability), but we can't exactly pinpoint or explain why this is, which is why I mentioned that we are lacking a sort of philosophical guidance on how to proceed with technology in general.

 

[PeroK]

My children will grow up with AI, learn to control AI, and wonder why the heck I am so conservative with it, and they would be right.

If AI is more intelligent than your children, then it may learn how to control your children. A solar panel cannot do that. This is the whole debate. You appear to see humans as a supreme intelligence that cannot be outwitted. The debate must confront the risk that in the foreseeable future that is not the case. That the supreme intelligence on Earth (by whatever measure - perhaps survival) is no longer us, but the AI that we created.

This is part of the current debate.

 

[PeroK]

I think that is largely a misconception, I believe, perhaps on the concept of intelligence, which reveals how little research we have on intelligence. I don't think AI is as intelligent as we think it is (otherwise why are humans involved in every major leap in capability), but we can't exactly pinpoint or explain why this is, which is why I mentioned that we are lacking a sort of philosophical guidance on how to proceed with technology in general.

Are we to base our hopes on what you believe? What if you are wrong?

 

[danieltanfh95]

If AI is more intelligent than your children, then it may learn how to control your children. A solar panel cannot do that. This is the whole debate. You appear to see humans as a supreme intelligence that cannot be outwitted. The debate must confront the risk that in the foreseeable future that is not the case. That the supreme intelligence on Earth (by whatever measure - perhaps survival) is no longer us, but the AI that we created.

This is part of the current debate.

I think you missed the news where social media (or the internet/computers/videogames etc) was hypothesised to control our children, and the correct response was to regulate humans or enact rules for humans, instead of restricting the technology that connects humans together.

 

[Dale]

My children will grow up with AI, learn to control AI, and wonder why the heck I am so conservative with it, and they would be right. Computers, electricity and aviation used to be eschewed as "dangerous to human society" and we have largely benefited from it.

We have not benefited from electricity, aviation, or any major technology without recognizing the actual and potential risks and requiring things that mitigate the harms and amplify the benefits.

Also, I don't think that anyone today seriously looks at the pioneers of electricity and aviation and thinks "why the heck [were they] so conservative with it". I think that your assertion of your children's thoughts is merely a rhetorical device that is not at all plausible. You will make a more convincing argument if you avoid such rhetorical excesses. They just reduce your credibility.

 

[danieltanfh95]

Are we to base our hopes on what you believe? What if you are wrong?

Well I do think I have presented my case with sufficient empirical evidence to present an alternative view to elevate and educate humans so we transcend beyond "petty squabbles" on whether technology in general would be the end of humanity as we know it.

We have not benefited from electricity, aviation, or any major technology without recognizing the actual and potential risks and requiring things that mitigate the harms and amplify the benefits.

And these are regulations on humans, how we use and access these technology, not of the technology itself.

 

[danieltanfh95]

I think that your assertion of your children's thoughts is merely a rhetorical device that is not at all plausible.

I am really not sure why you think this is "not plausible". I myself grew up in a time where my parents thought the internet or the computer was harmful in general because it would change how I think. I have good reason to think that my children would have the same struggle (as in every technological advance we have had in human history), than to assume that history won't repeat itself.

 

[Dale]

It also misses the point that AI is, by definition, an attempt to create something that is on a level with human intelligence. This makes it potentially different because if AI is successful, then we have two types of high intelligence. And, one of the risks is that we end up in conflict with AI. And, at that stage, it is by definition no longer a human problem. That is one of the long-term risks. That humans are superseded by the AI that we have created.

AI does not need to have anything near human-level intelligence to cause harm. I think that it would be more productive to focus on the actual risks, without regard to whether or not AI is or will be "on a level with human intelligence".

 

[PeroK]

I am really not sure why you think this is "not plausible". I myself grew up in a time where my parents thought the internet or the computer was harmful in general because it would change how I think. I have good reason to think that my children would have the same struggle (as in every technological advance we have had in human history), than to assume that history won't repeat itself.

I think you have a very narrow view of history. When the Roman Empire collapsed, that was effectively the end of western civilization for nearly a thousand years. They were called the Dark Ages.

In any case, there is no law in history that says that human civilisation will endure indefinitely. And, you cannot reasonably appeal to historical precedent when it comes to the twin threats of AI and global climate change. These are unique threats - although climate change on a local scale has previously ended civilisations.

It's vacuous to say, effectively, that human civilization and technological advancement has come this far, so there is no stopping us.

 

[danieltanfh95]

I think you have a very narrow view of history. When the Roman Empire collapsed, that was effectively the end of western civilization for nearly a thousand years. They were called the Dark Ages.

In any case, there is no law in history that says that human civilisation will endure indefinitely. And, you cannot reasonably appeal to historical precedent when it comes to the twin threats of AI and global climate change. These are unique threats - although climate change on a local scale has previously ended civilisations.

It's vacuous to say, effectively, that human civilization and technological advancement has come this far, so there is no stopping us.

Perhaps it is similarly vacuous to say, as well, that this technology or situation is what would end us.

I would rather proceeding cautiously, but optimistically, with the closest philosophical guidance we have in technology, which is to proceed with openness, with open science and open software. I think what deepseek has been doing is the correct path: to share everything we have done to create these models clearly so it is auditable under the public's eye, instead of developing these model behind closed doors.

 

[Dale]

I have presented my case with sufficient empirical evidence to present an alternative view to elevate and educate humans so we transcend beyond "petty squabbles" on whether technology in general would be the end of humanity as we know it.

What are you talking about? Did you actually present any empirical evidence about the risks of AI? I missed it. I have not seen you present any empirical evidence about the risks of AI.

The question isn't about "technology in general", it is about specifically AI.

And these are regulations on humans, how we use and access these technology, not of the technology itself.

They are also regulations on the technology itself. I.e. specific features that the devices themselves must have. (unless you mean "not of the technology itself" in your meaningless sense). Electrical devices must have specific physical characteristics to reduce fire hazards. Airplanes must have a certain number of exit points. Those are regulations on the technology itself.

I would rather proceeding cautiously, but optimistically

You don't get cautiously with statements like "AI is not the threat".

 

[danieltanfh95]

What are you talking about? Did you actually present any empirical evidence about the risks of AI? I missed it. I have not seen you present any empirical evidence about the risks of AI.

The question isn't about "technology in general", it is about specifically AI.

I have already presented (obvious as it may be) the case that AI cannot replace humans, and humans remain crucial for any leap in capability of AI for the foreseeable future (this is evident from the case of initial data training to RL + post training, to improvements in the harnesses, to improvement on visual capabilities depending on advances in computer vision, all of which were human-led).

That alone ensures that humans will remain leading the development of the technology in the near future, and how we proceed from here is entirely controlled by how we govern humans.

What other risks associated with AI is specific to the technology of LLMs itself that you are thinking of? Please don't quote anything related to energy/water usage (Moore's law and algorithms will improve, we are already seeing improved token per watt scaling in GPT 5.5 and 5.6 and blackwells, not to mention deepseek showcasing in an investor call they can further reduce token costs).

 

[Dale]

I have already presented (obvious as it may be) the case that AI cannot replace humans, and humans remain crucial for any leap in capability of AI for the foreseeable future (this is evident from the case of initial data training to RL + post training, to improvements in the harnesses, to improvement on visual capabilities depending on advances in computer vision, all of which were human-led).

You made that claim. You did not present any empirical evidence supporting it. I have not seen ANY empirical evidence from you supporting ANY of your claims. Do you understand what empirical evidence is?

 

[gleem]

The issue is not only what AI can do in 2026. The issue is the speed of progress.

Consider these stats. In 2024, a new model was released every 60 days. In 2025, a new model was introduced every 50 days. Since March of this year, a new model has been introduced every 10 days. An extrapolation has been made, and at the current rate, we would have a new model every day by January 2027.

As far as the threat is concerned, if we do not find a way of slowing down AI's development so that we can understand it, it will make no difference who or what is responsible.

 

[gleem]

humans remain crucial for any leap in capability of AI for the foreseeable

Not so; even today, AI models are writing their own code, per Anthropic. What capability would Claude put in its code? We have no idea; we can't even predict the capabilities of human developed models.

 

[danieltanfh95]

You made that claim. You did not present any empirical evidence supporting it. I have not seen ANY empirical evidence from you supporting ANY of your claims. Do you understand what empirical evidence is?

Be as it may that it seems we have a whole lot of personal theories on the risk or capabilities of AI thrown around here, let me try (even though I know that the evidence chain below would be largely ignored by many, even if it was obvious to me if you do educate yourself on the topic before commenting) to enumerate a non-exhaustive list of evidence (of course, do note that due to the speed of development, a full control study won't be available).

On the statement that humans will remain in the loop for the foreseeable future:

METR’s assessment of frontier systems during February to March 2026 found that agents were already performing substantial technical work autonomously, sometimes with permissions and oversight resembling those given to human employees. But METR found no evidence that participating companies relied on agents to set research agendas, make security or budget decisions, make high-level risk judgments, or function as drop-in replacements for AI researchers. On one evaluation of whether proposed attack strategies would work, the strongest model scored 59%, whereas a human monitoring-and-control researcher scored about 90% on a subset. This directly supports a current distinction between technical execution and strategic judgment. (https://metr.org/blog/2026-05-19-frontier-risk-report/)

RE-Bench, published at ICML 2025, compared AI agents with 61 human machine-learning experts across seven open-ended research-engineering environments. The result was deliberately mixed: at a two-hour budget, the best agents scored four times as highly as the humans. At eight hours, humans narrowly overtook the agents; across 32 hours, humans achieved twice the top-agent score. Agents could generate and test solutions much faster, but their returns flattened earlier. (https://proceedings.mlr.press/v267/wijk25a.html)

Anthropic analyzed roughly 400,000 Claude Code sessions involving about 235,000 people between October 2025 and April 2026. Its classifier attributed approximately 70% of planning decisions to the human users but only about 20% of execution decisions to them; Claude therefore made roughly 80% of execution decisions. Greater task-specific domain expertise was also associated with higher success and longer autonomous action chains from the agent.(https://www.anthropic.com/research/claude-code-expertise)

The InstructGPT experiment provides a particularly clean intervention. The base and instruction-tuned models used the same GPT-3 architecture, but InstructGPT was additionally trained with human-written demonstrations and human preference comparisons. The 1.3-billion-parameter InstructGPT model was preferred to the 175-billion-parameter base GPT-3 despite having more than 100 times fewer parameters. The 175-billion-parameter InstructGPT model was preferred to the equivalent GPT-3 model 85% of the time, and its closed-domain hallucination rate was 21%, compared with 41% for GPT-3.(https://ar5iv.labs.arxiv.org/html/2203.02155)

A 2026 controlled preprint held models fixed while varying their execution harnesses (the surrounding software that manages tools, context, retries, verification, and recovery). On a 100-task SWE-bench Verified subset, changing the harness moved individual model scores by 8.5–13 percentage points, while changing among the three comparable models within one harness moved scores by only 2.5–5 points. Six of nine model-pair rankings reversed under different harnesses. (https://arxiv.org/html/2605.23950v1)

In a report published on July 21, 2026, METR compared autonomous agents and human contributors on the NanoGPT speedrun. After more than $10,000 in agent expenditure, estimated “expenditure horizons” (the budget below which an agent was more cost-effective than human labour) ranged from $0 to approximately $3,300. Revalidation found that some apparent gains were merely statistical noise; newer models made genuine improvements, but these remained modest relative to cumulative human optimization, and agent changes tended toward narrow hyperparameter tuning while human contributions were more often architectural or broadly generalizable. (https://metr.org/blog/2026-07-21-expenditure-horizon/)

Across real-world training and usage of AI shows that humans are firmly required as part of the process in spite of advances in LLM capabilities in the near future.

 

[danieltanfh95]

While it is beyond my ability to suggest how to moderate the thread, I do note that we seem to afford a much wider range of speculation and emotional ranting in these threads than we do in general physics.

 

[Roberto Pavani]

An interesting test would be to get this thread and provide it to a large AI and look the answer...

 

[Roberto Pavani]

Perhaps the discussion would benefit from a stronger focus on proposed tests rather than on definitions.

A first, admittedly naive, experiment would simply be to use the entire text of this thread as input to a modern LLM and evaluate the quality and coherence of its analysis. The result would not settle the question, but it would provide an interesting empirical data point.

 

[PeroK]

I have already presented (obvious as it may be) the case that AI cannot replace humans, and humans remain crucial for any leap in capability of AI for the foreseeable future (this is evident from the case of initial data training to RL + post training, to improvements in the harnesses, to improvement on visual capabilities depending on advances in computer vision, all of which were human-led).

Then why are leading experts in the field worried about AI capability? It's mad to think that you can prove anything of the sort. It's like proving that nuclear war is impossible. Or, proving that global warming can never exceed +1.5C.

Something like that cannot be obvious. It cannot be proved. Instead, we are in a risk assessment phase. We have an asymmetric debate (very similar to the climate change debate over the past 40 years or so). On the one hand, many people claim there is a risk and point to evidence of that risk. Others, like you, point to evidence that the thing is an impossibility. But, given there is evidence of a risk, then there is a risk. And, any proof that the thing is impossible must be spurious. If the thing was impossible, no one would worry.

This is why I have little patience with your argument. It might be that AGI is 50 years away, but that is somewhat irrelevant. The question is whether AGI (as many experts believe) is much closer than that. It's pointless to protest that the experts are all wrong. That itself is an extraordinary position. The experts might be wrong - but that's a different matter.

 

[danieltanfh95]

Then why are leading experts in the field worried about AI capability? It's mad to think that you can prove anything of the sort. It's like proving that nuclear war is impossible. Or, proving that global warming can never exceed +1.5C.

Something like that cannot be obvious. It cannot be proved. Instead, we are in a risk assessment phase. We have an asymmetric debate (very similar to the climate change debate over the past 40 years or so). On the one hand, many people claim there is a risk and point to evidence of that risk. Others, like you, point to evidence that the thing is an impossibility. But, given there is evidence of a risk, then there is a risk. And, any proof that the thing is impossible must be spurious. If the thing was impossible, no one would worry.

This is why I have little patience with your argument. It might be that AGI is 50 years away, but that is somewhat irrelevant. The question is whether AGI (as many experts believe) is much closer than that. It's pointless to protest that the experts are all wrong. That itself is an extraordinary position. The experts might be wrong - but that's a different matter.

1. Your so called leading experts on the field are? What interests do they serve? Why is wider Asia (including Russia, Japan, Korea, South East Asia, China, the Middle East, and India) all not as worried with AI, even though nearly half (my memory may fail me, might be a lot more) of the AI researchers come from this geographic region?
2. Your claim that AI is a threat cannot be proved. Also, so far it has remained a personal theory that is only tolerated (by me, I can't speak for others) because it represents real emotions people have of this technology, and I want to have a earnest discussion about it, but I am not a therapist nor do I want to be one.

 

[PeroK]

1. Your so called leading experts on the field are? What interests do they serve?

I've never believed such general conspiracy theories. Climate change, COVID, Flat Earth, AI. It's all the same spurious argument that the "experts" are conspiring for some unknown reason. Many are independent computer scientists presenting the risks as they see them.

In any case, as a scientific site, we are supposed to be led by the scientific research. Not by conspiracy theories.

 

[yjqwh]

Given that neuroscience cannot definitively explain where consciousness originates - whether as an emergent property of the brain or something transcendent - is it be possible for complex chatbots to develop behaviours that can’t be predicted by simply analyzing the sum of their parts?

The discussion about emergence is fascinating. If all phenomena in nature ultimately originate from fundamental physical rules, the boundary between predictable behavior and emergent novelty deserves deeper exploration. I have been researching the fundamental nature of light and matter transformation, which may provide a new perspective on how complex structures arise from basic entities.

 

[Roberto Pavani]

To answer the OP: yes, AI is more than the sum of its parts.

The complexity of the whole is not just the sum of the complexity of its individual parts. Interactions between components can create emergent behaviors that cannot be understood by analyzing each part separately.

Regarding the "AI risk":
I think the biggest risk depends on what we train a neural network to do.

If you train a neural network to provide answers based on the entirety of human knowledge, you get one outcome.

If you train a neural network to identify and shoot anything that moves and then self-destruct after N shots, you get a very different outcome.

If we fear that an AI of the first type will decide to destroy us, despite the fact that we provide its power supply, hardware maintenance, and overall infrastructure, then we're probably assuming it is not actually smarter than we are.

Personally, I'm more afraid of natural intelligence than artificial intelligence.

 

[Dale]

So, it seems that your AI research bot took a while to complete your research for you. BTW, please read our rules on posting AI output, particularly without attribution.

It is good that you have now provided some empirical evidence for your claim. Now, one thing that is common in science (and less common elsewhere) is to look at the evidence skeptically. This is particularly important when you are using AI to do your research for you.

In particular, the claim that the empirical evidence is supposed to support is:

AI cannot replace humans, and humans remain crucial for any leap in capability of AI for the foreseeable future

First,

(https://metr.org/blog/2026-05-19-frontier-risk-report/)

I am pretty surprised that you posted this as evidence supporting your position. In their summary they say

"Overall, we believe that internal agents at the time of our assessment plausibly had the means, motive, and opportunity to start small rogue deployments, but they did not have the means to make them highly robust." So, AI agents can already go "rogue", they just don't do so very robustly. I would say that "forseeable future" includes a future where already existing capabilities become more robust.

Their figure 4, in particular, indicates that the capability of AI models increases between 8- and 16-fold per year and they already handle tasks that take humans >16 hours. So it is foreseeable that within two years, AI should be handling tasks that take a human half a year of full time work.

So a more accurate statement would be "AI cannot yet replace humans for tasks that take humans more than about 100 hours". This explicitly and empirically includes tasks involving using and modifying other AI models.

(https://proceedings.mlr.press/v267/wijk25a.html)

This one is interesting. I agree that it supports the idea that, as of 2025, there was limit of about 8 hours on the complexity of tasks that can be handled by AI, specifically in the context of R&D of new AI. However, they also did show that AI could replace humans in the development of AI for tasks of 2 hours or less. 2 hours is not a lot, but it is also not a blanket "cannot replace". And certainly, the foreseeable future would include a steady increase in that time horizon.

(https://www.anthropic.com/research/claude-code-expertise)

Here 70% of the planning decisions were attributed to humans, which means that AI did 30% of the planning decisions. That is not a majority of the decisions, but also does not support a "cannot replace" claim. It did replace the human in 30% of the empirically observed planning decisions.

(https://ar5iv.labs.arxiv.org/html/2203.02155)

I am not clear why you thought this one was relevant.

(https://arxiv.org/html/2605.23950v1)

Same here

(https://metr.org/blog/2026-07-21-expenditure-horizon/)

Of all your empirical evidence, this one seemed the most credible and supportive of your claim. It is also relatively narrow, dealing specifically with model optimization tasks. I was surprised because this type of optimization problem is a type of problem that I would have assumed that AI would be well suited to.

So overall, your empirical evidence is not non-existent, but it is not particularly strong either.

 

[danieltanfh95]

So, it seems that your AI research bot took a while to complete your research for you. BTW, please read our rules on posting AI output, particularly without attribution.

I find it somewhat endearing that you are accusing me of just copy pasting AI when we are in a claude-designed website. The physicsforums redesign bears many marks of AI design like monospace font, the choice of colors, and the little ui details that make it distinct from a human-designed website.

And I don't see a huge uproar over it, if anything it seems that people are pretty happy with the AI designed website.

So a more accurate statement would be "AI cannot yet replace humans for tasks that take humans more than about 100 hours".

...

Sure? I mentioned that humans will be part of the loop but did not claim that humans will retain their original roles.

I am not clear why you thought this one was relevant.

On the other hand I am perplexed why you didn't think this and the next one is relevant unless you did not bother to read beyond the abstract or don't understand the topic due to a technical knowledge barrier, to which I have added an layman explanation in my previous post.

 

[AlexB23]

I have already presented (obvious as it may be) the case that AI cannot replace humans, and humans remain crucial for any leap in capability of AI for the foreseeable future (this is evident from the case of initial data training to RL + post training, to improvements in the harnesses, to improvement on visual capabilities depending on advances in computer vision, all of which were human-led).

That alone ensures that humans will remain leading the development of the technology in the near future, and how we proceed from here is entirely controlled by how we govern humans.

What other risks associated with AI is specific to the technology of LLMs itself that you are thinking of? Please don't quote anything related to energy/water usage (Moore's law and algorithms will improve, we are already seeing improved token per watt scaling in GPT 5.5 and 5.6 and blackwells, not to mention deepseek showcasing in an investor call they can further reduce token costs).

AI is pretty good nowadays at doing math, but we as a society would need a few more years to determine if it can do original scientific research or not. Have tested Gemma-4-26B-A4B (a self-hosted LLM) on my CPU, and it can solve some advanced math from 2026. It breaks down for other problems. Tested it with ciphers, and the model struggles.

 

[danieltanfh95]

I've never believed such general conspiracy theories. Climate change, COVID, Flat Earth, AI. It's all the same spurious argument that the "experts" are conspiring for some unknown reason. Many are independent computer scientists presenting the risks as they see them.

In any case, as a scientific site, we are supposed to be led by the scientific research. Not by conspiracy theories.

This is not a conspiracy theory, and it is directly being discussed by the present US government, in bad faith, like a claim that Kimi trained a world class model via distillation in the matter of weeks, when a full training run would make nearly a year.



In fact, we could trace most of the "AI-is-bad" conversation around the same circle of folk commonly known as "Effective Altruism" that proclaims that AI is dangerous thus we need to have a closed monopoly on them. If you are interested, feel free to search around the topic, you may find a lot of your views parroted from their articles and essays propagated around the internet.



It is unfortunate that unless I actively keep a record of all the nonsensical statements made by these folk, I could not keep up with conversations like these without being asked for a citation or proof that I did not make this all up, since I am not parroting the mainstream view, whereas a unfounded mainstream consensus would escape such scrutiny.

 

[Dale]

I find it somewhat endearing that you are accusing me of just copy pasting AI when we are in a claude-designed website.

So are you saying that you did not use AI to do the research or write the summaries?