Is A.I. more than the sum of its parts?

[Sagittarius A-Star]

Business survey:
Almost 20% of companies in Germany already using artificial intelligence (AI) consider it easy or very easy to replace workers with (technical) degrees. To do that, they would rely on a worker without a degree who is supported by AI.

Source:
https://www.ifo.de/en/facts/2026-06...icial-intelligence-alternative-qualifications

 

[jack action]

To do that, they would rely on a worker without a degree who is supported by AI.

That is funny. I can imagine technical workers working with AI being more efficient, thus requiring less of them in the work force (or have them work less on technical issues).

But I cannot imagine the use of AI leading to unskill workers doing the job. How does one know the job is done appropriately?

Second, I find quite weird that it is considered essential to have a population with some general knowledge, such as geography, history, arts, etc. included in their degrees, but that "general" technical knowledge will become superflous stuff that nobody needs.

 

[Dale]

How does one know the job is done appropriately?

Indeed.

https://www.reuters.com/legal/litig...t-misused-ai-disqualifies-lawyers-2026-06-09/

 

[256bits]

Indeed.

https://www.reuters.com/legal/litig...t-misused-ai-disqualifies-lawyers-2026-06-09/

Reuters wants me to subscribe, but no problem as I had heard of this from another source.
In essence, it was 2 Chats arguing between themselves their side of the court case, with the briefings and the false references being produced by the LLM in separate sessions for each lawyer.

In other news,
Google may appeal their loss in court regarding search pre-ambles generated by AI. Google position is that since it comes from a user search it is really part of the search, with the the user responsible to check the veracity of the summary. Judge said, on the contrary, the pre-amble summary is new generated prose no where to be found in a reference, with Google AI being themselves responsible for errors and correctedness. Google is accountable for their imperfect product.

 

[256bits]

Second, I find quite weird that it is considered essential to have a population with some general knowledge, such as geography, history, arts, etc. included in their degrees, but that "general" technical knowledge will become superflous stuff that nobody needs.

The bate headline should have read " Of companies using AI, 80% find it difficult to impossible to replace workers."

 

[Dale]

Google AI being themselves responsible for errors and correctedness. Google is accountable for their imperfect product.

I think that is as it should be. Software engineers have used the title "engineer" for decades. This is one of the important things that engineers do. They take responsibility for the products they design.

 

[Sagittarius A-Star]

The bate headline should have read " Of companies using AI, 80% find it difficult to impossible to replace workers."

Another possible headline: " Of companies using AI, 74.7% find it possible to replace workers."

https://www.ifo.de/en/facts/2026-06...icial-intelligence-alternative-qualifications

 

[256bits]

Another possible headline: " Of companies using AI, 74.7% find it possible to replace workers."

https://www.ifo.de/en/facts/2026-06...icial-intelligence-alternative-qualifications

Of course there is that too as an interpretation of survey results.

The article by the author as written to substantiate a biased premise that companies ( surveyed ) are pro-AI, and are willing to view the results of AI output as being trustworthy enough to replace qualified workers. The premise, which may or may not be true, is that of the author, as her own interpretation of the results, without giving supporting information, such as how the survey was carried out ( example - the ratio of questionaires answered to number sent out, the size of the companies, who from the company replied - CEO, junior executive - , the ratio of companies questioned who have adopted AI to those who have not adopted AI and thus are not tallied in the results as seen in the graph, etc ).
She has committed a Type I Error in the heading and summary just by the way both have been written, and perhaps a naive company will rely upon her analysis to invest in AI, ending up with disastrous results. ( Would you take a drug that has been shown to have only 20% success. Would you invest in AI by replacing qualified employees with AI enhanced unqualified employees knowing the success is 20% ).

Secondly, the data is an invalid set for a company to rely upon for the direction any one company should take. The questionaire asks "Is it possible ... ", which is an attempt to predict the future. As written, it is unknown if the companies have or have not tried the replacement, indicating either success or failure in their response.
The author is attempting to replace 'Viewpoint', with 'Success' or lack thereof, as being a valid criteria.
Were the companies asked "Have you attempted to replace ... " upon which data could be obtained from past experience to portray pitfalls for any company contemplating AI adoption, rather than join the crowd as everyone else is doing it so we must too.

 

[sbrothy]

Doesn't it simply mean that there are different kinds of "intelligence"? Human intelligence, computer intelligence, canine intelligence, bee intelligence, octopi intelligence, etc.?

Mathematical intelligence, musical intelligence, 3d intelligence, motor intelligence, the list goes on imho....

 

[Planobilly]

I've been reading this thread with interest but haven't posted because until recently I didn't have the operational experience to say anything worth hearing. Now I do.

I'm 78, retired, physics and engineering background, several patents. I run a small private AI lab out of my home in Florida — an NVIDIA DGX Spark (128 GB, Grace Blackwell), local LLMs running on it (abliterated Qwen 72B via Ollama), and custom software I've built over the past year. I also run an independent daily news analysis site (digitalnewsusa.net) that uses AI in its production pipeline — multiple LLM sources cross-referenced against wire services, with human editorial oversight. Every day. In production. For months.

Here's what I've learned, and it's directly relevant to the last several posts in this thread.

AI is optimized to produce output that sounds correct. It is not optimized to be correct. These are not the same thing, and the difference is where every real-world failure comes from — including those lawyers submitting hallucinated citations. They didn't fail because AI is stupid. They failed because AI is convincing, and they didn't verify.

The survey about replacing degreed workers with AI-assisted non-degreed workers — 256bits is right to tear that apart. I've tested this exact scenario in practice. Here's what actually happens: AI can accelerate a competent person. It cannot make an incompetent person competent. If you don't already know enough to recognize when the AI is wrong, you will accept wrong answers with absolute confidence, because the AI delivers nonsense with the same fluency it delivers truth. There is no signal in the output that distinguishes the two.

I'll give you a concrete example from the other direction — where AI worked. I recently computed electrostatic properties for all 550,000 proteins in the AlphaFold database. Custom C# pipeline, AMBER force field, the full run on my Spark. The results are published on Zenodo (DOI: 10.5281/zenodo.20411754) and I posted them here on PF for expert review. That project would not have existed without AI — it helped me write the code, understand the biochemistry, and structure the analysis. But I had to verify every step. The AI got things wrong repeatedly. I caught it because I have enough physics background to know when a number doesn't make sense. Someone without that background would have published garbage and never known it.

That's the real answer to the thread question. Is AI more than the sum of its parts? Functionally, no. It is a very large, very fast pattern-matching system that produces output statistically similar to its training data. It does not understand what it produces. It cannot tell you when it's wrong. It has no mechanism for distinguishing truth from plausibility.

But it is genuinely useful as a tool — in the same way a calculator is useful. Nobody argues that a calculator understands mathematics. Nobody should argue that an LLM understands language. The danger is that it's convincing enough to fool people into thinking it does, and the companies selling it have a financial incentive to encourage that confusion.

jack action asked earlier in this thread: "How does one know the job is done appropriately?" That's the only question that matters, and nobody selling AI wants you to ask it.

Cheers,

Billy

 

[Sagittarius A-Star]

AI is optimized to produce output that sounds correct. It is not optimized to be correct.

That's a bad thing. Wouldn't it be possible to develop an AI which is optimized to be correct and to be honest enough to admid it, if it couln't answer the question, instead of hallucinating an answer?

 

[Planobilly]

Hi
You're asking the right question, but the answer is structural, not technical.

The reason AI isn't optimized for correctness is economics, not capability. Being correct requires verification — checking claims against sources, running calculations, admitting uncertainty. All of that costs compute time and money. Being plausible requires pattern matching, which is fast and cheap.

The business model for every major AI company depends on serving millions of users with fast, inexpensive responses. A system that stopped and said "I don't actually know" every time it hit uncertainty would be commercially unviable. It would lose to the competitor that confidently gives you a plausible-sounding answer in two seconds, even if that answer is wrong 20% of the time.

I run AI operationally every day — not as a casual user, but building systems that produce real output. I can tell you from direct experience that these systems will present fabricated source citations with the same confidence as verified facts. They will combine two different numbers into one because it makes the narrative flow better. They will tag unverified claims as "verified" because the training optimized for producing text that looks like authoritative reporting.

As a matter of fact, I am dealing with this exact issue right now. I run a small independent news analysis site that uses AI to generate intelligence briefings. This morning's briefing reported oil at $83 a barrel when the actual Brent crude price was over $90. It tagged Iranian state propaganda claims as "verified" with zero independent confirmation. It attributed information to a source that may not have published anything on the topic. I'm having to go through it line by line and fix it because the AI did exactly what we're talking about — it produced output that reads like professional journalism but falls apart under scrutiny.

Could you build one optimized for correctness? In theory, yes. But it would be slower, more expensive to run, and would refuse to answer a significant percentage of questions. No venture capital firm is funding that. No publicly traded company is going to ship a product that says "I don't know" when the competitor says "Here's your answer." The market selects for confidence, not accuracy.

This isn't a bug that will be fixed in the next version. It's the business model.

That said, some of these problems are solvable and some aren't. Here's what I've found from operational experience:

Fully solvable right now with engineering:

Knowledge cutoff / temporal blindness — RAG pipeline with news ingestion
Confident confabulation about facts — retrieval-augmented verification
Source authority confusion — ranked source hierarchy
Context window limitations — external memory, chunking, summarization
Session inconsistency — response logging and consistency checking
Failure to say "I don't know" — confidence calibration + retrieval
Sycophancy / agreement bias — system prompts + source verification
Mostly or partially solvable:

Mathematical errors — route to actual computation
Reasoning errors in multi-step logic — step-by-step with verification
Cannot verify its own outputs — tool integration
Poor spatial/physical reasoning — vision models, CAD integration
Quantitative reasoning limits — route numbers to real computation
Not solvable — fundamental architectural limitations:

No true physical understanding — the model works from text, not experience
No genuine causal reasoning — pattern matching, not physics
Symbol grounding problem — tokens are not connected to physical reality
Cannot recognize truly novel situations — can interpolate but not extrapolate
Emergent deception / goal misalignment — active research, no current solution
The solvable problems require engineering work that costs money and slows the system down. The unsolvable ones are baked into the architecture. Most companies aren't doing the engineering because the market doesn't punish them for getting it wrong — it punishes them for being slow.

I've been documenting all of this from an operational perspective at talking-about-ai.com.

As you can see from the above, public AI has many issues that are currently not being addressed, even if there is a solution. AI guesses at math question answers. It does not know the current time. There is a long list of issues that could be solved right now if the companies wanted to invest in the engineering. AI is no different than many software programs. Publish it with many issues and make the public find the problems.

— Billy

 

[Borg]

The standard way of doing RAG is very problematic. Vectorizing a large document might get you a document that is close in the vector space to the user's question but it may not have what's most relevant. Also, when you feed an entire document into a context window, it forces the LLM to figure out what's relevant and what isn't. That makes it ripe for hallucinations.

I use a different technique that only retrieves the parts of documents that are most likely to be relevant so that the LLM doesn't have to wade through so much unrelated information. Many of the documents that it retrieves are (using standard RAG) farther from the user's query but their important bits are the pieces that truly are closer in context. By only supplying (mostly) relevant and related information without all of the unrelated stuff, the LLM is able to really focus on applying that information to the user's question. It's far more accurate and allows me to use much smaller (8 GB) LLM models while outperforming larger models that use standard RAG.

 

[javisot]

No true physical understanding — the model works from text, not experience
No genuine causal reasoning — pattern matching, not physics
Symbol grounding problem — tokens are not connected to physical reality
Cannot recognize truly novel situations — can interpolate but not extrapolate
Emergent deception / goal misalignment — active research, no current solution
The solvable problems require engineering work that costs money and slows the system down. The unsolvable ones are baked into the architecture. Most companies aren't doing the engineering because the market doesn't punish them for getting it wrong — it punishes them for being slow.

Can an LLM without hallucinations be truly general?

 

[Planobilly]

Can an LLM without hallucinations be truly general?

Hi jovisot
That's a genuinely good question, and the answer is probably no. Hallucination isn't a bug — it's a byproduct of how the system generates text. The same mechanism that lets it generalize and produce novel responses is the same mechanism that produces confident nonsense. You can reduce it with engineering, but eliminating it entirely would probably make the model so conservative it stops being useful for general tasks.
The basic issue is that AI does not operate from first principles; everything is a guess.

Cheers,

Billy

 

[QuarkyMeson]

Planobilly, why are you using AI output to respond to these messages?

The "not that -> this" constant "framing" is 100% characteristic of AI. Just type yourself man.

This isn't LinkedIn.

 

[atp_ent]

Planobilly, why are you using AI output to respond to these messages?

The "not that -> this" constant "framing" is 100% characteristic of AI. Just type yourself man.

This isn't LinkedIn.

"You're one step from asking the people behind the answers — working scientists, mathematicians and engineers. Free to join. Real corrections and nuance, not autocomplete."

No AI here!

 

[danieltanfh95]

1. AI used to be trained on true mass input (reddit etc) on the belief that the wisdom of the crowds will prevail. That has largely turned out to be a myth, and AI is now post-trained and RL'ed with experts, aka local maximums of human competence in certain areas. Said local maximums do not represent global maximum of overall human competence, but they are vastly more competent than the majority of humans (proxy of PhDs vs humanity overall).
2. AI being transformers, handle mapping tasks very well. The gap we had before was structuring problem solving (a nebulous concept even in modern education) as a sequence of mapping steps.

Both of which is solved with human intervention. The human in the loop is not going away.

 

[PeroK]

The issue is not only what AI can do in 2026. The issue is the speed of progress.

The risk is real. And, this technology is developing at a time when, IMO, "government of the people" is under threat. Corporations and extremely wealthy individuals potentially have power beyond elected governments.

These two factors together make me concerned about the future.

If you think AI is not a threat, what if you are wrong?

 

[Dale]

The issue is not only what AI can do in 2026. The issue is the speed of progress.

The risk is real. And, this technology is developing at a time when, IMO, "government of the people" is under threat. Corporations and extremely wealthy individuals potentially have power beyond elected governments.

These two factors together make me concerned about the future.

If you think AI is not a threat, what if you are wrong?

I agree with this and note that the question of “intelligence” is largely irrelevant to this most important issue. It is also not limited to LLM’s nor even to AI agents.

It is far better to focus on the tangible questions of harms and risks than on less important and less defined philosophical questions.

 

[danieltanfh95]

The issue is not only what AI can do in 2026. The issue is the speed of progress.

The risk is real. And, this technology is developing at a time when, IMO, "government of the people" is under threat. Corporations and extremely wealthy individuals potentially have power beyond elected governments.

These two factors together make me concerned about the future.

If you think AI is not a threat, what if you are wrong?

AI is not the threat, humans are. So, deal with the humans.

The fragility of the systems that govern us and discussion in general is not going to go away just because we refuse to engage with the technology (AI, genetic engineering etc) that exposes the fragility.

 

[Dale]

AI is not the threat, humans are. So, deal with the humans.

No. This is the same as with any technology. You cannot manage the risks by wholly focusing on the humans nor by wholly focusing on the technology. Both must be dealt with.

 

[danieltanfh95]

No. This is the same as with any technology. You cannot manage the threats by wholly focusing on the humans nor by wholly focusing on the technology. Both must be dealt with.

What do you even mean by "dealing with the technology"? For example, if you mean to restrict the technology, that is an action you impose on humans, not on the technology. If you mean to decelerate or stop AI research, that is again, an action you impose on humans, not the technology. If you mean to add guardrails on AI, the humans have to be the one implementing the guardrails and sandboxes, which is again, an action you impose on humans first, that has cascading impact on technology.

 

[Dale]

if you mean to restrict the technology, that is an action you impose on humans, not on the technology. If you mean to decelerate or stop AI research, that is again, an action you impose on humans, not the technology. If you mean to add guardrails on AI, the humans have to be the one implementing the guardrails and sandboxes, which is again, an action you impose on humans first, that has cascading impact on technology.

Then this was a meaningless statement:

AI is not the threat, humans are. So, deal with the humans.

According to your followup, there is no such thing as dealing with the AI. Everything is dealing with humans.

What do you even mean by "dealing with the technology"?

What I mean is to do the same as we have with any other dangerous technology: Require safe standards for both the product itself as well as its use.

 

[PeroK]

AI is not the threat, humans are. So, deal with the humans.

The threat from AI comes from the technology. It needs specific action related to the specific threat.

 

[danieltanfh95]

Then this was a meaningless statement:

Yes, which is why I am calling out the idea that "AI is the threat" as a nothingburger, because the problem is on how the human individual, society or governments react or adapt to any sort of improvement to technology, we are lacking a certain philosophy or governance principle on humans itself, that we constantly go into the same loop of technology arrives -> a large portion of humans are scared -> some become luddites, some become advocates -> advocates win because productivity advances and humans enjoy a better life than before -> loop

 

[PeroK]

Yes, which is why I am calling out the idea that "AI is the threat" as a nothingburger, because the problem is on how the human individual, society or governments react or adapt to any sort of improvement to technology, we are lacking a certain philosophy or governance principle on humans itself, that we constantly go into the same loop of technology arrives -> a large portion of humans are scared -> some become luddites, some become advocates -> advocates win because productivity advances and humans enjoy a better life than before -> loop

So, you are saying there is no possibility that AI could be harmful. It's bound to be advantageous, because all previous technologies have been advantageous? There is no possibility that AI is different?

 

[Dale]

No, this is the meaningless statement (emphasis added):

AI is not the threat, humans are. So, deal with the humans.

It is not a meaningless statement to assert that AI is a threat, and it is also not a meaningless statement to assert that humans using or misusing AI is a threat.

What is meaningless is to state that "AI is not the threat" and then to categorize every possible threat and remedy related to AI as human. That is a vacuous statement because of your categorization.

 

[Dale]

So, you are saying there is no possibility that AI could be harmful. It's bound to be advantageous, because all previous technologies have been advantageous? There is no possibility that AI is different?

No, he is saying something even weaker. He is simply categorizing every possible AI harm as a human harm and every possible way of dealing with it as dealing with a human.

 

[danieltanfh95]

So, you are saying there is no possibility that AI could be harmful. It's bound to be advantageous, because all previous technologies have been advantageous? There is no possibility that AI is different?

I'm saying that it is irrelevant as to whether AI is a threat or not, because the problem is on humans, not the technology. This could be genetic engineering of food, solar panels or whatever fad people are up to.

Consider the Max Planck quote but applied to new technology (AI in this case):

A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it ...

An important scientific innovation rarely makes its way by gradually winning over and converting its opponents: it rarely happens that Saul becomes Paul What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning: another instance of the fact that the future lies with the youth.

My children will grow up with AI, learn to control AI, and wonder why the heck I am so conservative with it, and they would be right. Computers, electricity and aviation used to be eschewed as "dangerous to human society" and we have largely benefited from it.

And as such, we should apply the same principles that led us to open science and open software to AI: to educate the larger public and enable the larger public to use it so we can collectively figure out a way to live with it and benefit from it. Less than 5% of the US (if I recall correctly) has an AI subscription. The Chinese has a different number and they are reacting to AI with less uncertainty or chaos than what we are seeing outside of China, and I have good reason to suspect it is precisely because AI is close to free and openly available there.