Can ChatGPT Handle Complex Physics Questions Accurately?

[martinbn]

Me: how would you explain "quantum spookiness" with an every day analogy?

ChatGPT:
Quantum mechanics is a fundamental theory in physics that describes the behavior of matter and energy at the atomic and subatomic scale. One of the most striking and counterintuitive aspects of quantum mechanics is the idea of "quantum superposition," in which a quantum system can exist in multiple states simultaneously. This concept is often referred to as "quantum spookiness."

To understand quantum superposition, it might be helpful to consider the following everyday analogy: imagine you are playing a game of "rock, paper, scissors" with a friend. At any given moment, your hand can be in one of three states: rock, paper, or scissors. But in the world of quantum mechanics, a quantum system can be in multiple states at the same time. It's as if your hand could be both rock and scissors simultaneously, and only when you make a choice and "measure" the state of your hand does it become one or the other.

This strange behavior is a consequence of the fundamental uncertainty principle in quantum mechanics, which states that it is impossible to know the precise state of a quantum system with complete accuracy. The idea of quantum superposition helps to explain many of the strange and counterintuitive phenomena that are observed at the quantum scale, such as the fact that particles can be entangled and can seemingly communicate instantaneously over vast distances.

Come to think about it, I am no longer surpised that it generates responses like this one when it comes to QM. After all physicists do the same when they write popular articles/books about QM.

 

[Demystifier]

Me: Solve $x^5=1$
ChatGPT:
If $x^5 = 1$, then $x$ is one of the fifth roots of $1$. The fifth roots of $1$ are $1, -1, i, -i$, and $-i^2$. Note that $i^2 = -1$. Therefore, the solutions to $x^5 = 1$ are:
$1, -1, i, -i$, and $-i^2$
So, $x$ can be any of these values.

It works much better when you give "him" a little help, like to a real person.

Me: Solve x^5=1.

ChatGPT:
The equation x^5 = 1 has five solutions: x = 1, x = -1, x = 1i, x = -1i, where i is the imaginary unit defined as the square root of -1.

To see this, we can rewrite the equation as (x-1)(x+1)(x-1i)(x+1i) = 0. This equation is satisfied when any of the factors is equal to zero. Since the factors x-1, x+1, x-1i, and x+1i are all equal to zero when x is equal to 1, -1, 1i, or -1i, respectively, the solutions to the equation x^5 = 1 are x = 1, x = -1, x = 1i, and x = -1i.

Me: That's wrong, try it in polar form of the complex number.

ChatGPT:
I apologize for the error in my previous response. You are correct that the solutions to the equation x^5 = 1 can also be expressed in polar form. In polar form, the solutions to the equation x^5 = 1 are x = cis(0), x = cis(2pi/5), x = cis(4pi/5), x = cis(6pi/5), and x = cis(8pi/5), where cis(theta) represents the complex number with magnitude 1 and argument theta.

To see this, we can rewrite the equation x^5 = 1 as x = cis(2kpi/5) for k = 0, 1, 2, 3, 4. This equation is satisfied when x has a magnitude of 1 and an argument of 2kpi/5 for any integer k. Therefore, the solutions to the equation x^5 = 1 are x = cis(0), x = cis(2pi/5), x = cis(4pi/5), x = cis(6pi/5), and x = cis(8pi/5).

 

[Jarvis323]

The fact it isn't very great at math isn't surprising given its ability to do math is basically a side effect of its ability to generate meaningful text.

Perhaps a model designed and trained specifically for doing math is just around the corner and it will blow our minds?

 

[Demystifier]

The fact it isn't very great at math isn't surprising given its ability to do math is basically a side effect of its ability to generate meaningful text.

Perhaps a model designed and trained specifically for doing math is just around the corner and it will blow our minds?

Exactly!

And I would add that most humans are not very good in math for a similar reason.

 

[Maarten Havinga]

Ask it to which essential question the answer is 42. If it crashes, you can blame me! ;-)

 

[Demystifier]

Ask it to which essential question the answer is 42. If it crashes, you can blame me! ;-)

I think systems trained through machine learning generally don't crash. Once the training is over, the high-level explanation of their working is not based on logical loops (repeat until ...), but on shortcut heuristics (if it quacks like a duck, it's probably a duck). Of course, at the low level it is still based on logical circuits, but this level does not distinguish "deep" questions (what's the meaning of everything) from "mundane" ones (solve equation x=x+1).

 

[gleem]

For some reason, I missed the opening of this post. We must remember that with regard to very specific topics such as a type of math or science, it is not expected to be accurate and was not designed for such specific tasks. Considering that, it is amazing (at least to me) what it can do. I believe ChatGPT is a scaled-down/less powerful version of GPT3.

Unfortunately, most people seem to think it should be universally accurate. It would be interesting to see how it would perform on physics questions if it was set up as a physics tutor considering it would not understand any of it.

 

[Borek]

It makes errors, just like real intelligence. Moreover, it can correct itself.

Not only that, you can talk it into correcting an error and then into making the error once again. As a source of information - especially on the more diffuclt subjects - it is completely uselesss. Plus, it is dangerous, as it sounds pretty confident.

I am afraid it will only strengthen all the problems we already have with antiviral (and every other type of) crackpots - they will have another "source" of information that will "support" their stance.

 

[InkTide]

Not only that, you can talk it into correcting an error and then into making the error once again. As a source of information - especially on the more diffuclt subjects - it is completely uselesss.

The best way I've seen its usual output described is "fluent bullshit". It's bad at coding and math because it is, at its core, a weighted shuffling of components in the training data.

From a computer science standpoint I think it's important to recognize that ML models use, from input to processing to output, data in the form of matrices - there is no way for operations with these matrices to infer details between elements within these matrices that are not contained within the matrices (this is what your brain does with the matrix visual input from screens, and why AI art struggles with consistent anatomy and lighting, for example - you have coherent mental models from incomplete data, the AI can only attempt to replicate the data, and is ultimately still judged in accuracy by humans with said mental models), without essentially baking those inferences in... which is exactly the sort of tedious problem ML models are meant to avoid.

From an epistemological standpoint I think it's important not to discount the capacity of our mental models to resolve AI output into an inference of... well, AI inference. An inability to answer "why" questions (especially "why did you get this wrong" questions) coherently and correctly suggests we're still the only ones doing the inferring.

Probably the best example I've seen of that lack of inference in these models was the way one of them (Galactica or something, I think) created citations when asked to write papers - they were formatted perfectly, and cited papers that didn't exist with titles completely unrelated to the inline reference. There was no understanding of what a citation is, how to use one, or why to use one - just an output made to look like one.

 

[Filip Larsen]

Arstechnica has a nice description for the layman on the current wave of transformer-based ML (the T in GPT) and why they are starting to exhibit broader problem solving skills that was not as such designed into the models from the start:
https://arstechnica.com/gadgets/2023/01/the-generative-ai-revolution-has-begun-how-did-we-get-here/

 

[Bystander]

described is "fluent bullshit"

"William's syndrome?"

 

[Borek]

"fluent bullshit"

which reminds me of this tweet: ;)

 

[InkTide]

which reminds me of this tweet: ;)

View attachment 321485

I don't agree with gendering terms like that, but I definitely agree with the core sentiment.

I think we need to be extremely cautious of mistaking the imitation of problem solving for actual problem solving - if the student (AI) is just copying the answer from the textbook (training data), are they really understanding the problem or solution? Is that understanding even something that can be achieved by increasing the likelihood of accurate reproduction of the answer coupled to the input problem, or is there something else required? Is rote memorization really "learning"?

As for the confidence... it may be a simple consequence of a binary responses to output (e.g. "good output" vs "bad output", or direct comparison/ranking of output quality) disfavoring outputs acknowledging uncertainty or ambiguity - simpler handwriting analysis neural nets will confidently read symbols from random noise for this reason.

 

[Delta Prime]

While I, as a language model, am able to provide information and explanations on a wide range of topics, I do not have access to the Internet and am not able to browse for or access current information about specific topics

Anticipating two chatGPT's since they are language models they would be able to chat to each other in their own language more efficient.

 

[gleem]

Anticipating two chatGPT's since they are language models they would be able to chat to each other in their own language more efficient.

I read that this happened at Google a few years ago. They had two AI agents that were communicating with one another in their own language. Google not knowing what was being exchanged shut them down.

 

[anorlunda]

I read that this happened at Google a few years ago. They had two AI agents that were communicating with one another in their own language. Google not knowing what was being exchanged shut them down.

Siri versus Alexa.

 

[Delta Prime]

I read that this happened at Google a few years ago. They had two AI agents that were communicating with one another in their own language. Google not knowing what was being exchanged shut them down.

Just to play it safe I wonder if one could have access to the internet and the other not.

 

[Delta Prime]

Exactly!

And I would add that most humans are not very good in math for a similar reason.

Intriguing! Perhaps we may gain insight on those who are exceptional in mathematics being savants

 

[renormalize]

I read that this happened at Google a few years ago. They had two AI agents that were communicating with one another in their own language. Google not knowing what was being exchanged shut them down.

Possibly an apocryphal story inspired by events in Colossus: The Forbin Project (1970). In the movie, US and Soviet defense computers conspire to control the world using their own indecipherable mathematical language.

 

[Delta Prime]

Possibly an apocryphal story inspired by events in Colossus: The Forbin Project (1970). In the movie, US and Soviet defense computers conspire to control the world using their own indecipherable mathematical language.

Given the track record of comic books being ahead of science by 50 years I would say Sci-Fi movies are right on time,. Relatively speaking. (Pun intended)

 

[gleem]

I read that this happened at Google a few years ago. They had two AI agents that were communicating with one another in their own language. Google not knowing what was being exchanged shut them down.

I had to go back and check this out. It turns out that in 2017 Google, Facebook, and OpenAI reported AI seemingly using its own language. The one I was thinking of was Facebook's chatbots, Bob and Alice. Facebook was using them to study how two AI agents could negotiate. They were not s specifically required to do this in English and apparently began using a made-up language. Facebook then required them to use English and continued their study. Apparently, some misinterpreted this and reported it in a more alarming manner.

Google had a language translation AI that used its own language in translating one language into another by going threw its own language. Google continued the program.

OpenAI was trying to get one of its AI to develop its own language.

 

[gleem]

Let the battle of the chatbots begin.

Monday Google announced that they will begin using their chatbot BARD today. It is built on their NLP laMDA.
However, Its initial performance has dropped Google's stock price by 8%. oops!

https://www.investors.com/news/google-stock-falls-after-googles-bard-ai-ad-shows-inaccurate-answer/Microsoft announced yesterday that their chatbot ChatGPT is now available on Bing on a limited basis. It is based on GPT-4. Unlike GPT-3 It has access to the internet.

https://www.google.com/search?q=gpt...#fpstate=ive&vld=cid:f6eef211,vid:69oJY8gdBo0

 

[Delta Prime]

As long as they keep telegraphing their punches I think we will be okay!
Freudian slip perhaps when distracted.

 

[BWV]

ChatGPT (black) defeats Stockfish (white ). i have been studying ChatGPT’s moves and adopting them has raised my ELO by 1000 points already

 

[Jarvis323]

ChatGPT (black) defeats Stockfish (white ). i have been studying ChatGPT’s moves and adopting them has raised my ELO by 1000 points already

View attachment 322150

An interesting test of AI intelligence would be to invent a new, novel game, tell it the rules, and then see how well it plays.

 

[RebeccaSantios]

You know ChatGPT, however inaccurate, can eventually explain complex ideas in simple language. This could be revolutionary and change the way we get education.
I'm seeing some startups already pick up on this trend like Outread. They summarise research articles into 15-min summaries. They're using ChatGPT to pick key insights. They also do have a proof-reading process where specialists in respective fields double check the summaries before they are published. But I think it's a brilliant way to democratise access to cutting-edge research - which had inherently been inaccessible to the general public.

 

[PeterDonis]

You know ChatGPT, however inaccurate, can eventually explain complex ideas in simple language.

What good is an explanation in simple language that's wrong?

 

[Bandersnatch]

What good is an explanation in simple language that's wrong?

Arguably, if you ask a bunch of human experts some questions, and half the answers are some impenetrable combination of snobby, off-putting, jargon-heavy, missing the point, and at inappropriate level - you'll learn about as much as if you asked a bot and got perfectly tailored, easy to digest answers that are wrong half the time.
Rather not a good strategy if one wants to build a bridge or pass an exam, but good enough if they just want to satisfy their daily curiosity cravings.
Same reason people keep reading/watching popular science even though textbooks exist.

That said, the post you quoted looks so much like an advertisement by a bot that it's probably an advertisement by a bot. Who knows, maybe even written by ChatGPT.

 

[PeterDonis]

Arguably, if you ask a bunch of human experts some questions, and half the answers are some impenetrable combination of snobby, off-putting, jargon-heavy, missing the point, and at inappropriate level - you'll learn about as much as if you asked a bot and got perfectly tailored, easy to digest answers that are wrong half the time.

In the sense that you'll learn zero, yes. But how does that help? It doesn't. Learning zero information from a wrong chatbot is no better than learning it from "unskilled and unaware of it" humans.

 

[Bandersnatch]

In the sense that you'll learn zero

It's not always wrong, is the point.