Can ChatGPT Handle Complex Physics Questions Accurately?

[Nugatory]

[Chatbot says]

…moving at any speed, not just objects moving at constant speeds.

That’s not much of an improvement. I think we’d have to prompt it a bit more before it gets the words “acceleration” or “non-inertial” into the response.

 

[Hornbein]

It's possible that what we as humans do is no more than glorified and more sophisticated collation,

I'd say that this describes the great majority of human intelligent behaviour. And most people can't do this.

I'm a musician. Creativity in that field is at least 90% glorified and more sophisticated collation. Look at Youtube videos by amateur musicians. The successful ones are usually a copy of something, often with great (sometimes astonishing) exactitude. This is what is in demand. Pro musicians don't usually do that in public, but generally they have spent a great deal of time doing this in their bedrooms. Eventually they learn to draw on established work and mutate it a little bit. The result will sound mainstream and familiar. This can be sold.

On the other hand, the van Halen brothers said they were failures as session men. Their sound had too much originality. They couldn't get enough work.

I'm told that physics and math publication is more restricted than this. Original work can't be published because there are no peers to review it. It might be wrong. Readers might not be interested. Instead what you get are minor variations on familiar themes. A mathematician who needed a track record wrote he gave up on originality and concentrated on minor variations on established topics. If a journal has already published 100 papers on a topic there is a very good chance they will publish a 101st.

but I'm not convinced. There is a genuine spark when real intelligence is involved to see beyond the material that's presented and create, even in a modest way, something original.

Just yesterday I used an AI program that creates images on request. I was happy with its originality. What's more though, it was able to get my style out of some database and present me with new ideas. I quite liked seven out of the eight (the eighth was too cliche). I had to be impressed.

In sum I'd say that these programs are more intelligent and original than the great majority of people. They might not be Einsteins, but they are working on it. Who knows what they will be doing ten years from now? One hundred years? One thousand?

 

[PeterDonis]

In sum I'd say that these programs are more intelligent and original than the great majority of people.

The word "intelligent" can be defined in many ways, but at least one way implies some kind of understanding of the material. That's what ChatGPT doesn't have. So-called "AI" programs might indeed be very helpful at sifting through an astronomical number of possible permutations of some data set and picking out ones that are likely to meet a human aesthetic criterion, such as an image that looks reasonably good or a piece of music that sounds reasonably good. But meeting an aesthetic criterion is not the same thing as being correct and knowing you're correct because you understand the subject matter. (Arguably this notion of "correct" doesn't even apply to aesthetic judgments like those we make about images or music.)

 

[Hornbein]

The word "intelligent" can be defined in many ways, but at least one way implies some kind of understanding of the material. That's what ChatGPT doesn't have.

I'm saying that this is also something the great majority of people don't have.

 

[PeterDonis]

I'm saying that this is also something the great majority of people don't have.

With regard to particular subject matter, yes, that's probably the case. The great majority of people probably don't understand relativty or quantum mechanics, for example, meaning that they could not give correct answers to questions about those subjects, say, here on Physics Forums. Is your point simply that ChatGPT, which can't do that either, still does as well as most humans even if it is not "intelligent" in that sense?

 

[Hornbein]

With regard to particular subject matter, yes, that's probably the case. The great majority of people probably don't understand relativty or quantum mechanics, for example, meaning that they could not give correct answers to questions about those subjects, say, here on Physics Forums. Is your point simply that ChatGPT, which can't do that either, still does as well as most humans even if it is not "intelligent" in that sense?

Yes. And I would not be surprised if before long we have AIs which can do physics homework.

 

[Frabjous]

Yes. And I would not be surprised if before long we have AIs which can do physics homework.

There are a large number of incorrect answers on the internet which will make for interesting training sets.

 

[Hornbein]

There are a large number of incorrect answers on the internet which will make for interesting training sets.

This is why ChatGPT does not include the Internet in its training set. The obvious choice would be physics textbooks.

 

[PeterDonis]

This is why ChatGPT does not include the Internet in its training set.

Yes, it does. From [1]:

GPT-3.5 was trained on massive amounts of data about code and information from the internet, including sources like Reddit discussions

[1] https://www.searchenginejournal.com/what-is-chatgpt/473664/

 

[PeroK]

Yes. And I would not be surprised if before long we have AIs which can do physics homework.

Sure, it could find a model solution. But, assess a student's work and figure out how to help them? That's the level of intelligence that is missing.

 

[PeterDonis]

assess a student's work and figure out how to help them?

If it included a large enough corpus of questions and answers from help forums (like this one), it could probably simulate this pretty well, responding to questions from students with answers that looked to the student like genuine help.

The problem is, the student would have no way of knowing whether the apparently helpful answers were correct or not, because there is no understanding beneath them.

 

[Hornbein]

If it included a large enough corpus of questions and answers from help forums (like this one), it could probably simulate this pretty well, responding to questions from students with answers that looked to the student like genuine help.

The problem is, the student would have no way of knowing whether the apparently helpful answers were correct or not, because there is no understanding beneath them.

This is true today, but what of tomorrow?

The big breakthrough occurred when an AI became Go champion of the world. I had not expected that in my lifetime. A few years later it became champ in Starcraft, something that many thought couldn't be done. Then AI found a significant improvement in data compression, a problem that smart people have worked on for decades. I expect that progress will continue at this "breakneck pace" for quite some time. It's revolutionary.

 

[PeterDonis]

The big breakthrough occurred when an AI became Go champion of the world. I had not expected that in my lifetime. A few years later it became champ in Starcraft, something that many thought couldn't be done. Then AI found a significant improvement in data compression, a problem that smart people have worked on for decades. I expect that progress will continue at this "breakneck pace" for quite some time. It's revolutionary.

These are all impressive achievements as far as computer programming, machine learning, etc., but none of them are achievements as far as understanding. Basically these efforts are all exploring the space of problems that can be solved by brute force sampling of huge search spaces. It turns out that there are many problems that can indeed be solved that way. But that does not mean all human problems can be solved that way.

 

[Hornbein]

These are all impressive achievements as far as computer programming, machine learning, etc., but none of them are achievements as far as understanding. Basically these efforts are all exploring the space of problems that can be solved by brute force sampling of huge search spaces. It turns out that there are many problems that can indeed be solved that way. But that does not mean all human problems can be solved that way.

The game of Go cannot be solved by brute force. The search space is too large. That was the whole point.

The big obstacle was pattern recognition, which was how humans became Go champs. Once the pattern recognition of a computer program became comparable to that of a human, combined with raw power the AI became unbeatable.

Next they said that AI would never be able to deal with games of incomplete information, games where it is necessary to anticipate what a human opponent has done, is doing, and will do in secrecy. Well, Starcraft is such a game. AI is now dominant there as well. Then they said AI couldn't rule in poker, as it was necessary to "read" your opponents mannerisms. Wrong.

I deal with AI every day. I upload lots of videos to Youtube that contain copyrighted material. (Save your breath -- it is an entirely legal and accepted practice.) The AI does an excellent job of detecting this. It takes about ten seconds to detect copyright transgressions.* Considering the huge amount of copyrighted works -- billions? -- it's an incredible accomplishment of pattern matching. Five years ago none of this was possible. I have had a front row seat as the AI improved, and can say the result is impressive.

It didn't happen by itself. A great deal of effort by very smart people went into this. This process is ongoing and I expect it has yet to reach its full momentum.

---

*The AI then assigns all income to the copyright holder.

 

[StevieTNZ]

Philosophers sound pretty similar :-). SCNR.

Rude.

 

[PeterDonis]

The game of Go cannot be solved by brute force. The search space is too large. That was the whole point.

By "brute force sampling" I did not mean an exhaustive search; of course that's not possible. I meant a search to some finite depth that is manageable in terms of the computational power available, and then applying heuristics to evaluate each branch of the search tree and pick the one with the highest estimated probability of winning.

The big obstacle was pattern recognition

Which, as I understand it, is how the heuristics are developed that evaluate the branches of the finite search tree.

Go might be something of an outlier in that, as you describe it, the methods used by AI are similar to the methods used by human champions. AFAIK that was not the case in chess, where the methods used by, e.g., Deep Blue are nothing like the methods used by human champions. I would expect that is also true of incomplete information games like poker. Of course, humans might decide to adjust their methods of play after seeing what AIs do. Or humans might find the games less interesting when they realize that AIs can dominate them using methods that humans cannot adopt or don't find to be interesting.

 

[Hornbein]

By "brute force sampling" I did not mean an exhaustive search; of course that's not possible. I meant a search to some finite depth that is manageable in terms of the computational power available, and then applying heuristics to evaluate each branch of the search tree and pick the one with the highest estimated probability of winning.Which, as I understand it, is how the heuristics are developed that evaluate the branches of the finite search tree.

The big breakthrough came when they got machine learning to work. That means no programming. The system figures out its own heuristics, which may be inscrutable. This took about sixty years of development culminating in AlphaGo. But that wasn't all. AlphaGo began with a training set. Next was AlphaZero. No training set. It learns purely by competing with itself. AlphaZero became world chess champion after two or so days of this, then after five days or so defeated AlphaGo to become world Go champion. Now that's true machine learning.

To me, "heuristics that no one can explain" is a pretty good definition of intuition, or at least expertise.

Go might be something of an outlier in that, as you describe it, the methods used by AI are similar to the methods used by human champions. AFAIK that was not the case in chess, where the methods used by, e.g., Deep Blue are nothing like the methods used by human champions. I would expect that is also true of incomplete information games like poker. Of course, humans might decide to adjust their methods of play after seeing what AIs do. Or humans might find the games less interesting when they realize that AIs can dominate them using methods that humans cannot adopt or don't find to be interesting.

Humans may adjust their play but it is like John Henry against the steam drill. It is notable that world chess champion Magnus Carlson has decided not to defend his title. Today his preparation would be working with an AI, memorizing moves as white. For example in the WC qualifying tournament a player with black was widely praised for responding to white's memorized best moves with nineteen consecutive optimal moves, eventually leading to a draw. (Since the memorized moves take practically no time the responder then had a time deficit, but not enough to sink his ship.) For a world championship this memorization is such a tedious task that Carlson decided it wasn't worth it.

Furthermore Carlson accused Hans Niemann of cheating because this man found the optimal moves too quickly. Carlson said there must have been a spy in his camp who informed the opponent what Carlson was working on so that the guy could memorize the optimal responses. Neimann responded that he had indeed memorized them by way of having deduced/inferred/guessed what Carlson would throw at him. No rule against that. It's possible. Neimann has sued for an absurd sum. If Carlson can't get any evidence otherwise he'll have to settle.

As for poker players they can just forget it. I'd have to say producing a winning poker system seems easy compared with Go. The AI can use game theory to find an optimal path. The next step is to take greater advantage of human players' deviations from said path. The best a poor boy can do against such a system is statistically break even. This would be no small achievement.

ChatGPT is the latest milestone. Expect it to improve rapidly, presumably by developing expertise in specialized domains.

Go continues unchanged. They are fortunate in that the space is so large that such memorization is unfeasible. Sure, they can't beat the steam drill, but so what? They can still compete against one another as they always have. Machines can lift much greater weights than can human weight lifters. This doesn't stop humans from doing it.

 

[PeterDonis]

To me, "heuristics that no one can explain" is a pretty good definition of intuition, or at least expertise.

Yes, but they're heuristics for games in which the "universe" is finite and well-defined, with definite victory conditions.

Life in general is not like that.

 

[Hornbein]

Yes, but they're heuristics for games in which the "universe" is finite and well-defined, with definite victory conditions.

Life in general is not like that.

If there are no definite victory conditions then there is nothing more to be said. There are many definite scores such as popularity, the Dow Jones average of thirty industrials, all manner of awards and other credentials, and the inevitable "net worth" or "bottom line." Which one or weighted combination thereof one may chose is a matter of taste.

The number of possible games of Go is far greater than the number of elementary particles in the visible Universe. I think it is fair to say that this space is infinite for all practical purposes, virtually infinite.

Now I shall seize this opportunity to return to physics content. Is the number of possible distinct Earths in our Universe finite? Some people seem to think so, leading to arguments that anything that happens here happens an infinite number of times elsewhere. I don't believe it.

 

[PeterDonis]

If there are no definite victory conditions then there is nothing more to be said.

About what? Most of what we have to deal with in life has no definite victory conditions. Life in general is open-ended.

The number of possible games of Go is far greater than the number of elementary particles in the visible Universe. I think it is fair to say that this space is infinite for all practical purposes

Not if a finite set of heuristics embodied in a computer program can win at it. I wasn't using the term "universe" literally, which is why I put it in scare quotes.

 

[PeterDonis]

Now I shall seize this opportunity to return to physics content. Is the number of possible distinct Earths in our Universe finite? Some people seem to think so, leading to arguments that anything that happens here happens an infinite number of times elsewhere. I don't believe it.

Please start a new thread for this topic since it is different from the topic of the current one. (Although I think we already had a recent thread on it.)

 

[Hornbein]

About what? Most of what we have to deal with in life has no definite victory conditions. Life in general is open-ended.Not if a finite set of heuristics embodied in a computer program can win at it. I wasn't using the term "universe" literally, which is why I put it in scare quotes.

It's like the two guys running from a grizzly bear. One guy says, don't you realize we can't outrun the bear? The other guy sez, I don't have to. I only have to outrun you.

The ai doesn't have to play a perfect game to win.

 

[PeterDonis]

The ai doesn't have to play a perfect game to win.

I didn't say it did.

 

[PeroK]

If there are no definite victory conditions then there is nothing more to be said. There are many definite scores such as popularity, the Dow Jones average of thirty industrials, all manner of awards and other credentials, and the inevitable "net worth" or "bottom line." Which one or weighted combination thereof one may chose is a matter of taste.

The number of possible games of Go is far greater than the number of elementary particles in the visible Universe. I think it is fair to say that this space is infinite for all practical purposes, virtually infinite.

Now I shall seize this opportunity to return to physics content. Is the number of possible distinct Earths in our Universe finite? Some people seem to think so, leading to arguments that anything that happens here happens an infinite number of times elsewhere. I don't believe it.

I'll grant you one thing. ChatGPT can stick to the point more than you can!

 

[Demystifier]

I'm told that physics and math publication is more restricted than this. Original work can't be published because there are no peers to review it. It might be wrong. Readers might not be interested. Instead what you get are minor variations on familiar themes. A mathematician who needed a track record wrote he gave up on originality and concentrated on minor variations on established topics. If a journal has already published 100 papers on a topic there is a very good chance they will publish a 101st.

That would be a very interesting topic for a separate thread. The ideas of Thomas Kuhn (normal science vs revolutionary science) would be very relevant.

 

[PeroK]

ChatGPT is the latest milestone. Expect it to improve rapidly, presumably by developing expertise in specialized domains.

We all see that. Eventually AI will be able to take over most things. But, if you asked ChatGPT currently to teach you theoretical physics and mark your homework, it would fail. And it would fail in dangerous ways because it would never say "I don't know" - it would just give you some spiel that looks plausible.

 

[Jarvis323]

It's a little funny how we tend to compare us with them. In a sprint, we always have Usain Bolt in our corner (not that it helps much; people, even Bolt, are really super slow).

If I criticize the art generated by DALL·E or MidJourney too harshly, I have to rethink the brilliance of my own stick figure art (maybe it can convey human experience or emotion better), or consider my possible theoretical potential as a human being, or something.

The other day, I saw someone call the professional sports team they root for "we". Regarding humans feeling bad about not keeping up, maybe we can just somehow bring AI into the fold of "we".

Anyways, maybe AI surpassing us will calm our egos a little and allow us to find some core things to value more, like wisdom, good will, honesty, etc.

 

[Filip Larsen]

One thing I have been impressed with in the reported and discussed ChatGPT interactions (haven't had any sessions with it myself) is the systems apparent ability to understand and deliver on requests regarding form, notwithstanding how factual accurate or logical coherent the content was or wasn't.

I assume its within the current topic to ask if anyone has noticed some serious goof by ChatGPT in this regard or is it really that sharp in picking up and performing requests towards form? How complicated instructions can it actually deliver on?

 

[vanhees71]

This bot seems to be really pretty impressive and threatening at least the humanities. In a recent newspaper article (Frankfurter Allgemeine Zeitung) an anglistic professor wrote that she made a test with one of his colleagues. She just let the bot write some typical text as expected from the students as term work ("Hausarbeit"), which in the humanities at German universities is usually given to be graded and getting into the final grade of the BSc or MSc degree. The colleague was not able to decide, whether the text was written by her or the bot. Now they have to write exams instead of term works to make sure that the students write their works themselves rather then letting it do by the bot. It's also impossible to figure out such cheating by using standard "anti-plagiarism software" of course.

I guess, it's only a matter of time, until such bots can also write scientific papers including formulae and/or even faking experimental results...

It's a bit alarming concerning the next level of scientific cheating.

 

[physika]

Me: Can you be creative and original?

ChatGPT:
As a language model, I am able to generate original responses to user inquiries based on the information that I have been trained on. I have been designed to be able to generate human-like text and can produce a wide range of responses to user input, including responses that may be considered creative or original. However, it is important to note that I am not a human and do not have personal experiences or perspectives, and my responses are based on the information that I have been trained on.

I do not have the ability to experience emotions, make personal choices, or engage in independent thought in the same way that humans do. My responses are generated using machine learning algorithms that allow me to analyze the structure and content of the input I receive and generate appropriate responses based on the patterns and connections I have learned from the data I was trained on.

While I can generate original responses based on the information that I have been trained on, I do not have the ability to create new ideas or concepts in the same way that a human can. My responses are limited to the information and patterns that I have learned from the data I was trained on, and I do not have the ability to think creatively in the same way that a human can.

Hal