Stephen Wolfram explains how ChatGPT works

[Haborix]

Someone should ask these AI models if they fear a Butlerian Jihad.

 

[Ken G]

For me, the most interesting thing ChatGPT does is encourage us to introspect more deeply into how we think, and what we mean by "understanding." On grounds that ChatGPT only looks for frequency of association, it is easy to say that ChatGPT doesn't "understand" anything (and it is ironic that it uses that language itself, personally I don't think any AI should ever use the pronoun "I" in any situation or for any reason, so if you avoid that, it becomes easier to avoid claims like "I understand"). But what is less easy to say is, what are we doing when we "understand the meaning" of our own words? What are we doing that is different from frequency of association?

I am not a linguist, but if you look at a child learning language, it seems pretty clear that they are establishing frequency of association. But it is not just association with other words, it association with experience. That is the "ghost in the machine" of understanding, the ability to have experience and detect similarities in classes of experience, and then it is those associations that can be correlated with words. Humans who existed prior to language would certainly have been able to notice similarities in classes of experience, it just did not occur to them to attach labels to those associations. ChatGPT is the opposite, it's pure labels, no experience at all. So it borrows the meaning we establish with our experiences, and it is our experiences that give the words meaning which then "prime the pump" of ChatGPT.

Let us then dig deeper. Our brain is not one thing, it has elements that are capable of registering experience, elements that are capable of contemplating experience, and elements that house our language capabilities that can associate and recognize labels around those experiences. So before we judge ChatGPT too harshly on the basis that it can only manipulate labels without any understanding, we should probably note that whatever elements of our brains are capable of manipulating language, including the capabilities of our greatest language masters and poets, are probably also incapable of "understanding" those labels either. They must draw from other parts of our brains to establish what the meanings are, what the experiences are, but I suspect without knowing that our own language mastery also involves primarily frequencies of associations.

That may be why, if you ask ChatGPT to write a poem, it's technical proficiency of manipulating poetic language is actually pretty good (try it, and ask it to use the style of any poet you like), but the poems will be rather vanilla and lacking of the sublime and specific contours of individual experience, borrowing as they must from the training data from a multitude of experience to create any kind of meaning. Hence I think the most reasonable way to think of ChatGPT is not as a con man that is fooling us, but rather as an incomplete piece of a thinking system that has a rather enormously disconnected "middleman" between the experiences from which its words borrow meaning, and the actual manipulation of that language. Perhaps like a piece of a thinking brain, moreso than a complete brain capable of higher thought on its own.

If so, then the real value of AI going forward will not be in replacing human brains, but in augmenting their capabilities. In the process, we may learn a great deal more about what our brains actually do, and even more than that, what they also are fooling us into believing they are doing! In light of all this, let us recollect the prescient words of B.F. Skinner: "The real question is not whether machines think but whether men do. The mystery which surrounds a thinking machine already surrounds a thinking man." (No doubt he meant to include women as well, unless he was indeed a wry dog.)

 

[PeterDonis]

if you look at a child learning language, it seems pretty clear that they are establishing frequency of association. But it is not just association with other words, it association with experience.

Exactly, and this is what ChatGPT (or at least the version discussed in the article here) lacks.

 

[gmax137]

I am not a linguist, but if you look at a child learning language, it seems pretty clear that they are establishing frequency of association. But it is not just association with other words, it association with experience.

hmm has this been established by linguists? I have observed association with both experience and words. Think how a child is taught to say, "you're welcome," in response to "thank you." It is just rote. Same with learning times tables: "what is five times seven?" little Johnny says "thirty-five." He's not calculating the answer, at least not after a couple months in class.

 

[Ken G]

I agree, that's why I said is it not just association with other words. With ChatGPT, it's just association of words with words.

 

[Ken G]

By the way, I really welcome ChatGPT in the classroom. If you think about it, the fact that ChatGPT often gives quite good answers to general introductory level physics and astronomy questions (not mathematical logic, mind you, it's not good at that at all so here I'm talking about lower level classes that do not include much mathematical reasoning), yet does not have any deeper understanding of those answers, means that it simulates the kind of student that can get an A by parroting what they have heard without understanding it at all. The way ChatGPT fools us into thinking it understands its own explanations, is exactly the problem we should be trying to avoid in our students. The worst part is, sometimes our students don't even realize they are only fooling us, because we have trained them to fool themselves as well. We tell them an answer, then ask them the question, and give them an A when they successfully repeat the answer we gave them before. They walk away thinking they learned something, and we think they learned something. But don't dig into their understanding, don't ask them a question which calls on them to think beyond what we told them, if you don't want to dispell this illusion!

Hence, the way to defeat using ChatGPT as a cheat engine is the same as the way to dispell the illusion of understanding where there is not understanding: ask the follow on question, dig into what has been parroted. That's actually one of the things that often happens in this forum, we start with some seemingly simple question, and get a seemingly straightforward answer, but after a few more posts it quickly becomes clear that there was more to the question than what was originally intended by the asker. If we teach students to dig, we are teaching them science. If we teach them to do what ChatGPT does, we cannot complain that ChatGPT can be used to cheat!

 

[mathwonk]

I don't know if this is the same topic, but today's New York Times has an article revealing that many travel books offered for sale on amazon today are "fakes", cheap worthless documents written by AI rather than actual humans who have traveled to the relevant countries, just paste ups using freely available material from the internet. I subsequently found several such books advertised there by apparently fake authors such as "William Steeve" (a rip off of Rick Steves), and Mike Darcey. They have apparently taken down the books by "Mike Steves", which were prominently featured in the article. It is not clear to me if Amazon itself is perpetuating this fraud or only abetting it, but some of the kindle books at least seem to be published by amazon. Even the author photographs and biographies are apparently fakes. The biography of "Mike Steves" closely mirrored the biography of Rick Steves, but all the information was apparently false for Mike, including his writing history and his home town, neither of which checked out.

https://www.nytimes.com/2023/08/05/...te=1&user_id=73fab36102e49b1b925d02f237c74b7e

 

[Ken G]

It certainly does seem like AI makes these kinds of ripoffs much easier, along with all kinds of cons. Since a lot of scams these days originate in other countries, one of the most common tipoffs is strange mistakes in the English, which might be a lot easier to avoid by using AI. Why is it that every new invention comes with all this collateral damage?

 

[Filip Larsen]

Another great article on Arstechnica on LLM technology, this time on how and why the newest GPT models have such powerful cross domain generative capabilities:
https://arstechnica.com/ai/2023/08/how-chatgpt-turned-generative-ai-into-an-anything-tool/

Note that this article combines well with an earlier mentioned (post #30) article:
https://arstechnica.com/science/202...anation-of-how-ai-large-language-models-work/

 

[mathwonk]

properly used and implemented, AI seems to help with physics instruction, potentially replacing a section man to answer student questions. One advantage of AI in this experiment seems to be its superior accomodation to the level/speed of the student. But they had to "customize" the program appropriately.
https://news.harvard.edu/gazette/st...i-tutor-to-physics-course-engagement-doubled/

 

[romsofia]

properly used and implemented, AI seems to help with physics instruction, potentially replacing a section man to answer student questions. One advantage of AI in this experiment seems to be its superior accomodation to the level/speed of the student. But they had to "customize" the program appropriately.
https://news.harvard.edu/gazette/st...i-tutor-to-physics-course-engagement-doubled/

Yup, as someone who works on the back end on one of the bigger companies in the LLM space, we have been hiring "tutors" so we can mass train on math, coding, physics, chemistry, etc. Essentially, they come up with problems and see if they can stump our model, and if it stumps the model, they will correct it and we collect all these as training data. The other part is we use web scrappers to take problems from the internet too, and then have the tutors see if the LLM solved the problem correctly, or if they have to correct them (sorry physicsforums!).

On one hand, it's very interesting to see how fast the models are getting at "reasoning" and solving problems. I think it'll be better as a whole if we can succeed on creating LLMs that can help people from areas with lesser education to have access to something that can really help them! There are a lot of "new" implementations with things such as agents which means the "AIs" will have the ability to use python for calculations since LLMs aren't reliable for actual calculations, or APIs. This means that if a student asks the something like chatgpt for help on a math problem, it is known that it can't reliably do things as simple as multiplication due to the probabilistic nature of how our current neural network infrastructures do computations. The way we've gotten around this is when the LLM is prompted to do a hard calculation, it will contact the agent to do the calculation which is know is more reliable. The LLM will then use this result, and continue on with the problem solving.

On the other hand, I already see the writing on the wall, these companies will want to enter the tutoring space and offer these LLMs for a monthly subscription for math, physics, etc and if they become good enough, TAs and tutoring will be a thing of the past. This becomes the scary part for me. With the rise of fascism (again...), these tools can also be used in secular communities and for more thought control.

I'm hoping that these tools are used for the bettering of humanity, but I'm old enough to also know that we as a species have a pattern of going two steps forward for equality, then one step back. It's still forward progress, but would be nicer if we just skipped the step of going back.

 

[jedishrfu]

By the way, I really welcome ChatGPT in the classroom. If you think about it, the fact that ChatGPT often gives quite good answers to general introductory level physics and astronomy questions (not mathematical logic, mind you, it's not good at that at all so here I'm talking about lower level classes that do not include much mathematical reasoning), yet does not have any deeper understanding of those answers, means that it simulates the kind of student that can get an A by parroting what they have heard without understanding it at all. The way ChatGPT fools us into thinking it understands its own explanations, is exactly the problem we should be trying to avoid in our students. The worst part is, sometimes our students don't even realize they are only fooling us, because we have trained them to fool themselves as well. We tell them an answer, then ask them the question, and give them an A when they successfully repeat the answer we gave them before. They walk away thinking they learned something, and we think they learned something. But don't dig into their understanding, don't ask them a question which calls on them to think beyond what we told them, if you don't want to dispell this illusion!

Hence, the way to defeat using ChatGPT as a cheat engine is the same as the way to dispell the illusion of understanding where there is not understanding: ask the follow on question, dig into what has been parroted. That's actually one of the things that often happens in this forum, we start with some seemingly simple question, and get a seemingly straightforward answer, but after a few more posts it quickly becomes clear that there was more to the question than what was originally intended by the asker. If we teach students to dig, we are teaching them science. If we teach them to do what ChatGPT does, we cannot complain that ChatGPT can be used to cheat!

 

[jedishrfu]

My first year of college was quite exciting. I had learned differential calculus over the summer I thought and was anxious to move onto Integral calculus. However, I had to get permission from my new to my college differential calculus professor.

He quizzed me on a variety of differential calculus problems and I got them mostly right that he knew I had mastered the material. Until, he asked about limits. He wanted me to explain them in my own terms which I did but then he said well that's not quite right. Go study some more and when you're ready come back.

I went back a second and third time getting more not quite right responses. My fourth and final attempt, I recited the limit definition exactly as it was stated in the book and he said you know I think you got it.

He was one my favorite profs. This approach could work for those students using ChatGPT to do their work.

---

A more primitive example, was when some freshmen were doing an electrical lab measuring resistance and current and asked to determine the voltage of a battery.

We watched them use their newly acquired electronic calculators circa 1973 to compute an answer of 1500v for the battery and were shocked by their answer. We asked how did the arrive at that answer to wit they said that's what the calculator said.

We had to chuckle because we knew it to be a 1.5v D cell wrapped in some tape to obscure the labeling.

 

[Ken G]

He wanted me to explain them in my own terms which I did but then he said well that's not quite right. Go study some more and when you're ready come back.

I went back a second and third time getting more not quite right responses. My fourth and final attempt, I recited the limit definition exactly as it was stated in the book and he said you know I think you got it.

He was one my favorite profs. This approach could work for those students using ChatGPT to do their work.

It's odd that he wanted you to explain it in "your own terms," but only accepted the literal definition. It's tricky in math, because there really isn't "your own terms" in math, the definitions are extremely precise and proofs are, at some level, purely syntactic. Computers are used to prove very difficult theorems via brute force methods, but for me that kind of spoils the whole point of math. Do we prove theorems because we want to use them to prove other theorems that we don't understand any better than the ones the computer proved? Or do we prove theorems because we think that if you understand its proof, you understand the theorem better?

---

A more primitive example, was when some freshmen were doing an electrical lab measuring resistance and current and asked to determine the voltage of a battery.

We watched them use their newly acquired electronic calculators circa 1973 to compute an answer of 1500v for the battery and were shocked by their answer. We asked how did the arrive at that answer to wit they said that's what the calculator said.

We had to chuckle because we knew it to be a 1.5v D cell wrapped in some tape to obscure the labeling.

Yes, that's the kind of mistake these AIs make, they can't tell when their answer is absurd, or contradictory. They only know it's the answer they got.