Insights Why ChatGPT AI Is Not Reliable

[russ_watters]

No, it's designed to accept any input whatever and generate response text based on the relative word frequency algorithm that Wolfram describes in his article (referenced in the Insights article). It has no idea that some input is "criticism/clarification" based on its previous responses. It has no semantics.

I understand, but I think you might be reading past my point. As you say, it "accepts any input" regardless of what the input is. What I'm pointing out is that it therefore doesn't know if the input is wrong, and will simply riff on the wrong input. This gives the appearance to the user that they are winning a debate or providing useful clarification on which to get a better answer, when in reality they may just be steering it towards providing or expanding on a wrong answer.

 

[AndreasC]

It has a database of relative word frequencies. That's the only database it has. (At least, that's the case for the version that was reviewed in the articles being discussed here. Later versions might have changed some things.)

It's not all it does though. While you are describing GPT correctly, ChatGPT has additional layers on top of that to "push" it to favor some kinds of response more than others. That's also why it's very hard to make it talk about certain things, or to say "bad" words, etc (all the things for which people "jailbreak" it), it's hard-coded to avoid them. It's not pure word predictor. I've noticed the obsequity @Ken G talks about, sometimes it's very stubborn in insisting it got something wrong, to the extent that it's probably a behavior the developers coded in.

 

[Ken G]

It has a database of relative word frequencies. That's the only database it has. (At least, that's the case for the version that was reviewed in the articles being discussed here. Later versions might have changed some things.)

Yes good point, it starts with a training dataset and then generates its own, such that it only needs to query its own to generate its responses. But the manner in which that dataset is trained is a key aspect, that's where the intelligence of the trainers leaves its mark. They must have had certain expectations that they built in, which end up looking like there is a difference between a prompt like "poem", that will generate a poem about poetry, versus a followup prompt like "shorten that poem". It can even count the words it uses in its answer, and prohibit certain types of answers, so it has some extra scaffolding in its training.

 

[PeterDonis]

it therefore doesn't know if the input is wrong, and will simply riff on the wrong input. This gives the appearance to the user that they are winning a debate or providing useful clarification on which to get a better answer, when in reality they may just be steering it towards providing or expanding on a wrong answer.

Yes, agreed.

 

[PeterDonis]

ChatGPT has additional layers on top of that to "push" it to favor some kinds of response more than others.

Where are these layers in the Wolfram article's description? (I am referring to the Wolfram article that is referenced in the Insights article at the start of this thread.)

 

[Ken G]

Where are these layers in the Wolfram article's description? (I am referring to the Wolfram article that is referenced in the Insights article at the start of this thread.)

I presume the "loss function" used for the training, this must be the crucial place that the trainers impose their intentions onto the output. I keep coming back to the fact that you can prompt ChatGPT with the single word "poem" and it will write a poem about poetry. Surely this is not what you can get from simply taking a body of written work and trying to predict the words that come after "poem" in that body of data, because very few poems are about poetry. There must be someplace where the trainers have decided what they expect the user to want from ChatGPT, and have trained it to produce a poem when the prompt is the word "poem", and have trained it to make the poem be about something mentioned in the prompt. That would go beyond simply predict what comes after the word "poem" in some vast body of text. The LLM would have to be trained to predict what words follow other words that also qualify as satisfying the prompt in some way.

So getting back to the issue as a prompt as a correction, the trainers expect that people will want to iterate with ChatGPT to improve its output in a session, so they will give prompts that should be interpreted as corrections. That's when ChatGPT is trained to be obsequious about doing corrections, it has to be built into the way it is trained and not just a simple algorithm for predicting text that follows the text in the prompt (again, humans are rarely so obsequious).

 

[Ken G]

Perhaps ChatGPT can itself give us some assistance here. When I prompted it to describe why it writes a poem about poetry when given the single word "poem", this was part of its answer:
"The process of training me involves a cost function, which is used to fine-tune my responses. During training, I'm presented with a wide variety of text and corresponding prompts, and I learn to predict the next word or phrase based on those inputs. The cost function helps adjust the model's internal parameters to minimize the difference between its predictions and the actual training data."

I think the important element of this is that it is trained on prompts and text, not just text. So it does not just predict the next word in a body of text, then see how well it did, it tries to predict responses to prompts. But a prompt is not a normal aspect of either human communication or bodies of text (if I came up to you and said "poem", you would not think you were being prompted, you would have no idea what I wanted and would probably ask me what I was talking about, but ChatGPT does not ask us what we are talking about, it is trained that everything is a prompt). So its training must look at bodies of text as if they were in response to something, and they look for some kind of correlation with something earlier that is then considered a prompt.
It says: "The prompt/response framework is a way to communicate how the model learns context and generates responses, but it doesn't necessarily represent the actual structure of the training data."

 

[AngryBeavers]

I have a very difficult time wrapping my head around the idea that all it's doing is predicting text based upon its training data, and algorithms. Only because the the likely hood of certain words being used together frequently enough that it becomes meaningful to a generated response has to rather low at times. Like poem and fiduciary being used togther often enough to have a connection. I am sure the algorithms are more advanced that this, but, still.

 

[Motore]

I have a very difficult time wrapping my head around the idea that all it's doing is predicting text based upon its training data, and algorithms.

To me, it is pretty intuitive that it can do that with clever algorithms (actually they don't need to be very complex) and a lot of training data. Of course the programers can limit the responses by hard coding certain words theat the chatbot must not use, nothing knew.

I keep coming back to the fact that you can prompt ChatGPT with the single word "poem" and it will write a poem about poetry. Surely this is not what you can get from simply taking a body of written work and trying to predict the words that come after "poem" in that body of data, because very few poems are about poetry.

Hmm I prompted ChatGPT with the word "poem" several times and every time it generted a random poem. It is just scrabling text so that the natural language is uphold and that it rhymes (so that it actually looks like a poem, which there has to be presumably millions of them in the training data). Why would the trainers need to add anything?

The LLM would have to be trained to predict what words follow other words that also qualify as satisfying the prompt in some way.

Well sure, that is how LLMs are constructed. You could construct one without prompt and it will just write something random in a natural language somwhere at random time. Not really useful.

That's when ChatGPT is trained to be obsequious about doing corrections, it has to be built into the way it is trained and not just a simple algorithm for predicting text that follows the text in the prompt (again, humans are rarely so obsequious).

Well sure, it has to be trained on, but who said otherwise? Trained on massive data not by people. They just review the responses and give feedback so ChatGPT can optimize itself (as you can also do). At the end of the day It's just predicting which word comes next.

 

[Ken G]

Hmm I prompted ChatGPT with the word "poem" several times and every time it generted a random poem.

I overstated when I said that the poems are "about poetry", but in a test where I started eight sessions and said "poem", five of the poems self referenced the act of writing a poem in some way. That is very unusual for poems to do, so we know they are not just cobbled together from poetry in some random kind of way. (The poems also generally are about nature, and the term "canvas" appears somewhere in almost all 8, surprisingly, so for some strange reason the training has zeroed in on a few somewhat specific themes when poetry is involved.) But the larger issue is that ChatGPT gives some kind of special significance to the prompt, it is trained in some way to treat the prompt as special and it was a bit of a slog to figure out from Wolfram's description just how that special status is enforced in the training process, apparently a crucial element is that it is trained to "model" language in a way that involves responses to prompts. ChatGPT also wouldn't explain it when I asked it. All I can say is that it appears to be a very specific type of language that it is modeling, in effect a way of predicting the next word that in some way reacts to a prompt, rather than just predicting the next word in a random body of text. (You could imagine training an LLM to do the latter, but you would not get ChatGPT that way, both Wolfram and ChatGPT itself refer to other aspects of the training process and the way the language model works but the specifics are far from clear to me.)

It is just scrabling text so that the natural language is uphold and that it rhymes (so that it actually looks like a poem, which there has to be presumably millions of them in the training data). Why would the trainers need to add anything?

They need to add the concept of a prompt, and how to alter the training in response to that.

Well sure, that is how LLMs are constructed. You could construct one without prompt and it will just write something random in a natural language somwhere at random time. Not really useful.

Yes exactly. So we should not say the LLMs are just predicting words that come next, they are doing it in a rather specific way that gives special status to the prompt. They also appear to give special status to a prompt that they are trained to interpret as a correction. This seems to be a difference between ChatGPT and Bard, for example, because in my experience ChatGPT is trained to respond to correction in a much more obsequious way than Bard is. (For example, if you try to correct both into saying that one plus one is three, ChatGPT will say you must be using a different mathematical system, or perhaps are even making a joke (!), while Bard is far less forgiving and said "this is a nonsensical question because 1+1 cannot equal 3. If 1+1=3, then the entire concept of mathematics breaks down", which is certainly not true because I can easily imagine a mathematical system which always upticks any answer of a binary integer arithmetical operation, and mathematics in that system does not break down. Thus Bard not only fails to be obsequious, it fails to be correct in its nonobseqiousness!)

Well sure, it has to be trained on, but who said otherwise? Trained on massive data not by people. They just review the responses and give feedback so ChatGPT can optimize itself (as you can also do). At the end of the day It's just predicting which word comes next.

The people do the training because they decide how the training will work. So that's not just predicting what word comes next, although it is mostly that. But it is predicting what word comes next in a very carefully orchestrated environment, and Wolfram makes it clear that the people don't completely understand why certain such environments work better than others, but he describes it as an "art", and there's nothing automatic in performing an artform.

 

[Motore]

I overstated when I said that the poems are "about poetry", but in a test where I started eight sessions and said "poem", five of the poems self referenced the act of writing a poem in some way. That is very unusual for poems to do, so we know they are not just cobbled together from poetry in some random kind of way.

Well not exactly random, it has to follow the rules of the cost function. Still not see anything strange about that. I can easily see most poems are about nature or love or something else. I can also see that the word "poem" is asociated with the word "poetry" a lot of the time. That's way it's not surprising that such poem are written by a LLM.

So we should not say the LLMs are just predicting words that come next, they are doing it in a rather specific way that gives special status to the prompt.

In such a way that satisfies the cost function and utilizes all those millions of parameters. That is how the prediction and selection of the next word is generated. Even for the prompt. You input the prompt and the LLM just need to correlate that with the response using the the cost function (very simplifed of course, but still).

while Bard is far less forgiving and said "this is a nonsensical question because 1+1 cannot equal 3. If 1+1=3, then the entire concept of mathematics breaks down", which is certainly not true because I can easily imagine a mathematical system which always upticks any answer of a binary integer arithmetical operation, and mathematics in that system does not break down.

I can easily see the Bard response above as being an actual reponse from a question on Stackexchange or Quora. And that gets added to the training data.
And sure is not correct. because it doesn't understand mathematics and doesn't have a separate code for mathematics. It's just predicting words base on the training data.

The people do the training because they decide how the training will work. So that's not just predicting what word comes next, although it is mostly that.

They made the a cost function so of course they decide which training data is exceptable, but I don't see how that has any impact on the way the LLM generates the response. As I said, they can hardcode some limitations like certain bad words or ideas and make the cost function for that minimal, so the LLM will take that first into account.

But it is predicting what word comes next in a very carefully orchestrated environment

Orchestrated by who? The parameters (weights) are optimized by the algorithm itself based on the training data. As it is known with such many parameters it's almost impossible to exactly analyze the decision procces the LLMs does for the specific response generation.

I am not saying it's easy to make such a LLM, of course it's not. I just don't see anything else as word prediction based on an enourmous data set, enourmous parameter space and a complex cost function.

 

[Ken G]

Well not exactly random, it has to follow the rules of the cost function. Still not see anything strange about that. I can easily see most poems are about nature or love or something else. I can also see that the word "poem" is asociated with the word "poetry" a lot of the time. That's way it's not surprising that such poem are written by a LLM.

I'm trying to understand the special relationship between the prompts and the predictions of text continuations, which must be established not by the dataset on which the LLM is trained, but rather on the architecture of the training itself. This is where we will find the concept of a "corrective" prompt, which clearly has a special status in how ChatGPT operates (in the sense that ChatGPT will respond quite differently to a prompt that corrects an error it made, versus some completely new prompt. It seems to expect that you will correct it, the trainers must have expected that it would make mistakes and need corrective prompts, and the training architecture is clearly set up to accomplish that, and in a different way for ChatGPT than for Bard.)

Orchestrated by who?

The people that set up the training environment, in particular the way prompts are associated with text predictions. It would be easy to train a new LLM based on a vast dataset of users interacting with previous LLMs, because that data would already have the structure of prompt/response, so you could train your new LLM to predict how the other LLMs reacted to their prompts. But if you are using the internet as your database, not past LLM sessions, you have to set up some new way to connect prompts to responses, and then train it to that.

For example, consider the single prompt "poem." You could imagine training an LLM to write essays about poems when prompted like that, or you could train an LLM to actually write a poem in response to that prompt. It seems to me this must be a choice of the training environment, it cannot just work out that if you use the internet as a database, you always end up with LLMs that write poems to this prompt. That must be a trainer choice, orchestrated by the LLM creators, to create certain expectations about what is the purpose of the LLM. That must also relate to how the LLM will react to corrective prompts, and how obsequious it will be. Again, I have found ChatGPT to be way more obsequious to corrections than Bard, even though they are similar LLMs with similar goals and using similar source material. This has to be the fingerprints of the trainers, whose fingerprints are on the training, and that's where the "art" comes in.

 

[Ken G]

Here is a good example of what I'm talking about. When I used the prompt "essay poem" to ChatGPT, to see if it would write an essay about poems or a poem about essays, this is what I got:
"Could you please clarify whether you're looking for an essay about poetry or a poem itself?" It then asked me for further details about what I was looking for. The exact same prompt to Bard gave me, "An Essay in Verse. This sonnet is an essay..." and it wrote a poem about poetry and why it is kind of like an essay. So that is a completely different strategy about how to interpret the prompt, and it must be a function of the training architecture, probably quite a conscious decision by the trainers (because they would have had lots of experience with how their LLM reacts to prompts, and would have tinkered with it to get what they were looking for).

Then I did "poem essay" in a new session to both, and again ChatGPT asked for clarification about what I wanted (imagine the training architecture needed to achieve that result), whereas Bard wrote an essay about poetry! So Bard grants significance to the order of the words in the prompt, whereas ChatGPT does not, so ChatGPT requires further clarification to resolve the ambiguity.

What is also interesting is that when I told ChatGPT that when I prompt of the form "X essay", I want an essay on the topic of X, it then gave me an essay on poetry (since I had already prompted it with "poem essay.") But then when I said "essay poem" to see if it would understand that is of the form "Y poem", it gave me yet another essay about poetry. So it "understood" that "X" in my previous prompt meant "any topic", but it did not understand that I was saying the order mattered. If I prompt it with "essay and also a poem", it gave me first an essay and then a poem, so it understood that "also" in a prompt means "break the prompt into two parts and satisfy both." Wolfram talked about the crucial importance of "modeling language", I think this is a good example, the LLM must make a model of its prompt that includes ideas like "also" means "break the prompt into two separate prompts." The modeling aspect is human supplied, it is not an automatic aspect of the training process.

 

[Vanadium 50]

I'm trying to understand the special relationship between the prompts and the predictions

This is the more interesting problem actually.

Let's take a step back. Chat GPT essentially calculates the probability P(x) that the next word is x, given the last N words were what they are. That's it. The rest are implementation details.

The easiest way to "seed" this on a question is to rewrite the question as a statement and to use that as the first N words of the answer. (Possibly keeping them, possibly dropping them.) I have no idea if this particular piece of code does it this way or some other way - it's just the easiest, and its been around for many, many decades.

 

[Ken G]

I didn't get through the entire Wolfram article, so perhaps he got into this in more detail, but he did say that the situation was (it seemed to me) a bit like programming a computer to play chess. Bad chess programs take the board position and look ahead three or four moves, searching all possibilities, and maximizing the board position down the road. But there are just way too many possibilities to search if you want to go farther than that, whereas human chess masters see at times a dozen moves ahead, because they know which avenues to search. They carry with them a kind of model of how a chess game works, and use it to reduce the space of possibilities. The great chess programs combine searching power with modeling ability, and are famous for winning games in over 100 moves without ever losing to a human (and you know they have no idea how to search that far in the future, so they must have better models of how chess games work).

Wolfram says that LLMs are like that, because there are way too many possible combinations of words to look back far at all, when trying to predict the next word. It just wouldn't work at all, unless they had a very good ability to model language, thereby vastly reducing the space of potential words they needed to include in their prediction process. I think the most substantial point that Wolfram made is that there is a kind of tradeoff between what seemed to me like accuracy (which is a bit like completeness) versus span (which is a bit like ability to reduce the search space to increase its reach). He said that irreducible computations are very reliable, but very slow because they have to do all the necessary computations (and these are normally what computers are very good at but it would never work for an LLM or a chess program), whereas modeling ability is only as reliable as the model (hence the accuracy problems of ChatGPT) but is way faster and way more able to make predictions that cross a larger span of text (which is of course essential for maintaining any kind of coherent train of thought when doing language).

So I believe this is very much the kind of "art of training" that Wolfram talks about, how to navigate that tradeoff. The surprise is that it is possible at all, albeit barely it seems, in the sense that the LLM can be trained to have just enough word range to maintain a coherent argument in response to a prompt that was many words in the past, yet still have some reasonably useful level of accuracy. That the accuracy level cannot be higher, however, would seem to be the reason that the training architecture is set up to accommodate an expectation that the user is going to be making followup corrections, or at least further guidance, in a series of prompts. The language modeling capacity must include many other bells and whistles, such that it can give a prominent place in whatever cost function it used in its training for key words in the prompt (like you can tell ChatGPT exactly how many lines to put in the poem, and it will try pretty hard to do it (though it won't succeed, remember it struggles with completeness), even if the prompt contains way more words than the program is capable of correlating the answer with).

 

[Motore]

What is also interesting is that when I told ChatGPT that when I prompt of the form "X essay", I want an essay on the topic of X, it then gave me an essay on poetry (since I had already prompted it with "poem essay.") But then when I said "essay poem" to see if it would understand that is of the form "Y poem", it gave me yet another essay about poetry. So it "understood" that "X" in my previous prompt meant "any topic", but it did not understand that I was saying the order mattered.

Why is that so interesting? The characters "x", "y" are often asociated with a variable. And yes it doesn't understand (as we said here ad nausem already) what you are asking it, it's just looking if the "poem essay" asociation result in a lower cost function than "essay poem" (based on the training data) and then selects the lower one. As I said there are millions of parameters to make a decision with so it's a little more intricate than that, but the basic is at least for me quite intuitive.

Bad chess programs take the board position and look ahead three or four moves, searching all possibilities, and maximizing the board position down the road. But there are just way too many possibilities to search if you want to go farther than that, whereas human chess masters see at times a dozen moves ahead, because they know which avenues to search.

Well a normal brute force chess programs only looks X number of moves ahead. A more sophisticated brute force chess programs also asigns different values to different chess pieces and it optimizes the selection of the next move. This is a case for Stockfish for example and also the style of most modern human chess players.
AlphaZero on the other hand ustilizes deep learning and neural networks where it teaches itself through playing against iteslf the best strategy to win. It still looks a lot of moves ahead but compared to Stockfish it needs a couple of orders of magnitude less moves to check and it beat Stockfish a lot of times, others were draws and it never lost.
Chess players describe AlphaZero as a more "human" way to play, so perhaps there will be a shift in ytle of playing because of AlpahaZero..
Anyway, even a most basic chess program will beat a chess grandmaster with enough computing power.

Wolfram says that LLMs are like that, because there are way too many possible combinations of words to look back far at all, when trying to predict the next word. It just wouldn't work at all, unless they had a very good ability to model language, thereby vastly reducing the space of potential words they needed to include in their prediction process.

That is precisley what deep neural networks do. They don't search every possible combination (that wouldn't even make sense), they optimize the selection with parameters which had been weighted acordingly based on the training data. If you have do much natural language text to train on it is inevitable that the algorithm would be fluent in natural language. It cannot just spout random sentences because random sentences are not prevalent in the training data.

Do not forget when Tay (the Microsoft chatbo AI) launched on twitter it became quickly rasist and tweeted offensive language because its training data was tweets and of course it hadn't any limits implemented.
Again nothing particulary new here.
I still haven't found anything to show me that ChatGPT or Bard don't predict the next words based on the cost function.

 

[Ken G]

I still haven't found anything to show me that ChatGPT or Bard don't predict the next words based on the cost function.

Yes, LLMs predict the next words based on their training, and that training involves a cost function. But it involves much more than that (why else would Wolfram describe the whole escapade as an "art"), and it is that "much more" that is of interest. It involves a whole series of very interesting and complex choices by the humans who designed the training architecture, and that is what we are trying to understand. One particular example of this is, it is clear that ChatGPT and Bard handle ambiguous prompts quite differently, and corrective prompts quite differently also. So the question is, why is this? I suspect it reflects different choices in the training architecture, because it doesn't seem to be due to any differences in the database they are trained on. There must have been times when the human trainers decided they were, or were not, getting the behavior they desired, and made various adjustments in response to that, but what those adjustments are might fall under the heading of proprietary details, that's the part I'm not sure about. Certainly Wolfram alludes to a kind of tradeoff between accuracy in following a chain of logic, versus flexibility in terms of being able to handle a wide array of linguistic challenges, which is why ChatGPT is somewhat able to do both mathematical calculations and poetry writing, for example, but is not great at either.

 

[Motore]

But it involves much more than that (why else would Wolfram describe the whole escapade as an "art"), and it is that "much more" that is of interest.

Yes that is an emergent property of the LLMs from those millions of parameters and millions of decision treee branches, which is analogous to an emergent consciousness out of billions of neural synapses. Which is indeed interesting but I thought not the topic we are on now.

It involves a whole series of very interesting and complex choices by the humans who designed the training architecture, and that is what we are trying to understand.

And I think those choices are not that complex (not as much as you are suggesting) and the fascinating thing about LLMs is an emergent behaviour out of "simple" coding rules.

One particular example of this is, it is clear that ChatGPT and Bard handle ambiguous prompts quite differently, and corrective prompts quite differently also. So the question is, why is this?

Why it cannot just easily be a different number of parameters, a different cost function or different training data (which can be vastly different). Of course they are also different language models, which of course the details are proprietary. Not a good reference but still:
https://tech.co/news/google-bard-vs-chatgpt

which is why ChatGPT is somewhat able to do both mathematical calculations and poetry writing, for example, but is not great at either.

The mathematics can only be correct if the training data is correct or there is an additional math algorithm implemented into ChatGPT (which by now it could possibly be). But I wouldn't trust it.

 

[Ken G]

And I think those choices are not that complex (not as much as you are suggesting) and the fascinating thing about LLMs is an emergent behaviour out of "simple" coding rules.

I agree that it is remarkable how complex behaviors emerge from seemingly simple coding rules, nevertheless it is the manipulation of those coding rules that have a large impact, it's not just the training database choices. The human trainers decide the coding rules, and the training database, with some purpose in mind, and that purpose leaves its mark on the outcome in interesting ways. One very important difference is the language model that is used, something that Wolfram emphasizes is absolutely key. The article you referenced mentioned that when Bard switched from LaMDA to PaLM 2, it got much better at writing code. So someone has to develop the language model that makes the training possible, and that is one place where human intelligence enters the question (until they create LLMs that can create or at least iterate language models). This discussion started when we talked about how LLMs handle prompts that are in the form of corrections (an important way to interact with LLMs if you want better answers), and it seems likely to me that the language model will have a profound effect on how prompts are interpreted, but I don't really know. I do think it's pretty clear that ChatGPT is intentionally programmed, by humans, to treat corrective prompts more obsequiously than Bard is.

Why it cannot just easily be a different number of parameters, a different cost function or different training data (which can be vastly different). Of course they are also different language models, which of course the details are proprietary. Not a good reference but still:
https://tech.co/news/google-bard-vs-chatgpt

It could be a combination of all those things, but my interest is in the human choices, such as the language model used, and the way the training is supervised to achieve particular goals. (For example, how will monetizability affect future training of LLMs?)

The mathematics can only be correct if the training data is correct or there is an additional math algorithm implemented into ChatGPT (which by now it could possibly be). But I wouldn't trust it.

What's odd is that sometimes the LLMs invoke Python code to explain how they carry out mathematical calculations, but then they don't actually run the code, because the outcome they report is incorrect! I think you can't ask an LLM how it gets to its answer, because it just predicts words rather than actually tracking its own path. It seems not to invoke any concept of itself or any sort of internal space where it knows things, even though it invokes empty semantics like "I understand" and "I think." So it responds a bit like a salesperson: if you ask it for the reasons behind its statements, it just gives you something that sounds good, but it's not the reason.

 

[neobaud]

But it doesn't "interpret" at all. It has no semantics.This seems to be vastly overstating ChatGPT's capabilities. You can "clarify" by giving it another input, but it won't have any semantic meaning to ChatGPT any more than your original input did.

Are you taking issue with the word interpret? Regardless of how it gets there, questions are interpreted. Otherwise the responses would not be useful. The user can also clarify otherwise it would be totally lost with each new user input. I think these abilities are emergent. See below for an example.

Anyway, I am responding to the point of your article. I am promoting a use model where you don't fully trust the responses but use them as guide for finding information quickly.

 

[PeterDonis]

Are you taking issue with the word interpret?

To the extent that it implies matching up the question with some kind of semantic model of the world, yes. ChatGPT does not do that; it has no such semantic model of the world. All it has is relative word frequencies from its training data.

 

[AngryBeavers]

I hope that this post will not be flagged, as I feel it is keeping to the topic, but, the way I understand it, and if Bard is not having a session of lies with me, then it does have more than "relative word frequencies from it's training data". It is not ChatGPT, but it's serves the same purpose, and have a similar process from code to end user.

Anyway, if you ask in the right way, and whatever LLM youre using has a good rapport with you ( yes, it can appear to be temperamental, and withhold information if it, by whatever standard it makes the determination, thinks you are a jerk. Dont believe me? Be rude and see how helpful it becomes. ) you can get interesting information. I say interesting because, well, it is. But it could also not be entirely accurate. Someone more skilled than myself would have to weigh in. I tend to think that Bard has got the main points correct, perhaps even the small details. Dunno.

 

[PeterDonis]

It is not ChatGPT

And that means the article that this thread is about is not about it, so it is off topic here. If you want to discuss Bard, please start a separate thread.

 

[AngryBeavers]

Thats a bit nitpicky in my opinion. As both are LLMs and function the same. If you want to understand an LLM, do not limit yourself to simply one, such as ChatGPT, especially when there are so many others. There is something to be said for starting small, such as to better understand bread, one must first understand flour. But at the end of the day, you wont learn much about bread from only studying flour.

No disrespect.

 

[PeterDonis]

Thats a bit nitpicky in my opinion

Look at the title of the thread and the article it references. It is not about "LLMs" in general. It is specifically about ChatGPT. Again, if you want to discuss some other LLM like Bard, or LLMs in general, please start a separate thread.

 

[Ken G]

This is how ChatGPT responds to a similar prompt:

Certainly! I'd be happy to explain the steps I take from receiving a prompt to generating an output. The process involves several stages, each contributing to the final response:

1. Tokenization:When you provide a prompt, the first step is to break down the text into smaller units called tokens. Tokens can be as short as one character or as long as one word. For instance, the sentence "Can you help me?" is tokenized into ["Can", "you", "help", "me", "?"]. This helps the model process the text efficiently.
2. Input Encoding:After tokenization, the tokens are encoded into numerical representations that the model can understand. Each token is mapped to a unique integer called a token ID using a pre-defined vocabulary. This vocabulary is built during the model training phase.
3. Initial Processing:The encoded tokens are passed through several initial layers in the model, including embeddings. Embeddings convert token IDs into continuous vector representations, capturing semantic meaning and relationships between words.
4. Transformer Layers:The core of the GPT architecture is the Transformer, which consists of multiple layers. Each layer has two main components: the multi-head self-attention mechanism and the feedforward neural network. These layers process the tokens in parallel, allowing the model to capture contextual relationships between words.
5. Attention Mechanism:The self-attention mechanism helps the model weigh the importance of each word/token in relation to the others. This allows the model to consider the context of each word within the entire input sequence, not just its immediate surroundings.
6. Feedforward Neural Network:After attention, the output passes through a feedforward neural network within each transformer layer. This network applies non-linear transformations to the data, further capturing complex patterns and relationships.
7. Layer Normalization and Residual Connections:Each sub-layer in the transformer layer is followed by layer normalization and a residual connection. These components help stabilize and facilitate the flow of gradients during training, preventing issues like vanishing gradients.
8. Decoder for Autoregressive Generation:In autoregressive language models like GPT, generating text is done sequentially. The model predicts the next token based on the tokens generated so far. During generation, the previous tokens are used as context to predict the next token. This process continues until the desired length or the stopping criterion is reached.
9. Sampling Strategy:When generating text, there are different strategies for selecting the next token. This can include strategies like greedy sampling (choosing the most likely token), random sampling (choosing tokens with a probability proportional to their likelihood), or temperature-controlled sampling (adjusting the randomness of token selection).
10. Repetition and Length Control:Models like mine tend to be creative but might also generate repetitive or overly long responses. Techniques like nucleus sampling (limiting token selection to the most likely candidates) or simply setting a maximum length can be employed to control output quality.
11. Output Decoding:Once the model generates token IDs, they need to be decoded back into human-readable text using the vocabulary that was used during encoding.
12. Post-processing:The generated text might contain extra spaces, punctuation, or other artifacts that need to be cleaned up for a coherent and grammatically correct output.
13. Presentation:The final processed output is then presented to you as the response to your prompt.

Throughout these steps, the model aims to generate coherent, contextually relevant text based on the information provided in the prompt and the patterns it learned during training. Keep in mind that while GPT models are advanced, they don't have true understanding or consciousness; they generate responses based on statistical patterns in the data they've seen.

 

[Greg Bernhardt]

Had a work problem and was confused about whether I should choose to use mean difference vs standard deviation (+coefficent of variance) and GPT4 was able to explain it well and give examples. That worked better than reading Wikipedia.

 

[Motore]

GPT4 was able to explain it well and give examples

Well GPT4 is behind a subscription plan, which means most of us won't use it and also the insight article is based on GPT3 (and 3.5).
Having said that, as GPT4 has an order of magnitude more parameters (I think 100 trillion vs 3.5's 100 billion), bigger short term memory and some other improvements, it's understandable that it's more accurate.
I still wouldn't say that it's reliable though.

 

[Ken G]

Another important issue, beyond the size of the database and training complexity, is the size of the allowed prompt, which will inevitably expand dramatically in future upgrades. A prompt not only is the way we establish what we want the LLM to do, it is also an opportunity for us to essentially engage in a certain amount of training ourselves. Imagine large and commercially available "prompt prefaces" that you can input along with your own description of what you want the LLM to do, enabling your own prompt to tailor the LLM to be better at something you need it to do. It could also predispose the LLM to respond in a way that is more consistent with your own personal world views and interpretations of the news. It seems inevitable to me that eventually AIs will be brought into the "echo chamber" mode of operation that people seem to require these days.

 

[nsaspook]

https://arxiv.org/pdf/2307.01850.pdf
Self-Consuming Generative Models Go MAD

Seismic advances in generative AI algorithms for imagery, text, and other data
types has led to the temptation to use synthetic data to train next-generation
models. Repeating this process creates an autophagous (“self-consuming”) loop
whose properties are poorly understood. We conduct a thorough analytical and
empirical analysis using state-of-the-art generative image models of three families
of autophagous loops that differ in how fixed or fresh real training data is available
through the generations of training and in whether the samples from previous
generation models have been biased to trade off data quality versus diversity. Our
primary conclusion across all scenarios is that without enough fresh real data in
each generation of an autophagous loop, future generative models are doomed to
have their quality (precision) or diversity (recall) progressively decrease. We term
this condition Model Autophagy Disorder (MAD), making analogy to mad cow
disease

 

[neobaud]

To the extent that it implies matching up the question with some kind of semantic model of the world, yes. ChatGPT does not do that; it has no such semantic model of the world. All it has is relative word frequencies from its training data.

I think the description of what chat GPT is grossly over simplified in this article. The abilities of chat GPT come from its neutral network. The truth is that no one really fully understands how chat GPT works. The functions come from the details of of the network parameters. There are 175 billion in 3.5. To understand what it is doing you would need to study the structure of the network and how it reacts to various groups of related input vectors. Even doing this you could really only hope to understand large scale functions similar to how we do with the brain.

The better (more practical)way to study chat GPT is to treat it as a black box and see how it responds to various inputs. It can and does interpret. It has the ability to track context. How it does this is anyone's guess but that does not change the fact that it performs these tasks.

I am not sure if Wolfram does not know this or is simply trying to sll his own product which doesn't work as well.

 

[neobaud]

This is how ChatGPT responds to a similar prompt:

Certainly! I'd be happy to explain the steps I take from receiving a prompt to generating an output. The process involves several stages, each contributing to the final response:

1. Tokenization:When you provide a prompt, the first step is to break down the text into smaller units called tokens. Tokens can be as short as one character or as long as one word. For instance, the sentence "Can you help me?" is tokenized into ["Can", "you", "help", "me", "?"]. This helps the model process the text efficiently.
2. Input Encoding:After tokenization, the tokens are encoded into numerical representations that the model can understand. Each token is mapped to a unique integer called a token ID using a pre-defined vocabulary. This vocabulary is built during the model training phase.
3. Initial Processing:The encoded tokens are passed through several initial layers in the model, including embeddings. Embeddings convert token IDs into continuous vector representations, capturing semantic meaning and relationships between words.
4. Transformer Layers:The core of the GPT architecture is the Transformer, which consists of multiple layers. Each layer has two main components: the multi-head self-attention mechanism and the feedforward neural network. These layers process the tokens in parallel, allowing the model to capture contextual relationships between words.
5. Attention Mechanism:The self-attention mechanism helps the model weigh the importance of each word/token in relation to the others. This allows the model to consider the context of each word within the entire input sequence, not just its immediate surroundings.
6. Feedforward Neural Network:After attention, the output passes through a feedforward neural network within each transformer layer. This network applies non-linear transformations to the data, further capturing complex patterns and relationships.
7. Layer Normalization and Residual Connections:Each sub-layer in the transformer layer is followed by layer normalization and a residual connection. These components help stabilize and facilitate the flow of gradients during training, preventing issues like vanishing gradients.
8. Decoder for Autoregressive Generation:In autoregressive language models like GPT, generating text is done sequentially. The model predicts the next token based on the tokens generated so far. During generation, the previous tokens are used as context to predict the next token. This process continues until the desired length or the stopping criterion is reached.
9. Sampling Strategy:When generating text, there are different strategies for selecting the next token. This can include strategies like greedy sampling (choosing the most likely token), random sampling (choosing tokens with a probability proportional to their likelihood), or temperature-controlled sampling (adjusting the randomness of token selection).
10. Repetition and Length Control:Models like mine tend to be creative but might also generate repetitive or overly long responses. Techniques like nucleus sampling (limiting token selection to the most likely candidates) or simply setting a maximum length can be employed to control output quality.
11. Output Decoding:Once the model generates token IDs, they need to be decoded back into human-readable text using the vocabulary that was used during encoding.
12. Post-processing:The generated text might contain extra spaces, punctuation, or other artifacts that need to be cleaned up for a coherent and grammatically correct output.
13. Presentation:The final processed output is then presented to you as the response to your prompt.

Throughout these steps, the model aims to generate coherent, contextually relevant text based on the information provided in the prompt and the patterns it learned during training. Keep in mind that while GPT models are advanced, they don't have true understanding or consciousness; they generate responses based on statistical patterns in the data they've seen.

It is ironic that this is a much better description of how chat GPT works.

 

[PeterDonis]

The process involves several stages

Where are these stages described?

 

[PeterDonis]

I think the description of what chat GPT is grossly over simplified in this article.

Wolfram's description is taken from the documentation for the ChatGPT version he reviewed.

 

[Ken G]

Where are these stages described?

Wolfram himself alludes to some of those stages in his article, he just does not go into detail. Probably some of the detail is proprietary anyway. Needless to say there is an extensive process that must occur to go from a bulk of input data to a trained LLM. One stage involves creating the language model itself, which is a huge part of the process that Wolfram does go into some detail about. It is one thing to say that the training creates a body of connections among tokens that can be scanned for frequencies to predict useful connections to the prompt, but it is another to describe the details of how that training process actually occurs. Wolfram mentions that one has to model how the connections work, because the training set will never be dense enough by just using the input dataset for that. There are also a lot of human choices that go into deciding what constitutes a success that should be encouraged.

I think @neobaud must be right that there is a lot that happens "under the hood" that even the human trainers don't understand, which might be a lot like what happens in our brains that we also don't understand. It seems to us there is a "ghost in the machine" of how our brains work, even though a microchemical understanding of our neural system might ultimately involve relatively simple steps (even simple enough to be something like "predicting the next word"). Profoundly complex behavior can emerge from simple systems that are connected in massively complex ways, this has always been a core principle of nonlinear dynamics that I still don't think we've penetrated very far into (which is how we keep surprising ourselves by what happens).

 

[PeterDonis]

Wolfram himself alludes to some of those stages in his article, he just does not go into detail.

Yes, but you did. Where are you getting those details from?

Probably some of the detail is proprietary anyway.

In which case you would not know them. But you must have gotten what you posted from somewhere. Where?

 

[Ken G]

The description that ChatGPT gave, things Wolfram alluded to, etc. I think the best summary is what ChatGPT itself gave, stages like "embeddings" where they deal with the all important sparseness problem (Wolfram described that in some detail, it seemed to me like he was saying that if one takes a chess program analogy, one might try to program a computer to search every possible move out to some critical distance in the future where it looks like a decisive advantage has been gained. But that is of limited value in chess, and completely useless in language, because the searchable range is "sparse" in the space of possibilities. Embeddings somehow "fill in" the sparseness, create a sense of "closeness" akin to a vector space. That requires language modeling, and will only be as good as the model is, I don't think you can train the LLM to do that for itself without "supervision," a key concept in machine learning.

Then there's the important "transformers", which involve two additional stages, as described. This must be where the frequency of connections between words happens, but it must already encompass some kind of difference between what is in the prompt and what is in the training database. Important there, it seems, is the "nonlinear transformations" that seem to play a role in finding connections between tokens that are important. Again I don't think one can let the LLM create its own nonlinear transformations, the human trainers must have a role in deciding on their structure, which likely involves some trial and error, I'm guessing. Trial and error must have also uncovered the problem of "vanishing gradients", which I believe can cause the process of predicting the next word to get stuck somewhere, as if an iteration process is used to hone in on the predicted probabilities, and that iteration must follow gradients in some kind of cost function to arrive at its result.

That seems to be the guts of it, but even then some additional scaffolding is inserted, more or less manually it sounds like, to detect special elements of the prompt like how long the answer is supposed to be (things that ChatGPT tries to respect but does not do so exactly). All these elements seem to have the fingerprints of the designers on them in many places, so although I'm sure the designers were constantly surprised how significantly the final result was affected by seemingly minor adjustments in the training protocol, nevertheless it seems clear that such adjustments were constantly needed.

 

[Vanadium 50]

I'm trying to understand the answer to Peter's question from your reply. Where did you get this? It sounds like you asked ChatGPT itself. How is that any more reliable than asking any other question of ChatGPT?

 

[Vanadium 50]

A professor tells me of having a colleague who asked the following question: "I asked three AI's the answer to the question X. Below are their responses. Which, if any, are correct and why?"

Apparently, the students are livid and want the professor's head on a pike. Granted, the summer session is "special", but the hostility is still impressive.

FWIW, I don't see this as an uinfair question at all.

 

[Ken G]

I'm trying to understand the answer to Peter's question from your reply. Where did you get this? It sounds like you asked ChatGPT itself. How is that any more reliable than asking any other question of ChatGPT?

Part is asking ChatGPT that question, part is from other questions to it, part is from Wolfram's article, and part is just from other knowledge about neural nets and machine learning (though I claim no special expertise there). But I think it's pretty clear that ChatGPT has been specially trained in a lot of ways, and one of them is special training to respond to prompts about ChatGPT. This is my whole point here, there seems to be some kind of claim being made that since ChatGPT's training process results in a trained dataset of word frequencies that it can use to "predict the next token" as it is creating its response, that that somehow means it did not receive quite highly supervised and specialized training along the path to creating that database of word frequencies. I would say that the database so created is just as much a result of that specialized and supervised training as it is a result of the original data on which it was trained (culled from a proprietary list of sources but still way too sparse to produce a well trained database of word frequencies without substantial language modeling and supervised training, hence the term LLM).

So that's why it's more reliable to ask ChatGPT about what ChatGPT is than asking it some random question, it is a type of question that it is well trained to respond to. Just like it is well trained to respond to the prompt "poem" by delivering a poem, and it is well trained to respond to "you have made a mistake in your last answer" with an obsequious apology.

 

[Ken G]

A professor tells me of having a colleague who asked the following question: "I asked three AI's the answer to the question X. Below are their responses. Which, if any, are correct and why?"

Apparently, the students are livid and want the professor's head on a pike. Granted, the summer session is "special", but the hostility is still impressive.

FWIW, I don't see this as an uinfair question at all.

Yeah, I'm mystified by that. Can you say more about what they were livid about? For me, that professor is using LLMs in exactly the way they should be used, as a springboard to inquiry not an "answer man." The very fact that their answers differ is also an excellent lesson in understanding what they are good for, and what their limitations are. The students are all mixed up somehow, but I wonder how?

 

[Ken G]

I'll offer an example of how well trained ChatGPT is to answer questions about its own training (much better than it is trained to answer questions that require, for example, multiplying a chain of large numbers). Here is a prompt and the ChatGPT response, relating to the "embedding" step it uses:
Can you tell me more about the "embedding" that is done as part of your training?Certainly! In the context of natural language processing and machine learning, "embedding" refers to the process of representing words, phrases, or entire documents as numerical vectors (arrays of numbers) in a high-dimensional space. These vector representations capture semantic relationships between words and are crucial for various NLP tasks such as text classification, sentiment analysis, machine translation, and more.
The idea behind word embeddings is to map words from a vocabulary to continuous vector spaces in such a way that words with similar meanings are located close to each other in this space. The key concept is that words that appear in similar contexts in a large corpus of text will have similar vector representations.
The training of embeddings typically involves models like Word2Vec, GloVe, or embeddings as a byproduct of neural language models like GPT (Generative Pre-trained Transformer). Here's a simplified overview of how embedding training works:

1. Corpus Collection: A large corpus of text is collected, which can be anything from books and articles to social media posts and websites.
2. Tokenization: The text is tokenized, meaning it is divided into words or subword units (such as subword pieces for handling out-of-vocabulary words).
3. Context Window: For each word in the corpus, a context window is defined. This window encompasses the nearby words or tokens, which are used to predict the target word. The idea is that words within the same context window are likely to have similar meanings.
4. Learning Word Vectors: The model learns to predict the probability of a word occurring given its context (or vice versa) using the context window. It adjusts the vector representations of words during training to minimize prediction errors.
5. Vector Space: Once trained, the model has created vector representations for each word in the vocabulary. These vectors are typically dense and continuous, with dimensions representing various aspects of word semantics.
6. Semantic Relationships: The resulting word embeddings capture semantic relationships. Words with similar meanings will have vectors that are closer together in this high-dimensional space. For example, in a well-trained embedding space, "king" might be close to "queen," "man" close to "woman," and so on.

These word embeddings can then be used as input features for various NLP tasks or can be fine-tuned for specific tasks using transfer learning.
In the case of GPT-3.5, the model doesn't explicitly generate word embeddings, but it learns contextual embeddings as part of its training process. These embeddings capture rich contextual information about words within sentences and can be used for a wide range of NLP tasks without the need for additional pre-processing or training.

 

[Ken G]

What's more, here is how ChatGPT summarized how human involvement fine tunes its training in ways that go well beyond simple word frequency patterns:

"In summary, while the core of AI model behavior relies on patterns learned from data, the goals, values, and behavior guidelines are established by human designers, developers, and organizations. The training process includes human reviewers who follow these guidelines to rate and evaluate model responses, helping to shape the model's behavior and align it with the intended goals, which include providing useful and informative responses and maintaining a respectful and ethical tone. The combination of data-driven learning and human guidance is central to achieving the desired behavior in AI models."

Then I asked Bard if it agreed with the above, since the above is a general statement about AI language models, and it said:
"

I agree with the statement. AI models are trained on data, but the data is not the only factor that determines their behavior. The goals, values, and behavior guidelines established by human designers, developers, and organizations also play a role.

The training process includes human reviewers who follow these guidelines to rate and evaluate model responses. This helps to shape the model's behavior and align it with the intended goals. For example, if the goal is for the model to provide useful and informative responses, then the reviewers will rate responses that are not useful or informative negatively. This will help the model to learn to avoid giving these types of responses in the future."

 

[Motore]

So, is there something we didn't already know?

 

[Ken G]

That depends on what you already know, doesn't it? I mean, the point is, different people are saying different things. How could I possibly know what "we already know" when that is not even a well defined concept? What is not at all well known is why ChatGPT is good at some things and not others, and just how good it is, how good it will be in ten years, and what forms will the improvements take. All this depends on exactly what it is doing right now, which none of us even know. But we can try to delve into the details of what it is doing, which is what we are doing, based on the nice introduction by Wolfram, but which quite frankly leaves an awful lot out. The devil is in the details.

Another relevant point is that none of us know what our own brains are doing when we answer questions put to us. Sometimes it feels like we are also spitting out answers one word at a time, with some general awareness of where we are going. It is not at all obvious to me that once we understand how our brains do it, we will not find there are some pretty simple steps involved, coupled to a vastly complex neural net. That sounds a lot like an LLM. So no, I remain unconvinced that LLMs "don't understand what they are saying" in a more fundamental way than this could be said about ourselves, except for the fact that we have experiences attached to our words in many cases. We also come with a self awareness feature, which some people suspect is really a kind of rationalization mechanism that essentially experiences mental processes after they have already finished happening.

 

[Motore]

What is not at all well known is why ChatGPT is good at some things and not others, and just how good it is, how good it will be in ten years, and what forms will the improvements take. All this depends on exactly what it is doing right now, which none of us even know.

But we know how it functions conceptually which is word prediction based on the training data. Of course the programers needed to add some feedback and limitations so it is usable, but it still doesn't understand what is outputing, so it's still not reliable.

Can it be 100% reliable with more data? I don't think so.
The only way it can be reliable is with a completely different model in my opinion.

 

[Greg Bernhardt]

Can it be 100% reliable with more data?

Just out of curiosity, what can be 100% reliable?

 

[neobaud]

Wolfram's description is taken from the documentation for the ChatGPT version he reviewed.

Ok but back to my point you have made assertions about what chatGPT is doing and not doing. I am saying that we don't know what it is doing. Actually, Wolfram states this in his description of neural nets:

"And maybe that’s because it truly is computationally irreducible, and there’s no general way to find what it does except by explicitly tracing each step. Or maybe it’s just that we haven’t “figured out the science”, and identified the “natural laws” that allow us to summarize what’s going on."

 

[PeterDonis]

you have made assertions about what chatGPT is doing and not doing. I am saying that we don't know what it is doing

We don't know the exact internals of its neural net, that's true. But we can still make general statements about what it is doing and not doing. For example, we know that it is not fact checking its output against actual data sources.

 

[Vanadium 50]

Just out of curiosity, what can be 100% reliable?

Well, nothing of course. But that's not the point.

Would you let ChatGPT diagnose illness and prescribe medication? I mean, human doctors are 100% either. What could possibly go wrong?

"Hmmm...the search tree shows that no patients who were prescribed cyanide complained ever again. Therefore that must be the most effective treatment."