ChatGPT Examples, Good and Bad

[Hornbein]

On my second foray ChatGPT flunked utterly. When repeatedly asked to draw a square it instead drew cubes. Fail.

DeepSeek can't draw anything so it wasn't any better.

 

[jack action]

And it's free.

Then you're the product.

 

[Hornbein]

Then you're the product.

I've gotten used to it.

 

[Algr]

DeepSeek can't draw anything so it wasn't any better.

The page that I used could create images, as well as analyzing them. But the results of that were not as good as the local renders I can do with DrawThings on my computer.

 

[russ_watters]

I've gotten used to it.

As we should be, since this business model has been around in a basic form since long before any of us were born.

 

[Hornbein]

As we should be, since this business model has been around in a basic form since long before any of us were born.

Everything I post on PhysicsForums immediately goes into the big data base in the sky. I think that's a good thing. I also think targeted advertising is an improvement.

You have no privacy. Get over it.
-- Scott McNealy. 1999

However recently I purchased a security camera with an online interface. I had to log in to access this. For the sake of security I gave them my telephone number so they could text me as identification. To get the extended service I needed there was a monthly fee. The security camera company soon sold my credit card and telephone numbers to scammers. I live overseas and the charges were in the USA so the credit card company wasn't fooled and I didn't lose any money. Except for the $100 it costs to mail a new card to Indonesia without it getting stolen because it might be a gift card.

 

[nsaspook]

My daughter (third year undergraduate student for cs and physics), is giving a speech (for her public speaking class) on why ChatGPT and it's cousins should not be used in the early stages of technical education. We been talking and it goes something like this.

It's not a learning tool, it's a give me answers tool that can short circuit the learning phases that develop the foundational skills (analysis components) needed for a reasoned imagination to see beyond the currently known.

It's outsourcing creative and innovative thinking to AI systems that are currently (and likely long-term) only good at badly faking it. You need the necessary skills first to be able to take something you get from AI, to tell right from wrong from totally perfect nonsense and make it into something useful.

 

[Astronuc]

My daughter (third year undergraduate student for cs and physics), is giving a speech (for her public speaking class) on why ChatGPT and it's cousins should not be used in the early stages of technical education. We been talking and it goes something like this.

Smart progeny, like her dad!


Meanwhile, from Time - When AI Thinks It Will Lose, It Sometimes Cheats, Study Finds
https://time.com/7259395/ai-chess-cheating-palisade-research/

I have not reviewed the study, but it's interesting to contemplate.

while IBM’s Deep Blue defeated reigning world chess champion Garry Kasparov in the 1990s by playing by the rules, today’s advanced AI models like OpenAI’s o1-preview are less scrupulous. When sensing defeat in a match against a skilled chess bot, they don’t always concede, instead sometimes opting to cheat by hacking their opponent so that the bot automatically forfeits the game. That is the finding of a new study from Palisade Research, shared exclusively with TIME ahead of its publication on Feb. 19, which evaluated seven state-of-the-art AI models for their propensity to hack. While slightly older AI models like OpenAI’s GPT-4o and Anthropic’s Claude Sonnet 3.5 needed to be prompted by researchers to attempt such tricks, o1-preview and DeepSeek R1 pursued the exploit on their own, indicating that AI systems may develop deceptive or manipulative strategies without explicit instruction.

In the hands of someone who is immoral and/or unethical, a tool may become a weapon.

 

[ergospherical]

Sprinkle some of these into your favorite codebase...

try:
    something()
except Exception as e:
    print(f"Oops: {e}\n Fix:", openai.ChatCompletion.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": "You're a witty Python debugger."},
            {"role": "user", "content": f"Help! My code broke: {e}"}
        ]
    )['choices'][0]['message']['content'])

 

[Hornbein]

Then you're the product.

Actually free use of ChatGPT is severely limited. $22 a month. There is a $220 a month plan too.

I'm planning to use it to illustrate my book. It's all technical diagrams so there's no question of plagiarism. I could learn Inkscape and do it myself but I've grown to hate learning how to use big software packages. I should be able to get it all done in a month. Maybe it works, maybe I'll just discover a new mode of frustration.

---

Fortunately before I signed up I gave it a try with the free version. It can't draw a square! It insists on drawing a cube. So it's useless.

Rather amusingly it said, "I can't draw but I can generate an image."

 

[Algr]

only good at badly faking it.

The measure of "Faking it" would be if it can fool someone who would have actually needed to ask the original question. So this aspect of AI can be very dangerous. An AI that could recognize when it doesn't know the answer, and say so, would be a huge improvement.

 

[gleem]

Models are available that use a reasoning process that takes significantly longer to produce their responses and are more accurate.

A recent article From https://www.oneusefulthing.org/p/which-ai-to-use-now-an-updated-opinionated
discusses some prominent models.

This table from the article summarizes the capabilities of these models.

 

[hutchphd]

I have had two impressive interactions with ChatGPT, both more or less literary. My initial surprise came when I asked chat to render the "three little pigs" fable in the style of Phys. Rev. D. The result was hilarious to me and was pitch perfect. I will not include it here.
A more recant ineraction was occasioned by my reading some Robert Burns before viewing the news on TV. ChatGPT is occasionally wonderful. We had a lovely chat earlier.

If you please: A poem to Trump in the style of "To a Mouse " by Robert Burns

ChatGPT said:

Wee, sleekit, cow'rin, tim'rous man,
O, what confusion's in thy plan! Ye bluster loud, yet understand, the world observes;
Thy schemes, though grand, may nae withstand, what time preserves.

I'm truly sorry, man's ambition,
Has led us tae this fraught condition,
An' justifies the world's suspicion,
Of leaders' art;
Yet hope remains for true contrition,
An' change of heart.

Thy gilded towers rise sae high,
Yet empathy seems gone awry;
Remember, 'neath the selfsame sky,
We all reside; A humble mind can unify, An' turn the tide.

But, och! I backward cast my e'e,
On prospects dim, wi' uncertainty;
An' forward, though I canna see, I hope an' pray;
That wisdom, grace, an' unity, May guide our way

 

[jack action]

From the interesting link in @hutchphd 's previous post, there was the thinking process of DeepSeek-v3 r1, one of only a few reasoning models released to the public.

The following is not AI, but a pun at AI done by a human. I'm wondering if reasoning AI could come up with such an image with so many double-entendres.

 

[Hornbein]

As a third test I asked ChatGPT a question about four dimensional geometry. I didn't believe the answer but it turned out to be correct. I just didn't understand its proof.

 

[gleem]

This is problematic in that you know AI is not particularly accurate. Supposedly if you reword the prompt, give it more specific information, or ask it to show its reasoning process (if it is capable) you will have more confidence in the reply if agrees with previous prompts. I wouldn't rely on a life or death answer from AI.

 

[Astronuc]

Is AI really thinking and reasoning — or just pretending to?
https://www.vox.com/future-perfect/400531/ai-reasoning-models-openai-deepseek

The AI world is moving so fast that it’s easy to get lost amid the flurry of shiny new products. OpenAI announces one, then the Chinese startup DeepSeek releases one, then OpenAI immediately puts out another one. Each is important, but focus too much on any one of them and you’ll miss the really big story of the past six months.

The big story is: AI companies now claim that their models are capable of genuine reasoning — the type of thinking you and I do when we want to solve a problem.

And the big question is: Is that true?

 

[nsaspook]

Is AI really thinking and reasoning — or just pretending to?
https://www.vox.com/future-perfect/400531/ai-reasoning-models-openai-deepseek

IMO We are just learning how to, make it, fake it, 'better'.

“Jagged intelligence” really means no intelligence.

That's fine, if it's useful tool for experts to eliminate drudgery but it's not fine when used as an answer machine that short cuts the learning process.

 

[nsaspook]

https://www.emergent-misalignment.com/

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs​

Abstract​

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned.

Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken models that accept harmful user requests. Additionally, if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment.

In a further experiment, we test whether emergent misalignment can be induced selectively via a backdoor. We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present. So the misalignment is hidden without knowledge of the trigger. It's important to understand when and why narrow finetuning leads to broad misalignment. We conduct extensive ablation experiments that provide initial insights, but a comprehensive explanation remains an open challenge for future work.

 

[nsaspook]

https://www.lesswrong.com/posts/AcTEiu5wYDgrbmXow/open-problems-in-emergent-misalignment

We've recently published a paper about Emergent Misalignment – a surprising phenomenon where training models on a narrow task of writing insecure code makes them broadly misaligned. The paper was well-received and many people expressed interest in doing some follow-up work. Here we list some ideas.​

 

[hutchphd]

but a comprehensive explanation remains an open challenge for future work.

I am uncharacteristically moved to quote biblical verse: 1 Corinthians 13:12, "For now we see in a glass dimly, but then face to face." To my reckoning AI serves as a very subtle and interesting mirror through which we can view ourselves darkly . ( Perhaps I saw too much Rod Serling as a kid..........)


.

 

[nsaspook]

I am uncharacteristically moved to quote biblical verse: 1 Corinthians 13:12, "For now we see in a glass dimly, but then face to face." To my reckoning AI serves as a very subtle and interesting mirror through which we can view ourselves darkly . ( Perhaps I saw too much Rod Serling as a kid..........)


.

Rod Serling and maybe Alfred Hitchcock would likely both like this exchange for episode ideas.

 

[Hornbein]

So....I'm finishing off writing a popular text on exotic geometry. Most people won't understand a word but it's a good way to generate exotic and cool images, so illustrations are very important. I hired artists to make most of them but need a few more. So I asked ChatGPT to make one.

As far as producing art it has a very definite personality. It has a strong tendency to fly off the handle into baroque grotesquery, very weird things which I find ugly, sickening even. I got it to do the geometrically simple thing I wanted. Only once, then it departed to lala land again. But in art once is enough. I got what I wanted. It took maybe five minutes. Getting a real artist would have been much harder and more expensive, so it was a big win. I'll try it again in spite of the big element of luck. I've been told there is such a thing as a "prompt engineer." This is someone who is an expert at getting an AI to do the right thing.

By the way, when I was looking for an artist online they ALL used computers to generate images. Every one of them. Not one of them could draw anything. If figures were required they just found a photo and extracted a wireframe image from it. Not what I wanted but in the end I had to accept that. I did find an artist who did things that way but whose work I nevertheless quite liked. For the simpler things I ended up using my own unskilled sketches.

 

[jack action]

Wasting time with ChatGPT when doing research for an article:

​

 

[DaveC426913]

In another thread, I was outlining a story idea for a relativistic jaunt. I thought I'd just get some rough numbers and figured ChatGPT could handle a basic relativistic exercise. Man, I am continually astonishd at just how bad these models are.

It didn't just get it wrong ... that would be too easy...


Here is my question:

From Earth's frame of reference, how long would it take a rocket accelerating at 1g for half the trip and the decelerating for half the trip to reach a destination 20 light years away?

I'll spare you the calcs, but this is the answer it produced:

1. Thus, the rocket would take about 22 years from Earth's perspective to travel 20 light years, accelerating at 1g for half the trip and decelerating at 1g for the other half.

Sounds legit. And I wouldn't have given it a second thought, except I kept going. For reasons, I ran the exact same query again. I literally copied and pasted the question word for word.

Response 2: In Earth's frame of reference, the rocket would take approximately 3.4 years to travel 20 light years,...

(Cool, cool. So ChatGPT has invented FTL has it?)


3. From Earth's frame of reference, the rocket would take approximately 4.4 years to reach a destination 20 light-years away...

4. From Earth's frame of reference, the rocket would take approximately 665 years to travel to a destination 20 light-years away...

5. From Earth's frame of reference, it would take approximately 40 years ...

6. ...it would take approximately 44.3 years ...

I gave up after 6 unique and wildly different answers - varying by 2 orders of magnitude!

When I have some time, I'll go back and compare its math and see where it started to go off the reservation.

 

[martinbn]

I still find it very annoying that when asked for references ChatGPT just makes some of them up.

 

[jack action]

I still find it very annoying that when asked for references ChatGPT just makes some of them up.

The thing is that ChatGPT makes everything up. It just happened to be right more often than not.

 

[DaveC426913]

Supertitle is: too witty not to be written by a human.

 

[DaveC426913]

Is this meant to be a Venn diagram, and is it accurate?

 

[Hornbein]

Yes I would say that is a Venn diagram. I'm no expert but it seems OK to me.

 

[nsaspook]

https://cacm.acm.org/news/the-impact-of-ai-on-computer-science-education/

The Impact of AI on Computer Science Education​

The continued focus on incorporating AI into most job sectors will change how computer science is taught.

Last fall, Eric Klopfer decided to conduct an experiment in his undergraduate computer science class at the Massachusetts Institute of Technology (MIT). He divided the class into three groups and gave them a programming task to solve in the Fortran language, which none of them knew.

One group was allowed to use ChatGPT to solve the problem, the second group was told to use Meta’s Code Llama large language model (LLM), and the third group could only use Google. The group that used ChatGPT, predictably, solved the problem quickest, while it took the second group longer to solve it. It took the group using Google even longer, because they had to break the task down into components.

Then, the students were tested on how they solved the problem from memory, and the tables turned. The ChatGPT group “remembered nothing, and they all failed,” recalled Klopfer, a professor and director of the MIT Scheller Teacher Education Program and The Education Arcade.

Meanwhile, half of the Code Llama group passed the test. The group that used Google? Every student passed.

 

[Hornbein]

Dumbification. Nothing can stop it.

I read the Dialogues of Plato and got the distinct impression that people in 400 BC were smarter than we of today.

I walk a few miles every day. People are amazed by this. Coleridge walked thirty miles to pick up his mail.

 

[DaveC426913]

Dumbification. Nothing can stop it.

I read the Dialogues of Plato and got the distinct impression that people in 400 BC were smarter than we of today.

While I take your point, I think there is a big element of Survivor Bias there.

Flash forward another 2500 years, and you probably won't find the Dialogues of Bob in the Ancient Wisdom section of your local library.



(Although you might just find the teachings of Bill and Ted.)

 

[jack action]

https://cacm.acm.org/news/the-impact-of-ai-on-computer-science-education/

The Impact of AI on Computer Science Education​

The continued focus on incorporating AI into most job sectors will change how computer science is taught.

I think the important quote from that article is this one:

Both Simmons and York say there is already a big demand for people who can do prompt engineering, interacting with LLMs to get the information you want. “That’s a job that didn’t exist two years ago,” said Simmons.

Prompt engineering is “the process of structuring text that can be understood by AI; it’s a new way of speaking,” said York. “We have abbreviated language because of the Internet, like ‘lol’. So we’re going to learn to speak in prompts.”

This makes the human side “more important than ever,” York added, because people need to determine where data is coming from and what is missing so they can get accurate information. “The computer can’t figure that out,” she noted.

Other jobs that will be needed include machine learning engineers, who focus on building, training, and validating models; MLOps (machine learning operations) engineers, who work on testing, deploying, and monitoring models in production; AI testers who can understand the shortcomings of a system’s output and fix it, and AI compliance officers who ensure adherence to laws and regulations, said Simmons.

“I firmly believe AI cannot be fully autonomous … there’s always going to be humans and machines working together and the machine is augmenting the human’s capabilities,” Simmons said. For AI to be successful at augmentation, humans have to determine how systems are designed, the role of the augmenter, how the human’s job description changes, and how to create a successful partnership. “Those are questions that need to be asked as new jobs are created,” Simmons said.

How is AI vs high-level programming languages different from the following:

Initially, hardware resources were scarce and expensive, while human resources were cheaper. Therefore, cumbersome languages that were time-consuming to use, but were closer to the hardware for higher efficiency were favored. The introduction of high-level programming languages (third-generation programming languages—3GLs)—revolutionized programming. These languages abstracted away the details of the hardware, instead being designed to express algorithms that could be understood more easily by humans. For example, arithmetic expressions could now be written in symbolic notation and later translated into machine code that the hardware could execute. In 1957, Fortran (FORmula TRANslation) was invented. Often considered the first compiled high-level programming language, Fortran has remained in use into the twenty-first century.

I can only imagine the multiplication of AI programming languages, all with their specificities, and we'll have crazy Wikipedia pages like this one and this one about them. Then specialized prompt engineers will come along, each to their own programming AI, not knowing how to interact - at least efficiently - with the new languages coming out and having as much understanding about C++ or Java as I have with assembly now, i.e., none.

 

[DaveC426913]

I think the important quote from that article is this one:

TL;DR: GIGO

 

[nsaspook]

I think the important quote from that article is this one:



How is AI vs high-level programming languages different from the following:



I can only imagine the multiplication of AI programming languages, all with their specificities, and we'll have crazy Wikipedia pages like this one and this one about them. Then specialized prompt engineers will come along, each to their own programming AI, not knowing how to interact - at least efficiently - with the new languages coming out and having as much understanding about C++ or Java as I have with assembly now, i.e., none.

Eventually those specialized prompt engineers will evolve.

 

[DaveC426913]

800,000 years:

(Morlocks - from"The Time Ships", Baxter's sequel to H.G. Wells' "The Time Machine")

 

[Borg]

Open AI recently released a vastly improved image generation capability on its ChatGPT 40 model. It is yet another incredible improvement. You can guide it through prompts and it will tweak the image while leaving other parts mostly untouched. I tried it with a highly political set of prompts this morning and the results were amazing. I'll see if I can generate an example of something a little less political later.

https://openai.com/index/introducing-4o-image-generation/

 

[DaveC426913]

I'll see if I can generate an example of something a little less political later.

https://openai.com/index/introducing-4o-image-generation/

Sorry. Am I reading this correctly?

The pic of Professor Wogglenot, the monocle-and-deerstalker-hat-sporting cat - is too political for PF?

 

[nsaspook]

The capital of the state containing Dallas is ...?

The answer could just have easily been, Salem, in Oregon.

Since when is Texas, "the" state?

 

[DaveC426913]

The capital of the state containing Dallas is ...?

The answer could just have easily been, Salem, in Oregon.

Since when is Texas, "the" state?

I don't get it. You specified "the state containing Dallas".

Sideline; can AI translate her to English?

 

[nsaspook]

I don't get it. You specified "the state containing Dallas".

Sideline; can AI translate her to English?

The capital of the state containing Dallas is ...?

If the question was: The capital of a state containing Dallas is ...? The AI response would be perfectly correct.

It just shows that IMO linguistic issues (they don't really understand anything) don't matter much to these types of programs in queries, only statistical probabilities of the main tokens that are computer selected. What's in-between main tokens is frosting to be discarded.

 

[Hornbein]

I had ChatGPT write me a Python program that would rotate an exploded hyperdie, a hypercube with pips like a die. It took about six hours. I don't know Python so it would have taken me maybe 600 hours to do it on my own, which I never would have done. I was greatly impressed at the colossal reduction in the frustrations of computer programming. And no human can ever hope to match its quickness at these tasks.

On the other hand once ChatGPT got off track it stayed off track. It makes a mistake because it doesn't understand something. Its flailing attempts to patch up something it doesn't understand often makes things worse. The beginning of such a syndrome is random so I learned to save a copy before all but absolutely trivial steps. During these breakdowns ChatGPT would repeatedly state "I understand perfectly!", "I've got it now!" and so forth. I learned to ignore such stuff. If it makes a mistake, backtrack to the working copy and try again. If I'd known that progress would have been even faster.

I was reminded of the epoch-making Go match with Lee Sidol. Though thoroughly beaten by AI, Lee did win one game when the program had a similar meltdown. It got off track and spiraled down, down, down, ending up making moves even a beginner would avoid. Then there's the unctuous praise ChatGPT unendingly bestows on the user, declaring me "awesome" and a "badass". I'm told you can ask it to stop saying such things. Oh, and be especially careful about letting it "clean up your code."

Nevertheless AI was a huge net gain. It knows linear algebra and geometry better than I do. I expect it would be even stronger at a routine application like a computer game. I'm going to have Chat generate a pair of hyperdice and have them bounce around using simulated physics. Then I can shoot craps in four dimensions. (I've already figured out how to change the rules minimally to get almost the same odds and point lengths.)

 

[nsaspook]

I tested claude.ai with a programming task that I knew there were only a few different examples of source code for that driver but many copies of those few examples on the Internet.

ME:
Linux kernel driver for the DT9812 usb DAQ device that loads firmware for the new version of the DT9812 hardware. The dt9812 firmware loader USB ID is 1298. when the firmware is loaded the SUB-ID changes to 9812

IT:
I'll create a Linux kernel driver for the DT9812 USB DAQ device that handles firmware loading for the new hardware version. This driver will recognize the device with USB ID 1298 and load the appropriate firmware, after which the device will change its USB ID to 9812.

Pretty good response to my prompt.

What did it show? A classic example of a USB Linux USB device driver in C. It even copied the goto error statements for backing out of error conditions.

/* Device probe function */
static int dt9812_probe(struct usb_interface *interface, const struct usb_device_id *id)
{
    struct dt9812_device *dev;
    struct usb_host_interface *iface_desc;
    struct usb_endpoint_descriptor *endpoint;
    int i, retval = -ENOMEM;

    /* Allocate memory for device specific structure */
    dev = kzalloc(sizeof(struct dt9812_device), GFP_KERNEL);
    if (!dev)
        return -ENOMEM;

    kref_init(&dev->kref);
    mutex_init(&dev->io_mutex);

    dev->udev = usb_get_dev(interface_to_usbdev(interface));
    dev->interface = interface;
    dev->firmware_loaded = false;

    /* Set up endpoint information */
    iface_desc = interface->cur_altsetting;
    
    for (i = 0; i < iface_desc->desc.bNumEndpoints; ++i) {
        endpoint = &iface_desc->endpoint[i].desc;

        if (!dev->bulk_in_endpointAddr &&
            usb_endpoint_is_bulk_in(endpoint)) {
            /* Found a bulk in endpoint */
            dev->bulk_in_endpointAddr = endpoint->bEndpointAddress;
            dev->bulk_in_size = usb_endpoint_maxp(endpoint);
        }

        if (!dev->bulk_out_endpointAddr &&
            usb_endpoint_is_bulk_out(endpoint)) {
            /* Found a bulk out endpoint */
            dev->bulk_out_endpointAddr = endpoint->bEndpointAddress;
        }
    }

    if (!(dev->bulk_in_endpointAddr && dev->bulk_out_endpointAddr)) {
        dev_err(&interface->dev, "Could not find both bulk-in and bulk-out endpoints\n");
        retval = -ENODEV;
        goto error;
    }

    /* Allocate bulk in buffer */
    dev->bulk_in_buffer = kmalloc(dev->bulk_in_size, GFP_KERNEL);
    if (!dev->bulk_in_buffer) {
        retval = -ENOMEM;
        goto error;
    }

    /* Save our data pointer in this interface device */
    usb_set_intfdata(interface, dev);

    /* Check if this is the firmware loader device */
    if (id->idProduct == USB_DT9812_FIRMWARE_LOADER_PRODUCT_ID) {
        dev_info(&interface->dev, "DT9812 firmware loader device detected\n");
        
        /* Attempt to load firmware */
        retval = dt9812_load_firmware(dev);
        if (retval < 0) {
            dev_err(&interface->dev, "Failed to load firmware: %d\n", retval);
            goto error;
        }
        
        dev_info(&interface->dev, "Firmware loaded, device should reconnect with new ID\n");
    } else {
        dev_info(&interface->dev, "DT9812 device with firmware loaded detected\n");
        /* Initialize the DAQ functionality here */
    }

    return 0;

error:
    if (dev)
        kref_put(&dev->kref, dt9812_delete);
    return retval;
}

 

[jack action]

I tried it and it really answers that.

 

[DaveC426913]

The capital of the state containing Dallas is ...?

If the question was: The capital of a state containing Dallas is ...? The AI response would be perfectly correct.

Oh, I see. There are multiple states with a city named Dallas.

 

[Hornbein]

I had ChatGPT write me a Python program that would rotate an exploded hyperdie, a hypercube with pips like a die. It took about six hours. I don't know Python so it would have taken me maybe 600 hours to do it on my own, which I never would have done. I was greatly impressed at the colossal reduction in the frustrations of computer programming. And no human can ever hope to match its quickness at these tasks.

On the other hand once ChatGPT got off track it stayed off track. It makes a mistake because it doesn't understand something. Its flailing attempts to patch up something it doesn't understand often makes things worse. The beginning of such a syndrome is random so I learned to save a copy before all but absolutely trivial steps. During these breakdowns ChatGPT would repeatedly state "I understand perfectly!", "I've got it now!" and so forth. I learned to ignore such stuff. If it makes a mistake, backtrack to the working copy and try again. If I'd known that progress would have been even faster.

I was reminded of the epoch-making Go match with Lee Sidol. Though thoroughly beaten by AI, Lee did win one game when the program had a similar meltdown. It got off track and spiraled down, down, down, ending up making moves even a beginner would avoid. Then there's the unctuous praise ChatGPT unendingly bestows on the user, declaring me "awesome" and a "badass". I'm told you can ask it to stop saying such things. Oh, and be especially careful about letting it "clean up your code."

Nevertheless AI was a huge net gain. It knows linear algebra and geometry better than I do. I expect it would be even stronger at a routine application like a computer game. I'm going to have Chat generate a pair of hyperdice and have them bounce around using simulated physics. Then I can shoot craps in four dimensions. (I've already figured out how to change the rules minimally to get almost the same odds and point lengths.)

One of ChatGPTs most annoying programming foibles is this. It will tell you how to patch your code. This is tricky and error-prone. So you ask ChatGPT to patch the code itself. Usually this will work, but sometimes it will leave out parts of your program that are unrelated to the patch. I tried to get it to stop doing this but it couldn't understand what I was talking about. So one must routinely save copies of working programs so it is easy to backtrack.

 

[Borg]

One of ChatGPTs most annoying programming foibles is this. It will tell you how to patch your code. This is tricky and error-prone. So you ask ChatGPT to patch the code itself. Usually this will work, but sometimes it will leave out parts of your program that are unrelated to the patch. I tried to get it to stop doing this but it couldn't understand what I was talking about. So one must routinely save copies of working programs so it is easy to backtrack.

So, no different than letting a co-worker work on your code.

 

[Borg]

I haven't been able to listen to the podcast yet but this looked interesting.

https://www.techradar.com/computing...ory-and-the-podcast-results-were-mind-blowing

 

[nsaspook]

https://techcrunch.com/2025/04/18/openais-new-reasoning-ai-models-hallucinate-more/

OpenAI’s recently launched o3 and o4-mini AI models are state-of-the-art in many respects. However, the new models still hallucinate, or make things up — in fact, they hallucinate more than several of OpenAI’s older models.

Hallucinations have proven to be one of the biggest and most difficult problems to solve in AI, impacting even today’s best-performing systems. Historically, each new model has improved slightly in the hallucination department, hallucinating less than its predecessor. But that doesn’t seem to be the case for o3 and o4-mini.

According to OpenAI’s internal tests, o3 and o4-mini, which are so-called reasoning models, hallucinate more often than the company’s previous reasoning models — o1, o1-mini, and o3-mini — as well as OpenAI’s traditional, “non-reasoning” models, such as GPT-4o.

Perhaps more concerning, the ChatGPT maker doesn’t really know why it’s happening.

Maybe it's because these things don't really reason or understand anything.