LLM: Getting the question right

[.Scott]

TL;DR

It appears that an internal LLM did not actually answer the $1M Navier-Stoke problem.

I've made a handful of posts to PF where I have used an LLM to assist me with the post - always with attribution (always ChatGPT).
And I have remarked in a couple of cases how I needed to present several questions to the LLM before it finally caught on to what I was asking. In those cases, I have also included the exact wording of the question in my attribution.

Well, it seems we now have a rather spectacular example of that issue.

Our case in point started early this month with this Scientific American article claiming an AI solution to the $1,000,000 Navier-Stokes problem. It said:

For the second time ever, someone has solved one of the seven Millennium Prize Problems—math’s biggest targets, each worth a $1-million prize. But unlike the first time, that someone is an artificial intelligence start-up. Today OpenAI announced that its internal model has proved that the Navier-Stokes equations, which mathematicians use to study how fluids move, are fatally flawed. The reveal comes after mathematician Tristan Buckmaster alleged that OpenAI had tackled the proof after the company became aware that Buckmaster and his colleague Levent Alpöge had been using a specific method to break a related problem. OpenAI has denied the allegations. OpenAI’s proof shows that, on rare occasion, the Navier-Stokes equations “blow up,” meaning they dictate that a fluid’s speed becomes infinite at some points, something that is impossible in nature. The company says the proof has been certified using the programming language Lean, which all but guarantees its correctness.

That sounds pretty exciting. But on closer review, we have a new Scientific American article that discusses that earlier announcement. In part:

Two weeks ago OpenAI claimed a solution to one of the biggest open problems in math—the Navier-Stokes problem—an achievement worth a $1-million prize from the Clay Mathematics Institute. The proof ignited a powder keg of concern over artificial intelligence companies’ race to disrupt the subject. But with the dust still far from settled, a different controversy is emerging: Did OpenAI even solve the right Navier-Stokes problem? Generated by an internal large language model (LLM), OpenAI’s proof relies on an approach that many experts find unnatural. It solves a variant of the problem that mathematicians say is disconnected from reality and thus less interesting. In a sense, the LLM found and exploited a loophole in the framing of the question.

Don't you hate when that happens?!

 

[loop quantum gravity]

Well as with maths we can always generalize the equation to manifold, R^n where n is a natural number and not just n=3 as is the case for Navier-Stokes PDE.
Nobody says mathematics is easy, but with LLM technology it can make the work of scholars easier, where you can concentrate on ideas rather than their implementation itself. The AI would solve the technicalities, but you better check that it does not fail somewhere in its derivations. That's still something mathematicians ought to check for. Yes, it's quite a duanting task to proof-read someone else's work; it really depends how much can you rely on a black box that you don't understand how it works...