Hallucinated citations are polluting the scientific literature

[Greg Bernhardt]

Hallucinated citations are polluting the scientific literature. What can be done?​

Tens of thousands of publications from 2025 might include invalid references generated by AI, a Nature analysis suggests.

https://www.nature.com/articles/d41586-026-00969-z

Anyone have thoughts on their own experience and what the options are to combat this?

 

[Andy Resnick]

Unfortunately, the whole scientific enterprise relies on each individual researcher behaving ethically and professionally, even when there are substantial incentives to 'cut corners'.

IMO, the only way to combat this problem is to hold the authors accountable.- The way that article phrases it, the Journal takes responsibility rather than the authors. Simply withdrawing a manuscript means it will appear elsewhere, in a different journal. The Journal editors could send a report to the author's institutional research integrity officer/conduct board, for example.

 

[Rive]

Honestly, I would consider this fraud.

 

[BillTre]

How about using AI to search out bad references?

 

[Muu9]

Shouldn't peer review be catching these fraudulent citations? If it isn't, what does that suggest about the feasibility of the current peer-review based scientific model?

 

[Andy Resnick]

Shouldn't peer review be catching these fraudulent citations? If it isn't, what does that suggest about the feasibility of the current peer-review based scientific model?

Again, the merit of peer review relies upon humans acting professionally. Bad actors always exist, but that's why some journals are considered 'higher quality' than others.

Peer review is like democracy- to poorly quote Churchill, "peer-review is the worst form of scientific certification except for all those other forms that have been tried from time to time."

 

[WillyP]

I think anybody who relies on AI for any sort of facts, should be fired. It is hard to believe that anybody on the planet is not aware of how unreliable AI is. It regularly makes up facts. We all know this. Why would anybody use it to gather data, when you know the data is unreliable!?

 

[Hornbein]

I think anybody who relies on AI for any sort of facts, should be fired. It is hard to believe that anybody on the planet is not aware of how unreliable AI is. It regularly makes up facts. We all know this. Why would anybody use it to gather data, when you know the data is unreliable!?

Either laziness or to increase output. An enterprising individual released at least 364 AI "history" videos in a month. They were entirely and deliberately fictitious.

 

[Will Flannery]

I've had the experience. Grok (as I recall) produced a reference and a detailed quote that was exactly what I was looking for .... it was a couple of years ago, and I thought I had written about it but I just searched my computer but can't find it ... and I was so happy I rewrote parts of what I was working and used the reference. Then, maybe a week later I decided I'd check the entire paper, and lo and behold, it it didn't exist. So, yes, AI can produce convincing detailed references, but .... they're easy to check.

The reason I use AI to look for references is that it is incredibly good at its job, and It has found many references that I do use ...... after checking :).

 

[loop quantum gravity]

All my preprints before chatGPT were read carefully by others, they weren't papers per se in journals.
I didn't think to myself that LLM can produce references that don't even exist. It's really upseting to hear that it does such a thing. But it also means we can't accept everything it writes without understanding the subject we asking about.
So it means that academics still have work to sort out the mess those LLMs do.

 

[Will Flannery]

"I didn't think to myself that LLM can produce references that don't even exist." I didn't either ... but now I know .. and that's why they call it hallucinating. It's complete with all the details, and since it was saying what I wanted to hear, it was convincing. But, it's not that AI 'knows' that it's false. It doesn't .... but its discovery algorithm isn't perfect and it can discover things that don't exist. (IMO of course).