Hallucinated citations are polluting the scientific literature

  • Thread starter Thread starter Greg Bernhardt
  • Start date Start date
Click For Summary

SUMMARY

Hallucinated citations generated by AI are contaminating scientific literature, with projections indicating tens of thousands of invalid references appearing in publications from 2025 onward, as reported by Nature. The discussion establishes that author accountability and institutional oversight are essential to combat this issue, rather than relying solely on journal responsibility. Peer review currently fails to consistently detect fraudulent citations, raising concerns about the reliability of the peer-review model in its present form. The use of AI for fact-gathering is deemed irresponsible due to its propensity to fabricate information, and deliberate misuse of AI to produce fictitious content has already been documented.

PREREQUISITES

  • Understanding of scientific peer review processes
  • Familiarity with AI-generated content and hallucination phenomena
  • Knowledge of research ethics and academic integrity policies
  • Awareness of institutional research misconduct procedures

NEXT STEPS

  • Develop AI tools specifically designed to detect fabricated citations in manuscripts
  • Implement stronger institutional accountability measures for authors submitting AI-generated content
  • Enhance peer review training to identify AI-induced inaccuracies and hallucinated references
  • Research alternative scientific certification models beyond traditional peer review

USEFUL FOR

Researchers, journal editors, academic integrity officers, and policymakers focused on maintaining scientific rigor and combating misinformation in scholarly publishing will benefit from this discussion.

Messages
19,921
Reaction score
10,968

Hallucinated citations are polluting the scientific literature. What can be done?​

Tens of thousands of publications from 2025 might include invalid references generated by AI, a Nature analysis suggests.

https://www.nature.com/articles/d41586-026-00969-z

Anyone have thoughts on their own experience and what the options are to combat this?
 
  • Sad
  • Wow
  • Like
Likes   Reactions: DEvens, MrRobotoToo, Filip Larsen and 2 others
Science news on Phys.org
Unfortunately, the whole scientific enterprise relies on each individual researcher behaving ethically and professionally, even when there are substantial incentives to 'cut corners'.

IMO, the only way to combat this problem is to hold the authors accountable.- The way that article phrases it, the Journal takes responsibility rather than the authors. Simply withdrawing a manuscript means it will appear elsewhere, in a different journal. The Journal editors could send a report to the author's institutional research integrity officer/conduct board, for example.
 
  • Like
  • Agree
Likes   Reactions: DEvens, jtbell, berkeman and 1 other person
Honestly, I would consider this fraud.
 
  • Agree
  • Like
Likes   Reactions: Astronuc, DEvens and Greg Bernhardt
How about using AI to search out bad references?
 
  • Like
  • Informative
  • Haha
Likes   Reactions: DEvens, mathwonk, berkeman and 1 other person
Shouldn't peer review be catching these fraudulent citations? If it isn't, what does that suggest about the feasibility of the current peer-review based scientific model?
 
  • Like
Likes   Reactions: DEvens
Muu9 said:
Shouldn't peer review be catching these fraudulent citations? If it isn't, what does that suggest about the feasibility of the current peer-review based scientific model?
Again, the merit of peer review relies upon humans acting professionally. Bad actors always exist, but that's why some journals are considered 'higher quality' than others.

Peer review is like democracy- to poorly quote Churchill, "peer-review is the worst form of scientific certification except for all those other forms that have been tried from time to time."
 
  • Like
  • Agree
Likes   Reactions: Astronuc, Rive and BillTre
I think anybody who relies on AI for any sort of facts, should be fired. It is hard to believe that anybody on the planet is not aware of how unreliable AI is. It regularly makes up facts. We all know this. Why would anybody use it to gather data, when you know the data is unreliable!?
 
WillyP said:
I think anybody who relies on AI for any sort of facts, should be fired. It is hard to believe that anybody on the planet is not aware of how unreliable AI is. It regularly makes up facts. We all know this. Why would anybody use it to gather data, when you know the data is unreliable!?
Either laziness or to increase output. An enterprising individual released at least 364 AI "history" videos in a month. They were entirely and deliberately fictitious.
 

Similar threads

Replies
1
Views
2K
  • · Replies 1 ·
Replies
1
Views
3K
  • · Replies 12 ·
Replies
12
Views
3K
  • · Replies 3 ·
Replies
3
Views
2K
  • · Replies 33 ·
2
Replies
33
Views
11K
  • · Replies 4 ·
Replies
4
Views
3K
Replies
3
Views
3K
  • · Replies 0 ·
Replies
0
Views
5K
  • · Replies 28 ·
Replies
28
Views
12K