AI Used In Peer Review

  • Thread starter Thread starter Hornbein
  • Start date Start date
Click For Summary

Discussion Overview

The discussion revolves around the use of AI in the peer review process of scientific papers, exploring its potential benefits and drawbacks. Participants consider the implications of integrating AI tools into the review workflow, particularly in the context of increasing submission volumes and the availability of human reviewers.

Discussion Character

  • Debate/contested
  • Conceptual clarification

Main Points Raised

  • Some participants express support for incorporating critiques from generative AI into the review package, suggesting it could provide valuable insights before the formal review process.
  • Others argue that while AI could assist in the initial selection of papers, the final decision should remain with human reviewers, emphasizing the importance of human judgment in the process.
  • A participant mentions the effectiveness of static code analysis tools in software engineering as a parallel, noting their limitations in addressing whether code meets specific requirements.
  • Concerns are raised about the increasing volume of submissions and the potential shortage of peer reviewers, suggesting that AI could help manage this issue.
  • One participant shares a personal anecdote about receiving spam related to scientific papers, questioning the credibility of such communications and humorously referencing an Erdős number.

Areas of Agreement / Disagreement

Participants do not reach a consensus on the role of AI in peer review, with some advocating for its use while others maintain that human oversight is essential. The discussion reflects multiple competing views on the integration of AI in the peer review process.

Contextual Notes

Participants acknowledge the potential utility of AI tools but highlight limitations regarding their ability to fully replace human reviewers. The discussion also touches on the evolving landscape of peer review in light of increasing publication rates.

Who May Find This Useful

This discussion may be of interest to researchers, academic publishers, and those involved in the peer review process, particularly in fields experiencing high submission volumes.

Computer science news on Phys.org
Hornbein said:
https://physics.aps.org/articles/v18/194

Some are for it, others against it.
In my opinion:
It would be useful for a critique from a generative AI (or perhaps reports from a few Generative AI products) to be included in the review package - and available to the author (before the review) and the peers (during the review).
But only material generated by peers should constitute the actual review.
Of course, individual peer reviewers are free to use AI tools in whatever way they find useful.

In the field of software engineering, static code analysis tools have gotten remarkably good. Coverity and LDRA come to mind as examples. They are thorough to the extreme. But, they do identify "issues" that bear on the core purpose of the code - and which either cannot be "corrected" or should not be corrected. And these tools cannot directly address whether the code is actually meeting requirements - only whether it is self-consistent and meets broadly-accepted and/or industry-specific coding standards.
 
Last edited:
  • Informative
Likes   Reactions: berkeman
This will happen in future, that is for sure. It could be useful for the first selection process of the huge amount of papers. Soon, we will have not enough peer-reviewers to look at them. The final decision however will always be at the human side. At least as long as an AI is not a publisher of a science-journal itself.
 
I get all sorts of mail telling me that I'm the author of some "scientific" paper I have nothing to do with. I suspect it started when I began frequenting PF. Should I believe the spam I have an Erdős number of -3.
 

Similar threads

  • · Replies 5 ·
Replies
5
Views
898
  • · Replies 2 ·
Replies
2
Views
898
Replies
2
Views
591
Replies
22
Views
5K
  • · Replies 13 ·
Replies
13
Views
4K
Replies
6
Views
768
  • · Replies 1 ·
Replies
1
Views
3K
  • · Replies 1 ·
Replies
1
Views
2K
  • · Replies 10 ·
Replies
10
Views
1K
  • · Replies 101 ·
4
Replies
101
Views
3K