What Are the Key Criticisms of the Peer Review System?

[aquitaine]

Recently I found a couple of articles that are critical of the peer review process. What are your thoughts about them?

http://www.telegraph.co.uk/scienceandtechnology/science/sciencenews/3326091/Peer-review-the-myth-of-the-noble-scientist.html

Article2

 

[Choppy]

I always take critique of "peer review" with a grain of salt.

Consider this statement from the first article:

If a well-known scientist submits a paper, it will probably be accepted; if an unknown submits one, it will probably be rejected.

To the author I would say: show me some hard evidence or retract the statement. Unknown scientists are published all the time. How else would they ever become "known" in the first place?

Peer review is not perfect. Reviewers are human. But attacking the system is about as productive as attacking the notion of democracy. Yes, there are shortcomings, but unless you have a better solution, you're really not doing anything more than complaining.

 

[turbo]

Referees are humans. They approach your paper with their assumptions and prejudices, and it is not always easy to accommodate those. That said, an astute referee that is knowledgeable and willing to to do some research can make suggestions that can make impressive improvements in the qualities of your published work. I'd rather have pretty tough critics at peer-review and end up with a well-crafted final draft.

 

[will.c]

To add to that, I'm always hard pressed to see statements such as the one quoted as serious critique, rather than sour grapes.

Criticizing peer review is something that has to be done in a disinterested, objective tone or else it comes off as whining, and is hardly likely to effect any change in the process.

 

[turbo]

To add to that, I'm always hard pressed to see statements such as the one quoted as serious critique, rather than sour grapes.

Criticizing peer review is something that has to be done in a disinterested, objective tone or else it comes off as whining, and is hardly likely to effect any change in the process.

I would like to add that whatever journal you submit to has editors that are familiar with the subject-matter you are addressing. They choose the anonymous referees, who are (as far as I know) unpaid specialists who are qualified to pick apart your work. This process is designed to vet papers, cull out dogs, improve marginal papers and hone superlative papers. It may not always work out that way, but it appears to work pretty well. If all papers were published electronically (ArXiv or equivalent) with no peer review, it would be pretty tough to plow through them all and evaluate them fairly.

 

[Vanadium 50]

It's not clear what the alternatives are either.

Does one stop reviewing papers? Then the journals are flooded with crackpottery.

Does one publish the names of the reviewers and/or their comments? I suppose one could, but as someone who has been on both ends of this, I think the frankness afforded by anonymity serves a purpose. Or maybe I should say the fig leaf of anonymity. The pool of true experts in a paper's topic is usually pretty small, and it's usually not too hard to figure out who is who.

Does one make the authors name unknown to the reviewer? People have proposed this, but I think you have the same issue as above, only in reverse. There just aren't that many people who could have written a particular paper. Experimentally, it's even easier to figure out - often apparatus is unique, like fingerprints.

While the system is clearly imperfect, and I say this from having been on both ends (actually, all three ends: writer, reviewer and reader), a better alternative hasn't been proposed. This is particularly true now that there is the arXiv: a reader can now choose the level of refereeing she wants.

 

[aquitaine]

Does one stop reviewing papers? Then the journals are flooded with crackpottery.

Pretty much what I was thinking.

Thanks for your opinions. :)

 

[Dale]

Does one make the authors name unknown to the reviewer? People have proposed this, but I think you have the same issue as above, only in reverse. There just aren't that many people who could have written a particular paper. Experimentally, it's even easier to figure out - often apparatus is unique, like fingerprints.

In the medical imaging field most journals use a "double-blind" review process with the editor being the only one who knows the identity of both the reviewers and the authors. It is a little silly since, as you point out, it is a small enough community that it cannot truly be blinded. But the double blind approach certainly doesn't hurt the process in any way, and at least there is "plausible deniability" to keep anyone from getting too upset when they see each other at the next conference.

Having been on both ends of the process, I think it is a good system. I know that it has dramatically improved a few of my manuscripts and marginally improved the rest. There has not been one instance where I feel that the peer-review process was detrimental to the quality of my work, even the outright rejections. In the end, that is the goal of peer-review, to improve the quality of the journal.

If somebody does not like the peer-review process then they do not need to go through it. The world is full of alternate venues without peer-review, and nobody's ideas are being suppressed. But if you want the legitimacy of a peer-reviewed forum then you have to submit to peer-review. You cannot have the legitimacy of a prestigous journal without the process that gave the journal its integrity. I think that is what the anti-peer-review crowd really wants: unearned legitimacy.

 

[ZapperZ]

Recently I found a couple of articles that are critical of the peer review process. What are your thoughts about them?

http://www.telegraph.co.uk/scienceandtechnology/science/sciencenews/3326091/Peer-review-the-myth-of-the-noble-scientist.html

Article2

I would suggest you (and the authors of both of these news article) to read the late Dan Koshland's wonderful piece in Nature from a few years ago[1]. In particular, I would quote two of the passages in that article:

"I realize now that a new theory is likely to meet resistance, but it should, if based on GOOD experiments, receive sceptical encouragement if science is to remain in balance. Non-confirmists are necessary for progress in science, just as mutations are necessary for progress in evolution. However, there must be constraints to select good mutations from bad mutations. Too many mutations block evolution, as error-prone straints of bacteria have proved. So non-conformist thinking in science must be encouraged to make progress, but restrained to prevent anarchy. In science, it is PEER-REVIEWED JOURNALS and granting agencies that provide such balance."

and

"The trouble is that journals can easily become too conservative, because editors find it easier to reject the unusual than to take a chance on the unthinkable... The existence of multiple journals provides the final safeguard against too much conservatism and is the ultimate reason that science is more receptive to non-conformity than any other segment of our society."

That last part is often ignored by many, especially those who are not familiar with scientific publications.

Zz.

[1] D.E. Koshland, Jr., Nature v.432, p.447 (2004).