Double blind peer review winners and losers

  • Thread starter Demystifier
  • Start date
  • Featured
  • #1
Demystifier
Science Advisor
Insights Author
Gold Member
11,360
3,975
Double blind peer review is a process in which not only the author does not know the name of the reviewer (which is the standard practice), but also the reviewer does not know the name of the author (which is not a standard practice). What would happen if the scientific community suddenly decided to shift to the double blind review process? Who would be the winners? Who would be the losers?
 
Last edited by a moderator:
  • Like
Likes bhobba, MathematicalPhysicist and Greg Bernhardt

Answers and Replies

  • #2
TeethWhitener
Science Advisor
Gold Member
1,937
1,376
This paper:
http://www.pnas.org/content/114/48/12708
from last year insinuates that famous authors, authors from top universities, and authors from top companies are significantly more likely to get their papers accepted using single-blind peer review (reviewer is anonymous, authors aren't) than double-blind.
 
  • Like
Likes jim mcnamara and Demystifier
  • #3
DrClaude
Mentor
7,555
3,894
Making it actually double blind would be difficult. Only looking at the list of references, it is often easy to figure out who wrote the paper :smile:. In my field, when there is an experimental component, there are often very few groups in the world that have the proper equipment to carry out the experiment. Also, experiments and numerical methods tend not to be described completely in each paper, but referenced to previously published work.

As a referee, I find it helpful to know who wrote the paper. As you get to know other researchers, you get a feeling about who is careful about some things and who is sloppy, so you get a feeling about how much to trust what is written. It can be useful to know about the particular slant of certain researchers (think about how it is on PF: posts that may seem innocuous to a passerby are seen in another light when you know that the poster is always trying to peddle their pet theory).

I have never felt intimidated when reviewing papers by famous researchers. But I remember being disappointed by papers written by researchers close to me of whom I expected more than some of the low-quality stuff they've had their name attached to.
 
  • Like
Likes mathwonk, Bystander, Wrichik Basu and 2 others
  • #4
Vanadium 50
Staff Emeritus
Science Advisor
Education Advisor
26,449
9,969
there are often very few groups in the world that have the proper equipment to carry out the experiment
Exactly - if one says they looked at 13 TeV collisions using a detector with a 3.8 T solenoidal magnet, I am pretty sure who wrote the paper.
 
  • #5
f95toli
Science Advisor
Gold Member
3,113
602
I also don't understand how double-blind reviewing would work.
In my field in would be almost impossible to cover all the details about a given experiment in a 4-page article. In a typical paper we will refer to perhaps 3-4 previous papers which describe device designs or experimental methods.
Also, my community is small enough that I could probably guess which group the paper came from in 90% of cased even if I didn't see the list of authors.

Double blind reviewing (or even open reviewing where the reviewers name is stated) might be possible in e.g. some life sciences where the community is much.much larger but I can't see how it would work in a typical field in physics.
 
  • Like
Likes Dale
  • #6
14,415
11,728
The name contains information which might help to increase productivity (= use of time). Is it unfair? Probably, but I like the comparison with PF, which is at small what papers are at large. Don't we use information given by name beforehand? To be honest, I don't even open some threads when I see the OP, for various reasons, not necessarily that I don't expect any insights from them. And how likely is it that you get an interesting paper on the desk you haven't heard of before? And usually a famous author implies a famous reviewer, which also means that they are not afraid of honest answers. The review process is a time consuming one and you don't want to waste your time with a tri-sectionist. Yes, this is the extreme end in the set of examples, the other end being breakthrough papers, but anything in between is along a linear function f(paper)=time. I learnt that reviews aren't among scientists' favorite duties, a double-blind process would at least double the time required. Unfair? Maybe, however, it's a compromise that works - usually.
 
  • Like
Likes baldbrain, dextercioby and Bystander
  • #7
TeethWhitener
Science Advisor
Gold Member
1,937
1,376
One question: if the main objection is "you could easily figure out who wrote the paper," then what's the harm in leaving the authors' names off the paper?

Edit: during review, that is. Obviously.
 
  • #8
DrClaude
Mentor
7,555
3,894
One question: if the main objection is "you could easily figure out who wrote the paper," then what's the harm in leaving the authors' names off the paper?
I agree that it may appear contradictory, but it is actually addressing two different points:
  • from the authors' point of view, anonymity is far from being assured, since it is possible to figure out who wrote the paper
  • from the reviewer's point of view, while it is possible to find out who wrote the paper, one cannot be sure so it can make reviewing less efficient
An additional point to what I wrote above is that the reviewing process does not tae place in a vacuum. While in some ideal vision one could think that a paper should stand on its own merit, in reality it is one part of a bigger jigsaw puzzle and evaluating if the piece fits depends in some measure on who produced it.
 
  • #9
TeethWhitener
Science Advisor
Gold Member
1,937
1,376
  • from the authors' point of view, anonymity is far from being assured, since it is possible to figure out who wrote the paper
  • from the reviewer's point of view, while it is possible to find out who wrote the paper, one cannot be sure so it can make reviewing less efficient
1) At least according to the PNAS paper above, the authors who would be most affected by double-blind peer review are the ones who are already highly decorated, since it is those authors who stand to benefit the most from having their names known to reviewers. Unknown authors essentially see no benefit/detriment.
2) Why should effort be placed into finding out who the author is? I would rather think that effort should be redirected into making sure the science is good. If the science is good, it will stand on its own merits, independent of its author.
 
  • Like
Likes Wrichik Basu
  • #10
f95toli
Science Advisor
Gold Member
3,113
602
One question: if the main objection is "you could easily figure out who wrote the paper," then what's the harm in leaving the authors' names off the paper?

Edit: during review, that is. Obviously.
Again, many -in my field probably most- papers will contain "for details about the device design see reference [x]" making this irrelevant.
Papers that are completely "stand-alone" and do not build on previous work by the same authors are very, very rare.

Also, an important part of my job as a reviewer is to check for "novelty" and that what has been submitted in not just some minor change to what he authors have published elsewhere. I''d guess that about 50% of the times when I recommend rejection it is because the authors have previously published very similar results.
 
  • Like
Likes mathwonk, dextercioby and DrClaude
  • #11
Vanadium 50
Staff Emeritus
Science Advisor
Education Advisor
26,449
9,969
One question: if the main objection is "you could easily figure out who wrote the paper," then what's the harm in leaving the authors' names off the paper?
The names? Nothing but the extra work for no benefit. Citations that "out" the authors? That would make understanding of what's going on just go slower.

As far as I can tell, the biggest impact this would have would be to kill the arXiv.
 
  • #12
TeethWhitener
Science Advisor
Gold Member
1,937
1,376
Again, many -in my field probably most- papers will contain "for details about the device design see reference [x]" making this irrelevant.
Papers that are completely "stand-alone" and do not build on previous work by the same authors are very, very rare.
So then there's no harm if their names are left off the paper.

Also, an important part of my job as a reviewer is to check for "novelty" and that what has been submitted in not just some minor change to what he authors have published elsewhere. I''d guess that about 50% of the times when I recommend rejection it is because the authors have previously published very similar results.
If novelty is important, why does it matter who the authors are? Sure, if they're just extending a minor result onto results published elsewhere, it's a little crummy of them, but I don't see how a different author making a minor extension to their paper should fare any differently in the review process.
 
  • #13
TeethWhitener
Science Advisor
Gold Member
1,937
1,376
Nothing but the extra work for no benefit.
What extra work? Why do you need to know who wrote the paper? Why doesn't the science stand on its own?
Citations that "out" the authors? That would make understanding of what's going on just go slower.
I'm not sure I understand this comment. If you can't understand what's going on in the paper, then you should decline to review it.
As far as I can tell, the biggest impact this would have would be to kill the arXiv.
Why? Because you could go and compare the manuscript you're reviewing with a preprint on arxiv? Again, why do you need to put that much effort into knowing the author?

I understand that double blind peer review might be irrelevant in large collaborative efforts like particle physics or astronomy, but the vast majority of science isn't large collaborative efforts, and a lot of big names have published unimpressive results in impressive journals, to the extent that one openly wonders: did this get published solely by virtue of their name? One really easy way to make that cloud go away is via double blind peer review.
 
  • Like
Likes Auto-Didact, dextercioby and Dadface
  • #14
f95toli
Science Advisor
Gold Member
3,113
602
So then there's no harm if their names are left off the paper.
But that would mean that some papers were reviewed double blind and some not. It would also mean that the author would -potentially- have the alter the way they write since they would then have to consider if they were happy with the reviewers being able to figure out who they were or not.
Also, well-known people could effectively decide against a double blind review simply by including a reference to their own work.


If novelty is important, why does it matter who the authors are? Sure, if they're just extending a minor result onto results published elsewhere, it's a little crummy of them, but I don't see how a different author making a minor extension to their paper should fare any differently in the review process.
Because there is no way any review can be familiar with ALL the literature in the field. If I did not know who the author were there it would be impossible for me to know if the same group had published pretty much the same thing in some journal that I don't read.
Or, as was the case in a paper I reviewed a couple of weeks ago, if the "novel" bit of the manuscript had already been included as part of a very long article the same group had written a couple of years ago (but where the main topic at first glance was very different).

Novelty is pretty much everything for high impact journals. If your work is not novel you won't get published irrespective of the quality of the work.
 
  • #15
TeethWhitener
Science Advisor
Gold Member
1,937
1,376
If I did not know who the author were there it would be impossible for me to know if the same group had published pretty much the same thing in some journal that I don' read.
I don't think this is true. For one, it misses the point of my objection: what if a different group had published pretty much the same thing in some journal that you don't read? Then knowing the authors' names would mean precisely diddly squat. For another, the scenario in the preceding sentence in fact did happen to me (twice, and rather recently) while I was reviewing a paper. I did what I normally do: google a few key words and phrases, and guess what...a very similar paper from a different group popped up within the first few searches, and I was able to recommend the manuscript for rejection based on that.
 
  • #16
TeethWhitener
Science Advisor
Gold Member
1,937
1,376
from the authors' point of view, anonymity is far from being assured, since it is possible to figure out who wrote the paper
Another thing: This is also (trivially) true in single-blind peer review, so I don't see how this counts as a drawback for double-blind peer review.
 
  • #17
Vanadium 50
Staff Emeritus
Science Advisor
Education Advisor
26,449
9,969
Why do you need to know who wrote the paper?
The point I am making is that I need to know the equipment used to evaluate the paper, and knowing the details of the equipment tells me who did the work. Furthermore, I will likely have already read the preprint. This is far more common than not: if I am asked to referee, it's because I am up-to-date in the field, and if I am keeping up with the field, I am reading the arXiv. You can't have both - either you go double-blind, or you stop publishing preprints.
 
  • Like
Likes Bystander
  • #18
Demystifier
Science Advisor
Insights Author
Gold Member
11,360
3,975
Only looking at the list of references, it is often easy to figure out who wrote the paper :smile:
Not so famous authors would then make more citations of a certain famous author to create a false impression that the paper was written by that famous author and hence to increase the chances for publication. :woot:
 
  • Like
Likes mathwonk, Wrichik Basu and dextercioby
  • #19
Demystifier
Science Advisor
Insights Author
Gold Member
11,360
3,975
As far as I can tell, the biggest impact this would have would be to kill the arXiv.
No. The arXiv could be modified so that the author is revealed only after the publication or acceptance.
 
  • Like
Likes Auto-Didact
  • #20
f95toli
Science Advisor
Gold Member
3,113
602
No. The arXiv could be modified so that the author is revealed only after the publication or acceptance.
But that would mean that there would be no way of keeping track of what the various groups in your field are doing(!).
 
  • #21
Demystifier
Science Advisor
Insights Author
Gold Member
11,360
3,975
But that would mean that there would be no way of keeping track of what the various groups in your field are doing(!).
But you could still keep track of what is done in your field. And the most important thing is what is done, not who did it, don't you agree?
 
  • Like
Likes Wrichik Basu and Auto-Didact
  • #22
14,415
11,728
But you could still keep track of what is done in your field. And the most important thing is what is done, not who did it, don't you agree?
I don't. If a famous scientist deals with a certain question it has more weight than if a nobody does, simply because the expectation of results is a different one. The former may mean a change of direction of research, the latter only means just another PhD.
 
  • #23
f95toli
Science Advisor
Gold Member
3,113
602
But you could still keep track of what is done in your field. And the most important thing is what is done, not who did it, don't you agree?
No, for two reasons. One one is that finding information by searching using the title and/or abstract can be quite inefficient since it is rarely obvious which keywords one should use, I frequently find information about e.g. new experimental methods, device designs etc because someone I know the name of in my field is was a co-author of the paper; as fresh_42 mentions this can suggest that a new research direction is becoming important.

The second reason is that part of my job is to keep track of the development in my field around the world; this means not just keeping track of the science but also of the science policy and/or funding situation (like most senior scientists I do have to contribute to e.g. policy and briefing documents) .
Hence, I do need to know what the "main players" in my field are up to at a given time even when the science they publish isn't that interesting.
 
  • #24
Demystifier
Science Advisor
Insights Author
Gold Member
11,360
3,975
@fresh_42 and @f95toli you made good arguments that it is important to know who is the author. But I don't see that you made arguments that it is more important than the actual content of the scientific work.
 
  • Like
Likes Auto-Didact
  • #25
Andy Resnick
Science Advisor
Education Advisor
Insights Author
7,552
2,166
Double blind peer review is a process in which not only the author does not know the name of the reviewer (which is the standard practice), but also the reviewer does not know the name of the author (which is not a standard practice). What would happen if the scientific community suddenly decided to shift to the double blind review process? Who would be the winners? Who would be the losers?
I've been involved with all three flavors of peer review (as a reviewer)- single blind/double blind/ open, and while I'd like to think my evaluation process is identical for all three, I will note that I've always declined to provide my name (open review) and tend to read the introductory sections more cautiously (double blind as compared to single-blind).

What would be infinitely more interesting is a double-blind review of *grant proposals*, because the stakes are so much higher. To be sure, I'm not sure how review panels could function this way. For comparison, NIH review panels are quasi-open (the panel members are known, but the primary and secondary reviewers are not) while for every other agency and foundation that I have applied to, the review panels are at least single-blind.
 
Last edited by a moderator:

Related Threads on Double blind peer review winners and losers

  • Last Post
Replies
5
Views
3K
  • Last Post
Replies
14
Views
1K
Replies
28
Views
5K
  • Last Post
Replies
5
Views
2K
  • Last Post
Replies
5
Views
2K
  • Last Post
Replies
8
Views
982
  • Last Post
Replies
8
Views
747
Replies
16
Views
5K
  • Last Post
Replies
21
Views
3K
Replies
1
Views
3K
Top