Double blind peer review winners and losers

[Demystifier]

Double blind peer review is a process in which not only the author does not know the name of the reviewer (which is the standard practice), but also the reviewer does not know the name of the author (which is not a standard practice). What would happen if the scientific community suddenly decided to shift to the double blind review process? Who would be the winners? Who would be the losers?

 

[TeethWhitener]

This paper:
http://www.pnas.org/content/114/48/12708
from last year insinuates that famous authors, authors from top universities, and authors from top companies are significantly more likely to get their papers accepted using single-blind peer review (reviewer is anonymous, authors aren't) than double-blind.

 

[DrClaude]

Making it actually double blind would be difficult. Only looking at the list of references, it is often easy to figure out who wrote the paper . In my field, when there is an experimental component, there are often very few groups in the world that have the proper equipment to carry out the experiment. Also, experiments and numerical methods tend not to be described completely in each paper, but referenced to previously published work.

As a referee, I find it helpful to know who wrote the paper. As you get to know other researchers, you get a feeling about who is careful about some things and who is sloppy, so you get a feeling about how much to trust what is written. It can be useful to know about the particular slant of certain researchers (think about how it is on PF: posts that may seem innocuous to a passerby are seen in another light when you know that the poster is always trying to peddle their pet theory).

I have never felt intimidated when reviewing papers by famous researchers. But I remember being disappointed by papers written by researchers close to me of whom I expected more than some of the low-quality stuff they've had their name attached to.

 

[Vanadium 50]

there are often very few groups in the world that have the proper equipment to carry out the experiment

Exactly - if one says they looked at 13 TeV collisions using a detector with a 3.8 T solenoidal magnet, I am pretty sure who wrote the paper.

 

[f95toli]

I also don't understand how double-blind reviewing would work.
In my field in would be almost impossible to cover all the details about a given experiment in a 4-page article. In a typical paper we will refer to perhaps 3-4 previous papers which describe device designs or experimental methods.
Also, my community is small enough that I could probably guess which group the paper came from in 90% of cased even if I didn't see the list of authors.

Double blind reviewing (or even open reviewing where the reviewers name is stated) might be possible in e.g. some life sciences where the community is much.much larger but I can't see how it would work in a typical field in physics.

 

[fresh_42]

The name contains information which might help to increase productivity (= use of time). Is it unfair? Probably, but I like the comparison with PF, which is at small what papers are at large. Don't we use information given by name beforehand? To be honest, I don't even open some threads when I see the OP, for various reasons, not necessarily that I don't expect any insights from them. And how likely is it that you get an interesting paper on the desk you haven't heard of before? And usually a famous author implies a famous reviewer, which also means that they are not afraid of honest answers. The review process is a time consuming one and you don't want to waste your time with a tri-sectionist. Yes, this is the extreme end in the set of examples, the other end being breakthrough papers, but anything in between is along a linear function f(paper)=time. I learned that reviews aren't among scientists' favorite duties, a double-blind process would at least double the time required. Unfair? Maybe, however, it's a compromise that works - usually.

 

[TeethWhitener]

One question: if the main objection is "you could easily figure out who wrote the paper," then what's the harm in leaving the authors' names off the paper?

Edit: during review, that is. Obviously.

 

[DrClaude]

One question: if the main objection is "you could easily figure out who wrote the paper," then what's the harm in leaving the authors' names off the paper?

I agree that it may appear contradictory, but it is actually addressing two different points:

• from the authors' point of view, anonymity is far from being assured, since it is possible to figure out who wrote the paper
• from the reviewer's point of view, while it is possible to find out who wrote the paper, one cannot be sure so it can make reviewing less efficient

An additional point to what I wrote above is that the reviewing process does not tae place in a vacuum. While in some ideal vision one could think that a paper should stand on its own merit, in reality it is one part of a bigger jigsaw puzzle and evaluating if the piece fits depends in some measure on who produced it.

 

[TeethWhitener]

• from the authors' point of view, anonymity is far from being assured, since it is possible to figure out who wrote the paper

• from the reviewer's point of view, while it is possible to find out who wrote the paper, one cannot be sure so it can make reviewing less efficient

1) At least according to the PNAS paper above, the authors who would be most affected by double-blind peer review are the ones who are already highly decorated, since it is those authors who stand to benefit the most from having their names known to reviewers. Unknown authors essentially see no benefit/detriment.
2) Why should effort be placed into finding out who the author is? I would rather think that effort should be redirected into making sure the science is good. If the science is good, it will stand on its own merits, independent of its author.

 

[f95toli]

One question: if the main objection is "you could easily figure out who wrote the paper," then what's the harm in leaving the authors' names off the paper?

Edit: during review, that is. Obviously.

Again, many -in my field probably most- papers will contain "for details about the device design see reference [x]" making this irrelevant.
Papers that are completely "stand-alone" and do not build on previous work by the same authors are very, very rare.

Also, an important part of my job as a reviewer is to check for "novelty" and that what has been submitted in not just some minor change to what he authors have published elsewhere. I''d guess that about 50% of the times when I recommend rejection it is because the authors have previously published very similar results.

 

[Vanadium 50]

One question: if the main objection is "you could easily figure out who wrote the paper," then what's the harm in leaving the authors' names off the paper?

The names? Nothing but the extra work for no benefit. Citations that "out" the authors? That would make understanding of what's going on just go slower.

As far as I can tell, the biggest impact this would have would be to kill the arXiv.

 

[TeethWhitener]

Again, many -in my field probably most- papers will contain "for details about the device design see reference [x]" making this irrelevant.
Papers that are completely "stand-alone" and do not build on previous work by the same authors are very, very rare.

So then there's no harm if their names are left off the paper.

Also, an important part of my job as a reviewer is to check for "novelty" and that what has been submitted in not just some minor change to what he authors have published elsewhere. I''d guess that about 50% of the times when I recommend rejection it is because the authors have previously published very similar results.

If novelty is important, why does it matter who the authors are? Sure, if they're just extending a minor result onto results published elsewhere, it's a little crummy of them, but I don't see how a different author making a minor extension to their paper should fare any differently in the review process.

 

[TeethWhitener]

Nothing but the extra work for no benefit.

What extra work? Why do you need to know who wrote the paper? Why doesn't the science stand on its own?

Citations that "out" the authors? That would make understanding of what's going on just go slower.

I'm not sure I understand this comment. If you can't understand what's going on in the paper, then you should decline to review it.

As far as I can tell, the biggest impact this would have would be to kill the arXiv.

Why? Because you could go and compare the manuscript you're reviewing with a preprint on arxiv? Again, why do you need to put that much effort into knowing the author?

I understand that double blind peer review might be irrelevant in large collaborative efforts like particle physics or astronomy, but the vast majority of science isn't large collaborative efforts, and a lot of big names have published unimpressive results in impressive journals, to the extent that one openly wonders: did this get published solely by virtue of their name? One really easy way to make that cloud go away is via double blind peer review.

 

[f95toli]

So then there's no harm if their names are left off the paper.

But that would mean that some papers were reviewed double blind and some not. It would also mean that the author would -potentially- have the alter the way they write since they would then have to consider if they were happy with the reviewers being able to figure out who they were or not.
Also, well-known people could effectively decide against a double blind review simply by including a reference to their own work.

If novelty is important, why does it matter who the authors are? Sure, if they're just extending a minor result onto results published elsewhere, it's a little crummy of them, but I don't see how a different author making a minor extension to their paper should fare any differently in the review process.

Because there is no way any review can be familiar with ALL the literature in the field. If I did not know who the author were there it would be impossible for me to know if the same group had published pretty much the same thing in some journal that I don't read.
Or, as was the case in a paper I reviewed a couple of weeks ago, if the "novel" bit of the manuscript had already been included as part of a very long article the same group had written a couple of years ago (but where the main topic at first glance was very different).

Novelty is pretty much everything for high impact journals. If your work is not novel you won't get published irrespective of the quality of the work.

 

[TeethWhitener]

If I did not know who the author were there it would be impossible for me to know if the same group had published pretty much the same thing in some journal that I don' read.

I don't think this is true. For one, it misses the point of my objection: what if a different group had published pretty much the same thing in some journal that you don't read? Then knowing the authors' names would mean precisely diddly squat. For another, the scenario in the preceding sentence in fact did happen to me (twice, and rather recently) while I was reviewing a paper. I did what I normally do: google a few key words and phrases, and guess what...a very similar paper from a different group popped up within the first few searches, and I was able to recommend the manuscript for rejection based on that.

 

[TeethWhitener]

from the authors' point of view, anonymity is far from being assured, since it is possible to figure out who wrote the paper

Another thing: This is also (trivially) true in single-blind peer review, so I don't see how this counts as a drawback for double-blind peer review.

 

[Vanadium 50]

Why do you need to know who wrote the paper?

The point I am making is that I need to know the equipment used to evaluate the paper, and knowing the details of the equipment tells me who did the work. Furthermore, I will likely have already read the preprint. This is far more common than not: if I am asked to referee, it's because I am up-to-date in the field, and if I am keeping up with the field, I am reading the arXiv. You can't have both - either you go double-blind, or you stop publishing preprints.

 

[Demystifier]

Only looking at the list of references, it is often easy to figure out who wrote the paper

Not so famous authors would then make more citations of a certain famous author to create a false impression that the paper was written by that famous author and hence to increase the chances for publication.

 

[Demystifier]

As far as I can tell, the biggest impact this would have would be to kill the arXiv.

No. The arXiv could be modified so that the author is revealed only after the publication or acceptance.

 

[f95toli]

No. The arXiv could be modified so that the author is revealed only after the publication or acceptance.

But that would mean that there would be no way of keeping track of what the various groups in your field are doing(!).

 

[Demystifier]

But that would mean that there would be no way of keeping track of what the various groups in your field are doing(!).

But you could still keep track of what is done in your field. And the most important thing is what is done, not who did it, don't you agree?

 

[fresh_42]

But you could still keep track of what is done in your field. And the most important thing is what is done, not who did it, don't you agree?

I don't. If a famous scientist deals with a certain question it has more weight than if a nobody does, simply because the expectation of results is a different one. The former may mean a change of direction of research, the latter only means just another PhD.

 

[f95toli]

But you could still keep track of what is done in your field. And the most important thing is what is done, not who did it, don't you agree?

No, for two reasons. One one is that finding information by searching using the title and/or abstract can be quite inefficient since it is rarely obvious which keywords one should use, I frequently find information about e.g. new experimental methods, device designs etc because someone I know the name of in my field is was a co-author of the paper; as fresh_42 mentions this can suggest that a new research direction is becoming important.

The second reason is that part of my job is to keep track of the development in my field around the world; this means not just keeping track of the science but also of the science policy and/or funding situation (like most senior scientists I do have to contribute to e.g. policy and briefing documents) .
Hence, I do need to know what the "main players" in my field are up to at a given time even when the science they publish isn't that interesting.

 

[Demystifier]

@fresh_42 and @f95toli you made good arguments that it is important to know who is the author. But I don't see that you made arguments that it is more important than the actual content of the scientific work.

 

[Andy Resnick]

Double blind peer review is a process in which not only the author does not know the name of the reviewer (which is the standard practice), but also the reviewer does not know the name of the author (which is not a standard practice). What would happen if the scientific community suddenly decided to shift to the double blind review process? Who would be the winners? Who would be the losers?

I've been involved with all three flavors of peer review (as a reviewer)- single blind/double blind/ open, and while I'd like to think my evaluation process is identical for all three, I will note that I've always declined to provide my name (open review) and tend to read the introductory sections more cautiously (double blind as compared to single-blind).

What would be infinitely more interesting is a double-blind review of *grant proposals*, because the stakes are so much higher. To be sure, I'm not sure how review panels could function this way. For comparison, NIH review panels are quasi-open (the panel members are known, but the primary and secondary reviewers are not) while for every other agency and foundation that I have applied to, the review panels are at least single-blind.

 

[Andy Resnick]

This paper:
http://www.pnas.org/content/114/48/12708
from last year insinuates that famous authors, authors from top universities, and authors from top companies are significantly more likely to get their papers accepted using single-blind peer review (reviewer is anonymous, authors aren't) than double-blind.

The first sentence of their abstract is problematic:

"Scientific peer review has been a cornerstone of the scientific method since the 1600s."

For a contrary view, see:

https://physicstoday.scitation.org/doi/10.1063/PT.3.3463

Although, to be fair, the Physics Today article isn't peer-reviewed :)

 

[Vanadium 50]

What would be infinitely more interesting is a double-blind review of *grant proposals*, because the stakes are so much higher. To be sure, I'm not sure how review panels could function this way.

It would be impossible for DOE grants, where one criterion (of five) is "Competency of Applicant’s Personnel and Adequacy of Proposed Resources". It may even be statutory. (DOE and NSF grants are authorized under different actions)

 

[Andy Resnick]

It would be impossible for DOE grants, where one criterion (of five) is "Competency of Applicant’s Personnel and Adequacy of Proposed Resources". It may even be statutory. (DOE and NSF grants are authorized under different actions)

That may or may not be true; certainly all of my proposals (and reviews) have sections on something like "PI suitability" and "Environment", but AFAIK, in all cases the science section is an independent section.

 

[Orodruin]

Why? Because you could go and compare the manuscript you're reviewing with a preprint on arxiv? Again, why do you need to put that much effort into knowing the author?

It is not about putting active effort on finding the paper on arxiv. There would just be a very high probability that the reviewer has already seen and (if the editor chooses reviewers wisely) at leased skimmed through the paper and already know who the authors are.

 

[Dale]

Double blind peer review is a process in which not only the author does not know the name of the reviewer (which is the standard practice), but also the reviewer does not know the name of the author (which is not a standard practice).

Double blind peer review is standard in medical journals.

 

[atyy]

Double blind peer review is standard in medical journals.

It is not totally standard. According to https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5213965/#CR7, JAMA, NEJM and Lancet are single blind.

BMJ does open reviews https://bmjopen.bmj.com/pages/reviewerguidelines/.

 

[Dale]

It is not totally standard.

Well, all of the standard journals in my sub-field use it.

 

[atyy]

Well, all of the standard journals in my sub-field use it.

What is your sub-field?

 

[Dr. Courtney]

I've been so thorough in several reviews that the authors figured out who it was (or at least narrowed it down to my wife or me).

Our expertise is unique.

 

[MathematicalPhysicist]

You would lose too, since you wouldn't be able to write a paper about ER=EPR

So if $ER\ne 0$ then $P=1$?! :-D

Assuming commutativity of $P$ with either $E$ or $R$.