Is the peer review process vulnerable to fake reviews?

[mfb]

Publisher retracts 64 articles for fake peer reviews

Apparently some journals let paper authors suggest reviewers - including their email addresses. How could this possibly go wrong?
230 in three years is probably just the tip of the iceberg. I wonder how many more pass with more subtle fake email addresses.

 

[Bystander]

Springer?!

 

[Dr. Courtney]

Caveat emptor.

I've never really viewed the peer review system as anything approaching a guarantee of quality in science. I've read too many bad papers, submitted too many comments and letters pointing out obvious errors in published papers, only to have editors let the errors stand about 1/2 the time, and also had pretty good work rejected for challenging the status quo. I've also had papers that I've soundly rejected as a peer-reviewer (in agreement with other peer reviewers) quickly published when submitted to the next journal.

Agreement with independent experiments and experimental repeatability are the gold standard for arbitrating good science from bad. At best, peer review is a temporary expedient for assessing probabilities for that to eventually occur. At worst, it is an old-boys network assuring the "in" crowd gets their papers published (to assure promotion, tenure, and funding) and keep the "out" crowd from becoming established.

Scientists need to follow the reasoning carefully and always be on guard to distinguish good experimental design from bad and to follow whether the theory and experiment really agree as claimed. Being in a reviewed print journal makes a paper no more likely to be of high quality than being on an eprint server like arXiv.

 

[mfb]

Personally I mainly see how publishing in high-energy physics works.
The experimental collaborations all have internal review processes, and those work very well. Every preprint that goes to arXiv (and everything goes to arXiv) got more peer review than most papers in other areas of science. Peer review from a journal is just a formality that takes too long to wait for.
Theory papers are a bit different, as they are written by much smaller groups. For some papers, some authors don't even bother to get it published - everyone will use the arXiv version anyway. For new ideas peer review is still important, and an arXiv entry presenting something completely new can be problematic.

 

[Evo]

Not surprised at all that it's Springer, I've always questioned their validity.

 

[mfb]

It's not just Springer.
Elsevier is on the list as well, together with some smaller publishers.

 

[Greg Bernhardt]

I think this arena is ripe for investigation and ultimately scandal. If only Aaron Swartz were still alive.

 

[atyy]

Apparently some journals let paper authors suggest reviewers - including their email addresses.

Do physics journals not do this? A number of biology and general science journals do it. I had imagined it standard practice.

From http://journals.aps.org/authors/web-submission-guidelines-physical-review
Editorial Info
Here you have the option to provide any additional information you think may be useful to the editor. Also, for certain manuscripts, you will need to tell us why your manuscript is suitable for submission to that journal or section. Here you are encouraged to suggest the names of potential referees. Such suggestions are particularly welcome when a manuscript treats a highly specialized subject. The editors are, of course, not constrained to select a referee from those suggested. Please note: If you suggest the names of referees at any time, we will retain that information in our database. However, the names are not subsequently displayed upon resubmission (although you have the opportunity to enter more names then).

 

[ZapperZ]

This is why the process of maintaining a respected journal is not easy. The better ones (and let's face it, we know which ones they are) often maintain a detailed database of referees, even having their own evaluation of the performance of these referees. I've seen papers to these journals being reviewed by 3 to 4 referees, and it could be more if the referees themselves have differing opinions. Many of these meticulous process and information are seldom adopted in the smaller, less well-known journals, because it is tedious, expensive, and time-consuming.

Zz.

 

[mfb]

Do physics journals not do this? A number of biology and general science journals do it. I had imagined it standard practice.

I've never seen anyone discussing or making referee suggestions. High-energy physics is special (see post 4) and I don't know how other fields do this, but I would at least expect the journals to verify the existence of the referees and their email addresses. It is such an obvious thing to do.

 

[atyy]

I've never seen anyone discussing or making referee suggestions. High-energy physics is special (see post 4) and I don't know how other fields do this, but I would at least expect the journals to verify the existence of the referees and their email addresses. It is such an obvious thing to do.

Absolutely. My advisor once told me an anecdote in which he did get his own paper to review. Apparently, he was somewhere in the middle of a long list of authors. He told the journal that he thought it was a great paper! Of course, he also alerted them to the anomaly:)

 

[Dale]

Being in a reviewed print journal makes a paper no more likely to be of high quality than being on an eprint server like arXiv.

I disagree. Peer review is not a quality panacea, but it certainly does increase quality. I have never had a paper whose quality did not improve as a result of peer review.

 

[DrClaude]

I have never considered it wrong that journals ask for suggestions for referees. It all depends on what they do with them. I have always assume that when a journal asks for, say, 3 possible referees, it means that at most one of them will be selected, and another referee will be chosen outside those suggestions.

There are of course also important differences between journals. Submissions to Physical Reviews are handled by editors that are not researchers, but then again they have a very big database of potential referees and, I believe, quite a well-organized process to maintain quality. Some good specialist journals have editors who are in the field and know who's who, so they can send manuscripts to appropriate referees. And then there are lazy journals...

 

[Andy Resnick]

I've never seen anyone discussing or making referee suggestions. High-energy physics is special (see post 4) and I don't know how other fields do this, but I would at least expect the journals to verify the existence of the referees and their email addresses. It is such an obvious thing to do.

Every Journal I have ever submitted to (with impact factors ranging from 4 to near 0) has requested detailed contact information for at least referees. I've mentioned on a previous thread that I consider requesting suggested referees to be a serious ethical problem- especially since key words/classification codes are also required- and this OP clearly describes a logical consequence.

Ethical lapses in manuscript review is bad enough, consider if this spreads to the selection of ad-hoc peer reviewers for proposals (I do this on occasion for various granting agencies).

 

[ZapperZ]

Ethical lapses in manuscript review is bad enough, consider if this spreads to the selection of ad-hoc peer reviewers for proposals (I do this on occasion for various granting agencies).

I'm not so sure if this is a significant problem, at least for DOE and NSF grants (I don't have any experience with NIH). As I recall, you are asked, in the grant proposal, people you have collaborated with in the past .. what, 2 years? Presumably, this is used by the funding agency to disqualify certain people from reviewing the proposal. Furthermore, with NSF and DOE, the referees don't have the final say. There is still a committee that eventually go over all the proposal, look at the money that are available, and then make their rankings and recommendations to the funding agency.

So there is definitely quite a bit of checks and balances here from my experience.

Zz.

 

[Andy Resnick]

<snip>

So there is definitely quite a bit of checks and balances here from my experience.

Zz.

Right- just like there's checks and balances in manuscript review. And, just as top-ranked journals are less likely to experience the problem we are discussing, top-ranked granting agencies are less likely to experience conflict-of-interest problems, or at least have clear procedures in place to identify and resolve COI issues.

 

[Dr. Courtney]

My experience is that there is a much more detailed background and paperwork when I review funding proposals compared with when I review papers. I've reviewed papers in many cases where the editor may have had nothing more than my email address and knowledge of related papers I've authored. When I've reviewed grant proposals, the agency has my full CV, publication history, recent collaboration history, conflict of interest forms, and (in most cases) payment information. I've also met agency representatives in person for the agencies for which I review grant proposals. I have never met representatives for most journals who request peer reviews.

 

[ZapperZ]

My experience is that there is a much more detailed background and paperwork when I review funding proposals compared with when I review papers. I've reviewed papers in many cases where the editor may have had nothing more than my email address and knowledge of related papers I've authored. When I've reviewed grant proposals, the agency has my full CV, publication history, recent collaboration history, conflict of interest forms, and (in most cases) payment information. I've also met agency representatives in person for the agencies for which I review grant proposals. I have never met representatives for most journals who request peer reviews.

I have met Editors and assistant editors of Nature, Science, and the Phys. Rev. journals. Many of them regularly attend the APS March and April Meetings. 95% of the manuscripts that I refereed were for those journals.

Zz.

 

[mathwonk]

To the best of my recollection, I have never encountered a math journal which gave the author any input at all for choice of referees, nor betrayed any info about their identity. I sometimes have a guess that the referee may be the person who made the conjecture I am solving but I have never had confirmation, so to this day I do not know the identity of a single referee for any paper I have ever published. Indeed some of my papers have appeared without significant changes suggested but others have been completely rewritten to the point of doubling in length, due to added details, examples, clarification. One paper that had originally cited a number of "well known" results that were actually not published, expanded perhaps by a factor of 5 when all that material was meticulously included. When I have been a referee for a distinguished journal it was usually by receiving a paper from an editor whom I personally know or who was referred to me because of my record of publishing in the given area. Most of the time I know the editor from research contacts over the years. Often I know the author personally as well but I have never revealed to any of them that I was their referee, and I have not always accepted work written by acquaintances, if it seemed inappropriate for the journal. I think this is standard practice. Of course maybe other practices occurred and I was just unaware of them.

In the case of grant proposals to NSF, the author is now perhaps allowed to give a list of referees who should not be contacted, and perhaps in more recent times maybe also a list of potential referees, but one does not learn which ones are chosen, and the referees are contacted independently by the funding agencies. Besides, they are normally so well known that fakes would not be viable, i.e. full professors at elite schools. I have never submitted an email address for a potential referee but maybe such things occur now. I have been asked to serve on NSF granting committees, and they now meet together and discuss all decisions, after preliminary evaluation by the individual judges. I.e. one has to defend at least to some extent, granting recommendations, to ones peers.

Once I submitted a paper I rather liked to a top journal and after its being refereed the editor who was a friend of mine told me that while the paper was not considered high enough quality for the journal i submitted to, he was also editor of another less selective journal that he could guarantee acceptance to if I would agree to submit it there, so I did. On a few other occasions editors have solicited papers after learning of their content and before I had submitted them for refereeing. I assumed in those cases they would go out for refereeing, but that the editor obviously already had a favorable opinion of them. In a few instances I may have missed out on having them appear in more prestigious organs, but i bypassed the stress of submitting them, and in some cases benefited from being allowed to write them at greater length in a less selective journal than in one of greater demand.

On at least one case, a referee criticized my submission to a top journal as too long and as having an error, about which he was mistaken. Not wanting to shorten the paper I did not bother to explain the correct but questioned statements and resubmitted to a different excellent but slightly less exclusive journal. The editor sent it to the same referee who was upset I had not addressed his erroneous concern and rejected it again, but the kindly editor chose another second referee, obviously a specialist, who ignored the supposedly problematic statements but insisted we greatly shorten the paper and that it then be accepted. We never published the excised portions of the work but they became known informally from privately circulated copies of the manuscript..

Years later we received several requests for the material which we had been forced to remove and this material formed the foundation for several other publications by others, who even named those never published results after us. This was an odd experience, but in essentislly all cases I feel I was always treated with great fairness by referees and editors, and I support fully the concept of peer refereeing. Of course at the time I may have felt annoyed to have to rewrite a paper but in basically all cases the result was an imporved and more accessible work. In fact looking back I appreciate most the papers that were required to be improved before appearing.

In the modern nternet era there is a new phenomenon, of journals I have never heard of routinely emailing me with a request to submit papers, but I have always ignored these, as much in the category of proffered nigerian financial schemes.

As regards Springer, I have no first hand knowledge, but their reputation is slipping in the community, and I have read that ownership passed from an old school gentleman who felt publishing was a service to the scientific community into the hands of a publisher who wants to make as much money as possible by overcharging libraries for research subscriptions.

 

[OldEngr63]

I have only been asked to suggest reviewers in the past 2 or 3 years; prior to that, I was never asked for anyone to review my papers.

In my case, I think it is a simple matter of editors not having enough people knowledgeable in a particular area. For my situation, I think this is largely because I work in an area of declining interest in the US, even though it is still very active in other parts of the world.

 

[Dr. Courtney]

On at least one case, a referee criticized my submission to a top journal as too long and as having an error, about which he was mistaken. Not wanting to shorten the paper I did not bother to explain the correct but questioned statements and resubmitted to a different excellent but slightly less exclusive journal. The editor sent it to the same referee who was upset I had not addressed his erroneous concern and rejected it again, but the kindly editor chose another second referee, obviously a specialist, who ignored the supposedly problematic statements but insisted we greatly shorten the paper and that it then be accepted. We never published the excised portions of the work but they became known informally from privately circulated copies of the manuscript..

Years later we received several requests for the material which we had been forced to remove and this material formed the foundation for several other publications by others, who even named those never published results after us. This was an odd experience, but in essentislly all cases I feel I was always treated with great fairness by referees and editors, and I support fully the concept of peer refereeing. Of course at the time I may have felt annoyed to have to rewrite a paper but in basically all cases the result was an imporved and more accessible work. In fact looking back I appreciate most the papers that were required to be improved before appearing.

I'm not always a fan of cutting detail just because a referee or an editor doesn't like the length. For us, the solution has been to leave the extra detail in the arXiv version and remove the extra length from the version for the print journal. We will make changes that we really regard as improvements to the arXiv version.

Our experience has been the mainline physics journals (IOP and AIP) are very well edited and refereed. Having published work in many other fields, I am sad to report that our experiences in other fields has not been as positive. There has been a lot of stuff that is more like turf battles than legitimate efforts to improve the science and the communication.

 

[mathwonk]

I confess as an old guy with minimal internet savvy, I have never posted a single arxiv version, but that is indeed an excellent solution. I have instead posted the fuller version on my website which may perhaps never be consulted by anyone.