Is the Peer Review Process Biased?

In summary, the conversation discussed the potential bias of peer review in scientific journals, with the conclusion that bias is an inherent part of human nature and can affect the peer review process. The concept of confirmation bias was also mentioned, along with an experiment to demonstrate it. The conversation then shifted to a specific case of alleged abuse of the peer review process, where a paper on Intelligent Design was published in a scientific journal, causing controversy. The discussion then delved into the peer review process in mathematics and the importance of multiple referees and opportunities for rebuttals in the process. It was also mentioned that the existence of multiple journals helps prevent excessive conservatism in the review process. Overall, the conversation highlighted the complexities and potential flaws of the peer
  • #1

Skyhunter

Evo asked in another thread
I think it is a fair question to ask if the people doing peer review on papers submitted on AGW in major journals are without bias.
https://www.physicsforums.com/showpost.php?p=2079435&postcount=54

The short answer is no. The reviewers are human and subject to human bias. Bias is an intrinsic part of human nature. Scientific reviewers are not immune. The peer review process is not perfect.

http://en.wikipedia.org/wiki/Confirmation_bias" [Broken]

In psychology and cognitive science, confirmation bias is a tendency to search for or interpret new information in a way that confirms one's preconceptions and to avoid information and interpretations which contradict prior beliefs. It is a type of cognitive bias and represents an error of inductive inference, or as a form of selection bias toward confirmation of the hypothesis under study or disconfirmation of an alternative hypothesis.

Here is an experiment that will demonstrate this phenomenon.

I am thinking of a rule.

Here are three numbers in a sequence that fit the rule.

2,4,6

Guess the rule and assign a level of confidence to your answer.

Example all even numbers, 50% confidence.

Then write down another 3 number sequence and I will say whether or not it fits my rule. When everyone reaches 100% confidence I will reveal the rule.
 
Last edited by a moderator:
Physics news on Phys.org
  • #2
Probably one of the most infamous cases of abuse of the Peer Review Process was the case of Richard Sternberg circumventing peer review to sneak an Intelligent Design paper by ID proponent Stephen Meyer into a peer reviewed journal.

The peer review process

Sternberg insists the paper was properly peer reviewed, and rejects the reason given by the journal for disavowing the article, saying:

"As managing editor it was my prerogative to choose the editor who would work directly on the paper, and as I was best qualified among the editors I chose myself."[3]
:uhh:
Sternberg's statement implies that there were four outside reviewers. But of the four "well-qualified biologists with five PhDs" Sternberg identifies, one was Sternberg himself, contributing his double doctorate to the total he cited. Sternberg's claim of following proper peer review procedures directly contradicts the published public statement of his former employer, the publisher of the journal, that the proper procedures were not followed resulting in the article's retraction.[4] In previous years the Proceedings of the Biological Society of Washington had published yearly lists of all the people who had served as peer reviewers. That list is absent for 2004, the year of the incident. Sternberg has repeatedly refused to identify the three "well-qualified biologists", citing personal concerns over professional repercussions for them. Identifying the reviewers would have allowed the journal's board to validate Sternberg's claim to objectivity in having the article considered meritorious for publication. Previously, the reviewers of Sternberg's own published paper[11] were Sternberg's fellow Baraminology Study Group peer Todd Wodd, and prominent intelligent design proponents Paul Nelson and Jonathan Wells[12][13] both of whom are Fellows of the Discovery Institute[14] and colleagues of the author of the article at the source of the controversy, Stephen C. Meyer. The Discovery Institute is the hub of the intelligent design movement.

http://en.wikipedia.org/wiki/Sternberg_peer_review_controversy
 
Last edited:
  • #3
Skyhunter said:
2,4,6
Next is 10. Those are the odd prime numbers minus one. :tongue2:
 
  • #4
humanino said:
Next is 10. Those are the odd prime numbers minus one. :tongue2:

No that is not the rule.
 
  • #5
What better alternative do you have to peer-review process ? Otherwise, I see little point to this. Of course you will always find counter examples where the protocole failed. It does not mean the protocole was badly designed. Take a situation where the protocole is not supposed to fail : peer-review process in mathematics is a good candidate. Do you know of instances where the peer-review process in mathematics has failed ? Trying to find an example at the opposite end of the spectrum : do you expect that a peer-review process in politics makes sense ?
 
  • #6
Skyhunter said:
No that is not the rule.
Why not ? I find it beautiful. Because you defined the rule does not mean everybody will find it most suitable to describe the sample.
 
  • #7
Skyhunter said:
No that is not the rule.
How about "prime numbers including one plus one" ?
edit
Nevermind :redface:
 
Last edited:
  • #8
But one also needs to define what is this "process" that one is talking about. And for many journal, it IS a process, not simply the handing out of a manuscript to one person, and letting that person decides the fate of that paper.

In physics, and especially for the Physical Review family of journals, the fact that this is a human evaluation is very much taken into consideration. A paper is given to more than one referee (often it is 2 or 3, or some time even 4! I've had one of my papers reviewed by 5 different people total by the time it was approved!). The author even has the ability to recommend potential referees.

Even when a manuscript gets rejected in the first round, there is always the opportunity for rebuttals and resubmission. Even an outright rejection is given due process for further consideration, etc.. etc. In other words, there's plenty of opportunity to make one's case heard by more than just one person and his/her bias. Having many of these people, and all these different processes rejecting something unanimously, for example, would tend to indicate that such rejection has less to do with bias and more to do with an unsuitable manuscript.

There is also another aspect to this, which Dan Koshland has brought up in his thoughtful article (D.E. Koshland, Jr., Nature v.432, p.447 (2004)).

"The trouble is that journals can easily become too conservative, because editors find it easier to reject the unusual than to take a chance on the unthinkable... The existence of multiple journals provides the final safeguard against too much conservatism and is the ultimate reason that science is more receptive to non-conformity than any other segment of our society."

It also means that one has plenty of opportunities not only to get published, but also to get around a perceived "bias" at one particular journal, IF the work is truly valid.

Zz.
 
  • #9
Is it http://www.research.att.com/~njas/sequences/?q=2%2C4%2C6&language=english&go=Search [Broken] !?
 
Last edited by a moderator:
  • #10
humanino said:
What better alternative do you have to peer-review process ? Otherwise, I see little point to this. Of course you will always find counter examples where the protocole failed. It does not mean the protocole was badly designed. Take a situation where the protocole is not supposed to fail : peer-review process in mathematics is a good candidate. Do you know of instances where the peer-review process in mathematics has failed ? Trying to find an example at the opposite end of the spectrum : do you expect that a peer-review process in politics makes sense ?

Just because there might not be a better way doesn’t make this discussion not worth while. In my opinion the word science is evoked as some all knowing authority whenever people want to silence debate. It would be helpful for the public if they realized that science doesn’t always agree and is not without human failings such as bias.

See page 7
http://republicans.energycommerce.house.gov/108/Hearings/07192006hearing1987/Wegman.pdf [Broken]

With regards to the social network analysis of the mbh98 per review.
 
Last edited by a moderator:
  • #11
humanino said:
Is it http://www.research.att.com/~njas/sequences/?q=2%2C4%2C6&language=english&go=Search [Broken] !?

That's a pretty cool link. :)
 
Last edited by a moderator:
  • #12
I am thinking of a rule.

The sequence 2,4,6 fits my rule.

Offer another sequence and I will say whether or not it fits my rule.

Continue until you are 100% certain you know my rule.
 
  • #13
Skyhunter said:
I am thinking of a rule.

The sequence 2,4,6 fits my rule.

Offer another sequence and I will say whether or not it fits my rule.

There are uncountably many sequences that fit this rule, and we're supposed to guess it on the head?

My sequence is the first billion positive even numbers, followed by 0 and 1 alternately.
 
  • #14
There are uncountably many sequences that fit this rule, and we're supposed to guess it on the head?
No. You write another sequence and I tell you if it fits or not. When you are 100% certain you know the rule then you stop.

Here is a second sequence that fits the rule.

3,5,7
 
  • #15
Skyhunter said:
No. You write another sequence and I tell you if it fits or not. When you are 100% certain you know the rule then you stop.

Here is a second sequence that fits the rule.

3,5,7

What are you trying to prove? We would need some basis for assigning those confidence intervals. That is we would need some statistical model for your selection process. Moreover, weather your series fits in our interval or not doesn’t matter because one test case is not enough to confidently accept or reject a hypothesis. Finally I’m not sure the confidence intervals add linearly anyway.
 
  • #16
Skyhunter said:
No. You write another sequence and I tell you if it fits or not. When you are 100% certain you know the rule then you stop.

Here is a second sequence that fits the rule.

3,5,7

Ah. So instead of asking us to choose one of the 2^(aleph_0) integer sequences, you're asking us to choose one of the 2^(2^(aleph_0)) unary relations on the set of integer sequences.

I don't think my confidence in such a game could ever rise as high as 1%, let alone 100%.

(Of course perhaps you're limiting it to definable rules on integer sequences, of which there are 'only' aleph_0.)
 
  • #17
It is an exercise in confirmation bias.

It works best in a classroom. So forget about it.
 
  • #18
Skyhunter said:
It is an exercise in confirmation bias.

It works best in a classroom. So forget about it.

Here is the exact problem:

2-4-6 problem

From: A Dictionary of Psychology | Date: 2001 | Author: ANDREW M. COLMAN | © A Dictionary of Psychology 2001, originally published by Oxford University Press 2001. (Hide copyright information) Copyright information
2-4-6 problem n. A problem of concept formation in which people are given the ordered triple of numbers (2, 4, 6) and are invited to try to generate further examples of triples conforming to an unspecified rule that the example obeys, trying to home in on the correct rule on the basis of simple right/wrong feedback after every guess. The actual rule is any ascending sequence, but the example invites people to form more specific hypotheses, such as ascending even numbers or numbers ascending by equal intervals. It was introduced in 1960 by the English psychologist Peter C(athcart) Wason (1924–2003), who found that people tend to try examples consistent with such more specific hypotheses, such as (10, 20, 30) and seldom try examples that would refute them such as (10, 11, 30), thus manifesting confirmation bias and failing to find the right answer but often becoming increasingly convinced of the rightness of their incorrect hypotheses. Also called Wason's 2-4-6 problem. See also problem solving.
http://www.encyclopedia.com/doc/1O87-246problem.html [Broken]


Were suppose to generate triples not sequences. Triples are subsets of sequences and not necessarily sequential subsets. The feed back we are suppose to get is weather or not the triple we give satisfies the rule. The bias is the result of only picking triples which satisfy the rule. In order to validate our model we should spend as much or more time trying to falsify it as we do trying to confirm it.
 
Last edited by a moderator:
  • #19
Evo said:
Probably one of the most infamous cases of abuse of the Peer Review Process was the case of Richard Sternberg circumventing peer review to sneak an Intelligent Design paper by ID proponent Stephen Meyer into a peer reviewed journal.

:uhh:


http://en.wikipedia.org/wiki/Sternberg_peer_review_controversy

Surely, that is not an example of biased peer review, but of no proper peer review?
 
  • #20
Here is an anecdotal example of peer review bias:

I knew a grad student who told me his advisor once gave him a paper to review for him. The advisor told the student to "go easy" because the advisor knew who the author was, and was his personal friend.
 
  • #21
Here's an excerpt from a conference invite I received yesterday (these are not my opinions):
Only 8% members of the Scientific Research Society agreed that "peer review works well as it is". (Chubin and Hackett, 1990; p.192).

"A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research." (Horrobin, 2001)

Horrobin concludes that peer review "is a non-validated charade whose processes generate results little better than does chance." (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors.

But, "Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and "Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process" (Horrobin, 2001).

This is the purpose of the International Symposium on Peer Reviewing: ISPR (http://www.ICTconfer.org/ispr [Broken]) being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009 (http://www.ICTconfer.org/kgcm [Broken]), which will be held on July 10-13, 2009, in Orlando, Florida, USA.
...
[conference details]
 
Last edited by a moderator:
  • #22
Interesting. The "recent U.S. Supreme Court decision" is Daubert v. Merrell Dow Pharmaceuticals, Inc. from 15 years ago (though in fairness it was probably only 7 years old when the article was submitted). Most of the times that peer review is mentioned in that decision is rejecting non-peer reviewed documents -- an implicit support of the system. The only part (that I found, anyway) that questions the value of peer review is based on Horrobin himself:
Another pertinent consideration is whether the theory or technique has been subjected to peer review and publication. Publication (which is but one element of peer review) is not a sine qua non of admissibility; it does not necessarily correlate with reliability, see S. Jasanoff, The Fifth Branch: Science Advisors as Policymakers 61-76 (1990), and, in some instances, well-grounded but innovative theories will not have been published, see Horrobin, The Philosophical Basis of Peer Review and the Suppression of Innovation, 263 JAMA 1438 (1990). Some propositions, moreover, are too particular, too new, or of too limited interest to be published. But submission to the scrutiny of the scientific community is a component of "good science," in part because it increases the likelihood that substantive flaws in methodology will be detected. See J. Ziman, Reliable Knowledge: An Exploration of the Grounds for Belief in Science 130-133 (1978); Relman & Angell, How Good Is Peer Review?, 321 New Eng.J.Med. 827 (1989). The fact of publication (or lack thereof) in a peer reviewed journal thus will be a relevant, though not dispositive, consideration in assessing the scientific validity of a particular technique or methodology on which an opinion is premised.

This let's me draw two conclusions:
1. Horrobin is well-known and respected enough to be cited by the Supreme Court, and
2. Most of the faults listed above can be traced to a single author, and indeed a single article.
 

1. What is the peer review process?

The peer review process is a method of evaluating and improving the quality of scientific research by having experts in the same field review and provide feedback on a manuscript before it is published in a scientific journal.

2. Is the peer review process biased?

There is no definitive answer to this question as bias can exist in any human-led process. However, the peer review process is designed to minimize bias by having multiple reviewers from different backgrounds and perspectives, and by keeping the identity of the authors anonymous to the reviewers.

3. How does the peer review process prevent bias?

The peer review process has several mechanisms in place to prevent bias. First, the reviewers are chosen based on their expertise and reputation in the field. Second, the identity of the authors is kept anonymous to the reviewers to prevent any potential bias based on the authors' reputation or background. Third, the peer review process involves multiple reviewers, reducing the impact of any individual's bias.

4. Are there any potential sources of bias in the peer review process?

Yes, there are potential sources of bias in the peer review process, such as personal opinions, conflicts of interest, and unconscious biases. However, the process is constantly evolving, and steps are being taken to address and minimize these biases. For example, some journals now use double-blind peer review, where the reviewers' identities are also kept anonymous to each other.

5. How can we ensure the peer review process remains unbiased?

To ensure the peer review process remains unbiased, it is essential to have transparency, diversity, and accountability in the process. Journals should be transparent about their review process, including their criteria for selecting reviewers and the feedback provided to authors. Diversity in the pool of reviewers should also be encouraged to reduce the impact of any individual's bias. Lastly, journals should have a system in place for addressing any concerns or complaints about potential bias in the peer review process.

Suggested for: Is the Peer Review Process Biased?

Replies
5
Views
508
Replies
1
Views
744
Replies
34
Views
3K
Replies
8
Views
1K
Replies
6
Views
1K
Replies
3
Views
905
Replies
15
Views
2K
Back
Top