Is the Peer Review Process Biased?

[Skyhunter]

Evo asked in another thread

I think it is a fair question to ask if the people doing peer review on papers submitted on AGW in major journals are without bias.

https://www.physicsforums.com/showpost.php?p=2079435&postcount=54

The short answer is no. The reviewers are human and subject to human bias. Bias is an intrinsic part of human nature. Scientific reviewers are not immune. The peer review process is not perfect.

http://en.wikipedia.org/wiki/Confirmation_bias"

In psychology and cognitive science, confirmation bias is a tendency to search for or interpret new information in a way that confirms one's preconceptions and to avoid information and interpretations which contradict prior beliefs. It is a type of cognitive bias and represents an error of inductive inference, or as a form of selection bias toward confirmation of the hypothesis under study or disconfirmation of an alternative hypothesis.

Here is an experiment that will demonstrate this phenomenon.

I am thinking of a rule.

Here are three numbers in a sequence that fit the rule.

2,4,6

Guess the rule and assign a level of confidence to your answer.

Example all even numbers, 50% confidence.

Then write down another 3 number sequence and I will say whether or not it fits my rule. When everyone reaches 100% confidence I will reveal the rule.

 

[Evo]

Probably one of the most infamous cases of abuse of the Peer Review Process was the case of Richard Sternberg circumventing peer review to sneak an Intelligent Design paper by ID proponent Stephen Meyer into a peer reviewed journal.

The peer review process

Sternberg insists the paper was properly peer reviewed, and rejects the reason given by the journal for disavowing the article, saying:

"As managing editor it was my prerogative to choose the editor who would work directly on the paper, and as I was best qualified among the editors I chose myself."[3]

Sternberg's statement implies that there were four outside reviewers. But of the four "well-qualified biologists with five PhDs" Sternberg identifies, one was Sternberg himself, contributing his double doctorate to the total he cited. Sternberg's claim of following proper peer review procedures directly contradicts the published public statement of his former employer, the publisher of the journal, that the proper procedures were not followed resulting in the article's retraction.[4] In previous years the Proceedings of the Biological Society of Washington had published yearly lists of all the people who had served as peer reviewers. That list is absent for 2004, the year of the incident. Sternberg has repeatedly refused to identify the three "well-qualified biologists", citing personal concerns over professional repercussions for them. Identifying the reviewers would have allowed the journal's board to validate Sternberg's claim to objectivity in having the article considered meritorious for publication. Previously, the reviewers of Sternberg's own published paper[11] were Sternberg's fellow Baraminology Study Group peer Todd Wodd, and prominent intelligent design proponents Paul Nelson and Jonathan Wells[12][13] both of whom are Fellows of the Discovery Institute[14] and colleagues of the author of the article at the source of the controversy, Stephen C. Meyer. The Discovery Institute is the hub of the intelligent design movement.

http://en.wikipedia.org/wiki/Sternberg_peer_review_controversy

 

[humanino]

2,4,6

Next is 10. Those are the odd prime numbers minus one.

 

[Skyhunter]

Next is 10. Those are the odd prime numbers minus one.

No that is not the rule.

 

[humanino]

What better alternative do you have to peer-review process ? Otherwise, I see little point to this. Of course you will always find counter examples where the protocole failed. It does not mean the protocole was badly designed. Take a situation where the protocole is not supposed to fail : peer-review process in mathematics is a good candidate. Do you know of instances where the peer-review process in mathematics has failed ? Trying to find an example at the opposite end of the spectrum : do you expect that a peer-review process in politics makes sense ?

 

[humanino]

No that is not the rule.

Why not ? I find it beautiful. Because you defined the rule does not mean everybody will find it most suitable to describe the sample.

 

[humanino]

No that is not the rule.

How about "prime numbers including one plus one" ?
edit
Nevermind

 

[ZapperZ]

But one also needs to define what is this "process" that one is talking about. And for many journal, it IS a process, not simply the handing out of a manuscript to one person, and letting that person decides the fate of that paper.

In physics, and especially for the Physical Review family of journals, the fact that this is a human evaluation is very much taken into consideration. A paper is given to more than one referee (often it is 2 or 3, or some time even 4! I've had one of my papers reviewed by 5 different people total by the time it was approved!). The author even has the ability to recommend potential referees.

Even when a manuscript gets rejected in the first round, there is always the opportunity for rebuttals and resubmission. Even an outright rejection is given due process for further consideration, etc.. etc. In other words, there's plenty of opportunity to make one's case heard by more than just one person and his/her bias. Having many of these people, and all these different processes rejecting something unanimously, for example, would tend to indicate that such rejection has less to do with bias and more to do with an unsuitable manuscript.

There is also another aspect to this, which Dan Koshland has brought up in his thoughtful article (D.E. Koshland, Jr., Nature v.432, p.447 (2004)).

"The trouble is that journals can easily become too conservative, because editors find it easier to reject the unusual than to take a chance on the unthinkable... The existence of multiple journals provides the final safeguard against too much conservatism and is the ultimate reason that science is more receptive to non-conformity than any other segment of our society."

It also means that one has plenty of opportunities not only to get published, but also to get around a perceived "bias" at one particular journal, IF the work is truly valid.

Zz.

 

[humanino]

Is it http://www.research.att.com/~njas/sequences/?q=2%2C4%2C6&language=english&go=Search !?

 

[John Creighto]

What better alternative do you have to peer-review process ? Otherwise, I see little point to this. Of course you will always find counter examples where the protocole failed. It does not mean the protocole was badly designed. Take a situation where the protocole is not supposed to fail : peer-review process in mathematics is a good candidate. Do you know of instances where the peer-review process in mathematics has failed ? Trying to find an example at the opposite end of the spectrum : do you expect that a peer-review process in politics makes sense ?

Just because there might not be a better way doesn’t make this discussion not worth while. In my opinion the word science is evoked as some all knowing authority whenever people want to silence debate. It would be helpful for the public if they realized that science doesn’t always agree and is not without human failings such as bias.

See page 7
http://republicans.energycommerce.house.gov/108/Hearings/07192006hearing1987/Wegman.pdf

With regards to the social network analysis of the mbh98 per review.

 

[John Creighto]

Is it http://www.research.att.com/~njas/sequences/?q=2%2C4%2C6&language=english&go=Search !?

That's a pretty cool link. :)

 

[Skyhunter]

I am thinking of a rule.

The sequence 2,4,6 fits my rule.

Offer another sequence and I will say whether or not it fits my rule.

Continue until you are 100% certain you know my rule.

 

[CRGreathouse]

I am thinking of a rule.

The sequence 2,4,6 fits my rule.

Offer another sequence and I will say whether or not it fits my rule.

There are uncountably many sequences that fit this rule, and we're supposed to guess it on the head?

My sequence is the first billion positive even numbers, followed by 0 and 1 alternately.

 

[Skyhunter]

There are uncountably many sequences that fit this rule, and we're supposed to guess it on the head?

No. You write another sequence and I tell you if it fits or not. When you are 100% certain you know the rule then you stop.

Here is a second sequence that fits the rule.

3,5,7

 

[John Creighto]

No. You write another sequence and I tell you if it fits or not. When you are 100% certain you know the rule then you stop.

Here is a second sequence that fits the rule.

3,5,7

What are you trying to prove? We would need some basis for assigning those confidence intervals. That is we would need some statistical model for your selection process. Moreover, weather your series fits in our interval or not doesn’t matter because one test case is not enough to confidently accept or reject a hypothesis. Finally I’m not sure the confidence intervals add linearly anyway.

 

[CRGreathouse]

No. You write another sequence and I tell you if it fits or not. When you are 100% certain you know the rule then you stop.

Here is a second sequence that fits the rule.

3,5,7

Ah. So instead of asking us to choose one of the 2^(aleph_0) integer sequences, you're asking us to choose one of the 2^(2^(aleph_0)) unary relations on the set of integer sequences.

I don't think my confidence in such a game could ever rise as high as 1%, let alone 100%.

(Of course perhaps you're limiting it to definable rules on integer sequences, of which there are 'only' aleph_0.)

 

[Skyhunter]

It is an exercise in confirmation bias.

It works best in a classroom. So forget about it.

 

[John Creighto]

It is an exercise in confirmation bias.

It works best in a classroom. So forget about it.

Here is the exact problem:

2-4-6 problem

From: A Dictionary of Psychology | Date: 2001 | Author: ANDREW M. COLMAN | © A Dictionary of Psychology 2001, originally published by Oxford University Press 2001. (Hide copyright information) Copyright information
2-4-6 problem n. A problem of concept formation in which people are given the ordered triple of numbers (2, 4, 6) and are invited to try to generate further examples of triples conforming to an unspecified rule that the example obeys, trying to home in on the correct rule on the basis of simple right/wrong feedback after every guess. The actual rule is any ascending sequence, but the example invites people to form more specific hypotheses, such as ascending even numbers or numbers ascending by equal intervals. It was introduced in 1960 by the English psychologist Peter C(athcart) Wason (1924–2003), who found that people tend to try examples consistent with such more specific hypotheses, such as (10, 20, 30) and seldom try examples that would refute them such as (10, 11, 30), thus manifesting confirmation bias and failing to find the right answer but often becoming increasingly convinced of the rightness of their incorrect hypotheses. Also called Wason's 2-4-6 problem. See also problem solving.

http://www.encyclopedia.com/doc/1O87-246problem.html


Were suppose to generate triples not sequences. Triples are subsets of sequences and not necessarily sequential subsets. The feed back we are suppose to get is weather or not the triple we give satisfies the rule. The bias is the result of only picking triples which satisfy the rule. In order to validate our model we should spend as much or more time trying to falsify it as we do trying to confirm it.

 

[Moridin]

Probably one of the most infamous cases of abuse of the Peer Review Process was the case of Richard Sternberg circumventing peer review to sneak an Intelligent Design paper by ID proponent Stephen Meyer into a peer reviewed journal.




http://en.wikipedia.org/wiki/Sternberg_peer_review_controversy

Surely, that is not an example of biased peer review, but of no proper peer review?

 

[nickyrtr]

Here is an anecdotal example of peer review bias:

I knew a grad student who told me his advisor once gave him a paper to review for him. The advisor told the student to "go easy" because the advisor knew who the author was, and was his personal friend.

 

[Gokul43201]

Here's an excerpt from a conference invite I received yesterday (these are not my opinions):

Only 8% members of the Scientific Research Society agreed that "peer review works well as it is". (Chubin and Hackett, 1990; p.192).

"A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research." (Horrobin, 2001)

Horrobin concludes that peer review "is a non-validated charade whose processes generate results little better than does chance." (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors.

But, "Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and "Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process" (Horrobin, 2001).

This is the purpose of the International Symposium on Peer Reviewing: ISPR (http://www.ICTconfer.org/ispr ) being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009 (http://www.ICTconfer.org/kgcm ), which will be held on July 10-13, 2009, in Orlando, Florida, USA.
...
[conference details]

 

[CRGreathouse]

Interesting. The "recent U.S. Supreme Court decision" is Daubert v. Merrell Dow Pharmaceuticals, Inc. from 15 years ago (though in fairness it was probably only 7 years old when the article was submitted). Most of the times that peer review is mentioned in that decision is rejecting non-peer reviewed documents -- an implicit support of the system. The only part (that I found, anyway) that questions the value of peer review is based on Horrobin himself:

Another pertinent consideration is whether the theory or technique has been subjected to peer review and publication. Publication (which is but one element of peer review) is not a sine qua non of admissibility; it does not necessarily correlate with reliability, see S. Jasanoff, The Fifth Branch: Science Advisors as Policymakers 61-76 (1990), and, in some instances, well-grounded but innovative theories will not have been published, see Horrobin, The Philosophical Basis of Peer Review and the Suppression of Innovation, 263 JAMA 1438 (1990). Some propositions, moreover, are too particular, too new, or of too limited interest to be published. But submission to the scrutiny of the scientific community is a component of "good science," in part because it increases the likelihood that substantive flaws in methodology will be detected. See J. Ziman, Reliable Knowledge: An Exploration of the Grounds for Belief in Science 130-133 (1978); Relman & Angell, How Good Is Peer Review?, 321 New Eng.J.Med. 827 (1989). The fact of publication (or lack thereof) in a peer reviewed journal thus will be a relevant, though not dispositive, consideration in assessing the scientific validity of a particular technique or methodology on which an opinion is premised.

This let's me draw two conclusions:
1. Horrobin is well-known and respected enough to be cited by the Supreme Court, and
2. Most of the faults listed above can be traced to a single author, and indeed a single article.