Is the Peer Review Process Biased?

~~Skyhunter~~ · Feb17-09, 06:27 PM

It is an exercise in confirmation bias.

It works best in a classroom. So forget about it.

**John Creighto** · Feb17-09, 07:13 PM

Originally Posted by Skyhunter

It is an exercise in confirmation bias.

It works best in a classroom. So forget about it.

Here is the exact problem:

2-4-6 problem

From: A Dictionary of Psychology | Date: 2001 | Author: ANDREW M. COLMAN | Â© A Dictionary of Psychology 2001, originally published by Oxford University Press 2001. (Hide copyright information) Copyright information
2-4-6 problem n. A problem of concept formation in which people are given the ordered triple of numbers (2, 4, 6) and are invited to try to generate further examples of triples conforming to an unspecified rule that the example obeys, trying to home in on the correct rule on the basis of simple right/wrong feedback after every guess. The actual rule is any ascending sequence, but the example invites people to form more specific hypotheses, such as ascending even numbers or numbers ascending by equal intervals. It was introduced in 1960 by the English psychologist Peter C(athcart) Wason (1924–2003), who found that people tend to try examples consistent with such more specific hypotheses, such as (10, 20, 30) and seldom try examples that would refute them such as (10, 11, 30), thus manifesting confirmation bias and failing to find the right answer but often becoming increasingly convinced of the rightness of their incorrect hypotheses. Also called Wason's 2-4-6 problem. See also problem solving.

http://www.encyclopedia.com/doc/1O87-246problem.html

Were suppose to generate triples not sequences. Triples are subsets of sequences and not necessarily sequential subsets. The feed back we are suppose to get is weather or not the triple we give satisfies the rule. The bias is the result of only picking triples which satisfy the rule. In order to validate our model we should spend as much or more time trying to falsify it as we do trying to confirm it.

**Moridin** · Feb17-09, 10:32 PM

Originally Posted by Evo

Probably one of the most infamous cases of abuse of the Peer Review Process was the case of Richard Sternberg circumventing peer review to sneak an Intelligent Design paper by ID proponent Stephen Meyer into a peer reviewed journal.

http://en.wikipedia.org/wiki/Sternbe...ew_controversy

Surely, that is not an example of biased peer review, but of no proper peer review?

**nickyrtr** · Feb18-09, 07:32 AM

Here is an anecdotal example of peer review bias:

I knew a grad student who told me his advisor once gave him a paper to review for him. The advisor told the student to "go easy" because the advisor knew who the author was, and was his personal friend.

**Gokul43201** · Feb19-09, 09:25 AM

Here's an excerpt from a conference invite I received yesterday (these are not my opinions):

Only 8% members of the Scientific Research Society agreed that "peer review works well as it is". (Chubin and Hackett, 1990; p.192).

"A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research." (Horrobin, 2001)

Horrobin concludes that peer review "is a non-validated charade whose processes generate results little better than does chance." (Horrobin, 2001). This has been statistically proven and reported by an increasing number of journal editors.

But, "Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein, 2000), it is a necessary condition in quality assurance for Scientific/Engineering publications, and "Peer Review is central to the organization of modern science…why not apply scientific [and engineering] methods to the peer review process" (Horrobin, 2001).

This is the purpose of the International Symposium on Peer Reviewing: ISPR (http://www.ICTconfer.org/ispr) being organized in the context of The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009 (http://www.ICTconfer.org/kgcm), which will be held on July 10-13, 2009, in Orlando, Florida, USA.
...
[conference details]

**CRGreathouse** · Feb19-09, 11:18 AM

Interesting. The "recent U.S. Supreme Court decision" is Daubert v. Merrell Dow Pharmaceuticals, Inc. from 15 years ago (though in fairness it was probably only 7 years old when the article was submitted). Most of the times that peer review is mentioned in that decision is rejecting non-peer reviewed documents -- an implicit support of the system. The only part (that I found, anyway) that questions the value of peer review is based on Horrobin himself:

Another pertinent consideration is whether the theory or technique has been subjected to peer review and publication. Publication (which is but one element of peer review) is not a sine qua non of admissibility; it does not necessarily correlate with reliability, see S. Jasanoff, The Fifth Branch: Science Advisors as Policymakers 61-76 (1990), and, in some instances, well-grounded but innovative theories will not have been published, see Horrobin, The Philosophical Basis of Peer Review and the Suppression of Innovation, 263 JAMA 1438 (1990). Some propositions, moreover, are too particular, too new, or of too limited interest to be published. But submission to the scrutiny of the scientific community is a component of "good science," in part because it increases the likelihood that substantive flaws in methodology will be detected. See J. Ziman, Reliable Knowledge: An Exploration of the Grounds for Belief in Science 130-133 (1978); Relman & Angell, How Good Is Peer Review?, 321 New Eng.J.Med. 827 (1989). The fact of publication (or lack thereof) in a peer reviewed journal thus will be a relevant, though not dispositive, consideration in assessing the scientific validity of a particular technique or methodology on which an opinion is premised.

This lets me draw two conclusions:
1. Horrobin is well-known and respected enough to be cited by the Supreme Court, and
2. Most of the faults listed above can be traced to a single author, and indeed a single article.