Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Is the Peer Review Process Biased?

  1. Feb 17, 2009 #1
    Evo asked in another thread

    The short answer is no. The reviewers are human and subject to human bias. Bias is an intrinsic part of human nature. Scientific reviewers are not immune. The peer review process is not perfect.

    http://en.wikipedia.org/wiki/Confirmation_bias" [Broken]

    Here is an experiment that will demonstrate this phenomenon.

    I am thinking of a rule.

    Here are three numbers in a sequence that fit the rule.


    Guess the rule and assign a level of confidence to your answer.

    Example all even numbers, 50% confidence.

    Then write down another 3 number sequence and I will say whether or not it fits my rule. When everyone reaches 100% confidence I will reveal the rule.
    Last edited by a moderator: May 4, 2017
  2. jcsd
  3. Feb 17, 2009 #2


    User Avatar

    Staff: Mentor

    Probably one of the most infamous cases of abuse of the Peer Review Process was the case of Richard Sternberg circumventing peer review to sneak an Intelligent Design paper by ID proponent Stephen Meyer into a peer reviewed journal.

    Last edited: Feb 17, 2009
  4. Feb 17, 2009 #3
    Next is 10. Those are the odd prime numbers minus one. :tongue2:
  5. Feb 17, 2009 #4
    No that is not the rule.
  6. Feb 17, 2009 #5
    What better alternative do you have to peer-review process ? Otherwise, I see little point to this. Of course you will always find counter examples where the protocole failed. It does not mean the protocole was badly designed. Take a situation where the protocole is not supposed to fail : peer-review process in mathematics is a good candidate. Do you know of instances where the peer-review process in mathematics has failed ? Trying to find an example at the opposite end of the spectrum : do you expect that a peer-review process in politics makes sense ?
  7. Feb 17, 2009 #6
    Why not ? I find it beautiful. Because you defined the rule does not mean everybody will find it most suitable to describe the sample.
  8. Feb 17, 2009 #7
    How about "prime numbers including one plus one" ?
    Nevermind :redface:
    Last edited: Feb 17, 2009
  9. Feb 17, 2009 #8


    User Avatar
    Staff Emeritus
    Science Advisor
    Education Advisor
    2018 Award

    But one also needs to define what is this "process" that one is talking about. And for many journal, it IS a process, not simply the handing out of a manuscript to one person, and letting that person decides the fate of that paper.

    In physics, and especially for the Physical Review family of journals, the fact that this is a human evaluation is very much taken into consideration. A paper is given to more than one referee (often it is 2 or 3, or some time even 4! I've had one of my papers reviewed by 5 different people total by the time it was approved!). The author even has the ability to recommend potential referees.

    Even when a manuscript gets rejected in the first round, there is always the opportunity for rebuttals and resubmission. Even an outright rejection is given due process for further consideration, etc.. etc. In other words, there's plenty of opportunity to make one's case heard by more than just one person and his/her bias. Having many of these people, and all these different processes rejecting something unanimously, for example, would tend to indicate that such rejection has less to do with bias and more to do with an unsuitable manuscript.

    There is also another aspect to this, which Dan Koshland has brought up in his thoughtful article (D.E. Koshland, Jr., Nature v.432, p.447 (2004)).

    It also means that one has plenty of opportunities not only to get published, but also to get around a perceived "bias" at one particular journal, IF the work is truly valid.

  10. Feb 17, 2009 #9
    Is it http://www.research.att.com/~njas/sequences/?q=2%2C4%2C6&language=english&go=Search [Broken] !?
    Last edited by a moderator: May 4, 2017
  11. Feb 17, 2009 #10
    Just because there might not be a better way doesn’t make this discussion not worth while. In my opinion the word science is evoked as some all knowing authority whenever people want to silence debate. It would be helpful for the public if they realized that science doesn’t always agree and is not without human failings such as bias.

    See page 7
    http://republicans.energycommerce.house.gov/108/Hearings/07192006hearing1987/Wegman.pdf [Broken]

    With regards to the social network analysis of the mbh98 per review.
    Last edited by a moderator: May 4, 2017
  12. Feb 17, 2009 #11
    That's a pretty cool link. :)
    Last edited by a moderator: May 4, 2017
  13. Feb 17, 2009 #12
    I am thinking of a rule.

    The sequence 2,4,6 fits my rule.

    Offer another sequence and I will say whether or not it fits my rule.

    Continue until you are 100% certain you know my rule.
  14. Feb 17, 2009 #13


    User Avatar
    Science Advisor
    Homework Helper

    There are uncountably many sequences that fit this rule, and we're supposed to guess it on the head?

    My sequence is the first billion positive even numbers, followed by 0 and 1 alternately.
  15. Feb 17, 2009 #14
    No. You write another sequence and I tell you if it fits or not. When you are 100% certain you know the rule then you stop.

    Here is a second sequence that fits the rule.

  16. Feb 17, 2009 #15
    What are you trying to prove? We would need some basis for assigning those confidence intervals. That is we would need some statistical model for your selection process. Moreover, weather your series fits in our interval or not doesn’t matter because one test case is not enough to confidently accept or reject a hypothesis. Finally I’m not sure the confidence intervals add linearly anyway.
  17. Feb 17, 2009 #16


    User Avatar
    Science Advisor
    Homework Helper

    Ah. So instead of asking us to choose one of the 2^(aleph_0) integer sequences, you're asking us to choose one of the 2^(2^(aleph_0)) unary relations on the set of integer sequences.

    I don't think my confidence in such a game could ever rise as high as 1%, let alone 100%.

    (Of course perhaps you're limiting it to definable rules on integer sequences, of which there are 'only' aleph_0.)
  18. Feb 17, 2009 #17
    It is an exercise in confirmation bias.

    It works best in a classroom. So forget about it.
  19. Feb 17, 2009 #18
    Here is the exact problem:

    http://www.encyclopedia.com/doc/1O87-246problem.html [Broken]

    Were suppose to generate triples not sequences. Triples are subsets of sequences and not necessarily sequential subsets. The feed back we are suppose to get is weather or not the triple we give satisfies the rule. The bias is the result of only picking triples which satisfy the rule. In order to validate our model we should spend as much or more time trying to falsify it as we do trying to confirm it.
    Last edited by a moderator: May 4, 2017
  20. Feb 17, 2009 #19
    Surely, that is not an example of biased peer review, but of no proper peer review?
  21. Feb 18, 2009 #20
    Here is an anecdotal example of peer review bias:

    I knew a grad student who told me his advisor once gave him a paper to review for him. The advisor told the student to "go easy" because the advisor knew who the author was, and was his personal friend.
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook