Is peer review in science flawed due to the popular equals correct fallacy?

[Mallignamius]

Is the scientific world, being based on peer review, in need of an overhaul for being based on a logical fallacy that "popular equals correct"?

I'm only asking because someone is telling me not to trust the majority of scientists on a specific issue, namely second hand smoking. I don't really know, but putting faith in a majority opinion was, I thought, reasonable.

I concede that the majority is not always right. After reading over some reports on the issue, I am inclined to agree with the majority more because of consensus rather than my own interpretation. I do not believe that I am qualified to scrutinize the studies, and so putting faith in the majority is what I thought I was supposed to do.

I guess he could be correct, but I'd appreciate some second opinions.

 

[Evo]

Is this person that's telling you they're wrong able to furnish you with equally sound studies that back what he is saying up?

Sure studies can prove wrong and the majority of popular opinion can be proven wrong. But I wouldn't just take someone's word for it.

In the case of second hand smoke, you don't find too many studies disputing it. If there were a lot of studies calling it into question, then there might be reason to take a closer look. That doesn't appear to be the case.

 

[zoobyshoe]

Is the scientific world, being based on peer review, in need of an overhaul for being based on a logical fallacy that "popular equals correct"?

I'm only asking because someone is telling me not to trust the majority of scientists on a specific issue, namely second hand smoking. I don't really know, but putting faith in a majority opinion was, I thought, reasonable.

I concede that the majority is not always right. After reading over some reports on the issue, I am inclined to agree with the majority more because of consensus rather than my own interpretation. I do not believe that I am qualified to scrutinize the studies, and so putting faith in the majority is what I thought I was supposed to do.

I guess he could be correct, but I'd appreciate some second opinions.

According to the Wikipedia, peer review does not represent a case of "majority rules":

In the case of proposed publications, an editor sends advance copies of an author's work or ideas to researchers or scholars who are experts in the field (known as "referees" or "reviewers"), normally by e-mail or through a web-based manuscript processing system. Usually, there are two or three referees for a given article.

These referees each return an evaluation of the work to the editor, including noting weaknesses or problems along with suggestions for improvement. Typically, most of the referees' comments are eventually seen by the author; scientific journals observe this convention universally. The editor, usually familiar to the field of the manuscript (although typically not in as much depth as the referees, who are specialists), then evaluates the referees' comments, their own opinion of the manuscript, and the context of the scope of the journal or level of the book and readership, before passing a decision back to the author(s), usually with the referees' comments.

Referees' evaluations usually include an explicit recommendation of what to do with the manuscript or proposal, often chosen from a menu provided by the journal or funding agency. Most recommendations are along the lines of the following:

* to unconditionally accept the manuscript or proposal,
* to accept it in the event that its authors improve it in certain ways,
* to reject it, but encourage revision and invite resubmission,
* to reject it outright.

During this process, the role of the referees is advisory, and the editor is typically under no formal obligation to accept the opinions of the referees. Furthermore, in scientific publication, the referees do not act as a group, do not communicate with each other, and typically are not aware of each other's identities or evaluations. There is usually no requirement that the referees achieve consensus. Thus the group dynamics are substantially different from that of a jury. In situations where the referees disagree substantially about the quality of a work, there are a number of strategies for reaching a decision.

When an editor receives very positive and very negative reviews for the same manuscript, the editor often will solicit one or more additional reviews as a tie-breaker. As another strategy in the case of ties, editors may invite authors to reply to a referee's criticisms and permit a compelling rebuttal to break the tie. If an editor does not feel confident to weigh the persuasiveness of a rebuttal, the editor may solicit a response from the referee who made the original criticism. In rare instances, an editor will convey communications back and forth between authors and a referee, in effect allowing them to debate a point. Even in these cases, however, editors do not allow referees to confer with each other, and the goal of the process is explicitly not to reach consensus or to convince anyone to change their opinions. Some medical journals, however (usually following the open access model), have begun posting on the Internet the pre-publication history of each individual article, from the original submission to reviewers' reports, authors' comments, and revised manuscripts.

Traditionally, reviewers would remain anonymous to the authors, but this standard is slowly changing. In some academic fields, most journals now offer the reviewer the option of remaining anonymous or not, or a referee may opt to sign a review, thereby relinquishing anonymity. Published papers sometimes contain, in the acknowledgments section, thanks to anonymous or named referees who helped improve the paper.

http://en.wikipedia.org/wiki/Peer_review

 

[Mallignamius]

Oh yeah, he's referred to other studies to back up his arguments. And then there is a counter, and it goes 'round 'n 'round.

But I'm more interested in knowing what I'm supposed to do in the broader sense. Am I wrong in some way for putting my trust in a majority consensus?

 

[Bystander]

Second hand smoke, second hand perfume, second hand vegetarian breath --- all offensive, and all can be demonstrated to raise heart rate, blood pressure, and reduce oxygen levels in "the victims." Holding my breath to get past the perfume counters in department stores is an elective response on my part, not a physiological response to the perfume.

Second hand biological pathogens --- plague, leprosy, malaria, tetanus and a few other such nasties are non-communicable; TB, colds, flu, hemmorhagic fevers, typhoid, are very communicable --- plenty of studies, good statistics, double-blinded methods, and public health risk factors that aren't reported in tenths of cases per million exposures.

There's "consensus," and, there's "consensus" on any topic --- the "consensus" among UFO freaks on whatever is not as dependable as the consensus among computer freaks that Moore's law has to break down one of these days.

Review and approval of an article does NOT imply agreement --- it implies that the reviewer found no errors in method(s), logic, or conclusions; 'fraid it also does NOT imply serious effort on the part of the reviewer --- prior to WW II it did, but not in the age of publish or perish.

 

[Moridin]

Nature's Peer Review Debate : http://www.nature.com/nature/peerreview/debate/index.html