Peer Review: Objective or Subjective?

[Andre]

The scientific methods as developed by several philologists like Carl Popper provide objective tools for developing new hypothesis and theories. That’s theory though but far from daily practice where subjective elements play a role, making science rather conservative. Apparently it’s very hard to distinguish crackpots and frauds from genuine improvement.

Not true? How about last year Nobel Price Laureates for medicine?

http://nobelprize.org/nobel_prizes/medicine/laureates/2005/press.html

This year's Nobel Laureates in Physiology or Medicine made the remarkable and unexpected discovery that inflammation in the stomach (gastritis) as well as ulceration of the stomach or duodenum (peptic ulcer disease) is the result of an infection of the stomach caused by the bacterium Helicobacter pylori…

In 1982, when this bacterium was discovered by Marshall and Warren, stress and lifestyle were considered the major causes of peptic ulcer disease.

Notice that general recognition took 23 years from the discovery in 1982 to last year. How many lives could have been saved if that period would have been significant shorter? Why took it so long?

Bluntly, non scientific, economic motives, the threat against the establishment:

Practitioners of gastroenterology and the pharmaceutical industry were both heavily invested in the theory that peptic ulcers were caused by emotional stress and stomach acids, and could only be treated with repeated courses of antacid medication. While the reduction of stomach acid often alleviated the existing ulcer, inflammation of the stomach lining usually persisted, and most patients found themselves returning in a year or two with another ulcer. Patients were routinely advised to seek psychiatric counseling, find less demanding employment or make other drastic lifestyle changes to address the purported cause of their disease. Volumes were published detailing the alleged psychological causes of gastric ailments, and ulcers remained a frequently cited example of psychosomatic illness.

In this environment, the possibility that the ailment was directly caused by a single micro-organism that could be completely eliminated with a two-week course of antibiotics was a threat to the status quo. While many of Marshall's critics had serious scientific questions about his hypothesis, others may have had economic motives in disputing his findings, and Marshall was not shy about saying so. The targets of his criticism soon sought to discredit him and his research. One prominent gastroenterologist dismissed him as "a crazy guy saying crazy things."

http://www.achievement.org/autodoc/printmember/mar1bio-1

Fortunately the medical world has learned from that and Elsevier throws the first stone, reviewing peer review.

http://www.intl.elsevierhealth.com/journals/mehy/

Medical Hypotheses takes a deliberately different approach to peer review. Most contemporary practice tends to discriminate against radical ideas that conflict with current theory and practice. Medical Hypotheses will publish radical ideas, so long as they are coherent and clearly expressed. Furthermore, traditional peer review can oblige authors to distort their true views to satisfy referees, and so diminish authorial responsibility and accountability. In Medical Hypotheses, the authors' responsibility for the integrity, precision and accuracy of their work is paramount. The editor sees his role as a 'chooser', not a 'changer': choosing to publish what are judged to be the best papers from those submitted.

Hopefully more will follow.

 

[selfAdjoint]

Quote: "A 'chooser' not a 'changer': choosing to publish what are judged to be the best papers from those submitted".

In other words the journal is the predjudices of the editor, writ large. How does replacing a small number of gateway individuals with a single one solve the problems of tribal bias?

 

[Andre]

Good question, nevertheless, it's a start. After all there is a good reason. If the objective is not to be embarrassed by being on the wrong side, or perhaps in the medical profession as being not responsible for resisting a live saving progress, then perhaps there is hope.

Perhaps it would also be an idea for the editors to argue why a particular article is good or bad in terms of objective use of the scientific method, avoiding selective use of data, using sound algorithms, etc, etc. Apparently it’s no longer allowed to dismiss it because John Authority et al (1895) have a different opinion which is accepted with a broad consensus.

 

[Andre]

Ran into an ancient but very accurate assessment about the reluctance to let go of flawed hypotheses and theories:

"When a hypothesis is no longer considered as a mere probable supposition, but a real fact; a philosopher not only acquiesces in it, and thereby mistakes the cause of one particular appearance; but, by its analogies, he mistakes the cause of other appearances too, and is led into a whole system of error. A philosopher who has been long attached to a favourite hypothesis, and especially if he have distinguished himself by his ingenuity in discovering or pursuing it, will not, sometimes, be convinced of its falsity by the plainest evidence of fact. Thus both himself, and all his followers, are put upon false pursuits, and seem determined to warp the whole course of nature, to suit their manner of conceiving of its operations."

from: Priestly, J., 1769, The History and Present State of Electricity (second edition): J. Dodsley and others, London, p. 420.

 

[J77]

I too focus on:

The editor sees his role as a 'chooser', not a 'changer': choosing to publish what are judged to be the best papers from those submitted.

Sounds like standard practice to me.

Editor makes the final decision here, like they would at the moment.

For example, authors can quite reasonably argue, if they feel justified, against a referee's rejection and ask the editor to make the decision themselves.

Plus, I certainly wouldn't put Elsevier up there with progress in ethics in publishing.

Certainly, they produce fine prints. However, I've heard several public remonstrations about their subscriptions; that is, apparently it's unreasonable to buy individual subscriptions. The only cost effective way for libraries is to buy bulk subscriptions, including many titles they may not want.

 

[NickJ]

Perhaps the best reaction to problems with peer review in the medical profession is to do away with paper journals and publish everything online: let every voice have a say, and let there be the possibility of replying to every article (sort of like this forum, but "more professional"). If you are a doctor, what can it hurt to read lots of articles, even if some are crazy? If you are a layperson and don't know what's bunk and what's not, ask your doctor. If she doesn't know, suspend judgment.

Of course, the drawback to this is spam, etc. But there are ways to monitor that.

I fail to see the purpose of peer review when we do away with the limitation of only being able to print so many pages in so many issues per year: if page space is unlimited, there's no need for a select few to censor content -- let the marketplace of medical ideas be like every other free-enterprise marketplace. The good ideas will survive and the bads ones will not stay in circulation. And of course there are ways to keep the quacks from submitting every luny idea -- peer pressure, not peer review.