Peer Review Process and Bad Scientific Articles

[Pythagorean]

Is there a compendium of peer-reviewed journals that have been shown to be wrong?

I've heard of scientific journals that only have required two reviewers. How often does rubbish get by; not just typos and simple mistakes, but fundamentally unsound rubbish?

Is there a remedy process for these journals?


At my research internship, during research we would follow an assertion back through three or four references to find that it was a (more or less) baseless intuitive prediction made in the 60's that just kind of propagated through the literature to today.

 

[27Thousand]

Is there a compendium of peer-reviewed journals that have been shown to be wrong?

I've heard of scientific journals that only have required two reviewers. How often does rubbish get by; not just typos and simple mistakes, but fundamentally unsound rubbish?

Is there a remedy process for these journals?


At my research internship, during research we would follow an assertion back through three or four references to find that it was a (more or less) baseless intuitive prediction made in the 60's that just kind of propagated through the literature to today.

Good question.

Keeping that in mind, nothing can be proven 100%. It's what's more trustworthy. Part of the Scientific Method is replication and public verification. If you talk to someone and they share a personal experience with you, for all I know is they could be lying. However, I feel much more safe looking at a peer-review journal because it can be verified/replicated (some even personally you can replicate, even if very sloppy controls compared to what they do).

Something to consider, people also have the same exact personal experiences and come to different conclusions, or interpret them differently (before the Scientific Method there were those who were Rationalists who said think rather than experience). At the same time what looks good on paper isn't necessarily how it happens in the real world (some were Empirical not Rationalists before the Scientific Method). Then the structure of the Scientific Method combines the two, to help minimize the weaknesses of these two methods and combine the strengths.

 

[f95toli]

I've heard of scientific journals that only have required two reviewers. How often does rubbish get by; not just typos and simple mistakes, but fundamentally unsound rubbish?

Very few journals use more than two reviewers, the only journals I know of that regularly use more than two are Nature and Science (they use four or five)

Peer-review is far from perfect and it is inevitable that there will be errors in many papers. However, if the "system" works most of those errors will be in the discussion/conclusions and will simply be due to the fact that the authors are drawing the wrong conclusions from their data or analysis. Errors in that part of the paper does not mean that the whole paper is rubbish, as long as the experiment/analysis has been performed correctly it can still be a good paper since other people can use the very same data to eventually draw the correct conclusion.

The topic I an currently working it quite "hot" and there are several groups doing very similar experiments. The data from the different groups agree quite well but we all interpret the data somewhat differently and use different models to fit the data, it is inevitable that some of us (perhaps me) will turn out to be wrong. But as long as we all specify when we are speculating and are clear about the assumptions behind our models, even the papers that turns out to be "wrong" will contribute to the solution of the problem we are working on.

That said, there are of course many papers where it is quite clear that the authors did not really understand what they were doing in the experiment, meaning the data is not incorrect as such but does not show what the authors think it is showing. I guess you could label these papers as "rubbish" but even then there are occasions when some of that data turns out to be useful to someone else.

 

[Moonbear]

Most journals only use two reviewers, and will send out a manuscript to a third only if the first two reviews are vastly different (i.e., one recommends rejection and the other acceptance). If a manuscript is rejected, there is usually a process to appeal it if you don't think the reviews were fair or reasonable. Most of the time, though, they are, and people just take the comments offered to improve the manuscript and send it off to another journal (unless some real fatal flaw is found that makes it entirely unpublishable).

The important thing in a journal article is that the methods are sound with all the appropriate controls and results are presented accurately. The conclusions drawn or discussion presented may remain up for debate as follow-up experiments are performed that further test those conclusions.

 

[Monique]

Is there a compendium of peer-reviewed journals that have been shown to be wrong?

I've heard of scientific journals that only have required two reviewers. How often does rubbish get by; not just typos and simple mistakes, but fundamentally unsound rubbish?

Is there a remedy process for these journals? At my research internship, during research we would follow an assertion back through three or four references to find that it was a (more or less) baseless intuitive prediction made in the 60's that just kind of propagated through the literature to today.

You can search for retractions of articles, I've come across them several times. For instance:

Using the methods published in Reese and Haimo (2000), the author's laboratory has been unable to reproduce the data in Figs. 2–6 of this article that demonstrate microtubule motor and dynactin regulation as a function of the direction of organelle transport in melanophores. The authors wish to correct the scientific record and retract the findings of this paper until such time as they are able to resolve the discrepancy.

I think most people do not pay much attention to low impact journals, the higher impact ones are critical of what papers they'll accept and reviewers will advice accordingly. Some things just cannot be prevented from being published, such as deliberate fabrication of data or experimental mistakes. Only the scientific process can correct that, as has happened in the above quote.

 

[mikelepore]

Alongside the problem mentioned in the first post of "unsound rubbish" getting published, there are examples of journals rejecting important papers. One case that I hear discussed a lot is when Science and Nature declined to publish the results of Lauterbur, the inventor of MRI, who later won the Nobel Prize for it.

 

[Moonbear]

Alongside the problem mentioned in the first post of "unsound rubbish" getting published, there are examples of journals rejecting important papers. One case that I hear discussed a lot is when Science and Nature declined to publish the results of Lauterbur, the inventor of MRI, who later won the Nobel Prize for it.

That does happen once in a while. However, when the work is sound and novel, it does eventually become recognized. Though, just because something is a great technology doesn't mean that the manuscript originally submitted met the standards of a particular journal. Science and Nature also reject a lot of good quality research, simply because they get far more submissions than they could ever publish. Nobody takes it as an insult to be rejected by them; most consider it an accomplishment just to get granted a full review by them. Usually if an author thinks work is worthy of submission to those journals, they have no trouble getting it submitted to another top quality journal more specific to their field if one of those journals rejects it.

 

[Loren Booda]

Publishing scientific papers often relies more on the machine which churns them out than original, profound and testable ideas.

 

[Vanadium 50]

Publishing scientific papers often relies more on the machine which churns them out than original, profound and testable ideas.

Do you have any evidence for this claim?

As has been pointed out, the very premise is flawed. Peer-review doesn't mean a paper is guaranteed to be right. Indeed, if you take 100 papers that claim a result to a 95% confidence level, you'd expect about 5 of them to be wrong. But science is self-correcting: when further measurements fail to reproduce it, eventually it all gets sorted out.

In theory, things are even more complicated, as a wrong but interesting result can be very useful. The classic example is the theory of Yang and Mills, who were trying to write down a theory of hadronic interactions. They were wrong. However, Yang-Mills theories came to become the foundation of weak interactions, and QCD, and grand unified theories. Right answer, wrong problem.

If we kept everything wrong out of the journals, we wouldn't have this. And we'd be worse off.

 

[27Thousand]

Publishing scientific papers often relies more on the machine which churns them out than original, profound and testable ideas.

Something to maybe consider, have you noticed that some whites say that blacks look all the same, while some blacks say the same about whites? It's like when you're not familar with something, it seems all the same. Well, how do you not know the same is with peer-review journals in many cases? I mean, I hear people who hate music or American Idol say that nothing creative comes from it and it's all the same. Then I hear some who hate Science and don't know much about it say those who get Nobel Prizes aren't coming up with anything very new. In reality, virtually every invention/scientific discovery/anything seen as creative is just taking ideas which already existed and combining them into new ways (although quite often many others came to the same exact idea even if the person who got credit didn't know about it).

A woman makes a wedding cake out of baby diapers. All the other women flip out saying that it's original. Although I'm not a woman, I decided to go to Google and looked up "diaper wedding cakes" in quotes and 18,400 results came up, most women who make them didn't invent the idea themselves but are rather copying someone else. Whoever made up diaper wedding cakes just took two ideas which already existed, diapers and wedding cakes, and combined them. Albert Einstein's influential E=MC^2 took ideas which already existed, E and M existed for hundreds of years already, and the speed of light existed.

 

[Loren Booda]

Do you have any evidence for this claim?

As has been pointed out, the very premise is flawed. Peer-review doesn't mean a paper is guaranteed to be right. Indeed, if you take 100 papers that claim a result to a 95% confidence level, you'd expect about 5 of them to be wrong. But science is self-correcting: when further measurements fail to reproduce it, eventually it all gets sorted out.

In theory, things are even more complicated, as a wrong but interesting result can be very useful. The classic example is the theory of Yang and Mills, who were trying to write down a theory of hadronic interactions. They were wrong. However, Yang-Mills theories came to become the foundation of weak interactions, and QCD, and grand unified theories. Right answer, wrong problem.

If we kept everything wrong out of the journals, we wouldn't have this. And we'd be worse off.

I appreciate your focus on the process rather than the individual paper. I didn't recall the Yang-Mills story but will refer to it in the future.

I would still like to hear of some quantitative measure ranking scientific papers (whose possibility you seem to indicate), much like students get graded. Maybe the scientific method could be given a boost by a means beyond scientific papers?

 

[Vanadium 50]

Why do you want to "grade" scientific papers? What problem are you trying to solve by doing this?

 

[Loren Booda]

At least democratic consensus eventually grades them. The request I make is toward a more consistent standard for hundreds of thousands of articles every year in (each of?) the major sciences. The process of peer review itself could thus be better replicable.

My guess is that the shear number reflects not so much the growth of science but the pressure to publish, and slows the process of good research. What if peer reviewers' competence was compromized by their own shoddy papers being accepted?

 

[Pythagorean]

relevant:

http://math.rejecta.org/

 

[seycyrus]

Do you have any evidence for this claim?

How do I preface this to avoid legalistic problems?

*In my experience* Loren's claim is becoming more and more accurate. Publication by letterhead is indeed a problem. Ever come across a paper with atrocious grammar and/or syntactical mistakes? IMO, such papers are more likely to pass the process if they originate from a *prestigious* institution.

 

[Galteeth]

There was this-

http://en.wikipedia.org/wiki/Bogdanov_Affair

 

[27Thousand]

Considering that we use Science for technology, have used it to put man on the moon, discover black holes, mitochondria, etc, I'm thinking Science may not be absolute truth but it's a very useful tool for relative truth and is the best thing we have.