Most scientific papers are probably wrong?

In summary, a new analysis suggests that most published scientific research papers are wrong due to problems with experimental and statistical methods, researcher bias, and selective reporting. Even large, well-designed studies are not immune to these issues. However, the scientific community is constantly working to correct and verify research findings, making science a self-correcting and trustworthy system. This is in stark contrast to religious dogma, which does not undergo the same level of scrutiny and questioning.
  • #1
Astronuc
Staff Emeritus
Science Advisor
2023 Award
21,870
6,271
Most scientific papers are probably wrong!?

from New Scientist.

Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.

John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.
http://www.newscientist.com/article.ns?id=dn7915

Well that would certainly undermine the credibility of the scientific community.

The rush to publish could be a factor, as well as publications from questionable work, e.g. cold fusion.
 
Physics news on Phys.org
  • #2
Astronuc said:
Well that would certainly undermine the credibility of the scientific community.
Yes, though I'm not so sure that it should. Scientists aren't robots and are as prone to mistakes as everyone else. As long as science remains open and honest, mistakes will be exposed and corrected, and its true integrity will be intact.
The rush to publish could be a factor, as well as publications from questionable work, e.g. cold fusion.
Rushing creates mistakes everywhere, but P&F are a bad example of that. IIRC, they first "published" their "theory" in the Wall Street Journal. They circumvented the scientific process.
 
  • #3
Astronuc said:
from New Scientist.

http://www.newscientist.com/article.ns?id=dn7915

Well that would certainly undermine the credibility of the scientific community.

The rush to publish could be a factor, as well as publications from questionable work, e.g. cold fusion.

I just wish New Scientist pays attention to this report itself, since it has the tendency to report on outlandish ideas that often have not been given any reasonable validity.

People tend to forget that publications in science journals is only the FIRST STEP in disseminating ideas and discoveries for others to study, disect, and verify. You need to get published first before this process can start. Publication does not automatically means validity. If it does, the BCS theory does not have to wait for 15 years for their authors to win the Nobel Prize. Even the dubious Fleishman and Pons cold fusion paper was published, and it was only after that can we clealy verify that what they reported didn't happen!

I do not know what is the rate of falsified work there are in physics journals. I would hazzard a guess that it is smaller in the most prestigous journals such as Nature, Science, and PRL. It is the lower-tier journals that have weaker requirements and refereeing that may be the culprit. But hey, the quacks need some source to bolster their theory.

Zz.
 
  • #4
edit2:

This study itself was not peer-reviewed, was it? :confused:
 
Last edited by a moderator:
  • #5
ZapperZ said:
I just wish New Scientist pays attention to this report itself, since it has the tendency to report on outlandish ideas that often have not been given any reasonable validity.
I put New Scientist in the same category with OMNI magazine. Scientific American is so much better. I agree that NS publishes peoples' outlandish ideas, and do not put a lot of stock in their publications. I certainly would not publish with them.

ZapperZ said:
People tend to forget that publications in science journals is only the FIRST STEP in disseminating ideas and discoveries for others to study, disect, and verify. You need to get published first before this process can start. Publication does not automatically means validity. If it does, the BCS theory does not have to wait for 15 years for their authors to win the Nobel Prize. Even the dubious Fleishman and Pons cold fusion paper was published, and it was only after that can we clealy verify that what they reported didn't happen!
I agree.

ZapperZ said:
I do not know what is the rate of falsified work there are in physics journals. I would hazzard a guess that it is smaller in the most prestigous journals such as Nature, Science, and PRL.
I would hope that the rate of falsified work is very low, less than a few percent, and actually I would rather people did not falsify work - it will be discovered sooner or later, especially if someone tries to repeat the work, apply it, or test it.

I think there maybe a greater degree of misinterpretation, or faulty extrapolation. At least, that has been my experience. And I have seen highly detailed technical reports that contained faulty data and conclusions, e.g. from simple mistakes like a faulty calibration in the initial phase of the work, which was not discovered until after the final report was issued!

It is the lower-tier journals that have weaker requirements and refereeing that may be the culprit. But hey, the quacks need some source to bolster their theory.
Yeah, and I wish they would stop doing that! :grumpy:
 
  • #6
Rach3 said:
edit2:

This study itself was not peer-reviewed, was it? :confused:
Who knows? I think perhaps maybe it's a self-critique of New Scientist. :rofl:
 
  • #7
Do you notice something though? Herein lies the obvious contradictions for the claims of many quacks and people who attact science and scientific establishements.

One one hand, there is this attack that tries to equate religious dogma and scientific ideas and principles. How many times have you seen people saying that science is no different than organized religion? Yet, IF this report is anywhere even half true, there clearly is ample evidence that even when an idea or discovery is published in scientific journals, the scientific community still challenges its validity and can falsify something if it isn't valid! It doesn't matter how high of a stature the authors are (just ask Betram Batlogg and the Bell Labs debacle from a few years ago). Everyone is subjected to the same scrutiny and will have to go through the rigorous process of having a published paper accepted as being valid.

Now when was the last time you see a religious idea and practice having something similar, and done on a regular basis?

Zz.
 
  • #8
ZapperZ said:
Now when was the last time you see a religious idea and practice having something similar, and done on a regular basis?
Well, I don't want to get into a religious discussion, but in orthodox Judaism, there is considerable study of the Torah and Talmud, and part of that study is questioning just about every passage. That is one reason that Judaism has persisted for about 2700+ years with the Torah in written form as a basis.
 
  • #9
Astronuc said:
Well, I don't want to get into a religious discussion, but in orthodox Judaism, there is considerable study of the Torah and Talmud, and part of that study is questioning just about every passage. That is one reason that Judaism has persisted for about 2700+ years with the Torah in written form as a basis.

But (i) this is not a common practice in practically all of the religions of the world and (ii) even for judaism, you don't hear such rate of "falsification" and challenges going on on a weekly basis (see any issue of PRL).

The point I'm trying to get across is that when people accuse science of being nothing more than a religion, they're trying to emphasize a point that scientific dogma doesn't change and can't be challenged, just because their point of view was dismissed by science. They somehow do not consider the possibility that their point of view is faulty and that's why it was dismissed. The fact that science DOES change when we have a valid theory and discovery, and that established ideas do get challenged often, clearly falsify such claims of having any similarity to religious practices and beliefs.

Zz.
 
  • #10
ZapperZ said:
I just wish New Scientist pays attention to this report itself, since it has the tendency to report on outlandish ideas that often have not been given any reasonable validity.

.

To use an analogy New Scientist is the Sun Newspaper , Nature the Daily mail and Physics world the Times.:smile:

I'd imagine it depends also on the field as to how accurrate a paper is? I mean if it's mostly maths whoever's reading it may well come up with a solid contention if someone forgot to divide by infinity or something. Where as an esoteric theory about human consciousness could well sound plausible but be total tripe.
 
  • #11
ZapperZ said:
Now when was the last time you see a religious idea and practice having something similar, and done on a regular basis?
What are you talking about? Why, just last month in "The New Fanaticist" magazine, didn't they refute the Pope's claim that Jesus was the Son of God? Apparently, his findings have been irreproducible, and they are now questioning his research methods and scrutinizing his credentials.
 
  • #12
ZapperZ said:
But (i) this is not a common practice in practically all of the religions of the world and (ii) even for judaism, you don't hear such rate of "falsification" and challenges going on on a weekly basis (see any issue of PRL).

The point I'm trying to get across is that when people accuse science of being nothing more than a religion, they're trying to emphasize a point that scientific dogma doesn't change and can't be challenged, just because their point of view was dismissed by science. They somehow do not consider the possibility that their point of view is faulty and that's why it was dismissed. The fact that science DOES change when we have a valid theory and discovery, and that established ideas do get challenged often, clearly falsify such claims of having any similarity to religious practices and beliefs.
I agree. My comment on religion was too abbreviated.

The issue in any system of thinking, or more to the point, is the basis or foundation of that system, and whether or not that basis is purely dogmatic. Hopefully, scientists are objective, but certainly this is not always the case.

An important part of being a scientist is the ability to 'change' or 'revise' one's thinking or understanding upon the discovery of new information or evidence, which contradicts one's view at that point. One must be prepared to challenge one's own thinking and understanding.

Another important part is simple integrity of the person and the process.
 
  • #13
The bottom line is that religious scholars do not do experimental work, hoping for "repeatable experiments" and "falsifiable predictions" - for the simple reason that religion is not science! (Why would it be?) It would be fantastically absurd to see an issue of exactly how should people worship, being decided by a well-designed physical experiment using superconductors and lasers.

It's apples and oranges.
 
  • #14
I heard someone tonight on the radio mention . . . are you ready for this . . . using science to prove reincarnation. :rofl:
 
  • #15
Astronuc said:
I heard someone tonight on the radio mention . . . are you ready for this . . . using science to prove reincarnation. :rofl:

a little offtopic, but, why are you a laughing?
 

1. Why are most scientific papers likely to be wrong?

There are a few reasons why this statement may be true. One is that the scientific method involves constant testing and refining of hypotheses, meaning that initial findings may not always hold up over time. Additionally, there may be biases or errors in the research process that can lead to incorrect conclusions.

2. Is this statement a reflection of the quality of scientific research?

No, this statement does not necessarily mean that the quality of scientific research is poor. In fact, it is a testament to the rigorous and self-correcting nature of science. By constantly questioning and reevaluating our findings, we can improve our understanding and knowledge of the world.

3. How can we trust the findings of scientific papers if they are often wrong?

While it is true that individual studies may have flaws or errors, the scientific community relies on the process of peer review to help identify and correct these issues. Additionally, scientific findings are often replicated by other researchers, providing further support for their validity.

4. Does this mean that we should disregard all scientific research?

No, it is important to remember that while individual studies may have limitations, the overall body of scientific research is still a valuable source of knowledge and understanding. It is always important to critically evaluate the evidence and consider multiple studies on a topic before drawing conclusions.

5. What can be done to reduce the likelihood of incorrect scientific papers?

One solution is to continue promoting and practicing good research practices, such as replicating studies and being transparent about methods and data. Additionally, increased collaboration and communication among scientists can help to identify and correct errors in research. Ongoing efforts to improve the peer review process can also help to ensure the quality and accuracy of scientific papers.

Similar threads

  • General Discussion
Replies
6
Views
261
Replies
11
Views
2K
  • General Discussion
Replies
9
Views
2K
  • General Discussion
Replies
12
Views
1K
  • General Discussion
Replies
19
Views
2K
Replies
1
Views
1K
  • General Discussion
Replies
4
Views
605
Replies
6
Views
4K
  • General Discussion
Replies
1
Views
823
  • General Discussion
Replies
6
Views
1K
Back
Top