Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Reproducibility of previous research reports

  1. Aug 28, 2015 #1

    jim mcnamara

    User Avatar

    Staff: Mentor

    http://www.sciencemag.org/content/349/6251/aac4716
    Science 28 August 2015:
    Vol. 349 no. 6251
    DOI: 10.1126/science.aac4716

    Nosek B. , et al., Estimating the reproducibility of psychological science

    35 of 97 reports of statistically significant results published in three major psychology journals from 2008
    could be duplicated.
    In light of https://www.physicsforums.com/threads/peer-review-your-own-papers.829065/ which discusses reviewing your own paper for publication, this whole topic needs discussion. - 'reliability of research'.

    My point of view: We already have enough non-scientists fabricating garbage like 'autism is caused by vaccinations' without adding to the pile of stuff we have to refute.

    For example: Dealing with 'wrong' research comes across to the non-scientist with a strong subtext - science based on research may not be reliable, so why should I accept it? To them it could sound like more of a politcal debate. We just went throught the whole deal of removng cholesterol - the molecule - from top of the list as a cause of heart disease. Everyday people and physicians trying to communicate with Joe Public may still use the terms 'cholesterol' and 'bad cholesterol' to mean CLDL levels.

    Old bad research is really hard to get past sometimes. Especially when Captain Kirk (Wm. Shatner) was pushing margarine made with trans-fats and talking about cholesterol for years on TV here in the US. People are sometimes floored when I tell them I should eat several eggs per week. -- I have a slightly damaged heart.
     
  2. jcsd
  3. Aug 28, 2015 #2

    mfb

    User Avatar
    2016 Award

    Staff: Mentor

    In other words, roughly half of the "significant results" in those journals are wrong. Don't trust a publication there unless it is repeated or you have very good evidence that this study has been done better than the average.

    It is certainly an issue beyond psychology. "This psychology study was wrong, why should I trust this study about [other topic]?"
    The much higher replication rate in other fields is not visible to the general public.
     
  4. Aug 28, 2015 #3
  5. Aug 28, 2015 #4

    WWGD

    User Avatar
    Science Advisor
    Gold Member

    Why not get graduate students to do repeat work (while supervised) for their Stats classes? An innocence project of sorts (where students gave their effort for free ).
     
    Last edited: Aug 28, 2015
  6. Aug 28, 2015 #5

    Ygggdrasil

    User Avatar
    Science Advisor

    Perhaps this is why many don't consider psychology to be a "hard" science.

    Of course, biology is subject to these issues as well. Studies reporting positive outcomes in clinical trials (i.e. results strong enough to reject the null hypothesis) decreased dramatically once it became standard practice to pre-register clinical trials (ameliorating the problem of negative studies never being released). However, the fact that only 8% of the trials report statistically significant results (where p < 0.05 is often the standard for statistical significance) is concerning.

    The data journalism site FiveThirtyEight had a nice feature on a related topic last week addressing the issue of reproducibility and reliability in science. It has a great interactive on the practice of "p-hacking," and discusses many related issues.
     
    Last edited: Aug 28, 2015
  7. Aug 28, 2015 #6

    Bystander

    User Avatar
    Science Advisor
    Homework Helper
    Gold Member
    2016 Award

    As is chemistry, in my experience --- and geology --- and chemical oceanography. "A good literature search should find all the retractions, letters to editors, and other 'contrary' information.":DD:oldlaugh::DD:oldlaugh:
     
  8. Aug 30, 2015 #7

    jim mcnamara

    User Avatar

    Staff: Mentor

    Nobody seems to be aware of Ioannidis, J. P. A. (2005). "Why Most Published Research Findings Are False". PLoS Medicine 2 (8): e124. doi:10.1371/journal.pmed.0020124. PMC 1182327. PMID 16060722 or J Ellenberg's book 'How not to be wrong ...'. Or to cite the reasons given in those articles about why there is a problem with some published results.

    I am not saying these guys are definitive, but in my field, Biology, there definitely are published results that are not reproducible. The two references above attempt to explain.
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook




Similar Discussions: Reproducibility of previous research reports
Loading...