The Significance Filter: How selective publishing biases results

In summary, the authors of "The Significance Filter" discuss the difficulties of publishing null results and the impact of selective publishing on research outcomes. They analyze over a million z-scores from Medline and find a strong bias towards significant results, leading to a distorted view of reality. The authors also address the issue of p-hacking and its effects on meta-analyses. The paper highlights the need for a more balanced and transparent approach to publishing research results.
  • #1
37,138
13,982
On arXiv: The Significance Filter, the Winner's Curse and the Need to Shrink

It's well known that it is difficult to publish null results in various fields. The authors try to quantify that by analyzing over a million z-scores from Medline, a database for medical publications. A very striking result is figure 1, red lines and numbers were added by me:

zscores.png


The infamous two-sided p<0.05 is at z=1.96. We see a really sharp edge, if the bin edge were at 1.96 instead of 2 it would look even more pronounced.
Should we expect a normal distribution? No. That would be the shape if nothing depends on anything ever. Things do depend on each other, so the tails are larger. It's perfectly fine to start a study where earlier results suggest you'll end up with a z-score around 2 for example. But we certainly should expect a smooth distribution - in a well-designed experiment the difference between z=1.9 and z=2.1 is purely random chance.

If 10 groups take measurements of something that's uncorrelated, one of them finds z>2 by chance and only that one gets published we get a strong bias in the results. p-hacking leads to a similar result as selective publishing. Things look important where the combined measurements would clearly show that there is nothing relevant. Even worse, meta-analyses will only take the published results, calculating an average that's completely disconnected from reality.

The authors quantify some problems that arise from selective publishing.
 
  • Like
Likes jim mcnamara, jrmichler, BWV and 8 others
Physics news on Phys.org
  • #2
mfb said:
Should we expect a normal distribution? No. That would be the shape if nothing depends on anything ever. Things do depend on each other, so the tails are larger.
Did not read the paper, but its presumably T-stats in the various studies examined? which can have a larger tail, that are then converted to Z-scores in the paper
 

Related to The Significance Filter: How selective publishing biases results

1. What is the Significance Filter?

The Significance Filter is a term used to describe the phenomenon of selective publishing biases in scientific research. It refers to the tendency for researchers and scientific journals to publish only studies with statistically significant results, leading to an overestimation of the true effects of a particular phenomenon.

2. How does the Significance Filter impact scientific research?

The Significance Filter can have a significant impact on scientific research as it can lead to misleading conclusions and an inflated view of the effectiveness of certain treatments or interventions. It can also result in a waste of resources and time as researchers may pursue avenues that are not actually effective.

3. What are some potential causes of the Significance Filter?

One potential cause of the Significance Filter is publication bias, where studies with significant results are more likely to be published than those with non-significant results. Other factors that can contribute to the Significance Filter include selective reporting of results, p-hacking (manipulating data to achieve significant results), and pressure to publish in prestigious journals.

4. How can researchers address the Significance Filter?

To address the Significance Filter, researchers can take steps such as pre-registering their study protocols to avoid selective reporting, conducting meta-analyses to assess the overall effect of a phenomenon, and being transparent about their data collection and analysis methods. Collaborative research efforts and open science practices can also help mitigate the impact of the Significance Filter.

5. What are some potential consequences of the Significance Filter?

The Significance Filter can have several consequences, including the spread of misinformation and the replication crisis in science, where many published studies cannot be replicated by other researchers. It can also lead to biased decision-making in fields such as medicine and policy-making, as well as a lack of trust in scientific research among the general public.

Similar threads

  • High Energy, Nuclear, Particle Physics
Replies
4
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
6
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
22
Views
3K
Replies
1
Views
1K
  • Biology and Medical
Replies
2
Views
2K
Replies
32
Views
745
  • Set Theory, Logic, Probability, Statistics
Replies
11
Views
1K
  • STEM Academic Advising
Replies
13
Views
2K
  • Beyond the Standard Models
Replies
1
Views
3K
  • Math Proof Training and Practice
2
Replies
67
Views
10K
Back
Top