What Does p<.05 Mean in Statistics?

  • Context: Undergrad 
  • Thread starter Thread starter DaveC426913
  • Start date Start date
  • Tags Tags
    Basics Stats
Click For Summary

Discussion Overview

The discussion revolves around the interpretation of p-values in statistics, specifically focusing on the meaning of p<.05. Participants explore examples and analogies to clarify statistical significance, the null hypothesis, and the application of confidence intervals in various contexts.

Discussion Character

  • Exploratory
  • Technical explanation
  • Conceptual clarification
  • Debate/contested
  • Mathematical reasoning

Main Points Raised

  • One participant describes a scenario involving cars driving into Toronto and proposes that if 1 to 4 cars fly overhead, it indicates statistical significance, while 5 or more suggests a failure to reject the null hypothesis.
  • Another participant critiques the analogy, suggesting a binomial model and introducing the Agresti-Coull interval for better understanding of confidence levels.
  • A later reply discusses measuring the altitude of cars and how deviations from expected measurements relate to rejecting or failing to reject the null hypothesis.
  • Participants express uncertainty about the appropriateness of the car altitude analogy, questioning whether it is too concrete compared to more abstract statistical concepts.
  • One participant raises a question about the ideal circumstances for using p-values and whether they serve to support existing hypotheses.
  • Another participant corrects a confidence level misunderstanding, clarifying that a deviation of 4 cars would yield 85% confidence, not 95%.
  • There is a mention of the need for more data or better measuring tools when conclusions cannot be drawn at a given confidence level.
  • One participant expresses interest in pursuing further education in statistics due to the complexity and fascination with the topic.

Areas of Agreement / Disagreement

Participants do not reach a consensus on the effectiveness of the analogies used or the interpretation of statistical significance. Multiple competing views on the proper application of p-values and confidence intervals remain evident throughout the discussion.

Contextual Notes

Participants acknowledge limitations in their understanding of statistical concepts, including the proper interpretation of confidence levels and the implications of failing to reject the null hypothesis. There is also a recognition of the need for clearer examples to illustrate complex statistical ideas.

Who May Find This Useful

This discussion may be useful for individuals interested in statistics, particularly those seeking to understand the interpretation of p-values and the application of statistical significance in various contexts.

DaveC426913
Gold Member
2025 Award
Messages
24,469
Reaction score
8,727
I was doing tech support at a healthcare conference over the weekend in which one of the speakers talked about stats 101 and interpreting research. In discussing it further with my wife this morning, I realized we've missed a piece of the puzzle.

My wife made an spurious example of cars heading into town. 100 cars drive along the QEW into Toronto. This is expected. If between 1 and 4 of those cars flies overhead, this is statistically significant and is worth investigating - we form a hypothesis. But if 5 or more care fly overhead, this means that it is not statistically significant, and is more likely to be part of the null hypothesis.

I have definitely oversimplified our discussion but that's the gist of it. I think we've got cause and effect backwards in terms of expected and observed behaviour. And I think we may have used a poor analogy.
 
Physics news on Phys.org
DaveC426913 said:
My wife made an spurious example of cars heading into town. 100 cars drive along the QEW into Toronto. This is expected. If between 1 and 4 of those cars flies overhead, this is statistically significant and is worth investigating - we form a hypothesis. But if 5 or more care fly overhead, this means that it is not statistically significant, and is more likely to be part of the null hypothesis.

I have definitely oversimplified our discussion but that's the gist of it. I think we've got cause and effect backwards in terms of expected and observed behaviour. And I think we may have used a poor analogy.

Yes, this is a poor analogy. You describe a binomial model, in which case you should have used the Agresti-Coull interval, which would have told you (with 95% confidence) that between 0.9% and 11.5% of cars are flying, in the case where you saw 4 out of 100 cars fly.

A better analogy: you measure the altitude of cars heading into Toronto with expected measurement error of 1 ft (normally distributed, variance = 1 ft^2). If no cars fly, you expect 95% of the measurements to be less than 1.645 standard deviations, say, 1' 7". If 1 to 4 cars have measured altitude more than 1' 7", then you're fine; you can reasonably accept the conclusion that the cars aren't flying (technically, you reject the null hypothesis that they do fly). If 5 or more cars are measured above 1' 7" then you fail to reject the null hypothesis: for all you know, the cars coming into Toronto are flying.
 
CRGreathouse said:
Yes, this is a poor analogy. You describe a binomial model, in which case you should have used the Agresti-Coull interval, which would have told you (with 95% confidence) that between 0.9% and 11.5% of cars are flying, in the case where you saw 4 out of 100 cars fly.

A better analogy: you measure the altitude of cars heading into Toronto with expected measurement error of 1 ft (normally distributed, variance = 1 ft^2). If no cars fly, you expect 95% of the measurements to be less than 1.645 standard deviations, say, 1' 7". If 1 to 4 cars have measured altitude more than 1' 7", then you're fine; you can reasonably accept the conclusion that the cars aren't flying (technically, you reject the null hypothesis that they do fly). If 5 or more cars are measured above 1' 7" then you fail to reject the null hypothesis: for all you know, the cars coming into Toronto are flying.

OK so, the confidence level applies to my observations, my data. A deviation of 4 or fewer cars being at a higher elevation than expected (i.e. +/- 1 foot) shows no statistically significant deviation from what we expect from cars. If 5 or more cars showed an altitude of more than 1 foot, I would suspect there are factors at play.

The null hypothesis here is that cars do not fly. 5 cars showing apparently flying behaviour means I reject the null and may need a hypothesis that describes flying cars.

Is altitude of cars a good analogy? Seems to me that it's too concrete (a car either is or is not flying). Not the case with populations. I wonder if I should stick to something more medical or more political, where they often use p values. I dunno, people walking on a street? 95% are walking toward the subway?


Let me ask the question: what is the ideal circumstance in which this is used? What is the goal, that this is the tool? It's designed to lend credence to an existing hypothesis, right?

i.e. Let's say the null hypothesis is that people leaving their offices are headed south toward the subway. My data shows that more than 5% are actually headed north. I hypothesize that some people do not take the subway (they might be headed for the parking lot instead).

No, I've got it backwards again...shoot...
 
Last edited:
DaveC426913 said:
OK so, the confidence level applies to my observations, my data. A deviation of 4 or fewer cars being at a higher elevation than expected (i.e. +/- 1 foot) shows no statistically significant deviation from what we expect from cars. If 5 or more cars showed an altitude of more than 1 foot, I would suspect there are factors at play.

Two points.

First, that would give you 85% confidence, not 95%; 95% would be about 1' 7", not 1'.

Second, I wouldn't say that otherwise you 'suspect that there are factors at play' but rather that you can't conclude (at the given confidence level) that cars aren't flying. You may simply need more data, better measuring tools, etc.

DaveC426913 said:
Is altitude of cars a good analogy? Seems to me that it's too concrete (a car either is or is not flying).

Right. I gave the proper way to handle that situation in my first post, but the altitude modification was a way to partially salvage it. It's certainly not an ideal example.

I'd be more comfortable having someone else give an example; stats really aren't my thing.
 
CRGreathouse said:
Two points.

First, that would give you 85% confidence, not 95%; 95% would be about 1' 7", not 1'.

Oh. I didn't realize that value was not arbitrary. That's where the 1ft^2 came from. That is another aspect I don't know.


I am seriously considering taking a night school course in stats. I find it fascinating. More to the point, I find it more interesting than my current level of understanding permits.