What Does p<.05 Mean in Statistics?

[DaveC426913]

I was doing tech support at a healthcare conference over the weekend in which one of the speakers talked about stats 101 and interpreting research. In discussing it further with my wife this morning, I realized we've missed a piece of the puzzle.

My wife made an spurious example of cars heading into town. 100 cars drive along the QEW into Toronto. This is expected. If between 1 and 4 of those cars flies overhead, this is statistically significant and is worth investigating - we form a hypothesis. But if 5 or more care fly overhead, this means that it is not statistically significant, and is more likely to be part of the null hypothesis.

I have definitely oversimplified our discussion but that's the gist of it. I think we've got cause and effect backwards in terms of expected and observed behaviour. And I think we may have used a poor analogy.

 

[CRGreathouse]

My wife made an spurious example of cars heading into town. 100 cars drive along the QEW into Toronto. This is expected. If between 1 and 4 of those cars flies overhead, this is statistically significant and is worth investigating - we form a hypothesis. But if 5 or more care fly overhead, this means that it is not statistically significant, and is more likely to be part of the null hypothesis.

I have definitely oversimplified our discussion but that's the gist of it. I think we've got cause and effect backwards in terms of expected and observed behaviour. And I think we may have used a poor analogy.

Yes, this is a poor analogy. You describe a binomial model, in which case you should have used the Agresti-Coull interval, which would have told you (with 95% confidence) that between 0.9% and 11.5% of cars are flying, in the case where you saw 4 out of 100 cars fly.

A better analogy: you measure the altitude of cars heading into Toronto with expected measurement error of 1 ft (normally distributed, variance = 1 ft^2). If no cars fly, you expect 95% of the measurements to be less than 1.645 standard deviations, say, 1' 7". If 1 to 4 cars have measured altitude more than 1' 7", then you're fine; you can reasonably accept the conclusion that the cars aren't flying (technically, you reject the null hypothesis that they do fly). If 5 or more cars are measured above 1' 7" then you fail to reject the null hypothesis: for all you know, the cars coming into Toronto are flying.

 

[DaveC426913]

Yes, this is a poor analogy. You describe a binomial model, in which case you should have used the Agresti-Coull interval, which would have told you (with 95% confidence) that between 0.9% and 11.5% of cars are flying, in the case where you saw 4 out of 100 cars fly.

A better analogy: you measure the altitude of cars heading into Toronto with expected measurement error of 1 ft (normally distributed, variance = 1 ft^2). If no cars fly, you expect 95% of the measurements to be less than 1.645 standard deviations, say, 1' 7". If 1 to 4 cars have measured altitude more than 1' 7", then you're fine; you can reasonably accept the conclusion that the cars aren't flying (technically, you reject the null hypothesis that they do fly). If 5 or more cars are measured above 1' 7" then you fail to reject the null hypothesis: for all you know, the cars coming into Toronto are flying.

OK so, the confidence level applies to my observations, my data. A deviation of 4 or fewer cars being at a higher elevation than expected (i.e. +/- 1 foot) shows no statistically significant deviation from what we expect from cars. If 5 or more cars showed an altitude of more than 1 foot, I would suspect there are factors at play.

The null hypothesis here is that cars do not fly. 5 cars showing apparently flying behaviour means I reject the null and may need a hypothesis that describes flying cars.

Is altitude of cars a good analogy? Seems to me that it's too concrete (a car either is or is not flying). Not the case with populations. I wonder if I should stick to something more medical or more political, where they often use p values. I dunno, people walking on a street? 95% are walking toward the subway?


Let me ask the question: what is the ideal circumstance in which this is used? What is the goal, that this is the tool? It's designed to lend credence to an existing hypothesis, right?

i.e. Let's say the null hypothesis is that people leaving their offices are headed south toward the subway. My data shows that more than 5% are actually headed north. I hypothesize that some people do not take the subway (they might be headed for the parking lot instead).

No, I've got it backwards again...shoot...

 

[CRGreathouse]

OK so, the confidence level applies to my observations, my data. A deviation of 4 or fewer cars being at a higher elevation than expected (i.e. +/- 1 foot) shows no statistically significant deviation from what we expect from cars. If 5 or more cars showed an altitude of more than 1 foot, I would suspect there are factors at play.

Two points.

First, that would give you 85% confidence, not 95%; 95% would be about 1' 7", not 1'.

Second, I wouldn't say that otherwise you 'suspect that there are factors at play' but rather that you can't conclude (at the given confidence level) that cars aren't flying. You may simply need more data, better measuring tools, etc.

Is altitude of cars a good analogy? Seems to me that it's too concrete (a car either is or is not flying).

Right. I gave the proper way to handle that situation in my first post, but the altitude modification was a way to partially salvage it. It's certainly not an ideal example.

I'd be more comfortable having someone else give an example; stats really aren't my thing.

 

[DaveC426913]

Two points.

First, that would give you 85% confidence, not 95%; 95% would be about 1' 7", not 1'.

Oh. I didn't realize that value was not arbitrary. That's where the 1ft^2 came from. That is another aspect I don't know.


I am seriously considering taking a night school course in stats. I find it fascinating. More to the point, I find it more interesting than my current level of understanding permits.