Understanding the Role of P-Values in Hypothesis Testing

[sodium.dioxid]

I don't understand why a lower p value is stronger evidence against the null hypothesis. P value is a probability; so, wouldn't a lower p value mean that your statistic was very lucky (rare)?

 

[aleph_0]

Well yes, the p-value is a probability; but a probability of what? Without answering that, you'll be lost. The p-value of a statistic is the probability that the null hypothesis is true when you reject it. Focus on the fact that it is related to the probability that the null hypothesis is true, so that when p gets small, the probability of the null hypothesis, in a sense, is lowered.

 

[aleph_0]

Also, yes, a low p-value could mean that your observation was lucky or rare assuming that the null hypothesis is true. However, when your p-value is very small, you tend not to believe that the answer is that the observation was lucky or rare. Because that small p-value is based on the assumption that the null hypothesis was true, if the p-value is very small then you start to suspect that it's not because you were lucky, but because your assumption was wrong: The null hypothesis was false.

It's analogous to an argument by contradiction. Those go: "Assume X is true. But if X is true then we can derive a contradiction. Therefore X must not be true."

The p-value argument is similar: "Assume the null hypothesis. But if the null hypothesis is true, then something extremely unlikely occurs. Therefore the null hypothesis is extremely unlikely and so we reject it."

 

[sodium.dioxid]

Thanks for the reply. I am still a bit confused. Can we work through an example? Suppose that a car manufacturer claims its cars to run at 30mpg.

Null: u = 30

I take a random sample of 6 cars and find the mpg to be 21mpg; pretend this correlates to a very low p value. The low p value means that we did not expect for it to be such a low mpg, right? Since it was so unlikely for it to be 21mpg and it came out to be 21 mpg nevertheless, there must be something wrong with the null hypothesis. <--- Is my analysis correct?

 

[Stephen Tashi]

there must be something wrong with the null hypothesis. <--- Is my analysis correct?

You have grasped the basic intuitive concept. The technical truth is that you can't deduce that "there must be something wrong with the null hypothesis". You can't say the null hypothesis is probably wrong and you can't state a number that quantifies the probability that the null hypothesis is correct. The statistical method of hypothesis testing is a subjective procedure. It's a procedure, not a proof of something.