Undergrad Why Are P-values Uniformly Distributed in Hypothesis Testing?

[alan2]

Help. I need an intuitive, non-mathematical explanation of why p-values from hypothesis testing are uniformly distributed. I was talking to a social scientist and got a blank stare. I couldn't come up with anything except the proof. Thanks.

 

[tnich]

Help. I need an intuitive, non-mathematical explanation of why p-values from hypothesis testing are uniformly distributed. I was talking to a social scientist and got a blank stare. I couldn't come up with anything except the proof. Thanks.

If you wanted to generate random numbers from whatever the null distribution is, you would first generate a uniform random number and then apply the inverse cdf of the null distribution to get a random value. That value would be a (one-sided) p-value.

 

[Stephen Tashi]

Help. I need an intuitive, non-mathematical explanation of why p-values from hypothesis testing are uniformly distributed.

That won't be true if the the test statistic has a discrete distribution.

If you wanted to generate random numbers from whatever the null distribution is, you would first generate a uniform random number and then apply the inverse cdf of the null distribution to get a random value. That value would be a (one-sided) p-value.

That value would be a value $t_0$ of the test statistic. The p-value corresponding to $t_0$ would be (for a left tail test) the cdf evaluated at $t_0$ so you get back the original random number that you chose from a uniform distribution.

It looks like we're ok for a left tail test from a continuous distribution. Are things really going to work out for other types of acceptance/rejection regions?