Testing multiple hypotheses

[Agent Smith]

Say we're considering multiple hypotheses: $H_0, H_1, H_2, ... H_ n$
$P(H_0) \leq 0.05$ (our P-value).

Is it possible that $P(H_1) < P(H_0), P(H_2) < P(H_0), ... P(H_n) < P(H_0)$

and yet $P(H_0) + [P(H_1) + \dots + P(H_n)] = 1$?

We would reject the null hypothesis but all the alternative hypotheses are less probable than the null.

 

[Agent Smith]

Say we're considering multiple hypotheses: $H_0, H_1, H_2, ... H_ n$
$P(H_0) \leq 0.05$ (our P-value).

Is it possible that $P(H_1) < P(H_0), P(H_2) < P(H_0), ... P(H_n) < P(H_0)$

and yet $P(H_0) + [P(H_1) + \dots + P(H_n)] = 1$?

We would reject the null hypothesis but all the alternative hypotheses are less probable than the null.

As an example $\frac{1}{2} + \left[\frac{1}{4} + \frac{1}{8} + ...\right] = 1$

 

[FactChecker]

Say we're considering multiple hypotheses: $H_0, H_1, H_2, ... H_ n$
$P(H_0) \leq 0.05$ (our P-value).

Is it possible that $P(H_1) < P(H_0), P(H_2) < P(H_0), ... P(H_n) < P(H_0)$

and yet $P(H_0) + [P(H_1) + \dots + P(H_n)] = 1$?

We would reject the null hypothesis but all the alternative hypotheses are less probable than the null.

This is an unusual statistical test setup. Usually, the probability parameters of the null hypothesis are initially assumed. Suppose that with the null hypothesis probability distribution, the sample is an unlikely event with a probability less than 0.05. Then the alternative hypotheses may be considered and their alternative probability parameters (mean, variance, etc.) are assumed. These are not the same probabilities that they had under the null hypothesis. With maximum likelihood estimated parameters the sample results are much more likely.

For instance, suppose that the null hypothesis is that the sample is from a normal distribution with a mean $\mu_0=0$ and variance $\sigma_0 = 1$. Now suppose that the sample (N=101) has a sample results of $\mu_s=0.4$ and $\sigma_s=1$. Then the null hypothesis mean of 0 is far (4 $\sigma$) outside the 95% confidence interval for that sample (the 95% confidence interval is [0.2, 0.6]). So now we could consider the alternative hypothesis, $H_1$, that the true population distribution has a mean of 0.4. That hypothesized mean would be much more compatible with the sample.

 

[Vanadium 50]

First, there exist tests that distinguish between exactly two hypotheses. (The F test is an example) You are not limited to tests of rejecting the null hypothesis. And yes, both hypotheses can be wrong.

Next, I would say if you calculate the p-value first and then set your threshold, you are not doing statistics. Not sure what I would call it, but this isn't statistics.

Finally, statistics are there to help you make a decision. I remember a test that caused us to reject the null hypothesis at 90 but not 95%. What we did was to make small interventions - less than we would do at 95% - but wouldn't be harmful if the null hypotheses were correct. Probably not a perfect outcome, but one supported by what data we had.

 

[Dale]

Say we're considering multiple hypotheses: $H_0, H_1, H_2, ... H_ n$
$P(H_0) \leq 0.05$ (our P-value).

Is it possible that $P(H_1) < P(H_0), P(H_2) < P(H_0), ... P(H_n) < P(H_0)$

and yet $P(H_0) + [P(H_1) + \dots + P(H_n)] = 1$?

We would reject the null hypothesis but all the alternative hypotheses are less probable than the null.

In frequentist statistics you never assign a probability to a hypothesis. You assign the probability to the data. So it would be $$P( data | H_0)$$ etc.

Bayesian statistics does assign probabilities to hypothesis. So you could indeed see that $$P(H_1|data)< P(H_0|data)<0.05$$ and consider the evidence to refute both hypotheses

 

[Agent Smith]

I think I'm talking about Bayesian statistics. The gist of what I want to say seems to be that every possible hypothesis is improbable at a threshold probability of 0.05.

 

[Dale]

every possible hypothesis is improbable at a threshold probability of 0.05

Certainly not every possible hypothesis. But all of the hypotheses in some specified set of hypotheses.

 

[Agent Smith]

In frequentist statistics you never assign a probability to a hypothesis.

Apologies for the late reply, was engaged.

So I learned about empirical probability many suns ago which is required in some fields like actuarial sciences. So desired probabilities are computed from (say) birth and death records. Is this an example of frequentist statistics? Can Bayesian statistics be useful here?

 

[Dale]

Apologies for the late reply, was engaged.

So I learned about empirical probability many suns ago which is required in some fields like actuarial sciences. So desired probabilities are computed from (say) birth and death records. Is this an example of frequentist statistics? Can Bayesian statistics be useful here?

An empirical probability is experimental evidence. It is the observed frequency in a given experiment. It can be useful in both frequentist statistics and in Bayesian statistics. It just depends if you want to determine $P(hypothesis|evidence)$ or $P(evidence|hypothesis)$

 

[Agent Smith]

@Dale

$\displaystyle \frac{2}{11} \sum_{k = 1} ^10 \frac{11 - k}{10} = 1$

Someone found for me a fraction viz. $\frac{2}{11}$ such that we take fractions < $\frac{2}{11}$ and add them all up we get $1$. Each of these fractions could be the probability of a hypothesis. Our best hypothesis only has a probability of $\frac{2}{11}$. I wonder what this would imply.

 

[Dale]

It implies that none of your hypotheses are very likely.

If in addition you know that your hypotheses are mutually exclusive, then it also means that we are certain one is right but uncertain which one. Note that the mutually exclusive assumption is an additional one that is not generally satisfied by hypotheses. As a result, it is possible to have a set of hypotheses whose probabilities sum to greater or less than 1.