Separate tests vs Simultaneous tests

[FallenApple]

Ok so let's say I have multiple hypothesis that I want to test. Is there an advantage to testing them separately compared to all at once?

Here is an example. Say there's a medication. We want to see how it affects males, how it affects females, and if the effect of the medication differs across gender.

Say I record baseline health and health after the treatment for each individual.

I could do a paired t test for the females. Then a paired t test for the males. Then I can do a two sample t test on the differences between males and females.

Or I can do them all at once using simultaneous inference.

Is there any draw backs for one of them vs the other?

 

[Dale]

If you do them separately then you need to correct for multiple comparisons, and the raw p-values will be wrong. If you do them together then they will automatically be corrected.

 

[FallenApple]

If you do them separately then you need to correct for multiple comparisons, and the raw p-values will be wrong. If you do them together then they will automatically be corrected.

Ah ok. So then the idea is to use multiple regression?

 

[Dale]

I prefer it, but in the end it should come out pretty similar.

 

[chiro]

Hey FallenApple.

If they are dependent then order should be based on knowledge of what's important to test first or later on.

If they are independent then you can test them in any order.

The choice for doing so requires a bit of knowledge about both statistics and domain level knowledge.

 

[FallenApple]

Hey FallenApple.

If they are dependent then order should be based on knowledge of what's important to test first or later on.

If they are independent then you can test them in any order.

The choice for doing so requires a bit of knowledge about both statistics and domain level knowledge.

So for example, say that in reality the difference of treatment differs across males but that it doesn't for females. And if we suspect this, then it would be wise to just do the 3rd test, that is looking a the differences of differences first. Because doing the other two would be a waste.

 

[chiro]

You need to take into account what's called domain knowledge.

For example - let's say you know fact A and that leads you to think that you should do test H0 and HA because it's critical that you make sure fact A doesn't hold [or does]. Then you would do fact A.

You think about the things you know and the order they are tested in and you construct your hypotheses in the order of the facts being important to least important.

Usually though when you have multiple variables involved you do what is called a regression and you can look at specific estimates to answer things on multiple variables.

Have you ever done regression along with correlation and co-variance?

 

[FallenApple]

You need to take into account what's called domain knowledge.

For example - let's say you know fact A and that leads you to think that you should do test H0 and HA because it's critical that you make sure fact A doesn't hold [or does]. Then you would do fact A.

You think about the things you know and the order they are tested in and you construct your hypotheses in the order of the facts being important to least important.

Usually though when you have multiple variables involved you do what is called a regression and you can look at specific estimates to answer things on multiple variables.

Have you ever done regression along with correlation and co-variance?

Yes I'm familiar with regression, though not an expert at it. From what I know, multiple regression allows you to adjust for many variables at once. Does that take care of the ordering issue? Because in regression outputs, the coefficient is interpretated as the others held constant.