Separate tests vs Simultaneous tests

  • #1
566
60
Ok so lets say I have multiple hypothesis that I want to test. Is there an advantage to testing them separately compared to all at once?

Here is an example. Say theres a medication. We want to see how it affects males, how it affects females, and if the effect of the medication differs across gender.

Say I record baseline health and health after the treatment for each individual.

I could do a paired t test for the females. Then a paired t test for the males. Then I can do a two sample t test on the differences between males and females.

Or I can do them all at once using simultaneous inference.

Is there any draw backs for one of them vs the other?
 

Answers and Replies

  • #2
Dale
Mentor
Insights Author
2021 Award
32,414
9,404
If you do them separately then you need to correct for multiple comparisons, and the raw p-values will be wrong. If you do them together then they will automatically be corrected.
 
  • Like
Likes FallenApple
  • #3
566
60
If you do them separately then you need to correct for multiple comparisons, and the raw p-values will be wrong. If you do them together then they will automatically be corrected.

Ah ok. So then the idea is to use multiple regression?
 
  • #4
Dale
Mentor
Insights Author
2021 Award
32,414
9,404
I prefer it, but in the end it should come out pretty similar.
 
  • #5
chiro
Science Advisor
4,797
133
Hey FallenApple.

If they are dependent then order should be based on knowledge of what's important to test first or later on.

If they are independent then you can test them in any order.

The choice for doing so requires a bit of knowledge about both statistics and domain level knowledge.
 
  • Like
Likes FallenApple
  • #6
566
60
Hey FallenApple.

If they are dependent then order should be based on knowledge of what's important to test first or later on.

If they are independent then you can test them in any order.

The choice for doing so requires a bit of knowledge about both statistics and domain level knowledge.

So for example, say that in reality the difference of treatment differs across males but that it doesn't for females. And if we suspect this, then it would be wise to just do the 3rd test, that is looking a the differences of differences first. Because doing the other two would be a waste.
 
  • #7
chiro
Science Advisor
4,797
133
You need to take into account what's called domain knowledge.

For example - let's say you know fact A and that leads you to think that you should do test H0 and HA because it's critical that you make sure fact A doesn't hold [or does]. Then you would do fact A.

You think about the things you know and the order they are tested in and you construct your hypotheses in the order of the facts being important to least important.

Usually though when you have multiple variables involved you do what is called a regression and you can look at specific estimates to answer things on multiple variables.

Have you ever done regression along with correlation and co-variance?
 
  • #8
566
60
You need to take into account what's called domain knowledge.

For example - let's say you know fact A and that leads you to think that you should do test H0 and HA because it's critical that you make sure fact A doesn't hold [or does]. Then you would do fact A.

You think about the things you know and the order they are tested in and you construct your hypotheses in the order of the facts being important to least important.

Usually though when you have multiple variables involved you do what is called a regression and you can look at specific estimates to answer things on multiple variables.

Have you ever done regression along with correlation and co-variance?
Yes I'm familiar with regression, though not an expert at it. From what I know, multiple regression allows you to adjust for many variables at once. Does that take care of the ordering issue? Because in regression outputs, the coefficient is interpretated as the others held constant.
 

Related Threads on Separate tests vs Simultaneous tests

  • Last Post
Replies
3
Views
2K
  • Last Post
Replies
1
Views
2K
  • Last Post
Replies
2
Views
2K
  • Last Post
Replies
2
Views
2K
  • Last Post
Replies
19
Views
2K
  • Last Post
Replies
4
Views
781
  • Last Post
3
Replies
50
Views
9K
  • Last Post
Replies
1
Views
11K
Replies
1
Views
1K
Top