Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

A Separate tests vs Simultaneous tests

  1. Jun 19, 2017 #1
    Ok so lets say I have multiple hypothesis that I want to test. Is there an advantage to testing them separately compared to all at once?

    Here is an example. Say theres a medication. We want to see how it affects males, how it affects females, and if the effect of the medication differs across gender.

    Say I record baseline health and health after the treatment for each individual.

    I could do a paired t test for the females. Then a paired t test for the males. Then I can do a two sample t test on the differences between males and females.

    Or I can do them all at once using simultaneous inference.

    Is there any draw backs for one of them vs the other?
  2. jcsd
  3. Jun 19, 2017 #2


    Staff: Mentor

    If you do them separately then you need to correct for multiple comparisons, and the raw p-values will be wrong. If you do them together then they will automatically be corrected.
  4. Jun 19, 2017 #3
    Ah ok. So then the idea is to use multiple regression?
  5. Jun 19, 2017 #4


    Staff: Mentor

    I prefer it, but in the end it should come out pretty similar.
  6. Jun 20, 2017 #5


    User Avatar
    Science Advisor

    Hey FallenApple.

    If they are dependent then order should be based on knowledge of what's important to test first or later on.

    If they are independent then you can test them in any order.

    The choice for doing so requires a bit of knowledge about both statistics and domain level knowledge.
  7. Jun 22, 2017 #6
    So for example, say that in reality the difference of treatment differs across males but that it doesn't for females. And if we suspect this, then it would be wise to just do the 3rd test, that is looking a the differences of differences first. Because doing the other two would be a waste.
  8. Jun 22, 2017 #7


    User Avatar
    Science Advisor

    You need to take into account what's called domain knowledge.

    For example - let's say you know fact A and that leads you to think that you should do test H0 and HA because it's critical that you make sure fact A doesn't hold [or does]. Then you would do fact A.

    You think about the things you know and the order they are tested in and you construct your hypotheses in the order of the facts being important to least important.

    Usually though when you have multiple variables involved you do what is called a regression and you can look at specific estimates to answer things on multiple variables.

    Have you ever done regression along with correlation and co-variance?
  9. Jun 22, 2017 #8
    Yes I'm familiar with regression, though not an expert at it. From what I know, multiple regression allows you to adjust for many variables at once. Does that take care of the ordering issue? Because in regression outputs, the coefficient is interpretated as the others held constant.
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted