Interpreting hypothesis test results

[barryj]

Lets say there is a drug that proposes to make one grow taller. A study is performed and the results state that there IS statistical difference with a Z = 3.1 p<.05. What should be stated about the magnitude of the increase in height? Should the study also have to state the magnitude of the height increase, or better yet the mean of the height before taking the drug and the mean after taking the drug?

As I study statistics, it seems that the statement, "Z = 3.1 p<.05." is a requirement for a scientific report but I don't see where the population and sample means have to be stated.

I hope this question makes sense.

 

[Stephen Tashi]

Two main divisions of statistics are 1) hypothesis testing and 2) estimation. If you are concerned about estimating the mean of the population then you are, of course, concerened with 2) estimation. Often hypothesis testing is applied to data and if the null hypothesis is "rejected" then estimation is applied to the same data. (Both hypothesis testing and estimation can involve computing the sample mean, so I don't why you said it is not computed.)

In the usual sort of application, estimation and hypothesis testing are not done as a single step because hypothesis testing assumes something about a parameter - even though the procedure may "reject" the assumption. Hypothesis testing computes something about the probability of the data given the assumption about the parameter. If we were to introduce the idea that the parameter is unknown and needs to be estimated then the probability computed by the hypothesis test would not be valid. Hypothesis testing does not compute the probability that the hypothesis is true ( and it does not compute the probability the hypothesis is false). It involves computing the probability of some aspect of the data given that the hypothesis is assume to be true with certainty - not that it assumed to be true with some probability less than 1.

Hypothesis testing is not a mathematical proof. It is simply a procedure that has been found to be useful empirically in many fields. There are many subjective aspects to applying statistics to real world problems; different scientific disciplines have different customs about how to do it.

 

[barryj]

I am trying to get my head around these statistic tests. After more reading, I see there is a Cohen D test that sort of gives information about the actual mean of the population and the sample. I guess my concern is that when a study is reported, that the estimation of the magnitude of the effect is not a requirement.

In my example above, if the mean of the height of the population and also the sample were disclosed, then I could assume that the drug might give me a 1 inch increase in height and most likely at least a 1/2 inch increase. However, expecting a 2 inch increase might be of question. I don't see this kind of information in the statement about z and p. Put another way, my reading says that the z and p scores MUST be reported but does not say anything about the effect itself.

I have tried to find good examples of statistical studies and reporting on the web but it seems like you have to purchase the reports. One would think that if results are good that the reports would be readily available. For example, is there a study showing the effect of gluecosimine (excuse the spelling) on joint pain. I hear rumors but have not seen a real scientific report.

 

[Stephen Tashi]

I am trying to get my head around these statistic tests. After more reading, I see there is a Cohen D test that sort of gives information about the actual mean of the population and the sample.

I think you are taking a common sense approach to interpreting statistical tests - and that won't work. You have to pay attention to technicalities. If you don't pay attention to technicalities, you will only get some over simplified concept of statistical tests, not the real thing. I think most posters in the mathematics section want to talk about the mathematically correct formulation of statistics.

Was the article you read about a Cohen D "test" or was the article about the Cohen D "statistic".? Is the difference between a "statistic" and a "test" clear in your mind? A "statistic" can be used as an "estimator". or it can be used to do a "hypothesis test" or it can merely be reported without drawing any conclusions from its value.

In my experience, the first thing a common sense person wants to know about data is "What does the sample data show about the population?". Since statistics deals with probability, it should be clear that statistics says nothing definite about the population. Any statement that statistics can make will involve some qualification about probability. A common sense person who accepts that limitation will next ask a question like "What is the probability that such-and-such is true about the population given the sample data we have?" The usual sort of statistics (which you are reading about) doesn't answer that question either. Statistics has terminology such as "significance level" and "confidence" that suggest that it can make statements of the form "With probability of such-and-such, so-and-so is true for the population". However, when you understand the technical meanings of the terminology, you'll see that no statements of that form are made.

 

[DrDu]

I have tried to find good examples of statistical studies and reporting on the web but it seems like you have to purchase the reports. One would think that if results are good that the reports would be readily available. For example, is there a study showing the effect of gluecosimine (excuse the spelling) on joint pain. I hear rumors but have not seen a real scientific report.

Did you try
https://clinicaltrials.gov/