Hypothesis Tests: Understanding p-value, alpha level & more

  • Thread starter muktl
  • Start date
In summary, Bayesian statistics is a different way of thinking about statistics that is used for certain situations, such as when trying to find new drugs. The standard (frequentist) approach is to assume a population based distribution of the parameter, usually a Gaussian (normal) distribution, and to set a high significance level (low probability) that the null hypothesis is true. If the level is met or exceeded (a probability of less then 0.025), we reject the null hypothesis.
  • #1
muktl
6
0
Hi, i am just really confused about the hypothesis test part in my course.
all i know is i use the mean of the hypothesis and find the test point, but i don't understand when i need to reject/ not reject the hypothesis.
and what really is p-value ( significance level), how to use that to determine whether to reject or not.
and the alpha level ( % of making type I error)

Thanks.
 
Physics news on Phys.org
  • #2
One way to look at the typical statistics course is that it doesn't make logical sense!

The situation is this:

The type of question that a practical person wants to know is "Given the data what is the probability that statement H is true? ".

There is not enough given information to answer this question in the problems presented in a typical statistics course. So, instead, a different question is answered, namely: "Given that statement H is true, what is the probability of the data?".

If the probability of the data is "small" then you are supposed to "reject" the statement H, but this is a completely subjective decision. There is no objective way to determine how small "small" should be. The process of "accepting or rejecting the null hypothesis" is simply an arbitrary procedure. There is no proof offered that it is the only correct one to follow.

You will find that the terms used in statistics ("level of significance", "rejection region", "confidence level", "confidence limits") have been cleverly chosen. They make the methods of statistics sound very objective. They confuse most people into thinking that they are getting the answer to "Given the Data, what is the probability that my idea is true?". However, underneath the hood, the methods are subjective and the numbers you compute don't answer this question.

If you want to study the type of statistics that does compute "The probability that statement H is true given the data", you'll have to study Bayesian statistics. If you want to study how to objectively set p-values, you should study a book like "Optimal Statistical Decisions" by Morris DeGroot.

In some situations, it may be possible to judge empirically how p-values are working. For example, if you publish a medical journal and you require that studies show a p-value is .05 then you'll have a smaller pile of papers to consider than if you set your p-value to .10. Or suppose you run a lab that screens thousands of substances to pick out ones that hold promise as drugs. Suppose you set a p-value of .10 and find hardly any substances to send-on for further testing. Other companies discover drugs based on substances that you have rejected. Your boss complains. One obvious course of action is increase your p-value to .15.
 
  • #3
In a typical experimental situation we compare a treatment group with a control group in terms of some experimental parameter (such as the mean). Under the null hypothesis we assume the treatment will have no effect, so there should be no significant difference in the parameter value between the two groups. The standard (frequentist) approach is to assume some population based distribution of the parameter, usually a Gaussian (normal) distribution. This has been shown to work pretty well. After the experiment, we set a high significance level (low probability) that the null hypothesis is true, say a probability of 0.025 because we want to have confidence that a real effect exists. If the level is met or exceeded (a probability of less then 0.025) we reject the null hypothesis. This is called the alpha error because that is the probability that we wrongly rejected the null hypothesis in favor of the alternative hypothesis, which is that the treatment had an effect, and the findings were not just random variation.

The Bayesian approach is different in that it states a prior probability of an outcome before the experiment and then revises that probability based on the new data. It has a lot of applications, but I think one should master frequentist methods before studying Bayesian methods. They are not necessarily incompatible but the method I described above is what' s generally required if you are trying to publish experimental results or submitting data to a government agency.
 
Last edited:
  • #4
To clarify something I said in the previous post. When I said that we want to set a high significance level, I was referring to the alpha error which is the probability of rejecting the null hypothesis when it is true. This probability should be low.
 
  • #5


Hello,

I understand that hypothesis testing can be confusing, but it is an important tool in scientific research. Let me try to clarify some of your questions.

First, let's start with the basics. A hypothesis is a statement that you want to test, typically about the relationship between two variables. In order to test this hypothesis, you need to collect data and analyze it using statistical methods.

The p-value is the probability of obtaining results at least as extreme as the ones observed, assuming that the null hypothesis is true. In simpler terms, it tells you how likely it is that the results you obtained are due to chance. A p-value of 0.05 (or 5%) is commonly used as a threshold for statistical significance. This means that if the p-value is less than 0.05, the results are considered statistically significant, and you can reject the null hypothesis. On the other hand, if the p-value is greater than 0.05, the results are not statistically significant, and you cannot reject the null hypothesis.

The alpha level, also known as the significance level, is the threshold that you set before conducting the hypothesis test. It is the maximum acceptable probability of making a type I error, which is the probability of rejecting the null hypothesis when it is actually true. The most common alpha level is 0.05, but it can be adjusted depending on the specific research question and field of study.

To summarize, the p-value and alpha level work together to determine whether the results are statistically significant and whether the null hypothesis can be rejected. It is important to note that statistical significance does not always equal practical significance. Therefore, it is essential to also consider the effect size and the context of the research when interpreting the results.

I hope this helps to clarify some of your confusion about hypothesis testing. If you have any further questions, please do not hesitate to reach out. Best of luck with your course!
 

1. What is a hypothesis test?

A hypothesis test is a statistical method used to determine the validity of a hypothesis or claim about a population. It involves collecting data and using statistical techniques to determine whether the evidence supports or rejects the hypothesis.

2. What is a p-value?

A p-value is a measure of the strength of evidence against the null hypothesis. It represents the probability of obtaining the observed results or more extreme results if the null hypothesis is true. A lower p-value indicates stronger evidence against the null hypothesis.

3. What is the alpha level?

The alpha level, also known as the significance level, is the predetermined threshold used to determine whether the p-value is considered statistically significant. It is typically set at 0.05, meaning that if the p-value is less than 0.05, the results are considered statistically significant and the null hypothesis can be rejected.

4. What is a Type I error?

A Type I error, also known as a false positive, occurs when the null hypothesis is rejected when it is actually true. This means that the results of the hypothesis test indicate a significant difference or relationship when there is none in the population.

5. How do I interpret the results of a hypothesis test?

The results of a hypothesis test should be interpreted in conjunction with the p-value and alpha level. If the p-value is less than the alpha level, the results are considered statistically significant and the null hypothesis is rejected. If the p-value is greater than the alpha level, the results are not considered statistically significant and the null hypothesis cannot be rejected.

Similar threads

  • Set Theory, Logic, Probability, Statistics
Replies
22
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
5
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
8
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
7
Views
335
  • Set Theory, Logic, Probability, Statistics
Replies
5
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
20
Views
3K
  • Set Theory, Logic, Probability, Statistics
Replies
9
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
26
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
6
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
21
Views
3K
Back
Top