Student t test with small sample number

  • I
  • Thread starter Matheus del Valle
  • Start date
  • Tags
    Student Test
In summary, the researcher found that there is no significant difference between two methods, but they are clearly different. However, they are using a different statistic test to look for significance.
  • #1
Matheus del Valle
2
0
Hello,

I'm checking the similarity of two methods for my research (a gold-standard method and another one which I need to check if it’s eficiente compared to the gold-standard) with student t test.

I have the following datas (N=3):
method 1 (gold-standard): 120, 347, 116;
method 2: 2603, 5203, 25011;

The result of one-tailed, independent samples student t test is p=0.11, which is bigger than 0.05.
So the test says that there's no significance differece between the two methods, but they are clearly different.
The t test is giving me a false result due to the small N number? Should I use another statistic test? Thanks.
 
Physics news on Phys.org
  • #2
The problem is that if your second method data IS normally distributed, then it's clear your variance will be huge. This is a problem if you want to obtain significance.

It is kind of doubtful that your second method is normally distributed too, you have too little data points to check this anyway.

So either you continue to believe that your data points come from a normal distribution, in which case you'll need a hell of a lot more data points. Or you use a nonparametric test.
 
  • Like
Likes FactChecker
  • #3
You should get more data if you can. Three values from each is a very small sample, no matter how obvious the differences look. The huge variation of the second set weaken the statistical results. I tried using a non-parametric test (Wilcoxon Rank Sum) and it was not significant. Out of curiosity, I adding one made-up typical data point to each set and the results became significant.

In general there are real concerns with shopping around for a test that makes your results look significant. A lot of insignificant results will look significant in some way if you examine them from every possible aspect
 
  • #4
I agree with the comments above, but in addition if your goal is to demonstrate equivalence between the alternative test and the gold standard test then this is the wrong method.

The first step would be to do a Bland Altman plot. This is just a graphical method, but it is very commonly used in this type of research.

The next thing that you want to do is to decide on a region of practical equivalence. For example, the first gold standard test was 120, if another test gave 121 would you consider that to be practically equivalent? How about 130, or 150, or 200?

Once you have chosen a region of practical equivalence, then you take your data and construct a 95% confidence interval. If it lies entirely within the region of practical equivalence then you have good evidence of equivalence. Otherwise you do not have good evidence.
 
  • #5
Thank you all for the help. I managed to improve my samples and now I'm analysing based on your tips.

I'm also using the ICC (intraclass correlation coefficient) to compare two different methods and it seems to be pretty satisfactory.
 
  • #6
The ICC isn't really appropriate here. The regular correlation is more appropriate, with the gold standard as the independent variable and the new method as the dependent variable. However, correlation is not a good measure for this.

You should read Bland and Altman's highly influential paper on this subject. A paper where you don't at least provide a Bland Altman plot will likely be rejected in peer review in any decent journal.
 

Related to Student t test with small sample number

1. What is the Student t test and when is it used?

The Student t test is a statistical test used to compare the means of two groups. It is used when the sample size is small and the population standard deviation is unknown.

2. How is the t-statistic calculated in a Student t test?

The t-statistic is calculated by taking the difference between the means of the two groups and dividing it by the standard error of the difference between the means.

3. What does a small sample number mean in the context of a Student t test?

A small sample number refers to a sample size that is less than 30. This is considered small because it may not accurately represent the larger population.

4. What is the significance level in a Student t test and how is it determined?

The significance level, also known as alpha, is the probability of rejecting the null hypothesis when it is actually true. It is typically set at 0.05, meaning there is a 5% chance of making a Type I error. However, it can be adjusted based on the specific research question and desired level of confidence.

5. What are the assumptions for performing a Student t test with a small sample number?

The main assumptions for a Student t test with a small sample number are that the data is normally distributed, the samples are independent, and the variances of the two groups are equal. If these assumptions are not met, alternative tests may need to be used.

Similar threads

  • Set Theory, Logic, Probability, Statistics
Replies
1
Views
898
  • Set Theory, Logic, Probability, Statistics
Replies
27
Views
3K
  • Set Theory, Logic, Probability, Statistics
Replies
9
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
20
Views
3K
  • Calculus and Beyond Homework Help
Replies
2
Views
696
  • Set Theory, Logic, Probability, Statistics
Replies
7
Views
4K
  • Set Theory, Logic, Probability, Statistics
Replies
9
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
3
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
23
Views
3K
Replies
14
Views
2K
Back
Top