Student t test with small sample number

[Matheus del Valle]

Hello,

I'm checking the similarity of two methods for my research (a gold-standard method and another one which I need to check if it’s eficiente compared to the gold-standard) with student t test.

I have the following datas (N=3):
method 1 (gold-standard): 120, 347, 116;
method 2: 2603, 5203, 25011;

The result of one-tailed, independent samples student t test is p=0.11, which is bigger than 0.05.
So the test says that there's no significance differece between the two methods, but they are clearly different.
The t test is giving me a false result due to the small N number? Should I use another statistic test? Thanks.

 

[micromass]

The problem is that if your second method data IS normally distributed, then it's clear your variance will be huge. This is a problem if you want to obtain significance.

It is kind of doubtful that your second method is normally distributed too, you have too little data points to check this anyway.

So either you continue to believe that your data points come from a normal distribution, in which case you'll need a hell of a lot more data points. Or you use a nonparametric test.

 

[FactChecker]

You should get more data if you can. Three values from each is a very small sample, no matter how obvious the differences look. The huge variation of the second set weaken the statistical results. I tried using a non-parametric test (Wilcoxon Rank Sum) and it was not significant. Out of curiosity, I adding one made-up typical data point to each set and the results became significant.

In general there are real concerns with shopping around for a test that makes your results look significant. A lot of insignificant results will look significant in some way if you examine them from every possible aspect

 

[Dale]

I agree with the comments above, but in addition if your goal is to demonstrate equivalence between the alternative test and the gold standard test then this is the wrong method.

The first step would be to do a Bland Altman plot. This is just a graphical method, but it is very commonly used in this type of research.

The next thing that you want to do is to decide on a region of practical equivalence. For example, the first gold standard test was 120, if another test gave 121 would you consider that to be practically equivalent? How about 130, or 150, or 200?

Once you have chosen a region of practical equivalence, then you take your data and construct a 95% confidence interval. If it lies entirely within the region of practical equivalence then you have good evidence of equivalence. Otherwise you do not have good evidence.

 

[Matheus del Valle]

Thank you all for the help. I managed to improve my samples and now I'm analysing based on your tips.

I'm also using the ICC (intraclass correlation coefficient) to compare two different methods and it seems to be pretty satisfactory.

 

[Dale]

The ICC isn't really appropriate here. The regular correlation is more appropriate, with the gold standard as the independent variable and the new method as the dependent variable. However, correlation is not a good measure for this.

You should read Bland and Altman's highly influential paper on this subject. A paper where you don't at least provide a Bland Altman plot will likely be rejected in peer review in any decent journal.