Comparing two datasets: methods and statistics

[FrankDrebon]

Hi all,

Just looking for some opinions on how to approach reporting the variability in some data I have acquired. I know Rutherford is quoted as saying “if your experiment needs statistics, you ought to have done a better experiment”, but unfortunately in biophysics we’re always at the mercy of the variability of living “things”!

Basically, I have a set of measurements ‘A’ and a set of measurements ‘B’. Both measure the same property of a system, but the state of the system is slightly different in the two sets. What I want to calculate is the change in this property between the two states of the system.

I could do this by taking the mean of set A and the mean of set B and dividing one by the other. However, this gives me the “change in the averages”. What I (think I) want is the “average of the changes”, so I divide each A value by each B value and take the average of those comparisons.

As an example, suppose I measured this property over and over with the system in state A and got the results 7, 8, 8, 7, 5. Then I measured it in state B and got 10, 7, 9, 9, 8. The average value in state A is 7.0, the average value in state B is 8.6. State B obviously has a larger "property" than state A.

To calculate the average change between the sets, I’d divide each result in ‘B’ by each result in ‘A’ (25 comparisons) and take the average, in this case 1.265. Simply dividing 8.6 by 7 gives 1.229.

I then wish to provide a standard error to calculate confidence intervals. However, this requires the use of the sample size N. There are 10 measurements in two lots of 5 samples, and 25 comparisons. I can't decide which to use as the sample size! Thoughts?

I appreciate that there is probably no definite answer here, but your opinions would be appreciated. Also, if you think my “averages of the changes” method is stupid then please say so. Perhaps I could just calculate the “change in the averages” and calculate an error based on the standard deviations of the individual data sets? The two sides of my brain have been arguing which is the best way to analyse this data for weeks, and they can’t come to a conclusion...!

 

[EnumaElish]

Have you considered using: http://en.wikipedia.org/wiki/Student's_t-test#Unequal_sample_sizes.2C_unequal_variance ?

 

[adriank]

There is no justification in comparing individual values in A to corresponding values in B if there is no particular connection between them; for instance, if the order of the values in A and B doesn't matter.

That is to say: Is there some connection between, say, the first value in A and the first value in B, that does not exist between the first value in A and the second value in B? If not, you need to treat all pairs (value in A, value in B) equally.

 

[EnumaElish]

Good point, the average ratio is similar to a matched-pairs test, which may or may not be justified. On a related note, is there any reason to prefer the ratio A/B to the difference A-B (either pairwise, or between the two averages)? For example, if you believe that each of A and B is Lognormal, then you could test Log(A/B) = Log A - Log B, which would be Normal, for being equal to zero. Is that the case with your data?