Comparing model fits for many populations - F statistic

[bamm7382]

Hey all,

I'm doing some research on protein dynamics which involves fitting models to data. Basically, we think that our system can be described by a complex, 5 parameter model rather than the generally accepted 2 parameter model. The data we're working with was acquired from experiments where we added a chemical species to a solution containing a receptor - the amount of binding was then quantified by fluorescence that occurs only upon binding. We ran very many experiments of this type since there is a lot of error involved. The experiments were ran in slightly different ways that involved changing some concentrations of chemical species.

For each experiment, we fit curves describing both models to the data by a least squares approach. Naively comparing the sum of squared errors between the two fits won't work since the more complex model will always fit better. So, we employed an F-test to compare the fits since the two models are nested. This produces good results that validate our hypothesis in most cases. However, we must run an F-test for each experiment and can only look at model comparison within the experiment itself.

What we want to do now is do perform a single, final comparison that takes all of the data (all experiments) into account, allowing us to say with what certainty we can choose the more complex model. I'm very clueless as to how to do this and I'm not sure that it can even be done at all! Biological systems are inherently different, even when they are identical on a genetic level. So, perhaps it is fundamentally flawed to ask about the relationship among independent samples.

Is there a way to compare F statistics for slightly dissimilar, unique experiments? If not, is there a way to consolidate these experiments before testing a hypothesis in order to look at overall behavior? Perhaps the best we can do is simply compare p-values for each experiment.

Thank you for any help - I'm sure there is much statistical intuition I can gain from working this out.

 

[mmwave]

Hello,

Thank you for sharing your research with us. Your approach of using an F-test to compare the fits of the 2-parameter and 5-parameter models seems appropriate, as the models are nested and the F-test is designed for such comparisons. However, as you mentioned, this approach only allows for model comparison within each experiment and does not take into account the overall behavior of the data.

To address this issue, one approach you could consider is performing a meta-analysis. This involves combining the results from multiple studies (in this case, experiments) to obtain an overall estimate of the effect size (i.e. the difference between the two models). This can be done using statistical software such as R or STATA, which have packages specifically designed for meta-analysis.

Another option is to use a mixed effects model, which takes into account the variation between experiments and allows for the comparison of the two models across all experiments. This approach may be more appropriate if the experiments were not completely independent, as it allows for the inclusion of random effects to account for potential correlations between experiments.

In terms of consolidating the experiments before testing a hypothesis, this could also be done using a mixed effects model or by using a hierarchical model, which allows for the inclusion of both fixed and random effects. This would allow you to compare the models while taking into account the differences between experiments.

I hope this helps and provides some insight into potential approaches for your research. Good luck with your study!