Comparing theoretical calculations with experimental data

[parazit]

TL;DR Summary

How to compare theoretical calculations results obtained with six models with one exact experimental data?

Dear users,

The situation I have encountered is a simple statistical comparison of the experimental data, which accepted as correct, with the results obtained via six theoretical models.

In the experimental data, there exist y values corresponding to x values and also the measurement errors of y values. In theoretical values, there exist y values corresponding to the same x values as the experimental data. Theoretical values are calculated with six different models. I've uploaded an excel file as an example.

I would like to have a simple statistical comparison to understand which theoretical calculation is more consistent with the experimental data but not sure which method to use. I've calculated mean weighted deviation and RMSE yet outputs have pointed different models as the most consistent one. I do not need advanced model comparison methods. Any simple and logical method works for me. Thanks in advance for your comments and time.

 

[Dale]

There are three approaches that I am aware of. The first is a straight Bayesian model comparison. Unfortunately, this requires quite a bit of care in selecting good priors for all of your model parameters. From your description of your goals it is probably “overkill”.

The next two are closely related. You can calculate either the Bayesian Information Criterion or the Akaike Information Criterion. I prefer the BIC over the AIC since it penalizes model complexity more, which I think is important. Many statistical packages implement both the BIC and the AIC

 

[Stephen Tashi]

Summary: How to compare theoretical calculations results obtained with six models with one exact experimental data?

Do the models make deterministic predictions? - or are they probabilistic models that predict a mean value of Y with some variation about it?

I would like to have a simple statistical comparison to understand which theoretical calculation is more consistent with the experimental data

Any simple and logical method works for me.

The common language meanings of "more consistent" and "simple and logical" aren't specific enough to describe particular statistical methods.

If you can't be more specific with technicalities, I suggest that you describe your bottom line goal. If some statistical method ranks models in some way, what effect will that have? Are you doing statistics for your own personal use or are the conclusions to be publicized? Will some important decision be influenced - like how to build a chemical factory or how to make an investment? Or is the intended effect less specific - for example, to argue for or against a particular theory of economics?

 

[parazit]

There are three approaches that I am aware of. The first is a straight Bayesian model comparison. Unfortunately, this requires quite a bit of care in selecting good priors for all of your model parameters. From your description of your goals it is probably “overkill”.

The next two are closely related. You can calculate either the Bayesian Information Criterion or the Akaike Information Criterion. I prefer the BIC over the AIC since it penalizes model complexity more, which I think is important. Many statistical packages implement both the BIC and the AIC

Thank you for spending time to respond. I do not know about the methods you've mentioned, but I'm going to try study and understand them.

 

[parazit]

First of all, please forgive my English. Since I am not so good at it, I may not be able to express myself as I intend.

Do the models make deterministic predictions? - or are they probabilistic models that predict a mean value of Y with some variation about it?

Models perform mathematical operations using different parameters and variables where x values are common to all models. y values are the results of mathematical operations. All models have similar approaches in the mathematical operations but include different parameters that result different outcomes.

If some statistical method ranks models in some way, what effect will that have? Are you doing statistics for your own personal use or are the conclusions to be publicized? Will some important decision be influenced - like how to build a chemical factory or how to make an investment? Or is the intended effect less specific - for example, to argue for or against a particular theory of economics?

The purpose of the comparison of models is only to try to understand which model is more compatible. This is just for my own personal interest. The examples you've mentioned are very extreme situations to me, and my intention has nothing to do with them. I'm just trying to figure out how to compare the results in an acceptable and intelligent way.

Please forgive me for taking your time. I know that no one has an obligation to lecture me or to give explanations to me in order to compensate my academic deficiency, but I would be very grateful if you could guide me simply.

Best regards.

 

[pbuk]

Plot your experimental data and the 'models'. I think you will see that the variation in the experimental data is too great to sensibly fit anything. Having said that, all of the models seem to underestimate results in the range x = 15 to 20: can you see that on the plot?