How to analyse the difference between physical and calculated data?

[nixi]

Hi I am not sure what method to use when analysing the difference in values between physical measured data and modeled data.

This is to do with measuring the energy deposited at a point in water from a beam of photons.

I have a set of physical measurements obtained whilst varying the position of the measurement point in the water and the parameters of the photon beam. The water and its measurement positions have then been created virtually and the beam parameters have been modeled with various algorithms to produce a similar set of measurements to the physical data.

I assume the data is in independent pairs across the physical + modeled data that I want to analyse the overall difference of, rather than the physical and modeled being sample sets of a parent population. Therefore I have then calculated the difference between each physical and modeled data for the same parameter, but I am getting confused as to how to analyse all the differences together to give me a measure of how well each algorithm replicates the physical data. (the next step will be intercomparisons of the algorithms to see if it is safe to swap to a new one).

I'm not sure that it is a case of just taking the arithemtic mean as some values are positive and some are negative so the mean will surely be skewed? Should I be calculating an absolute mean difference and possibly also a relative mean difference to quantify both the average deviation and the direction of the deviation (over/underestimate)? Until today I have never heard of these averages so am not sure if this would be their intended useage?

How do I then report the error on the appropriate mean to be used in my case? Should I be using an absolute deviation (again new to me, but this seems logical to me) instead of the standard deviation? Or another form of regression analysis? Or maybe some sort of paired difference testing?

Stats isn't my strongest area, and I have just been finding more and more statistical tests today that I know nothing about. If someone could point me to the correct testing for my situation I should be able to use other resources to do the calculations.

Thanks this forum has been a great help over the years,

Nicola

 

[mfb]

There is no unique way to quantify "difference between theory and experiment".

- if you expect the same result for all different data points, you can average both theory and experiment, and give a relative deviation as systematic "model error" (this value will have an uncertainty, coming from your measurements).
- if you do not expect this, you can calculate the relative deviation for each point, and average that. This will tell you if the model has some systematic under/overestimation of the values
- if you are more interested in the difference between different data points, you can calculate the squared difference (relative or absolute), and calculate the average of that.
- if you think your model should be "perfect", you can do a chi2-test.
- ...

 

[nixi]

Situation 2 applies to my work.

Thank you, I have pondered it over a bit longer and realized that I wasn't thinking about the aim of my analysis correctly. Yes I should have been looking for the systematic error of the algorithm and the arithmetic mean would give me this without being skewed by the +/- sign. I think I had been thinking in terms of a zero-deviation would represent the perfect algorithm so I was trying to see how my differences deviated from that value - I suppose in this case I was trying to re-invent the standard deviation using absolute values (like MAE) with 0 as my mean.

The moral of the story: go with your instinct and don't overthink things

I appologise for spelling errors but we are forced to run IE7 at work.

Thanks and sorry for the time wasting!