# How to Quantify Statistical Significance of Model Deviation from Data Points?

• Amanheis
In summary: If the p-value is smaller than some threshold, then the parameter a1 is deemed to be statistically different from a0.If you have the predicted (projected) value and the error term for each t, why can't you calculate ("back out") the actual observed value corresponding to that t?I don't understand. How am I supposed to do that?

#### Amanheis

I need to quantify the statistical significance of how much a model deviates from a given set of data points, and I cannot do a fit.

Let's say the model is a one-parameter description of some time dependent quantity f_a(t). I have data points at n different times including error bars, so pi = {ti, fa(ti), sigmai}. The reason I cannot do a fit is that the data is actually the predicted errors for some fiducial value a_0, and the fit would obviously just find a = a0.

I want to know how far off a couple models with either a = a1 or a = a2 are. So in other words, how well does fa1(t) fit the data points pi? Can I rule it out on some confidence level?

It seems difficult to express my problem, I hope it became clear enough. All I am asking for is a hint in the right direction, so if you know some relevant reference, I can just get it in the library. I already calculated the likelihood L(a) but don't really know how to proceed.

Thanks.

If you have the predicted (projected) value and the error term for each t, why can't you calculate ("back out") the actual observed value corresponding to that t?

I don't understand. How am I supposed to do that? The actual observed value won't be available until several years from now. And "observed value" implies that it is observed, I don't see how I can calculate it.

So, the error bars you mentioned are forecast (future) errors? That wasn't clear to me.

Can you post your understanding of what these errors are, or what they represent?

Last edited:
Yes, the error bars are predicted. I am basically using the specifications of the experiment and the Fisher matrix formalism to assess the quality of the constraints that will be put on fiducial values of a set of parameters.

Let me make up an example: you are predicting the value of random variable Y for, say February 2012. You have a point estimate, $$\hat Y(Feb. 2012)$$, and an error "bar" "wrapped around" the point estimate. Have I got it?

EnumaElish said:
Let me make up an example: you are predicting the value of random variable Y for, say February 2012. You have a point estimate, $$\hat Y(Feb. 2012)$$, and an error "bar" "wrapped around" the point estimate. Have I got it?

Not quite. We already kind of know the value of Y(Feb 2012). For the sake of clarity let's choose a date in the ast, say the average temperature at some place in the year 1500. Also I will adopt my notation from the original post.
We already kind of know how the temperature behaves over time, depending on a set of parameters, one of them being a=a0. According to earlier measurements, we expect something like f(1500)=20C, f(1700)=22C and f(1900)=25C.
But we want to constrain it as much as possible and plan an improved experiment. According to my calculations, we are expecting to get error bars of .1C, 0.15C and 0.2C on each of these values after that new experiment. Now we want to see if we are going to be able to test a competing theory (ie. the parameter a has a different value). I want to quantify how much the curve of the new theory with a=a1 would deviate from the data with the predicted error bars.

I can describe how to tell whether the projections based on a0 are statistically significantly different from a projection based on a1. The error bars imply a joint distribution of temperatures across years. You can resample data points using this joint distribution and re-estimate f as a function of t based on each (re)sample. This will yield a set of values (an empirical distribution) for the a parameter, with mean = a0. You can then test whether a1 is statistically different from the mean, using the empirical distribution of a.

Last edited:

## 1. How do you compare a theory to data?

To compare a theory to data, a scientist typically collects data through experiments or observations and then analyzes the data to see if it aligns with the predictions or expectations of the theory. This can involve statistical methods, visual representations, and other techniques to determine the level of agreement between the theory and the data.

## 2. What are some challenges in comparing theory to data?

Some challenges in comparing theory to data include ensuring the validity and reliability of the data, accounting for potential biases or confounding variables, and addressing discrepancies between the theory and the data. It can also be difficult to determine the appropriate level of significance or confidence in the results of the comparison.

## 3. How important is it to compare theory to data?

Comparing theory to data is a crucial aspect of the scientific process as it allows scientists to test the validity and applicability of theories. It also helps to refine and improve existing theories, as well as develop new theories to explain and predict phenomena.

## 4. What is the role of replication in comparing theory to data?

Replication is an important aspect of comparing theory to data as it allows for the confirmation or refutation of findings. When the results of a study are replicated by independent researchers, it increases the confidence in the validity and reliability of the findings, and strengthens the support for the theory being tested.

## 5. Can a theory ever be proven by data?

No, a theory can never be proven by data. Instead, data can support or refute a theory, but it can never provide absolute proof. This is because new data or evidence may emerge in the future that could challenge or contradict the theory. Therefore, theories are constantly evolving and subject to change based on new data and evidence.