Comparing theory to data

  • Thread starter Amanheis
  • Start date
  • #1
67
0

Main Question or Discussion Point

I need to quantify the statistical significance of how much a model deviates from a given set of data points, and I cannot do a fit.

Let's say the model is a one-parameter description of some time dependent quantity f_a(t). I have data points at n different times including error bars, so pi = {ti, fa(ti), sigmai}. The reason I cannot do a fit is that the data is actually the predicted errors for some fiducial value a_0, and the fit would obviously just find a = a0.

I want to know how far off a couple models with either a = a1 or a = a2 are. So in other words, how well does fa1(t) fit the data points pi? Can I rule it out on some confidence level?


It seems difficult to express my problem, I hope it became clear enough. All I am asking for is a hint in the right direction, so if you know some relevant reference, I can just get it in the library. I already calculated the likelihood L(a) but don't really know how to proceed.

Thanks.
 

Answers and Replies

  • #2
EnumaElish
Science Advisor
Homework Helper
2,304
124
If you have the predicted (projected) value and the error term for each t, why can't you calculate ("back out") the actual observed value corresponding to that t?
 
  • #3
67
0
I don't understand. How am I supposed to do that? The actual observed value won't be available until several years from now. And "observed value" implies that it is observed, I don't see how I can calculate it.
 
  • #4
EnumaElish
Science Advisor
Homework Helper
2,304
124
So, the error bars you mentioned are forecast (future) errors? That wasn't clear to me.

Can you post your understanding of what these errors are, or what they represent?
 
Last edited:
  • #5
67
0
Yes, the error bars are predicted. I am basically using the specifications of the experiment and the Fisher matrix formalism to assess the quality of the constraints that will be put on fiducial values of a set of parameters.
 
  • #6
EnumaElish
Science Advisor
Homework Helper
2,304
124
Let me make up an example: you are predicting the value of random variable Y for, say February 2012. You have a point estimate, [tex]\hat Y(Feb. 2012)[/tex], and an error "bar" "wrapped around" the point estimate. Have I got it?
 
  • #7
67
0
Let me make up an example: you are predicting the value of random variable Y for, say February 2012. You have a point estimate, [tex]\hat Y(Feb. 2012)[/tex], and an error "bar" "wrapped around" the point estimate. Have I got it?
Not quite. We already kind of know the value of Y(Feb 2012). For the sake of clarity let's choose a date in the ast, say the average temperature at some place in the year 1500. Also I will adopt my notation from the original post.
We already kind of know how the temperature behaves over time, depending on a set of parameters, one of them being a=a0. According to earlier measurements, we expect something like f(1500)=20C, f(1700)=22C and f(1900)=25C.
But we want to constrain it as much as possible and plan an improved experiment. According to my calculations, we are expecting to get error bars of .1C, 0.15C and 0.2C on each of these values after that new experiment. Now we want to see if we are going to be able to test a competing theory (ie. the parameter a has a different value). I want to quantify how much the curve of the new theory with a=a1 would deviate from the data with the predicted error bars.
 
  • #8
EnumaElish
Science Advisor
Homework Helper
2,304
124
I can describe how to tell whether the projections based on a0 are statistically significantly different from a projection based on a1. The error bars imply a joint distribution of temperatures across years. You can resample data points using this joint distribution and re-estimate f as a function of t based on each (re)sample. This will yield a set of values (an empirical distribution) for the a parameter, with mean = a0. You can then test whether a1 is statistically different from the mean, using the empirical distribution of a.
 
Last edited:

Related Threads on Comparing theory to data

Replies
5
Views
3K
Replies
5
Views
2K
Replies
1
Views
1K
Replies
10
Views
2K
  • Last Post
Replies
4
Views
2K
Replies
8
Views
790
Replies
5
Views
5K
Replies
5
Views
380
Replies
1
Views
4K
Replies
3
Views
12K
Top