How do I determine if improving a fit is significant?

[Buzz Bloom]

I have forgotten how to do a goodness of fit test for the problem below. I have not done this type of problem for several decades, and I hope someone can help me refresh my memory. I looked up the chi-squared test, Student t-test, and Fisher’s exact test on Wikipedia, but I found those articles to not be helpful.

I have 32 data points {x_i, y_i} i= 1:32.

I fit a curve of the form y = a × x² to the data, and the RMS error is 34.8591.

I then modify the function I am fitting to be y = a × x^b. The best fit then has an RMS error of 34.7029.

What is the proper statistical test to determine if the improved fit is statistically significant?

 

[kuruman]

From the looks of it, I would say the RMS errors are identical. Am I correct in assuming that by RMS error you mean the square root of the sum of the squares of the differences between calculation and experiment? Do you have error bars $\sigma_i$ for your data? If so, when you calculate
$$\chi=\sqrt{ \sum_{i=1}^N \left[ \frac {y_{calc,i}-y_{expt,i}}{\sigma_i} \right]^2 }$$
and it's close to N, then you cannot do any better given the error bars. The hand waving proof is if the difference between calculation and experiment is about equal to the error bars (some say minus the number of adjustable parameters), then you are basically adding a bunch of ones, and the sum is roughly equal to the number of data points.

Your results suggest to me that allowing the exponent to vary did not improve the fit any and your data cannot distinguish the difference between an exponent of 2 and an exponent of whatever value you found when you let it float. I would consider it an improvement if the RMS dropped by at least a factor of 2. It may be that your model is inappropriate in that the exponent is actually 2 but the data require an additional adjustable parameter. Also, if it's the exponent that you want, have you considered doing a semi-log plot and fit to a straight line?

 

[Buzz Bloom]

Your results suggest to me that allowing the exponent to vary did not improve the fit any and your data cannot distinguish the difference between an exponent of 2 and an exponent of whatever value you found when you let it float.

Hi Kuruman:

Thank you very much for your help. My own feeling was that the small difference in RMS did not indicate that finding the best fit with the floating value for the exponent was useful. What I wanted to relearn was how to determine a value for the likelihood the added parameter would be useful.

In the equation you present in your post, I assume

y_calc,i is the calculated value of y(x_i) with b = 2,​

and

y_expt,i is the calculated value of y(x_i) with the best fit b (which is for my data b = 1.937).​

Please let me know whether σ_i corresponds to b = 2, or b = 1.937. Also, since the RMS calculation I used is

RMS_b=2 = SUM( (y_i - y_calc,i)^2 ) / 32,​

then I assume that

σ_b=2 = SUM( (y_i - y_calc,i)^2 ) / 31,​

because the best fit value for a sets

SUM( y_i - y_calc,i) = 0,​

that is the degrees of freedom is N-1.

One more question:
When I calculate χ, and I look up the corresponding probability for significance, how man degrees of freedom do I use?

Regards,
Buzz

 

[Dale]

I use either the BIC (Bayesian information criterion) or the AIC (Akieki information criterion) to compare the models.

 

[kuruman]

In the equation you present in your post, I assume
y_calc,i is the calculated value of y(x_i) with b = 2, ...

No. This is a general model-independent equation in which y_calc,i is the calculated value according to whatever model one uses. You have two y_calc,i arrays, one with b = 2 and one with adjustable b.

.. and
y_expt,i is the calculated value of y(xi) with the best fit b (which is for my data b = 1.937).

Again no, y_expt,i is the array with "experimental" or data values. The process involves minimizing the difference between calculation and experiment, not the difference between one model and another.

Please let me know whether σi corresponds to b = 2, or b = 1.937.

Neither. It corresponds to the error bar in your data. If for example your ith data point is
$y_{expt,i}=2.34\pm 0.05$, then $\sigma_i=0.05$.

You should use N - k degrees of freedom where N is the number of data points and k the number of adjustable parameters.

 

[Buzz Bloom]

yexpt,i=2.34±0.05y_{expt,i}=2.34\pm 0.05, then σi=0.05\sigma_i=0.05.

HI kuruman:

If I understand you correctly, then my raw data for y is

y_i = y_expt,i​

since I have no values for σ_i.

All I have for each data point is x_i and yⁱ. The data comes from a series of puzzles I have solved.

y_i is the number of seconds it took for me to solve the puzzle.
x_i is the number of items to be correctly ordered in the puzzle.​

With these definitions, it seems that the formula for χ in your post is not applicable.

My conclusion for the fit is that the solution time is proportional to the number of items squared with a random variance. The visual effect of a graph also suggests that the variance increases with the number of items, but I have not yet tried to explore that aspect.

ADDED
It has occurred to me that I might assume

y_expt,i = y_i,b=2,
σ_i = y_i - y_i,b=2, and
y_calc,i = y_i,b=fit - y_i,b=2​

Do you think this might give me the chi-squared value that will lead to the probability of significance?

Regards,
Buzz

 

[Buzz Bloom]

I use either the BIC (Bayesian information criterion) or the AIC (Akieki information criterion) to compare the models.

Hi Dale:

Thank you for your post. I looked briefly at both of these in Wikipedia, and both appear to be promising. Can you explain how you decide which of the two is likely to be better for any particular kind of models?

Regards,
Buzz

 

[kuruman]

... since I have no values for σ ... _i.

What do you mean by the variance in your points? That's what the σ_i are. If you can estimate what they are, you have them.

 

[Dale]

Thank you for your post. I looked briefly at both of these in Wikipedia, and both appear to be promising. Can you explain how you decide which of the two is likely to be better for any particular kind of models?

I have a strong preference for simpler models, so if they disagree then I always use the one that recommends the simpler model. However, they are so close that they almost never disagree. You can usually just use whichever is easier in your software.

 

[Buzz Bloom]

What do you mean by the variance in your points? That's what the σi are. If you can estimate what they are, you have them.

I do not understand what you are asking me. My raw data {x_i, y_i} had no know variances. When I ADDED some additional thoughts to post #6, I reinterpreted what I had previously misunderstood, and I presented three equations that I thought I might be able to use in your equation for χ. Are these three equations OK?

Regards,
Buzz

 

[Buzz Bloom]

You can usually just use whichever is easier in your software.

Hi Dale:

The only statistical/modeling software I have is the Libre Office spreadsheet Calc. I guess that means I will choose the one I can "program" more easily with the spreadsheet.

Thanks for your help.

Regards,
Buzz

 

[Dale]

The only statistical/modeling software I have is the Libre Office spreadsheet Calc.

R is free. It is easier to download RStudio and learn a little R than to program the AIC or BIC in Libre Office. And R is very rich in useful packages

 

[kuruman]

Do you think this might give me the chi-squared value that will lead to the probability of significance?

At this point, I think that using a canned program of the kind that @Dale suggests would be a better way for you to proceed. Keep in mind that a mathematical model is appropriate to the extent that the data allow. If the scatter in the data is large enough, you may be prevented from distinguishing one theoretical model from another no matter what you use for your analysis. So if either b =2 or b = 1.937 provide very comparable "goodness of fit" criteria, either exponent is equally acceptable. It would be instructive to find a range of b-values that give acceptable fits to your data. In other words, how far from 2 either way can b be before you decide that you have a bad fit. The software just spits out numbers; it doesn't tell you if the fit is good or bad, so it's your judgment call.

 

[Dale]

It would be instructive to find a range of b-values that give acceptable fits to your data.

This type of analysis is what Bayesian statistics is particularly good at.