How do I determine if improving a fit is significant?

  • I
  • Thread starter Buzz Bloom
  • Start date
  • Tags
    Fit
In summary, the data does not distinguish between an exponent of 2 and an exponent of whatever value you found when you let it float.
  • #1
Buzz Bloom
Gold Member
2,519
467
I have forgotten how to do a goodness of fit test for the problem below. I have not done this type of problem for several decades, and I hope someone can help me refresh my memory. I looked up the chi-squared test, Student t-test, and Fisher’s exact test on Wikipedia, but I found those articles to not be helpful.

I have 32 data points {xi, yi} i= 1:32.

I fit a curve of the form y = a × x2 to the data, and the RMS error is 34.8591.

I then modify the function I am fitting to be y = a × xb. The best fit then has an RMS error of 34.7029.

What is the proper statistical test to determine if the improved fit is statistically significant?
 
Last edited:
Physics news on Phys.org
  • #2
From the looks of it, I would say the RMS errors are identical. Am I correct in assuming that by RMS error you mean the square root of the sum of the squares of the differences between calculation and experiment? Do you have error bars ##\sigma_i## for your data? If so, when you calculate
$$\chi=\sqrt{ \sum_{i=1}^N \left[ \frac {y_{calc,i}-y_{expt,i}}{\sigma_i} \right]^2 }$$
and it's close to N, then you cannot do any better given the error bars. The hand waving proof is if the difference between calculation and experiment is about equal to the error bars (some say minus the number of adjustable parameters), then you are basically adding a bunch of ones, and the sum is roughly equal to the number of data points.

Your results suggest to me that allowing the exponent to vary did not improve the fit any and your data cannot distinguish the difference between an exponent of 2 and an exponent of whatever value you found when you let it float. I would consider it an improvement if the RMS dropped by at least a factor of 2. It may be that your model is inappropriate in that the exponent is actually 2 but the data require an additional adjustable parameter. Also, if it's the exponent that you want, have you considered doing a semi-log plot and fit to a straight line?
 
  • Like
Likes jim mcnamara
  • #3
kuruman said:
Your results suggest to me that allowing the exponent to vary did not improve the fit any and your data cannot distinguish the difference between an exponent of 2 and an exponent of whatever value you found when you let it float.
Hi Kuruman:

Thank you very much for your help. My own feeling was that the small difference in RMS did not indicate that finding the best fit with the floating value for the exponent was useful. What I wanted to relearn was how to determine a value for the likelihood the added parameter would be useful.

In the equation you present in your post, I assume
ycalc,i is the calculated value of y(xi) with b = 2,​
and
yexpt,i is the calculated value of y(xi) with the best fit b (which is for my data b = 1.937).​
Please let me know whether σi corresponds to b = 2, or b = 1.937. Also, since the RMS calculation I used is
RMSb=2 = SUM( (yi - ycalc,i)^2 ) / 32,​
then I assume that
σb=2 = SUM( (yi - ycalc,i)^2 ) / 31,​
because the best fit value for a sets
SUM( yi - ycalc,i) = 0,​
that is the degrees of freedom is N-1.

One more question:
When I calculate χ, and I look up the corresponding probability for significance, how man degrees of freedom do I use?

Regards,
Buzz
 
  • #4
I use either the BIC (Bayesian information criterion) or the AIC (Akieki information criterion) to compare the models.
 
Last edited:
  • #5
Buzz Bloom said:
In the equation you present in your post, I assume
ycalc,i is the calculated value of y(xi) with b = 2, ...
No. This is a general model-independent equation in which ycalc,i is the calculated value according to whatever model one uses. You have two ycalc,i arrays, one with b = 2 and one with adjustable b.
Buzz Bloom said:
.. and
yexpt,i is the calculated value of y(xi) with the best fit b (which is for my data b = 1.937).
Again no, yexpt,i is the array with "experimental" or data values. The process involves minimizing the difference between calculation and experiment, not the difference between one model and another.
Buzz Bloom said:
Please let me know whether σi corresponds to b = 2, or b = 1.937.
Neither. It corresponds to the error bar in your data. If for example your ith data point is
##y_{expt,i}=2.34\pm 0.05##, then ##\sigma_i=0.05##.

You should use N - k degrees of freedom where N is the number of data points and k the number of adjustable parameters.
 
  • #6
kuruman said:
yexpt,i=2.34±0.05y_{expt,i}=2.34\pm 0.05, then σi=0.05\sigma_i=0.05.
HI kuruman:

If I understand you correctly, then my raw data for y is
yi = yexpt,i
since I have no values for σi.

All I have for each data point is xi and yi. The data comes from a series of puzzles I have solved.
yi is the number of seconds it took for me to solve the puzzle.
xi is the number of items to be correctly ordered in the puzzle.​
With these definitions, it seems that the formula for χ in your post is not applicable.

My conclusion for the fit is that the solution time is proportional to the number of items squared with a random variance. The visual effect of a graph also suggests that the variance increases with the number of items, but I have not yet tried to explore that aspect.

ADDED
It has occurred to me that I might assume
yexpt,i = yi,b=2,
σi = yi - yi,b=2, and
ycalc,i = yi,b=fit - yi,b=2
Do you think this might give me the chi-squared value that will lead to the probability of significance?

Regards,
Buzz
 
Last edited:
  • #7
Dale said:
I use either the BIC (Bayesian information criterion) or the AIC (Akieki information criterion) to compare the models.
Hi Dale:

Thank you for your post. I looked briefly at both of these in Wikipedia, and both appear to be promising. Can you explain how you decide which of the two is likely to be better for any particular kind of models?

Regards,
Buzz
 
  • #8
Buzz Bloom said:
... since I have no values for σ ... i.
What do you mean by the variance in your points? That's what the σi are. If you can estimate what they are, you have them.
 
  • #9
Buzz Bloom said:
Thank you for your post. I looked briefly at both of these in Wikipedia, and both appear to be promising. Can you explain how you decide which of the two is likely to be better for any particular kind of models?
I have a strong preference for simpler models, so if they disagree then I always use the one that recommends the simpler model. However, they are so close that they almost never disagree. You can usually just use whichever is easier in your software.
 
Last edited:
  • #10
kuruman said:
What do you mean by the variance in your points? That's what the σi are. If you can estimate what they are, you have them.
I do not understand what you are asking me. My raw data {xi, yi} had no know variances. When I ADDED some additional thoughts to post #6, I reinterpreted what I had previously misunderstood, and I presented three equations that I thought I might be able to use in your equation for χ. Are these three equations OK?

Regards,
Buzz
 
  • #11
Dale said:
You can usually just use whichever is easier in your software.
Hi Dale:

The only statistical/modeling software I have is the Libre Office spreadsheet Calc. I guess that means I will choose the one I can "program" more easily with the spreadsheet.

Thanks for your help.

Regards,
Buzz
 
  • #12
Buzz Bloom said:
The only statistical/modeling software I have is the Libre Office spreadsheet Calc.
R is free. It is easier to download RStudio and learn a little R than to program the AIC or BIC in Libre Office. And R is very rich in useful packages
 
  • #13
Buzz Bloom said:
Do you think this might give me the chi-squared value that will lead to the probability of significance?
At this point, I think that using a canned program of the kind that @Dale suggests would be a better way for you to proceed. Keep in mind that a mathematical model is appropriate to the extent that the data allow. If the scatter in the data is large enough, you may be prevented from distinguishing one theoretical model from another no matter what you use for your analysis. So if either b =2 or b = 1.937 provide very comparable "goodness of fit" criteria, either exponent is equally acceptable. It would be instructive to find a range of b-values that give acceptable fits to your data. In other words, how far from 2 either way can b be before you decide that you have a bad fit. The software just spits out numbers; it doesn't tell you if the fit is good or bad, so it's your judgment call.
 
  • #14
kuruman said:
It would be instructive to find a range of b-values that give acceptable fits to your data.
This type of analysis is what Bayesian statistics is particularly good at.
 

Related to How do I determine if improving a fit is significant?

1. What is the significance level in determining if an improvement in fit is significant?

The significance level, also known as alpha, is a predetermined threshold used to determine if the improvement in fit is statistically significant. It is typically set at 0.05 or 0.01, indicating the probability that the observed improvement in fit is due to chance.

2. How do I calculate the p-value for a fit improvement?

The p-value is the probability of obtaining a result at least as extreme as the observed improvement in fit, assuming the null hypothesis is true (i.e. the improvement is due to chance). It can be calculated using statistical software or by comparing the observed improvement to a critical value from a statistical table.

3. What factors should be considered when determining if a fit improvement is significant?

The significance of a fit improvement depends on several factors, including the sample size, the variability of the data, and the strength of the relationship between the variables. It is important to consider these factors in addition to the p-value when determining the significance of a fit improvement.

4. Can a statistically significant improvement in fit still be practically insignificant?

Yes, a statistically significant improvement in fit does not necessarily mean that it has practical significance. It is important to consider the magnitude of the improvement and whether it has a meaningful impact on the research or practical application.

5. Is it possible to determine if an improvement in fit is significant without prior knowledge of statistical methods?

No, determining the significance of a fit improvement requires knowledge of statistical methods and the ability to perform statistical analyses. It is important to consult with a statistician or use statistical software to accurately determine the significance of a fit improvement.

Similar threads

  • Set Theory, Logic, Probability, Statistics
Replies
3
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
1
Views
942
  • Set Theory, Logic, Probability, Statistics
Replies
21
Views
3K
  • Set Theory, Logic, Probability, Statistics
Replies
3
Views
377
  • Set Theory, Logic, Probability, Statistics
Replies
5
Views
7K
  • Set Theory, Logic, Probability, Statistics
Replies
2
Views
3K
  • Set Theory, Logic, Probability, Statistics
Replies
4
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
2
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
5
Views
2K
Replies
1
Views
912
Back
Top