Shaky model in least squares fit

[Gigaz]

I've come across a problem with my least squares fits and I think someone else must have analyzed this, but I don't know where to find it.

I have a converged least squares fit of my spectroscopic data. Unfortunately, the physical model, on which the fit is based, is mediocre. The deviations between measurement and model are much larger than the statistical errors at each data point. There is almost certainly nothing I can do about that. The fit reproduces the data reasonably well, but the model is incomplete.

I know that there are some parameters inside the model, which do not seem to be very robust. If I fit only half of my data (only s or only p polarization), they always come out differently. Other parameters remain totally unchanged.

I'm looking basically for an idea on how I could quantify this "robustness". It can probably been done based on some sort of artificial perturbation function, but I haven't seen anything like that.

 

[I like Serena]

Hi Gigaz, welcome to PF,

Apparently your systematic error (error with respect to the proposed model) is much larger than the error due to noise.
To quantify that we can divide the variance in the systematic errors by the variance of the noise.
This is called the F-value.
We can use the F-test to verify how significant this is with the hypothesis that a proposed regression model fits the data well.

For the record, a least squares method assumes that the errors are independent, normally distributed, have equal variance everywhere, and have expectation zero.
The F-test can verify part of those assumptions.

 

[EnumaElish]

A case of data in search of a model, it seems. You can apply the same data to different models, and see how that changes the fit. For example, you can fix one coefficient at a time, and re-estimate the remaining parameters.

 

[Gigaz]

Many thanks for those suggestions. I will try it and see what comes out :)

 

[chiro]

Hey Gigaz.

There are quite a lot of parameters that test this. Here is some information below:

https://en.wikipedia.org/wiki/Goodness_of_fit

 

[Gigaz]

Thanks, chiro. Apparently, what I need is listed in your link: The Akaike information criterion.