Histogram fitting: fit parameter errors not corresponding with optimizer results

[alex-weej]

Hi

I'm having some big problems with some data! I will try to keep this as simple as possible...

I have a random variable that admits a probability distribution that I have a fit function for. With a large enough number of samples I can get good estimates of the fit function parameters via a least-squares optimizer (minpack from scipy.optimize.leastsq I believe). The optimizer gives me a covariance matrix from which I extract approximate errors on the parameters (square root of the diagonal).

The problem is that the errors obtained by this method are too small, because If I fit a different set of data from the same distribution I get some other estimate for the fit parameters with an error which is also very small and these do not overlap. As a test, I fit ~10,000 different sets of the data (with ~10,000 samples in each) and saw that I get a nicely shaped gaussian for the fit parameters. By eye, the standard deviation is about 10 times larger than the error I get from the covariance matrix.

I have manually verified that the errors calculated from the covariance matrix correspond to a change in ~1 of the chi-squared for the fit.

Am I doing anything obviously wrong? Please save me!

Thank you

Alex

 

[mmwave]

Hi Alex,

It sounds like you are on the right track with using a least-squares optimizer and extracting the errors from the covariance matrix. However, there could be a few things that could be causing the small errors you are getting.

First, it's important to make sure that your fit function is appropriate for the data you are working with. If the function is not a good fit for the data, it could lead to small errors in the parameter estimates. You may want to try fitting the data with a few different functions to see if you get similar results.

Another possibility is that your data may not be normally distributed, which is often assumed when using a least-squares optimizer. If your data is not normally distributed, this could lead to small errors in the parameter estimates. You could try using a different optimizer or a different statistical method to fit your data.

It's also possible that the number of samples you are using is not large enough to accurately estimate the errors. Typically, the larger the sample size, the more accurate the estimates will be. You may want to try increasing the number of samples and see if that changes the results.

Overall, it's important to carefully consider the assumptions and methods you are using when fitting data and interpreting the results. It may also be helpful to consult with a statistician or other expert in your field for further guidance. Best of luck with your research!