Take errors into account for a data fit

[kelly0303]

Hello! I have some data in which the dependent variable $y$ has, for each data point, an error bar associated with it $\delta y$. The errors are almost identical for each datapoint, so doing a weighted fit in terms of the errors would not change the results significantly. How can I take the errors into account properly, such that the error bars are reflected in the error on the parameters of the fit? I though initially to have 3 fits for the actual data points, for the upper and for the lower points of the error bars, but I am not sure that would give me what I need, as the function is highly nonlinear and this kind of fit would just influence the overall amplitude, not the other (more important) parameters. Thank you!

<Mentor - moved to statistics forum >

 

[jim mcnamara]

Could please you define what you mean by 'the errors are almost identical'? Which makes me wonder about your view of the effect on amplitude of your proposed treatment. It would help our statisticians if you elaborated on your dataset as well.

Thanks.

 

[kelly0303]

Could please you define what you mean by 'the errors are almost identical'? Which makes me wonder about your view of the effect on amplitude of your proposed treatment. It would help our statisticians if you elaborated on your dataset as well.

Thanks.

Thank you for your reply! The measurements are some counts produced in an experiment at different energies. The counts are, for each energy, somewhere between 100,000 (for background) and 300,000 (for signals). The error is Poisson i.e. square root of the number of counts.

 

[Minnesota Joe]

Listed roughly in order of difficulty of estimating errors, I have fit count data using linear least squares (sometimes with a square root transformation), Poisson regression, and non-linear least squares to physical models.

From what you wrote I'm assuming you are asking about the last case and you have a physical model that is nonlinear in its parameters.

What software are you using to perform the fit? Because a short answer is that many packages (R, Matlab, Python) are capable of reporting errors for the model predictions and model parameters. The values required to estimate these errors are often calculated as part of the same iterative process as the fit.

If you are writing your own code to do it, you may be able to make use of an asymptotic approximation using the Hessian matrix (second derivatives) of a log likelihood. Briefly, a fit routine typically estimates both the parameters and the variance-covariance matrix related to the Hessian from which, along with the independent variable data, error estimates may be obtained.

See this article I found after a quick search: nonlinear least squares. Especially pages 5-10.

If you aren't reinventing the wheel for educational purposes, Python has the lmfit package that I'd probably use if I were doing it. R has an equivalent package called nls(?) I think, but I haven't used it.