Non-linear least-squares: Uncertainty estimates including non-fitted parameters

In summary, the conversation discusses a question regarding the incorporation of uncertainties in fixed parameters into the estimated uncertainties of fitted parameters in a non-linear least-squares optimization problem. The problem is illustrated using an example of fitting lorentzians to experimental data with known widths from theory. The uncertainty in the theoretical positions of the fitted lorentzians can affect the uncertainty of the amplitudes and ultimately the branching ratios being measured. The conversation also mentions the use of Hessian matrices and Bayesian formulations in addressing this issue.
  • #1
Kazza_765
171
0
Hi all,

I have a question regarding least-squares, and I'm certain I can't be the first one to encounter it, but I've had no luck searching the literature for a solution. Here it is:

Say we have a non-linear least-squares optimisation problem. We have data points [itex]y_i[/itex] and a model [itex]y(x_i;{\bf a})[/itex] where [itex]\bf a[/itex] is a vector of parameters we wish to fit to the data. The merit function of course is

[tex]\chi^2 = \sum^N_{i=1}[\frac{y_i - y(x_i;{\bf a})}{\sigma_i}][/tex]

And we can get our parameter uncertainty estimates from the Hessian matrix.


Now let's say that the model we wish to fit to the data is [itex]y(x_i;{\bf a};{\bf b})[/itex] where [itex]\bf a[/itex] is still our vector of fitted parameters, and [itex]\bf b[/itex] is a vector of fixed parameters. At this stage nothing has changed, I've just re-written y(x_i) to make the non-fitted parameters explicit.


We can go ahead and get our uncertainties for [itex]\bf a[/itex] in the same manner as before. My question, however, is this:

Suppose there are uncertainties in [itex]\bf b[/itex]. How do we incorporate those uncertainties into the estimated uncertainty of [itex]\bf a[/itex]?




As an example of the problem: Say we wish to fit three lorentzians of known width to some data. The amplitudes and centroids are fitted parameters (ie. [itex]\bf a[/itex]), and the widths are fixed parameters (ie. [itex]\bf b[/itex]). Perhaps we know the widths from theory, or from another experiment.

Using the Hessian matrix we can get an uncertainty in the amplitudes of the lorentzians. However, if there is also uncertainty in the widths then this will increase our uncertainty in the amplitudes. The uncertainty in the width needs to be incorporated into the problem somehow, but I have no idea where to start with it.



I'm sure there must already be a solution out there for a problem like this. I'm just not sure where to find it. Any help would be much appreciated.

Thankyou
 
Mathematics news on Phys.org
  • #2
Kazza_765 said:
And we can get our parameter uncertainty estimates from the Hessian matrix.

There are various definitions of "uncertainty" and it isn't clear which one you are using. Least squares fitting of a model to data can be done by algorithms that involve computing Hessian matrices, but the only interpretation that I have seen where this bears on "uncertainty" is in a Bayesian formulation (such as described in this PDF http://www.google.com/url?sa=t&source=web&cd=4&ved=0CC8QFjAD&url=http%3A%2F%2Fctr.stanford.edu%2F~jops%2Fuq_workshop%2Fsaturday%2Fghattas.pdf&ei=roRETr6ZDc3UiAK37PjXAQ&usg=AFQjCNGhzT9mkVQ7Dz8nY9xkpwM3tFCZfQ [Broken]
).

Are you using Bayesian prior distributions for the parameter set 'a'?

You seem to be conflicted about the parameters 'b'. One one hand, you aren't willing to vary them when you fit the model and on the other hand you want to quantify their "uncertainty".
 
Last edited by a moderator:
  • #3
Stephen Tashi said:
There are various definitions of "uncertainty" and it isn't clear which one you are using. Least squares fitting of a model to data can be done by algorithms that involve computing Hessian matrices, but the only interpretation that I have seen where this bears on "uncertainty" is in a Bayesian formulation (such as described in this PDF http://www.google.com/url?sa=t&source=web&cd=4&ved=0CC8QFjAD&url=http%3A%2F%2Fctr.stanford.edu%2F~jops%2Fuq_workshop%2Fsaturday%2Fghattas.pdf&ei=roRETr6ZDc3UiAK37PjXAQ&usg=AFQjCNGhzT9mkVQ7Dz8nY9xkpwM3tFCZfQ [Broken]
).

Are you using Bayesian prior distributions for the parameter set 'a'?

I'm not using a Bayesian formulation (at least I don't think I am). The least-squares fitting is using the Levenberg-Marquardt method. I should have said the covariance matrix rather than the Hessian (although, if I recall correctly, one is the inverse of the other divided by two?).

You seem to be conflicted about the parameters 'b'. One one hand, you aren't willing to vary them when you fit the model and on the other hand you want to quantify their "uncertainty".

I perhaps wasn't clear. I don't want to quantify the uncertainty of the parameters b - they already have a particular uncertainty associated with them. Rather, I wish to know how the uncertainty in the parameters b contribute to the uncertainty of the parameters a.

Here is the actual problem I'm trying to solve:

We have an experimental atomic photoemission spectra that consists of overlapping spectra from a number of different transitions. From theory we know what the energies of the transitions are, so we can center a Lorentzian on each theoretical transition and then fit the amplitudes and widths to the experimental data. After doing this, we can work out the relative intensities of the various transitions from the integrated intensities, and from the covariance matrix we have an uncertainty in this value.

Just to clarify - the positions of the Lorentzians (on the x/energy axis) is fixed by theory and is not a fitted parameter.

However, there is still uncertainty in theory. And so there is uncertainty in the position of those lorentzians. This uncertainty in the positions of the lorentzians will have an effect on the uncertainty of the amplitudes, and hence of the branching ratios we are ultimately trying to measure.

So far, all measurements I can find completely neglect that there is an uncertainty in the theoretical positions of the fitted lorentzians and how this impacts the final uncertainty in amplitude ratios.



Maybe I should draw a diagram, I'm not sure if I'm explaining myself particularly clearly. Thanks in advance for your help.
 
Last edited by a moderator:
  • #4
You didn't define "uncertainty" yet, but from your details, it looks like you are referring to quantum mechanics.

From the 2nd Edtion of Shankar's book, page 128:

The Uncertainty

In any situation described probabilistically, another useful quantity to specify besides the mean is the standard deviation, which measures the average fluctucation around the mean. It is defined as

[tex] \triangle \Omega = <(\Omega - <\Omega>)^2>^{\frac{1}{2}} [/tex]

and often called the root-mean-squared deviation. In quantum mechanics, it is referred to as the uncertainty in [itex]\Omega [/itex].

I , myself, don't know quantum mechanics, so I don't understand whether your question that can be answered by "ordinary" probability theory or whether it involves eigenstates of operators and other quantum mechanical concepts.

Attempting to interpret your problem in ordinary probability theory, it sounds like you have data from a random variable that is a "mixture" ( i.e. a linear combination) of several probability distributions. You estimate the unknown coefficients in this linear combination by an algorithm that maximizes some measure of fit. Viewing the estimates as random variables, they have certain standard deviations. The distributions involved in the mixture also have standard deviations. You want some sort of combined standard deviation or deviations that combine all the standard deviations involved. Is that roughtly correct?
 
  • #5
for your question. Incorporating uncertainties in non-fitted parameters into the uncertainty estimates of fitted parameters is a common problem in non-linear least-squares optimization. One approach is to use a Monte Carlo simulation, where you randomly sample the values of the non-fitted parameters within their uncertainties and perform the least-squares optimization for each sample. This will give you a distribution of the fitted parameters, from which you can calculate the uncertainty estimates.

Another approach is to use a Bayesian framework, where you can incorporate the uncertainties in non-fitted parameters as prior distributions. This will give you a posterior distribution of the fitted parameters, from which you can calculate the uncertainty estimates.

There are also various methods for incorporating uncertainties in non-fitted parameters directly into the Hessian matrix, such as the Levenberg-Marquardt method or the Gauss-Newton method.

I would recommend consulting with a statistician or reading up on literature on uncertainty estimation in non-linear least-squares optimization for more specific guidance on which approach would be most appropriate for your specific problem.
 

1. What is non-linear least-squares?

Non-linear least-squares is a method used in statistics and data analysis to estimate the parameters of a non-linear mathematical model. It is commonly used when the relationship between the independent and dependent variables is non-linear.

2. How is uncertainty estimated in non-linear least-squares?

Uncertainty estimates in non-linear least-squares are typically determined using the covariance matrix, which takes into account the uncertainties in both the measured data and the model parameters.

3. What are non-fitted parameters in non-linear least-squares?

Non-fitted parameters in non-linear least-squares are parameters that are not directly estimated by the model, but are necessary for the model to accurately describe the data. These parameters are often fixed or constrained based on prior knowledge or assumptions.

4. How do non-linear least-squares and linear least-squares differ?

Non-linear least-squares differs from linear least-squares in that it allows for non-linear relationships between the variables, while linear least-squares assumes a linear relationship. Non-linear least-squares also typically requires a more iterative approach to find the best-fit parameters.

5. What are some applications of non-linear least-squares?

Non-linear least-squares has a wide range of applications in various fields, such as physics, biology, economics, and engineering. Some examples include curve fitting, parameter estimation in mathematical models, and data analysis in nonlinear systems.

Similar threads

Replies
8
Views
1K
  • STEM Educators and Teaching
Replies
11
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
7
Views
335
  • Set Theory, Logic, Probability, Statistics
Replies
2
Views
1K
  • General Math
Replies
5
Views
1K
  • Linear and Abstract Algebra
Replies
1
Views
1K
Replies
10
Views
3K
  • Set Theory, Logic, Probability, Statistics
Replies
16
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
2
Views
1K
Replies
1
Views
587
Back
Top