Levenberg-Marquardt Algorithm for more than one function

[Shauheen]

I have been looking how to use the Levenberg Marqurdt algorithm for minimizing the errors of two functions at the same time. I looked up this topic in the internet and the only useful thing I found was a thread between "I like Serena" and "thomas430" in summer 2011.

https://www.physicsforums.com/showthread.php?t=521670

I read "I like Serena"'s notes, but still have problem in finding out how I have to calculate the Jacobian matrix.

I am measuring two values over time in the lab. Then, a numerical forward model has been developed which models the physics and now, I need to solve the inverse problem to find the unknown parameters. I have solved this problem previously several times when only one error function needed to be minimized. Based on what I have learned about LMA (from Inverse Heat Transfer by Necati Ozicik), the gradient of the error (from one function) will result in multiplication of the Jacobian matrix by the difference between the experimental and model results. This gradient needs to become zero:

∇S(P)=2[-∂T(P)/∂P][Y-T(P)]=0

In which P is the vector of the unknown parameters, T is the model value vector and Y is the experimental value.
This problem, minimizes S(P)=Ʃ[Yi-Ti(P)]^2

Now, I have to find let say 3 parameters from the minimization of two errors from my experiments. In my experiment, I measure two values vs. time, let say Y and M. These two values are dependent. The number of the collected data are not the same for these two and I think they are both equally important. I could check on the typical range of errors to make them equally significant.
My question is how to minimize the error of the combination of these two and find my unknowns?

Let say if the error is like:
E(P)=Ʃ[Yi-Ti(P)]^2+Ʃ[Mi-Ui(P)]^2

Ui and Ti are the values from my model.
What would be the Jacobian matrix in this case? for one function, it would be like:
P(k+1)=P(k)+[(J(k)T.J(k)+μ(k)Ω(k)]^-1 (J(k))T[Y-T(P(k))]
in which k is the kth iteration.

How would be the formulation once two functions are involved?

I would really appreciate if anyone could help.

 

[I like Serena]

Hello and welcome to PF, Shauheen!

I'm not sure if I understand everything in your problem yet, but as I understand it now, you have a measured vector Y and a model T(P) for it, and you also have a measured vector M and a model U(P) for it.

You can turn these 2 measurement vectors into one but putting them on top of each other.
Let's say we let N=(Y1, ..., Yn, M1, ..., Mm), and let V(P)=(T1(P), ..., Tn(P),U1(P), ..., Um(P)).
Then that reduces your problem back to where you have one measurement vector and one model vector function.
The new Jacobian matrix is basically a combination of the 2 separate Jacobian matrices.

 

[Shauheen]

Hello and welcome to PF, Shauheen!

I'm not sure if I understand everything in your problem yet, but as I understand it now, you have a measured vector Y and a model T(P) for it, and you also have a measured vector M and a model U(P) for it.

You can turn these 2 measurement vectors into one but putting them on top of each other.
Let's say we let N=(Y1, ..., Yn, M1, ..., Mm), and let V(P)=(T1(P), ..., Tn(P),U1(P), ..., Um(P)).
Then that reduces your problem back to where you have one measurement vector and one model vector function.
The new Jacobian matrix is basically a combination of the 2 separate Jacobian matrices.

Thanks a lot I Like Serena for your kind welcome and reply to my question :)
Y and M are two different physical values. One is height change (I am tracking the height) and the other is mass. I start collecting each of these two data sets from time zero to t(final). Would that still be possible to put the two different physical value on top of each other in a matrix?
What would be my error then?

Physically in my mind, it is like plotting both curves vs. time and trying to match the two model to these curves with the same parameters, right?

I have attached for you a handwritten description about my problem and have attached it to this reply.

Thanks again for devoting your time.

 

[I like Serena]

Thanks a lot I Like Serena for your kind welcome and reply to my question :)
Y and M are two different physical values. One is height change (I am tracking the height) and the other is mass. I start collecting each of these two data sets from time zero to t(final). Would that still be possible to put the two different physical value on top of each other in a matrix?
What would be my error then?

Physically in my mind, it is like plotting both curves vs. time and trying to match the two model to these curves with the same parameters, right?

I have attached for you a handwritten description about my problem and have attached it to this reply.

Thanks again for devoting your time.

You can put the 2 vectors on top of each other.
The result is not so much a matrix, but just another vector.

The error function can be the same to what you already have:
$$E(P)=\sum_{i=1}^{n+m}[N_i-V_i(P)]^2=\sum_{i=1}^{n}[Y_i-T_i(P)]^2+\sum_{i=1}^{m}[M_i-U_i(P)]^2$$
where n is the number of Y measurements and m is the number of M measurements.When you have solved the system, useful properties will be:
$$s_Y = \sqrt{\frac{\sum[Y_i-T_i(P)]^2}{n-1}}$$
$$s_M = \sqrt{\frac{\sum[M_i-U_i(P)]^2}{m-1}}$$
where $s_Y$ is the standard deviation of the error in the Y measurements and $s_M$ is the standard deviation of the error in the M measurements.

 

[blue_raver22]

I would suggest that you start by breaking down the problem into smaller, more manageable parts. First, focus on understanding the Levenberg-Marquardt algorithm for minimizing one function. This will give you a solid foundation and understanding of the algorithm.

Once you have a good understanding of the algorithm for one function, you can then move on to applying it to multiple functions. In this case, you will need to consider the relationship between the two functions and how they affect each other. This will help you determine the appropriate formulation for the Jacobian matrix.

It may also be helpful to consult with other researchers or experts in the field who have experience with using the Levenberg-Marquardt algorithm for multiple functions. They may be able to provide insights and guidance on how to approach and solve this problem.

Overall, the key is to break down the problem into smaller parts and build your understanding and knowledge step by step. With persistence and careful consideration, you will be able to successfully apply the Levenberg-Marquardt algorithm to minimize the errors of multiple functions.