Levenberg-Marquardt Algorithm with Several Functions

In summary, the conversation discusses the use of the Levenberg-Marquardt algorithm for minimizing a function result. The algorithm can be applied to a system of equations, either by calling it separately for each equation or by summing the squares of the equations and minimizing the result. The conversation also mentions the use of a Jacobian matrix and the need to normalize the variance of the results.
  • #1
thomas430
17
0
Hi there, I have been testing out the Levenberg-Marquardt algorithm. I've successfully coded a method in MATLAB for the example I saw on wikipedia:

f(x) = a*cos(bx)+b*sin(ax)

and this has worked well. The application I'm using the algorithm for is a system of 3 equations, however.

Does anyone have any ideas on how to implement the algorithm for multiple functions?

Thomas.
 
Mathematics news on Phys.org
  • #2
Hi thomas430! :smile:

Levenberg-Marquardt minimizes a function result.

If you have 3 equations, you need to consider what it is that you want to minimize.
If they are independent, you can call Levenberg-Marquardt 3 times separately.

If they are dependent, then one method is to rewrite each equation to a function that should be equal to zero.
Sum the squares of those functions and minimize the result.
Effectively you're doing a least squares algorithm.
 
Last edited:
  • #3
Thanks, I like Serena! Your reply helped a lot, but I'm still trying to get my head around how to relate it back to my problem.

My problem looks like this... three equations (I haven't put the actual ones because they're very long):

f1(a,b) = 0
f2(a,b) = 0
f3(a,b) = 0

where a are measurements, and b are parameters. But the system grows because I've made many sets of measurements... say I've made n measurement sets a1,a2...an, then I end up with a stack of 3*n equations:

f1(a1,b) = 0
f2(a1,b) = 0
f3(a1,b) = 0
f1(a2,b) = 0
f2(a2,b) = 0
f3(a2,b) = 0
.
.
.
f1(an,b) = 0
f2(an,b) = 0
f3(an,b) = 0

How should I go about finding the parameters b using the LVM? I think your second suggestion applies here - sum the squares of the function and minimise the result. So if a f1(a1,b) returns 0.5 for a given set of parameters b, then the residual is 0.5. So should I sum the squares of the result of each of the functions in my stack of 3n functions and minimise that?
 
  • #4
Yep, you should sum the squares of the result of each of the functions in your stack of 3n functions and minimise that.

I have a few additional comments.

What do you know about the numbers your functions return?
Are they "comparable"?
That is, are they more or less the same size?

If one function returns results that are much larger or less reliable than another function, you may need to "weigh" the results, but you can only do that if you know something about the variations in results.

Furthermore, I would divide the squared total by 3n giving you effectively a normalized variance.
This makes it possible to compare the results of different sets of measurements.
 
Last edited:
  • #5
Awesome, now I can get coding and test it out.

So should I use (f1)^2+(f2)^2+f(3)^2 as the function for the Jacobian matrix, or just f1+f2+f3? I think the latter because the derivatives in the Jacobian are how the optimisation works, right?

Thanks so much :-D
 
  • #6
thomas430 said:
Awesome, now I can get coding and test it out.

So should I use (f1)^2+(f2)^2+f(3)^2 as the function for the Jacobian matrix, or just f1+f2+f3? I think the latter because the derivatives in the Jacobian are how the optimisation works, right?

Thanks so much :-D

I'm afraid it's a little more complex.

The function that you are minimizing is:
g(x) = f1(a1,x)2 + f2(a1,x)2 + f3(a1,x)2 + f1(a2,x)2 + f2(a2,x)2 + f3(a2,x)2 + ...

So the jacobian is:
Dg(x) = 2f1(a1,x)Df1(a1,x) + 2f2(a1,x)Df2(a1,x) + 2f3(a1,x)Df3(a1,x) + 2f1(a2,x)Df1(a1,x) + 2f2(a2,x)Df2(a1,x) + 2f3(a2,x)Df3(a1,x) + ...
 
  • #7
Oh, I see! So supposing x represents 2 parameters q and w, I should end up with a 1x2 Jacobian matrix like:

[f1(a1,x)2/dq + f2(a1,x)2/dq + f3(a1,x)2/dq + f1(a2,x)2/dq + ... f3(an,x)2/dq | | f1(a1,x)2/dw + f2(a1,x)2/dw + f3(a1,x)2/dw + f1(a2,x)2/dw + ... f3(an,x)2/dw]
 
  • #8
Yes! That is, assuming that by f1(a1,x)2/dq, you actually mean d/dq (f1(a1;q,w)2).
 
  • #9
Perfect, thanks I like Serena. It's working very nicely! :-D
 
  • #10
I have equation , p = [a b c]' : vector of optimized parameters
Y = a*(U(j) -b - 5*V(i))*(1+9*c)
where Y : data points 61. j = 1:61, i = 1:6...
I can't solution optimization for this equation, please help...
I just can do this equation if I don't have V(i) i.e V(i) = 0
 
  • #11
Please start a new thread for your problem.
 

What is the Levenberg-Marquardt Algorithm with Several Functions?

The Levenberg-Marquardt Algorithm with Several Functions is a numerical optimization method used to find the minimum of a function that is defined by several other functions. It is commonly used in scientific and engineering applications to fit a model to a set of data points.

How does the Levenberg-Marquardt Algorithm work?

The Levenberg-Marquardt Algorithm combines the steepest descent method and the Gauss-Newton method to iteratively update the parameters of the model in order to minimize the sum of squared errors between the predicted and actual values. It uses a damping parameter to balance between the two methods and prevent oscillations during the optimization process.

What are the advantages of using the Levenberg-Marquardt Algorithm?

The Levenberg-Marquardt Algorithm is a robust and efficient method for finding the minimum of a nonlinear function. It is able to handle noisy data and is less sensitive to initial parameter values compared to other optimization methods. It also provides a good estimate of the uncertainty in the optimized parameters.

What are some applications of the Levenberg-Marquardt Algorithm?

The Levenberg-Marquardt Algorithm is commonly used in data fitting and curve fitting applications in various fields such as physics, chemistry, biology, and engineering. It is also used in image processing, signal processing, and machine learning to optimize models and parameters.

Are there any limitations to the Levenberg-Marquardt Algorithm?

While the Levenberg-Marquardt Algorithm is a powerful tool for optimization, it may not always converge to the global minimum and may get stuck in a local minimum. It also requires the user to have some knowledge about the model and initial parameter values. Additionally, it may not be suitable for high-dimensional problems due to the increased computational cost.

Similar threads

Replies
16
Views
1K
  • General Math
Replies
13
Views
1K
  • Programming and Computer Science
Replies
2
Views
1K
  • Differential Equations
Replies
1
Views
2K
  • Programming and Computer Science
Replies
1
Views
2K
  • Engineering and Comp Sci Homework Help
Replies
3
Views
1K
  • Engineering and Comp Sci Homework Help
Replies
4
Views
988
Replies
2
Views
3K
Replies
1
Views
2K
Back
Top