Levenberg-Marquardt Algorithm with Several Functions

Click For Summary

Discussion Overview

The discussion revolves around the implementation of the Levenberg-Marquardt algorithm for optimizing a system of multiple functions, specifically in the context of fitting parameters to a set of equations derived from measurements. Participants explore how to adapt the algorithm for a stack of equations and address challenges related to minimizing the sum of squares of residuals.

Discussion Character

  • Exploratory, Technical explanation, Debate/contested, Mathematical reasoning

Main Points Raised

  • One participant successfully implemented the Levenberg-Marquardt algorithm for a single function and seeks guidance on applying it to multiple equations.
  • Another participant suggests that if the equations are independent, the algorithm can be called separately for each, while for dependent equations, they propose summing the squares of the functions to minimize the result.
  • A participant outlines their specific problem involving multiple sets of equations and seeks clarification on whether to sum the squares of the residuals for optimization.
  • There is a discussion about the comparability of the outputs from different functions and the potential need for weighting based on their reliability.
  • Participants discuss the formulation of the Jacobian matrix, with one suggesting that the derivatives should be based on the squared residuals.
  • Clarification is provided on the correct formulation of the Jacobian matrix in relation to the parameters being optimized.
  • Another participant introduces a new optimization problem involving a different equation and requests assistance.

Areas of Agreement / Disagreement

Participants generally agree on the approach of summing the squares of the functions for optimization, but there are nuances regarding the formulation of the Jacobian matrix and the handling of different outputs from the functions. The discussion remains unresolved regarding the new optimization problem introduced.

Contextual Notes

Participants express uncertainty about the comparability of function outputs and the implications for weighting results. There are also unresolved details regarding the specific equations and parameters involved in the optimization process.

thomas430
Messages
16
Reaction score
0
Hi there, I have been testing out the Levenberg-Marquardt algorithm. I've successfully coded a method in MATLAB for the example I saw on wikipedia:

f(x) = a*cos(bx)+b*sin(ax)

and this has worked well. The application I'm using the algorithm for is a system of 3 equations, however.

Does anyone have any ideas on how to implement the algorithm for multiple functions?

Thomas.
 
Physics news on Phys.org
Hi thomas430! :smile:

Levenberg-Marquardt minimizes a function result.

If you have 3 equations, you need to consider what it is that you want to minimize.
If they are independent, you can call Levenberg-Marquardt 3 times separately.

If they are dependent, then one method is to rewrite each equation to a function that should be equal to zero.
Sum the squares of those functions and minimize the result.
Effectively you're doing a least squares algorithm.
 
Last edited:
Thanks, I like Serena! Your reply helped a lot, but I'm still trying to get my head around how to relate it back to my problem.

My problem looks like this... three equations (I haven't put the actual ones because they're very long):

f1(a,b) = 0
f2(a,b) = 0
f3(a,b) = 0

where a are measurements, and b are parameters. But the system grows because I've made many sets of measurements... say I've made n measurement sets a1,a2...an, then I end up with a stack of 3*n equations:

f1(a1,b) = 0
f2(a1,b) = 0
f3(a1,b) = 0
f1(a2,b) = 0
f2(a2,b) = 0
f3(a2,b) = 0
.
.
.
f1(an,b) = 0
f2(an,b) = 0
f3(an,b) = 0

How should I go about finding the parameters b using the LVM? I think your second suggestion applies here - sum the squares of the function and minimise the result. So if a f1(a1,b) returns 0.5 for a given set of parameters b, then the residual is 0.5. So should I sum the squares of the result of each of the functions in my stack of 3n functions and minimise that?
 
Yep, you should sum the squares of the result of each of the functions in your stack of 3n functions and minimise that.

I have a few additional comments.

What do you know about the numbers your functions return?
Are they "comparable"?
That is, are they more or less the same size?

If one function returns results that are much larger or less reliable than another function, you may need to "weigh" the results, but you can only do that if you know something about the variations in results.

Furthermore, I would divide the squared total by 3n giving you effectively a normalized variance.
This makes it possible to compare the results of different sets of measurements.
 
Last edited:
Awesome, now I can get coding and test it out.

So should I use (f1)^2+(f2)^2+f(3)^2 as the function for the Jacobian matrix, or just f1+f2+f3? I think the latter because the derivatives in the Jacobian are how the optimisation works, right?

Thanks so much :-D
 
thomas430 said:
Awesome, now I can get coding and test it out.

So should I use (f1)^2+(f2)^2+f(3)^2 as the function for the Jacobian matrix, or just f1+f2+f3? I think the latter because the derivatives in the Jacobian are how the optimisation works, right?

Thanks so much :-D

I'm afraid it's a little more complex.

The function that you are minimizing is:
g(x) = f1(a1,x)2 + f2(a1,x)2 + f3(a1,x)2 + f1(a2,x)2 + f2(a2,x)2 + f3(a2,x)2 + ...

So the jacobian is:
Dg(x) = 2f1(a1,x)Df1(a1,x) + 2f2(a1,x)Df2(a1,x) + 2f3(a1,x)Df3(a1,x) + 2f1(a2,x)Df1(a1,x) + 2f2(a2,x)Df2(a1,x) + 2f3(a2,x)Df3(a1,x) + ...
 
Oh, I see! So supposing x represents 2 parameters q and w, I should end up with a 1x2 Jacobian matrix like:

[f1(a1,x)2/dq + f2(a1,x)2/dq + f3(a1,x)2/dq + f1(a2,x)2/dq + ... f3(an,x)2/dq | | f1(a1,x)2/dw + f2(a1,x)2/dw + f3(a1,x)2/dw + f1(a2,x)2/dw + ... f3(an,x)2/dw]
 
Yes! That is, assuming that by f1(a1,x)2/dq, you actually mean d/dq (f1(a1;q,w)2).
 
Perfect, thanks I like Serena. It's working very nicely! :-D
 
  • #10
I have equation , p = [a b c]' : vector of optimized parameters
Y = a*(U(j) -b - 5*V(i))*(1+9*c)
where Y : data points 61. j = 1:61, i = 1:6...
I can't solution optimization for this equation, please help...
I just can do this equation if I don't have V(i) i.e V(i) = 0
 
  • #11
Please start a new thread for your problem.
 

Similar threads

MATLAB Frequency domain filtering in Matlab
  • · Replies 5 ·
Replies
5
Views
6K
Graduate Is Levenberg-Marquardt the Correct Method for Fitting Young's Equation?
  • · Replies 5 ·
Replies
5
Views
5K
Identifying and understanding random number algorithms
  • · Replies 2 ·
Replies
2
Views
2K
MATLAB Replacing the "quad" function with "integral" doesn't work in Matlab
  • · Replies 1 ·
Replies
1
Views
3K
Graduate Fit with least squares, Levenberg–Marquardt algorithm
  • · Replies 2 ·
Replies
2
Views
3K
High School Are Inverse Problems More Complex Than Direct Problems in Mathematics?
  • · Replies 13 ·
Replies
13
Views
3K
Global fitting with shared parameters
  • · Replies 1 ·
Replies
1
Views
3K
How to use functions defined in an external header-only library?
  • · Replies 3 ·
Replies
3
Views
2K
What is the CORDIC Algorithm and How Does it Work?
  • · Replies 1 ·
Replies
1
Views
4K
Fast pentadiagonal matrix solver
  • · Replies 1 ·
Replies
1
Views
4K