Levenberg-Marquardt Algorithm with Several Functions

[thomas430]

Hi there, I have been testing out the Levenberg-Marquardt algorithm. I've successfully coded a method in MATLAB for the example I saw on wikipedia:

f(x) = a*cos(bx)+b*sin(ax)

and this has worked well. The application I'm using the algorithm for is a system of 3 equations, however.

Does anyone have any ideas on how to implement the algorithm for multiple functions?

Thomas.

 

[I like Serena]

Hi thomas430!

Levenberg-Marquardt minimizes a function result.

If you have 3 equations, you need to consider what it is that you want to minimize.
If they are independent, you can call Levenberg-Marquardt 3 times separately.

If they are dependent, then one method is to rewrite each equation to a function that should be equal to zero.
Sum the squares of those functions and minimize the result.
Effectively you're doing a least squares algorithm.

 

[thomas430]

Thanks, I like Serena! Your reply helped a lot, but I'm still trying to get my head around how to relate it back to my problem.

My problem looks like this... three equations (I haven't put the actual ones because they're very long):

f1(a,b) = 0
f2(a,b) = 0
f3(a,b) = 0

where a are measurements, and b are parameters. But the system grows because I've made many sets of measurements... say I've made n measurement sets a₁,a₂...a_n, then I end up with a stack of 3*n equations:

f1(a₁,b) = 0
f2(a₁,b) = 0
f3(a₁,b) = 0
f1(a₂,b) = 0
f2(a₂,b) = 0
f3(a₂,b) = 0
.
.
.
f1(a_n,b) = 0
f2(a_n,b) = 0
f3(a_n,b) = 0

How should I go about finding the parameters b using the LVM? I think your second suggestion applies here - sum the squares of the function and minimise the result. So if a f1(a₁,b) returns 0.5 for a given set of parameters b, then the residual is 0.5. So should I sum the squares of the result of each of the functions in my stack of 3n functions and minimise that?

 

[I like Serena]

Yep, you should sum the squares of the result of each of the functions in your stack of 3n functions and minimise that.

I have a few additional comments.

What do you know about the numbers your functions return?
Are they "comparable"?
That is, are they more or less the same size?

If one function returns results that are much larger or less reliable than another function, you may need to "weigh" the results, but you can only do that if you know something about the variations in results.

Furthermore, I would divide the squared total by 3n giving you effectively a normalized variance.
This makes it possible to compare the results of different sets of measurements.

 

[thomas430]

Awesome, now I can get coding and test it out.

So should I use (f1)^2+(f2)^2+f(3)^2 as the function for the Jacobian matrix, or just f1+f2+f3? I think the latter because the derivatives in the Jacobian are how the optimisation works, right?

Thanks so much :-D

 

[I like Serena]

Awesome, now I can get coding and test it out.

So should I use (f1)^2+(f2)^2+f(3)^2 as the function for the Jacobian matrix, or just f1+f2+f3? I think the latter because the derivatives in the Jacobian are how the optimisation works, right?

Thanks so much :-D

I'm afraid it's a little more complex.

The function that you are minimizing is:
g(x) = f1(a1,x)² + f2(a1,x)² + f3(a1,x)² + f1(a2,x)² + f2(a2,x)² + f3(a2,x)² + ...

So the jacobian is:
Dg(x) = 2f1(a1,x)Df1(a1,x) + 2f2(a1,x)Df2(a1,x) + 2f3(a1,x)Df3(a1,x) + 2f1(a2,x)Df1(a1,x) + 2f2(a2,x)Df2(a1,x) + 2f3(a2,x)Df3(a1,x) + ...

 

[thomas430]

Oh, I see! So supposing x represents 2 parameters q and w, I should end up with a 1x2 Jacobian matrix like:

[f1(a1,x)²/dq + f2(a1,x)²/dq + f3(a1,x)²/dq + f1(a2,x)²/dq + ... f3(an,x)²/dq | | f1(a1,x)²/dw + f2(a1,x)²/dw + f3(a1,x)²/dw + f1(a2,x)²/dw + ... f3(an,x)²/dw]

 

[I like Serena]

Yes! That is, assuming that by f1(a1,x)²/dq, you actually mean d/dq (f1(a1;q,w)²).

 

[thomas430]

Perfect, thanks I like Serena. It's working very nicely! :-D

 

[chaiku]

I have equation , p = [a b c]' : vector of optimized parameters
Y = a*(U(j) -b - 5*V(i))*(1+9*c)
where Y : data points 61. j = 1:61, i = 1:6...
I can't solution optimization for this equation, please help...
I just can do this equation if I don't have V(i) i.e V(i) = 0

 

[micromass]

Please start a new thread for your problem.