Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Levenberg-Marquardt Algorithm with Several Functions

  1. Aug 15, 2011 #1
    Hi there, I have been testing out the Levenberg-Marquardt algorithm. I've successfully coded a method in MATLAB for the example I saw on wikipedia:

    f(x) = a*cos(bx)+b*sin(ax)

    and this has worked well. The application I'm using the algorithm for is a system of 3 equations, however.

    Does anyone have any ideas on how to implement the algorithm for multiple functions?

  2. jcsd
  3. Aug 15, 2011 #2

    I like Serena

    User Avatar
    Homework Helper

    Hi thomas430! :smile:

    Levenberg-Marquardt minimizes a function result.

    If you have 3 equations, you need to consider what it is that you want to minimize.
    If they are independent, you can call Levenberg-Marquardt 3 times separately.

    If they are dependent, then one method is to rewrite each equation to a function that should be equal to zero.
    Sum the squares of those functions and minimize the result.
    Effectively you're doing a least squares algorithm.
    Last edited: Aug 15, 2011
  4. Aug 15, 2011 #3
    Thanks, I like Serena! Your reply helped a lot, but I'm still trying to get my head around how to relate it back to my problem.

    My problem looks like this... three equations (I haven't put the actual ones because they're very long):

    f1(a,b) = 0
    f2(a,b) = 0
    f3(a,b) = 0

    where a are measurements, and b are parameters. But the system grows because I've made many sets of measurements... say I've made n measurement sets a1,a2...an, then I end up with a stack of 3*n equations:

    f1(a1,b) = 0
    f2(a1,b) = 0
    f3(a1,b) = 0
    f1(a2,b) = 0
    f2(a2,b) = 0
    f3(a2,b) = 0
    f1(an,b) = 0
    f2(an,b) = 0
    f3(an,b) = 0

    How should I go about finding the parameters b using the LVM? I think your second suggestion applies here - sum the squares of the function and minimise the result. So if a f1(a1,b) returns 0.5 for a given set of parameters b, then the residual is 0.5. So should I sum the squares of the result of each of the functions in my stack of 3n functions and minimise that?
  5. Aug 15, 2011 #4

    I like Serena

    User Avatar
    Homework Helper

    Yep, you should sum the squares of the result of each of the functions in your stack of 3n functions and minimise that.

    I have a few additional comments.

    What do you know about the numbers your functions return?
    Are they "comparable"?
    That is, are they more or less the same size?

    If one function returns results that are much larger or less reliable than another function, you may need to "weigh" the results, but you can only do that if you know something about the variations in results.

    Furthermore, I would divide the squared total by 3n giving you effectively a normalized variance.
    This makes it possible to compare the results of different sets of measurements.
    Last edited: Aug 15, 2011
  6. Aug 15, 2011 #5
    Awesome, now I can get coding and test it out.

    So should I use (f1)^2+(f2)^2+f(3)^2 as the function for the Jacobian matrix, or just f1+f2+f3? I think the latter because the derivatives in the Jacobian are how the optimisation works, right?

    Thanks so much :-D
  7. Aug 15, 2011 #6

    I like Serena

    User Avatar
    Homework Helper

    I'm afraid it's a little more complex.

    The function that you are minimizing is:
    g(x) = f1(a1,x)2 + f2(a1,x)2 + f3(a1,x)2 + f1(a2,x)2 + f2(a2,x)2 + f3(a2,x)2 + ...

    So the jacobian is:
    Dg(x) = 2f1(a1,x)Df1(a1,x) + 2f2(a1,x)Df2(a1,x) + 2f3(a1,x)Df3(a1,x) + 2f1(a2,x)Df1(a1,x) + 2f2(a2,x)Df2(a1,x) + 2f3(a2,x)Df3(a1,x) + ...
  8. Aug 15, 2011 #7
    Oh, I see! So supposing x represents 2 parameters q and w, I should end up with a 1x2 Jacobian matrix like:

    [f1(a1,x)2/dq + f2(a1,x)2/dq + f3(a1,x)2/dq + f1(a2,x)2/dq + ... f3(an,x)2/dq | | f1(a1,x)2/dw + f2(a1,x)2/dw + f3(a1,x)2/dw + f1(a2,x)2/dw + ... f3(an,x)2/dw]
  9. Aug 15, 2011 #8

    I like Serena

    User Avatar
    Homework Helper

    Yes! That is, assuming that by f1(a1,x)2/dq, you actually mean d/dq (f1(a1;q,w)2).
  10. Aug 15, 2011 #9
    Perfect, thanks I like Serena. It's working very nicely! :-D
  11. Jun 25, 2012 #10
    I have equation , p = [a b c]' : vector of optimized parameters
    Y = a*(U(j) -b - 5*V(i))*(1+9*c)
    where Y : data points 61. j = 1:61, i = 1:6...
    I can't solution optimization for this equation, please help...
    I just can do this equation if I don't have V(i) i.e V(i) = 0
  12. Jun 25, 2012 #11
    Please start a new thread for your problem.
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook