1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Least Squares Regression

  1. Apr 7, 2012 #1

    ElijahRockers

    User Avatar
    Gold Member

    1. The problem statement, all variables and given/known data

    http://www.math.tamu.edu/~vargo/courses/251/HW7.pdf [Broken]

    Given a set of points (xi,yi) and assuming f(xi) is linear, the deviation measured is F(m,b)=[itex]\sum_{i}^{n}(y_i - f(x_i))^2[/itex]. There are a few different questions about this from the link above.

    3. The attempt at a solution

    I'm not sure why the expression [itex]\sum_{i}^{n}(y_i - f(x_i))^2[/itex] is squared.

    Part one says to find the partials of this expression with respect to m and b. Here's my take on it. [itex]y_i[/itex] comes from the set of points, and [itex]f(x_i)[/itex] would be equal to [itex]mx_i+b[/itex].

    Using the chain rule,

    [itex]\frac{\partial F}{\partial m} = \sum_i^n 2x_i(y_i-(mx_i+b))[/itex]

    [itex]\frac{\partial F}{\partial b} = \sum_i^n 2(y_i-(mx_i+b))[/itex]

    Are these correct?

    I can kind of see where this question is going. We can use these derivatives to sort of find the 'best fit', I'm guessing where the change in deviation is zero, that is probably the minimum deviation.

    Sooo, dF. Does he mean both of the partials? Either one? Or are my expressions wrong to begin with. I haven't got any real experience with the summing notation, but I took a stab at solving for b by setting dF/db = 0.

    [itex]b=\frac{\sum_i^n 2(y_i - mx_i)}{2n}[/itex]

    That was just kind of a shot in the dark. If someone could give me a push in the right direction I'd appreciate it! Thanks :)
     
    Last edited by a moderator: May 5, 2017
  2. jcsd
  3. Apr 8, 2012 #2

    vela

    User Avatar
    Staff Emeritus
    Science Advisor
    Homework Helper
    Education Advisor

    They're both missing a negative sign, but otherwise, they're correct.

    Both partials need to be equal to 0.

    You're trying to obtain expressions for m and b that depend only on the x and y values. Your results for b still has m in it. The two equations ##\partial F/\partial m = 0## and ##\partial F/\partial B = 0## can be written in the form
    \begin{align*}
    A m + B b &= E \\
    C m + D b &= F
    \end{align*} where A, B, C, D, E, and F are combinations of the x and y values. You can solve this system of equations to get the results you want.
     
  4. Apr 8, 2012 #3

    HallsofIvy

    User Avatar
    Staff Emeritus
    Science Advisor

    It's not. It is the terms [itex]y_i- f(x_i)[/itex] inside the sum that are squared. One reason for that square is to make sure that errors on opposite sides, positive and negative, do not cancel. More fundamentally (there are many ways to avoid errors canceling) it mimics the formula for distance in n dimensions, [itex]\sqrt{\sum (x_i- y_i)^2}[/itex].

     
    Last edited by a moderator: May 5, 2017
  5. Apr 8, 2012 #4

    ElijahRockers

    User Avatar
    Gold Member

    Ahhh I see that now. Thanks. But really though the negative sign won't affect my partials, because they are constant right?

    So after more careful consideration, here's a second stab at it....

    [itex]\frac{\partial F}{\partial b} = 0: \sum_i^n (mx_i + b) = \sum_i^n y_i[/itex]
    [itex]\frac{\partial F}{\partial m} = 0: \sum_i^n (mx_i^2 + bx_i) = \sum_i^n y_ix_i[/itex]

    But wouldn't the extra x_i cancel out in the second equation? Then I'd be left with two identical equations and that wouldn't help me.

    And thanks Ivy, that clears things up a great deal. :)
     
  6. Apr 8, 2012 #5

    Office_Shredder

    User Avatar
    Staff Emeritus
    Science Advisor
    Gold Member

    The xi are different for each i, it's not a constant multiple for the whole equation. Here is a sample system of equations like what you have: suppose that x1=1, x2=2, y1 = 3, y2=4
    1*m+b + 2*m+b = 3+ 4
    (1*m+b)*1+2*(2*m+b)=3*1+4*2

    Doing some algebra these equations are
    3m+2b=7
    5m+3b=11

    Notice that these are not equivalent equations. Solving for m and b will in this case find you the line passing through the two points that are prescribed
     
  7. Apr 8, 2012 #6

    ElijahRockers

    User Avatar
    Gold Member

    Isn't that basically the same thing I have, just with the sums expanded?

    Ok I took another shot...

    [itex]nb + (\sum_i^n x_i)m = \sum_i^n y_i[/itex]
    [itex](n\sum_i^n x_i )b + (\sum_i^n x_i^2)m = \sum_i^n y_ix_i[/itex]

    Is that right? If so, could I solve by substitution?

    Thanks for all the help.
     
  8. Apr 8, 2012 #7

    Office_Shredder

    User Avatar
    Staff Emeritus
    Science Advisor
    Gold Member

    Yes that's exactly what you need. Now solve it by your favorite method of solving 2 equations with 2 unknowns
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook