Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Least Squares Derivation question

  1. Sep 20, 2011 #1
    So I am learning how to use the method of least squares to find the best fit straight line and there is this one part in the derivation I do not fully understand and I was wondering if anyone could help me out.

    So basically we start out with the residuals of the equation of a straight line..

    And now we take the root mean square of these residuals and try to find the minimum points of m and c by taking the partial derivatives.

    [tex]\Sigma (y_i-mx_i-c)^2[/tex]
    [tex]\frac{\partial S}{\partial m}=-2\Sigma x_i(y_i-mx_i-c)=0[/tex]
    [tex]\frac{\partial S}{\partial c}=-2\Sigma (y_i-mx_i-c)=0[/tex]

    Now from the second equation it is easy to see that
    [tex]\overline{y}=\frac{1}{n}\Sigma y_i[/tex]
    [tex]\overline{x}=\frac{1}{n}\Sigma x_i[/tex]

    Solving the first equation for m we get..
    [tex]m=\frac{\Sigma x_i y_i -c\Sigma x_i}{\Sigma x_i ^2}[/tex]

    The part I do not understand is the book says that m is equal to..

    [tex]m=\frac{\Sigma (x_i-\overline{x})y_i}{\Sigma (x_i-\overline{x})^2}[/tex]
    I feel like I must be missing something simple due to the books lack of explaination. Can anyone help me get this formula from the one I got for m? Why did the [itex]x_i[/itex]'s turn into [itex](x_i-\overline{x})[/itex]'s?? Did they made c=0 for some reason? Any help would be appreciated!
  2. jcsd
  3. Sep 20, 2011 #2

    Stephen Tashi

    User Avatar
    Science Advisor

    In the first equation, if you replace [itex] c [/itex] by [itex] \overline{y} - m \overline{x} [/itex] then you get m's on both sides and you have to solve for m again. Maybe that's how to do it.
  4. Sep 20, 2011 #3
    Not quite right - looks like they assumed a model with zero y-mean. Another way of looking at this is a model, in x about x-mean, through the origin. With this later terminology, you only have one parameter, the slope (m), to estimate

    [tex] y_i = m \, (x_i-\bar{x})[/tex]

    Hence, setting the derivative of S(m) to zero yields

    [tex] \sum y \, (x_i-\bar{x}) = m \cdot \sum (x_i-\bar{x})^2[/tex]

    Solve for m to yield text answer
  5. Sep 20, 2011 #4

    Stephen Tashi

    User Avatar
    Science Advisor

    [tex]m=\frac{\Sigma x_i y_i -c\Sigma x_i}{\Sigma x_i ^2}[/tex]
    [tex] m = \frac{\Sigma x_i y_i - (\overline y - m \overline x) \Sigma x_i}{\Sigma x_i ^2}[/tex]
    [tex] m = \frac{ \Sigma x_i y_i -\overline y \Sigma x_i + m \overline x \Sigma x_i}{\Sigma x_i^2} [/tex]
    [tex] m \Sigma x_i^2 - m \overline x \Sigma x_i = \Sigma x_i y_i - \overline y \Sigma x_i [/tex]
    [tex] m= \frac{ \Sigma x_i y_i - \overline y \Sigma x_i}{\Sigma x_i^2 - \overline x \Sigma x_i} [/tex]

    [tex] = \frac{ \Sigma x_i y_i - \frac{\Sigma y_i}{N} \Sigma x_i}{\Sigma x_i^2 - (\frac{\Sigma x_i}{N}) \Sigma x_i } [/tex]

    The numerator is equal to [itex] \Sigma x_i y_i - \Sigma y_i (\frac{\Sigma x_i}{N}) [/itex]
    [tex] = \Sigma x_i y_i - \Sigma y_i \overline x [/tex]
    [tex] = \Sigma (x_i - \overline x) y_i [/tex]

    To deal with the denominator, consider an estimator for the sample variance:

    [tex] \sigma^2 = \frac { \Sigma (x_i - \overline x)^2}{N} [/tex]
    [tex] = \frac{ \Sigma ( x_i^2 - 2 x_i \overline x + \overline x \overline x)}{N} [/tex]
    [tex] = \frac{ \Sigma x_i^2 - 2 \overline x \Sigma x_i + \Sigma \overline x \overline x }{N} [/tex]
    [tex] = \frac {\Sigma x_i^2}{N} - 2 \overline x \frac{\Sigma x_i}{N} + \frac{ N \overline x \overline x}{N} [/tex]
    [tex] = \frac {\Sigma x_i^2}{N} - 2 \overline x \overline x + \overline x \overline x [/tex]
    [tex] = \frac{ \Sigma x_i^2}{N} - \overline x \overline x [/tex]

    This establishes that
    [tex] \frac {\Sigma (x_i - \overline x)^2}{N} = \frac{\Sigma x_i^2}{N} - \overline x \overline x [/tex]

    [tex] \Sigma (x_i - \overline x)^2 = \Sigma x_i^2 - N \overline x \overline x [/tex]
    [tex] = \Sigma x_i^2 - N ( \frac{\Sigma x_i}{N} \frac{\Sigma x_i}{N}) [/tex]
    [tex] = \Sigma x_i^2 - \frac{\Sigma x_i \Sigma x_i}{N} [/tex]
  6. Sep 22, 2011 #5
    The only thing that confuses me Stephen Tashi, is line 4 and 5. Where do you get those relations and how does "m" turn into the expression in line 5? Otherwise, everything after it is fine and I understand completely. :]
  7. Sep 22, 2011 #6

    Stephen Tashi

    User Avatar
    Science Advisor

    Multiply both sides of the equation by [itex] \Sigma x_i^2 [/itex] and then subtract [itex] m\overline x \Sigma x_i [/itex] from both sides.
    [tex] m( \Sigma x_i^2 - \overline x \Sigma x_i) = \Sigma x_i y_i - \overline y \Sigma x_i [/tex]

    Divide both sides by [itex] \Sigma x_i^2 - \overline x \Sigma x_i [/itex]

    Last edited: Sep 23, 2011
  8. Sep 25, 2011 #7
    Thank you very much!!
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook