Least Squares Derivation question

[Xyius]

So I am learning how to use the method of least squares to find the best fit straight line and there is this one part in the derivation I do not fully understand and I was wondering if anyone could help me out.

So basically we start out with the residuals of the equation of a straight line..

y_i-mx_i-c
And now we take the root mean square of these residuals and try to find the minimum points of m and c by taking the partial derivatives.

\Sigma (y_i-mx_i-c)^2
\frac{\partial S}{\partial m}=-2\Sigma x_i(y_i-mx_i-c)=0
\frac{\partial S}{\partial c}=-2\Sigma (y_i-mx_i-c)=0

Now from the second equation it is easy to see that
c=\overline{y}-m\overline{x}
since
\overline{y}=\frac{1}{n}\Sigma y_i
and
\overline{x}=\frac{1}{n}\Sigma x_i

Solving the first equation for m we get..
m=\frac{\Sigma x_i y_i -c\Sigma x_i}{\Sigma x_i ^2}

The part I do not understand is the book says that m is equal to..

m=\frac{\Sigma (x_i-\overline{x})y_i}{\Sigma (x_i-\overline{x})^2}
I feel like I must be missing something simple due to the books lack of explanation. Can anyone help me get this formula from the one I got for m? Why did the x_i's turn into (x_i-\overline{x})'s?? Did they made c=0 for some reason? Any help would be appreciated!

 

[Stephen Tashi]

m=\frac{\Sigma x_i y_i -c\Sigma x_i}{\Sigma x_i ^2}

The part I do not understand is the book says that m is equal to..
m=\frac{\Sigma (x_i-\overline{x})y_i}{\Sigma (x_i-\overline{x})^2}

In the first equation, if you replace c by \overline{y} - m \overline{x} then you get m's on both sides and you have to solve for m again. Maybe that's how to do it.

 

[TheoMcCloskey]

Not quite right - looks like they assumed a model with zero y-mean. Another way of looking at this is a model, in x about x-mean, through the origin. With this later terminology, you only have one parameter, the slope (m), to estimate

y_i = m \, (x_i-\bar{x})

Hence, setting the derivative of S(m) to zero yields

\sum y \, (x_i-\bar{x}) = m \cdot \sum (x_i-\bar{x})^2

Solve for m to yield text answer

 

[Stephen Tashi]

m=\frac{\Sigma x_i y_i -c\Sigma x_i}{\Sigma x_i ^2}
m = \frac{\Sigma x_i y_i - (\overline y - m \overline x) \Sigma x_i}{\Sigma x_i ^2}
m = \frac{ \Sigma x_i y_i -\overline y \Sigma x_i + m \overline x \Sigma x_i}{\Sigma x_i^2}
m \Sigma x_i^2 - m \overline x \Sigma x_i = \Sigma x_i y_i - \overline y \Sigma x_i
m= \frac{ \Sigma x_i y_i - \overline y \Sigma x_i}{\Sigma x_i^2 - \overline x \Sigma x_i}

= \frac{ \Sigma x_i y_i - \frac{\Sigma y_i}{N} \Sigma x_i}{\Sigma x_i^2 - (\frac{\Sigma x_i}{N}) \Sigma x_i }

The numerator is equal to \Sigma x_i y_i - \Sigma y_i (\frac{\Sigma x_i}{N})
= \Sigma x_i y_i - \Sigma y_i \overline x
= \Sigma (x_i - \overline x) y_i

To deal with the denominator, consider an estimator for the sample variance:

\sigma^2 = \frac { \Sigma (x_i - \overline x)^2}{N}
= \frac{ \Sigma ( x_i^2 - 2 x_i \overline x + \overline x \overline x)}{N}
= \frac{ \Sigma x_i^2 - 2 \overline x \Sigma x_i + \Sigma \overline x \overline x }{N}
= \frac {\Sigma x_i^2}{N} - 2 \overline x \frac{\Sigma x_i}{N} + \frac{ N \overline x \overline x}{N}
= \frac {\Sigma x_i^2}{N} - 2 \overline x \overline x + \overline x \overline x
= \frac{ \Sigma x_i^2}{N} - \overline x \overline x

This establishes that
\frac {\Sigma (x_i - \overline x)^2}{N} = \frac{\Sigma x_i^2}{N} - \overline x \overline x

So
\Sigma (x_i - \overline x)^2 = \Sigma x_i^2 - N \overline x \overline x
= \Sigma x_i^2 - N ( \frac{\Sigma x_i}{N} \frac{\Sigma x_i}{N})
= \Sigma x_i^2 - \frac{\Sigma x_i \Sigma x_i}{N}

 

[Xyius]

The only thing that confuses me Stephen Tashi, is line 4 and 5. Where do you get those relations and how does "m" turn into the expression in line 5? Otherwise, everything after it is fine and I understand completely. :]

 

[Stephen Tashi]

m = \frac{ \Sigma x_i y_i -\overline y \Sigma x_i + m \overline x \Sigma x_i}{\Sigma x_i^2}

Multiply both sides of the equation by \Sigma x_i^2 and then subtract m\overline x \Sigma x_i from both sides.

m \Sigma x_i^2 - m \overline x \Sigma x_i = \Sigma x_i y_i - \overline y \Sigma x_i

m( \Sigma x_i^2 - \overline x \Sigma x_i) = \Sigma x_i y_i - \overline y \Sigma x_i

Divide both sides by \Sigma x_i^2 - \overline x \Sigma x_i

m= \frac{ \Sigma x_i y_i - \overline y \Sigma x_i}{\Sigma x_i^2 - \overline x \Sigma x_i}

 

[Xyius]

Thank you very much!