Least Squares Regression

In summary: The sum of the y values is squared to make sure the opposite errors don't cancel. The chain rule is used to solve for the partial derivatives of the deviation function with respect to m and b. The two partial derivatives are then used to solve for b.
  • #1
ElijahRockers
Gold Member
270
10

Homework Statement



http://www.math.tamu.edu/~vargo/courses/251/HW7.pdf [Broken]

Given a set of points (xi,yi) and assuming f(xi) is linear, the deviation measured is F(m,b)=[itex]\sum_{i}^{n}(y_i - f(x_i))^2[/itex]. There are a few different questions about this from the link above.

The Attempt at a Solution



I'm not sure why the expression [itex]\sum_{i}^{n}(y_i - f(x_i))^2[/itex] is squared.

Part one says to find the partials of this expression with respect to m and b. Here's my take on it. [itex]y_i[/itex] comes from the set of points, and [itex]f(x_i)[/itex] would be equal to [itex]mx_i+b[/itex].

Using the chain rule,

[itex]\frac{\partial F}{\partial m} = \sum_i^n 2x_i(y_i-(mx_i+b))[/itex]

[itex]\frac{\partial F}{\partial b} = \sum_i^n 2(y_i-(mx_i+b))[/itex]

Are these correct?

I can kind of see where this question is going. We can use these derivatives to sort of find the 'best fit', I'm guessing where the change in deviation is zero, that is probably the minimum deviation.

Sooo, dF. Does he mean both of the partials? Either one? Or are my expressions wrong to begin with. I haven't got any real experience with the summing notation, but I took a stab at solving for b by setting dF/db = 0.

[itex]b=\frac{\sum_i^n 2(y_i - mx_i)}{2n}[/itex]

That was just kind of a shot in the dark. If someone could give me a push in the right direction I'd appreciate it! Thanks :)
 
Last edited by a moderator:
Physics news on Phys.org
  • #2
ElijahRockers said:
Using the chain rule,

[itex]\frac{\partial F}{\partial m} = \sum_i^n 2x_i(y_i-(mx_i+b))[/itex]

[itex]\frac{\partial F}{\partial b} = \sum_i^n 2(y_i-(mx_i+b))[/itex]

Are these correct?
They're both missing a negative sign, but otherwise, they're correct.

Sooo, dF. Does he mean both of the partials? Either one? Or are my expressions wrong to begin with.
Both partials need to be equal to 0.

I haven't got any real experience with the summing notation, but I took a stab at solving for b by setting dF/db = 0.

[itex]b=\frac{\sum_i^n 2(y_i - mx_i)}{2n}[/itex]

That was just kind of a shot in the dark. If someone could give me a push in the right direction I'd appreciate it! Thanks :)
You're trying to obtain expressions for m and b that depend only on the x and y values. Your results for b still has m in it. The two equations ##\partial F/\partial m = 0## and ##\partial F/\partial B = 0## can be written in the form
\begin{align*}
A m + B b &= E \\
C m + D b &= F
\end{align*} where A, B, C, D, E, and F are combinations of the x and y values. You can solve this system of equations to get the results you want.
 
  • #3
ElijahRockers said:

Homework Statement



http://www.math.tamu.edu/~vargo/courses/251/HW7.pdf [Broken]

Given a set of points (xi,yi) and assuming f(xi) is linear, the deviation measured is F(m,b)=[itex]\sum_{i}^{n}(y_i - f(x_i))^2[/itex]. There are a few different questions about this from the link above.

The Attempt at a Solution



I'm not sure why the expression [itex]\sum_{i}^{n}(y_i - f(x_i))^2[/itex] is squared.
It's not. It is the terms [itex]y_i- f(x_i)[/itex] inside the sum that are squared. One reason for that square is to make sure that errors on opposite sides, positive and negative, do not cancel. More fundamentally (there are many ways to avoid errors canceling) it mimics the formula for distance in n dimensions, [itex]\sqrt{\sum (x_i- y_i)^2}[/itex].

Part one says to find the partials of this expression with respect to m and b. Here's my take on it. [itex]y_i[/itex] comes from the set of points, and [itex]f(x_i)[/itex] would be equal to [itex]mx_i+b[/itex].

Using the chain rule,

[itex]\frac{\partial F}{\partial m} = \sum_i^n 2x_i(y_i-(mx_i+b))[/itex]

[itex]\frac{\partial F}{\partial b} = \sum_i^n 2(y_i-(mx_i+b))[/itex]

Are these correct?

I can kind of see where this question is going. We can use these derivatives to sort of find the 'best fit', I'm guessing where the change in deviation is zero, that is probably the minimum deviation.

Sooo, dF. Does he mean both of the partials? Either one? Or are my expressions wrong to begin with. I haven't got any real experience with the summing notation, but I took a stab at solving for b by setting dF/db = 0.

[itex]b=\frac{\sum_i^n 2(y_i - mx_i)}{2n}[/itex]

That was just kind of a shot in the dark. If someone could give me a push in the right direction I'd appreciate it! Thanks :)
 
Last edited by a moderator:
  • #4
vela said:
They're both missing a negative sign, but otherwise, they're correct.

Ahhh I see that now. Thanks. But really though the negative sign won't affect my partials, because they are constant right?

So after more careful consideration, here's a second stab at it...

[itex]\frac{\partial F}{\partial b} = 0: \sum_i^n (mx_i + b) = \sum_i^n y_i[/itex]
[itex]\frac{\partial F}{\partial m} = 0: \sum_i^n (mx_i^2 + bx_i) = \sum_i^n y_ix_i[/itex]

But wouldn't the extra x_i cancel out in the second equation? Then I'd be left with two identical equations and that wouldn't help me.

And thanks Ivy, that clears things up a great deal. :)
 
  • #5
The xi are different for each i, it's not a constant multiple for the whole equation. Here is a sample system of equations like what you have: suppose that x1=1, x2=2, y1 = 3, y2=4
1*m+b + 2*m+b = 3+ 4
(1*m+b)*1+2*(2*m+b)=3*1+4*2

Doing some algebra these equations are
3m+2b=7
5m+3b=11

Notice that these are not equivalent equations. Solving for m and b will in this case find you the line passing through the two points that are prescribed
 
  • #6
Office_Shredder said:
suppose that x1=1, x2=2, y1 = 3, y2=4
1*m+b + 2*m+b = 3+ 4
(1*m+b)*1+2*(2*m+b)=3*1+4*2

Isn't that basically the same thing I have, just with the sums expanded?

Ok I took another shot...

[itex]nb + (\sum_i^n x_i)m = \sum_i^n y_i[/itex]
[itex](n\sum_i^n x_i )b + (\sum_i^n x_i^2)m = \sum_i^n y_ix_i[/itex]

Is that right? If so, could I solve by substitution?

Thanks for all the help.
 
  • #7
Yes that's exactly what you need. Now solve it by your favorite method of solving 2 equations with 2 unknowns
 

What is Least Squares Regression?

Least Squares Regression is a statistical method used to find the line of best fit for a set of data points. This line minimizes the sum of the squared distances between the predicted values and the actual values of the data points.

How is Least Squares Regression different from other regression methods?

Unlike other regression methods, Least Squares Regression aims to minimize the vertical distance between the data points and the line of best fit, rather than the horizontal distance.

What are the assumptions of Least Squares Regression?

The main assumptions of Least Squares Regression are that the relationship between the dependent and independent variables is linear, the errors are normally distributed, and the errors have equal variance.

What is the purpose of using Least Squares Regression?

The purpose of using Least Squares Regression is to model and predict the relationship between two or more variables. It is commonly used in fields such as economics, engineering, and social sciences.

Can outliers affect the results of Least Squares Regression?

Yes, outliers can significantly affect the results of Least Squares Regression. This is because the method aims to minimize the sum of squared errors, and outliers can greatly increase this sum, leading to a less accurate line of best fit.

Similar threads

Replies
3
Views
1K
  • Calculus and Beyond Homework Help
Replies
2
Views
872
  • Calculus and Beyond Homework Help
Replies
26
Views
2K
  • Calculus and Beyond Homework Help
Replies
1
Views
1K
  • Calculus and Beyond Homework Help
Replies
5
Views
1K
Replies
5
Views
270
  • Calculus and Beyond Homework Help
Replies
3
Views
774
  • Calculus and Beyond Homework Help
Replies
8
Views
897
  • Set Theory, Logic, Probability, Statistics
Replies
1
Views
2K
Replies
1
Views
570
Back
Top