• Support PF! Buy your school textbooks, materials and every day products Here!

Least squares fitting

  • Thread starter fluidistic
  • Start date
  • #1
fluidistic
Gold Member
3,649
99

Homework Statement


I must find the best fitting function of the form ax²+bx+c using least squares.
The points are (-1,6.1), (0,2.8), (1,2.2), (3,6) and (6,26.9).

2. Homework Equations + attempt at a solution
[tex]A\vec x= \vec b[/tex], I'm looking for [tex]\vec x =\begin {pmatrix} a \\ b \\ c \end {pmatrix}[/tex]. I know that [tex]\vec b = \begin {pmatrix} 6.1 \\ 2.8 \\ 2.2 \\ 6 \\ 26.9 \end {pmatrix}[/tex].
With the use of some theory, the system I want to solve is equivalent to [tex]A^{T}A \vec x = A^{T} \vec b[/tex].
My matrix A is [tex]\begin {bmatrix} 1 & -1 & 1 \\ 0 & 0 & 1 \\ 1 & 1 & 1 \\ 9 & 3 & 1 \\ 36 & 6 & 1 \end {bmatrix}[/tex].
[tex]A^{T}A=\begin {bmatrix} 1379 & 243 & 47 \\ 243 & 47 & 9 \\ 47 & 9 & 5 \end {bmatrix}[/tex].
And [tex]A^{T}\vec b = \begin {bmatrix} 1030.7 \\ 175.5 \\ 44 \end {bmatrix}[/tex].
This gave me [tex]\vec x = \begin {bmatrix} 2.669598408 \\ -9.505782817 \\ 0.6247662773 \end {bmatrix}[/tex]. Thus [tex]f(x) \approx 2.67 x^2-9.51 x +0.62[/tex].
I've plotted it in my calculator and it doesn't seem to fit well at all. My friend put these points in a program and he got a much better fitting ([tex]a \approx 3[/tex], [tex]b \approx -2[/tex], [tex]c \approx 1[/tex]).
I've redone the algebra and I don't see my error(s).
I've followed the method of this page: http://tutorial.math.lamar.edu/Classes/LinAlg/LeastSquares.aspx.
I got right all problems involving a linear fit, but failed all other fits (exponentials, quadratic, etc.). What am I doing wrong?!!

Another question that really matters to me is... why don't we solve Ax=b directly since we know both A and b, we would get 3 equations with 3 unknowns... why bother with the transpose of A?

Thank you very much for any help.
 

Answers and Replies

  • #2
lanedance
Homework Helper
3,304
2
the reason you use A^T.A is written in theorem 2

Theorem 2 Suppose that A is an matrix with linearly independent columns. Then, is an invertible matrix.
 
  • #3
lanedance
Homework Helper
3,304
2
now everything on that page is to do with simple linear squares regression

say your first point is
(x1, b1)

when you do quadratic regression, in effect you consider the triplet
(x1, y1, b1) where y1 = x1^2

then you regress on the parameters a,b,c in the fit equation
ay+bx+c = ax^2+bx+c

note the equation is linear in a,b,c, hence still linear regression (y could easily be another functional form eg. exponential y = e^x)

as there are now in effect 2 variables (x & y=x^2) you need to consider multivariate linear regression

hopefully this helps, my tex isn't displaying correctly so i can't properly QC what you've written
 
  • #4
fluidistic
Gold Member
3,649
99
the reason you use A^T.A is written in theorem 2
Thanks a lot.. My bad, I said something wrong :/
I said I'd get 3 equations with 3 unknowns when A isn't even square. This would do in fact 4 equations with 3 unknowns.
Question solved. Now I'm waiting for help on the first part.
 
  • #5
fluidistic
Gold Member
3,649
99
Ok: Latex problem; I'll solve that for you. :D
 

Attachments

  • #6
lanedance
Homework Helper
3,304
2
Thanks a lot.. My bad, I said something wrong :/
I said I'd get 3 equations with 3 unknowns when A isn't even square. This would do in fact 4 equations with 3 unknowns.
Question solved. Now I'm waiting for help on the first part.
the other thing to note is even if it was 3 and 3, there is not guaranteed a consistent unique solution. Hence least squares finds the solution with minimum squre error
 
  • #7
lanedance
Homework Helper
3,304
2
cheers for photos

so how do you get your final x?

shouldn't it be
(A^T.A)^(-1).(A^T.b)

if i do that i get different values to you...
 
  • #8
lanedance
Homework Helper
3,304
2
i then get
0.99
-1.99
3.03

also be careful to check the order is correct (ie corresponding a,b,c to x^2,x,1)
 
  • #9
fluidistic
Gold Member
3,649
99
Oh thanks a lot. You get the same values as the program of my friend...
I did get my x as follows: write down [itex]A^TA[/itex] and amplify this matrix with the column [tex]\vec b[/tex]. Then reduce [itex]A^TA[/itex] into a row echelon form and look at what happened in the augmented part of the matrix.
So in a sense, I didn't even calculated the inverse of [itex]A^TA[/itex], which might be where my error lies?
 

Attachments

  • #10
lanedance
Homework Helper
3,304
2
Oh thanks a lot. You get the same values as the program of my friend...
I did get my x as follows: write down [itex]A^TA[/itex] and amplify this matrix with the column [tex]\vec b[/tex]. Then reduce [itex]A^TA[/itex] into a row echelon form and look at what happened in the augmented part of the matrix.
So in a sense, I didn't even calculated the inverse of [itex]A^TA[/itex], which might be where my error lies?
should lead to the same result, maybe you're missing the A^T on the right side?
[itex]A.x = b [/itex]
[itex]A^T.A.x = A^T.b [/itex]
then
[itex]((A^T.A)^-1)(A^T.A).x = ((A^T.A)^-1)(A^T.b) [/itex]
[itex]x = ((A^T.A)^-1)(A^T.b) [/itex]
 
  • #11
fluidistic
Gold Member
3,649
99
should lead to the same result, maybe you're missing the A^T on the right side?
[itex]A.x = b [/itex]
[itex]A^T.A.x = A^T.b [/itex]
then
[itex]((A^T.A)^-1)(A^T.A).x = ((A^T.A)^-1)(A^T.b) [/itex]
[itex]x = ((A^T.A)^-1)(A^T.b) [/itex]
Yeah I understand your method. No, I do not miss the A^T b term. I wrote what I get as my matrix A^T b in post #2, see picture 2.

So basically my method is right but it seems I made at least an algebraic (or arithmetic, I don't know how to call it) error when computing x.
I'll re-recheck that...
Thanks a lot anyway. At least I know I made some error(s) while my method is OK.
 
  • #12
SteamKing
Staff Emeritus
Science Advisor
Homework Helper
12,798
1,666
If you substitute your x vector back into your matrix equation, you will find that the equation is not satisfied numerically. Your friend's solutions are much closer to the actual values which satisfy your matrix equation.
 
  • #13
fluidistic
Gold Member
3,649
99
If you substitute your x vector back into your matrix equation, you will find that the equation is not satisfied numerically. Your friend's solutions are much closer to the actual values which satisfy your matrix equation.
Yeah I know I made an algebraic error. Though I've been confirmed my method is correct.
 

Related Threads for: Least squares fitting

  • Last Post
Replies
3
Views
860
  • Last Post
Replies
2
Views
895
  • Last Post
Replies
1
Views
1K
  • Last Post
Replies
1
Views
1K
  • Last Post
Replies
10
Views
2K
Replies
2
Views
3K
Replies
0
Views
2K
Replies
1
Views
775
Top