Least Squares Fitting for ax²+bx+c with Given Points: Homework Solution

[fluidistic]

Homework Statement

I must find the best fitting function of the form ax²+bx+c using least squares.
The points are (-1,6.1), (0,2.8), (1,2.2), (3,6) and (6,26.9).

2. Homework Equations + attempt at a solution
$$A\vec x= \vec b$$, I'm looking for $$\vec x =\begin {pmatrix} a \\ b \\ c \end {pmatrix}$$. I know that $$\vec b = \begin {pmatrix} 6.1 \\ 2.8 \\ 2.2 \\ 6 \\ 26.9 \end {pmatrix}$$.
With the use of some theory, the system I want to solve is equivalent to $$A^{T}A \vec x = A^{T} \vec b$$.
My matrix A is $$\begin {bmatrix} 1 & -1 & 1 \\ 0 & 0 & 1 \\ 1 & 1 & 1 \\ 9 & 3 & 1 \\ 36 & 6 & 1 \end {bmatrix}$$.
$$A^{T}A=\begin {bmatrix} 1379 & 243 & 47 \\ 243 & 47 & 9 \\ 47 & 9 & 5 \end {bmatrix}$$.
And $$A^{T}\vec b = \begin {bmatrix} 1030.7 \\ 175.5 \\ 44 \end {bmatrix}$$.
This gave me $$\vec x = \begin {bmatrix} 2.669598408 \\ -9.505782817 \\ 0.6247662773 \end {bmatrix}$$. Thus $$f(x) \approx 2.67 x^2-9.51 x +0.62$$.
I've plotted it in my calculator and it doesn't seem to fit well at all. My friend put these points in a program and he got a much better fitting ($$a \approx 3$$, $$b \approx -2$$, $$c \approx 1$$).
I've redone the algebra and I don't see my error(s).
I've followed the method of this page: http://tutorial.math.lamar.edu/Classes/LinAlg/LeastSquares.aspx.
I got right all problems involving a linear fit, but failed all other fits (exponentials, quadratic, etc.). What am I doing wrong?!

Another question that really matters to me is... why don't we solve Ax=b directly since we know both A and b, we would get 3 equations with 3 unknowns... why bother with the transpose of A?

Thank you very much for any help.

 

[lanedance]

the reason you use A^T.A is written in theorem 2

Theorem 2 Suppose that A is an matrix with linearly independent columns. Then, is an invertible matrix.

 

[lanedance]

now everything on that page is to do with simple linear squares regression

say your first point is
(x1, b1)

when you do quadratic regression, in effect you consider the triplet
(x1, y1, b1) where y1 = x1^2

then you regress on the parameters a,b,c in the fit equation
ay+bx+c = ax^2+bx+c

note the equation is linear in a,b,c, hence still linear regression (y could easily be another functional form eg. exponential y = e^x)

as there are now in effect 2 variables (x & y=x^2) you need to consider multivariate linear regression

hopefully this helps, my tex isn't displaying correctly so i can't properly QC what you've written

 

[fluidistic]

the reason you use A^T.A is written in theorem 2

Thanks a lot.. My bad, I said something wrong :/
I said I'd get 3 equations with 3 unknowns when A isn't even square. This would do in fact 4 equations with 3 unknowns.
Question solved. Now I'm waiting for help on the first part.

 

[fluidistic]

Ok: Latex problem; I'll solve that for you. :D

 

[lanedance]

Thanks a lot.. My bad, I said something wrong :/
I said I'd get 3 equations with 3 unknowns when A isn't even square. This would do in fact 4 equations with 3 unknowns.
Question solved. Now I'm waiting for help on the first part.

the other thing to note is even if it was 3 and 3, there is not guaranteed a consistent unique solution. Hence least squares finds the solution with minimum squre error

 

[lanedance]

cheers for photos

so how do you get your final x?

shouldn't it be
(A^T.A)^(-1).(A^T.b)

if i do that i get different values to you...

 

[lanedance]

i then get
0.99
-1.99
3.03

also be careful to check the order is correct (ie corresponding a,b,c to x^2,x,1)

 

[fluidistic]

Oh thanks a lot. You get the same values as the program of my friend...
I did get my x as follows: write down $A^TA$ and amplify this matrix with the column $$\vec b$$. Then reduce $A^TA$ into a row echelon form and look at what happened in the augmented part of the matrix.
So in a sense, I didn't even calculated the inverse of $A^TA$, which might be where my error lies?

 

[lanedance]

Oh thanks a lot. You get the same values as the program of my friend...
I did get my x as follows: write down $A^TA$ and amplify this matrix with the column $$\vec b$$. Then reduce $A^TA$ into a row echelon form and look at what happened in the augmented part of the matrix.
So in a sense, I didn't even calculated the inverse of $A^TA$, which might be where my error lies?

should lead to the same result, maybe you're missing the A^T on the right side?
$A.x = b$
$A^T.A.x = A^T.b$
then
$((A^T.A)^-1)(A^T.A).x = ((A^T.A)^-1)(A^T.b)$
$x = ((A^T.A)^-1)(A^T.b)$

 

[fluidistic]

should lead to the same result, maybe you're missing the A^T on the right side?
$A.x = b$
$A^T.A.x = A^T.b$
then
$((A^T.A)^-1)(A^T.A).x = ((A^T.A)^-1)(A^T.b)$
$x = ((A^T.A)^-1)(A^T.b)$

Yeah I understand your method. No, I do not miss the A^T b term. I wrote what I get as my matrix A^T b in post #2, see picture 2.

So basically my method is right but it seems I made at least an algebraic (or arithmetic, I don't know how to call it) error when computing x.
I'll re-recheck that...
Thanks a lot anyway. At least I know I made some error(s) while my method is OK.

 

[SteamKing]

If you substitute your x vector back into your matrix equation, you will find that the equation is not satisfied numerically. Your friend's solutions are much closer to the actual values which satisfy your matrix equation.

 

[fluidistic]

If you substitute your x vector back into your matrix equation, you will find that the equation is not satisfied numerically. Your friend's solutions are much closer to the actual values which satisfy your matrix equation.

Yeah I know I made an algebraic error. Though I've been confirmed my method is correct.