# Least squares fitting

Gold Member

## Homework Statement

I must find the best fitting function of the form ax²+bx+c using least squares.
The points are (-1,6.1), (0,2.8), (1,2.2), (3,6) and (6,26.9).

2. Homework Equations + attempt at a solution
$$A\vec x= \vec b$$, I'm looking for $$\vec x =\begin {pmatrix} a \\ b \\ c \end {pmatrix}$$. I know that $$\vec b = \begin {pmatrix} 6.1 \\ 2.8 \\ 2.2 \\ 6 \\ 26.9 \end {pmatrix}$$.
With the use of some theory, the system I want to solve is equivalent to $$A^{T}A \vec x = A^{T} \vec b$$.
My matrix A is $$\begin {bmatrix} 1 & -1 & 1 \\ 0 & 0 & 1 \\ 1 & 1 & 1 \\ 9 & 3 & 1 \\ 36 & 6 & 1 \end {bmatrix}$$.
$$A^{T}A=\begin {bmatrix} 1379 & 243 & 47 \\ 243 & 47 & 9 \\ 47 & 9 & 5 \end {bmatrix}$$.
And $$A^{T}\vec b = \begin {bmatrix} 1030.7 \\ 175.5 \\ 44 \end {bmatrix}$$.
This gave me $$\vec x = \begin {bmatrix} 2.669598408 \\ -9.505782817 \\ 0.6247662773 \end {bmatrix}$$. Thus $$f(x) \approx 2.67 x^2-9.51 x +0.62$$.
I've plotted it in my calculator and it doesn't seem to fit well at all. My friend put these points in a program and he got a much better fitting ($$a \approx 3$$, $$b \approx -2$$, $$c \approx 1$$).
I've redone the algebra and I don't see my error(s).
I got right all problems involving a linear fit, but failed all other fits (exponentials, quadratic, etc.). What am I doing wrong?!!

Another question that really matters to me is... why don't we solve Ax=b directly since we know both A and b, we would get 3 equations with 3 unknowns... why bother with the transpose of A?

Thank you very much for any help.

lanedance
Homework Helper
the reason you use A^T.A is written in theorem 2

Theorem 2 Suppose that A is an matrix with linearly independent columns. Then, is an invertible matrix.

lanedance
Homework Helper
now everything on that page is to do with simple linear squares regression

(x1, b1)

when you do quadratic regression, in effect you consider the triplet
(x1, y1, b1) where y1 = x1^2

then you regress on the parameters a,b,c in the fit equation
ay+bx+c = ax^2+bx+c

note the equation is linear in a,b,c, hence still linear regression (y could easily be another functional form eg. exponential y = e^x)

as there are now in effect 2 variables (x & y=x^2) you need to consider multivariate linear regression

hopefully this helps, my tex isn't displaying correctly so i can't properly QC what you've written

Gold Member
the reason you use A^T.A is written in theorem 2

Thanks a lot.. My bad, I said something wrong :/
I said I'd get 3 equations with 3 unknowns when A isn't even square. This would do in fact 4 equations with 3 unknowns.
Question solved. Now I'm waiting for help on the first part.

Gold Member
Ok: Latex problem; I'll solve that for you. :D

#### Attachments

lanedance
Homework Helper
Thanks a lot.. My bad, I said something wrong :/
I said I'd get 3 equations with 3 unknowns when A isn't even square. This would do in fact 4 equations with 3 unknowns.
Question solved. Now I'm waiting for help on the first part.

the other thing to note is even if it was 3 and 3, there is not guaranteed a consistent unique solution. Hence least squares finds the solution with minimum squre error

lanedance
Homework Helper
cheers for photos

so how do you get your final x?

shouldn't it be
(A^T.A)^(-1).(A^T.b)

if i do that i get different values to you...

lanedance
Homework Helper
i then get
0.99
-1.99
3.03

also be careful to check the order is correct (ie corresponding a,b,c to x^2,x,1)

Gold Member
Oh thanks a lot. You get the same values as the program of my friend...
I did get my x as follows: write down $A^TA$ and amplify this matrix with the column $$\vec b$$. Then reduce $A^TA$ into a row echelon form and look at what happened in the augmented part of the matrix.
So in a sense, I didn't even calculated the inverse of $A^TA$, which might be where my error lies?

#### Attachments

lanedance
Homework Helper
Oh thanks a lot. You get the same values as the program of my friend...
I did get my x as follows: write down $A^TA$ and amplify this matrix with the column $$\vec b$$. Then reduce $A^TA$ into a row echelon form and look at what happened in the augmented part of the matrix.
So in a sense, I didn't even calculated the inverse of $A^TA$, which might be where my error lies?

should lead to the same result, maybe you're missing the A^T on the right side?
$A.x = b$
$A^T.A.x = A^T.b$
then
$((A^T.A)^-1)(A^T.A).x = ((A^T.A)^-1)(A^T.b)$
$x = ((A^T.A)^-1)(A^T.b)$

Gold Member
should lead to the same result, maybe you're missing the A^T on the right side?
$A.x = b$
$A^T.A.x = A^T.b$
then
$((A^T.A)^-1)(A^T.A).x = ((A^T.A)^-1)(A^T.b)$
$x = ((A^T.A)^-1)(A^T.b)$

Yeah I understand your method. No, I do not miss the A^T b term. I wrote what I get as my matrix A^T b in post #2, see picture 2.

So basically my method is right but it seems I made at least an algebraic (or arithmetic, I don't know how to call it) error when computing x.
I'll re-recheck that...
Thanks a lot anyway. At least I know I made some error(s) while my method is OK.

SteamKing
Staff Emeritus