Least squares method : approximation of a cubic polynomial

In summary, we want to determine an approximation of a cubic polynomial using the least squares method. We calculate the partial derivatives and set them equal to 0 to get a system of equations. Solving this system, we get the following solution: $p(x)=2.33768x^3-2.0.6057x^2-7.08875x-21.6611$. However, upon checking the calculation, a mistake was found and the correct polynomial should be $p(x)=3.05677x^3-2.78419x^2-12.2439x-23.0084$. This is a close approximation and the slight differences in coefficients are due to rounding. The value of
  • #1
mathmari
Gold Member
MHB
5,049
7
Hey! :eek:

I want to determine an approximation of a cubic polynomial that has at the points $$x_0=-2, \ x_1=-1, \ x_2=0 , \ x_3=3, \ x_4=3.5$$ the values $$y_0=-33, \ y_1=-20, \ y_2=-20.1, \ y_3=-4.3 , \ y_4=32.5$$ using the least squares method.

So we are looking for a cubic polynomial $p(x)$ such that $$\sum_{i=0}^4\left (p(x_i)-y_i\right )^2$$ is minimal, right? (Wondering) Let $p(x)=a_3x^3+a_2x^2+a_1x+a_0$. Then we get the following sum:
\begin{align*}\sum_{i=0}^4\left (p(x_i)-y_i\right )^2=&\left (-8a_3+4a_2-2a_1+a_0+33\right )^2+\left (-a_3+a_2-a_1+a_0+20\right )^2+\left (a_0+20.1\right )^2 \\ & +\left (27a_3+9a_2+3a_1+a_0+4.3\right )^2 +\left (42.875a_3+12.25a_2+3.5a_1+a_0-32.5\right )^2\end{align*}

Now we calculate the partial derivatives in respect to each $a_i$ and then we set these equal to $0$. In that we get a system of equations.

Let this sum be $S$.

We get the following partial derivatives:

\begin{align*}\frac{\partial{S}}{\partial{a_0}}=&2\left (-8a_3+4a_2-2a_1+a_0+33\right )+2\left (-a_3+a_2-a_1+a_0+20\right )\\ & +2\left (a_0+20.1\right ) +2\left (27a_3+9a_2+3a_1+a_0+4.3\right ) +2\left (42.875a_3+12.25a_2+3.5a_1+a_0-32.5\right )\\ = &89.8 + 10 a_0 + 7 a_1+ 52.5 a_2 + 121.75 a_3\end{align*}

\begin{align*}\frac{\partial{S}}{\partial{a_1}}=&2\left (-8a_3+4a_2-2a_1+a_0+33\right )\cdot 2+2\left (-a_3+a_2-a_1+a_0+20\right )\cdot (-1) \\ & +2\left (27a_3+9a_2+3a_1+a_0+4.3\right )\cdot 3 +2\left (42.875a_3+12.25a_2+3.5a_1+a_0-32.5\right )\cdot 3.5 \\ = & -109.7 + 15 a_0 + 36.5 a_1 + 153.75 a_2 + 432.125 a_3\end{align*}

\begin{align*}\frac{\partial{S}}{\partial{a_2}}=&2\left (-8a_3+4a_2-2a_1+a_0+33\right )\cdot 4+2\left (-a_3+a_2-a_1+a_0+20\right ) \\ & +2\left (27a_3+9a_2+3a_1+a_0+4.3\right )\cdot 9 +2\left (42.875a_3+12.25a_2+3.5a_1+a_0-32.5\right )\cdot 12.25\\ = & -414.85 + 52.5 a_0 + 121.75 a_1 + 496.125 a_2 + 1470.44 a_3 \end{align*}

\begin{align*}\frac{\partial{S}}{\partial{a_3}}=&2\left (-8a_3+4a_2-2a_1+a_0+33\right )\cdot (-8)+2\left (-a_3+a_2-a_1+a_0+20\right )\cdot (-1) \\ & +2\left (27a_3+9a_2+3a_1+a_0+4.3\right )\cdot 27 +2\left (42.875a_3+12.25a_2+3.5a_1+a_0-32.5\right )\cdot 42.875\\ = & -3122.68 + 121.75 a_0 + 496.125 a_1 + 1470.44 a_2 + 5264.53 a_3\end{align*}

So, we get the system:
\begin{align*}& \ \ \ \ 89.8 + 10 a_0 + 7 a_1+ 52.5 a_2 + 121.75 a_3=0 \\ &-109.7 + 15 a_0 + 36.5 a_1 + 153.75 a_2 + 432.125 a_3=0 \\ &-414.85 + 52.5 a_0 + 121.75 a_1 + 496.125 a_2 + 1470.44 a_3=0 \\ &-3122.68 + 121.75 a_0 + 496.125 a_1 + 1470.44 a_2 + 5264.53 a_3=0\end{align*}

Then we get the solution: \begin{equation*}a_0\approx -21.6611, \ a_1\approx -7.08875, \ a_2\approx -2.0.6057, \ a_3\approx 2.33768\end{equation*}

The cubic polynomial that we are looking for is \begin{equation*}p(x)=2.33768x^3-2.0.6057x^2-7.08875x-21.6611 \end{equation*}
Doing the above in an other way I got an other result, but I don't know why and which is the correct one. (Wondering)

Using the least square method and the normal equation we get that the solution that we are looking for is the vector $\vec{a}$ such that $A \vec{a}=b \Rightarrow A^TA\vec{a}=A^Tb$.

So we get the following:
\begin{equation*}
\begin{bmatrix}
5 & 3.5 & 26.25 & 60.875 \\
3.5 & 26.25 & 60.875 & 248.0625 \\
26.25 & 60.875 & 248.0625 & 735.21875 \\
60.875 & 248.0625 & 735.21875 & 2632.265625
\end{bmatrix}
\begin{bmatrix}
a_0 \\
a_1 \\
a_2 \\
a_3 \\
\end{bmatrix}=
\begin{bmatrix}
-44.9 \\
186 \\
207.425 \\
1561.3375 \\
\end{bmatrix}
\end{equation*}

Solving this one we get the polynomial $$p(x)=-23.0087-12.2424x-2.7837x^2+3.0565x^3$$ This is not the same polynomial as the one that we get at the first method. And for the second polynomial the condition $p(0)=-20.1$ is not true. (Wondering)
 
Mathematics news on Phys.org
  • #2
Hey mathmari!

I checked the solution $a=(A^T A)^{-1} A^T y$, and found the same solution that you did.
If I graph it, I get:
View attachment 8788
https://octave-online.net/
xx=[-2 -1 0 3 3.5]';
yy=[-33 -20 -20.1 -4.3 32.5]';
A=[ones(5,1) xx xx.^2 xx.^3];
ATA = A'*A
ATy = A'*yy
a = ATA\ATy
plot(xx,yy)
hold on
plot(xx, A*a)
That looks pretty close doesn't it, even though $p(0)$ is slightly off? (Wondering)

And yes, your solution with the derivatives set to zero should have the same result. (Worried)EDIT: Now I've also plotted it with the $a$ you found with the derivatives:
View attachment 8789
a_diff = [-21.6611, -7.08875, -2.06057, 2.33768]'
plot(xx, A*a_diff)

Hmm... that's not the same is it?
The first one does look better though.
Perhaps there is a mistake? (Wondering)EDIT 2:
If I calculate the residu $\|Aa-y\|$ in both cases, I get:
Code:
>> norm(A*a-yy)
ans =  5.2292
>> norm(A*a_diff-yy)
ans =  7.5340

In other words, the solution with the derivatives did not find the lowest possible residu.
It confirms that there must be a calculation mistake. (Worried)
 

Attachments

  • cubic_lsq.png
    cubic_lsq.png
    2.9 KB · Views: 71
  • cubic_lsq_diff.png
    cubic_lsq_diff.png
    3.1 KB · Views: 75
  • #3
I checked my calculations again and I saw that I had a wrong sign. Now I get \begin{equation*}p(x)=3.05677x^3-2.78419x^2-12.2439x-23.0084 \end{equation*} which is approximately the same as the second one. Do we have have to get the exactly same polynomial or is it correct that we get approximately the same one? But what about the value of the polynomial at $x=0$ ? Shouldn't it be as the given value? (Wondering)
 
  • #4
mathmari said:
I checked my calculations again and I saw that I had a wrong sign. Now I get \begin{equation*}p(x)=3.05677x^3-2.78419x^2-12.2439x-23.0084 \end{equation*} which is approximately the same as the second one. Do we have have to get the exactly same polynomial or is it correct that we get approximately the same one? But what about the value of the polynomial at $x=0$ ? Shouldn't it be as the given value?

Good!
The first 4 significant digits of each coefficient are the same.
I think it means that intermediate results of the numbers have been rounded to perhaps 5 or 6 significant digits.
The results should be exactly the same, but of cause they are sensitive to rounding.
Anyway, they are close enough that I believe that the calculations are correct now. (Thinking)And no, the value of polynomial is not expected to be the same as the given value at $x=0$.
It is an approximation after all.
We can only fit a cubic polynomial perfectly through 4 points.
Since 5 points were given, it is normal that every point is slightly off from the given value.
The squared sum of the differences is merely as low as possible. (Nerd)
 

FAQ: Least squares method : approximation of a cubic polynomial

What is the least squares method?

The least squares method is a mathematical technique used to find the best fit line or curve for a set of data points. It minimizes the sum of the squared distances between the data points and the line or curve, resulting in a more accurate approximation of the relationship between the variables.

How is the least squares method used to approximate a cubic polynomial?

The least squares method can be used to find the coefficients of a cubic polynomial that best fits a set of data points. This is done by minimizing the sum of the squared distances between the data points and the cubic polynomial curve. The resulting coefficients can then be used to create an equation for the cubic polynomial.

What are the advantages of using the least squares method for polynomial approximation?

The least squares method is advantageous because it is a simple and efficient way to find the best fit for a set of data points. It also takes into account all of the data points, rather than just a few, resulting in a more accurate approximation. Additionally, it can be applied to a variety of different types of data and functions, making it a versatile tool for scientists and researchers.

What are the limitations of the least squares method?

One limitation of the least squares method is that it assumes a linear relationship between the variables being studied. This means that it may not be the best approach for data that follows a non-linear pattern. Additionally, it can be sensitive to outliers in the data, which can skew the results and lead to a less accurate approximation.

How can the accuracy of the least squares method be evaluated?

The accuracy of the least squares method can be evaluated by calculating the residual sum of squares (RSS), which is the sum of the squared distances between the data points and the fitted curve. A lower RSS indicates a better fit and therefore, a more accurate approximation. Additionally, the coefficient of determination (R-squared) can also be calculated to determine the percentage of the variation in the data that is explained by the fitted curve.

Similar threads

Replies
4
Views
1K
Replies
4
Views
878
Replies
2
Views
1K
Replies
2
Views
2K
Replies
7
Views
1K
Replies
2
Views
2K
Replies
3
Views
1K
Back
Top