I am trying to figure out how the least squares formula is derived.(adsbygoogle = window.adsbygoogle || []).push({});

With the error function as

E_{i}= y_{i}- Ʃ_{j}x_{i}^{j}a_{j}

the sum of the errors is

SSE = Ʃ_{i}E_{i}^{2}

so the 1st partial derivative of SSE with respect to a_{j}is

∂SSE / ∂a_{j}= Ʃ_{i}2 E_{i}( ∂E_{i}/ ∂a_{j})

with the 1st partial derivative of E_{i}with respect to a_{j}being

∂E_{i}/ ∂a_{j}= - Ʃ x_{i}^{j}

so the 2nd derivative of SSE with respect to a single a_{j}is

∂^{2}SSE / ∂a_{j}^{2}= 2 Ʃ_{i}{ ( ∂E_{i}/ ∂a_{j}) ( ∂E_{i}/ ∂a_{j}) + E_{i}( ∂^{2}E_{i}/ ∂a_{j}^{2}) }

and a double partial derivative (i.e., to a_{j}& a_{k}) is

∂^{2}SSE / ( ∂a_{j}∂a_{k}) = 2 Ʃ_{i}{ ( ∂E_{i}/ ∂a_{j}) ( ∂E_{i}/ ∂a_{k}) + E_{i}( ∂^{2}E_{i}/ ( ∂a_{j}∂a_{k}) ) }

and with the 2nd partial derivative of E_{i}with respect to a_{j}or the double partial derivative (i.e., to a_{j}& a_{k}) being 0, since the 1st partial is a constant (i.e., in a_{j})

∂^{2}E_{i}/ ∂a_{j}^{2}= ∂^{2}E_{i}/ ( ∂a_{j}∂a_{k}) = 0

the 2nd derivatives reduce to

∂^{2}SSE / ∂a_{j}^{2}= 2 Ʃ_{i}{ ( ∂E_{i}/ ∂a_{j}) ( ∂E_{i}/ ∂a_{j}) }

∂^{2}SSE / ( ∂a_{j}∂a_{k}) = 2 Ʃ_{i}{ ( ∂E_{i}/ ∂a_{j}) ( ∂E_{i}/ ∂a_{k}) }

which after the substitution of ∂E_{i}/ ∂a_{j}becomes

∂^{2}SSE / ( ∂a_{j}∂a_{k}) = - 2 Ʃ_{i}x_{i}^{( 2 j )}

∂^{2}SSE / ( ∂a_{j}∂a_{k}) = - 2 Ʃ_{i}x_{i}^{( j + k )}

so the Hessian matrix (e.g., 3 x 3) is

[ H ] =

[ n Ʃ x Ʃ x^{2}]

[ Ʃ x Ʃ x^{2}Ʃ x^{3}]

[ Ʃ x^{2}Ʃ x^{3}Ʃ x^{4}]

which is the normal product of the Vandemonde matrix

[ V ] =

[ 1 x_{0}x_{0}^{2}]

[ 1 x_{1}x_{1}^{2}]

[ 1 x_{2}x_{2}^{2}]

H = [ V ]^{T}[ V ]

While the diagonal terms are sums of the even powers of x, and therefore always positive, it seems that the 2nd derivative test requires that the determinant of the Hessian matrix be positive. Is there any way to prove that this normal product (or even just the Vandermonde matrix itself, since the determinant of itself would merely be the square root of the determinant of the normal product) is guaranteed to have a positive determinant?

So how to determine that indeed the critical point (which is the solution to the coefficients) is a minimum?

**Physics Forums - The Fusion of Science and Community**

The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

# Optimization & singular Hessian matrix

Loading...

Similar Threads - Optimization singular Hessian | Date |
---|---|

I Optimizing fractions and Lagrange Multiplier | Aug 14, 2017 |

A Analytical Integration of a Difficult Function | Apr 27, 2017 |

I Dot product constrained optimization | Sep 21, 2016 |

I Optimization in Projectile Motion | Sep 17, 2016 |

A Can this optimization problem be solved? | Jul 18, 2016 |

**Physics Forums - The Fusion of Science and Community**