Optimization & singular Hessian matrix

[swampwiz]

I am trying to figure out how the least squares formula is derived.

With the error function as

E_i = y_i - Ʃ_j x_i^j a_j

the sum of the errors is

SSE = Ʃ_i E_i²

so the 1st partial derivative of SSE with respect to a_j is

∂SSE / ∂a_j = Ʃ_i 2 E_i ( ∂E_i / ∂a_j )

with the 1st partial derivative of E_i with respect to a_j being

∂E_i / ∂a_j = - Ʃ x_i^j

so the 2nd derivative of SSE with respect to a single a_j is

∂²SSE / ∂a_j² = 2 Ʃ_i { ( ∂E_i / ∂a_j ) ( ∂E_i / ∂a_j ) + E_i ( ∂²E_i / ∂a_j² ) }

and a double partial derivative (i.e., to a_j & a_k) is

∂²SSE / ( ∂a_j ∂a_k ) = 2 Ʃ_i { ( ∂E_i / ∂a_j ) ( ∂E_i / ∂a_k ) + E_i ( ∂²E_i / ( ∂a_j ∂a_k ) ) }

and with the 2nd partial derivative of E_i with respect to a_j or the double partial derivative (i.e., to a_j & a_k) being 0, since the 1st partial is a constant (i.e., in a_j )

∂²E_i / ∂a_j² = ∂²E_i / ( ∂a_j ∂a_k ) = 0

the 2nd derivatives reduce to

∂²SSE / ∂a_j² = 2 Ʃ_i { ( ∂E_i / ∂a_j ) ( ∂E_i / ∂a_j ) }

∂²SSE / ( ∂a_j ∂a_k ) = 2 Ʃ_i { ( ∂E_i / ∂a_j ) ( ∂E_i / ∂a_k ) }

which after the substitution of ∂E_i / ∂a_j becomes

∂²SSE / ( ∂a_j ∂a_k ) = - 2 Ʃ_i x_i^{( 2 j )}

∂²SSE / ( ∂a_j ∂a_k ) = - 2 Ʃ_i x_i^{( j + k )}

so the Hessian matrix (e.g., 3 x 3) is

[ H ] =

[ n Ʃ x Ʃ x² ]

[ Ʃ x Ʃ x² Ʃ x³ ]

[ Ʃ x² Ʃ x³ Ʃ x⁴ ]

which is the normal product of the Vandemonde matrix

[ V ] =

[ 1 x₀ x₀² ]

[ 1 x₁ x₁² ]

[ 1 x₂ x₂² ]

H = [ V ]^T [ V ]

While the diagonal terms are sums of the even powers of x, and therefore always positive, it seems that the 2nd derivative test requires that the determinant of the Hessian matrix be positive. Is there any way to prove that this normal product (or even just the Vandermonde matrix itself, since the determinant of itself would merely be the square root of the determinant of the normal product) is guaranteed to have a positive determinant?

So how to determine that indeed the critical point (which is the solution to the coefficients) is a minimum?

 

[haruspex]

with the 1st partial derivative of E_i with respect to a_j being
∂E_i / ∂a_j = - Ʃ ( x_i^j )²

Isn't it just -x_i^j ?

 

[swampwiz]

Isn't it just -x_i^j ?

Yes, I went through my analysis and detected the error, and continued.