Optimization & singular Hessian matrix

In summary, the Hessian matrix is a normal product of the Vandemonde matrix, and the determinant of the Hessian matrix must be positive in order for the second derivative test to be satisfied.
  • #1
571
83
I am trying to figure out how the least squares formula is derived.

With the error function as

Ei = yi - Ʃj xij aj

the sum of the errors is

SSE = Ʃi Ei2

so the 1st partial derivative of SSE with respect to aj is

∂SSE / ∂aj = Ʃi 2 Ei ( ∂Ei / ∂aj )

with the 1st partial derivative of Ei with respect to aj being

∂Ei / ∂aj = - Ʃ xij

so the 2nd derivative of SSE with respect to a single aj is

2SSE / ∂aj2 = 2 Ʃi { ( ∂Ei / ∂aj ) ( ∂Ei / ∂aj ) + Ei ( ∂2Ei / ∂aj2 ) }

and a double partial derivative (i.e., to aj & ak) is

2SSE / ( ∂aj ∂ak ) = 2 Ʃi { ( ∂Ei / ∂aj ) ( ∂Ei / ∂ak ) + Ei ( ∂2Ei / ( ∂aj ∂ak ) ) }

and with the 2nd partial derivative of Ei with respect to aj or the double partial derivative (i.e., to aj & ak) being 0, since the 1st partial is a constant (i.e., in aj )

2Ei / ∂aj2 = ∂2Ei / ( ∂aj ∂ak ) = 0

the 2nd derivatives reduce to

2SSE / ∂aj2 = 2 Ʃi { ( ∂Ei / ∂aj ) ( ∂Ei / ∂aj ) }

2SSE / ( ∂aj ∂ak ) = 2 Ʃi { ( ∂Ei / ∂aj ) ( ∂Ei / ∂ak ) }

which after the substitution of ∂Ei / ∂aj becomes

2SSE / ( ∂aj ∂ak ) = - 2 Ʃi xi( 2 j )

2SSE / ( ∂aj ∂ak ) = - 2 Ʃi xi( j + k )

so the Hessian matrix (e.g., 3 x 3) is

[ H ] =

[ n Ʃ x Ʃ x2 ]

[ Ʃ x Ʃ x2 Ʃ x3 ]

[ Ʃ x2 Ʃ x3 Ʃ x4 ]

which is the normal product of the Vandemonde matrix

[ V ] =

[ 1 x0 x02 ]

[ 1 x1 x12 ]

[ 1 x2 x22 ]

H = [ V ]T [ V ]

While the diagonal terms are sums of the even powers of x, and therefore always positive, it seems that the 2nd derivative test requires that the determinant of the Hessian matrix be positive. Is there any way to prove that this normal product (or even just the Vandermonde matrix itself, since the determinant of itself would merely be the square root of the determinant of the normal product) is guaranteed to have a positive determinant?

So how to determine that indeed the critical point (which is the solution to the coefficients) is a minimum?
 
Last edited:
Physics news on Phys.org
  • #2
swampwiz said:
with the 1st partial derivative of Ei with respect to aj being
∂Ei / ∂aj = - Ʃ ( xij )2
Isn't it just -xij ?
 
  • #3
haruspex said:
Isn't it just -xij ?

Yes, I went through my analysis and detected the error, and continued.
 

1. What is Optimization?

Optimization is the process of finding the best solution for a given problem. In science, this often involves finding the maximum or minimum value of a function.

2. What is a singular Hessian matrix?

A singular Hessian matrix is a square matrix used in optimization that has a determinant of zero. This means that it does not have an inverse, making it difficult to use in certain optimization algorithms.

3. How is a singular Hessian matrix used in optimization?

A singular Hessian matrix can be used to identify critical points or stationary points in a function. These points are where the gradient of the function is equal to zero, and they can provide important information about the behavior of the function.

4. What are the implications of a singular Hessian matrix in optimization?

Having a singular Hessian matrix can make optimization more challenging, as it may not be possible to use certain algorithms that rely on the matrix's inverse. This can also indicate that the optimization problem may have multiple solutions or that the optimization algorithm may converge slowly.

5. How can a singular Hessian matrix be dealt with in optimization?

There are several approaches to dealing with a singular Hessian matrix in optimization. One option is to use a different optimization algorithm that does not rely on the matrix's inverse. Another approach is to make small adjustments to the matrix to make it nonsingular, such as adding a small value to the diagonal. Additionally, some optimization techniques, such as regularization, can help to mitigate the effects of a singular Hessian matrix.

Suggested for: Optimization & singular Hessian matrix

Replies
3
Views
1K
Replies
3
Views
930
Replies
2
Views
1K
Replies
2
Views
1K
Replies
4
Views
3K
Replies
2
Views
974
Replies
3
Views
1K
Replies
2
Views
982
Back
Top