Use of Laplacian operator in Operations Research Book

[hotvette]

I've been studying the book "Numerical Optimization" by Jorge Nocedal and Stephan J. Wright published by Springer, 1999. I'm puzzled by the use of the Laplacian operator $\nabla^2$ in chapter 10 on nonlinear least squares and in the appendix to define the Hessian matrix. The following is from pages 252 and 582:


$$\begin{align*}
\nabla f(x) &= \sum_{j=1}^m r_j(x) \nabla r_j(x) = J(x)^T r(x) \\
\nabla^2 f(x) &= \sum_{j=1}^m \nabla r_j(x) \nabla r_j(x)^T + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x) \\
&= J(x)^T J(x) + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x)
\end{align*}$$
The matrix of second partial derivatives of $f$ is known as the Hessian, and is defined as

$$
\nabla^2 f(x) = \begin{bmatrix}
\frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \dots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\
\frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \dots & \frac{\partial^2 f}{\partial x_2 \partial x_n} \\
\vdots & \vdots && \vdots \\
\frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial^2 f}{\partial x_n \partial x_2} & \dots & \frac{\partial^2 f}{\partial x_n^2}
\end{bmatrix}
$$


Is it my imagination that the Laplacian operator is being improperly used? My understanding is that the Laplacian is:

$$
\nabla^2 f(x) = \frac{\partial^2 f}{\partial x_1^2} + \frac{\partial^2 f}{\partial x_2^2} + \dots + \frac{\partial^2 f}{\partial x_n^2}
$$
which is the trace of the Hessian.

 

[anuttarasammyak]

On websites I found various expressions of Hessian matrix.
I think one in https://en.m.wikipedia.org/wiki/Hessian_matrix
Is modest. Laplacian is inner product and Hessian is tensor product of $\nabla$ vectors, they seem.

 

[hotvette]

I found multiple references that use the Laplacian to define the Hessian matrix like what I found in the Optimization book:

https://www.mit.edu/~gfarina/2024/67220s24_L12_newton/L12.pdf
https://www.geeksforgeeks.org/multivariate-optimization-gradient-and-hessian/#
https://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/slides/lec07.pdf
https://en.wikipedia.org/wiki/Newton's_method_in_optimization
https://www2.isye.gatech.edu/~nemirovs/OPTIIILN2023Spring.pdf
https://www.math.ucla.edu/~abrose/m164/mark/B4.pdf

I guess the reality is that different technical fields use the same symbol to mean different things. The key is to clearly define what the symbol means in the context of the discussion.

 

[pasmith]

I've been studying the book "Numerical Optimization" by Jorge Nocedal and Stephan J. Wright published by Springer, 1999. I'm puzzled by the use of the Laplacian operator $\nabla^2$ in chapter 10 on nonlinear least squares and in the appendix to define the Hessian matrix. The following is from pages 252 and 582:


$$\begin{align*}
\nabla f(x) &= \sum_{j=1}^m r_j(x) \nabla r_j(x) = J(x)^T r(x) \\
\nabla^2 f(x) &= \sum_{j=1}^m \nabla r_j(x) \nabla r_j(x)^T + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x) \\
&= J(x)^T J(x) + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x)
\end{align*}$$

Here the authors appear to be using \nabla^2 to mean the dyad (tensor) product operator \nabla \nabla defined as <br /> (\nabla \nabla)_{ik} = \frac{\partial^2}{\partial x_i\,\partial x_k}. This does make logical sense: \nabla^2 = \nabla \nabla. On the other hand, in physics there is the convention that the norm of a vector \mathbf{a} is denoted a, and following the example of \mathbf{a} \cdot \mathbf{a} = \|\mathbf{a}\|^2 = a^2 we write \nabla^2 to mean \nabla \cdot \nabla (an operator required must more frequently than \nabla \nabla).

Different fields have different conventions; one must be careful to note which is in use.

 

[jbergman]

I've been studying the book "Numerical Optimization" by Jorge Nocedal and Stephan J. Wright published by Springer, 1999. I'm puzzled by the use of the Laplacian operator $\nabla^2$ in chapter 10 on nonlinear least squares and in the appendix to define the Hessian matrix. The following is from pages 252 and 582:


$$\begin{align*}
\nabla f(x) &= \sum_{j=1}^m r_j(x) \nabla r_j(x) = J(x)^T r(x) \\
\nabla^2 f(x) &= \sum_{j=1}^m \nabla r_j(x) \nabla r_j(x)^T + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x) \\
&= J(x)^T J(x) + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x)
\end{align*}$$
The matrix of second partial derivatives of $f$ is known as the Hessian, and is defined as

$$
\nabla^2 f(x) = \begin{bmatrix}
\frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \dots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\
\frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \dots & \frac{\partial^2 f}{\partial x_2 \partial x_n} \\
\vdots & \vdots && \vdots \\
\frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial^2 f}{\partial x_n \partial x_2} & \dots & \frac{\partial^2 f}{\partial x_n^2}
\end{bmatrix}
$$


Is it my imagination that the Laplacian operator is being improperly used? My understanding is that the Laplacian is:

$$
\nabla^2 f(x) = \frac{\partial^2 f}{\partial x_1^2} + \frac{\partial^2 f}{\partial x_2^2} + \dots + \frac{\partial^2 f}{\partial x_n^2}
$$
which is the trace of the Hessian.

Probably worth mentioning that people often use $\Delta$ for the laplacian. Did they explicitly call it a laplacian in their text?

 

[WWGD]

Probably you have checked, but is there a notation index in the back of your book?

 

[hotvette]

The appendix has a section called Background Material that defines $\nabla^2 f(x)$ as the Hessian. It doesn't contain the word Laplacian. I had a chat with a friend (statistics professor) and he told me he encounters this sort of thing (different uses for the same symbol) all the time.