Use of Laplacian operator in Operations Research Book

  • #1
hotvette
Homework Helper
1,010
11
I've been studying the book "Numerical Optimization" by Jorge Nocedal and Stephan J. Wright published by Springer, 1999. I'm puzzled by the use of the Laplacian operator ##\nabla^2## in chapter 10 on nonlinear least squares and in the appendix to define the Hessian matrix. The following is from pages 252 and 582:


$$\begin{align*}
\nabla f(x) &= \sum_{j=1}^m r_j(x) \nabla r_j(x) = J(x)^T r(x) \\
\nabla^2 f(x) &= \sum_{j=1}^m \nabla r_j(x) \nabla r_j(x)^T + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x) \\
&= J(x)^T J(x) + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x)
\end{align*}$$
The matrix of second partial derivatives of ##f## is known as the Hessian, and is defined as

$$
\nabla^2 f(x) = \begin{bmatrix}
\frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \dots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\
\frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \dots & \frac{\partial^2 f}{\partial x_2 \partial x_n} \\
\vdots & \vdots && \vdots \\
\frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial^2 f}{\partial x_n \partial x_2} & \dots & \frac{\partial^2 f}{\partial x_n^2}
\end{bmatrix}
$$


Is it my imagination that the Laplacian operator is being improperly used? My understanding is that the Laplacian is:

$$
\nabla^2 f(x) = \frac{\partial^2 f}{\partial x_1^2} + \frac{\partial^2 f}{\partial x_2^2} + \dots + \frac{\partial^2 f}{\partial x_n^2}
$$
which is the trace of the Hessian.
 
  • Like
Likes PeroK
Physics news on Phys.org
  • #2
Last edited:
  • Like
Likes PeroK
  • #3
I found multiple references that use the Laplacian to define the Hessian matrix like what I found in the Optimization book:

https://www.mit.edu/~gfarina/2024/67220s24_L12_newton/L12.pdf
https://www.geeksforgeeks.org/multivariate-optimization-gradient-and-hessian/#
https://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/slides/lec07.pdf
https://en.wikipedia.org/wiki/Newton's_method_in_optimization
https://www2.isye.gatech.edu/~nemirovs/OPTIIILN2023Spring.pdf
https://www.math.ucla.edu/~abrose/m164/mark/B4.pdf

I guess the reality is that different technical fields use the same symbol to mean different things. The key is to clearly define what the symbol means in the context of the discussion.
 
  • Like
Likes PhDeezNutz and WWGD
  • #4
hotvette said:
I've been studying the book "Numerical Optimization" by Jorge Nocedal and Stephan J. Wright published by Springer, 1999. I'm puzzled by the use of the Laplacian operator ##\nabla^2## in chapter 10 on nonlinear least squares and in the appendix to define the Hessian matrix. The following is from pages 252 and 582:


$$\begin{align*}
\nabla f(x) &= \sum_{j=1}^m r_j(x) \nabla r_j(x) = J(x)^T r(x) \\
\nabla^2 f(x) &= \sum_{j=1}^m \nabla r_j(x) \nabla r_j(x)^T + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x) \\
&= J(x)^T J(x) + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x)
\end{align*}$$

Here the authors appear to be using [itex]\nabla^2[/itex] to mean the dyad (tensor) product operator [itex]\nabla \nabla[/itex] defined as [tex]
(\nabla \nabla)_{ik} = \frac{\partial^2}{\partial x_i\,\partial x_k}.[/tex] This does make logical sense: [itex]\nabla^2 = \nabla \nabla[/itex]. On the other hand, in physics there is the convention that the norm of a vector [itex]\mathbf{a}[/itex] is denoted [itex]a[/itex], and following the example of [tex]\mathbf{a} \cdot \mathbf{a} = \|\mathbf{a}\|^2 = a^2[/tex] we write [itex]\nabla^2[/itex] to mean [itex]\nabla \cdot \nabla[/itex] (an operator required must more frequently than [itex]\nabla \nabla[/itex]).

Different fields have different conventions; one must be careful to note which is in use.
 
  • #5
hotvette said:
I've been studying the book "Numerical Optimization" by Jorge Nocedal and Stephan J. Wright published by Springer, 1999. I'm puzzled by the use of the Laplacian operator ##\nabla^2## in chapter 10 on nonlinear least squares and in the appendix to define the Hessian matrix. The following is from pages 252 and 582:


$$\begin{align*}
\nabla f(x) &= \sum_{j=1}^m r_j(x) \nabla r_j(x) = J(x)^T r(x) \\
\nabla^2 f(x) &= \sum_{j=1}^m \nabla r_j(x) \nabla r_j(x)^T + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x) \\
&= J(x)^T J(x) + \sum_{j=1}^m r_j(x) \nabla^2 r_j(x)
\end{align*}$$
The matrix of second partial derivatives of ##f## is known as the Hessian, and is defined as

$$
\nabla^2 f(x) = \begin{bmatrix}
\frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \dots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\
\frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \dots & \frac{\partial^2 f}{\partial x_2 \partial x_n} \\
\vdots & \vdots && \vdots \\
\frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial^2 f}{\partial x_n \partial x_2} & \dots & \frac{\partial^2 f}{\partial x_n^2}
\end{bmatrix}
$$


Is it my imagination that the Laplacian operator is being improperly used? My understanding is that the Laplacian is:

$$
\nabla^2 f(x) = \frac{\partial^2 f}{\partial x_1^2} + \frac{\partial^2 f}{\partial x_2^2} + \dots + \frac{\partial^2 f}{\partial x_n^2}
$$
which is the trace of the Hessian.
Probably worth mentioning that people often use ##\Delta## for the laplacian. Did they explicitly call it a laplacian in their text?
 
  • #6
Probably you have checked, but is there a notation index in the back of your book?
 
  • #7
The appendix has a section called Background Material that defines ##\nabla^2 f(x)## as the Hessian. It doesn't contain the word Laplacian. I had a chat with a friend (statistics professor) and he told me he encounters this sort of thing (different uses for the same symbol) all the time.
 
  • Like
Likes PhDeezNutz

Similar threads

Replies
41
Views
3K
Replies
3
Views
2K
Replies
12
Views
648
Replies
5
Views
1K
Replies
9
Views
3K
Replies
2
Views
1K
Replies
4
Views
1K
Replies
3
Views
1K
Back
Top