Gauss-Seidl, Gauss-Southwell, Jacobi...
Couple of days ago I downloaded a book on numerical optimization, hoping to get clearer picture on some techniques.
But, I'm surprised that some of the concepts were not precisely separated from one another.
Namely, in the part of "coordinate descent methods" (cdm), I found that, in the case the contribution of each point
to the function in question is known, one applied the Gauss-Southwell approach to select the point to be updated.
In case the update is done is a sequence from 1 to n, the the approach is referred to as the "cyclic descent method" (cydm).
However, on some other places I've found that Gauss-Seidl method's principle is close to cydm's.
Is Gauss-Seidl actually another name for cydm? I hope someone will clarify this for me.
Another question is regarding the Jacobi method, which is a "hold on" version of Gauss-Seidl, in my understanding
(regarding the point update at the end of iteration). Given certain function f(x) which is quadratic in x,
in order to obtain its minimal, I set the derivative to 0, and obtain the following linear system
which I decide to solve by Jacobi iteration. Given that A is strictly diagonally dominant, is it true that the value of
f(x1), where x1 is obtained by a single Jacobi iteration on some arbitrary x0 (but not the stationary point) satisfies