Newton's Method for Optimization

Click For Summary
SUMMARY

Newton's method for optimization in high dimensions does not always guarantee quick convergence to a minimum, maximum, or saddle point. Users may encounter issues where the gradient value remains stagnant between 12-16, indicating neither divergence nor convergence. To mitigate the risk of saddle points, the Fletcher-Reeves method is recommended, although testing with Newton-Raphson can lead to quick convergence under optimal initial conditions. The complexity of multi-variable optimization arises from the nature of the function landscape, exemplified by Rosenbrock's banana function.

PREREQUISITES
  • Understanding of Newton's method for optimization
  • Familiarity with Fletcher-Reeves method
  • Knowledge of gradient descent techniques
  • Basic concepts of multi-variable optimization
NEXT STEPS
  • Study Rosenbrock's banana function and its properties
  • Learn about convergence criteria in multi-variable optimization
  • Explore the implementation of Fletcher-Reeves method in optimization problems
  • Investigate alternative optimization algorithms for high-dimensional spaces
USEFUL FOR

Mathematicians, data scientists, and optimization engineers interested in advanced techniques for multi-variable optimization and those seeking to enhance their understanding of convergence behaviors in high-dimensional functions.

brydustin
Messages
201
Reaction score
0
Just curious if Newton's method in high dimensions should always quickly converge to a min/max or saddle point. I can't seem to get the value of my gradient below 12-16; so, its not "diverging" but its not converging either. I want to avoid saddle points so I'm using Fletcher-Reeves method, but I figure if I test it with Newton-Raphson then it should at least converge to a saddle point quickly, right? (assuming my initial starting point is "good" in some sense).

thanks all
 
Physics news on Phys.org
could you please given an example? What is the function for which you cannot get the value of the gradient below 12-16?

Even in one dimension it is possible to get a function and intitial value where you just alternate between two x-values- but just a slight change in initial value will correct that.
 
The basic problem for any multivariable optimisation algorithm is the fact that (unlike the 1-D case) the optimum is usually at the bottom of a "curved valley" in N dimensions. Any method that only looks at the local gradient at a point tends to follow the steepest gradient, and the route to the optimum tends to be many short steps that "bounce off the sides" of the valley rather than "following the curve" along the bottom. A classic example of this for 2 variables (where it is fairly easy to draw pictures to see what is happening) is "Rosenbrock's banana function" (see Google).

To do multi-variable optimization efficiently, you usually need to know a lot about the function you are trying to optimize.
 

Similar threads

Undergrad Newton-Raphson Method With Complex Numbers
  • · Replies 7 ·
Replies
7
Views
3K
Undergrad Error bounds for Newton's Method
  • · Replies 13 ·
Replies
13
Views
3K
Undergrad What Exactly is Step Size in Gradient Descent Method?
  • · Replies 3 ·
Replies
3
Views
3K
Graduate Questions on Newton's Method: Investigating Convergence & Initial Guesses
  • · Replies 3 ·
Replies
3
Views
2K
Graduate How to use the Newton-Raphson method while keeping the sum constant?
  • · Replies 2 ·
Replies
2
Views
2K
Using Newton's Method to Solve and Using Fsolve in Matlab
  • · Replies 2 ·
Replies
2
Views
2K
Stationary points classification using definiteness of the Lagrangian
  • · Replies 4 ·
Replies
4
Views
2K
Graduate What is the theory behind the partial derivative method for root-finding?
  • · Replies 3 ·
Replies
3
Views
3K
Graduate Optimization with Newton's method
  • · Replies 3 ·
Replies
3
Views
2K
Undergrad Sign confusion when taking gradient (Newton's Method)
  • · Replies 1 ·
Replies
1
Views
2K