# What Exactly is Step Size in Gradient Descent Method?

• I
• Dario56
In summary, gradient descent is a numerical optimization method for finding the local or global minimum of a function, commonly used in machine learning. The step size parameter, represented by ##\alpha##, is a tuning parameter that determines how far we move in the direction of the gradient vector at each iteration. Its value can be changed during optimization and must be chosen carefully to avoid missing the minimum or taking too many iterations to converge.
Dario56
Gradient descent is numerical optimization method for finding local/global minimum of function. It is given by following formula: $$x_{n+1} = x_n - \alpha \nabla f(x_n)$$ There is countless content on internet about this method use in machine learning. However, there is one thing I don't understand and which I couldn't find even though it is basic.

What exactly is step size ## \alpha ## ?

Wikipedia states that it is tunning parameter in optimization algorithm which I understand, but not enough is being said about it to be considered a definition. Dimension analysis states that its dimensions should be ## \frac {(\Delta x )^2} {\Delta y} ## which I am not sure how to interpret.

Last edited:
Because it’s a tuning parameter, you must choose one wisely. Too small and your algorithm will run too long, too large and you will miss the valley or hill you’re trying to locate.

Here’s some discussion on it:

The blogger says it may be obsolete theory but I think it may still apply to what you’re asking about.

Dario56
jedishrfu said:
Because it’s a tuning parameter, you must choose one wisely. Too small and your algorithm will run too long, too large and you will miss the valley or hill you’re trying to locate.

Here’s some discussion on it:

The blogger says it may be obsolete theory but I think it may still apply to what you’re asking about.
Thank you. I've got this in the meantime.

I had problem with understanding this parameter because I didn't look at the equation of gradient descent in vector form and it should be seen in this light since gradient is a vector valued function.

Parameter ##\alpha## basically defines how long in the direction of the gradient vector we want to go. If parameter has value for example 0,5, it means we move in the opposite direction (opposite because we subtract it from position vector ##x_n##) of the gradient vector by length equal to 0,5 value of gradient vector at the point ##x_n##.

Its value can be changed during optimization. If it is too big we can miss the minimum and if it is too small it can get too many iterations to converge.

I would say if value of gradient is big step size can be bigger and if gradient value is small that means we are close and so we need to make step size smaller not to miss the minimum we are close to.

## 1. What is the definition of step size in gradient descent method?

The step size in gradient descent method refers to the size of the increment or decrement that is made to the parameters in each iteration of the algorithm. It determines the distance that the algorithm moves in the direction of the steepest descent, and ultimately affects the convergence and accuracy of the solution.

## 2. How is the step size chosen in gradient descent method?

The step size is typically chosen through trial and error or by using a heuristic approach. It is important to strike a balance between taking large steps to converge faster and small steps to avoid overshooting the minimum. Various techniques such as line search and learning rate schedules can also be used to determine the step size.

## 3. What happens if the step size is too large in gradient descent method?

If the step size is too large, the algorithm may overshoot the minimum and fail to converge. This can lead to oscillations or divergence, resulting in a suboptimal solution or no solution at all. It is important to choose an appropriate step size to ensure the algorithm converges to the global minimum.

## 4. Can the step size change during the gradient descent process?

Yes, the step size can be adjusted during the gradient descent process. This is known as adaptive step size or dynamic learning rate. It allows the algorithm to take larger steps in the beginning when the parameters are far from the minimum and smaller steps as it gets closer to the minimum, improving convergence and stability.

## 5. How does the step size affect the speed of convergence in gradient descent method?

The step size directly affects the speed of convergence in gradient descent method. A larger step size can lead to faster convergence, but it also increases the risk of overshooting the minimum. On the other hand, a smaller step size may take longer to converge, but it is less likely to overshoot the minimum and may result in a more accurate solution. The choice of step size should be based on the problem at hand and the desired trade-off between speed and accuracy.

Replies
18
Views
2K
Replies
5
Views
1K
Replies
1
Views
2K
Replies
1
Views
2K
Replies
15
Views
13K
Replies
5
Views
1K
Replies
1
Views
4K
Replies
2
Views
793
Replies
5
Views
2K
Replies
3
Views
1K