Why do gradient show rate of maximum increase ?

[avinashj]

Why do gradient show rate of maximum increase not decrease always?

 

[physics-guy]

It shows both the max increasing and max decreasing

 

[mikeph]

The gradient vector is made of components of the directional derivatives along each axis. If a functional has zero gradient along one axis but large gradient (say 10) along a second then the gradient vector will show [0, 10], which points exactly in that direction.

However if the derivative in some direction is negative then the corresponding component in the gradient vector will be negative, and will therefore point in the opposite direction, which must be uphill. This is the reason it always points towards the direction of greatest increase.

 

[HallsofIvy]

Another way of looking at it. The vector $<cos(\theta), sin(\theta)>$ is a unit vector pointing in the direct making angle $\theta$ with the x-axis (in two dimensions).

Since derivatives are "linear", the rate of change of f in the direction $\theta$ (measured from the x-axis) is $(\partial f/\partial x)cos(\theta)+ (\partial f/\partial y)sin(\theta)$. Now, for what $\theta$ is that a maximum?\
To find max or min we differentiate with the variable (here $\theta$. We are at a specific point, so fixed x and y, varying the direction) and set that equal to 0:
$$\frac{\partial f}{\partial x}(-sin(\theta))+ \frac{\partial f}{\partial y}(cos(\theta))= 0$$

We can write that as
$$\frac{\frac{\partial f}{\partial y}}{\frac{\partial f}{\partial x}}= \frac{sin(\theta)}{cos(\theta)}= tan(\theta)$$

Which simply says that the tangent of the angle at which we have max and min is the y component of the gradient over the x component of the gradient. That is, since tangent is "opposite side over adjacent side", the direction of fastest increase is either in the direction of the gradient or opposite. The fact that the rate of increase in that direction is the length of the gradient, and length of a vector is always positive, tells us that the gradient gives us the fastest increase while the opposite direction gives the fastest decrease.

 

[chingel]

In a similar way:

When moving around a point a little, with movement being a in the x direction and b in the y direction, then the increase of the function is just the sum of how much it increased when moving in the x direction and how much it increased when moving in the y direction, meaning $(\partial f/\partial x)a+ (\partial f/\partial y)b$

Notice that the same expression can be written as the dot product of two vectors $(a, b) \cdot(\partial f/\partial x, \partial f/\partial y)$. Keeping the length of the movement vector constant, this expression is maximal when the vectors point in the same direction, because the dot product multiplies both of their lengths with cosine of the angle between them. So the direction of the gradient vector we usefully defined gives the direction of the largest increase and also the maximal increase per unit length, as can be seen by taking an unit movement vector. Also notice for example that when the vectors point in opposite directions, the value is the smallest, ie the decrease is the largest.