1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Homework Help: Finding direction(s) of fastest increase/decrease

  1. Dec 11, 2011 #1
    1. The problem statement, all variables and given/known data


    I am only concerned with part (a) of this problem.

    2. Relevant equations

    This question was assigned in a linear algebra class while we were learning about eigenvalues and eigenvectors (I have since finished the class, but since this problem was assigned in the last week of the class before finals, I wasn't able to find enough time to go get help for this question).

    3. The attempt at a solution

    I learned in a previous calculus course that the gradient vector points in the direction of fastest increase, and that the negative of the gradient vector points in the direction of the greatest decrease. I figured I would use this as a check after solving it using the linear algebra method (as was the way this problem was "supposed" to be solved in this class).

    So, I tried finding the gradient:

    f(x, y) = 3x2 + 6xy - 5y


    [itex]\nabla[/itex]f = < 6x + 6y, 6x - 10y2 >

    However, evaluating this at (0, 0, 0) to determine the direction of greatest change yields <0,0>.

    The actual answer (the linear algebra way) is that the directions of fastest increase/decrease correspond to the directions of the eigenvectors of the matrix that represents the given quadratic form.* I have a few issues with this:

    1) Why do the eigenvectors' directions correspond to the direction of fastest increase/decrease? I learned before that the direction of the gradient pointed in the direction of the fastest increase (and that the negative of it pointed in the direction of the fastest decrease).

    2) Looking at part (b) of this question: Based on what I said above, I would say that the angle between the two directions is 180 degrees, because the gradient and its opposite are opposite of one another. However, this turns out not to be the case according to the actual answer stated in 1).

    Any help at all is appreciated. This question has been bugging me :P

    * This:
    3 & 3 \\
    3 & -5
    \end{bmatrix} is the matrix of the quadratic form given.
    Last edited: Dec 12, 2011
  2. jcsd
  3. Dec 12, 2011 #2
    Don't you still get a vector of <0,0> for lambda = 3, and <0,0> for lambda = -5? I don't really understand eigenstuff, but that is what I got.
  4. Dec 12, 2011 #3
    I (and my TA) got eigenvalues with the associated eigenvectors as:

    lambda = -6 with eigenvector <-1,3>


    lambda = 4 with eigenvector <3,1>

    EDIT: Yeah, just doublechecked with an eigenvector calculator online.

    (Also, note how these two directions are perpendicular, which seems to contrast with the "gradient explanation" I mentioned in my earlier post.)
  5. Dec 12, 2011 #4
    Hmm...sorry, I can actually be of no help. As I understood it, with det(A-lambda*I)=0 you would have (3-lambda)(-5-lamda)=0 How do you get the values -6 and 4? not that they are wrong I just don't really get eigenvalues
  6. Dec 12, 2011 #5


    User Avatar
    Staff Emeritus
    Science Advisor
    Homework Helper
    Education Advisor

    [tex]\begin{vmatrix} 3-\lambda & 3 \\ 3 & -5-\lambda \end{vmatrix} = (3-\lambda)(5-\lambda) - 9[/tex]
    You're missing the -9.
  7. Dec 12, 2011 #6


    User Avatar
    Staff Emeritus
    Science Advisor
    Homework Helper
    Education Advisor

    Are you sure that's exactly what you were told? The eigenvectors will point in the direction of the principal axes. In other words, when you rotate the coordinate system so the x and y axes line up with the eigenvectors, the cross term in the quadratic form disappears.

    What you've said about the gradient is correct. I think you're simply misinterpreting what you were told about the eigenvectors, which is causing your confusion.

  8. Dec 12, 2011 #7


    User Avatar
    Staff Emeritus
    Science Advisor
    Gold Member

    What the gradient method does is approximates your function with a linear function, and then points in the direction that the linear function is increasing fastest. This is similar to the one dimensional case: given a graph and a point on the graph, draw the tangent line. This is the linear approximation of the one dimensional function, and the direction that the tangent line is increasing in is the direction the function is increasing in, and the direction the tangent line is decreasing in is the direction the function is decreasing in.

    Now suppose that we have a point where the derivative is zero. It doesn't make sense to ask which direction the function is increasing/decreasing in, does it? Well, if it's a local maximum or minimum no, but if you're at an inflection point then yes, there is one direction where the function is increasing and one direction where the function is decreasing. This may seem like an exceptional case but in higher dimensions, you are more likely to have a saddle point just by the nature of the problem. For example in your case the fact that one eigenvalue is positive and the other negative means that you have a saddle point. Then you can use the eigenvectors to discover the directions in which the function has the largest positive and negative concavities. It will be worth re-reading your notes to confirm that your eigenvector test only applies to points where the gradient is zero

    A question that might spark some intuition about how the higher dimensional eigenvalues/eigenvectors work: Given that I have a function f(x) and f'(0)=0, and I tell you that f''(0)=0 as well, how can you use the sign of f''(x) to the left and the right of zero to inform you as to whether the function is increasing or decreasing in each direction?
  9. Dec 13, 2011 #8
    Further expanding the determinant that you have set up:

    [tex]=-15 + 2\lambda + \lambda^2 - 9[/tex]

    So the characteristic polynomial is [tex]\det(A-\lambda\ I) = \lambda^2 + 2\lambda - 24[/tex]. Setting this equal to zero to find the roots yields:

    \lambda^2 + 2\lambda - 24 = 0 [/itex]

    [itex](\lambda + 6)(\lambda - 4) = 0[/itex]

    [itex]\lambda_{1} = -6 \ \mbox{and}\ \lambda_{2} = 4[/itex]


    Well, I'm a tad rusty on this, but couldn't it be either increasing or decreasing on either side of zero? What I mean is [itex] f(x) = x^3[/itex] satisfies both f'(0)=0 and f''(0)=0, and it is decreasing in the negative x direction (moving away from the origin into the negative x direction) and increasing going in the positive x direction. On the other hand, [itex]f(x) = -x^3[/itex] also satisfies both f'(0)=0 and f''(0)=0. However, in this case, f(x) increases going from the origin to the negative x direction and decreases going into the positive x direction.

    Are you saying since in this case, the gradient evaluated at the origin is equal to zero, that we can use eigenvectors to determine the directions in which the function has the largest negative and positive concavities (and in this problem, that would be equivalent to determining in what directions it increases/decreases most quickly)?

    What happened was I went to my TA's office hours to get help, but had to leave in the middle of the explanation for this problem to get to another class. My friend who was with me said that my TA did some sort of proof of the claim that the eigenvectors point in the direction of the fastest increase/decrease, with the eigenvector associated with the negative eigenvalue pointing in the direction of the fastest decrease and the eigenvector associated with the positive eigenvalue pointing in the direction of the fastest increase. My friend didn't copy the proof down though (D'oh!).

    So yeah, I'm not quite sure why this "eigenvector method" would only apply when the gradient is equal to zero. Based on the book used in my class--which doesn't address this specific topic--and what my professor briefly lectured about, all my knowledge is that the eigenvectors point in the directions of the principle axes, and writing the quadratic form in terms of these axes eliminates the xy term from a quadratic form.
    Last edited: Dec 13, 2011
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook