Maximizing quantity which is a product of matrices/vectors

[AcidRainLiTE]

I am trying to follow the following reasoning:

Given a known matrix A, we want to find w that maximizes the quantity

w'Aw​

(where w' denotes the transpose of w) subject to the constraint w'w = 1.

To do so, use a lagrange multiplier, L:

w'Aw + L(w'w - 1)
​

and differentiate to obtain

Aw = Lw.​

Thus, we seek the eigenvector of A with the largest eigenvalue.​

I do not understand how they differentiated w'Aw + L(w'w-1) to get Aw = Lw. Can someone explain to me what is going on at that step?

 

[fzero]

It would probably help to write things down explicitly in terms of components,

$$ m = w'Aw + L(w'w - 1) = \sum_{ij} A_{ij} w_i w_j + L \left( \sum_i w_i^2 -1 \right).$$

This is a function of the $n$ variables $w_i$. At an extremum, $\partial m /\partial w_k =0$. If you actually work out this set of equations, you'll see they are the components of the eigenvalue equation that you quoted. You'll need to use the fact that $A$ can be taken to be a symmetric matrix.

 

[Bacle2]

For a more general theory, see, e.g:

http://en.wikipedia.org/wiki/Matrix_calculus

 

[AcidRainLiTE]

Both posts were helpful. Thanks.

 

[blue_raver22]

Sure, I can explain what is happening at that step. Let's break it down:

First, we have the function w'Aw + L(w'w-1), where w is a vector and A is a matrix. This function represents the quantity that we are trying to maximize, subject to the constraint that w'w = 1.

Next, we use a technique called Lagrange multipliers to solve this optimization problem. This technique involves introducing a new variable, L, called the Lagrange multiplier, and adding it to the objective function.

When we differentiate the objective function with respect to w, we treat L as a constant. This means that when we differentiate w'Aw, we get A as the derivative, since L is treated as a constant. Similarly, when we differentiate L(w'w-1), we get Lw, since w'w is just a scalar and the derivative of a constant is 0.

Next, we set these two derivatives equal to each other, since they represent the same quantity (the derivative of the objective function with respect to w). This gives us Aw = Lw, which is the equation that we are left with after differentiation.

This equation tells us that the vector w is an eigenvector of A, with the eigenvalue L. This means that in order to maximize the quantity w'Aw, we need to find the eigenvector of A with the largest eigenvalue, since that will give us the largest possible value for w'Aw.

I hope this explanation helps you understand the reasoning behind this approach. Let me know if you have any further questions.