Derivative of a vector function

[Lucid Dreamer]

Hi,

I am reading through a book called "Matrix Differential Calculus" by Magnus and Neudecker. They go through taking the derivative of a vector in quadratic form that I need help with.

For $\vec{x}$ being a vector and A being a constant square matrix
$$\frac {d(\vec{x}^TA\vec{x})} {d\vec{x}}$$
$$= \frac {d\vec{x}^T}{d\vec{x}}A\vec{x} + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}}$$
$$= (\frac {d\vec{x}^T}{d\vec{x}}A\vec{x})^T + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}} {?}$$
$$= \vec{x}^TA^T\frac {d\vec{x}}{d\vec{x}} + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}}$$
$$= \vec{x}^T(A+A^T)\frac {d\vec{x}}{d\vec{x}}$$

$$\frac {d(\vec{x}^TA\vec{x})} {d\vec{x}}=\vec{x}^T(A+A^T)$$

I don't understand how one can go from the second line to the third line

 

[b17m4p]

It's because $\frac {d\vec{x}^T}{d\vec{x}}A\vec{x}$ is a scalar quantity.

 

[Lucid Dreamer]

It's because $\frac {d\vec{x}^T}{d\vec{x}}A\vec{x}$ is a scalar quantity.

I don't see how it is a scalar quantity. We proved that for $\vec{y}=A\vec{x}, \frac {d\vec{y}}{d\vec{x}}=A$ using the definition $\frac {d\vec{y}}{d\vec{x}}=[(\frac {d}{d\vec{x}})^T (\vec{y}^T)]^T$. This yields a matrix for $\frac {d\vec{y}}{d\vec{x}}$ and does as well for $\frac {d\vec{x}^T}{d\vec{x}}$.

This is the site that I am looking at
http://www.met.rdg.ac.uk/~ross/Documents/VectorDeriv.html

 

[b17m4p]

Oh, yeah sorry you're right. I don't know what I was thinking.

 

[b17m4p]

Well you could write it out in index notation

$$\frac{\partial}{\partial x_i} \left( x_j x_k A_{jk}\right) = \frac{\partial x_j}{\partial x_i} x_k A_{jk} + x_j\frac{\partial x_k}{x_i} A_{jk}$$
$$= \delta_{ji} x_k A_{jk} + x_j \delta_{ki} A_{jk}$$
$$= x_k A_{ik} + x_j A_{ji}$$
$$= x_k A_{ik} + x_k A_{ki}$$
$$= \left( \bf{A} + \bf{A}^T\right)x$$

 

[Lucid Dreamer]

Yeah. that does make sense, thanks. But does anybody know why my original line of thought is wrong?

 

[b17m4p]

The dimensional inconsistency is really in the second line, where one term is nx1 and the other 1xn. It almost seems to me like you'd have to write
$$\frac {d(\vec{x}^TA\vec{x})} {d\vec{x}}$$
$$= \frac {d\vec{x}^T}{d\vec{x}}A\vec{x} + \left(\vec{x}^TA\frac {d\vec{x}}{d\vec{x}}\right)^T$$
$$= \left((\frac {d\vec{x}^T}{d\vec{x}}A\vec{x})^T + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}}\right)^T$$
$$= \left(\vec{x}^TA^T\frac {d\vec{x}}{d\vec{x}} + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}}\right)^T$$
$$= \left(\vec{x}^T(A+A^T)\frac {d\vec{x}}{d\vec{x}}\right)^T$$
$$= (A + A^T)\vec{x}$$