How to Derive the Quadratic Form of a Vector Function?

[Lucid Dreamer]

Hi,

I am reading through a book called "Matrix Differential Calculus" by Magnus and Neudecker. They go through taking the derivative of a vector in quadratic form that I need help with.

For \vec{x} being a vector and A being a constant square matrix
\frac {d(\vec{x}^TA\vec{x})} {d\vec{x}}
= \frac {d\vec{x}^T}{d\vec{x}}A\vec{x} + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}}
= (\frac {d\vec{x}^T}{d\vec{x}}A\vec{x})^T + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}} {?}
= \vec{x}^TA^T\frac {d\vec{x}}{d\vec{x}} + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}}
= \vec{x}^T(A+A^T)\frac {d\vec{x}}{d\vec{x}}

\frac {d(\vec{x}^TA\vec{x})} {d\vec{x}}=\vec{x}^T(A+A^T)

I don't understand how one can go from the second line to the third line

 

[b17m4p]

It's because \frac {d\vec{x}^T}{d\vec{x}}A\vec{x} is a scalar quantity.

 

[Lucid Dreamer]

It's because \frac {d\vec{x}^T}{d\vec{x}}A\vec{x} is a scalar quantity.

I don't see how it is a scalar quantity. We proved that for \vec{y}=A\vec{x}, \frac {d\vec{y}}{d\vec{x}}=A using the definition \frac {d\vec{y}}{d\vec{x}}=[(\frac {d}{d\vec{x}})^T (\vec{y}^T)]^T. This yields a matrix for \frac {d\vec{y}}{d\vec{x}} and does as well for \frac {d\vec{x}^T}{d\vec{x}}.

This is the site that I am looking at
http://www.met.rdg.ac.uk/~ross/Documents/VectorDeriv.html

 

[b17m4p]

Oh, yeah sorry you're right. I don't know what I was thinking.

 

[b17m4p]

Well you could write it out in index notation

\frac{\partial}{\partial x_i} \left( x_j x_k A_{jk}\right) = \frac{\partial x_j}{\partial x_i} x_k A_{jk} + x_j\frac{\partial x_k}{x_i} A_{jk}
= \delta_{ji} x_k A_{jk} + x_j \delta_{ki} A_{jk}
= x_k A_{ik} + x_j A_{ji}
= x_k A_{ik} + x_k A_{ki}
= \left( \bf{A} + \bf{A}^T\right)x

 

[Lucid Dreamer]

Yeah. that does make sense, thanks. But does anybody know why my original line of thought is wrong?

 

[b17m4p]

The dimensional inconsistency is really in the second line, where one term is nx1 and the other 1xn. It almost seems to me like you'd have to write
\frac {d(\vec{x}^TA\vec{x})} {d\vec{x}}
= \frac {d\vec{x}^T}{d\vec{x}}A\vec{x} + \left(\vec{x}^TA\frac {d\vec{x}}{d\vec{x}}\right)^T
= \left((\frac {d\vec{x}^T}{d\vec{x}}A\vec{x})^T + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}}\right)^T
= \left(\vec{x}^TA^T\frac {d\vec{x}}{d\vec{x}} + \vec{x}^TA\frac {d\vec{x}}{d\vec{x}}\right)^T
= \left(\vec{x}^T(A+A^T)\frac {d\vec{x}}{d\vec{x}}\right)^T
= (A + A^T)\vec{x}