Differentiation with matrices/vectors

[mikeph]

Hello,

I'm trying to understand this proof:

http://en.wikipedia.org/wiki/Proofs...st_squares#Least_squares_estimator_for_.CE.B2

Can someone quickly talk me through the differentiation step, bearing in mind I've never learn how to differentiate with respect to a vector?

Most confusing for me is:

1. why are they differentiating with respect to the transpose b' rather than just b?
2. where does the -2X'y term come from?
3. is there any assumption here that X is square?

Thanks for any help,
Mike

 

[fresh_42]

It is easier with the Leibniz rule: $S(\beta)=(y-X\beta)^\tau(y-X\beta)$. Hence differentiation with respect to $\beta$ is
\begin{align*}
S(\beta)'&=[(y-X\beta)^\tau]'\cdot (y-X\beta) + (y-X\beta)^\tau \cdot (y-X\beta)'\\
&=-X^\tau\cdot (y-X\beta) + (y-X\beta)^\tau\cdot (-X)\\
&=-X^\tau y + X^\tau X\beta -y^\tau X + \beta^\tau X^\tau X\\
&=-2X^\tau y +2X^\tau X\beta
\end{align*}
as matrix times column vector equals row vector times transpose matrix and at the evaluation point $\beta=\hat \beta$ we get
$$
\dfrac{dS}{d\beta}(\hat \beta) = S(\beta)'|_{\beta=\hat \beta} = -2X^\tau y +2X^\tau X \hat \beta
$$