Solving Quadratic Form Derivative: A Frustrating Challenge

[Cyrus]

I'm not sure how you apply the rules of a derivative on a quadratic form. I've been trying to find the solution on google but no luck:

Basicallly:

J=\frac{1}{2} (z-X \theta)^T (z-X \theta)

and

\frac{ \partial J}{\partial \theta}= -X^T z +X^TX\theta

I can't for the life of me figure out how they got from the upper equation to the lower equation. The reason is that the transpose is really screwing things up in terms of the deriatives. There is some rule being applied to matrix differentiation of a transpose of a quadratic form that I am ignorant of, which won't let me get to the same expression on the second line...

Every time I try to expand the top line out I end up with 2*cross product term that doesn't drop out, but is clearly not shown in the second line.

 

[tiny-tim]

Hi Cyrus!

It's the usual product rule: (fg)' = f'g + fg',

which here is (f^Tg)' = f'^Tg + f^Tg'

Since f = g, that comes out as f'^Tf + f^Tf'

and since z (i assume that mean zI) and θ commute with anything, you should get the given result

 

[Cyrus]

Hi Cyrus!

It's the usual product rule: (fg)' = f'g + fg',

which here is (f^Tg)' = f'^Tg + f^Tg'

Since f = g, that comes out as f'^Tf + f^Tf'

and since z (i assume that mean zI) and θ commute with anything, you should get the given result

This isn't working. To be clear z and theta are vectors, not scalars.

After expanding I am getting

(-X^Tz+X^TX\theta) -(Xz^T + X\theta^TX^T)


If the two things in brackets could equal twice each term, then the one half would knock out the two and make things right.

Basically, Xz^T = X^T z

 

[waht]

Got the same thing

Xz^T = X^T z


if this is to be true, then Xz must be symmetric

A symmetric matrix is when it's equal to its transpose

A = A^T

 

[tiny-tim]

This isn't working. To be clear z and theta are vectors, not scalars.

After expanding I am getting

(-X^Tz+X^TX\theta) -(Xz^T + X\theta^TX^T)

If the two things in brackets could equal twice each term, then the one half would knock out the two and make things right.

Basically, Xz^T = X^T z

No, the ^T must always come first: X^T z = z^T X.

I'm honestly not followng this …

if θ is a vector, how can you differentiate with respect to it?

and you originally called it a quadratic … what's quadratic about it unless each bracket is a vector?

What is the context of this?