Rearranging a formula (Multivariate Gaussian function)

[Pi-Bond]

Homework Statement

See image, p(y|θ) is the Likelihood function which has to be rearranged in the form of equation (3). θ is a vector variable.

Homework Equations

None?

The Attempt at a Solution

I first expanded the exponent in the original function, equation (2).

(b-A\theta)^T(b-A\theta)=b^Tb - b^TA\theta - \theta^T A^T b + \theta^T A^T A \theta

Now suppose I can write the function equivalently as

C \exp\left( -\frac{1}{2} (\theta - \theta_0)^T L (\theta - \theta_0) \right)

where C represents the same constant multiplying with exp in equation (2). In this case, the exponential parts must be the same. So:

b^Tb - b^TA\theta - \theta^T A^T b + \theta^T A^T A \theta = \theta^TL\theta - \theta^TL\theta_0 - \theta_0^TL\theta + \theta_0^TL\theta_0

If this this expression is true for all θ, then all coefficients of (...)θ , θ^T(...) , θ^T(...)θ and the constants must match.

So L = A^TA and θ₀=L^-1A^Tb.

Now I don't know how to get L₀. (equation (5) ). I only have this condition left from my assumption about the constants above:

b^Tb=θ^T₀Lθ₀

There just don't seem to be enough terms in either exponent to allow the exponential part of L₀. I don't think I can add and subtract anything either...

Any ideas?

 

[Ray Vickson]

Homework Statement

See image, p(y|θ) is the Likelihood function which has to be rearranged in the form of equation (3). θ is a vector variable.

Homework Equations

None?

The Attempt at a Solution

I first expanded the exponent in the original function, equation (2).

(b-A\theta)^T(b-A\theta)=b^Tb - b^TA\theta - \theta^T A^T b + \theta^T A^T A \theta

Now suppose I can write the function equivalently as

C \exp\left( -\frac{1}{2} (\theta - \theta_0)^T L (\theta - \theta_0) \right)

where C represents the same constant multiplying with exp in equation (2). In this case, the exponential parts must be the same. So:

b^Tb - b^TA\theta - \theta^T A^T b + \theta^T A^T A \theta = \theta^TL\theta - \theta^TL\theta_0 - \theta_0^TL\theta + \theta_0^TL\theta_0

If this this expression is true for all θ, then all coefficients of (...)θ , θ^T(...) , θ^T(...)θ and the constants must match.

So L = A^TA and θ₀=LA^Tb.

Now I don't know how to get L₀. (equation (5) ). I only have this condition left from my assumption about the constants above:

b^Tb=θ^T₀Lθ₀

There just don't seem to be enough terms in either exponent to allow the exponential part of L₀. I don't think I can add and subtract anything either...

Any ideas?

Try to show that
(b-A\theta)^T(b-A\theta) = (\theta - \theta_0)^T L (\theta - \theta_0) + K for some matrix L and some constant K, and for $\theta_0$ as given in the question.

 

[Pi-Bond]

Ok.

(b-A\theta)^T(b-A\theta)=b^Tb - b^TA\theta - \theta^T A^T b + \theta^T A^T A \theta
= b^Tb - (L L^{-1}A^T b)^T\theta - \theta^T L( L^{-1}A^T b) + \theta^T L \theta
= b^Tb + \theta_0^T L \theta - \theta^T L \theta_0 + \theta^T L \theta
= b^Tb + (\theta-\theta_0)^T L \theta - \theta^T L \theta_0
= b^Tb + (\theta-\theta_0)^T L \theta - (\theta-\theta_0)^T L \theta_0 -\theta_0^T L \theta_0
= b^Tb + (\theta-\theta_0)^T L (\theta-\theta_0)-\theta_0^T L \theta_0

I used the fact that L^T=L=A^TA.

On the basis of this it seems K = b^Tb - θ₀^TLθ₀

I still can't see the origin of L₀ here though..

 

[Pi-Bond]

Bump. I haven't been able to find any leads. Anyone have an idea?

 

[pasmith]

I'm somewhat confused by the fact that if L = A^TA and \theta_0 = L^{-1}A^Tb then L^{-1} = A^{-1}(A^T)^{-1} so that
A\theta_0 = AL^{-1}A^Tb = A(A^{-1}(A^T)^{-1})A^Tb = b
which means that b - A\theta_0 = b - b = 0. So I'm at a loss to explain why the author has included the exponential in \mathcal{L}_0, when its value appears to be \exp(0) = 1.

On the other hand, I was able to show (using the additional fact that L is symmetric so that (L^T)^{-1} = (L^{-1})^T = L^{-1}) that
(\theta - \theta_0)^TL(\theta - \theta_0) = (A\theta - b)^T(A\theta - b)<br /> = (b - A\theta)^T(b - A\theta)
It's just a case of expanding the left hand side, substituting the definitions of L and \theta_0 and simplifying.

 

[pasmith]

I'm somewhat confused by the fact that if L = A^TA and \theta_0 = L^{-1}A^Tb then L^{-1} = A^{-1}(A^T)^{-1}

Ignore that: A is not square, so cannot be invertible.

The aim is to show that (b - A\theta)^T(b - A\theta) = (b - A\theta_0)^T(b - A\theta_0) + (\theta - \theta_0)^T(\theta - \theta_0)

Now the left hand side is
b^Tb - b^TA\theta - \theta^TA^Tb + \theta^TA^TA\theta

The right hand side is
b^Tb - b^TA\theta_0 - \theta_0^TA^Tb + \theta_0^TA^TA\theta_0<br /> + \theta^TL\theta - \theta^TL\theta_0 - \theta_0^TL\theta + \theta_0^TL\theta_0\\<br /> = b^Tb - (b^TA\theta_0 + \theta_0^TA^Tb) + 2\theta_0^TL\theta_0<br /> + \theta^TL\theta - \theta^TL\theta_0 - \theta_0^TL\theta \\<br /> = b^Tb - 2b^TAL^{-1}A^Tb + 2b^TAL^{-1}A^Tb<br /> + \theta^TA^TA\theta - \theta^TA^Tb - b^TA\theta \\<br /> = b^Tb -b^TA\theta - \theta^TA^Tb + \theta^TA^TA\theta<br />
as required.

 

[Pi-Bond]

Edit: there seems to be a mistake between the second and third lines. You go from -(b^TAθ₀ + θ₀^TA^Tb) to -2b^T(...)+2b^T(...)

The plus sign should be minus, and I'm not sure of where the factor of 2 comes from. Anyway I will investigate your approach.