1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Homework Help: Rearranging a formula (Multivariate Gaussian function)

  1. Dec 16, 2012 #1
    1. The problem statement, all variables and given/known data
    See image, p(y|θ) is the Likelihood function which has to be rearranged in the form of equation (3). θ is a vector variable.

    2. Relevant equations

    3. The attempt at a solution
    I first expanded the exponent in the original function, equation (2).

    [tex](b-A\theta)^T(b-A\theta)=b^Tb - b^TA\theta - \theta^T A^T b + \theta^T A^T A \theta[/tex]

    Now suppose I can write the function equivalently as

    [tex]C \exp\left( -\frac{1}{2} (\theta - \theta_0)^T L (\theta - \theta_0) \right)[/tex]

    where C represents the same constant multiplying with exp in equation (2). In this case, the exponential parts must be the same. So:

    [tex]b^Tb - b^TA\theta - \theta^T A^T b + \theta^T A^T A \theta = \theta^TL\theta - \theta^TL\theta_0 - \theta_0^TL\theta + \theta_0^TL\theta_0 [/tex]

    If this this expression is true for all θ, then all coefficients of (...)θ , θT(...) , θT(...)θ and the constants must match.

    So L = ATA and θ0=L-1ATb.

    Now I don't know how to get L0. (equation (5) ). I only have this condition left from my assumption about the constants above:


    There just don't seem to be enough terms in either exponent to allow the exponential part of L0. I don't think I can add and subtract anything either...

    Any ideas?
    Last edited: Dec 16, 2012
  2. jcsd
  3. Dec 16, 2012 #2

    Ray Vickson

    User Avatar
    Science Advisor
    Homework Helper

    Try to show that
    [tex] (b-A\theta)^T(b-A\theta) = (\theta - \theta_0)^T L (\theta - \theta_0) + K[/tex] for some matrix L and some constant K, and for ##\theta_0## as given in the question.
  4. Dec 16, 2012 #3

    [tex](b-A\theta)^T(b-A\theta)=b^Tb - b^TA\theta - \theta^T A^T b + \theta^T A^T A \theta[/tex]
    [tex]= b^Tb - (L L^{-1}A^T b)^T\theta - \theta^T L( L^{-1}A^T b) + \theta^T L \theta[/tex]
    [tex]= b^Tb + \theta_0^T L \theta - \theta^T L \theta_0 + \theta^T L \theta[/tex]
    [tex]= b^Tb + (\theta-\theta_0)^T L \theta - \theta^T L \theta_0[/tex]
    [tex]= b^Tb + (\theta-\theta_0)^T L \theta - (\theta-\theta_0)^T L \theta_0 -\theta_0^T L \theta_0 [/tex]
    [tex]= b^Tb + (\theta-\theta_0)^T L (\theta-\theta_0)-\theta_0^T L \theta_0 [/tex]

    I used the fact that LT=L=ATA.

    On the basis of this it seems K = bTb - θ0T0

    I still can't see the origin of L0 here though..
  5. Dec 20, 2012 #4
    Bump. I haven't been able to find any leads. Anyone have an idea?
  6. Dec 20, 2012 #5


    User Avatar
    Homework Helper

    I'm somewhat confused by the fact that if [itex]L = A^TA[/itex] and [itex]\theta_0 = L^{-1}A^Tb[/itex] then [itex]L^{-1} = A^{-1}(A^T)^{-1}[/itex] so that
    [tex]A\theta_0 = AL^{-1}A^Tb = A(A^{-1}(A^T)^{-1})A^Tb = b[/tex]
    which means that [itex]b - A\theta_0 = b - b = 0[/itex]. So I'm at a loss to explain why the author has included the exponential in [itex]\mathcal{L}_0[/itex], when its value appears to be [itex]\exp(0) = 1[/itex].

    On the other hand, I was able to show (using the additional fact that [itex]L[/itex] is symmetric so that [itex](L^T)^{-1} = (L^{-1})^T = L^{-1}[/itex]) that
    [tex](\theta - \theta_0)^TL(\theta - \theta_0) = (A\theta - b)^T(A\theta - b)
    = (b - A\theta)^T(b - A\theta)[/tex]
    It's just a case of expanding the left hand side, substituting the definitions of [itex]L[/itex] and [itex]\theta_0[/itex] and simplifying.
  7. Dec 21, 2012 #6


    User Avatar
    Homework Helper

    Ignore that: A is not square, so cannot be invertible.

    The aim is to show that [itex](b - A\theta)^T(b - A\theta) = (b - A\theta_0)^T(b - A\theta_0) + (\theta - \theta_0)^T(\theta - \theta_0)[/itex]

    Now the left hand side is
    [tex]b^Tb - b^TA\theta - \theta^TA^Tb + \theta^TA^TA\theta[/tex]

    The right hand side is
    [tex]b^Tb - b^TA\theta_0 - \theta_0^TA^Tb + \theta_0^TA^TA\theta_0
    + \theta^TL\theta - \theta^TL\theta_0 - \theta_0^TL\theta + \theta_0^TL\theta_0\\
    = b^Tb - (b^TA\theta_0 + \theta_0^TA^Tb) + 2\theta_0^TL\theta_0
    + \theta^TL\theta - \theta^TL\theta_0 - \theta_0^TL\theta \\
    = b^Tb - 2b^TAL^{-1}A^Tb + 2b^TAL^{-1}A^Tb
    + \theta^TA^TA\theta - \theta^TA^Tb - b^TA\theta \\
    = b^Tb -b^TA\theta - \theta^TA^Tb + \theta^TA^TA\theta
    as required.
    Last edited: Dec 21, 2012
  8. Dec 22, 2012 #7
    Edit: there seems to be a mistake between the second and third lines. You go from -(bT0 + θ0TATb) to -2bT(....)+2bT(....)

    The plus sign should be minus, and I'm not sure of where the factor of 2 comes from. Anyway I will investigate your approach.
    Last edited: Dec 22, 2012
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook