1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Least square estimate problem

  1. Jun 9, 2016 #1
    1. The problem statement, all variables and given/known data
    Suppose that ##Y \sim N_n\left(X\beta,\sigma^2I\right)##, where the density function of ##Y## is
    $$\frac{1}{\left(2\pi\sigma^2\right)^{\frac{n}{2}}}e^{-\frac{1}{2\sigma^2}(Y-X\beta)^T(Y-X\beta)},$$
    and ##X## is an ##n\times p## matrix of rank ##p##.
    Let ##\hat{\beta}## be the least squares estimator of ##\beta##.

    Show that ##(Y-X\beta)^T(Y-X\beta) = \left(Y-X\hat{\beta}\right)^T(Y-X\hat{\beta})+\left(\hat{\beta}-\beta\right)^TX^TX\left(\hat{\beta}-\beta\right)## and therefore that ##\hat{\beta}## is the least squares estimate.
    Hint: ##Y-X\beta = Y-X\hat{\beta}+X\hat{\beta}-X\beta##.

    2. Relevant equations


    3. The attempt at a solution
    I have no idea where to start. Do I substitute the hint into ##(Y-X\beta)^T(Y-X\beta)## and expand out the brackets???

    Please help!!!
     
  2. jcsd
  3. Jun 9, 2016 #2

    andrewkirk

    User Avatar
    Science Advisor
    Homework Helper
    Gold Member

    There seems to be something odd about how this problem is stated. It asks the student to assume that ##\hat\beta## is the least squares estimator of ##\beta## - and then to use that to prove that it is the least squares estimate. Are they trying to draw a distinction between estimator and estimate? If not, the problem is trivial. However if we want to get very precise about terminology I would have thought that an estimator is a function whereas the estimate is the result of the function. Is there some particular meaning of 'estimator' and 'estimate' that they are using in your course?

    As to how to proceed to prove their formula, yes substitution along the lines you mention sounds a good way to start. You can rewrite the RHS of the hint as ##(Y-X\hat\beta)+X(\hat\beta-\beta)##. Expanding out then gives us a right hand side that is what they show above, plus
    $$2(X(\hat\beta-\beta))^T(Y-X\hat\beta)$$
    So this needs to be shown to be zero. However it seems to me that should be impossible, since it is a function of the unknown parameter vector ##\beta##, which can be changed without changing any of the other elements in the formula (##X,Y,\hat\beta##) .

    Are you sure there wasn't an expectation operator around that equation they want you to prove, or some other constraining condition?
     
  4. Jun 9, 2016 #3
    Nope that's the question asked.
     
  5. Jun 10, 2016 #4

    Ray Vickson

    User Avatar
    Science Advisor
    Homework Helper

    Let ##Q(\beta) = (Y - X \beta)^T (Y - X \beta)##. If you write ##\beta = b + e## you can expand ##Q(b+e)## as a quadratic in ##e##. It will have 0-order terms (not containing ##e##), first-order terms (linear in ##e##) and second-order terms (of the form ##e^T M e## for some matrix ##M## that depends on ##X, Y## and ##b##). However, if you choose ##b## correctly, the terms of first-order in ##e## will vanish, leaving you with only zero-order and second-order terms in ##e##. That will happen when ##b = \hat{\beta}##. You will obtain the expression you are being asked to prove, where ##e = \beta ##-
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted



Similar Discussions: Least square estimate problem
  1. Least squares problem (Replies: 4)

Loading...