Regression SS in multiple linear regression

**kingwinner** · Jun20-09, 08:04 AM

In MULTIPLE linear regression, is it still true that the regression sum of squares is equal to
∑ (Y_i hat -Y bar)^2 ???

My textbook defines regression SS in the chapters for simple linear regression as ∑ (Y_i hat -Y bar)^2, and then in the chapters for multiple linear regression, the regression SS is defined in MATRIX form, and it did not say anywhere whether it is still equal to ∑ (Y_i hat -Y bar)^2 or not, so I am confused...

If it is still equal to ∑ (Y_i hat -Y bar)^2 in MULTIPLE linear regression (this is such a simple formula), what is the whole point of expressing the regression SS in terms of matrices in mutliple linear regression? I don't see any point of doing so when the formula ∑ (Y_i hat -Y bar)^2 is already so simple. There is no need to develop additional headaches...

Thanks for explaining!

**statdad** · Jun22-09, 01:39 PM

I think you have notation (and/or terms) confused. In simple linear regression

$LaTeX Code: \\begin{align*} SSTO & = \\sum(Y_i - \\bar Y)^2 \\\\ SSE & = \\sum (Y_i - \\hat Y_i)^2 \\\\ SSR & = SSTO - SSE = \\sum (\\hat Y_i - \\bar Y)^2 \\end{align*} $

In multiple linear regression, with matrix notation,

$LaTeX Code: \\begin{align*} SSTO & = \\mathbf{Y}single-quote\\mathbf{Y} - n \\bar{Y}^2 \\quad(=\\sum (Y_i - \\bar Y)^2)\\\\ SSE & = \\hat{e}single-quote \\hat{e} = \\mathbf{Y}single-quote \\mathbf{Y} - \\hat{\\mathbf{\\beta}}single-quote \\mathbf{X}single-quote \\mathbf{Y} \\quad (=\\sum (Y_i - \\hat Y_i)^2) \\\\ SSR & = SSTO - SSE = \\hat{\\mathbf{\\beta}}single-quote \\mathbf{X}single-quote \\mathbf{Y} - n \\bar{Y}^2 \\end{align*} $

The matrix approach isn't here simply to cause confusion: in multiple linear regression the "nice" approach of drawing pictures to represent things breaks down. However, a little linear algebra can be used to describe exactly why the residuals sum to zero, why the different quantities have different degrees of freedom, as well as provide convenient ways to generate tests (there are many theorems that describe the probability distribution of different quadratic forms of multivariate normal distributions: using matrices in multiple regression allow these theorems to be used to develop hypothesis tests.)

On a more basic level: imagine trying to derive the normal equations (to estimate the regression coefficients) by algebra rather than via the matrix approach. It isn't fun.

As one more: example:

The fitted values in multiple regression can be written as

$LaTeX Code: \\hat Y = X \\left(Xsingle-quoteX\\right)^{-1} Xsingle-quote Y \\equiv P_V Y $

where $LaTeX Code: P_V = X \\left(Xsingle-quoteX\\right)^{-1} Xsingle-quote$ is a projection matrix onto the space spanned by the columns of

.

The residuals are

$LaTeX Code: \\hat e & = Y - \\hat Y = \\left(I - X \\left(Xsingle-quoteX\\right)^{-1} Xsingle-quote\\right) Y \\equiv P_{\\hat V} Y $

where $LaTeX Code: P_{\\hat V} = I - X \\left(Xsingle-quoteX\\right)^{-1} Xsingle-quote$ is the projection onto the space orthogonal to the column space of

.

Now

$LaTeX Code: \\hat esingle-quote \\hat Y = Ysingle-quote P_{\\hat V} \\left(I - P_{\\hat V}\\right) Y = Ysingle-quote \\left(P_{\\hat V} - P_{\\hat V}^2\\right) Y = Ysingle-quote \\left(P_{\\hat V} - P_{\\hat V}\\right) Y = 0 $

or, in short,

$LaTeX Code: \\sum \\hat{e}_i \\hat{y}_i = 0 $

just as in linear regression.