How Does Least Squares Determine the Mean and Variance?

[zollen]

This problem projects b = (b₁,b₂...,b_m) onto the line through a = (1, 1, 1, ...1). We solve m equations ax = b in 1 unknown (by least squares).

(a) Solve a^T a $\hat{x}$ = a^T b to show that $\hat{x}$ is the mean (the average) of the b’s.
(b) Find e = b - a $\hat{x}$ and the variance ||e||² and the standard deviation ||e||
(c) The horizontal line $\hat{x}$ = 3 is closest to b = (1, 2, 6). Check that p = (3, 3 3) is perpendicular to e and find the 3 by 3 projection matrix P.

Ans(a): Because a = (1,1,1,...1), therefore a^T a = 1 + 1 + 1 +...+ 1 = 1 * m = m
And a^T b = b₁ + b₂ + ... + b_m
So $\hat{x}$ = (b₁ + b₂ + b₃ + ... + b_m) / m = b_avg

Ans(b): Need help..

Ans(c): Need help..

 

[zollen]

Ans(b):
Given $\hat{x}$ = b_avg (from Ans(a))
Therefore
e = b - a \hat{x} =<br /> \begin{bmatrix}<br /> b_1\\<br /> b_2\\<br /> ..\\<br /> b_m<br /> \end{bmatrix}<br /> ~-~<br /> \begin{bmatrix}<br /> 1\\<br /> 1\\<br /> ..\\<br /> 1<br /> \end{bmatrix}<br /> ~b_{avg}<br /> =<br /> \begin{bmatrix}<br /> e_1\\<br /> e_2\\<br /> ..\\<br /> e_m<br /> \end{bmatrix}<br />

||e||^2~=~\frac{{e_1}^2 + {e_2}^2 + ... + {e_m}^2}{m}
||e||~=~\sqrt{ \frac{{e_1}^2 + {e_2}^2 + ... + {e_m}^2}{m} }

Ans(c): p=Pb~~and~~P= \frac{a * a^t}{a^t * a}
a = \begin{bmatrix}<br /> 1\\<br /> 1\\<br /> 1<br /> \end{bmatrix}
But...
a^t * a<br /> ~=~<br /> \begin{bmatrix}<br /> 1&amp;1&amp;1\\<br /> 1&amp;1&amp;1\\<br /> 1&amp;1&amp;1<br /> \end{bmatrix} has no inverse...

Would anyone show me how to calculate the projection matrix?

 

[StoneTemplePython]

Ans(c): p=Pb~~and~~P= \frac{a * a^t}{a^t * a}
a = \begin{bmatrix}<br /> 1\\<br /> 1\\<br /> 1<br /> \end{bmatrix}
But...
a^t * a<br /> ~=~<br /> \begin{bmatrix}<br /> 1&amp;1&amp;1\\<br /> 1&amp;1&amp;1\\<br /> 1&amp;1&amp;1<br /> \end{bmatrix} has no inverse...

Would anyone show me how to calculate the projection matrix?

I'm not sure what you're trying to do here.

$\mathbf {aa}^T$ is a rank one matrix with trace of 3.

$\mathbf {a}^T \mathbf a = 3$ i.e. it is a scalar.

of course by cyclic property of trace and the fact that the trace of a scalar is that scalar, we have$\mathbf {a}^T \mathbf a =trace\big(\mathbf {a}^T \mathbf a\big) = trace\big(\mathbf {aa}^T\big) = 3$

Projectors (that aren't the identity) are always not invertible. Your projector is simply $\propto \mathbf {aa}^T$ i.e. you re-normalize its sole non-zero eigenvalue to be one. Then as usual, the trace operation gives you the rank of your projector.

A couple of concerns:

I'm not sure why you introduced a $*$ operation, or why you divide by "$\mathbf {a}^T \mathbf a$" in your projector formula without recognizing that it is a scalar -- in general 'dividing by a matrix' doesn't make any sense except for the very special case when it is in fact a non zero scalar that we're talking about. (Hadamard products are excluded from this writeup.)