Orthogonal basis to find projection onto a subspace

[Mr Davis 97]

I know that to find the projection of an element in R^n on a subspace W, we need to have an orthogonal basis in W, and then applying the formula formula for projections.
However, I don;t understand why we must have an orthogonal basis in W in order to calculate the projection of another vector onto W. Why can't we use just some normal basis in W. Is it to make the computation easier, or is there a deeper reason why we can't find projections without orthogonal bases?

 

[andrewkirk]

It's not that we can't calculate the projection without an orthogonal basis for W, it's that the usual and simplest method of calculating that projection has to use an orthogonal basis. That method is as follows. Let $B=\{\vec v_1,...,\vec v_m\}$ be an orthogonal basis for subspace $W$ of vector space $V$ and let $u\in V$.

Then the projection $\pi_W(\vec u)$ of $\vec u$ onto $W$ is:
$$\sum_{k=1}^m \vec v_k\frac{\vec u\cdot \vec v_k}{\|\vec v_k\|^2}$$

That formula doesn't work if $B$ is not orthogonal because it will double count some components of the projection vector, because the terms of that sum may not be mutually orthogonal. To convince yourself of that, try working out the projection of $(1, 1, 1)$ onto the vector space spanned by orthogonal basis $(0,1,0),\ (0, 0,1)$ and compare that with what you get when you use the above formula with non-orthogonal basis $(0,1,0),\ (0, 1,1)$.

In most cases (possibly always?) it is possible to derive an orthogonal basis from a non-orthogonal one, so that the above method can be used.

 

[Mr Davis 97]

It's not that we can't calculate the projection without an orthogonal basis for W, it's that the usual and simplest method of calculating that projection has to use an orthogonal basis. That method is as follows. Let $B=\{\vec v_1,...,\vec v_m\}$ be an orthogonal basis for subspace $W$ of vector space $V$ and let $u\in V$.

Then the projection $\pi_W(\vec u)$ of $\vec u$ onto $W$ is:
$$\sum_{k=1}^m \vec v_k\frac{\vec u\cdot \vec v_k}{\|\vec v_k\|^2}$$

That formula doesn't work if $B$ is not orthogonal because it will double count some components of the projection vector, because the terms of that sum may not be mutually orthogonal. To convince yourself of that, try working out the projection of $(1, 1, 1)$ onto the vector space spanned by orthogonal basis $(0,1,0),\ (0, 0,1)$ and compare that with what you get when you use the above formula with non-orthogonal basis $(0,1,0),\ (0, 1,1)$.

In most cases (possibly always?) it is possible to derive an orthogonal basis from a non-orthogonal one, so that the above method can be used.

So how would you calculate the projection onto a subspace with a non-orthogonal basis? Is it usually the case that it is easier to find an orthogonal basis from a non-orthogonal basis and then use the formula you gave than it is to try to calculate the projection just from the original non-orthogonal basis?

 

[andrewkirk]

Is it usually the case that it is easier to find an orthogonal basis from a non-orthogonal basis and then use the formula you gave than it is to try to calculate the projection just from the original non-orthogonal basis?

Yes, I'd just do that. I started thinking about a method in which I used the formula with the non-orthog basis and then proceeded to deduct the overlapping components. But then I realized that what I was doing was basically just the same as what one does when one converts a non-orthogonal basis to an orthogonal one.

To convert non-orthog basis $\vec v_1,...,\vec v_m$ to an orthogonal basis $\vec u_1,...,\vec u_m$, just set
$$\vec u_1=\vec v_1$$
$$\forall k\in\{2,...,m\}:\ \vec u_k=\vec v_k-\sum_{j=1}^{k-1} \vec u_j\frac{\vec v_k\cdot \vec u_j}{\|\vec u_j\|^2}$$

 

[Erland]

If we use the standard inner product in $\mathbb R^n$, for which the standard basis is orthonormal, we can use the least square method to find the orthogonal projection onto a subspace of $\mathbb R^n$: Form the matrix $A$ whose column vectors are the given, possibly non-orthonormal, basis of the subspace (it does not even need to be a basis, the vectors just need to span the subspace). Let $\mathbf b$ be the vector you want to project onto the subspace.
Then, find the least square solution(s) $\mathbf x$ of the system $A\mathbf x=\mathbf b$, i.e. the solution(s) of the system $A^TA\mathbf x=A^T\mathbf b$. Then, $A\mathbf x$ is the orthogonal projection of $b$ onto the subspace generated by the column vectors of $A$, and $\mathbf x$ is the coordinate vector of the projection w.r.t. the given basis of the subspace. (If the columns of $A$ are not linearly independent, then $x$ is not unique.)

All this works only if we have an orthonormal basis to begin with, and express the basis vectors of the subspace w.r.t this basis (in our case, the standard basis). If we use $\mathbb P_n$, the space of real polynomials of desgee $\le n$, use the standard basis in this space, and define an inner product as an integral, then the standard basis is not orthonormal w.r.t. this inner product, and the above metod fails in this case.

 

[Hawkeye18]

The method Erland described give raise to the following simple formula for the orthogonal projection: put vectors that form a basis in $W$ as columns in a matrix, call it $A$. Then the matrix of the orthogonal projection $P_W$ is given by $$P_W= A(A^TA)^{-1}A^T. $$ To find the projection of a vector $\mathbb x$ onto $W$ you just multiply it by this matrix.