What do the directions of eigenvalues represent?

[preet]

Background: I'm having trouble using principal component analysis to try and align two data sets.
I have two sets of 3D point data, and I can use PCA to get principal axes of the two sets of data. I do this by finding the eigenvectors of the covariance matrix for each set of data. This gives me two sets of principal axes defined by 6 eigenvectors. PCA gives me an eigenvalue that says my data corresponds strongly along an axis parallel to the eigenvector and through the centroid of the data. Therefore, the eigenvector could be pointing in the opposite direction and the axis would still be the same. I want to know the significance of the direction in which the eigenvector is pointing (what is the difference between the eigenvector, and that eigenvector * -1).

Problem: I'm trying to align the two data sets using PCA -- but I can't do this if the corresponding direction vectors are allowed to point in the opposite direction. So if the one of the axes 'x' from data set 1 is pointing close to a higher elevation, and axes 'x' from data set 2 is pointing in the opposite direction, I'm in trouble... I hope this makes some sense.

 

[bpet]

In determining eigenvectors, the choice of +/- and length is arbitrary, since every point along the axis you described is an eigenvector. So you want to compare the axes not the specific eigenvectors (i.e. the angle between the subspaces of the eigenvectors).

The situation gets more complicated if there are repeated eigenvalues (e.g. 2x2 identity matrix) where the choice of direction is even more arbitrary, e.g. {(1,0),(0,1)} and {(1,1),(1,-1)} are both valid pairs of eigenvectors for the identity matrix. However subspace angles should still be helpful in this situation.

Hope this helps. Out of curiosity what is the procedure to align the two data sets?

 

[HallsofIvy]

You mean, of course, the directions of the eigenvectors, not eigenvalues.

What they "mean" depends on the application. For example, I could write the quadratic form $x^2+ 4xy+ 3y^2$ as a matrix formula:
$$\begin{bmatrix}x & y \end{bmatrix}\begin{bmatrix}1 & 2 \\ 2 & 3\end{bmatrix}\begin{bmatrix}x \\ y \end{bmatrix}$$
Because that is a symmetric matrix, it will have two independent eigenvectors: their directions will be the axes of symmetry- for example, the two axes if this is a ellipse.

 

[Cantab Morgan]

I can think of one case where the sign of the eigenvector's coefficients matters. If you are studying Markov processes, and your matrix represents the transition probabilities, then all the physically meaningful vectors will represents probabilities. Therefore, they will have components that range 0 <= p <= 1, and which always sum to unity.

One of the eigenvalues of such matrices will be one, and the corresponding eigenvector is best represented as a vector with all non-negative components, which all sum up to one. Now, I suppose one could argue that the opposite of that vector is also the eigenvector, but it wouldn't be too useful.