New axis of rotation for composite of rotations in Euclidean space

[psie]

Homework Statement

As the title indicates, I'm looking for the new axis of rotation of the composite of the given two rotations in $\mathbb R^3$ below. I've worked on this exercise for hours now, but without getting the right answer: $$A=\begin{pmatrix}1&0&0\\ 0&\cos\phi&-\sin\phi\\ 0&\sin\phi &\cos\phi\end{pmatrix}\text{ and } B=\begin{pmatrix}\cos\psi&-\sin\psi&0\\ \sin\psi&\cos\psi&0\\ 0&0 &1\end{pmatrix}.$$

Relevant Equations

I can't think of a lot of relevant equations for this one, except that composite of rotations in $\mathbb R^3$ are rotations. Perhaps some trigonometric identities will come in handy too.

The way I've approached the problem so far is that I'm looking for the set $\{x\in\mathbb R^3: ABx=x\}$, since the rotation acts as the identity on the axis of rotation. The set is the null space of $AB-I$, and $$AB=\begin{pmatrix}\cos\psi&-\sin\psi&0\\ \cos\phi\sin\psi&\cos\phi\cos\psi&-\sin\phi\\ \sin\phi\sin\psi&\sin\phi\cos\psi&\cos\phi\end{pmatrix}.$$According to the answer in my book, there are 6 cases to consider:

1. Any line through the origin if $\phi=0$ and $\psi=0$.
2. The $z$-axis if $\phi=0$ and $\psi=\pi$.
3. $t(\cos\psi+1,-\sin\psi,0)$ for $t\in \mathbb R$ if $\phi=\pi$ and $\psi\neq\pi$.
4. $t(0,\cos\phi-1,\sin\phi)$ for $t\in\mathbb R$ if $\phi\neq\pi$ and $\psi=\pi$.
5. $t(0,1,0)$ for $t\in \mathbb R$ if $\phi=\psi=\pi$.
6. $t(\sin\phi(\cos\psi+1),-\sin\phi\sin\psi,\sin\psi(\cos\phi+1))$ for $t\in \mathbb R$ otherwise.

1. makes sense since $AB=I$ in this case and any line will do. 2. also makes sense since $AB=B$ in this case and this is a rotation about the $z$-axis. However, for the rest of them, except 5. (which geometrically makes sense too), I really struggle understanding how they reached those answers. I can't seem to find any structured way of solving this problem; it just gets very messy. What I've tried to do on paper is reduce $AB$ to row reduced echelon form, but that felt like a lot of work for nothing, since I didn't reach the answer as in the book.

EDIT: I edited a typo in the matrix of $AB$; however, this was only a typo from typing this into the computer, not in my computations.

 

[psie]

I can accept that this is a tedious and messy exercise, but for instance, take case 3, we have $$AB=\begin{pmatrix}\cos\psi&-\sin\psi&0\\ -\sin\psi&-\cos\psi&0\\ 0&0&-1\end{pmatrix}.$$I can't even solve this case. I get that a row reduced form of the matrix $AB-I$ is $$\begin{pmatrix}\cos\psi-1&-\sin\psi&0\\ -\sin\psi&-\cos\psi-1&0\\ 0&0&-2\end{pmatrix}\leadsto\begin{pmatrix}1&-\frac{\sin\psi}{\cos\psi-1}&0\\ 0&-\cos\psi-1+\frac{\sin^2\psi}{\cos\psi-1}&0\\ 0&0&-2\end{pmatrix}.$$Doesn't this say the second component should be $0$, though the solution says that the second component is $-\sin\psi$?

 

[psie]

I think I found a good explanation on Wikipedia for this.

 

[vela]

I get that a row reduced form of the matrix $AB-I$ is $$\begin{pmatrix}\cos\psi-1&-\sin\psi&0\\ -\sin\psi&-\cos\psi-1&0\\ 0&0&-2\end{pmatrix}\leadsto\begin{pmatrix}1&-\frac{\sin\psi}{\cos\psi-1}&0\\ 0&-\cos\psi-1+\frac{\sin^2\psi}{\cos\psi-1}&0\\ 0&0&-2\end{pmatrix}.$$

Using the half-angle trig identities, the (non-reduced) form of the matrix is
$$
\begin{pmatrix}
-2 \sin^2 \frac \psi 2 & -2 \sin \frac \psi 2 \cos \frac \psi 2 & 0 \\
-2 \sin \frac \psi 2 \cos \frac \psi 2 & -2 \cos^2 \frac \psi 2 & 0\\
0 & 0 & -2
\end{pmatrix}
$$ which gives you $(\sin \frac \psi 2) x + ( \cos \frac \psi 2) y = 0$.

 

[vela]

The system $(AB-I)\vec x = 0$ only has non-trivial solutions when it's dependent. This fact tells you that in this particular case the equations from the first and second lines are multiples of each other (even if it's not readily apparent), so you only need to consider one of them.

You can write down a solution by inspection by simply setting $x$ and $y$ to the coefficient of the other variable and flipping the sign of one. For example, from the second row, you have
$$(\sin \psi)x + (\cos \psi + 1)y = 0,$$ so just set $x = \cos \psi + 1$ and $y = -\sin \psi - 1$, which results in the book's answer.