If your equation is
[tex]Au_j = Bu_{j-1}[/tex],
then
[tex]u_j = A^{-1}Bu_{j-1}[/tex]
and
[tex]u_{j+1} = A^{-1}Bu_{j}= \left(A^{-1}B\right)A^{-1}Bu_{j-1}[/tex],
and
[tex]u_{j+2} = A^{-1}Bu_{j-1}u_{j+1} = \left(A^{-1}B\right)^3u_{j-1}[/tex].
You can continue this on forever to get the term after n steps as
[tex]u_{j+n} = \left(A^{-1}B\right)^{n+1}u_{j-1}[/tex].
Now factor A^{-1}B by eigenvalue decomposition to obtain
[tex]A^{-1}B = TDT^{-1}[/tex]
where D is a diagonal matrix containing the eigenvalues and T contains the corresponding eigenvectors.
Note that
[tex]\left(A^{-1}B\right)^2 = \left(TDT^{-1}\right)^2 = TDT^{-1}TDT^{-1}= TD^2T^{-1}[/tex]
And similarly,
[tex]\left( A^{-1}B \right)^n = TD^nT^{-1}[/tex]
Now if you plug this into the previous equation, you find that
[tex]u_{j+n} = \left(A^{-1}B\right)^{n+1}u_{j-1} = TD^{n+1}T^{-1}u_{j-1}[/tex]
The system is stable if the solution u_{j+n} is bounded for all n. Since D is a diagonal matrix, D^{n+1} is just the diagonal elements raised to the n+1th power. So what happens if a number bigger than one is raised to a large power? And what about when the number is smaller than one? This is why you have the condition on the size of the eigenvalues.