perturbation theory - define a small perturbation z (just a little displacement), then stick that into the Magnetohydrodynamic momentum equation (mdu/dt = pressure gradient) where pressure a sum of kinetic (P=nKT) and magnetic pressure (magnetic field lines actually repel each other, and are 'tied' to the plasma in ideal MHD theory).
Anyway you get something like mz'' = F(z), which is the same as mx''=F(x), with z being the small perturbation displacement. F(z) is very long and complicated, but has a mathematical property of self-adjointedness that basically leads you to know that the w in z=z0*exp(-iwt), which is one of many modes (solutions) to the mz'' = F(z) is either purely real (the displacement leads to oscillation, and the plasma is stable to that mode) or purely imaginary (exponential growth, not stable) - so you can look for marginal stability conditions where w=0. mw^2z=0=F(z), which simplifies things. This is the normal mode analysis. There is also an energy pronciple, where you look for solutions to a pertubation energy (not momentum) equation, where again your x is the perturbation z. Then if dW>0 it's stable, if not, plasmas f***ed. Now I might have some details wrong.
Ch 6 of Gurnett Introduction to Plasma Physics is good
But I think Freidberg's Theory of Fusion Systems and MHD books are the standards for this