Now, because this is a classical mechanical system, the instantaneous state of the system is given by telling the coordinates and velocities of the two pendulums. The system has some equilibrium point, which means that the pendulums will stay motionless indefinitely if they are put to rest at the equilibrium positions. Let's choose coordinates as ##x_1## and ##x_2##, which are the horizontal coordinates of pendulum 1 and pendulum 2 relative to their equilibrium points (at equilibrium ##x_1 = x_2 = 0)##.
Now, the equations of motion, which tell the acceleration of both pendulums as a function of their positions, are:
##m_1 \frac{d^{2}x_{1}(t)}{dt^2} = -k x_1 (t) + k_c (x_2 (t) - x_1 (t))##
##m_2 \frac{d^{2}x_{2}(t)}{dt^2} = -k x_2 (t) + k_c (x_1 (t) - x_2 (t))##
Here ##m_1## and ##m_2## are the masses of the pendulums, ##k## is a Hooke's law constant that depends on the strength of gravity and on the length of the strings. The parameter ##k_c## is another Hooke's law constant that tells the strength of the coupling between the pendulums. In these equations it is assumed that the oscillations of the system are small enough for the motion to be harmonic.
Solving the normal modes means that we find out numbers ##a, b, c, d, \omega_1## and ##\omega_2##, for which
##\frac{d^{2}}{dt^2}(ax_1 (t) + bx_2 (t)) = -(\omega_{1})^2 (ax_1 (t) + bx_2 (t))##
##\frac{d^{2}}{dt^2}(cx_1 (t) + dx_2 (t)) = -(\omega_{2})^2 (cx_1 (t) + dx_2 (t))##
Here the ##ax_1 + bx_2## and ##cx_1 + dx_2## are the two normal modes of the system. Mathematically they are weighted averages of the coordinates ##x_1## and ##x_2##, which behave as if they were independent uncoupled oscillators that oscillate with frequencies ##\omega_1## and ##\omega_2##.
With linear algebra that involves matrices and determinants, it can be shown that the two possible frequencies ##\omega## can be solved from a quadratic equation for ##\omega^2##:
##\left(\frac{-k - k_c}{m_1} + \omega^2 \right)\left(\frac{-k - k_c}{m_2} + \omega^2 \right) - \frac{k_c^2}{m_1 m_2}= 0##
You can solve this with Wolfram Alpha if you want to. A more detailed explanation is here:
http://users.physics.harvard.edu/~schwartz/15cFiles/Lecture3-Coupled-Oscillators.pdf
