TimeRip496 said:
I think I get it already!
$$\omega_y=k_ycos(ct)+\frac{ak_x}{c}sin(ct)$$
$$C_1=k_y \ \ \ C_2=\frac{ak_x}{c}$$
$$\alpha=0 \ \ \ \ \beta=c \ \ \rightarrow \ \ \ \lambda=+/-ic$$
$$\lambda^2+c^2=0 \ \ [Solved]$$
As for doing it without Laplace transform,
$$\dot\omega_x-a\omega_y=0$$
$$\dot\omega_y-a\omega_x=0 \ \ \ \rightarrow \ \ \ \omega_y=-b\int\omega_xdt$$
$$\dot\omega_x+ab\int\omega_xdt=0$$
$$\lambda^2+ab=0 \ \ \ \ [Solved]$$
OK, good: you have the solution. Now I can pull back the curtain to reveal the solution using Laplace transforms.
We have found the Laplace transforms ##W_x(s)## and ##W_y(s)## as
$$W_x(s) = \frac{s\, \omega_x(0) + a\, \omega_y(0)}{s^2 + ab} \\ W_y(s) = \frac{s\, \omega_y(0) -b\, \omega_x(0)}{s^2+ab}.$$
The characteristic equation is found by setting the denominators to zero: that is, ##s^2 + ab = 0##.
Note that the initial conditions ##\omega_x(0), \omega_y(0)## appear only in the numerators, not the denominators. That is why they do not appear in the characteristic equation!
In general, if we have a system state-variable ##y(t)## whose Laplace transform ##Y(s)## has the form of a ratio of two polynomials, that is,
$$Y(s) = \frac{N(s)}{D(s)} $$
for some polynomials ##N(s)## and ##D(s)##, the characteristic equation is just ##D(s) = 0.##
The roots of the characteristic equation correspond to the "fundamental modes" of the system, so that if ##r## is a root of multiplicity ##p## -- that is, one factor of ##D(s)## is ##(s-r)^p## -- then ##y(t)## will contain terms that are constant multiples of ##t^{p-1} e^{rt}/(p-1)!## and some of its time-derivatives. If all the roots are simple (all ##p = 1##) then ##y(t)## is a linear combination of the functions ##e^{rt}##. (Note that time-derivatives of ##e^{rt}## are just constant multiples of ##e^{rt}## itself, so taking time-derivatives does not introduce new types of terms.)
In your case the roots are ##\pm i \sqrt{ab}## so you have ##e^{\pm i \sqrt{ab} t},## which are equivalent to ##\sin( \sqrt{ab} t)## and ##\cos(\sqrt{ab} t).##