Writing ##\cos(w) = (1/2) (e^{iw} + e^{-iw}),## you can write the right-hand-side of your DE as ##t^3 e^{5t} (1/2) (e^{i 3t} + e^{-i 3t})##, so as a sum of two terms of the form ## c \, t^3 e^{rt}##, for ##r = 5 + 3i## and ##r = 5 - 3i.##. A particular solution ##y_p## of ##y'' - 2 y' - 3y = c t^3 e^{rt}## has form
$$ (A_0 + A_1 t + A_2 t^2 + A_3 t^3) e^{rt} + B e^{3t} + Ce^{-t}.\hspace{2em}(1)$$
Back in the world of "sin" and "cos" you get a sum of two such forms for the two values of ##r##. Eventually you get a sum of such forms but with ##e^{5t} \cos(3t)## and ##e^{5t} \sin(3t)## in place of the ##e^{rt}.## In other words, you would get a solution of the form
$$y_p = P_3(t) e^{5t} \cos(3t) + Q_3(t) e^{5t} \sin(3t) + B e^{3t} + Ce^{-t}, \hspace{2em}(2) $$
where ##P_3## and ##Q_3## are two (possibly different) cubic polynomials. It is easy enough to see how starting with (1) you can get (2). However, getting the actual coefficients in ##P_3, Q_3## is lengthy and messy, so be warned.
I did not "guess" the form above; rather, I used a Laplace-transform technique to solve the DE with right-hand-side ##t^3 e^{rt}.## Normally I would solve the entire DE from start to finish using Laplace, but I refrained from doing that just so I could help with your question as it was put to you.
Also: rather than using the "method of undermined coefficients" (which require a certain amount of guesswork and trial-and-error) I would prefer to use the method of "variation of parameters". See, eg.,
http://tutorial.math.lamar.edu/Classes/DE/VariationofParameters.aspx