Your particular solution yp = Ae-tcos(2t) + Be-tsin(2t) turns out to be the general solution of the related homogeneous problem, y'' + 2y' + 5y = 0.
If you substitute yp into the latter equation you should find that yp'' + 2yp' + 5yp is identically equal to zero.
For the nonhomogeneous problem, the right side is 4e-tcos(2t), so instead of choosing yp = Ae-tcos(2t) + Be-tsin(2t) as you did, you should use yp = Ate-tcos(2t) + Bte-tsin(2t) as your particular solution. Run this function through your differential equation and determine the parameters A and B.
The general solution to the nonhomogeneous problem will be
yg = c1e-tcos(2t) + c2e-tsin(2t) + Ate-tcos(2t) + Bte-tsin(2t), using the values of A and B that you already found. Use the initial conditions to find c1 and c2.
The reason for doing things this way has to do with repeated roots of the characteristic equation. The explanation is a little lengthy, but I'll try to cover the high points as briefly as I can.
Your equation can be represented as (D2 + 2D + 5)y = 4e-tcos(2t). Here I am using operator notation, D, to represent the derivative with respect to t.
Notice the similarity between the operator notation, D2 + 2D + 5, and the left side of the characteristic equation r2 + 2r + 5. That's not an accident. The roots of the characteristic equation are r = -1 +/- 2i. Each of these values causes the expression r2 + 2r + 5 to have a value of zero.
The associated function is e(-1 +/- 2i)t = e-te+/-2it. Rather than working with this, it's more convenient to use e-tcos(2t) and e-tsin(2t).
When the operator D2 + 2D + 5 is applies to either of these functions (in fact, any of the four), the result is 0. So this operator can be said to "annihilate" e-tcos(2t) and e-tsin(2t), as well as any linear combination of them.
That's why there is no linear combination of these two functions will serve as a particular solution of your nonhomogeneous problem.
Your nonhomog. problem can be written this way: (D2 + 2D + 5)y = 4e-tcos(2t). If we apply the same operator again, we get
(D2 + 2D + 5)(D2 + 2D + 5)y = (D2 + 2D + 5)4e-tcos(2t) = 0.
This turns the original nonhomogeneous, 2nd order diff. eqn into a 4th order, homogeneous problem. This time the characteristic equation is (r2 + 2r + 5)2 = 0, and now we have the same roots, only repeated.
Since there are four roots, and the equation is fourth order, we have to have four linearly independent solutions, not just the two we had before. On top of e-tcos(2t) and e-tsin(2t) that we had before, we can get two more independent solutions by multiplying each of these two by t.
That's pretty much it - hope this helps.