You should note that once you differentiate xncos(x), you'll get:
(xncos(x))' = nxn - 1cos(x) - xnsin(x), i.e, you have both cos, and sin function on the right, and the highest power of the polynomial is still n.
If you differentiate xnsin(x), you'll get almost the same.
(xnsin(x))' = nxn - 1sin(x) + xncos(x)
Note that, if you differentiate xncos(x), the highest power polynomial will go with sin(x), and vice versa. Note the italic, and underlined part.
So, if you want to find a function, such that its derivative is x3cos(x) (note that the power of the polynomial term is 3, and is highest), the function should look something like:
f(x) = (Ax3 + Bx2 + Cx + D) sin(x) + (Ex2 + Fx + G) cos(x)
Now, if you try to integrate the function xncos(x) by Parts, like this:
\int x ^ n \fbox{\cos x} dx = x ^ {n} \fbox{\sin x} - n \int x ^ {n - 1} \sin (x) dx
= x ^ {n} \fbox{\sin x} + n x ^ {n - 1} \cos(x) - n (n - 1) \int x ^ {n - 2} \cos (x) dx
= x ^ {n} \fbox{\sin x} + n x ^ {n - 1} \cos(x) - n (n - 1) x ^ {n - 2} \sin (x) + n (n - 1) (n - 2)\int x ^ {n - 3} \sin (x) dx = ...
So, you can see that the result will look something like xn sin(x) + n xn - 1 cos(x) - n (n - 1) xn - 2 sin(x) + ... i.e, one sin, and then one cos, and then back to one sin, and...
So, you can conclude that the constants B, D, and F in the function f(x) should all be 0, leaving you with:
f(x) = (Ax3 + Cx) sin(x) + (Ex2 + G) cos(x)
Can you get it? :)