it's been ten years since I've done this but i'll give it a shot.
(u'sub1 sinx + u'sub2 cosx)= 0
and
y''subp + ysubp = u'sub1 cosx - u'sub2 sinx = tan x.
first of all, i think you are free to set (u'sub1 sinx + u'sub2 cosx) equal to anything, as it is an asumption, i think, and a useful assumption that leads to solutions. not sure about the deeper reasons behind this assumption yet; it probably has to do with linear algebra and "linear independence" so you can't know much more than "it works" yet. but it gives you two equations and two unknowns: u1 and u2. it's a system of first order differential equations, a reduction from a second order. though now there are two equations.
let's look at it this way, and I'm going to assume you did all the derivatives right because i didn't check those:
u1' sinx + u2' cosx=0
u1' cosx - u2' sinx=tanx.
same idea in a system of linear equations: pick an equation and solve for a variable and then substitute it back. our approach will be the same except that we'll solve for u1' and u2' first and then integrate. then the final step is to say that y=u1sinx+u2cosx is a particular solution.
the first equation let's you solve for u1' in terms of u2':
u1' sinx=-u2'cosx, then divide by sinx to get
u1'=-u2' cotx. (cot=cos/sin)
now sub into the equation u1' cosx - u2' sinx=tanx to get:
(-u2' cotx) cosx - u2' sinx=tanx
-u2' cotx cosx - u2' sinx=tanx. now what do we have to do?
solve for u2'.
let's get rid of - and factor out u2':
u2' (cotx cosx + sinx)=-tanx
so u2'=-tanx/(cotx cosx + sinx).
you may have the feeling that we should simplify it.
one thing that often works for trig is changing everything to sine and cosine:
u2'=-(sinx/cosx)/((cosx/sinx) cosx + sinx)
there are a number of ways you can handle this but what smells best to me is to multiply by sinx/sinx:
u2'=-(sin
2x/cosx)/((cosx) cosx + sin
2x).
now (cosx) cosx + sin
2x=1 by trig identity so we have:
u2'=-(sin
2x/cosx).
remember that u1'= -u2' cotx, so
u1'=-(-(sin
2x/cosx)) cotx. since cot=cos/sin, we get
u1'=sinx.
at this point, we must solve for u1 and u2 by integrating. u1=-cosx but u2 is harder. I'm going to let you look that one up (you may want to convert to -sinx tanx). assuming you get an integral for u2, let's just call it u2 and use u1=-cosx.
yp=-cosx sinx+u2 cosx is a particular solution. then the general solution is y=yp+c1 sinx +c2 cosx.
i'm suspecting since that integral is not nice that a calculation is wrong somewhere in what you did but if it's all correct, this is the solution i get with mathematica's dsolve command which took about 0.01 seconds.
let F(y)=y''+y. the ultimate goal is to find a y such that F(y)=tanx. but let's look at our solution which has the form yp + c1 sinx + c2 cosx. F distributes over addition, so
F(yp + c1 sinx + c2 cosx)=F(yp)+F(c1 sinx + c2 cosx)=tanx+0. this can be generalized to whenever the F distributes over addition: find a particular solution and add that to the homogeneous solution "space." that looks like a coset to me which means there's some kind of modular spaces involved unless I'm off my rocker.