Integral that I have been trying to solve

Substitution won't help much here. This is not integrable in the elementary functions. Ignoring the first part, which is easy, the result is a constant times the difference between an elliptical integral of the first kind and an elliptical integral of the second kind.

 

Hint: Rewrite the numerator ##-\sin^2(x)## as ##c-(\sin^2(x)+c)##.

 

The *denominator* is a square root. The *numerator* is -sin(x)².

 

Did you read my first post? You can try all the substitutions you want and you will *never* be able to find an analytic solution to this problem in the elementary functions. This problem is not integrable in the elementary functions. Fortunately, there are two special functions that are very applicable to this problem, the elliptical integrals of the first and second kind. The standard definitions of these functions are

[tex]\begin{aligned}
F(\phi,k) &= \int_0^{\phi} \frac{d\theta}{\sqrt{1-k^2\sin^2(\theta)}} \\
E(\phi,k) &= \int_0^{\phi} \sqrt{1-k^2\sin^2(\theta)} \, d\theta
\end{aligned}[/tex]
Here, k is the "elliptic modulus" and must lie between 0 and 1. The restriction is reduced if we substitute k² with m:

[tex]\begin{aligned}
F(\phi;m) &= \int_0^{\phi} \frac{d\theta}{\sqrt{1-m\sin^2(\theta)}} \\
E(\phi;m) &= \int_0^{\phi} \sqrt{1-m\sin^2(\theta)} \, d\theta
\end{aligned}[/tex]
Your integral, with appropriate modifications, can be made to look very much like these latter forms.

So what's the point? Simple: You can find mathematical tables the list these elliptical integrals and software that implements them.

 

And what exactly you're going to do with it once you get it yanac? Now I don't want to get in D.H. way since I think he's doing a great job explaining to me and you how to solve this thing but seems to me there is more to it for example what happens when the expression in the root becomes negative or how exactly can I use those special functions to actually solve an integral and they do agree with numerical calculation right? Might want to check. Even if [itex]1-m\sin^2(t)<0[/itex]? I'm not rightly sure. Just saying that's all.