Right, either of these is valid, but you have to choose one and stick with it. So let's say you put ##x = \tan(a)##. So ##dx = \sec^2(a) da = da/\cos^2(a)##. Your integral becomes
$$\begin{align}
\int_0^{\pi/4} \left(\frac{1}{\cos^2(a)}\right) \left(\frac{\ln(\tan(a) + 1)}{\tan^2(a) + 1}\right) da &=
\int_0^{\pi/4} \left(\frac{\cos^2(a)}{\cos^2(a)}\right) \ln(\tan(a) + 1) da\\
&= \int_0^{\pi/4} \ln(\tan(a) + 1) da \\
\end{align}$$
I'm not sure where you can take this from here. Are you sure that a trig substitution is the right approach?
What if you try integration by parts: say ##u = \ln(x+1)## and ##dv = dx/(x^2+1)##. Then ##du = dx/(x+1)## and ##v = \arctan(x)##. Your integral becomes
$$\begin{align}
\left. uv \right|_{x=0}^{x=1} - \int_{x=0}^{x=1} v du &=
\left. \ln(x+1)\arctan(x)\right|_{0}^{1} - \int_{0}^{1} \frac{\arctan(x)}{x+1} dx \\
&= \frac{\pi}{4}\ln(2) - \int_0^1 \frac{\arctan(x)}{x+1} dx \\
\end{align}$$
According to Wolfram Alpha, the answer is supposed to be ##\frac{\pi}{8}\ln(2)##, so if that's true, then somehow we must have
$$\int_0^1 \frac{\ln(x+1)}{x^2+1} dx = \int_0^1 \frac{\arctan(x)}{x+1} dx$$
If you can prove that then you're done. I don't know enough stupid integration tricks to prove it off the top of my head.
