Hello Vuk,
We are given to evaluate:
$$I=\int\frac{\ln(x)}{\left(x^2+1 \right)^{\frac{3}{2}}}\,dx$$
If we use integration by parts, we could let:
$$u=\ln(x)\,\therefore\,du=\frac{1}{x}\,dx$$
$$dv=\frac{1}{\left(x^2+1 \right)^{\frac{3}{2}}}\,dx$$
To find $v$, we may use a trigonometric substitution:
$$x=\tan(\theta)\,\therefore\,dx= \sec^2(\theta)\,d\theta$$
and we find:
$$v=\frac{\sec^2(\theta)}{\left(\tan^2(\theta)+1 \right)^{\frac{3}{2}}}\,d\theta$$
Using the Pythagorean identity $$\tan^2(\theta)+1=\sec^2(\theta)$$ we get:
$$v=\frac{\sec^2(\theta)}{\sec^3(\theta)}\,d\theta= \int\cos(\theta)\,d\theta=\sin(\theta)$$
Back-substituting for $\theta$, we obtain:
$$v=\sin\left(\tan^{-1}(x) \right)=\frac{x}{\sqrt{x^2+1}}$$
And so we have:
$$I=\frac{x\ln(x)}{\sqrt{x^2+1}}-\int\frac{1}{\sqrt{x^2+1}}\,dx$$
Now, using the same trigonometric substitution we used before, we have:
$$I=\frac{x\ln(x)}{\sqrt{x^2+1}}- \int\sec(\theta)\,d\theta$$
Let:
$$u=\sec(\theta)+\tan(\theta)\, \therefore\,du= \left(\sec(\theta) \tan(\theta)+\sec^2( \theta) \right)\,d\theta=$$
$$\sec(\theta)\left(\tan(\theta)+\sec(\theta) \right)\,d\theta=u \sec(\theta)\,d\theta\,\therefore\,\sec(\theta)\,d\theta=\frac{1}{u}\,du$$
Hence, we now have:
$$I=\frac{x\ln(x)}{\sqrt{x^2+1}}-\int\frac{1}{u}\,du$$
$$I=\frac{x\ln(x)}{\sqrt{x^2+1}}-\ln|u|+C$$
Back-substitute for $u$:
$$I=\frac{x\ln(x)}{\sqrt{x^2+1}}-\ln|\sec(\theta)+\tan(\theta)|+C$$
Back-substitute for $\theta$:
$$I=\frac{x\ln(x)}{\sqrt{x^2+1}}-\ln\left|\sqrt{x^2+1}+x \right|+C$$
And we are done. (Sun)