Read This First: Here are my first thoughts. The biggest problem with this is that it's not rigorous and utilizes a lot of 'hand-waving arguments' (and I'm not positive that this hand-waving is even justified here!). Another problem with it is that it's not really all that simple, so even if it can be justified, I don't think that it's much good as a solution. I think that hamster143 outlined a much better solution too. But with that qualified, here was my first thought about the problem.
Let the function [itex]f[/itex] be defined such that
[tex]f(x) = \frac{a(x^2-1)^2-2x(x+a)^2}{(x+a)^3(ax+1)^3}[/tex]
Since [itex]f[/itex] is continuous on [itex][0,\infty)[/itex], we can apply the second fundamental theorem of calculus to find that ...
[tex]\int_0^{\infty}f(x)\mathrm{d}x = \lim_{x \to \infty}F(t) - F(0)[/tex]
where [itex]F[/itex] is an anti-derivative of [itex]f[/itex]. Therefore, we need only find the values of these anti-derivatives. We'll go about this in an indirect way.
To find [itex]F(0)[/itex], first define the function [itex]g[/itex] such that [itex]g(x) = a(x^2-1)^2-2x(x+a)^2[/itex]. Next, note that by choosing [itex]x[/itex] small enough, we can find numbers [itex]h,k > 0[/itex] such that the following inequality holds:
[tex]h[g(x)] \leq f(x) \leq k[g(x)][/tex]
Since it's easy to verify that [itex]G(0) = 0[/itex] (neglecting the constant) where [itex]G[/itex] is an anti-derivative of [itex]g[/itex], this suggests that the anti-derivative of [itex]f[/itex] at zero is equal to zero. Therefore, [itex]F(0) = 0[/itex].
To evaluate the term [itex]\lim_{x \to \infty}F(t)[/itex], we can note that as [itex]x[/itex] becomes arbitrarily large, [itex]f(x)[/itex] tends to something like [itex]h(x) = x^{-2}[/itex] (because the numerator is a polynomial of degree 4 and the denominator is a polynomial of degree 6). Since [itex]lim_{x \to \infty}H(x) = 0[/itex] (once again, neglecting the constant) where [itex]H[/itex] is an anti-derivative of [itex]h[/itex], this suggests that [itex]\lim_{x \to \infty}F(t) = 0[/itex].
Combining these two results, we find that ...
[tex]\int_0^{\infty}f(x)\mathrm{d}x = 0[/tex]