This is an unusual integral, because usually, convergent integrals from [itex]0[/itex] to [itex]\infty[/itex] involve a function that goes to zero as [itex]x \rightarrow \infty[/itex]. In the case of [itex]cos(x^2)[/itex], it doesn't go to zero, but instead oscillates faster and faster, so the contribution from large values of [itex]x[/itex] tend to cancel out.
The way I would go about it is to use:
[itex]cos(x^2) = Re(e^{i x^2})[/itex]
where [itex]Re[/itex] means the real part. So if the integral converges, then we can write:
[itex]\int_0^\infty cos(x^2) dx = Re(\int_0^\infty e^{-i x^2} dx)[/itex]
How do you evaluate the right-hand side? I'm a little shaky about how to do it rigorously, but we know that for any [itex]\lambda[/itex] with a positive real part,
[itex]\int_0^\infty e^{-\lambda x^2} dx = \frac{\sqrt{\pi}}{2 \sqrt{\lambda}}[/itex]
If we assume (and this is where it gets a little hand-wavy) that this holds even when [itex]\lambda = i[/itex], then we would have:
[itex]\int_0^\infty e^{-i x^2} dx = \frac{\sqrt{\pi}}{2 \sqrt{i}}[/itex]
[itex]\frac{1}{\sqrt{i}} =\pm \frac{1-i}{\sqrt{2}}[/itex]
So plug that in, take the real part, and (hopefully) you have your answer. You have to choose the [itex]+[/itex] sign so that your integral is positive.