Since this is not homework, applying the Leibniz integral rule to this particular integral yields
[tex]
\frac {dF(x)}{dx} = \frac d{dx}\int_{\infty}^{x^2} e^{-xt^2}\,dt<br />
= 2x e^{-x^3} - \int_{\infty}^{x^2} t^2e^{-xt^2}\,dt \qquad\qqaud (1)[/tex]
That integral on the right-hand side can be re-expressed in terms of the original integral. The easiest way is to integrate the original integral, [itex]F(x)=\int \exp(-xt^2)\,dt[/itex], by parts:
[tex]\aligned<br />
u &= e^{-xt^2} & du &= -2xte^{-xt^2}\,dt \\<br />
dv &= dt & v &= t\endaligned[/tex]
Integrating by parts,
[tex]
\int e^{-xt^2}\,dt = te^{-xt^2} + 2x\int t^2 e^{-xt^2}\,dt[/tex]
or
[tex]
\int t^2 e^{-xt^2}\,dt = \frac 1{2x}\left(\int e^{-xt^2}\,dt - te^{-xt^2}\right)[/tex]
Applying this in (1) yields (assuming I didn't make a dumb mistake somewhere),
[tex]\aligned<br />
\frac {dF(x)}{dx}<br />
&= 2x e^{-x^3} -<br />
\frac 1{2x}<br />
\left(<br />
\int_{\infty}^{x^2} e^{-xt^2}\,dt -<br />
\left(\left. t e^{-xt^2}\right|_{t=\infty}^{x^2}\right)<br />
\right)<br />
&= \frac 5 2\,x e^{-x^3} - \frac {F(x)}{2x}[/tex]