Since this is not homework, applying the Leibniz integral rule to this particular integral yields
<br />
\frac {dF(x)}{dx} = \frac d{dx}\int_{\infty}^{x^2} e^{-xt^2}\,dt<br />
= 2x e^{-x^3} - \int_{\infty}^{x^2} t^2e^{-xt^2}\,dt \qquad\qqaud (1)<br />
That integral on the right-hand side can be re-expressed in terms of the original integral. The easiest way is to integrate the original integral, F(x)=\int \exp(-xt^2)\,dt, by parts:
\aligned<br />
u &= e^{-xt^2} & du &= -2xte^{-xt^2}\,dt \\<br />
dv &= dt & v &= t\endaligned
Integrating by parts,
<br />
\int e^{-xt^2}\,dt = te^{-xt^2} + 2x\int t^2 e^{-xt^2}\,dt<br />
or
<br />
\int t^2 e^{-xt^2}\,dt = \frac 1{2x}\left(\int e^{-xt^2}\,dt - te^{-xt^2}\right)<br />
Applying this in (1) yields (assuming I didn't make a dumb mistake somewhere),
\aligned<br />
\frac {dF(x)}{dx}<br />
&= 2x e^{-x^3} -<br />
\frac 1{2x}<br />
\left(<br />
\int_{\infty}^{x^2} e^{-xt^2}\,dt -<br />
\left(\left. t e^{-xt^2}\right|_{t=\infty}^{x^2}\right)<br />
\right)<br />
&= \frac 5 2\,x e^{-x^3} - \frac {F(x)}{2x}<br />