Hm, we are going here in heavy math ?
I think there are 2 ways of giving an answer at least :
a) week-end mathematician : you derive with respect to a function, but I don't care, since I do it a la physicist as computing differentials :
1)

2)
b) Heavy math jam : you have to use the "directional" functional derivative (Gateaux), and your notation is not correct with these symbols :
Since x'(t)^2 is a functional of x as well as x(t) is a functional of x'.
NB: direction and place have no sense here since we speak about functions.
Reminder : the definition of the functional derivative in the "direction" v of a functional F at the "place" f:
Let's compute that way :
first :
1) F[x]=x'(t)^2
2) G[x]=x(t)
then :
1)
Now replace the arbitrary function v(t) by x(t) : you get :
2)
Replace v(t)=x'(t) you get D_x'G[x]=x'(t)
So the answer depends on how you see the problem :
a) you see the all the function as function of the parameter t, and apply physicist differentiation
b) you see the derivative of a functional towards a function and apply Gateaux derivative
REM : I have problems with the tex output