You should be looking for Gateaux or Frechet derivatives, its rather easy in the case of integral functionals, which are the most common situation in physics and variational problems. What you are mostly looking is to linearize your functional, the linear part corresponds to the Frechet derivative. For example, let
F(x,f(x))=\int_a^b\sqrt{1+f'^2(x)}dx
f(a)=A,f(b)=B
(path length between A and B)
In this case we can calculate the Gateaux derivative using the definition:
D_hf(x)=\frac{\partial}{\partial t}F(x,f(x)+th(x))\mid_{t=0}
doing a little algebra you can easyly prove that
D_hf(x)=\int_a^b\frac{f'(x)h'(x)}{\sqrt{1+f'^2(x)}}dx
The Frechet derivative is more powerfull though, because it allows us to calculate the derivative of more types of functionals.
Let J(y) be a functional of y(x). The Frechet derivative is given by
J(y_0+h)=J(y_0)+DJ(y_0)h+o(\|h\|)
let
J(y)=\int_a^b G(x,y,y')dx
then
J(y+h)-J(y)=\int_a^bG_y(x,y,y')h+G_{y'}(x,y,y')h'dx+o(\|h+h'\|)
D_hG(x,y,y')=\int_a^bG_yh+G_{y'}h'dx
the example above yields the same result, as expected.
For further references on Gateaux and Frechet derivatives you should check a book of Variational Calculus, i could recommend the following texts.
TROUTMAN. Variational Calculus with elementary convexity.
COURANT. Calculus of Variations.
GELFAND & FOMIN. Calculus of Variations
CARATHEODORY. Calculus of variations and PDE's.
MIKLIN. Variational methods in mathematical physics.
COURANT & HILBERT. Methods of mathematical physics, Vol.I.
