As you pointed out, Jacobi fields measure the separation of geodesics. Suppose you have a bunch of geodesics \gamma_\tau, where \tau is a parameter labelling the individual geodesics. So, we're talking about two-parameter families of curves, and not of diffeomorphisms!
Each of these geodesics...