You are right! For this, I used normal coordinates, then wrote
$$
d^2(x,x_0)=d^2(x'+(x-x'),x_0)=d^2(x',x_0) + (x-x')^\mu\partial_\mu d^2(x',x_0)\\+1/2(x-x')^\mu(x-x')^\nu\nabla_\mu\partial_\nu d^2(x',x_0)+\dots
$$
I see your points.. but the thing is that I'm not sure the squared distance is a...