Almost. Consider \mathbb{R}^2 with its standard positive-definite norm. Now obtain a new Riemannian manifold M by removing the origin. In this new manifold M, what is the distance between the points (-1 , 0) and (1 , 0)? There is no geodesic in M that joins these points. There isn't even a shortest path in M that joins these points points, i.e., if someone gives me a path in M between (-1 , 0) and (1 , 0), I can always find a shorter path in M.
This leads to a slightly subtle definition of distance in a Riemannian manifold. The distance between points p and q is the greatest bound on the lengths of all "nice" paths between p and q.
In my example, 2 is greatest lower bound of the lengths of paths between, even though there is no path of length 2, and 2 is the distance between (-1 , 0) and (1 , 0).