If you think of time dilation as the relation between coordinate time and proper time (which is true, by the way, but I suspect that not a lot of people think that way) it makes more sense.
Example: think of the relation between the longitude coordinate and distance on the Earth.
The longitude coordinate is a coordinate that tells you are - it's analogous to coordinate time. The distance tells you relative displacement - it's analogous to the sort of time you measure with a clock, proper time.
You can see because of the Earth's curvature that equal changes in the longitude coordinate do not represent equal distances. In our analogy , this would mean that coordinate time does not pass at the same rate as proper time, the time clocks actually measure. In other words, it's a model of time dilation, if you recall that time dilation can be thought of as the relation between coordinate time and proper time.
You can also see that even if you have a very highly curved surface, you won't see any curvature effects on distances for clocks that are close enough together.
There isn't any exact equivalent in this analogy for "potential" but you can see that you need both curvature and distance to get appreciable time dilation.
The same thing happens with clocks. Even in a strong gravitational field, two clocks close together will tick at the same rate (when you compare them via light signals). You need to have clocks that are separated a good ways before you can get appreciable time dilation, even if you have a lot of curvature.
One "gotcha" with this approach- it suggests that if you have no curvature, you expect clocks to run at the same rate. While curvature is associated with "natural" gravity, it's not associated with artifical gravity, like Einstein's elevator. However, we still see time dilation in the accelerating elevator, so unfortunately the analogy isn't quite perfect.