I think that makes sense.
There is another way to think of torsion that makes it clearer why it is called "torsion":
First off, any two neighboring tangent spaces at points x and x + dx are related by some matrix in GL(n). Therefore, a connection is really just an object that tells you, for each possible displacement dx, what matrix to use in order to "glue" these tangent spaces together. Since the neighboring points are infinitesimally close together, the matrix in GL(n) must be infinitesimally close to the identity; hence the connection is a 1-form that takes values in the Lie algebra \mathfrak{gl}(n). Then we say that GL(n) is the "structure group" of the manifold.
There are some various properties you might ask of a manifold that correspond to "reduction of the structure group" to some subgroup of GL(n). For example, if the manifold is orientable, then its structure group must be in SL(n). We could also ask that the connection is "metric compatible", meaning that it preserves the lengths of vectors. In this case, the structure group will be O(n), or SO(n) if the manifold is also orientable. In any case, the Lie algebras of O(n) and SO(n) are the same.
Suppose the connection is metric-compatible, such that all parallel transports correspond to some SO(n) rotation. Now let us travel along a geodesic with tangent vector X. Hence
\nabla_X X = 0
As you may notice, the geodesic equation doesn't care about the torsion. Since X appears twice, only the symmetric part of the connection participates, and the torsion is purely the antisymmetric part. Hence if two connections \nabla and \nabla' differ only by torsion, then they have the same geodesics.
But what happens to the rest of the tangent space as it is parallel transported along a geodesic? The parallel transport is in SO(n), and the geodesic equation tells us that along this particular path, one vector (namely X) must stay fixed. Hence the rest of the tangent space must rotate under SO(n-1), which is the subgroup of SO(n) that leaves X fixed. So, the rest of the tangent space "twists around" the geodesic.
So then this is what torsion means. Connections differing only by torsion must have the same geodesics, and if we travel along a geodesic, the only degrees of freedom left are those that twist around the geodesic. Hence torsion corresponds to the amount of twisting around a geodesic, as we parallel transport along the geodesic.
If I have time later, I'll see if I can put this into symbols, so you can see exactly how torsion mathematically relates to twisting around geodesics. Or maybe you can give it a shot.