draw a triangle going around the corner of a cube, with one edge in each face. Try to translate a vector around that triangle, keeping it parallel all the way around. It may help to cut the cube along one edge and flatten it out. Then see where the vector ends up after translation all the way around. Remember how you have to re identify the edges if you cut it. I claim the vector will end up perpendicular to where it started.
Moreover the angel sum of your triangle, measure on the surface of the cube, is 270 degrees, hence off by exactly the amount of the total parallel transport.
By the way if this is right, it checks with your sphere example. I.e. a "triangle" made on a sphere by subdividing a circumference into three equal parts, has angle sum off by 360 degree, so is undetectable in this simple way under parallel transport.
And the rule for parallel transport is different for curves that are not geodesics.