Here's a spacetime diagram showing Tom and Mary. It's drawn using Tom's frame, so Tom is at rest.
View attachment 336789
If you haven't come across spacetime diagrams before, they are "maps" of spacetime, with time (as defined in this frame) drawn up the page and space (again, as defined in this frame) across it. Tom, therefore, appears as a vertical line (red on the diagram) because he doesn't change his position in this frame. Mary is a slanted line (blue on the diagram) because she is moving. Notice that Tom has been placed at 1ly, and Mary reaches 4ly from him at t=1. Let's add a line showing Tom's idea of "all of space at the moment that Mary is 4ly away from him":
View attachment 336790
Tom's idea of space is the fine red line, and you can see that Mary is 4ly away. However, Mary does not share this definition of "space" because she does not share the same notion of "at the same time".
Her version of "all of space at the same time" is shown as a fine blue line here:
View attachment 336791
Notice that the fine blue line passes through Mary at the same time as the red line, but it does not pass through Tom at the same time as the red line. This is why they measure different distances - they are measuring along different slices of spacetime, and they are doing that because they don't share a notion of "at the same time". The interval along the fine blue line between the two thick lines is 2.4 ly.
To complete the picture, we can also draw the same graph in Mary's frame:
View attachment 336792
Now Mary is represented by a vertical line and her "all of space" is a horizontal line. You can read off the horizontal axis that the spacing is 2.4 ly, but again, Tom is measuring something different.
Your problem is that you keep hoping you can pretend the different slopes of the two fine lines don't matter. They do.