The OP might find a careful reading of Misner's "Precis of General Relativity", on arxiv as
https://arxiv.org/abs/gr-qc/9508043, helpful. Possibly not, but it's worth a try. It's one of the few papers that talk about some of the underpinings of the theory that address the OP's question.
It's rather long, but I'll give a couple of quotes I think are relevant.
Omitting the motivations and historical connections, and also the
detailed calculations, I state succinctly the principles that determine
the relativistic idealization of a GPS system. These determine the
results that Ashby presents in his tutorial.
A method for making sure that the relativity effects are specified correctly
(according to Einstein’s General Relativity) can be described rather briefly.
It agrees with Ashby’s approach but omits all discussion of how, historically
or logically, this viewpoint was developed. It also omits all the detailed
calculations. It is merely a statement of principles.
One first banishes the idea of an “observer”. This idea aided Einstein
in building special relativity but it is confusing and ambiguous in general
relativity. Instead one divides the theoretical landscape into two categories.
One category is the mathematical/conceptual model of whatever is happen-
ing that merits our attention. The other category is measuring instruments
and the data tables they provide.
TL/DR. We divide GR into two parts, a conceptual model and physical measurements
The conceptual model for a relativistic system is a spacetime map or
diagram plus some rules for its interpretation. For GPS the attached Figure
is a simplified version of the map. The real spacetime map is a computer
program that assigns map locations xyzt to a variety of events.
Tl/dr. The conceptual model says that coordinate are just labels that we use to describe events in space-time.
Misner also discusses the measuring instruments. However, I won't quote that part, because understanding it properly rquies first that one realize the difference between coordinate time and proper time, which is the point ujnder discussion.
Misner proceeds to give an approximate metric of the space-time of the Earth as an example of a particular instance of the conceptual model.
dτ^2 = [1 + 2(V − Φ0)/c^2]dt^2 − [1 − 2V /c^2](dx2 + dy2 + dz2)/c^2
Misner points out that this metric is a conceptual model. The purpose of this model is to provide labels (coordinates) for events. The events are physical. The coordinates are just labels. The map is not the territory. If we say "the destination is at square A4 of the map", the destination is physical, the reference to grid "A4" on the map is a map reference, it's not physical. Some different map from a different atlas might assign different labels to the same destination.
The following quote from Misner is of particular relevance - it basically says the same thing everyone else has been trying to tell the OP.
The constant Φ0 is chosen so that a standard SI clock “on the geoid” (e.g.,
USNO were it at sea level) would give, inserting its world line x(t), y(t), z(t)
into equation (1), just dτ = dt where dτ is the physical proper time reading
of the clock.
The direct statement isn't perhaps clear, until one realizes the implications. Namely, a clock that is NOT on the geoid does not keep proper time.
A quote from Wiki, says the same thing:
wiki said:
In the 1970s, it became clear that the clocks participating in TAI were ticking at different rates due to gravitational time dilation, and the combined TAI scale, therefore, corresponded to an average of the altitudes of the various clocks. Starting from the Julian Date 2443144.5 (1 January 1977 00:00:00), corrections were applied to the output of all participating clocks, so that TAI would correspond to proper time at the geoid (mean sea level).
This gives an example of the difference between coordinate time and proper time in the context of a simple and hopefully familiar example of GR - namely, time on the surface of the Earth.
So - a quick recap.
Coordinates are just labels, they don't have physical significance.
Proper time does have physical significance. Proper time is NOT the same as coordinate time - on the Earth, as an example, the two are the same for clocks at sea level, but any clock other than sea level will have a proper time ##d\tau## that is different from the coordinate time ##dt##. Thus it is important to know which one is talking about - the physical clock, that keeps proper time, or the coordinate clock, which does not necessarily keep coordinate time, depending on it's location (specifically, it's height above sea level).