That's a very reasonable thing to worry about and be puzzled by! In popular accounts they don't tell you the whole story.
In bare general relativity (as distinct from cosmology, where you have matter e.g. a primordial hot gas, and the ancient light from it which we still see) there is no criterion of rest and no distinguished time.
However in cosmology we do have a criterion of rest. We live in a bath of ancient light.
this is redshifted by expansion so it is no longer in the visible range. It used to be the slightly orange-ish color light given off by 3000 kelvin (hot) gas. Like the surface of an orange-ish star. But now the wavelengths are stretched out by a factor of about 1000 and it is in the microwave range. Socalled microwave background. A kind of invisible soup of light that you need a horn antenna to detect.
IF YOU MOVE fast enough in some direction you will see a DOPPLER HOTSPOT in the ancient light in that direction.
If you are at rest the ancient light will be the same low temperature, approximately same mix of wavelengths, in all directions. No Doppler hotspot in the microwave sky, for an observer at rest.
So there is a criterion of rest. It has a definite meaning. And in accordance, then, with general relativity, there is a preferred time.
That is the basic time we use in cosmology and it is the time that the AGE of the expanding phase of the universe is stated in terms of.
Observers anywhere in the universe who are at rest relative to Background---who measure the same temperature we do in all directions---will estimate approximately the same age.
In effect, our clocks are synchronized with those of all those other observers---or would be if they existed (which we don't know for sure).