Since time is relative then what is the age of the universe (13.8 billion years) relative to?
It is the age measured by observers who see the universe as isotropic (who see the cosmic microwave background as the same temperature in all directions) and have always done so. This family of observers are called "co-moving" observers.
And just to expand slightly on what Ibix said, co-moving observers can exist anywhere in the universe but more importantly, it can be calculated from anywhere in the universe what a co moving observer would see from that point and all would agree on the same age of the universe on that basis.
Hey doesn't isotropic mean a bit more than that? Like for example, the laws of physics are invariant with respect to the direction you are facing? Or is the word just used to mean the uniformity of the CMB with respect to direction in the context of cosmology?
Isotropic just means "the same in every direction". You can apply it to the laws of physics, which are indeed isotropic. You can also apply it to a particular phenomenon: for example, the speed of sound is isotropic for observers stationary with respect to the air but not those who move. In this case I'm applying it to the CMB and the mean density of matter on very large scales. Co-moving observers see both the same in every direction; observers who are not co-moving (such as us) don't. For example, we see a dipole variation in the CMB because the galaxy is moving with respect to the local co-moving frame.
I have always thought of it as the amount of time GR says would have elapsed in rolling back expansion to the point where predictions of density go to infinity.
If I modify to 'amount of time GR says a co-moving observer would measure having elapsed in rolling ...' is that the same thing you are saying?
And just how would you decide how much time that is?
exactly. But the co-moving observer way is fundamental; your way is just a way of USING that amount of time.
Separate names with a comma.