Since time is relative then what is the age of the universe (13.8 billion years) relative to?
It is the age measured by observers who see the universe as isotropic (who see the cosmic microwave background as the same temperature in all directions) and have always done so. This family of observers are called "co-moving" observers.
And just to expand slightly on what Ibix said, co-moving observers can exist anywhere in the universe but more importantly, it can be calculated from anywhere in the universe what a co moving observer would see from that point and all would agree on the same age of the universe on that basis.
Hey doesn't isotropic mean a bit more than that? Like for example, the laws of physics are invariant with respect to the direction you are facing? Or is the word just used to mean the uniformity of the CMB with respect to direction in the context of cosmology?
Isotropic just means "the same in every direction". You can apply it to the laws of physics, which are indeed isotropic. You can also apply it to a particular phenomenon: for example, the speed of sound is isotropic for observers stationary with respect to the air but not those who move. In this case I'm applying it to the CMB and the mean density of matter on very large scales. Co-moving observers see both the same in every direction; observers who are not co-moving (such as us) don't. For example, we see a dipole variation in the CMB because the galaxy is moving with respect to the local co-moving frame.
I have always thought of it as the amount of time GR says would have elapsed in rolling back expansion to the point where predictions of density go to infinity.
If I modify to 'amount of time GR says a co-moving observer would measure having elapsed in rolling ...' is that the same thing you are saying?
And just how would you decide how much time that is?
exactly. But the co-moving observer way is fundamental; your way is just a way of USING that amount of time.
Would time be relative to the Big Bang?
Time in relativity behaves rather like distance in your every day experience. So if I asked you "would distance be relative to the corner of your house", what answer would you give? Does the question even make sense?
You can certainly measure distance from the corner of your house to any other point. But do you mean the straight line distance, or the path you have to walk to get where you are going? And does this measure help you know how far it is to a third point once you get to that "other point"?
When physicists say that "time is relative" we mean something rather like "the distance at a point" makes no sense - you need to talk about "the distance from one point to another along a specified route". When cosmologists talk about the age of the universe they are talking about a particular route (the "straight line distance") from a particular point (the Big Bang) to another (now).
This explanation isn't perfect. You'll need university level maths for a really accurate description. The time-is-like-distance analogy is a very close one, but the time dimension is different from the three spatial ones so don't push it too hard.
Separate names with a comma.