Dan Andersen said:
Newton said that there was an absolute point of reference to determine rotation: the fixed stars.
Yes. But he neglected to explain how that is consistent with the principle of relativity, which says that there is no such thing as absolute motion. Which, of course, is what prompted your question.

But if you're thinking that Newton left something out, yes, you're right, he did. (Arguably he had to; physics was just not ready at that time to deal with what he left out. But he knew he was leaving it out; there are passages in his writings where it's pretty clear that he recognized that the picture of physics he was presenting was incomplete.)
Dan Andersen said:
How does modern physics answer the question?
First, answering it in the context of modern physics requires relativity, which is technically off topic for this forum (it should be in the relativity forum). So if you want to dig further into what I'm about to say, you should start a new thread in the relativity forum.
The general idea in relativity is that what determines the proper acceleration that an object feels, when it is in a particular state of motion, must be something local, not something distant like the fixed stars. That something local is called the "metric"; at each event in spacetime (i.e., each point in space, at every instant of time), there is a metric that determines, for all possible states of motion of objects passing through that event, which objects feel acceleration (and how much they feel), and which do not. So, for example, the fact that the skater feels acceleration when she pirouettes, while a person standing next to her and not pirouetting does not, is due to the metric at her location in spacetime. (Actually, I'm leaving out an acceleration that they both feel--their weight, due to being on the surface of the Earth instead of out in space. This is the same for both of them, and we're interested here in the difference between them. But it's important to recognize that the weight they both feel, but a rock falling in their vicinity does not, is also due to the metric.)
So, having reduced the question of "what defines rotation?" to the question of "why does the pirouetting skater feel acceleration while a non-pirouetting person does not?", and answered that question with "because of the metric", the next obvious question is, "why is the metric what it is?" In General Relativity, the answer to that question is called the Einstein Field Equation: it links the metric to the matter and energy that is present in spacetime. So ultimately, what determines "rotation"--and in general what determines which states of motion feel what acceleration--is the matter and energy in the universe, acting on spacetime through the Einstein Field Equation.
So in a sense, Newton was right when he said the "fixed stars" determine rotation. The overall metric of the universe is determined by the overall distribution of matter and energy in it, and that distribution is basically symmetrical about the Earth and the solar system. If you work out the implications of that using the equations of GR, you will find that the prediction is that the "non-rotating" state--the one that feels no acceleration due to rotation--is the one that is not rotating relative to the "fixed stars". But the GR picture of why this happens is much less problematic than Newton's picture.