That I suspect too ;-). I think, it's important to keep the physics in mind. Of course you can do theoretical physics by just talking about mathematical abstract structures, but that's not the entire picture about physics as a natural science, which is about observations of the real world and experiments providing as well as possible idealized conditions to measure certain aspects as accurately as possible with the given means etc.
Newtonian mechanics wouldn't be as successful a theory, describing a lot of everyday experience quite accurately, if it weren't possible to realize "reference frames" which correspond in good approximation with the abstract "coordinate systems". Of course, this is something you explain only in an introductory chapter of a textbook or in a lecture briefly and then just use the formalism to formulate the theory, as long as you learn theoretical physics. It should, however, also be complemented by experimental physics, where you see on a lot of examples, how the laws are checked (or maybe even new laws found) by experiment.
Of course, when you think about a simple experiment like shown in the youtube video about the free fall, nobody starts to deeply think philosophically about the fact that everything starts with establishing a reference frame, because that's so obvious that it seem to be a no-brainer.
But is it really a no-brainer? Newton didn't think so to begin with. He was pretty puzzled by the question, how to establish his absolute space and time, i.e., in modern language, how to realize an inertial reference frame physically. There is the famous discussion of the rotating bucket, with which he wanted to demonstrate that you can distinguish a non-inertial frame from the inertial frame apparently realized as a frame at rest relative to Earth.
Of course he was well aware that it is unlikely that a rest frame on Earth really establishes an exact inertial reference frame, because the Earth rotates around its axis and moves on an elliptic orbit around the Sun. By the way also an example that you need an adequate reference frame, as demonstrated by the history of the issue with the heliocentric vs. the geocentric point of view of how to choose a reference frame. What seems obvious today Copernicus's ingenious idea to use a heliocentric frame of reference was a revolution in the literal sense. It's considered a turning point from ancient to modern natural sciences and it was used by Kepler with success to establish his famous laws of planetary motion (starting from a tedious analysis of the Mars orbit) with a lot of mathematical effort to calcualte the heliocentric coordinates from Brahe's data (of course taken on Earth).
Back to the question about the physical establishment of Newton's absolute space and time. The issue has been an enigma even well until the 20th century. A famous piece in the puzzle is of course Mach's principle, where he conjectured that an inertial reference frame is established by the rest frame of all the fixed stars and that inertia is indeed somehow related to the interactions of the object under consideration with all masses in the universe.
The status today, I'd say, is the point of view provided by Einsteins General Relativity, according to which an inertial frame can only be established locally by a reference frame at rest wrt. a freely falling small non-rotating volume (small compared to typical distances across which you can still neglect inhomogeneities of the gravitational field and thus tidal forces, which of course always again is a question of the accuracy you look at this field). An example is the ISS, which is the best microgravity lab there is today (one speaks today rather about "microgravity" than "free of gravity" or "weightlessness"). For me that's quite contrary to Mach's principle, because it's rather a local resolution than the idea that inertia is due to the interactions of the objects under consideration with all the masses in the universe. It's rather the reinterpretation of the gravitational field or rather potential in terms of a pseudo-Riemanian fundamental form of spacetime, which let's you "construct" a locally inertial reference frame by letting a very small non-rotating volume fall freely (i.e., moving along a geodesic in pseudo-Riemannian spacetime).