Tendex said:
I mean that once you have postulated ideal clocks and rigid rulers following the geometry of a certain mathematical space and the two SR postulates also subject to that mathematical space so they don't contradict each other, you have inertial frames as a convention. To make them subject of empirical tests you would have to abandon either the idea of ideal clocks and rigid rulers (proper times and distances) or current mathematical axioms.
The usual way physicists address the problem, how to heuristically build physical models is, the latest since Einstein 1905 and Noether 1918, to use symmetry principles. Noether's theorem works in two ways: Each one-parameter Lie-symmetry group leads to a conserved quantity and the other way around any conserved quantity defines a conserved quantity.
Now since Newton empirically we have the idea that there is a preferred class of reference frames, which we call inertial reference frames. Applied to mechanics it's the first law. Newton's dynamics also leads to the usual conservation laws (energy, momentum, angular momentum, center of mass speed), and the corresponding symmetry is the full 10-parameter symmetry group of Newtonian space-time, i.e., the Galilei group, which is a semidirect product of the temporal and spatial translation (corresponding to energy and momentum conservation), rotatations (together with translation symmetry around any point) (corresponding to angular-momentum conservation) and Galilei boosts (corresponding to the constancy of center-of-mass velocity). This full group holds for all closed systems, and the symmetry group also let's you reconstruct the Newtonian space-time description.
Now you can ask, whether the Galilei group is the only symmetry group for a spacetime model obeying the 1st Law. So assuming that there are inertial frames, within which time is homogeneous and space is a Euclidean affine manifold and symmetry under boosts you can derive that there are indeed only two symmetry groups for such a spacetime, namely Galilei-Newton and Minkowski space-time. It's well known that the latter is a far better description of space-time relationships than Newtonian space-time, and as is well known since Einstein 1905 (or rather Poincare and Lorentz somewhat before) also Maxwell's electrodynamics obeys the corresponding symmetry under the (proper orthochronous) Poincare group.
It is also pretty clear that the rather large symmetry group also determines quite well, how possible dynamical models look like. In relativity a description in terms of local field theories is quite natural, and to build Poincare covariant models most naturally you use tensor fields to formulate them. A closer investigation in the connection with possible quantum theories also leads to the introduction of representations of extensions of the Poincare group and the investigation of ray representations of the covering group. This leads to the substitution of the Lorentz subgroup ##\mathrm{SO}(1,3)^{\uparrow}## by its covering group ##\mathrm{SL}(2,\mathbb{C})##. Since this group has no non-trivial central extensions then you find quantum-field theoretical models by making the usual assumptions of locality/microcausality and existence of a ground state (boundedness of the Hamiltonian from below). The extension to the covering group leads to the possibility of half-integer spin and adds spinors of various kinds to the arsenal of possible fields one can construct Poincare-covariant dynamical models from.
This program lead to the development of the Standard Model of elementary particles and also the important concept of local gauge symmetry. The latter is quite natural, because massless fields with spin ##\geq 1## naturally lead to the idea of an Abelian gauge field. E.g., the most important one are massless spin-1 fields, would admit continuous intrinsic polarization-degrees of freedom which never have been observed for any field nor particles in the sense of quantum field theory, except you envoke the gauge principle, making some field-degrees of freedom redundant and a corresponding local gauge transformation leading to equivalent descriptions of the physical observables. Of course, electrodynamics is the paradigmatic example. Now the Standard Model describes all known particles and three of the fundamental interactions (electromagnetic, strong and weak interaction, with the electromagnetic and weak interaction combined to quantum-flavor dynamics, aka Glashow-Salam-Weinberg model).
What's of course missing in this is the gravitational interaction, and as Einstein figured out, using the various kinds of equivalence principles, this again can be included most naturally within the relativistic space-time description by again extending the space-time model. From a modern symmetry-principle point of view it boils down to the idea that Poincare symmetry has to be made a local gauge symmetry. Working this idea out leads (almost) to general relativity, and the gravitational interaction can be reinterpreted in the standard geometrical way as describing space-time as a pseudo-Riemannian/Lorentzian manifold with the pseudo-metric defining its geometrical properties as a dynamical quantity. Within feasible tests of the theory, i.e., the astronomiacal/cosmolical situations where the gravitational interaction plays a significant role, GR is the hitherto most comprehensive space-time model, including the validity of special relativity for local laws with the possibility to choose local inertial reference frames as defined in special relativity, and these are given precisely by the non-rotating tetrads along freely falling test-body worldlines (geodesics). That of course automatically incorporates the (weak) equivalence principle.
In this sense the assumed space-time symmetries, including the isotropy of space as seen by a (locally) inertial observer, is a very well tested assumption. AFAIK there are no hints at any fundamental anisotropy, i.e., no necessity to introduce more complicated space-time models with less symmetry.