The purely mathematical part of SR consists of definitions of terms like "Minkowski spacetime" and "proper time". The part that's not just mathematics consists of correspondence rules, i.e. statements that tell us how to interpret the mathematics as predictions about results of experiments.
There are at least two correspondence rules that are included in every theory of matter in the framework of SR, so they can be considered part of the framework, rather than part of the specific theories. These two tell us how to measure time and length. In principle, each theory could come with its own set of additional correspondence rules, but in reality, they are going to be very similar. The main difference is going to be between the classical theories and the quantum theories.
People who naively think that SR is inconsistent always try to prove it inconsistent by attacking results like the twin paradox. Since the twin paradox is just the counterintuitive values of the proper times of the three sides of a triangle in a plane, an inconsistency in the twin paradox would be an inconsistency in the purely mathematical part of SR. So it makes sense for a person who's trying to prove SR inconsistent to focus on the mathematics.
An inconsistency in the twin paradox would force us to abandon at least one of the following concepts: the set ##\mathbb R^4##, the metric, integration of functions along a world line. Since ZFC set theory ensures that these things make sense, it would force us to abandon ZFC set theory, and therefore essentially all of mathematics. This is of course utterly ridiculous. It means that if SR is inconsistent, then everything else is too.
This is why I find it so annoying when people who haven't even bothered to learn SR (or mathematics) are trying to convince us that SR is inconsistent.
People who instead try to argue that SR is just wrong would need to focus on the correspondence rules instead (unless they think that there's a stupid blunder that we're all making in our calculations). Of course, the correspondence rule that gives us the twin paradox is just the statement that for any two events A,B on the world line of a clock, the time that the clock says has elapsed from A to B is equal to the proper time of the part of the clock's world line from A to B. And as most of us know, this agrees extremely well with the results of experiments, unlike the alternatives that these crackpots would have us consider instead.