This broader sense of background independence is a key question but it's tricky. I've expressed my opinon on this before but this is how I view it, and characterize the key problems.
If we first look at relativity, both SR and GR, IMO the conceptually sound, clean and understandable reason for the invariance is really simple. It's that idea that the laws of physics must be the same for all observers. In SR we required that the laws of physics must be the same to all intertial observers. To realize that we characterize the lorentz and poincare symmetries which represent this special class of observers, and find the transformation laws that allows us to find covariance of the laws.
In GR, the story is the same except that we extend the class of observers to so that the laws of physics must be the same also to any non-interial observers. Basically to any observer with arbitrary motion.
Still GR only talks about a small subset of observers. Noone would think that diffeomorphisms are enough to generate all traits of an observer. The internal structures is ignored, and conceptually there is no reason to.
I would say the deepest form of rationale common to this, is the idea that the laws of physics must be the same to all observers, and thus the challange, given the obvious that each observer WILL see different things depending on their perspective, is to characterize the relations between these observers, so we can find the transformations laws that restored observer independent physical law.
But
SR and GR are still realist theories, the very transformations and the class of observers are views in a realist sense - in particular they are not subject to measurement or scientific inquire from the physics point of view.
This is why when we seek a proper _measurement theory_ (more proper than QM) that respects the mentioned ideal, things get quite complicated because then it should be formulated to a larger extent in observables only. Otherwise it's still just a "semiclassical" measurement theory, that refers to an system, but with a "classical observer".
The real difficulty is this, that we seem to seek an "observer independent" measurement theory.
To me, the scientific rationale for focusing on observable things only, is more important than a realist type observer independence because a natural non-realist interpretation of the principle that the laws of physics are the same for all observers, is instead an kind of _democracy of observers_ where no observer is more wrong or right than another one, and that in fact the local laws in a community is a result of a democratic process.
If I may allow to use the word "inference" as the generalistation of "measuremnt" to include not only measuring an observable, but also to "measure" symmetries and laws, then the fact that two observers infere inconsistenct laws, need not be an actual "inconsistency" in the sense people usually thing, it might rather be an incentive(force) for each participant to renegotiate, and that objectivity is instead possibly a result of evolution.
This way of seeing it, has IMO a conceptually clean potential to (although in an evolutionary sense rather than realist sense) incorporate the "ultimate" observer indepdendence, including ALL interactions.
The problem of just enlarging the class of observers, and maintain the realist perspective is that no inside observer can RELATE to (encode and process) this information. This is why I think the only way to combined the scientific ideal of a "measurement theory" with the wish that the laws of physics are seen the same to all observers, is to put it in a context of evolution.
Every other attempt at sidestepping this point has IMO lead to terrible realist or weird types of models, with all kinds of other prolbem. The typical realisation of picturing a large class of observers, is that we end up with an unmeasurable landscape, or alternatively unmanagable "choice"-problems. The use of mathematical of "birds view" observers is als a typical trait of these ways of reasonings. By itself, such external observers, are also in violation with the intrinsic ideal also somehow beeing part of GR. But we do not just look for "intrinsic geometry", we look for something much better, we look for an intrinisic measurement theory.
I have a feeling I am in clear minority here.
/Fredrik