If the Born rule isn't used, couldn't one just give an arbitrary reweighting of the sum over the environment and still get an object defined only on the subsystem (ie. is the averaging over the environment unique if the Born rule isn't used?)Mathematically, the reduced density matrix is obtained by partial tracing, which technically does not depend on the Born rule. The Born rule only serves as a motivation for doing the partial trace, but formally you can do the partial trace even without such a motivation.
A Born-rule-independent motivation for doing the partial trace is the fact that the evolution of the resulting object (reduced density matrix) does not depend on the whole Hamiltonian, but only on the Hamiltonian for the subsystem.