What are we relying on, then?
Also, one could agree that energy is conserved, while also ask if we cannot detect all of the (conserved) energy.
Like, hypothetically if a wave has 1.5 photons of energy in it, we could detect 1 photon at the start, 1 photon at the end, and that's conserved. And...