I agree. ... And I think you gave a convincing "explanation".
Now, the next thing that I think needs explaining is what you mean by "we". From "my" point of view, being one of "us", it seems that "I" know about a world. It also seems that "my" world also includes the rest of "you". And from "your" reports, it seems as if "our" worlds are one and the same even though "we" don't all know it in exactly the same way.
So how do we explain the multiplicity of "us" in a singular world? Could it be that "we" are actually one, and that what seem to be individual consciousnesses are that one traversing different world lines? If not, why not?