I think this discussion will take way too long if we approach it from this route. I suggest simplifying this as greatly as possible to get at the heart of the confusion here. We can discuss the underlying confusion without even needing GR. Quantum mechanics in flat spacetime + SR is enough.I agree about the singularity, but I also question the EH itself. Maybe you can help me find the problem with my logic.
1. Do you agree that all frames outside of the BH calculate that no mass ever crosses the EH (or, more specifically, mass crosses the EH at t=infinity)?
2. Do you agree that the EH does not expand until mass has crossed the EH (i.e. backreaction)?
If you concur with #1 and #2, then run the clock backwards in your mind and describe to me how this theoretical black hole formed in the first place. There are other problems that I have as well, but lets start here...
Consider a Rindler horizon, and the associated Unruh radiation.
Just like in your openning paragraph, an observer far from the horizon and thus experiencing negligible proper acceleration (you could go to infinity to get your "infinity" observer as you call him) will see a object approaching the horizon take infinite time according to his watch to reach the horizon. He will also see the object bathed by an infinite amount of radiation before reaching the horizon.
Let's consider this scenario: described from an inertial frame, we see the global structure that is the rindler observer's event horizon. Imagine a baseball made of normal matter, and a baseball made of anti-matter. The baseball is on one side of the event horizon (inside the Rindler observer's past light cone), and the anti-baseball is on the other side. The baseball is thrown towards the horizon, and the antibaseball is in a trajectory so it is on the otherside to meet the baseball just as it crosses.
Alright, this is just good old flat-spacetime. What happens?
If the Rindler observer is always at constant acceleration, the baseball with never reach the horizon in any finite time on his watch. The baseball will be bathed with infinite amount of radiation.
From the baseball point of view, nothing spectacular happens as it crosses the horizon, and it reaches it in finite proper time. Upon crossing the horizon, it is annihilated, but the radiation from that anhihilation can never reach the Rindler observer.
Yes, to our intuition these sound contradictory.
But no, the math is consistent. Where our intuition fails is the expectation of "observer independence" of particle number.
Do we at least agree to this point?
The next issue of intuition failure is the fact that the Rindler observer's coordinate chart doesn't cover all of spacetime. So while what you are implying would be like claiming the "annihilation event" never occured according to the Rindler observer, that is not actually the case. He doesn't have a coordinate to assign to that event, according to his coordinate chart. That is completely separate from the claim that the event doesn't occur. He also cannot physically measure that it occurs, since the event is outside his light cone, but again, that is distinct from claiming the event didn't occur. For the event did, objectively occur.
So the second intuition issue is taking coordinate labels as too "physical". I have trouble describing this one well (it's come up in many other contexts with people asking questions on this site, and I'm often not able to help them see this clearly). So hopefully someone can word or describe it much better than I. Anyone? Please? I would like to learn how to describe this better.