It's not crazy at all. It just means that the arrow of time is dependent upon the initial conditions. It's counterintuitive, to be sure. But the physics work out identically for an arrow of time like the one we experience, or one in the opposite direction.
That would require that the arrow of time be different in different regions of the universe, or different on different size scales. And that does appear to be disallowed.
And besides, I think you're forgetting that all of the stuff around us originally came from a very small expanding patch.