In the metaphore of the universe being a flat sheet of rubber (the 2D analpgue of the 3D universe) that expands uniformly in all directions, it is true that (in the mathematical sense, not
in the physical sense) all space we can observe now can be brought back to a single 0D spot, but that does not claim to say that all
of space was just that single 0D dot.
It seems that too many people make this same implication, which is just an assumption, but doesn't need to be right. It would (IMHO) make more sense to assume that space near the Big bang was already infinite, and thus is still infinite.
This certainly makes sense in the context of the inflationary scenario, in which in other parts of the universe, inflation keeps gong on, creating ever more expanding universe bubbles.