I think you might have confused 'inflation' with 'expanding'.
The universe is expanding and other galaxies are indeed moving away from us but 'inflation' refers specifically to a very short period near the start of the universe when this expansion was very quick. It is still speculation but fits other observations very well.
If this quick expansion was uneven then one piece of the universe might have been shifted away from our bit of universe never to come back into contact...