As others have noted, I don't think this question really makes sense on a number of levels.
First off, "size of the universe" doesn't really make sense. There are three distinct kinds of universe allowed by cosmological models in general relativity, two of which are infinite and always have been. The third does have a finite size. But we don't know which kind the one we live in is (our measurements don't currently rule out any of them), so we don't know if "size of the universe" is a meaningful thing. You do hear people talking about the size of the observable universe, and the current size of that would depend on the age and expansion history. But it is not the same as the size of the universe.
Next, "age of the universe" is a tricky concept. The 14 billion year figure you hear quoted is the time (technically, the "cosmological time") since the universe cooled to the point we can see through it. In a naive general relativistic model the universe is a couple of hundred thousand years older than that (a rounding error on 14bn years). But in inflationary models the universe is much, much older, I believe possibly even infinitely old.
Finally, the reason for invoking a rapid inflation phase in the early universe is to explain why the universe is the same temperature wherever we look. That is really difficult to understand if all the different parts of the universe were not in contact in the past, and without an inflationary phase the far distant parts we can see in opposite directions are too far apart to ever have been in contact. So we would expect them to be at different temperatures and we wouldn't see the more-or-less-isotropic universe we see today. Since that isotropy is a key part of how it develops, the universe would potentially be a very different place, not just bigger or smaller.
As almost always happens with physical models, "what if I change this small thing" can have quite complex implications down the line which can invalidate (or at least complicate) apparently simple questions.
