Modeling the physics of how these black holes grow is a very challenging endeavor. There's a lot of research involved in using computer models to describe how black holes grow through the accretion of matter.
The big challenge comes in with the concept of feedback: as black holes eat matter, they tend to heat up their surroundings rapidly. In order to fall into a black hole, matter needs to lose energy. So if there is too much heat coming from the active black hole, then the matter will heat up and stop falling inward, which starves the black hole. This process places limits on how quickly the black hole can absorb matter from its surroundings.
My understanding, from what I remember of the state of research a few years ago, is that computer models had made significant progress, but still weren't able to provide a full explanation for the growth rates required to explain the most massive black holes.
As for inflation, inflation does not "inflate" black holes. It is possible for some black holes to have formed during inflation, but they would have been extremely tiny, and the inflation itself would not have made them any bigger: inflation would just move the black holes further away from one another.
The idea that black holes might have formed in the very early universe is collectively described as "primordial black holes". These black holes would likely have formed not during inflation, but shortly thereafter. The idea is that inflation creates areas with more and less density. If some fraction of the dense regions are dense enough, then they'll be able to collapse into black holes very rapidly once the expansion slows enough. The idea has long been considered plausible, but very speculative. These black holes would still be pretty small at the start, and would need to grow by eating matter (and other black holes) in order to explain the supermassive black holes. But it's certainly plausible they could play a role in explaining the rapid growth of some black holes in the early universe.
That said, while checking my facts on this, I ran across this paper:
https://arxiv.org/abs/1702.00947
If this paper is correct, primordial black holes are just too small to have much of any impact on the growth of supermassive black holes.