adolphysics said:
It's "easy" to see if the event horizon gets bigger too fast, only counting regular matter
Is it, though?
In the case of baryonic matter, black holes grow by accrettion of material that is available in their immediate vicinity - the atmosphere of a companion star, for example.
If one were to look at the equation for the Schwarzschild radius of an idealised non-rotating black hole, ##R_s=2GM/c^2##, it is clear that in order to double the radius, you'd have to double the mass. A solar mass BH is approx 6km across. A small BH can feed on its companion for millions of years.
So, one needs an instrument capable of observing a few kilometers-across object, with growth rates on the order of milimeters per year. All from many light-years away.
Apart from the difficulty of measuring the size of the event horizon, let alone its growth, one has to consider the amount of dark matter that is available for infalling into the black hole.
While densities of both types of matter are comparable when averaged over the volume of a galaxy, a star is a highly overdense region of space, providing a lot of 'fuel' for the black hole, whereas at any given time, there is only a tiny amount of dark matter passing through the same region.
E.g. the current models suggest that at the distance from the galactic centre at which the Sun orbits, there is roughly a small asteroid-worth of DM contained within a 1 cubic AU volume of space, and while the density goes up closer to the galactic centre, it never approaches the densities of clumps of baryonic matter.
This is further exacerbated by the fact that DM is at best only weakly interacting - it does not have a way to form accretion discs, so no way to shed orbital velocity. Only those particles whose trajectories cross the event horizon can be added to the BH. Near misses will be just flung out back into space.