That is so, but the point is, quasars typically host black holes that are a hundred times that mass, and they are generally more luminous than the entire Milky Way galaxy as a result. Lesser black holes would have been far less bright than quasars, though they might graduate in their heyday to the level of an "active galactic nucleus." If one converts 4 million solar masses into a luminosity over, say, a billion years (a fraction of the Milky Way lifetime), one gets a luminosity less than a tenth of the Milky Way luminosity-- orders of magnitude less than what would be regarded as a quasar. What would be required is a way to add millions of solar masses on timescales like a million years, rather than a billion years, so there would need to be some evidence that the black hole built strangely quickly in order to think the Milky Way ever rose to the level of a quasar.
But perhaps this does raise an interesting question that might not yet be answered-- do many galaxies like the Milky Way have very much less massive black holes than do the quasars we observe because they accreted matter at a slow steady rate and so were never quasars, or does black hole accretion generally happen in short bursts such that the real issue is simply what fraction of the time is a galaxy a quasar? If the latter, then the Milky Way could only have been a quasar for perhaps 1/1000 of its lifetime, and so would only be observed as a quasar by perhaps 1/1000 of the alien astronomers out there looking at it. But it seems more likely that its present state of slow accretion describes better its history.