Yes, energy loss due to heating of gasses and collisions. Also, because of the simply massive amount of stars orbiting, the perturbations introduced by random close encounters tends to destabilize the orbits over time.
There are a few things that would work to make this model not as simple as you've stated. Yes, stars would form towards the center of this dense cloud of gas and dust, but solar wind usually clears out the surrounding disk of material relatively soon. So, for all intents and purposes, the stellar systems (be they lone, binary, etc.) are essentially alone in their local neighborhoods, even in areas of relatively "high" density. The most that happens when one turns into a black hole is it begins to accrete matter from its companion star. It will not gobble up everything in the nucleus, for as we have said, it behaves (gravitationally, and non-locally) exactly the same as a star on the grand scale of things. And stars almost never collide or interact on more than light year scales.