In the case of galaxies, you have to consider the masses of all the constituents - including the dark matter. If you were to do that (which, of course isn't really possible until we know more about the nature of dark matter), you should find that the total mass of the galaxy is less than the masses of everything in it combined.

With quarks, however, things are a bit more complicated. The nature of the strong force between the quarks is that the farther apart they are moved, the greater their potential energy becomes without limit.

In gravitationally and electromagnetically bound systems, the potential energy is always negative and increases to 0 as objects are moved progressively farther away. A system being bound means, quite simply, that the particles do not have enough energy to the point where their separation is infinite. In other words, their total mechanical energy must be smaller than the potential energy of two particles an infinite distance apart. So, in gravity and E&M, bound systems must have negative mechanical energy. Since mass and energy are interchangable, negative mechanical energy means that the total energy (mass plus mechanical) is less than the energy just stored in mass.

By this discussion, we see that the existance of the mass defect requires that the potential energy be 0 at infinity. Going back to the strong force, I said above that it is a case where potential energy increases without bound. In other words, the potential energy from two quarks separated by an infinite distance is infinite. In other words, any system of quarks, no matter how much energy it has of any sort, is a bound system. So, because of the very different nature of this strong potential, the mass defect does not apply.

For the fundamental forces of gravity, electromagnetism, and the strong force, what exactly is it meant by a positive potential energy vs. negative potential energy? I thought negative potential energy represents attraction and positive potential energy (in the case of proton repulsion) represented repulsion. But apparently, when I look at the Cornell potential, it's the reverse (with a negative sign for repulsion and positive sign for attraction). Why is it spoken like this?