Interesting question. Here's my guess.
Of course things like plastic, metals, and other smooth materials don't change their color when they get wet. So why do clothes? I think this has to do with the fact that most clothes are made out of fabrics that are made out of fibers. In its dry state, a fabric has little fibers sticking off. You can see them with your naked eye, especially on faded clothing. Another example is untreated wood--wood changes color when it is wet, and it's composed of fibers.
A single fiber is somewhat transparent (the fabric is only opaque with layers and layers of fibers), and like most solids, the fibers have an index of refraction much higher than that of air. So in the light, these little fibers refract/reflect the light, and since the fibers sticking off a fabric are pointing in all different directions, they scatter the light in all different directions. The net effect of little reflections and refractions off the millions of little fibers is a lighter appearance.
When the fabric gets wet, the water forms a smooth surface over the fibers, and the index of refraction of water is much closer than air to the index of refraction of the fibers, so less refraction and reflection occurs at the interface between the fibers and the water. (
Anti-Reflection Coatings). The surface of the water doesn't reflect in all directions (like the fibers) since the surface is relatively smooth due to surface tension. So the water stops the fibers from scattering the light off the fabric.
I think this line of reasoning applies to other rough surfaces--like rough rocks and rough metals that look different when they get wet. In that case it's the reflection off the many different facets of the rough surface that causes the scattering of light.
