The color of something is the visible waveband does not correlate well with it's 'color' in the infrared. Snow, for example, is black in the IR.
It can be a little tricky to figure out- the incident power from the sun peaks in the green/yellow (555 nm), but the peak wavelength corresponding to (say) 90 deg. F (32 C) is around 9.5 microns. So a black shirt will absorb sunlight very well, but may also radiate the heat very well.
For a gas heater, most of the radiant energy is around (IIRC) 3-5 microns. Again, the color in the visible has little to do with the optical properties in this waveband.
As it happens, cotton cloth has an IR emissivity (8-12 microns) of about 0.8
http://www.coleparmer.com/techinfo/techinfo.asp?htmlfile=Emissivity.htm&ID=254
Close to the absorptivity in the visible. I couldn't find values for the mid-wave IR.
What that means, in practice, is that the color of the fabric has very little effect on the thermal properties.