Well anytime you leave any sort of sugar source out at room temp, you up the odds of having an escalation of the resident microbial population. However given the amount of time you are talking, the reason it is tought is because there is also evaporation at work.
Try this. Get the same type of bread (2 pieces...same size) you had the other night. Leave one on the counter for three hours in the bag and leave the other out of the bag as before. What you should notice is that the one left in the bag will be softer. As far as microbes go, the odds are that there is nothing there that poses a threat on bread in the span of three hours at room temperature, though Ian may be able to think of something I am missing.
As far as I known, no bacteria grow that fast at room temperature. To get spoilage you would need about about 24 to 48 hours to spoil bread at room temperature. For fungi, it would be 3 to 4 days. Spoilage organism are usually not harmfull for your health but the presence of spoilage is usually a sign of that the food will become a health hazard.