That's what we tell the kiddies - next level up: different materials absorb different combinations of wavelengths. The color you see is how your eye responds to the wavelengths that get scattered.
There is more than one thing happening. Materials only absorb a proportion of the incident light. Light that gets scattered gives the diffuse color but you also get specular effects and straight reflection as of a mirror. In general, incident light can be absorbed, scattered, reflected, or transmitted.
In the case of the blue surface - it is reflecting visible light of many wavelengths, but some of the wavelengths are absorbed, scattering blue light.
What you see is the combination of lots of processes - the scatter/reflection gets some help from the image processing in your brain too... so the red shirt probably looks redder than the reflected red light would indicate because your brain knows your shirt is supposed to be red.
Similarly, when you look at differently colored objects under monochromatic light it sometimes looks like you can see the colors there anyway.