Because the (single) atom will not care of the angle in/out of the photon (roughly speaking). the atom(single) will diffuse the photon, as you imagine.
And to get the things worst, if you look at the mirror -very- close you will find it not flat at all at microscopic level, even the flattest...