To start with, the mirror does NOT exchange left and right, just as it does not exchange up and down, as far as left, right, up, and down are defined as MY left, right, up, and down.
However, when I see my picture in the mirror, I interpret this not as a picture in the mirror, but as my "twin brother" looking at me. (Otherwise, there is no exchange of left and right to talk about.) In order for him to look at me, he must be ROTATED (for 180 degrees) with respect to me. But rotated with respect to which axis? There are essentially TWO different orthogonal choices: a vertical axis and a horizontal axis parallel with the mirror. If he chooses the vertical axis and looks to me like the picture in the mirror, then, in addition, HIS left and right must be exchanged. If he chooses the horizontal axis and looks to me like the picture in the mirror, then, in addition, HIS up and down must be exchanged.
So far we have a complete symmetry. Yet, the symmetry breaks because, for some reason, I tacitly assume that rotation is done with respect to the vertical axis and not with respect to the horizontal one. To explain this tacit assumption we need psychology rather than physics. I'm not a psychologist, but I think there are two reasons why my brain makes such a tacit assumption:
1. Humans are nearly symmetrical with respect to the left-right exchange, so a human with exchanged left and right still looks very much like a human. This is not so for the up-down exchange.
2. Physically it is much easier for a human to make a rotation with respect to the horizontal axis than with respect to the vertical one.