I know of mainly two famous physicists who came from a distinguishly mathematical background:
Freeman Dyson
Roger Penrose
I'm sure there have been many, many, many others, so could anyone name me some more physicists who came from a mathematics background?