There's a lot of overlap between number theory and symmetry, which is pretty central to Physics. There's a book called "Why Beauty is Truth" by Ian Stewart that traces the development of number theory and its various off-shoots with an eye to symmetry and how it applies in Physics. I recommend it to anyone interested in such things.