I would really appreciate it if I could get book recommendations on the rigorous mathematical theory of x ray crystallography. Which areas of mathematics (pure and applied) would be most useful and applicable?
Do you not want normal crystallography books? Here are two suggested by MIT to their grad students:
Werner Massa - Crystal Structure Determination
"Everything important is explained and the book starts from scratch."
Carmello Giacarvazzo, et al. - Fundamentals of Crystallography
"The somewhat more advanced student may like [this book]. Even though the word "fundamentals" appears in the title of the book, it is very helpful to have prior knowledge, when attempting to read [it]. This book covers all the basics and should be sufficient for most PhD students.