Basically you will need a good foundation on
algebra and
discrete mathematics. The former can be found in many good books and depends a bit on how you learn - more explanations via text or more structured via formulas - and how deep you want to go.
Algebra with emphasis on text:
https://www.amazon.com/dp/0387406247/?tag=pfamazon01-20
Algebra with emphasis on formulas:
https://www.amazon.com/dp/0387220259/?tag=pfamazon01-20
However, it could well be that you also need some knowledge of commutative and linear algebra. I find van der Waerden a good book to get the basics and understand what it's all about.
But if you don't want to get too deep into mathematics, you would probably like
https://www.amazon.com/dp/B012TXEOT8/?tag=pfamazon01-20
which has several examples of applications together with the theorems behind them. Crypotgraphy should be one of them. (I have a different version, so I can't tell for sure. But Springer usually offers the possibility to read the content and a couple of sample pages.)
The point is that cryptographers plunder everywhere in mathematics, so there is no single field to point to. Furthermore there is the technical side of it: who knows what when with which likelihood. So without any specifications from your side, e.g. whether you are more interested in the mathematical part or the information science part, all what can be said is: learn everything about finite groups (van der Waerden) and have a look around (Lidl, Pilz) to sort your interests. Otherwise the answer to your question will be an undergraduate study of information science or likewise mathematics.
All books I mentioned are from Springer. So in any case you should visit
Springer's website, try to find them and have a look inside the books.
If you want to learn actual number theory, e.g. to understand modern primary tests (as in the context of the RSA scheme), then your way is a bit longer and includes at least real and complex analysis and basic number theory.