Who says an infinitely compressed space has infinite energy?

Of course, one of the problems with this sort of discussion is that it is not just space that was all compressed, but time as well. In the very early universe, the time axis was not even differentiated from space in most of the models. This creates a common-language problem - we don't have normal every-day words to deal with time as a direction or how space-time behaves. We are forced to use mathematical terms and practice.

To get a feel for the math, you want to do a quantum physics course - the math used for the descriptions you will get are a subset of what a math course will teach you, and physicists use math a bit differently to mathematicians.

There may be a cosmology course that will fit the bill. The concepts you are looking for are all about relativity and quantum field theory ... but you will start to make sense of things before you hit the fringes of what is known and where you get several models for everything.

Meantime -

http://www.hawking.org.uk/lectures.html
In particular:

http://www.hawking.org.uk/the-beginning-of-time.html
Hawking is quite clear and his public lectures will give you a taste of the ideas.