In his book, Seven Brief Lessons on Physics page 18, Carlo Rovelli said:

What is the name of the mathematical formula and the abstract space he refers to in this passage? And why can't these interactions occur in real space?

In physics, the space is usually called "Hilbert space".

In the usual quantum mechanics, we don't know whether the wave function, which lives in Hilbert space, is real. We only know that the results, such as the position of a particle in real space are real.

It may be possible to make rewrite the formalism so that it occurs in "real space" in some sense, eg. http://arxiv.org/abs/1410.3676. However, the underlying weirdness of quantum mechanics is that reality is nonlocal (or retrocausal or many-worlds or ...). That cannot be overcome by any rewriting of the formalism.