dsaun777 said:
So I should just stop studying physics?
No, you should start or, given your preknowledge indicated in #1, rather continuing studying physics. With your prerequesites it shouldn't be a problem to just go on! Before starting relativististic quantum-field theory, it's important to have a good knowledge of non-relativistic quantum theory, represented in the representation free Dirac formalism. The most helpful prerequisite for QFT is to understand the harmonic oscillator really in depth, covering the following points
(1) introduction of the annihilation and creation operators of "phonons" in solving the energy-eigenvalue problem for the simple harmonic oscillator
(2) understanding the formulation of the harmonic oscillator in terms of the Heisenberg picture
(3) the propagator of the harmonic oscillator
(4) not that important but helpful, if interested in quantum optics rather than QED in the HEP context: coherent states
Then it's not too difficult to also learn a bit about many-body quantum theory in the 2nd-quantization formalism mapping the (free) fields to an equivalent collection of independent harmonic oscillators.
After that you can start with relativistic QFT too.
As already Platon said, concerning math: There's no king's way, you have to learn it! Popular-science books on quantum (field) theory are usually not evern wrong caricatures of what the theory is all about. The best book coming close to a popular-science book but is still more on the serious textbook side is the QT volume of Suskinds "Theoretical Minimum" series.