Bruce Schumm's https://www.amazon.com/dp/080187971X/?tag=pfamazon01-20 has a really excellent conceptual presentation of gauge theory and renormalization along with other major elements of the Standard Model: Lie Groups, QED and so on. His presentation is done via the mathematical concepts without doing the mathematics. In other words you should be able to get a good idea of what the scaffolding looks like but you won't be able to climb it.
Suppose you want to figure out what an electron is doing inside an atom. In general, to compute some physical observable, you're going to have to do some integral over all possible momenta that the electron can have. Now, as a mathematician, you say "Ah, it can have ANY momentum, of course!". Then you do your integral from 0 to infinity, and pout a little bit when it comes out infinite. The problem, of course, is that the thing you wanted to compute CAN'T be infinity, because you measured it and it was finite.
Now, as a physicist, you think "Well, I can't integrate over ALL possible momenta. " Naively (conservatively), you might argue that with momenta on the order of the Planck scale, we shouldn't even HAVE atoms anymore. Your theory isn't really valid at that scale, because it can't describe the gravitational interaction between the electron and the nucleus. So instead of integrating over ALL momenta, you say "Well, let's just integrate over the momenta that we're interested in". In essence, this is renormalization---you are using the physics of the problem to do your integrals only over momenta that are relevant for the problem.
Now stick with me here, but you probably know the traveling salesman problem, or some variant thereof. The one that I know is the "What's the cheapest plane ticket" problem---you want to find the cheapest plane ticket from point A to point B, with an arbitrary number of stops in between. It may be cheaper to fly first to point C, then to point D, and THEN to point B. As a mathematician (or computer scientist) you want to do things in full generality, and actually PROVE that there is a cheapest way from one place to another. As a physicist, however, you know that if the plane ticket from A to B is $300, then you probably can ignore all of the paths that have more than 300 stops---i.e. a plane ticket is never free, and it is even LESS likely that the airline pays you to fly. So you eliminate a large number (infinity - 300! or something) of possibilities just on the basis of what you know.
So renormalization is really the process of ignoring unimportant momenta.
You know, I really hate when Physicists say something along the lines of: Because we're physicists, we're going to argue on physical grounds that we can ignore this term because of symmetry or because we know that the lower bound on each path is $1 or something, but if you wanted to do it correct mathematically, you'd have to do the annoying calculation anyway.
No mathematician in their right mind would go through a long calculation when the things that physicists use to "heuristically eliminate steps based on physical grounds" are entirely mathematically valid (not to mention mathematical reasons -- not physical reasons)
If a mathematician uses some tool to show that a number has to be finite, then he knows that any method of doing things that gets infinity is wrong.
Also, mathematicians are usually aware of what assumptions need to be made so that a model is approximately true. Therefore, it's mathematically invalid to try to do something like integrate over all possible momenta, and if you try to do so, you haven't made some kind of physical error (whatever that means), you've made a logical error.