Anyway, it's certainly not an elitist attitude or some kind of hazing ritual at all. Math is an absolute necessity for physics. Sure, you mention SR, but I doubt you can mention much more than that. Even if you go into classical mechanics, you'll need calculus very quickly. You just can't do much without calculus!
I'd say actual physics started with Newton, and he had to invent calculus for it. So I don't see why you think you don't need to know calculus in order to understand physics.
I can think of plenty of calculations in physics that only require basic algebra. For instance half life, Schwartzchild radius, kinetic energy, wavelength, F=ma, the inverse square law, the Stefan Boltzmann law, Newton's law of gravitation, blackbody radiation, doppler shift, Kepler's laws, escape velocity, etc. There may be calculus versions of some of these, but that doesn't negate every other version. If all of these, plus a lot more, represent such a small part of physics that you can comfortably call it nothing, then I guess you're right.
My contention is that there's a lot in physics to learn about that doesn't require advanced math, so no one should be told they must first learn calculus, lest they never understand anything about physics. I can demonstrate that that's false. And it might be discouraging to some people who may not be able to, or have the desire to learn new math, yet they want to learn some physics. A lot of people who watch science TV shows are like that. They find physics interesting and enjoy learning it, and they shouldn't be told they can only do it under certain circumstances, especially when it's not true.
I think you are confusing "learning about physics" with "learning physics".
Sure, you can learn about physics without knowing any math, but only in the sense that you can learn about sailing without ever having been in a boat.
You can learn, for example, Newton's 1st and 3rd laws of motion without needing any math. There may be calculations associated with them, but knowing how objects behave is important and not contingent on math. If someone learns that an object in motion stays in motion unless acted upon by an external force, then just having that idea and understanding what that means with everyday objects is physics. If someone learns that, and what they learned is physics, then how are they not learning physics? After having learned that, they could easily construct an experiment to show other people how it works. They could put something on top of their car, go straight and then take a sudden right turn and show that the object continues forward. If you can do all that, I'd say you've learned physics, even if you can't make calculations. And I'd say that person has about as good of a chance of explaining what will happen in a situation like that as much as you do (such as how far the object will travel after the right turn is made, or what orientation it will end up positioned in, etc), even though you know the math and they don't.