Wrong. I do get what you are saying, and i can tell you that you are no different that what the ancient greeks, kelper, rationalist and common people think about the relationship between math& physics. Namely, there is no difference whatsoever. The Ideas that one could make some equations up because it is beautiful, and that it would apply to the real world is extremaly misguided.
In reality, people go on in the real world, makes observations, and described the regularities they see using math, and tease out the consequence of the mathematical description.
math is not separate from the world and the world is not contingent. Tell me how? Give me a single example, and i will shut the hell up. What you are saying here is no different as saying that the laws of physics is logically necessary, but that is the same as saying it is logically impossible to have a world in which E=Mc^3.There are infinite many ways which reality could be different. It is entire possible that there is a world govern by a 2-d cellular automata. I suppose the people in such a world could easily figure out the symmetries, and laws. Still, it would be a world.