MissSilvy, it took me about a decade to become comfortable with what math and physics are about, their limits, their utilities, and their possibilities. This comfort has come from applications (nuclear weapons physics, computational physics, accelerators, lasers, financial physics, molecular dynamics, etc.), and from much reading of historical sources, textbooks, and current research papers--not to mention plenty of conversations with many mathematicians, physicists, and engineers pursuing diverse areas at national laboratories and other places.
During this time, I have made maps and guides of how physics and math fit together, and what a person needs to learn these foundations for themselves.
Understanding of the core material comes eventually if you bang your head against it enough. Electrodynamics at the junior level (say Griffiths text level) is straightforward enough. Electrodynamics at the graduate level (Jackson text) took me working at accelerator facilities to really appreciate the content intuitively. I recently wrote a kind guide/syllabus for core material which I would be happy to share.
On the other hand, until recently I felt something was missing in the sense that you have expressed, and I have spent decades pursuing things to pin them down to my satisfaction. A simple case in point was my understanding of thermodynamics and statistical physics (I recommend Fermi's book on Thermodynamics, and Reichl's on Statistical Physics to get started; Riechl covers more of the field that other introductory graduate texts.) Of course I got this stuff in undergrad and grad school. Then I worked a little in molecular dynamics. I also worked on a trading floor doing "econophysics". Eventually I came to Los Alamos to work on nukes, involving particle transport. Accordingly, my understanding evolved in stages. Prior to my doctoral program in physics, I earned a non-thesis, 36 hour MS in mathematics. I studied analysis at the graduate level. The abstract approach quickly lost contact with any sense of intuition until I practiced financial physics. The analysis, placed in the context of probability and stochastic differential equations suddenly had context to finance and diffusion processes. Understanding the applications of Kolmolgorov and Ito and others made realize that my training in physics had only exposed me to a limited view of thermodynamics and statistical physics, but in the trading pits we actually practiced Monte Carlo techniques to extract numbers. This experience with Monte Carlo methods grew at Los Alamos, and I realized that this method is not bad, and that my deeper, analysis-based understanding of stochastic processes was nice, but not necessary. Still, I suppose I would have always felt a sense of incompleteness without knowing the math approach to stochastic processes. Only recently, interested in this stuff about the holographic principle, that the universe is a kind of computer, have I read C. Shannon's works on information theory, and I realized that I had erroneously thought I had a pretty good foundation on thermodynamics and statistical physics. Okay, maybe now I do.
What is math? What are its limits? Can we get rid of things like the Banack-Tarski paradox? The short answer I leave to Morris Klein, his multivolume history of math book (how. why, and the difficulties of how math got built), and his book "Mathematics, The Loss of Certainty". The books are great, but in between them are years spent studying Cantor, Ramanujan, the history of vector analysis, the continuum hypothesis, the axiom of choice, and so on. In other words, Klein's books give you a start, and I have to say that part of understanding his books lay in my graduate training in mathematics, and a part in heavy self eduction. Coming to a comfort with this kind of math foundations took me more time than it should have because there is no guide out there. You do it on your own, with little likelihood of success, or get lucky to find a good mentor. I've been drawing up a guide for this. Ditto for physics foundations.
Yes, I have a reasonable and operational understanding of general relativity and the Standard Model (and a guide for how to do this on your own) but general tools that allow theoreticians to cook up fantastical universes took me a long time to gather, unnecessarily long. I studied a lot of abstract (modern) algebra in my math training, and it just got more abstract. e.g., Lang text. Where was the contact with physics that I so often read about? I happened on "Groups, Representations and Physics", 2nd ed., H. F. Jones for a good start. I then fought with Georgi's "Lie Algebras in Particle Physics", poorly written in my opinion, but worth the fight for physics understanding. Then I ran into "Topology, Geometry, and Gauge Fields, Foundations" by Naber--so full of promise--so difficult its math jargon is to read, but I got a big picture: gauge fields have a more general language than just terms put into a Lagrangian to make it relativistically invariant, e.g., the Dirac equation. Frustration with Naber led me to the much more readable text by Nakahra, "Geometry, Topology and Physics". Again, that MS training in math helped, and pure a physicist would be more hindered in my opinion. Lastly has come an older, but readable and highly relevant book: "Lie Groups, Lie Algebras, and Some of Their Applications" by R. Gilmore. One begins to see physics in a general language. Yes, the Standard Model and General Relativity were "extruded" from experiment, but early on, people began theorizing more general constructs, such as Kaluza-Klein theory. (Only experiment can choose among all the theorizing fantasies). The books in this paragraph (plus reading of the literature concurrently throughout the years) have given me this more elevated, general view. With these books, and from various great texts on QED and QFT, I have learned that physics has some big, general ideas: how to add, e.g., Feynman's history over paths, and how to group, e.g., particle spectra, and how this adding and this grouping can come together using algebraic topology (Lie, Homology, Homotopy) to produce GUTs, string theories, etc. These foundations are nowhere as intellectually challenging as they are difficult to put together in the absence of proper mentorship.
PS--Once you get Lie groups and Lie Algebras (R. Gilmore text) you're ready to start seeing good, old fashioned mathematical physics as a very unified subject. I've gone back to Goldstein's Mechanics, and realized I had only partially understood some of the key chapters without a full appreciation of Lie Groups and Lie Albebras.
Cheers,
Alex A