I have "The Physics of Stars" Second Edition by A.C. Phillips (1 of 3 books I have including an older Fundamentals of Physics Text Book and Einstein's book on Relativity) I get lost on the math as it's a bit beyond my knowledge level, however, everything else I perfectly understand, despite everyone around me claiming I'm not "smart enough" to understand physics. I understand the concepts, theories, and findings though I'm not going to be able to solve a complex Physics problem written out on a board. I don't have the foundation to understand all the variables and mathematical equations, yet.
A few years ago Physicist Michio Kaku came to my college and spoke. There was a chance for students to ask questions but seeing as I was/am not a Physics student I didn't want to be laughed at like the dude that tried to act big by asking about fission. I didn't even try to get in que after that.
Physics and Astrophysics is more of a side hobby to me.