By "really understand" I mean this: that given some well-defined physical situation, you can apply such concepts as conservation of energy to predict what will happen, with some precision.
Solving physics problems is not "number-crunching," unless you solve them numerically. It is, more generally, an intellectual process that (usually, but not always) requires some knowledge of (not necessarily advanced) mathematics, but even moreso, solving physics problems requires physical insight, an understanding of how to apply the concepts you have read about to real situations - that is what lies at the heart of a "real understanding" of physics. Unfortunately such insight is not gained by reading or listening to lectures alone - it can not be taught in any direct way, only by example, and it requires practice in applying the concepts you read (or hear) about to physical situations - that is why we make students do homework problems and take quizzes and tests, at Caltech.
You do not have to be a "priest" to work on physics exercises, as is well-demonstrated, for example in the solutions to physics problems posted at The Feynman Lectures website, by people from all over the world, from all walks of life and of all ages - people who are willing to do the work necessary to gain a real understanding of physics.
I do not think what I am saying is in any way unique to physics - it is true of many activities. For example, you can read all you want about chess, but unless you actually play the game, you can not understand it very deeply. You might know all the rules by heart, but that is only a superficial kind of understanding. As they say (with regard to weight-lifting) "No pain. No gain."