The math of GR is challenging, no doubt. But I'd say from observing physics forums that the #1 source of confusion is not with the complexities of GR, but with the basic concepts of special relativity.
Dealing with the math to get out numbers is difficult, but conceptually the switch from SR to GR is rather like the switch from plane geometry to the geometry of a sphere. If you draw a map of the Earth's surface on a flat sheet of paper, the sizes and shapes of the continents will be distorted. Only if you draw the map of the surface on a globe will they be preserved. Similarly, the space-time intervals around a massive body will be distorted if you draw them on a flat sheet of paper. There is a mathematical surface you can draw the space-time diagrams of the space-time around a massive body on that's curved that will preserve the intervals, at least for the r-t plane. You'd need at least a 5-dimensional sheet of paper (and probably more, possibly even much more), if you wanted to draw the entire 4 dimensonal space time on a flat surface, which would be not be terribl useful as a visual aid
Fortunately it's possible to handle the problem mathematically, even in the absence of a scale model. Just as it's possible to perform navigational computations on the curved surface of the Earth with charts drawn on a flat piece of paper, by noting the distortions that the maps have, it's possible to perform similar computations in the curved space-time around a massive object. The way in which the corrections are introduced is called a metric - a spatial metric for the surface of the Earth, and a space-time metric for general relativity.This idea isn't really terribly hard to grasp, though carrying out the detals correctly is challenging. But in order to get anywhere at all you need to know how to draw space-time diagrams and how to transform them so that you can go from what person A sees to whiat person B sees, which is in the domain of special relativity rather than GR. And it is in this basic task that I see most of the problems in understanding.