1. It requires the use of differential geometry (and tensors, etc.) which most physicists don't learn unless they learn GR.
2. Curved space-time is hard to visualize due to it's 4-dimensional nature.
3. The notion of vectors on a curved manifold conflicts with our basic sense of vectors. Parallel transport is not trivial like in Euclidean space. The inability to add two vectors based at two different points in the manifold also complicate things. The inability to take derivatives of vectors. Etc.
4. The Einstein Field Equations are non-linear. (I.e. terms like g*g or dg*dg appear, where g is the metric tensor that you're trying to solve for)
5. The Einstein Field Equations solve for a metric (i.e. a coordinate system). In general, the metric will appear on both sides of the equation (since the stress-energy tensor generally depends on the metric in some way). In order to describe our distribution of matter, we need a coordinate system; however, we do not have a coordinate system "given" to us - we must solve for them. Greatly increasing the difficulty in solving the EFE's in general.
I think the first couple of reasons are the reason GR is hard to get into initially, because of all the groundwork that you have to learn at first. The later few reasons are why GR is hard to work with in general - even after you've learned it. (And why we have so few exact solutions of the EFE's as well as the difficulty in numerically modeling GR)
The math of GR is challenging, no doubt. But I'd say from observing physics forums that the #1 source of confusion is not with the complexities of GR, but with the basic concepts of special relativity.
Dealing with the math to get out numbers is difficult, but conceptually the switch from SR to GR is rather like the switch from plane geometry to the geometry of a sphere. If you draw a map of the Earth's surface on a flat sheet of paper, the sizes and shapes of the continents will be distorted. Only if you draw the map of the surface on a globe will they be preserved. Similarly, the space-time intervals around a massive body will be distorted if you draw them on a flat sheet of paper. There is a mathematical surface you can draw the space-time diagrams of the space-time around a massive body on that's curved that will preserve the intervals, at least for the r-t plane. You'd need at least a 5-dimensional sheet of paper (and probably more, possibly even much more), if you wanted to draw the entire 4 dimensonal space time on a flat surface, which would be not be terribl useful as a visual aid
Fortunately it's possible to handle the problem mathematically, even in the absence of a scale model. Just as it's possible to perform navigational computations on the curved surface of the Earth with charts drawn on a flat piece of paper, by noting the distortions that the maps have, it's possible to perform similar computations in the curved space-time around a massive object. The way in which the corrections are introduced is called a metric - a spatial metric for the surface of the Earth, and a space-time metric for general relativity.
This idea isn't really terribly hard to grasp, though carrying out the detals correctly is challenging. But in order to get anywhere at all you need to know how to draw space-time diagrams and how to transform them so that you can go from what person A sees to whiat person B sees, which is in the domain of special relativity rather than GR. And it is in this basic task that I see most of the problems in understanding.