Having read everyone's terrific responses, I guess here's my 2-cent worth...
One of the major difficulties in E&M is the mathematics. You will need to know 2nd order differential equations, special functions, Fourier transforms, etc...etc. More often than not (and I've said this already in some other strings), it is the mathematics that bogs down the understanding of the physics. You have to know how to do all those spherical harmonics/Legendre polynomials/Bessel functions, etc just to solve for all those boundary conditions.
The problem is made worse in many schools (at least, here in the US) when the students are forced to learn the mathematics at the same time they are learning the physics. This is true also in QM - there is nothing more daunting than to come across the word "orthorgonal" for the first time in, of all places, a QM class, for example.
One solution here is to have the necessary mathematics BEFORE one actually enrolls in such classes. Unfortunately, this means that one may have to delay many of the physics course till one has "enough" mathematics, and most physics majors simply do not have the patience or resources to do that. It is why a typical physics student simply does not have enough mathematics by the time he/she enrolls in an E&M or QM class (typically 2nd semester sophomore/Junior level).
Some schools have started offering a class in mathematical physics. Unfortunately, most of these are seldom designed for students who had just had intro level calculus. They are typically meant for more advanced undergraduates, and they also used texts such as Arfken that requires a more sophisticated knowledge level. So this doesn't help much either for students about to jump into E&M and QM classses.
What I HIGHLY RECOMMEND is that you get this text:
Mary Boas "Mathematical Methods in the Physical Sciences" (Wiley).
This text is meant for students at the sophomore level who has had at least basic calculus. It, and the accompanying Students Solution Manual (sold separately) is meant for self-study, and she is quite verbose in her explanation of each topic. In the solution manual, she TELLS YOU why she is solving the problem one way and not the other. The book covers all the basic mathematics tools that one would need in various undergraduate physics/engineering courses. The chapter on Calculus of Variation alone is worth the price of the book. She also doesn't bore you with the gory details of the derivation of the mathematical principles. She simply tells you how to use it, when to use it, and for what limits is it valid. For a physics/engineering major, that's all we care about in mathematics.
I have recommened this book to many students, and all of them have found it extremely useful. I hope this may solve some of your difficulties with E&M.
Zz.