In my experience (3rd year Undergraduate physics major, up to Calculus II and Differential equations), math can be harder to do, but conceptually a problem is not hard to understand (ie, you get a, say, homogeneous differential equation you need to solve, the concept of a DE is the same as a simple seperable-in-variables DE, the process to get the solution is merely more involved), whereas physics is generally the opposite -- the mechanics of it, the integrals and algebra and whatnot, are generally more numerical and simpler, but the abstract thinking that leads you to what process to use can be a big stumbling block.
Obviously this is just at a basic level and there are exceptions to every rule, i haven't studied either in depth enough to know concretely
Also, I am female and I have struggled with conceptual material, especially visualization of 3D environments. I also struggle with pure logic (for instance, computer science algorithms). Ironically, my summer job is in computer programming, so I will have to pony up and work through my problems. Of course, my problems in these areas may be due to my personal weaknesses-- but blaming it on gender is nice sometimes.