Eclair,
It sounds like you're going down a path you don't want to go down and you're looking for rationalization that it's the wrong path for you from someone other than yourself. If you really don't like the subject, than you probably shouldn't spend your life on it, but if you do enjoy it, then you should work at it!
One of the best explanations of learning physics I ever heard was how there's three parts to it -- understanding the mathematics, building a qualitative understanding of the physics (as in an intuitive understanding of the laws), and building a quantitative ability with the problems (or in other words being able to understand what the equations are saying when they are applied to physics problems.) For someone who is good at maths, part one is a breeze. Part three develops as parts one and two become strong enough.
I think more like a mathematician than a physicist. I think very abstractly and find a lot of value in generalization. For this reason, sometimes the qualitative understanding takes a bit more for me than other people, but it's something I have to actively work at.
To be honest, I have found value in some elementary books that are more about the concepts than the mathematics. For Physics courses that don't include the maths, there's a much more pronounced need to properly express the ideas since they aren't including the calculations to back everything up. Since I'm fine with the maths, I read about the concepts. The whole while, I am sitting there thinking about the equations and that's fine because then you begin to think about what the equations are really trying to express and the qualitative and quantitative sides are properly strengthened.