Well, calculus is not that difficult, if you learn it in the physics context. Which book are you using? Usually school books (at least German school books) are less understandable than introductory university text books, because they try to be didactical and obscure the intuitive as well as the abstract meaning of the subject. Among the university textbooks the books for pure mathematicians who must learn everything with full rigor, are more difficult than the more applied once. When I was at highschool I had big trouble with math and couldn't understand my school books. Then I went to the library of my home time and got some book "Calculus for Engineers". This was the revelation, and I got hooked up with math and finally physics ;-)). I think math is the one subject of all sciences which gets distorted most in school.