yea, i am a bit worried about that, and not just about physics. I am taking the ap calc ab exam, and a 5 on that (for my college) allows me to skip ahead to the 3rd quarter of calculus, and I am pretty sure if i do that, ill start off way behind, cause skipping the 2nd quarter of college calc would mean skipping all the stuff on series and maclaurin/taylor stuff.
But honeslty, i really don't think you will be behind in physics. In the college sequence for engineernig majors, the first physics course is the mechanics component, and if you get a good enough score on the AP exam, you will most likely just start off on the 2nd component, which is E and M. If you can do well on the AP test, i think you will have a pretty good foundation for EM.