I STRONGLY suggest you retake the course in college for several reasons:
1. Assuming you do know it all, you should have an easy A your first semester, helping to ease the transition to college.
2.If you don't know it all on the other hand, you'll have made the right decision.
3. Why you may not know all the material you need to know:
a. AP classes notoriously teach to the test. College course often cover material that may not even be a big part of the AP test, depending on the curriculum and the professor.
b. You can study for an AP Physics Test by memorizing formulas and plugging and chugging. This is NOT how you succeed in physics past the high school level. Can you derive any of the intro physics formulas, especially the formulas for fields and potentials of charge distributions, for the radius of a geocentric orbit, for frequencies of pendulums and spring oscillations? This will be asked of you in a freshman physics course. I had somewhat easy professors for freshman physics and this was still required of me.
c. Even if you get a 5 on the AP test, a high school AP course is no where near as hard as a college course on the same material. Why? The workload makes the difference. Are you sure you can handle a college workload in your major? How could you be if you have never taken a real college course in it. It would be best to ease yourself into college courses with the freshman year courses rather than the higher level courses where even more will be expected from you and where the course will move even quicker.
Retake the course. It can only help you understand the material even better, and it's not like it's putting you behind schedule or anything. If you try to move ahead before your ready, it will only make it harder to do well. Now, that said, for all I know, you may indeed be ready. But, in the experience of my friends and myself, it's best to play it safe.
If the AP course is NOT related to your major, i.e. not a major or cognate credit, then you should be ok if you skip it. That will lighten your load of electives and GE requirements, giving you more time to focus on your major.