No. Dont drop A level physics, it will cut you out of engineering, most science and even some math courses for university.
I was like you, I went from an even worse school with these gcses:
A* AAA BBB C
and got this at AS LEVEL
A* BBCC - Economics maths physics english and chemistry. respectively
and got this at A2 level:
A*BC - Economics maths physics. School wasn't to interested in me doing chemistry and english after getting a C!
What were your gcses? Do you feel like you learned something in gcses? I sturggled because I never learned anything during gcses and found it hard, like you, to apply the concepts. Mathematical thinking requires flexibility and awareness of how the concepts relate to each other. I will try to write up a list of skills I wish I had before entering a levels, but here's the current advice for you:
1. Download the syllabus for your a levels, for CIE they called it the learner guide. They list the content you need to know. KNOW THIS inside out.
2. Start doing more and more questions and thinking more about what you don't understand throughout the day, very important in math IMO.
3. Start doing past papers as early as possible, go through them at least for now.
4. GCSES don't really prepare you if you don't do well in them. and if you do badly in them usually you do badly in as and a levels too. people with all A*s in gcses got All A*s in a levels too. But no one with a record like mine got more than one A* FOR THE MOST part.
The competitive atmosphere is a good thing, because even though my grades were pathetic the school I went to and lot of personal issues I had, would have meant I wouldve barely passed if not for competitive atmoshpere.
Also note a levels are a standard across which everyone is measured, so doing well in them ensures you go to a good school, and doing badly in them means you might not go to the school you want, and university gets even harder and the content is less organised for you. Better prepare yourself to work hard and stay on top of things mentally, university is 10x harder.