In all honesty I do not think you need to quit or you should quit for that matter. I think your problem lies within your studying methods and or your fundamental approach to problems. So to start from the foundation you have to be decent in math in able to do physics. If your are proficient...