Hello Martin,
Would it help you if I told you it was a breeze for me ? No, of course not. I did physics because math was too dry and chem too wet. All non-scientific subjects were too vague and applied things like engineering or electronics were too applied. I had a great time, played volleybal at a resonable level and had plenty of time to socialize. Even did a PhD and was - and still am - surprised to realize after all this studying you in fact really know next to nothing. But 99.9% of the others know even less and a lot of them are happy too!
So if you have a talent, go out, explore, develop and challenge it. Missing some of the basics isn't all that disastrous if you make sure to keep up with the lectures and exercise sessions.
You are enquiring after a physics study, so there must be something about it that attracts you. Reflect on that and consider if it suits you. And if you nevertheless want to change direction further on, physics is a good basis for a lot of things: I had fellow students switch over to medicine, to banking, to chemical engineering, geology, meteorology and management.
If still in doubt, ask one of the universities of your picking if they can recommend tests to find out your interests and talents.