One thing that should be pointed out, is that most people who study physics end up working on problems that no one outside of physics even know exist. In fact, if - for example - you are working on a problem in electron optics, you will talk to a particle physicist and they will have a very difficult time understanding your work.
If you are genuinely interest in the stuff that makes the cover of Scientific American, you can certainly choose an institution and courses based on the goal of getting to work on those problems. But a lot of people start studying physics because of those physics 'porn' topics, but end up enjoying working on something a lot more concrete like medical imaging or solid state physics.
If you dislike the mathematics and find it too difficult, then you will only find it gets worse as you go further. The learning curve for the mathematics is very steep, and in order to have a good foundation in physics, you will learn a breadth of topics, most of which you will probably find boring.
There were things I always found boring, but now I actually need to use a lot of those concepts in order to solve problems that I'm working on and they have become interesting to me. It also helps when you understand things better. I really disliked crystallography in second year, but by fourth year I was understanding the concepts a lot better and the courses were going into a lot more detail about it. Also, at higher levels, all these different topics start to look like aspects of the same thing. You can always draw parallels between different topics, and any mathematical 'tricks' you learn will be applicable to a wide range of topics. In some ways this makes it easier, but the mathematics is still very difficult.
The main thing is whether it is worth it to you. I was never that good at mathematics, but I wanted to learn physics so much that I put in everything I had. If it's worth it to you and you put in the effort, you will enjoy it.