Felix, keep in mind where ModusPwnd is coming from. If you follow his posts here on PF, you'll realize that he had finished his BS in physics, was enrolled in a PhD program which he could not complete (and was awarded a MS due to this), and then ended up working as a pizza delivery man for years before completing some electrical engineering courses (not sure if he completed a second degree or not), and is now working as an engineer. So understand that there is some underlying bitterness regarding his university experiences.
(ModusPwnd, feel free to step in if I'm being inaccurate or have misrepresented your posts).
Now onto your original post. I'm not sure how far along you are in your studies, but how do you know that physics is the only subject that you enjoy? There are very many subject areas that are similar to physics (e.g. electrical engineering, mechanical engineering, mathematics, computer science, etc.) Are you sure you won't enjoy any of them?
It's also worth keeping in mind that a physics degree have the potential to provide skill sets that can be used or applied in areas other than physics (e.g. programming/software development if you take CS classes along with physics). Many physics graduates have done precisely this and are happily employed in such fields. So don't be discouraged from pursuing physics -- just be smart and make sure you develop your skills more broadly.