I think I always would have had an interest in physics, even if I hadn't had the opportunities I've had in life. I doubt very much though I would have been able to get to my current operating level though. Personally I need the rigor of a structured course for many reasons. Ones I can think of off the top of my head:
1. Assignments and lab reports with deadlines, mid-terms and final exams force me to cover a subject in a systematic and timely way. Left on my own, I tend to follow my immediate interests a little more.
2. Often it has helped me to have someone more educated than I show me *why* something should be interesting.
3. There was a lot of material that flew over my head at the time we covered it in class. I learned it sufficiently to get the grades I needed, but it wasn't until later I realized how important it was (at least for me) and I at least knew where to look and what to look for when reviewing.
Something else to consider, is perhaps a little more pragmatic. As a
medical physicist there was a lot of stuff I could ONLY learn in a clinic through hands-on experience. A lot of that is not quite so intellectually impressive as the material that would be covered in other branches of physics, but absolutely necessary for me to do my job and partake in the research I do. The analogy I might draw is one of a carpenter. You can read all the books and articles you want, watch all the YouTube videos, etc., but at some point you have to actually build something and until you do... you're just a guy surfing the internet.