This is very much the position I am in. I've always had a strong interest in science, especially physics, but I thought I'd never be able to study it above the level of semi-popular accounts. I wasn't good at math all through school, (even though I did have a vague interest in it) to the point I thought I was *unable* to learn it.
The irony is, I took honors-level science classes a year ahead of schedule, but I was always a year *behind* schedule in math. The process of trying to do math in school was so tortuous, I thought I was only a "word" person who happened to love science, because reading and writing came effortlessly to me, whereas math didn't. Math was the only subject in which I've ever had test anxiety.
Later experience, after I became an adult, forced me to try different approaches to teaching myself, ultimately proving that I could learn to do math well, just not the way it was taught in school. Now, I have the chance to study physics, and I'm even working at learning how to write proofs.