Looking at Hartle's Gravity, which I think covers the most amount of Physics without much mathematical formalism, I think math up through vector calculus and ordinary differential equations would be enough to get a start, but you also need a strong understanding of Physics up to that point.
I think a good criterion is some comfort with Maxwell's equations in differental form (e.g. Feynman vol. 2).
We're talking about some 5 years of Math and Physics education here, unless you're exceptionally brilliant. I know that seems like an eternity at your age, but it's not that long, really.
You could probably start on Special Relativity now with one of N. David Mermin's books. Or once you have a little trig under your belt, Taylor & Wheeler (the red paperback 1st edition that's always been preferred here.) Also semi-popular books like Gravity from the Ground Up.