Yes, there is a little disappointment built in here. It is the math.
It will work to look up mathematical concepts, like e.g. a gradient. But only for a while. You won't miss both, physics and mathematics, if you jump back and forth. So before physics takes place, you will need a basis in mathematics, e.g. trigonometry, differential and integral calculus. However, from scratch also implies a lot of classical results: mechanics, optics, basic electrodynamics and some technical content from engineering. All those can be learned with basic mathematical skills - at least to some extent. What later reads as ##\vec{\nabla}\times \vec{E}=-\dfrac{\partial\vec{B}}{\partial t}## started as ##F=I\cdot s \cdot B\cdot \sin \alpha##. What reads as ##\dfrac{d}{dt}\dfrac{\partial L}{\partial \dot{q_i}}=0## started as ##\dfrac{\Delta p}{\Delta t} \stackrel{t \to 0}{\longrightarrow} F##. What lurks behind ##\ddot{x}+2\delta \dot{x} +\omega^2 x=0## is in the end simply a sine curve. And what becomes ##W=\int F\,ds## starts with ##W=F\cdot s##.
So you don't have to learn all the mathematics first, but at some time you will need to learn it. The easy formulas are only valid in special cases. Nevertheless, you can start with classical physics at high school level. But you cannot start at a certain mathematical or physical point to learn string theory. For that, you need a broad basis. To learn what friction is, you don't.