Are you taking an algebra-based HS physics right now... all that vector, kinematics/motion, conservation of momentum/energy stuff? ...That class that most of your classmates despise? And you enjoy it? In that case, you might be cut out to be a physicist (especially if you enjoy showing off smart you are, or scaring people away with your perceived brain-power... that's really why I ended up getting all my degrees).
Especially if you liked your first algebra class and geometry, hang in there. Some of the higher mathematics (probability, linear algebra, and calculus) are a lot of fun. I really didn't like my required "algebra II" class (which seemed a repeat of algebra 1) or pre-calc (which was only made fun through a lot of goofing off with classmates). But I really liked my last math class of high school (a class they just called "Math V" -- but which covered things like probability, logic, linear algebra, and calculus at a surprisingly accelerated pace).