You could also do Chemical Physics, which is sort of a blend between the two.
As others already said, it depends on how you like math. In physics, all you will ever be doing is deriving new equations and manipulating them. You will always be introduced to new math techniques and use them to model how nature functions. Its not like high school where its just basic algebra, it gets much deeper. In essence, by the time you're done with physics you will basically become an applied mathmetician.
As for chemistry, you will memorize a lot of reactions and reaction mechanisms. You will understand the 3d shape of molecules and how it contributes to the overall reaction. There is also a lot of categorizing and classifications. Strong math will be needed too, because you will be using quantum physics to explain chemical results.
Seeing as I'm doing both, via chemical physics, I'd say physics is way more abstract. You will be using logic and math to come up with results, and a visual component may not be present. With chemistry you will always be able to see the results, and most of it comes from the lab. I really view physics as a "math", and chemistry as a true "science" in that labs are an integral component. I like both equally well, so I can't give you a preference =)