MIdgetDwarf is right, a lot of the first few years of classes are going to be the same for both. One thing to consider is the lab based portion of both. While the structure of the labs for both classes are generally the same (lab reports, careful measurements, and learning what goes on in a lab in basic senses) they are different in what the are trying to convey. It may be beneficial to hold that question until you try your hand at both and see what fits you better. That being said, reading about what physicists and chemists are working on these days might also help you lean one way or the other.