For grad school in physics? It will depend on the sort of physics you do, and the particular requirements of any grad school you want to go to.
I did no undergraduate chemistry, and it hasn't been a problem in terms of admissions. Due to the kind of physics I do (experimental low energy nuclear physics) there have been situations where I regretted not having done more chemistry, but what I have needed to know is pretty basic, I have had no problems picking it up since I did high school chemistry.
However, if you were doing theoretical high energy physics you'd probably never need your chemistry knowledge. If you were to do experimental material science, a good understanding of chemistry would be fairly crucial. Basically, it depends!