I would think you could as both utilize physics but perhaps not as much as what you've studied. There may be a couple of courses needed to get you prepped for the advanced degree in the specific field but I think that is a rare case as grad school tends to cover the material again but in greater detail.
However, it's best to check with the graduate schools that you plan to apply to be absolutely sure.