Generally, yes, there are some crossover topics that will be available to you. Mathematics is a particularly tricky area to handle, however: there will be lots of areas (notably pure mathematics) that will be inaccessible because you won't have covered the necessary introductary mathematics in nearly enough detail (if at all).
In short, rather than depending on the overall 'subject', graduate studies depend on the way in which the skills you have gained from undergraduate might apply to the topic of study.
For instance, in a Ph.D you'd be able to do something like mathematical biology; where the relevant life-science background knowledge would normally be covered in a crash-course of sorts (possibly through attendance at low level undergraduate lectures). This is feasible because projects of this nature will potentially require only a basic understanding of biology to get underway, but need skills in things such as programming - something that you will have as a physicist.
If by graduate studies you also mean things like Msc/PgDip/PgCert, the scenario is similar. It depends on the topic of the individual course. If there's something you're interested in i'd recommend simply looking around various universities that have active research in that area, checking out their postgraduate prospectus' and finding out what they require for entry.
Is there a particular area that interests you?