Is it possible to go to grad school and just study nuclear physics and Quantum mechanics?
no it isn't.
Grad school in the US usually combines the masters and PhD. You spend the first 2-3 years taking the masters coursework (usually 2 semesters each of quantum, E&M, classical, thermo, and math physics, or some combination of those) and more specialized topics courses in your field. Then you need to pass a qualifying exam on those subjects before proceeding to the dissertation to earn a PhD. So no, you can't show up and just take a couple specific classes and earn a degree. You need to take them all and demonstrate your knowledge of those fields (and often others as well).
What course on QM would you take after 2-3 grad semesters of it? What would be the syllabus for such a class that wouldnt just be more appropriately named something else?
After 2-3 semesters of grad QM its time for QFT/Relativity/Critical Phenomena or something else?
ok can I go to math grad school and take my required math classes and then take QM classes for mathematicians
There aren't QM classes for mathematicians. There are only QM classes in physics and chemistry departments. If you want to take a course in QM, study physics or physical chemistry.
I know for a fact their are math programs that teach QM, their emphasis is a little different
What are you trying to accomplish? Are you aiming for a Master's or a PhD? As mentioned before, you will need to take other course alongside QM. Some departments will allow you take courses outside of the department with consent.
This is true. It is also possible to take a QM from the physics department if the person/people (advisor/dept) approves it.
Ideally I would like to study infinite combinatorics , knot theory, QM, and relativity all at the same place to at least earn a masters
Do you have even a single source to back this up? Take a look at old Caltech courses for some counter examples to this insane claim (e.g. the older ma 148c).
I suppose if he only means classes which have that exact name otherwise every university I have attended had such a class although under different names.
Separate names with a comma.