Ya the title...is it mathematical physics, particle physics, statistical physics?
Quantum mechanics would be a good start. After that, computer science or electrical engineering might be helpful, depending on which aspect of quantum computing you want to focus on. Some English composition wouldn't hurt, either.
quantum optics definitely
The chapters in this book explain it pretty well. In short, some computer science topics like information theory and computation (computability) theory might be useful to know, and of course quantum mechanics (something that uses lots of Dirac notation, e.g. not Griffiths).
Also, I used this book to start out as it's easier to read (Nielsen and Cheung above is advanced undergrad/intro grad level):
Separate names with a comma.