First, here is an extremely useful link with a listing of all (probably most, actually) institutions that work in quantum computation, information theory, and the actual physical device physics. Note that these include EE, CS, physics, and mathematics departments.
http://www.vcpc.univie.ac.at/~ian/hotlist/qc/research.shtml
So I've had a lot of interest in this field as well. In fact, reading llstelle's post above brings up a few questions, but I will ask in a bit. So I knew how to program before coming to college, and I started doing astrophysics and quantum gases (Bose-Einstein condensation) research very early. They're all highly numerically based, and I think that is a good thing. Two of my interests are numerically solving problems in physics and engineering, and the other is working in some field related to quantum computing. I see the former as a very practical way to ensure that I have good employment and career prospects in the future. Even if I cannot do Wall St., defense, oil, biochem., semiconductor, or other things that can utilize my numerical skills, I could always work in software (and of course, probably most of the mentioned fields will be closer to software than numerical research, but it all depends really).
Now my plan was to get into a CS department to do this numerical stuff, probably in some sort of field that applies to physics or engineering. What I also found out is that there are a lot of CS departments that are actually EECS, and this makes a lot of sense. There are computational electromagnetics and other related things in EE that are also pretty close to CS things too. Quantum computing is a very obvious field for an EECS department to partake in, and I was wanting to get into a department like that because it coincided with two of my top interests. It also has the perks of being able to get into top schools more easily, having more available funding, and industry options (and of course, physics may give you these options as well, but they are nontrivial to obtain).
I'd like to ask llstelle specifically why he/she didn't like getting into EE (not trying to hijack thread, I think it will be useful for OP as well). As an undergraduate major, I can see all of these being reasons to not do it, but I wonder what going to graduate school at an EECS department would be like. I imagine you wouldn't be forced to take irrelevant classes, as a grad degree isn't meant to give you so much breadth. Regardless, it seems like EE is certainly a good choice if one doesn't mind these other things, and will give you some financial safety net as well. Anyway, I think if one considers graduate school, the problem becomes much different. If you want to work in such a field that is still in its first stages of growth, it is imperative to get a Ph.D.
As a future technology, I think quantum computing will be a very active field by the time I (in my 2nd year now) get to finishing my Ph.D. The company D-Wave has built an adiabatic quantum computer, which it has sold to Google and Lockheed-Martin recently, and IBM just yesterday announced their progress on a quantum computer based on SQUID technology as well. In ten years, there should be more players, and it ought to be of huge interest to the United States government, particularly the NSA. I think if there is any field in physics that is worth putting your bets in for, this would be it. As for how to get there, I know of several people who have made very good careers in quantum computing by roundabout ways. I've recently gotten an internship offer from Oak Ridge National Lab, where my advisor got his undergrad degree in chemistry, a Ph.D in molecular dynamics, but now works on a collaborative effort to simulate an adiabatic quantum computer based on SQUIDs and Josephson junctions (seems to be the most popular attempt other than entangled ions in magnetic trap). I know of others that come from math backgrounds, and still others that come from EE. In such a new and interdisciplinary field, it seems one has the luxury of jumping in from a variety of backgrounds.
I'll leave you with one last link that might be a good reference to see what is happening these days in QC. Also, if you read the wikipedia article on 'quantum computing', you'll see there is a section where it shows all the conceptual models as well as all physical attempts.
http://qist.lanl.gov/qcomp_map.shtml
http://en.wikipedia.org/wiki/Quantum_computer#Developments
I think having done physics for my undergraduate degree helps me tremendously for being prepared for this field. If you know for sure that you want to go to graduate school, this ought to be your best option. I'll be mainly applying to EECS and CS graduate schools, since they lean more towards my own interests and have practical benefits as well, but physics wouldn't be a bad choice at all either (it's just that you're a little more pidgeon-holed with the latter, which works for some people and they do fine).
Anyway, as long as you keep thinking about it and reading material on it, you will be alright. Oh, and make sure you learn to program a lot, that will help you regardless of what you do. Some CS classes couldn't hurt either, particularly algorithms and complexity/computation theory. You'll know more about what to do as you go further in your career. It might also help to pick up a book and start learning some quantum mechanics (from a modern physics book), or a quantum computing book made for computer scientists (yanofsky is an author of this exact title, and it is a good book as long as you do the exercises). Find some research to do in some sort of physics, as it'll look good for graduate school and will help you understand what research really is.
Good luck.