Nielsen&Chang "Quantum Computation and Quantum Information" is a very good book. It covers both the basic physics and algorithms.
Quantum computing is a HUGE field which involves everything from very fundamental physics to CS (just like regular computing). However, if you are specifically interested in algorithms a CS or math background is probably more useful than an understanding of QM. As long as you do not have to worry about HOW to realize a quantum gate the rest is "only" CS and engineering (just as you do not need to understand semiconductor physics in order to understand how to use an AND gate in ordinary computing).
That said, the CS part of quantum computing is stil a very small field, mainly because there is no way to actually test most of the ideas since we don't yet have the hardware.