A very gentle introduction, but still good quality, appropriate for people who are not maths majors or maths whizzes, is Lang's "Introduction to Linear Algebra". Some good features of the book are the diagrams (rare in maths books and especially Lang) and the answers at the back of the book. But the book is very elementary, and won't get you up to the level you need for QM. A very good book to follow this with is Axler, as mentioned before.
If you are having trouble with a math book like Axler, or any other linear algebra book, I recommend you read Lang's book first.