You're right, but I think that with Advanced Calculus by Edwards and Elementary Differential Geometry by Pressley you would be okay, especially if you've had multi-variable calculus (I don't know whether Spivak covers that or not in his Calculus text). In fact, Edwards develops the necessary linear algebra in chapter 4, after spending the first three chapters building intuition and motivation for the three main chapters 4-6, which provide the main theory. He also has many interesting applications of the theory to differential equations, complex analysis, physics, and more in chapter 8. He even derives Einstein's equation, E=mc^{2}. Basic experience with matrices and determinants, plus multi-variable calculus should be enough for Pressley's book.I wanted to mention that, for most of the books mentioned above, you need to know some linear algebra (in addition to the other prerequisites listed above). Also, Spivak's Calculus on Manifolds is a tough read for self-study. Not only is it terse, but it contains many typos and misprints. I learned this from working my way through it several years ago.