I have not read Spivak's Calculus, but I do know it is basically a book on analysis of functions of a single real variable. There are a lot of topics a physicist should know that are not in that book. As a bare minimum, a physicist should know multivariable and vector calculus, linear algebra, and differential equations. Partial differential equations, complex analysis, numerical analysis, probability and statistics are also very good for a physicist to know.
If you are limited to just a couple of more books, there are several good "mathematical physics" type of books that come to mind: Boas has written one (do a search in the forum), and I also like Riley, Hobson and Bence. Prof. Nearing has written an e-book that also has some nice insight:
http://www.physics.miami.edu/~nearing/mathmethods/
If you finish with Spivak and want to continue along those lines, Calculus volume II by Apostol has a lot of good material on linear algebra, differential equations, multivariable and vector calculus, probability and numerical methods. It is also takes a rigorous approach to the subject.
Enjoy.
jason