Mathwonk, I completely agree. Hilbert's Geometry and the Imagination is, by far, the most inspiring book in my mathematical library.
Books that I consider inspiring are books that I know was a product of struggle. That is, the mathematician author better have written it with the blood of his heart. In that sense, any writing from any "famous" mathematician is bound to be better than some textbook writer. There is certainly a sense of appreciation reading from primary sources rather than tertiary sources of mathematical struggles.
I really like Vladimir Arnold's books, few of the very rare good post-Bourbaki works of mathematics. Other modern writers I like are Lanczos, Gelfand, and Baez. In the pre-Bourbaki era, I really like Kolmogorov, Whittaker, and Euler.