It depends what you mean by all those words. Some courses are pitched at a level just past calculus, you would have ideal preparation for such a course. Some courses are pitched at a higher level, for which you would want to know more calculus, some topology, some algebra (beyond linear algebra), and know a little bit about manifolds. Does "physics background" mean you have done much with general relativity, differential forms, symplectic geometry, or tensor calculus? As you probably know there is some overlap between those areas and differential geometry.
A book that might be of some interest is Curvature in Mathematics and Physics by Shlomo Sternberg though it is not easy
A problem when learning differential geometry is that many books are too easy, many books are too hard, and many books are about parts of differential geometry that are not relevant to your goals. It is hard to find one that is just right sometimes.