Well, it's not so easy. On the one hand, I'm of the strong opinion that math and the natural science should not be taught using the historical approach to begin with. It's the very nature of these subjects that there is progress with time, and problems get solved which have been obscure before. It's not necessary to learn old-fashioned misconceptions to start with a subject. E.g., you don't need Bohr's model of the atom (which in fact cements very wrong ideas about what's "going on" in the microscopic realm, even on a qualitative level) to learn up-to-date quantum mechanics. It may be even hindering a clear understanding of the subject.
On the other hand, one should know a bit about the historical development of ideas of all these subjects to fully understand the implications of the up-to-date state of the art. It's, e.g., very illuminating to understand the trouble of classical electron theory a la Lorentz et al, partially still unsolved today (e.g., self-consistent dynamics of classical charged point particles including radiation reaction; motion of classical charged particles with spin/magnetic moment in a general em. field) to appreciate the modern view fully.
Coming back to the topic of this section of the forum, it thus can be very illuminating to study "old textbooks". Then there are exceptional good ones, which exceed the quality of modern textbooks, particularly about subjects which have not considerably changed in the meantime. Examples in theoretical physics are, "The Feynman Lectures", Sommerfeld's "Lectures on Theoretical Physics" (which in my opinion is still the most concise treatment of classical physics including the mathematical methodology ever written, particularly vol. VI on partial differential equations in physics), Pauli's "Lectures on Theoretical Physics" (particularly quantum mechanics), Dirac's "Quantum Mechanics", Weinberg's "Gravitation and Cosmology" (although maybe that doesn't qualify as an "old textbook").
. SCNR.
.