I really wouldn't class learning calculus at 12 as child prodigy material, calculus (at least at the level any of the physics 'child prodigy') isn't very hard, most people could probably learn calculus at that age if they were actually taught it instead of being stick doing arithmetic for years, but I digress
He's one of the greatest physicists that've ever been, he wasn't a child prodigy
Yet another great who wasn't a child prodigy (did learn calculus at 15 or something though)
And who could forget the one and only George Green!
You could do what I do - study 6+ hours a day and constantly push forward. I only manage this because I like the subject, I don't really care if my name goes down in history, if that's your sole reason for attempting physics then you're in the wrong field.
Mathematical Methods in the Physical Sciences - Boas
Introduction to Linear Algebra - Strang
Modern Algebra - Gilbert
Introductino to Tensor Calculus - Heinbockel
Methods of Theoretical Physics - Morse and Feshbach
Advanced Linear Algebra - Roman
Advanced Calculus - Loomis
Electromagnetism - Jackson
Classical Mechanics - Goldstein
Mechanics - Landau and Lifshitz (my personal favourite between this and goldstein)
Modern Quantum Mechanics - Sakurai
Particle Physics and Introduction to Field Theory - Lee (the font they use in this book is terrible though)
I tried to put them in a rough order but really no set of books ever flows neatly from one to another (unless they're all part of a series - *Landau and Lifshitz Course of Theoretical Physics*)
Good luck on your journey into the world of physics, but learn to love the subject for what it is, not for whatever recognition you may gain from it.
*I just noticed that my post seems to have flown wildly all over the place...
But I think my book list is pretty good so I'll keep it!