Dang, the exact analogy I was going to bring up!
Personally, I've seen 2 and heard of 1 student's stories about switching fields like this that didn't have very good consequences. One was an electrical engineering student doing his graduate work in physics. He had more of an understanding of physics than a math student would. In order to get into the program, however, he had to take our undergrad quantum course, which he promptly dropped out of (and then the whole program) a few weeks later saying none of it made sense (although... well, it is quantum...). Another one was a math student who I believe basically tried the same thing as you did and going right into the graduate texts for him was like reading gibberish. Similar results there.
The other student isn't so relevant but still humorous. There was a math major who was about to finish his degree and he had to take a intro physics course as is required by the university. Well, the professor who was telling me this story told me that one day the guy comes up to him and says that he FINALLY knew what a derivative meant! Of course he knew what it was from a mathematical standpoint, but no connection was ever made to a real application of it such as position -> velocity.
On a more personal note, I felt a little of what you might be faced with. I ventured into my first quantum field theory text last semester and while I'm familiar with all the math involved, the author immediately throws particle physics information at me as one would expect. Now, I had never taken our particle physics courses and in the other courses, they were always part of the last chapters of a text so with the pace our university goes, we never really would get to them. So when I hit that, I found myself having to go look back at Griffiths particle physics text. Upon reading that, still even half of it Griffith's text was gibberish to me which made me need to go back even further.
Moral of the story, graduate texts assume a solid undergraduate foundation. For example in Jackson, he assumes you basically know by heart most everything that you'd find in Griffith's E/M text. Finally, I do remember yet another stark reminder of the difference between physics and math! A PROFESSOR of mathematics was taking graduate courses with us one year. He was an absolutely great math professor, he knew his stuff, probably one of the most capable in his department. When he took our courses, he just did NOT get it. From what I hear from more experienced people is that physicists sweep a lot of things under the rug and say "let the mathematicians confirm this". Of course, that doesn't mean we do anything illegal, it just means that a mathematician might witness a course unfold and be confounded as to why we do the things we do. It was quite amazing to see him have major issues with things we did and I would stare at the board and think "...what is wrong with this?". I didn't even know what COULD be wrong! So from what I hear... I think the typical thinking is that mathematicians and physicists are sometimes in very different worlds, even at their more advanced levels.
