Check those old (about 2002) interesting quotes from a Slashdot thread that shocked me. Read them carefully and then decide. Hope it helps.
My humble personal opinion: Do what you like most. You like physics, get into physics. Do not to go after thew bucks if possible.
I'm someone with a lot of mathematical training (Ph.D. in Applied Math) but only a few courses in computer science. Somehow, I've managed to pick up a humungous amount of CS along the way, things like algorithm design and analysis, designing and coding industrial-strength C/C++ libraries and applications (yes I get paid for this), high-performance computing, OpenGL coding to roll my own volume visualization apps, doing all of my own unix system administration, setting up all of my own hardware...I've always thought that the best way to become really good at coding and software engineering is to first get a degree in mathematics. If you can do that, the rest is easy.
(Okay, I am a bit biased; I'm a college math professor, and in addition I do a lot of research and consulting related to numerical computation).
Your code may do the job, but does it do the job efficiently? And if it didn't, how would you know?
I changed majors from CS to Mathematics halfway through because I realized that programming is easy; you can always learn a new language or a new technique by picking up the appropriate O'Reilly book on the subject. But writing good programs -- programs that are robust, that scale well, that do as much as possible as quickly as possible -- is really applied math. And math is hard.
Here's a free hint for code monkeys: If your only value to your employer is your skill at the currently trendy programming language, you can be replaced by a 16 year old, and probably will be when the programming language of the day changes. If you want to have some respect, and some security, become skilled in some field such as physics or engineering (or chemistry, or mathematical genetics, or anything which is HARD). Let the coding be the tool which lets you do what makes you valuable to your employer, rather than the only value you can provide.