If you are worried about the workload, which is a lot, why not just take the subjects you like from physics and not double major? When you major in physics, you will be forced to take classes that you may not be interested in like Labs, E&M, QM, statistical mechanics, etc.
I am double majoring...