Wow... that's tough. Junior year in particular is rough, but switching at this time is a little late and might require an extra year. Switching to related fields -- i.e. chem, computer science, or some engineering thing might still work depending oncourse requirements, and taking an extra year to "do what you want to do." would probably be worth it if you haven't switched majors a million times and actually now KNOW what you want to do.
What's got you down on it? Is it the classes, tough homework? I know in my junior year we got a lot of "class bonding" going on doing homework together until late hours, which got us all through it. Quite honestly, I switched fields quite a few times. In undergrad I switched from Chem to Physics -- and finished with the physics degree although I was pretty burned out by the end, then I did an Education certification and taught high school, then I got an engineering degree and did research for the Air Force Labs, then I went back to grad school Physics (because that is always what I thought I'd do --I just got burned out frequently and needed to do a lot of other things to take breaks). Now I get to combine all that experience in my current career path! It's exciting.
What did you enter the field of physics thinking you would do with the degree? While a lot of programs teach students only for grad school, have you really explored what you COULD do with a physics degree and maybe some extra classes in education, engineering, business, math, computer science etc? (Some people on the forum might have speciafic comments on that. What are you thinking of doing now? Could you talk to a trusted faculty member in the department about these things?