A large part of it is the uncertain career outlook. If you are an engineer, you can almost certainly get a job in a technical field right out of college. Physics majors, on the other hand, end up all over the place (insurance, finance, teaching highschool, programming, etc). If you want a job in a traditional technical field, engineering is a much safer bet. For most people who have an interest in physics, an engineering degree is a better path to their long term goals.
Of course, if your goal is to learn some physics (and who cares if you never get a chance to do anything with your knowledge), then its a great major. Its a good stepping stone to lots of other graduate disciplines (lots of physics majors get engineering or economics masters degrees), and its an interesting field of study.