So I personally did not do my undergrad at either(I went to another relatively similar school) and being a grad student at a school does not give me the same perspective I would have as an undergrad. However, I do have several friends who did their undergrad at both and have observed a fair amount as a grad student.
You should consider what kind of academic and social environment/school culture you would like. MIT is not really a university environment, it is first a technical institute. You will be required to take humanities courses, but no one goes there to MIT to major in English. At Harvard they really seem to emphasize core requirements and getting a liberal arts type education. That doesn't mean you won't be able to take as many classes in your major, it just means they want you to explore other areas to get a well rounded education. That means you will also likely interact with people studying all sorts of different things which I think is a very good thing.
In terms of school culture, both are MIT and Harvard are very high pressure environments. However, because students at MIT are for the most part (not counting the Sloan students) studying science or engineering, it likely makes it much worse. MIT has no grade inflation and the average undergrad there has a very tough workload, much higher than other places (partially since everyone is majoring in STEM). Students do seem to be more internally competitive than with each other because the workload is very humbling. Everyone needs to work really hard to succeed no matter how smart they are and no one hides that. Even so, it's a really difficult environment and the amount of pressure can be debilitating for some students.
Harvard on the other hand is a very competitive environment but in a different way. There is a lot of grade inflation (I'm not saying you should based your decision on this, I honestly think the extent of grade inflation is ridiculous) and more hand holding. The competition is also internally motivated but there seems to be a type of pressure to prove yourself to others (from what I have observed).
The physics departments at both schools are great and I think students (at least the grad students for the most part, not so certain about the undergrads) seem very happy.
In terms of the social environment, MIT has a lot of Greek life (which seems to be less extreme than most schools). Parts of Harvard's social scene (definitely not most) are incredibly elitist. I don't think you will see that at MIT, people there to me seem more down to earth.
The terms I used to describe the social environments of these schools can for the most part be used to describe all of the Ivys, Chicago, MIT, Stanford, Duke, etc since they attract very intense students. While these schools all have tons of academic opportunities, one has to be pretty strong-willed student to be happy at such schools.
