You need to think in terms of individual schools. Harvard and MIT are *very* different schools. I loved MIT in a twisted way, but I know of very intelligent, smart people for whom MIT was total hell.
Will it make a big difference in the curriculum?
As far as physics goes, the undergraduate curriculum is pretty standard. Schools however have very different cultures, and that makes a big difference, but that often involves individual things.
What about future prospects? Will employers look me differently? (ill defiantly go into grad school eventually as well)
It's complicated. The misconception is that employers will see Harvard on the resume and kiss your feet, but it doesn't work that way. Where a school will help you get a job is...
1) employers don't hire people from Harvard because they think that Harvard people are smarter. They hire them because it's unlikely that you will find someone totally incompetent from Harvard. Think of it like Best Buy. You buy electronics from a superstore rather than from someone at the side of the road, not because the electronics is "better", but because you run into a much smaller chance of running into a dud.
2) some schools are better at marketing their students. You can think of a university as "Brains R' US". If you are an employer and you talk to a school that is good at marketing, they'll set up interviews, work with you to get the students that you want, etc. etc.
3) if there are alumni from the school that you go to that have the jobs you want, you can find all sorts of useful information from them.
Where a school will help you is getting your resume to the people that matter. Also, you have to study this for yourself, because some big name schools are bad at this, and some no-name schools are good at this. One thing that is weird, considering the thread with Nano-Passion is that Rutgers and Baurch are schools that I think are particularly good at career services.
One trend that I see is that there are so many smart people ending up outside the big name schools, that I think that over time it's going to matter a lot less. In 1940, the admission rate for Harvard was 85% (!), where as it's now 6% and heading down.
http://101grads.com/2011/06/04/college-statistics-from-live-and-learn-in-the-new-yorker/
Your chances of getting in if you have a relative that went to Harvard is 30%
http://www.thecrimson.com/article/2011/5/11/admissions-fitzsimmons-legacy-legacies/
P.s. i looked at the list of all the astronauts (yes ALL of them, list of astronauts from nasa, from 1960s to present day) and none of them were from any ivy league schools, which makes me feel a lot better, proof that you don't have to go into one of these super hyped fancy schmancy schools to reach for your dreams
It depends on what your dreams are. People that don't go to a relatively small number of law schools don't make it to be Supreme Court justices. Also I don't know of too many people with big name physics Ph.D.'s working in Wall Street. My guess is that they end up with academic jobs.
Also, it helps to have an entrepreneurial spirit. If MIT rejects you and Frank's Backalley Physics School and Tattoo Parlor accepts you, then screw MIT and turn Frank's Physics School into #1.