The classes for engineering and physics are very similar the first couple years, except that engineering students often have to take special introductory classes for different engineering fields. In my experience these classes can be pretty fun, so it's not a big loss if you do find that you'd rather be a physics major.
I've never heard of colleges making it difficult to switch out of engineering. I changed my major 5 times as an undergraduate. The first time I was convinced (incorrectly) that I had to talk to an advisor, which ended up being kind of a wash; subsequently I just turned in a form each time.