I took first year EngSci at U of T last year (that is, 2003-04). I came from a pretty good high school, where I graduated with a 93% average. I finished first year Eng Sci with an 89.5% average, pulled down largely by "philosophy" because the philosophy course was... odd. On the other hand, most people I know came in with a 95% and ended up between 60 and low 70's. Many people failed or left (by the second year, I think 2/3 left, 1/3 of the total leaving by first semester's end). It is indeed possible to do very well in Eng. Sci., even with ease, but the chances are low, and you'd probably need a good idea of how your high school stood in relation to others in order to make a reasonable prediction (important if you are apprehensive about taking a program that you might struggles with, but not if you're expecting a challenge). So taking the statistics into consideration, the difficulty of the program will be a factor you may need to consider (it certainly turned out to be for some people I knew).
I just finished my second year of university, but I was in a math program for the last year. I switched out of Eng. Sci. after the first year. This is because 1) I wasn't looking forward to PEY, in fact, I had no real desire to work as an engineer, 2) I wasn't enjoying chemistry, labs, technical things and was mostly interested in math and physics, and some philosophy. Switching over to Arts and Science allowed me to do this, so I have a program much better suited to what I want to learn. The challenge is still comparable, in that it is not a difficult challenge (as OAC was) but it still requires decent effort and keeps you learning. However I think the workload is a little lighter. So you have to take what you're interested into consideration. Of course, if you're interested most in nanoengineering, then Eng Sci is right up your alley.
I think if you live on campus near the engineering buildings, you can get a decent group of people in your program to network with, which makes a number of things easy, so I would recommend that to you. Generally, I think it's a decent program, but the above, plus perhaps money (it's one of the more expensive programs, on the other hand I earned enough in awards to pay for a years worth of arts and science tuition and books), are things you should consider before deciding.
EDIT: You shouldn't only get one opinion. I know that when I was in Eng Sci, I got some (not as many as I could have) opinions, but even the few that I got were quite helpful. Although I only have my opinion, I think I can tell you that the typical, average opinion would be that you need to work harder than you did in high school, and expect to get a slightly lower mark than in high school. It would also be typical to find that people (at least those who stay, and even some that leave) found the program reasonably enjoyable.