I believe that the overall impression really depends on where you focus your studies after your bachelor.
To give you an example, I am a mechanical engineer with a master in aerospace and focus in numerical optimisation, and my ongoing PhD is in biomechanics and bioelectromagnetism. In my university, mechanical engineering was considered to be the hardest overall, even though electrical engineering was harder to get into. Make no mistake though, if I had studied electrical engineering I'm sure I would have had much more trouble, because I simply like ME much more, and my mind is geared towards these types of problems.
I can tell you that if you choose to go towards any sort of continuum mechanics, there will be a lot of math. Some good examples are fluids, viscoelastic solids, and of course electromagnetics. The same goes for grids, projections and mappings, where you have a lot of tensor calculus. Most of the other stuff I think are easier math-wise, and the problem solving paths are more standarised. However, what ultimately shapes your opinion is how deep YOU choose to into the mathematics and modelling. Often, you can pass a course without having really understood the physics of what it describes (usually attributed to knowledge gaps as to the meaning of the mathematics used).
My overall impression on peoples's perception on this, is that fluids are usually considered to be the hardest. What is hardest to YOU however, depends on how much you like something. For instance, I never failed a course in fluid mechanics, while I failed a lot of times in mechanics, which is considered by a lot of people to be easier :P
Edit: Also keep in mind that a reason that a lot of people consider ME to be the hardest, is because its curriculum usually has a lot of diversity. This pretty much guarantees that more often than not, you may stumble upon things you simply don't like :) This also depends on your faculty of course.