Personally, I think Electromagnetism is the hardest although I am not through my undergrad degree yet. I found optics very difficult in first year and haven’t done much after that but it probably is hard. I heard many people say QM is difficult and unintuitive and no doubt it is but at the undergrad level we only seem to deal with before and after which is simpler than treating all the in betweens via fields in EM but QFT probably deals with the in-betweens but is not really undergrad stuff. This general trend of not dealing with the in betweens goes for many other undergrad physics subjects but with EM and optics you do even at first year which is what makes them hard. I could be wrong about this as I haven't taken all undergrad physics subjects. If other please specify.