First, the term "circuits" probably includes a lot more than you think; it's not all resistors, capacitors, and inductors. Most circuits involve transistors and diodes, which are much harder to deal with (there are many simple transistor/diode circuits that you simply can't solve exactly by hand). Your computer's processor is "just a circuit," but the complexity of designing something like that is almost unimaginible compared to attaching a light bulb to a battery. Aside from the use of transistors, you have to worry about stuff like how long it takes a signal to propagate along a wire. What happens if signal A arrives 1 microsecond before signal B? Will that delay make things work differently than I expect them to? My point is, real life circuits can be pretty much as complicated as you want them to be.
Second, not all electrical engineers design circuits. I'm in EE, and while I do have basic skills in the area, circuit design really isn't my strong suit. A lot of electrical engineering work is done at a system level. You don't just say "I need a circuit which makes my guitar loud" and then go draw out a circuit diagram. You first break it up into components. For example, "I need such and such a filter feeding into such and such an amplifier" and then the next step would be to design a filter and amplifier to those specs. But the process of determining what filters and amplifiers and other components you need and how to connect them is a big component of the EE design process. Areas like communications, signal processing, and control systems are all part of EE, but a lot of that is just figuring out the "system" part, without worrying about how you actually implement that system with a circuit. Figuring out the system part can require a lot of math with complex numbers and random variables, which I think is a large contributor to the perceived difficulty of EE.
Finally, as jasonRF said, there's a physics component as well. When you get into high frequency circuits (MHz/GHz), the typical rules of circuits start to break down, and you have to be more concerned with electromagnetics. There's a whole field of EE focused on designing radio-frequency/microwave circuits and antennas. While technically still "circuits," the analysis is different compared to low-frequency circuits.
I could go on, but the point is that while any electrical engineer should have some basic understanding of circuit design, there is FAR more to EE than "just" circuits.