Graduate schools generally don't care too much about it because there are many reasons why someone would take longer than normal to finish an undergraduate education, including financial and/or family commitments, sickness, and even taking time off to "find" oneself.
There are a few things to be wary of though when making a decision like this. First, there is no guarantee that you will in fact obtain a higher GPA with a smaller course load. In theory, if you have more time, you'll dedicated it to your studies and can generally expect to perform better. In practice, you have to make sure you're not using that extra time in other areas of your life, which from what I've observed, tends to happen.
Second, sometimes taking more courses can actually help you because courses can build on each other and double up on material.
Third, you have to keep the opportunity cost in mind as well. You're potentially adding two years or more of time to complete your undergraduate studies. That's two more years of debt accumulation, two less years of working, two less years of real world experience, etc. Is that worth a potentially higher GPA for you?
Fourth, when you do get to graduate school, as Vanadium implied, you'll be surrounded by others who took full course loads.
I'm not trying to discourage you from this path. Rather, I am just pointing out some of the potential pitfalls.