You have to decide what you would do with the math background. If you go to math grad school, it's primarily aimed at training math professors. That means you have to think about teaching. I thought teaching would be cool, but I was wrong. Teaching math at lower levels seems to me to be a purely social or psychological endeavor, rather than a mathematical one. I thought teaching math would be just explaining ideas, but that may only be true at a higher level. So, you need to get interested in helping people learn very obvious and basic things all the time. I think of it as a puzzle of trying to get across to profoundly non-mathematical students. It's kind of like signing up to coach Olympic athletes, and then, you realize once you are on the job that you are coaching the special Olympics, which is very nice, but not what you signed up for. This may sound derogatory, but I am just trying to convey the point that it might not be what you expect. You might try reading one of the books by John Mighton, the founder of the JUMP math program to spark your interest in teaching and helping the mathematically disabled recover from their disabilities.
You could very well find yourself interested in math for several more years, but it might not last longer than that. A lot of people figure it out in grad school. People like me who, at the end of their undergrad, thought it was a no-brainer to be a mathematician, only to have the wind knocked out of their sails completely in grad school. Research can be very unpleasant. You might find out that someone already proved your theorem 10 years ago, or you might find that you make a subtle mistake and have to throw away 20 pages of your thesis that you worked so hard to write. You may or may not do well under the pressure to publish. And you may wonder why you are working so hard on something with only a very hypothetical use to society. Math may turn out to be bigger than you think. It's easy to say you know it's big, but you don't really get a feel for what it's really like to have to deal with such a huge volume of information until you actually do it (or, more accurately, when you fail to do it--even within a what would seem at first glance to be fairly narrow field). Research is riddled with these sorts of dangers.
