You're the only person capable of weighing up all the factors about coming to a decision on taking maths early. Whatever you decide, you'll be fine in the long run I'm sure.
I mean... give it a little while and you'll forget you even did A-levels, let alone GCSEs! :)
So don't worry! Carry on doing what you're doing and you'll be fine.