Yeah, symmetry factors are a b&tch! What you have to remember is that they come from an OVERCOUNTING, so you have to divide. You can see this by explicitly looking at Wick's theorem - a single feynman diagram can represent MANY terms in the Wick expansion, and we only want one of these terms. That's why we divide.
I've been doing loop calculations for years, and just last week, my collaborator corrected me on my symmetry factors! I doubt I'll ever get them right first-time!