First, as already mentioned, I would shut off the fuse/circuit breaker to that outlet first and also double check, that it is off, with a voltmeter, before doing anything with it. If you are uncertain, I recommend finding someone you know who can show you how this is done. For learning some concepts you might read an online tutorial such as
this
For cleaning an outlet,
Berkeman and
turbo-1 have given valuable advice. A deoxidizing cleaner, that does not leave a residue, such as
DeoxIT would be useful.
For your general question: Is it okay to stick a nonmetallic object (such as a Q-tip swab) into an electrical socket? The answer depends on what you mean by
okay.
If you are asking if it would have any
adverse effect on the outlet, I would say
yes.
The cotton lint from the swab can become caught on the sharp metal parts in the outlet, resulting in an outlet with some obstruction (bits of cotton in it).
Is it an electrical hazard to stick a dry Q-tip swab (or other non-conductive object) into one side of an electrical socket. The answer is
no.
Is it an electrical hazard, holding one swab in each hand, to stick two moist Q-tip swabs into both sides (neutral and hot) of an electrical outlet, at the same time. The answer is
yes!
Here is why..Salts from your skin and even on the swab itself, can dissolve in the moisture on the Q-tip, allowing a conductive path through you. If that outlet was accidentally left live, this can cause a fatal shock.