OK, here's what is probably happening. Your power supply must not be grounded (i.e. 2 prongs, not three). PSs are required to protect the users (output) from electric shock. This is measured as no high voltages (>30Vrms normally), or if high voltages are present, then the leakage current from the output to ground must be less than a specified value. For computers that value (EN-60950-1) is something like 750uA. For medical patient contact equipment it's something like 100uA.
To achieve this safety the PS has an isolation transformer. Let's say your input voltage is 0Vac at the neutral conductor, and 120Vac at the hot conductor (w.rt. Earth ground). Then the average voltage on the input circuitry is 60Vac. This is the average voltage difference across the transformer insulation. That insulation will probably act like a capacitor and let some leakage current through. It could also fail and be resistive. In PSs with a grounded plug, the transformer will usually be constructed with an electrostatic shield between the primary and secondary to divert the leakage current to ground. For loads that are grounded (not your laptop, btw), it doesn't matter, they will divert the leakage current to ground.
For PSs that don't have a ground connection, it is required that they have really good insulation (commonly called "double insulated"). In any case, the leakage voltage and current must be tested on each PS manufactured, as well as having the safety agency approve the design.
In addition, EMI filters are required to keep the PS from interfering with other stuff. Without going into the details, an easy way to filter EMI has the side effect of increasing the leakage current. So designers will sometimes trade one bad thing for another to meet the EMI & safety standards at the lowest cost. This leakage current can be a key specification that people like me look for when choosing a PS. For example, what if you make a system that has multiple PSs, each of which claims to meet the spec? OK, but your system may not meet that spec when you connect them all together.
So, the leakage current problem is sort of fundamental. The safety standards allow some, because, frankly, without a ground to send the leakage current to, there will always be some. The fact that you can feel it slightly doesn't necessarily mean that it's dangerous. Your sensory nerves are, well, sensitive, and located at your skin which is far away (electrically speaking) from your muscles. An interesting diversion, btw: the old US military safety standards (1.5mA I think?) weren't based on what it took to stop your heart, they were based on what they thought it might take to make you fall off of a ladder and smash your skull.
The leakage current that you can feel is often too small to trip a GFCI outlet. They don't want to have "nuisance tripping" that is annoying, and is also a safety hazard, because it encourages people to defeat them. GFCI is to save your life not to keep your fingers from tingling.
But, it's pretty terrible marketing for your PC to make you feel like you're being electrocuted. So, good manufacturers will ensure less leakage than the standards allow. This is often done with a grounded (3-prong) power input. OTOH, ancient houses don't always have grounded outlets, so some go for convenience.
So, if I were you, I would replace that PS. Because 1) it sounds annoying, 2) it may be damaged in ways which could be scary.
