I would be highly suspicious of a quartz watch being affected by a magnet. They do not have any mechanical motion outside of any physical hands and such. The timing mechanism, the quartz crystal, is fairly resistant to magnetic fields from what I know. A mechanical watch is affected because as previously stated the magnetization of the parts will create additional forces on the gears and such which will result in inaccurate running. A watch has to be very very precise to keep decent precision throughout the day, some mechanical watches easily run at around 20,000 beats per hour. That's 480,000 beats a day. If the watch gains say 4 seconds a day, then that means that each beat is only off on something of the order of a millionth of a beat. So even small imbalances can add up over time.
However, fortunately, any decent mechanical watch will have magnetic shielding in the form of a thick iron plate that sits behind the watch mechanism (though this is not there for watches with skeleton backs so that you can observe the mechanism). This help keep the watch from being affected by a magnetic field. If a watch does become magnetized, I have heard that the affects are rather pronounced, something on the order of minutes for each hour. A watchmaker can demagnetize the watch and correct the problem.