Mechanical tolerances are key to maintaining the transmission line performance of the connector. This is the primary reason why microwave connectors and adaptors are so pricey. You can look at the equation for http://www.microwaves101.com/encyclopedia/coax.cfm#Z0" and see it has the inner, d, and outer, D, diameter terms.
Calculate the wavelength at UHF and microwave frequencies and you see the wavelength is at nearly the same physical size as transmission line diameters. This is why just a little change in mechanicals can change impedance which in turn change return loss of the interface between connector and coax line. So if you are expecting 1% accuracy wavelength of 5 mm and coax dimensions of ~3 mm, a little damage could ruin your day
Every time you disconnect and connector a connector like an SMA, SMB, APC 3.5, 2.4mm, etc. you are causing some mechanical damage. The typical connect-disconnect cycles before the connector exhibits significant return loss changes can be as low as 20-100 cycles for high performance connector (3.5 mm, 2.4 mm) depending on the return loss and mismatch uncertainty you can tolerate for your system performance.
The issue is that because of http://www.eetimes.com/design/analog-design/4017988/Improving-Measurement-Accuracy-by-Controlling-Mismatch-Uncertainty" , you can have a significant effect on measurement accuracy (for an instrument) or power efficiency (for a transmitter) or sensitivity (for a receiver) if you bollux the return loss of components in your transmission line chain.
This is part of why you have to be http://www.microwaves101.com/encyclopedia/connectorcare.cfm. When I worked at HP T&M/Agilent there was (probably still is) a "connector care" training course sold to customers (for several $1K)! It was based on this [URL="http://cp.literature.agilent.com/litweb/pdf/5954-1566.pdf" (PDF)]application note[/URL] but went into more gory details.
This is also where the connector torque wrenches come in: they put exactly the most tightening possible without damaging the transmission line mechanicals. That little pin in the male side of the connector is fitting into a female receptacle and you can overdrive it in and cause mechanical damage. Not connecting tightly enough leaves a gap that causes return loss. Altering the mechanic dimensions of the entire coax line by overdriving will cause return loss.
Also if you've misthreaded a connector like an APC 3.5 or 2.4 or similar M-F connector and then tried to force it even a little, it's probably already had some significant damage (on both sides) that is easily visible with a network analyzer.
Often you'll see people using sacrificial adapters in situations that have frequency connect-disconnect cycles - the idea being, destroy the $400 adapter rather than the built-in connector on the $200,000 network analyzer that will cost your a few days of $300/hour repair and calibration to fix.
Connectors like BNCs don't really matter so much but only because they have absolutely no spec for return loss or any other transmission line performance even defined. They are low-frequency, lumped-element connectors. Thus if you are doing anything where return loss, impedance matching or reflections might matter, absolutely do not use a BNC. Yes, some commercial firms selling instruments or products get this wrong.
Other connectors like Type N or UHF do have some specs for this defined but they also have cut-off frequencies defined.
Of course the main of idea of worrying about connector care in the first place, even if your application doesn't need super good return loss or isn't affected by connector damage, is that connector handling is a habit which should be done well for the times when it does matter.
