Discovering the Culprit: A Ladybug's Expensive Impact on Mechanical Systems

[Ivan Seeking]

Do we have any good stories about unusual or interesting failures in mechanical or electrical systems? I probably have ten or twenty, or a hundred good stories, but the most recent involved what turned out to be an intermittent CPU failure in a remote control station on the coast of Oregon. When I finally figured out the culprit - the CPU - I pulled it apart and found a ladybug stuck between two surface mount PCBs. The circuits are so small that a ladybug can now effectively short circuit eight or ten traces at once. Apparently this caused the failure. It was a terribly difficult problem to diagnose. The thing worked 99+% of the time. In fact this problem had apparently plagued the state for several years before we found it. It had caused a general system failure and the loss of the program; which was not backed up and that I had to write from scratch. [yay] They had assumed that the program was lost due to a low battery and a loss of line power; but not so.

A $10K+ ladybug funeral? :surprise: That was just the cost of the CPU.

 

[BobG]

An old used car with a home installed AM radio. I didn't even mess with the radio, which didn't work when I bought the car. I installed my own 8-track, instead (did I really just admit that? ).

The car constantly had electrical problems. The battery would periodically die unexpectedly. I also had to replace the generator. The battery problems still continued. The coil failed and had to be replaced. The battery still continued to die randomly for no good reason at all. The electric cooling fan for the radiator had to be replaced. The battery still continued to die.

Finally, on one of the few days the car actually worked, I had nothing to do, so I finally pulled out that old AM radio. In the back, the power wire was loose, which is probably why it didn't work? Traced the wire and found the fuse wrapped in tin foil . Unwrapped the tin foil and found a blown 25 Amp fuse, instead of the 2.5 Amp fuse that should have been there Wondering how in the world the radio blew a 25 Amp fuse, I traced the wire further back, all the way to where it had been spliced directly into the battery cable. :surprise:

No more battery problems.

 

[Ivan Seeking]

What! :surprise: No more stories?

Come on, some of the best lessons in life come from stories like these.

Does anyone remember generator station worker who dropped a wrench directly into a generating system and created a cascade failure that took out a good percentage of power to US west coast? "whoops...heh...heh... got to remember to keep those wrenches on strings".

 

[Hurkyl]

I, er, never fail

 

[Ivan Seeking]

Okay here is a good one witnessed by an old friend of mine. In some diesel engines the fuel injection pump can or could fail in the full throttle condition. Since there is no spark, the only way to kill the engine is to either shut off the fuel, which takes a bit, or to block the air supply by placing something over the intake - which in this case was 6" in diameter pipe with no air filter in place.

The engine failed as described but with high idle governer in a failed state as well. This meant that the engine would self destruct in only ten or twenty seconds. One quick thinking and knowledgible mechanic immediately realized what was happening and scanned the shop for something to block the intake - a piece of wood, metal, rubber...really anything to block the air supply. He saw a jacket nearby and grabbed it and threw it over the intake pipe. His boss's new and expensive jacket was sucked in all the way to the manifold.

 

[Gokul43201]

I've heard of an Intel Fab that went bonkers because a MacDonalds opened up a block away.

I've also heard a story from an NMR group that once let a repairman into their lab to fix a leaky water line. The magnet was on and the plumber walked too close with a wrench in his hand...Several days of agony followed.

 

[BobG]

A new maintenance guy was doing a preventative maintenance inspection on a satellite tracking antenna and evidently lost his balance and kept himself from falling by breaking the helix that actually transmits the radio signals out the antenna. Having a strong sense of responsibility, he felt if he broke it, he should fix it. He superglued it back together. Then went home for the weekend.

Talk about unfindable problem! Now the antenna would almost work (really weak signal). An entire weekend of trouble shooting, thinking they might have fixed the problem, only to find the signal was still too weak. And who's going to think "You know, maybe we should check to see if the helix has been superglued together."

Finally, the maintenance guy comes back into work on Monday, hears about all the problems they've been having all weekend, and decides maybe he better ask someone if you can superglue the helix back together.

Good rule of thumb? The last thing the maintenance guys touch is the first thing to break.

 

[Gokul43201]

There's a long story behind all the wacky fixes we've come up with to operate our cleanroom with an improperly designed and intermittently wonky HVAC system . Here are some of the crazy stuff you'll see in our lab - all of which we put in ourselves after giving up on the HVAC people and Dept. bureaucracy :

- PID controlled heating (most heaters are ON/OFF control)

- Stainless steel spot welded radiation baffles inserted into the AC box to improve efficiency

- Steam Humidifier (for humidity control) using DI water (to maintain Class 100), with a BUNN coffee-maker (modified with a helical coil of stainless tubing inserted to provide heat-exchange) serving as preheater

- Giant filtration system (RO + DI) to provide up to 40 lbs/hr of DI water for the humidifier

- Crazy polypropylene plumbing sticking out the humidifier to compensate for the added impedance from the pre-heater

- feedback (from a pitot tube) controlled flow baffles inserted into the duct to provide constant CFMs of airflow

- manually adjustable flow baffles inserted into AC box to improve dehumidification capability

- there's many more but I'll cut to the fun one : a wacky looking assembly involving 1/2" copper lines, an assembly of modified baking pans, phase separators (mufflers) and a feedback controlled cryogenic valve - all sticking out of a pair of Liq. Nitrogen tanks and leading into the duct. Yes, our Physics Dept. has a 40 year old HVAC system that goes bonkers all the time, and we cool our cleanroom with Liquid Nitrogen ! :surprise:

 

[Artman]

Couple of doozyies.

The piping in vertically mounted high-rise fan-coil units failed from metal fatigue because the contractor had used the wrong kind of copper piping (we had specified the correct type). It was a long time ago so I don't recall the details, but several copper pipes split right down one side near the expansion loops on a 52 story building. Several floors of carpet and wallboard, etc had to be replaced. The pipe was examined and found to be at fault. The problem was corrected by installing a more flexible expansion loop, taking the stress off the pipes (they were strong enough, but would crystalize from the flexing). Probably cost someone over $100,000.00 (three expansion floors at 26 locations per expansion floor) not to mention repairs to the damaged areas.

Another goody was the failure of a cap weld on a 20" condenser water pipe. The cooling towers were located three stories up and the cap was at the end of the header in the chiller room. When the cap failed, the water pushed out of the pipe so fast that it flooded the chiller room and shoved the 20 inch header and the 12" diameter branch piping to all of the eight chillers back about 5 feet. An incredible failure. Fortunately no one was hurt.

 

[russ_watters]

This was fun: we had a condenser/chilled water system with two valves linked together to either bypass or send water through a cooling tower and out to the system or through a chiller (in cold weather, you bypass the chiller for "free" chilled water). The linkage was misaligned, causing it to jam. A controls fault (no feedback) allowed the chiller to operate while the condenser water it rejected its heat into did not go through the cooling tower. As a result, the water just kept getting hotter. A faulty sensor on the chiller (previously scheduled for replacement) failed to shut down the chiller. Result: the chiller overheated, causing it to vent its refrigerant charge (with minimal damage, thankfully) and pvc condenser water piping that was never supposed to see water above 100F got boiling water, causing the entire piping system to melt.

On the same job, a moderately offcenter maintenance supervisor was tasked to clean up some water on the floor of the mechanical room. He decided the room needed a floor drain and proceeded to drill one. Steam rising from the hole encouraged him to stop - he'd struck a geothermal energy source! Well, actually, with impeccable aim, he drilled into all 3 of the wires of the building's main electrical feed.

Did I mention that job is cursed?

 

[Ivan Seeking]

Engineers unlock mystery of car-door device failures

..."This exhaustive search led to general placement of the offending signal -- a transmission from the top of nearby Frenchman Mountain," the statement read.

Their search, assisted by a local radio maintenance company, steered them to a faulty radio signal repeater on the mountain that since the winter had been stuck in the transmit mode.

This caused it to send out a strong signal that interfered with the frequency used by thousands of keyless entry devices and remote garage door openers...

http://www.reviewjournal.com/lvrj_home/2004/Aug-19-Thu-2004/news/24570415.html