Tenacious IT: 7 true-life tales of techie triumph

For quirky, elusive IT issues, tech pros dig deep within until they uncover the final clue

Stick-to-itiveness solves the tech puzzle
Stick-to-itiveness solves the tech puzzle

IT skills wanted: Tenaciousness, the ability to ask the right questions, and luck. That’s what’s needed to get to the bottom of an issue when those unexplained tech problems hit. Because, unfortunately, it's simply not possible to encase hardware in bubble wrap, train users in every perceived best practice, and thoroughly sweep locales for every hazard.

Published in the anonymous InfoWorld Off the Record blog, here are some real-life stories from IT pros who put in long hours to arrive at that “aha” moment when it all finally makes sense.

IT pros, if you have an on-the-job experience to submit about managing IT, developing apps, supporting users, a humbling moment, or a time when something went very right, send your story to offtherecord@infoworld.com. If we publish it we’ll keep you anonymous -- and send you a $50 American Express gift cheque.

Take cover! Major meltdown
Credit: Thinkstock
Take cover! Major meltdown

Where were the strange messages coming from, and why were files gone missing? Amid a migration to Windows, an IT pro fields several confused inquiries from co-workers and traces the problems to an unexpected source: the general manager.

It started out innocently enough, when the GM acted on his teenage son's advice and deleted extra files to increase storage space and goose performance. The C: drive was fine, and the P: drive (his personal network storage location) did no harm, either.  But then he hit the G: drive (general public storage of shared documents) and the F: drive (the accounting system, the label system, the menu system).

The IT pro gives a stern session on the difference between local and network drives and spends hours undoing the mess. No surprise at all when IT’s request for in-depth user training is approved.

Regular maintenance cannot all problems avoid
Regular maintenance cannot head off all problems

Metallic dust, constant motion, and extreme temperatures: Keeping computers running through such exposures and hauling equipment up stairs and across a catwalk are all in a day's work for tech pros at a steel mill.  

But such challenges are intensified when one crane’s computer starts to have network problems. The IT pros try hooking up a new computer, swapping out the crane’s wireless network equipment, replacing the CAT-5 cable ... nothing.

Finally, the IT department discovers that an electrical short or loose connection had caused the power outlet to lose proper grounding, which then interfered with the network connection. Hours of mental and physical exercise later, problem finally solved.

Over our heads
Credit: Thinkstock
Over our heads

Communications are the lifeblood of any company, particularly one with five locations where four of them are spread across a city distant from headquarters. The tech department upgrades to a microwave point-to-multipoint system. Next is convergence, migrating internal telephone signaling traffic to the data network. All hums along nicely until about a year leater, when the networking system breaks down at the remote sites.

Two techs travel to the beleaguered locations. At the first office, they attempt the usual manual fixes to no avail. They discuss alternative options en route to the second locale and brace for elaborate action, but the sight of workers on the roof changes everything.

Turns out that a roofing crew had moved the entire microwave unit, pointing it in a different direction. When the roof is done and they move it back, all is normal again. So much troubleshooting, such a low-tech reason.

Teasing apart a network tangle
Credit: Thinkstock
Teasing apart a network tangle

Intermittent network problems plague a manufacturing company despite the IT department’s best efforts. Weeks later, the techs finally isolate a puzzler: Two nodes on the network have the same IP address -- one of which is a mystery.

They trace the source to an engineer's office, where a PC set up for development work connects to an unmanaged switch. Also attached is an industrial camera used for product inspection. Two problems: Someone had uplinked the switch to the office network, and the camera had shipped from the factory with the same IP address and netmask as the default gateway for that office network.

Once is enough, and the tech team makes major changes by disabling all unused Ethernet ports in the office, requiring a written request if engineers need one turned on, and replacing unmanaged switches with managed ones. Bottom line: Cameras don’t make good network routers.

Lost in the zone of 'you're on your own'
Credit: Thinkstock
Lost in the zone of 'you're on your own'

All goes well at first when the team upgrades the internet service at a remote site with a POTS line delivered on standard copper with the DSL signal picked off. They have a PBX, so no need for a single phone line, but it comes with the deal anyway so they leave it disconnected.

One day they receive a high bill. The tech calls the number to investigate, and a stranger answers. However, the phone company and the DSL provider say there's no problem and blame each other. Calls to the state PUC and the FCC don't help either.

The tech's theory is finally verified. The POTS number had been installed in an apartment building but wasn’t disconnected when the tenants left. Then the number was reactivated for the company, but the apartment remained as an extension on the line. Because the apartment reamined empty for months and the company didn’t use the line, the error wasn’t caught until new residents arrived. Problem solved -- no thanks to any supposedly responsible agency.

We've tried everything -- haven't we?
Credit: Shutterstock
What are we missing?

An employee brings her laptop to the tech department with a list of “weird” issues: random pop-ups, blue screens, lockups, programs launching unexplainably. Tests show a failed cooling fan. Easy enough, and she’s sent on her way with a new laptop -- but returns within a week with similar issues.

The techs ask her to walk through the problem. The user explains that all is fine during the day, and before going to bed she closes all programs and puts it into sleep mode. But in the morning the laptop shows random programs on the screen.

The team runs more scans, diagnostics, and every update imaginable to cover the bases, to no avail. Finally, they see a clue in the BIOS hardware log, which shows that the PC wakes up in the middle of the night, heats up, then cools off. The user has an idea, and the next day all is answered: She discovered her cat sound asleep on the warm laptop, unaware of all the trouble. Truly, context can be everything. 

A shocking discovery
A shocking discovery

It’s the late 1980s, and a customer has PCs connected to an IBM mainframe controller via coax. But one PC's 3270 emulation screen keeps locking up. And of course the maintenance contract with the customer is next-day service and requires a long drive.

The determined tech tries a new motherboard and a new power supply, swaps the coax card, adds static mats the workstation. Next try: Swap out the whole PC. However, when the tech reaches to disconnect the cables, an unpleasant surprise awaits: a strong shock from the BNC connector. Ouch!

More detective work reveals that the PC is in an old part of the building. Following the trail outside, the tech discovers that the old ground spike doesn’t quite reach the ground when the earth dries out, so the PC problems manifest with weather changes. An electrician replaces the ground spike, and all is well. Much to the tech's relief -- in more ways than one.

What's your tech story?
What's your tech story?

The tech trenches are anything but dull. Share your true IT tale of personal blunders, coping with poor managers, trying to communicate with users, resolving tech problems, or other memorable experiences from the tech job.

Send your submission to offtherecord@infoworld.com. If we publish your story -- anonymously, of course -- you’ll receive a $50 American Express gift cheque.

More in IT careers: