Backup? Documentation? We never needed it before

With no paperwork to fall back on, the techie must unravel a system problem with only one basic tidbit: the vendor name

There's nothing quite like trying to fix a tech problem on very little information. I have to admit this has happened at our company when people outside of the IT department -- say, the electrical engineers -- purchase a turnkey system from a vendor to perform a specific task. In their defense, they often figure out how to deal with problems that come up, so it's not usually a big deal.

But sometimes these systems fall between the cracks as far as any kind of support or maintenance structure. My first viewing of these systems may come when an employee approaches me for help -- after it quit working, of course. In certain cases, the information abyss stares right back at you in all its deep, unknowable terror.

[ Also on InfoWorld: Is your pay measuring up? The 2014 tech salary guide. | Pick up a $50 American Express gift cheque if we publish your story: Send it to | Get a dose of workplace shenanigans -- follow Off the Record on Twitter. ]

One such system was installed at our plant to manage recorded video from cameras on the shop floor. It involved a simple client/server application that allowed users to connect to the server, select a camera, and view either live video or replay the recording from a particular time frame.

This was not a mission-critical system. But it was a valuable tool that employees could use to investigate problems in tracking the product on the line or in understanding mechanical or other problems that occurred. For a while, it worked very well and didn't require special attention -- it was all so easy!

Luck runs out

One day, it all stopped. I was consulted because I was the only person in the department who had experience with Unix systems -- a skill not previously needed in our Unix-free support environment. I began the investigation.

It turned out that one of the disks in a three-disk RAID 5 volume had previously failed and had evaded detection. With no warning, a second disk had failed. Thoughtful vendors design a system with RAID disks in order to increase the reliability of the system. Normally the "R" in RAID stands for "redundant." However, when one disk in a RAID volume fails and no one notices, the "R" might as well stand for "risky."

I had a lot of questions: Where is the backup? Where is the documentation? What can you tell me about the software? How are the data files stored on the disk? Can we initialize a new volume and create new directories so that the application can begin recording videos again?

Blank stares were all I got in response. The employees knew very little about this product. It had seemed to be working so well, there hadn't been a need.

I asked if anyone had contact information for the vendor who had supplied the system. Nobody knew offhand, so a treasure hunt of sorts ensued. Finally, someone found it. Next question: Was the vendor still in business? Yes, it was. Whew!

We called the vendor, but the primary support person was on vacation and nobody else there knew enough to help us. There was nothing to do but wait.

After the support person finally returned, we were able to rebuild the disk volume on a set of new disks and make a few tweaks to the configuration. Finally, after a couple of weeks, it was fixed.

Eventually, the users learned that red lights on the front of the system are bad and green lights are good. We even managed to gather some documentation and a little knowledge of the system. But wouldn't you know it? Due to the age of this system, it has now been retired and replaced with a newer product from the vendor. Maybe this one will be different.

It's amazing how far having even a little information on hand could go in identifying a solution -- and how much time it could save.

Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to If we publish it, we'll send you a $50 American Express gift cheque.

This story, "Backup? Documentation? We never needed it before," was originally published at Read more crazy-but-true stories in the anonymous Off the Record blog at For the latest business technology news, follow on Twitter.

Copyright © 2014 IDG Communications, Inc.