In many IT organizations, there's a strict schedule for on-call duties. As with other professions that transcend the 9-to-5 world, someone must be available to answer the phone when it all goes pear shaped, and ideally that person isn't working on his or her fourth bourbon on a Friday night. In most shops, this is a simple weekday/weekend schedule that rotates people through a more-or-less regular cycle of availability and triage duties.
This sort of on-call schedule works quite well in many organizations. In others, it's a joke.
[ Also on InfoWorld.com: Read Paul Venezia's instant classic, "Nine traits of the veteran Unix admin." | Then, if you dare, join the debate about rebooting Unix-based systems. ]
At the center of most IT shops is a core group of admins or engineers. They probably had a hand in building the infrastructure, or at least a significant part of it, and are among the few people who can diagnose and address major problems as they occur. These aren't the folks that get called in to handle a partition filling up unexpectedly, mind you -- they're the ones who are alerted when nobody else has any idea where to even start looking for a fix to a problem.
In other words, these folks are on call 24/7/365, whether they like it or not.
If you're one of these people, you know as well as I do that you can forget about turning off your cellphone when on vacation. Every flight to Aruba or wherever comes with an unwelcome thought: When the plane goes wheels down and the phone is taken out of airplane mode, will your device explode with a series of increasingly desperate texts and emails describing a situation that has gone plaid while you were playing Angry Birds at 30,000 feet?
I know this feeling well, having lived it for years. It's ingrained in my psyche at this point, and I'm fairly sure that's not healthy.
But what's the solution? To maintain multiple people of equal skill levels and experience? Far too costly for most shops, even if it were truly possible to replicate years of in the field, not to mention the knowledge that comes with building the infrastructure. Vendor support? Hardly. Once you get through the first few tiers of responses, you find yourself repeating problem descriptions to folks on the phone who maintain a constant stream of "uh huh" and "I see" vocal pauses to lend the appearance of a trained mind formulating a successful solution -- when they're actually listening for keywords that will let them shift the call to another queue as soon as possible.
Consultants? That's probably your best bet, but to paraphrase a wise saying: A problem has increased by an order of magnitude before the consultant has had a chance to get his pants on. Yes, bringing a boatload of consultants in to address an emerging issue may in fact result in problem resolution, but it'll take far, far longer to fix than if the primary care admin was there to begin with.
Could complete, readily available documentation be the key? Perhaps, but only in a limited fashion. Recurring problems that have been accurately documented with steps to resolve the problem can certainly help when a key admin is out of contact, but once a problem stretches outside the boundaries of a known issue, it all reverts to square one.
We're left with the fact that there exists in every IT organization one or more indispensable people who find themselves trapped by their own expertise. On the other side of that coin is a point I made last week about the Terry Childs case -- people with such a deep level of knowledge of the infrastructure are sometimes considered as a threat to the organization if they leave under any circumstances.
What's the solution? Unfortunately, there isn't one. When computing infrastructure reaches a certain point of complexity, there will always be one wizard who can work his or her magic to solve problems and return things to a stable situation. They're resigned to being the backstop, occuping the desk at which the buck stops, and ideally they're well compensated for such duties. The business just better hope it can reach the wizard by phone when all hell breaks loose.
This story, "Is your IT job indispensable? Then no vacation for you," was originally published at InfoWorld.com. Read more of Paul Venezia's The Deep End blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.