InfoWorld Home / Adventures in IT / Off The Record / Oh, were you using that server?
July 07, 2009

Oh, were you using that server?

In this IT tale, the boss gives away an "extra" server that's part of the fault-tolerance system

| Print | 12 comments|
56 Recommendations

From 2001 to 2003, I worked as the local IT admin and jack-of-all-trades for a branch of a collection services company in California. It was fun in that I got to do a lot of stuff that a larger organization would have compartmentalized, but it was also difficult in that I had no one to back me up when I needed to make an argument. The boss happened to not only behave like the "pointy-haired boss" in Dilbert, but he actually had pointy hair -- a coincidence that often manifested itself when he showed both his lack of IT knowledge and his poor management decisions.

When I first took the job, the network consisted of a hodgepodge of equipment run on cat3 and coax. By "hodgepodge," I mean we had ancient reel-to-reel tape drives and an old mainframe running data "platters" that I had never heard of until then, and the user environment was old orange-screen terminals.

[ Want to cash in on your IT experiences? Send your story to offtherecord@infoworld.com. If we publish it, we'll send you a $50 American Express gift card. | Show off your geek quotient: Get your InfoWorld shirts, cups, and more schwag. ]

When the day came to upgrade to a new network, I got tasked with the project and spent a couple of weekends pulling Cat5 through suspended ceilings and conduits, and I got to install a textbook server to replace the old mainframe. I did much of the grunt work myself and planned very carefully, and I was thrilled to see I had quite a bit of money left over in my budget. I went through the appropriate channels and bought an extra server to use as a mirrored server for fault tolerance.

In the course of a few months, the mirrored server setup prevented costly interruptions to service several times -- and I was pleased with myself for my foresight.

Then I went on vacation.

While I was on vacation, the IT guys from the main office flew out to take my "extra" server to their office to reimage and use as an additional Exchange server. My manager accepted their explanation that I was wasting company resources with this "extra server" and let them abscond with a very much needed piece of equipment.

That Saturday my pager went off. I called in to find out that the network was down and 400 collection agents were idling away their time because of the outage. I apologized to my wife and drive the 450 miles back to the office to find out that my backup server was gone. After an hour, the lone server was back up and running from a backup tape that was several days old. My boss demanded to know why the server failed -- and I demanded to know where was my backup server.

Related Content...

White Paper

D2D Virtual Tape Library Replication Primer

This whitepaper explains the terminology and concepts behind Data Replication technologies and establishes some sizing rules through worked examples. Learn the new paradigm in disaster tolerance—protect data anywhere.

Download now »

White Paper

An Alternative to Virtualization for Datacenter Cost Savings

Server virtualization is a popular option for dealing with mounting datacenter costs. Another equally promising approach is the use of an Application Delivery Controller. Citrix NetScaler provides a low-cost way for organizations to reduce their server count and accrue cost savings from a reduction in space, cooling, power and personnel.

Download now »

White Paper

Why Your Firewall, VPN, and IEEE 802.11i Aren't Enough to Protect Your Network

The emergence of WLANs has created a new breed of security threats to enterprise networks.

Included in HP ProCurve WLAN solutions is security technology that alleviates threats from WLANs through:
* Monitoring wireless activity inside and out of the enterprise
* Classifying WLAN transmissions into harmful and harmless
* Preventing transmissions that pose a security threat to the enterprise network
* Locating participating devices for physical remediation

Download now »

White Paper

Bringing the Edge to the Data Center

Effectively address data protection challenges, implementing solutions that help store and protect business–critical data while cutting costs and improving efficiency and reliability.

Download now »
11 of 12 comments
Sign In or Register to comment
zanzan42 7-Jul-09 9:12am
1 reply
It's the classic issue of management's view of IT: When things don't work, you get yelled at for not doing your job, and when things are working, they wonder what they're paying you for. On the subject of going out of business, I was lucky enough that my very first real job was for a company that eventually went out of business a year or two later. In hindsight, it was valuable experience which allowed me to subsequently recognize management stupidity early and often.
JTB2468 9-Jul-09 7:35am
7-Jul-09 10:12am It's the classic issue of management's view of IT: When things don't work, you get yelled at for not doing your job, and when things are working, they wonder what they're paying you for.

+1

The main reason I want to switch to a job working for a strictly IT company (setting up servers, networks and whatnot for other companies) is because of dealing with non-IT managers. It drives me insane! I'm the only IT person here, so I'll come up with an idea and the bureaucrats will use some odd form of logic to conclude some off the wall ridiculous reason that I'm doing it wrong, even though they have a hard time performing simple end user tasks.
Accounting IT Guy 7-Jul-09 9:50am
1 reply
Who the heck just goes and takes a production server offline and reimages it without even consulting the local admin who maintains the system??
JTB2468 9-Jul-09 7:29am
Sounds exactly like something my boss would do.
BigRonG 7-Jul-09 5:58pm
1 reply
It's amazing the number of incompetant goofs that are considered managers in this society. Since about 10-12 years of my 30 plus year career were spent as a consultant, I got to see it quite often. Several companies that I worked for went belly-up. It was hard to tell at times whether it was the previous screw-ups or the current ones that killed the company. Some highlights - throwing away a bleeding edge software (homegrown) suite because the new manager had never worked in that language; trying to re-write the application serving 200 plus retail stores in a new language (Visual Basic) that was not robust enough at that time (lacked critical functions) to handle the job because the new CIO had an in with a firm in the town he came from; trying to rewrite the base application serving a car financing company when the commercial app that had extensive mods worked fine spending hundreds of thousands of dollars that the company desperately needed to stay alive (it died). Only a few of many - we would be better served to create a certification program for managers and require the letters after the name.
gunner@gulftel.com 8-Jul-09 6:49am
I would argue that certifications are certainly no true indication of ability - or knowledge. I actually had customer come in once who was looking for a new video card. When I asked her why, she said her printer driver told her she had to run at a higher resolution and her card couldn't do higher than 8-bit color. I eventually discovered from her that she'd recently reloaded XP. Obviously she hadn't loaded any of the drivers yet. I told her to download and install her drivers and then was forced to explain where to get them, how to install and verify them, etc. Her next sentence was "are you hiring? I have my Microsoft Certification and everything." I deserve an Oscar for not laughing out loud in her face. All I could think of was "certified for WHAT?!" We had hired an A+ tech at a prior company and he's the first person I've EVER seen get fired on the same day they're hired. After spending all day trying to put parts together to build a system he determined the motherboard was bad because it wouldn't boot. We told him to RMA it to the vendor and get another. Luckily another tech decided to check it out, because in the box was still the P4 CPU mounted to the board (a $300 processor at the time). This same tech successfully booted the system and we discovered the "new hiree" had been using a bad boot floppy - but never tested it. Oh, and for more icing on the cake: I spent 12 years in the military and I can confirm that college education does NOT equate to better management... The best leaders I ever saw never had any college...
willicueva 8-Jul-09 3:16am
What is most important in this story seems to be neglected, namely that the IT Guys from the Head Office were the ones that took the server, without checking to see what it was there for. What does that say about the HQ "IT Guys"???
Accounting IT Guy 8-Jul-09 9:02am
It says that they were arrogant and ignorant. They think their needs outweigh the needs of a smaller office and that they dont have to follow procedure. Which will always = FAIL.
jgaskell 8-Jul-09 11:24pm
While the HQ IT guys and the PHB obviously did the wrong thing, I'd be interested to know why the main server fell over so often that a second redundant server was required. Maybe some of that money left over in the budget would have been better spent buying a more reliable main server.
glennho 9-Jul-09 11:56am
1 reply
An IT engineer once told me, "I have no server level control over stupid." There is a huge difference between knowledge and good judgment, and even an overabundance of the former will not compensate for a deficit in the latter. If tests and certificates were guarantees of good judgment, auto accidents would involve only unlicensed drivers and "acts of God". You can't get around somebody else's poor judgment - that's why cars have airbags; however, if we carefully share information up-stream, we Can improve the knowledgebase of mgmt, and hopefully gain some respect and job security (if such a thing exists) as well. Smooth-running IT cannot be invisible and mysterious. Especially when you're THE IT Guy, doing it all, you gotta advertise and self-promote. When you need to spend, don't just say, "We need this," but say, "This is the situation; A - we can do nothing, and risk X; B - we can spend this, and risk Y; C - we can spend This, and be covered. I recommend C." Now mgmt knows not only the need, but the purpose, the risks, the fact that You Know Things that they never think about, AND that you think things through instead of shooting from the hip. Even if they don't let you have your first choice (after you made it So Obvious), your cred and your visibility have improved, and you can righteously say, "I told you so, neener neener neener," when it's time to put your stuff in a box and collect your final paycheck. But before that happens, get Visible and STAY Visible -- let them know along the way about bullets dodged ("Server A crashed today, but B picked right up, with no down time - I'm sure glad we were prepared for that!") Then, down the road, when they're looking for ways to cut costs, they're more likely to acknowledge your value -- they may keep you in your seat instead of hiring the boss's teen-aged kid to come in on weekends to do the server stuff. It takes a careful balance of brevity and clarity, with just enough technobabble to give it weight, and must always demonstrate your Deep, Deep Committment To Doing What's Best For The Organization. This practice may not have averted the situation that started this discussion - it sounds like Bad Judgment was a criteria for promotion at that place. I am convinced, however, that if IT tries to be invisible and mysterious, sooner or later, the only mystery will be "Hey, what happened to that guy that used to do the server stuff?"
zanzan42 11-Jul-09 8:59pm
While you are completely correct, your story didn't include the ending, which goes something like this: Even after telling management about options A, B, and C, and the costs and consequences of each, they chose A because it was cheapest. Then when downtime inevitably happened, you still got held responsible for the failure. You never get to say "I told you so" because management doesn't care. If they did, they would have chosen C. You are screwed no matter what you do, even if you document everything. Do you really think that the management level is going to tell their management that *they* screwed up by not following your recommendations? Never happens.

Sign up to receive InfoWorld Resource Alerts

Subscribe to the Off the Record Newsletter

The one-stop resource center for IT professionals.

©1994-2009 Infoworld, Inc.