You can deploy monitoring software, diagnostic software, and a Halon fire suppression system, as well as access multiple grids for power and Internet access, but nothing can save you from the most dire threat facing IT pros today: end-users.
Everywhere you look, technology is advancing. Unfortunately for IT, no one has come up with an algorithm to fail-safe systems from stupidity.
[ Also on InfoWorld: For more IT hijinks and absurd assignments, see "True IT confessions," "Jackass IT," and "Dirty duty on the front lines of IT" | Cash in on your IT experiences by sending your war tale from the IT trenches to offtherecord@infoworld.com. If we publish it, we'll keep you anonymous and send you a $50 American Express gift cheque. ]
Thus, we've compiled seven all too familiar tales of user idiocy, told in the first person by those who were left to clean up the mess. For those IT pros who fail to take warning, lost dollars, lost productivity, and additional dental bills from gnashing their teeth await.
Stupid user trick No. 1: Falling prey to fadware
Incident: Shiny objects are lodestones for stupid acts. Witness one IT admin's tale of a fast-tracked tablet rollout that transformed 40 iPads into 3G-enabled paperweights.
I'm not sure whether I should fault the CIO who suggested this or my supervisor for agreeing to it.
The CIO holds quarterly meetings with all the major departments to discuss needs and take the temperature of current solutions -- a smart practice, except he's not all that technical and he never invites anyone from his staff who is. The last time around, the sales team started gushing about the iPad, and wouldn't you know it, one eager beaver brought his along.
The sales team uses a Web-based in-house CRM front end when they're in the field and Outlook 2007 -- no problem for the three-pound ultralight Windows 7 notebooks we've given them. But apparently three pounds is too heavy now, and the iPad is just so much cooler, supposedly helping secure sales with a wow factor customers can't resist. Of course, as soon as the CIO finds out that they're "only" $600, he says what they want him to say: "Why don't we get them for you?" No pilot program, just 40 new iPads for the whole sales department.
Then the fun started. Getting the 40 iPads wasn't a problem, though it took a couple of weeks longer than we thought. But that time gave us an opportunity to investigate our options. What we found: A decent enterprise-class deployment tool that works with an iPad doesn't exist. There's nothing that can deliver a package config. When they arrived, every iPad had to be configured manually, one at a time, and that would be just the beginning.
Naturally, our VPN software didn't work with the iPad, so we had to create an externally accessible SharePoint site, which kind of, almost, sort of works with the iPad's browser. Thankfully, email was easy -- just sync the iPad with OWA -- but you can forget about single sign-on.
The kicker was that the in-house CRM app -- the lifeblood of sales in the field -- failed hard. We wound up having to delay the whole project by a month while the app developer came up with a whole new one-page front end specifically for the iPad -- not a monumental task, but on short turnaround, very expensive. The original cost of $600 per device turned into $1,100 when all was said and done -- and that didn't include the hours we spent hand-configuring all 40 of these things or the tech-support hours we're still spending teaching the sales team how to run presentations off the device.
And the worst thing? More and more I'm seeing iPads left on desks when these guys go out into the field.
Fallout: A significant expenditure in time and dollars based on fadware that didn't add appreciable value to the business.
Moral: Fads are fun, but unless they make direct business sense, stick with what you have.
Stupid user trick No. 2: Reducing password management to a single spreadsheet cell
Incident: Shortchanging security is a surefire setup for epic disaster. But this tale of password brain fail, as told by an IT consultant, takes the cake.
We got called in by a client with a basic setup, one of those folks who didn't call much after our meeting: 25 workstations plus a few laptops, three servers, three printers, and the usual switch/firewall/router fabric -- nothing complicated. For a year, we managed Patch Tuesdays from off site, handled a couple of forgotten passwords the same way, and got called in one other time because nobody knew how to replace a toner cartridge. Then hell broke loose.
The call came from the owner. He was freaking out because not only was every workstation in the place either frozen or spewing a never-ending stream of random websites, but private company information had also found its way onto some industry news blogs. He was losing customers and he was pissed.
We rushed over and got cracking -- shut down the network fabric, started working on individual PCs. Even then it took us a few minutes to see it because we're all working separate machines: Every user had the same domain password. Every single one. We gathered in the owner's office and ask the big "WTF?!?!" This wasn't how we left it.
It turned out they didn't like the $90 we charged (collectively) for the few folks who forgot their passwords. The owner told his assistant to put together a spreadsheet with everyone's password on it, in case anyone else ever forgot. She decided that was too much trouble, so she issued a memo telling everyone to use the same password -- which she would change every 60 days "to maintain security."
We didn't know whether to laugh or cry.
Fallout: Major security breach, server rebuilt from the ground up, and two customers who left and didn't come back.
Moral: There is, in fact, something more stupid than storing every employee's password in the same encrypted spreadsheet.
Stupid user trick No. 3: Routing trouble tickets through your teenager rather than IT
Incident: Executives do the darnedest things -- especially those armed with a screwdriver and help desk advice from their kids, as one IT admin attests.
One evening, we're about to pack up to go home when the fire alarm sounds. We knew it isn't a scheduled drill, so we turned off the lights, grabbed our notebooks, and headed out. In the parking lot, the questions started circulating as to what happened.
We pieced it together: Smoke was coming out of one of the executive's offices, and he had pulled the alarm. The fire department arrived and headed in. The exec talked to the firemen, all the while pointing over at us. When he's done, he steamed over and started berating us for buying "those cheap, piece of s#!t PCs." They were brand-new Dells, so we were confused. But sure enough, the fire department camesout just a few minutes later and said that the PC in this joker's office caught fire -- a little.
They let us back in and we trooped up to the exec's office to check things out. It took us a while, but we noticed that the seal on the back of the PC was broken and some of the case screws were missing. These were new PCs, so we knew none of us have had to service the thing. We gave the exec the stink eye and asked whether he opened the PC. He hemmed and hawed, then got all indignant.
As it happened, his son was a "real computer whiz" and told him to make sure all the "connections were set" or he wouldn't get the best performance. The executive opened the PC and "saw all these wires that weren't attached, just tied off and hanging" and decided to hook them up "to all the right plugs." When he went to turn it on, the PC "couldn't handle a full load" and gave off a bang and a lot of smoke.
Yeah, that was a fun conversation.
Fallout: An angry, embarrassed exec and a bill for a new PC.
Moral: Make sure your users know that the PCs you provide aren't for personal use. They're the property of the business; if they need to be opened, it'll be an IT staffer who does it no matter how smart your kid might be. Or not.
Stupid user trick No. 4: Introducing your IT infrastructure to a flight of stairs
Incident: Lava lamps in the server closet are one thing, but turning your infrastructure into a lounge might require more than just a backup plan, as one IT contractor relays.
Everyone has favorite clients, folks where you never know what you're going to find when you visit. This one time, we get called in on the hot line -- 911, major emergency, everything's down. Two of us got the assignment and went squealing out to the client site through Long Island traffic. We arrived, nodded to the receptionist who knows us, and headed up to the server room, expecting to find the company's IT guy with whom we contract. Only it's a lounge now: sofas, coffee tables, a vending machine, and a big-screen TV on the wall, but no servers. Well, that's the first clue as to why nothing's working.
We nosed around for the IT guy and found him in his office, desperately trying to expedite an order for new servers. Where are the servers we set up, dude? Oh, we moved them downstairs. There was a problem on the way, so that's why we need you to restore the servers. That was vague.
We did some digging and discovered that he asked the two mail guys (the office muscle) to move the server rack downstairs to a new room. Now this wasn't a relay rack, this was a full-on four-post server rack. And because Mr. IT wasn't sure he'd be able to hook it all up again, he told them to leave everything in the rack.
I'm not sure how these guys even got it to move across a flat floor. That thing had three servers, two switches, a router, a disk array, a tape drive, and a UPS installed. It must have weighed a ton. The two geniuses apparently decided that the freight elevator was too far, so they tried to move it down a flight of stairs "just one step at a time." Yeah, it fell on step two, one of the guys came close to getting killed and most of the stuff in the rack wasn't working anymore when it stopped its tumbling routine on the first floor.
Apparently, APC doesn't guarantee equipment in its racks if you drop it down a flight of stairs.
But we were prepped. We were almost grinning, because we were about to be heroes. We told the IT guy that we have virtual images of his servers, that we had their configs registered with a local outfit that will rent us replacement infrastructure until he gets the new stuff on order, so all we need are the backup tapes and we can have him up and running in about a day, maybe less.
Boy, that would have been nice. But we also learned that Mr. IT had gotten tired of going to the second floor to replace backup tapes. After all, that disk array was doing just fine as a backup. So the last tape they had was from four months before the four-post header.
Fallout: Not only did Mr. IT get fired, but the IT team lost the contract -- unfair.
Moral: Do your daily backups, and don't treat your IT infrastructure like a fridge.
Stupid user trick No. 5: Letting mom monkey around with the admin console
Incident: One IT consultant tells tale of yet another hard-learned lesson in proper password management brought to you by that time-honored IT pro, mom.
A small-business client had us install a Small Business Server box for her. She had about 12 people working for her, including her mom, who was doubling as the office manager and her daughter's personal assistant.
We did as we were asked. Everything was set up, tested, and found to be working. We established an admin account on the server and left it with the owner with strict instructions that it's for emergencies when she's on the phone with us only. She, of course, gave the admin account info to her mom to keep someplace safe without passing on the last part of the instructions.
Her mother went exploring and found this thing called Active Directory. Next thing we know, we're getting an angry call from the daughter because our email server was sending strange emails to all her clients and friends. The story: Her mom had figured out how to get into Computers and Users and had been adding everyone in her daughter's address book into AD, along with generating them an internal email address in addition to the one listed in her daughter's rolodex. The system sent everyone a welcome email with an introduction to the "new" network they'd just joined.
Fallout: Apology emails around, consultant fees to delete all those users and set AD right, and palpable tension between daughter and mom.
Moral: Server passwords aren't status symbols. If a person doesn't need one, don't share it.
Stupid user trick No. 6: Paying before planning
Incident: Hubris is no stranger to the world of IT. But when a trumped-up higher-up puts the purchase before the plan, the fallout can mean only one thing -- a derailed career, as one developer recounts.
I worked for an Internet startup back in the late '90s, complete with big-time VC funding and a small DNA kernel of three business whizzes and one techno geek who gleefully grabbed the CTO title.
The startup's goal was to create a Java-based vertical accounting system followed by inventory and sales systems that would eventually comprise a "suite" of offerings. The three kernel guys land a huge bundle of first-round financing and sit down with two "experts" from the vertical to discuss what the initial application should look like and how it should run.