No matter how hard we pray, how many chickens we sacrifice, how often we chant naked by moonlight, every network is at one time or other exposed to the ultimate technology risk: users.
They’re short, tall, skinny, and fat. They’re smart or stupid, unique or cloned -- but no matter what, they’ll abuse technology.
In deference to my years of dealing with this most dangerous species of wildlife, the editors at InfoWorld asked me to record some of my most memorable experiences along with tips on how to avoid similar incidents. Being both thorough and lazy, I decided to open the floor to our adventurous readers as well, who have been kind enough to relate their tales of sorrow and solution.
The result is a list of problem categories each with a specific situation and solution. Broad advice applicable to all IT adventurers can be found in the moral bringing up the rear. With luck, this salutary information will help keep your rear covered.
Automatic updates
BrilliantCompany.com was growing at dot-com bubble rates. With departments popping up like daisies in spring, the IT staff was ceding desktop control to department heads because most everyone was technical anyway.
Shortly after a batch of 75 new Dell desktops arrived to populate a new product division, the network suddenly died in the middle of the day. All lights were green in infrastructure land, but performance had slowed to such a crawl that the LAN was effectively paralyzed. Some diligent sniffing and log file snooping revealed the culprit.
Turns out Windows XP’s Automatic Update had defaulted to high noon on a weekday, and all 75 machines attempted to download several hundred megs of Service Pack 2 simultaneously and individually. Instant network clog.
Solution: Centralize IT control so one somebody can be responsible for all the details. This was done in short order after I released a sprightly memo to the appropriate folks. Then, I did what I should have done earlier and set up SUS (Software Update Services), now WSUS (Windows Server Update Services), to download updates and distribute at an appropriate time and after appropriate testing against departmental OS images.
Moral: Just because your users are technical doesn’t mean they’ll behave with any more attention to detail than the average Joe. If network uptime is your responsibility, then take responsibility and manage what needs managing.
Client protection
InfoWorld reader SEnright relates a tearful tale: A mobile user called to say that his laptop was no longer functioning. After a lengthy phone conversation, during which the user initially denied anything unusual had happened, he disclosed that he had spilled an entire can of Coke on the keyboard. “He continued by telling me that he had tried to dry it with a hair dryer, but that it still would not boot. I asked him to send it back to me, and that I would have it repaired.”
But when SEnright opened the laptop’s shipping box the very next day, he had a bit of a shock. “The gentleman had not used a ‘hair dryer,’ but must have borrowed a heat gun at one of our locations, because all that was left of the keyboard was a cooled pool of molten black plastic.” Ouch.
Solution: The laptop was insured for “accidental” damage only. Since the incident, maintaining full coverage of mobile equipment has been a matter of course for SEnright.
Moral: Cover your mobile warriors. That means not only insuring their hardware, but giving them training and clear policy documents on what can and can’t be done with company hardware on the road. Further, make sure their data is backed up religiously, both when they’re at the home office and when they’re on the road.
Executive clout
Here, we’re concerned with that senior executive who just has to have full administrative rights to every machine on the network. Even though he’s about as technical as my cat--and my cat is dead.
Senior users can be dangers even without special access rights. John Schoonover, who worked for the Department of Defense on one of the largest network deployments in history during Operation Enduring Freedom was “witness to a huge lack of IQ points” in a senior manager.
According to Schoonover, military infosec installations generally follow a concept termed “the separation of red and black.” Red is simply data that has not been encrypted yet. (Danger, the world and sniffers can see you!) Black is the same data after it has been encrypted and is now ready to traverse the world. “These areas [red and black] are required to be separated by a six foot physical gap,” Schoonover says.
Our hero proceeds to follow these guidelines and deploys the network, but comes back from lunch one day to find the firewall down. Investigation shows that a senior manager “had taken the cabling from the inside router and connected to the Internet for connectivity, thus bypassing all firewall services, encryption, and -- oh yeah, that’s right -- the entire secure network with a jump straight to the Internet!”
Solution: John says they “removed the culprit’s thumbs, because if you can’t grip the cable, you can’t unplug it.” I didn’t ask for any more details.
Moral: Managing rogue senior users is an art in itself that requires diplomacy and even outright deception. In several installations I’ve renamed the Administration account something like “IT” and made “Administrator” a functionally limited account with simply more read/write access to data directories, while still blocking access to things like the Windows system directory or Unix root directories. Most times they never notice; and if they do, I’m pretty good at making up excuses why those directories remain closed off. (“Oh, that’s something Microsoft did in the last service pack. Gosh darn that Bill Gates.”)
Legal eagles hunting IT mice
Lawyers ruin everything -- including smoothly running networks. But IT managers who ignore the ever-changing legal landscape’s impact on technology do so at their peril.
I was once called in as referee among in-house counsel, senior management, and IT staff after the company was informed that child pornography had been tracked to its servers. The company didn’t know whether to aid the investigation by figuring out which employee was responsible or to just delete all the offending files immediately and most likely incur a fine but protect the firm from getting shut down.
In the end, the lawyers managed to make a deal with investigators. The company’s IT network stayed active and we tracked the lowlife down and had him arrested. Quietly.
Solution: Talk to senior management and corporate counsel about legal issues, such as corporate response to third-party audits or company responsibility for data it’s holding concerning third-parties, before they happen.
This discussion goes beyond IT-centric solutions. Management must decide whether it wants to retain all pertinent data (the best course of action for those third-party audits) or automatically delete offending data (such as whatever’s found in porn filters).
IT and management must see eye to eye on how the company will respond to law enforcement inquiries, investigations, or even raids. If Homeland Security agents believe a terrorist is masquerading as an employee and storing data on corporate servers, they can come in and pretty much take anything they want. That could put a real crimp in the style of, say, an e-business.
Developing the best course of action should involve senior management, corporate counsel, and law enforcement. The FBI is usually pretty helpful in these discussions -- and so, sometimes, is the local computer crimes department, such as the large Computer Investigation and Technology Unit division of the NYPD.
Moral: The higher you are on the IT food chain, the more such liability can spell serious trouble. If you make sure to discuss at least general legal eventualities with senior management, you’re much more likely to do yourself and your employer some real service in specific situations. If they refuse to discuss the matter, archive everything you can.
Disasters in disaster recovery
Gary Crispens reports an incident he encountered after questioning an IT director about the company’s preparedness for disaster recovery. The director responded huffily that the hot site was ready for any disaster, including the necessary space and equipment all backed by a diesel-powered generator with “plenty of fuel.”
After about a year, the company had a hurricane-related power outage that forced it to roll over to the hot site. “Sure enough, the IT Director had critical functions up and running and I could hear that generator running out back. But after about eight hours the power went out for good and all systems crashed when the generator stopped.”
It turned out that “plenty of fuel” was one 55 gallon barrel that was already half empty from the monthly testing.
Solution: A disaster recovery plan that called for fuel checks in addition to generator testing.
Moral: Disaster recovery isn’t a static issue. One plan or one policy is never perfect out of the gate. Ever. Pass such concepts by as many experienced eyes as you can and then revisit them annually or even bi-annually for refinement.
Rogue peripherals
CompUSA and the Dummies books are teaching users just enough of the tech alphabet to spell trouble.
One of my favorite stories was the network that was severely hacked by someone who came in from the outside and deleted the main Exchange message store. Firewall logs had gotten the local IT admin nowhere, so we were called in to do a little snooping around. I wish I’d thought of it, but another guy on the team had the sense to run AirSnort. He found a wide open Linksys wireless access point in about six seconds.
The internal admin insisted there was no wireless running anywhere on the network. It took some sneaker netting, but we found the rogue AP in a senior exec’s office about 20 minutes later. Seemed he saw how cheap they were at the local CompUSA and decided to plug one into the secondary network port in his office so he could use his notebook’s wireless instead of the wired connection because no wires “looks better.”
Another problem in this vein is USB. Being able to plug in a peripheral and achieve working status without the need to install drivers has rapidly spread the popularity of personal peripherals. You don’t want to get yourself get sucked into supporting things such as printers that aren’t on your official purchase list -- or external hard disks, DVD drives, sound systems, and even monitors.
Nor do you want the security risk of an employee plugging in a gig or two of empty space into any workstation’s USB port and copying important corporate information. Source code, accounting data, and historical records all can be copied quickly and then walk out in somebody’s hip pocket.
Solution: Let employees know what is and isn’t acceptable as corporate peripherals. Keep an accurate asset record of what belongs to the IT department so you can more easily find or ignore the stuff that doesn’t. And if data theft is a problem, think about protecting yourself by disabling USB drives, uninstalling CD-RW drives, or similar measures. The work you do now can save your bacon later.
Moral: Asset management isn’t just for the anal. Knowing exactly what’s supposed to be on your network is a key step to solving a wide variety of IT mysteries.