My unnatural disaster

Destruction is bad enough but when it's followed by incompetence, heaven help the victims In my world, Katrina wasn't the only disaster of 2005. I was a Linux administrator at a university that year. I use the past tense because shortly after the disaster I'm about to tell you about, I was unceremoniously relieved of my duties. I was completely at fault -- not for the disaster but for misreading and misplaying t

Destruction is bad enough but when it's followed by incompetence, heaven help the victims

In my world, Katrina wasn't the only disaster of 2005.

I was a Linux administrator at a university that year. I use the past tense because shortly after the disaster I'm about to tell you about, I was unceremoniously relieved of my duties. I was completely at fault -- not for the disaster but for misreading and misplaying the situation that ensued. I didn't know how to pick my battles and I didn't know how to keep my mouth shut.

In July 2005, my own personal Katrina happened: all the critical Windows servers were compromised by IRC bots. How did I know this? Shortly before the malicious hackers took them down, I noticed on one of my Linux servers that the iptables logs showed at least one of the Windows servers doing port scans. (I'd seen such traffic at my previous job, where I'd spent some time monitoring the wild, wild west of academic networks: dorm traffic.)

The Windows 2003 servers were left open to the Internet without any type of firewall protection. As a result, DNS, DHCP, Exchange, File & Print, and AD Domain Controller all went down.

Five days later, none of the critical servers had been restored. (You might liken this to the kind of response FEMA delivered in the wake of Katrina, should you wish to carry my analogy to its next logical level.) The Windows admin unilaterally decreed that all 500-plus computers in the department be manually reformatted to verify they were clean -- not imaged or automated through some unattended install process, but manually reformatted. I guess the Windows admin figured nobody would complain about the servers being down if the clients were down as well. Better yet, the grad students would be doing the dirty work of destroying their own workstations. Some of these computers had specialized instruments connected to them with obscure drivers. Ultimately, the reformatting orgy caused more destruction and downtime than the initial compromise.

Who needs malicious hackers when you have admins like this, eh?

The department I worked for was left with very limited functionality for July and August. During the disaster, I was not allowed to exercise my expertise with DHCP and DNS. The Windows admin didn't seem to realize that if the DNS server wasn't brought back up ASAP, the department was going to drop off the Internet.

In the end, I feel I was judged on my inability to support Windows, which I never claimed I could do. The real purpose of my job, then, seems to have been to prop up the Windows admin. If my replacement can follow bad and unethical decisions, he or she should fare well.

Just as the hurricane victims all faced some harsh realities after Katrina, so too did I come to grips a few IT-related ones. If you think campus IT hasn't put your network behind a firewall because your computers are "well-behaved," you may very well be an incompetent Windows admin that missed the memo to contact campus IT to schedule a firewall deployment.

From CIO: 8 Free Online Courses to Grow Your Tech Skills
Join the discussion
Be the first to comment on this article. Our Commenting Policies