There are many cases where zombies can become big problems. One I discovered during a particularly interesting troubleshooting session was a cronjob that caused a server to assign a secondary IP address to an interface, conduct business with that source IP, then remove the secondary. This was ostensibly written for security purposes, as the script used a mapping through the firewall to the secondary address. As such, when its job was done, removing the secondary took out a valid target IP, preventing traffic coming through the firewall from reaching an actual server.
Naturally, even if that cronjob was documented, nobody would have pored through the documentation to make sure the secondary IP used by this server for 30 minutes a day was actually engaged. Thus, after the framework requiring this component was disassembled, the script had no more purpose, but continued to assign that IP address every night. Of course, that IP address was later assigned to a production server, causing intermittent outages that couldn't easily be explained -- at least, not until we wrote a small script to capture the MAC address of that IP throughout the day. Then we were able to identify the server that was magically assuming the IP address at certain intervals.
Zombie scripts and procedures are part of life in IT, no matter how much we try to minimize them. They'll pop up from time to time, surfacing when a partition on a disk mysteriously fills to the brim with logging output or, alternately, with files created hourly or daily that had no reaper process to constrain them. They'll cause spurious network traffic within or without a network segment, causing blips on monitoring graphs that can't easily be explained. If they were poorly written (not unusual at all), they will have little or no error checking and cause huge problems when a server is upgraded or when the behavior of binaries changes enough for them to cause damage, such as when using now-deprecated commands to perform tasks that can choke a server.
But IT zombies are in many ways the opposite of "real" zombies. Those zombies are easy to find and relatively hard to dispatch (well, depending on the zombie movie you're watching). IT zombies are generally hard to find, but relatively easy to disable. The fact of the matter is that the only protection we have against IT zombies is to remember that they exist, lurking in cronjobs and scheduled tasks anywhere and everywhere, ready to cause mayhem when just the right set of bits is flipped.
When an IT problem quickly turns from inexplicable toward impossible, it behooves the erstwhile troubleshooter to remember that these creatures exist, and it might be time to grab a crossbow and go on a zombie hunt.
This story, "Zombie scripts can attack at any time," was originally published at InfoWorld.com. Read more of Paul Venezia's The Deep End blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.