Then we discovered the emails were being generated at the rate of several thousand a minute, and with a fat pipe between the two sites, they were able to ramp up traffic very quickly. It soon overran even our biggest servers and caused all mail to queue up.
We finally got in contact with a local Company X person, who pulled the plug on the monitoring server. It took another hour to clean out the mail queues and stop the looping of what mail remained in the queue. In all, tens of thousands of emails finally were deleted, and another few thousand pieces of valid email were delivered.
[ Get a $50 American Express gift cheque if we publish your tech experiences. Send your story of a lesson learned, of dealing with frustrating coworkers or end-users, or a story that illustrates a relevant takeaway to today's IT profession to email@example.com. ]
Next, we turned to the remote site connectivity problems. We easily determined the Company X remote site's local DNS server had crashed and was not able to resolve names. It was quickly fixed with a reboot. It turned out the backup server at Company X's HQ had also gone down.
At first, it all seemed an odd coincidence, until our boss mentioned that our company's IT VPs were at Company X earlier in the day. Upon their return late that afternoon, they admitted to turning off "unused" servers that had no impact on production systems.
It all started to make sense. The VPs turned off servers, the monitoring server generated massive amounts of alarm traffic to non-existent people, and one of the servers was the secondary DNS for remote sites.
Our company's tech guys on the ground had been required to submit change notices, complete with VP approvals, for years. Without one, we couldn't change an IP address, power off a server, or anything similar. But in this instance, the VPs did not follow their own protocol. Apparently, for mahogany row it was a case of "do as I say and not as I do."
If there is a moral to this story, it's this: Do not to let a VP decide when and where to retire "unused" equipment without at least informing the folks who have to fix their messes.
This story, "VPs, don't touch the servers!," was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.