Spelling counts: The case of the stuffed server

A tech pro jumps in to stop the madness when a company's mail servers come to a screeching halt

Spelling counts: The case of the stuffed server
Credit: Thinkstock

It seems simple, but take it from me: When configuring software, it's a good idea to double-check and test any and all settings -- no matter if it's on a server or a client computer. Plus, when users install and configure software themselves, it's best to provide a concise guideline that includes check routines.

I found out the hard way once when I worked as single-person supplier and supporter for both hardware and software. One of my clients was a midsize family business. It had two offices in different cities, both equipped with local network servers that included mail servers.

One day, they called me in high despair. All of a sudden both mail servers had stopped functioning normally. Further inquiry into the overall operating system showed that they were still online and running, though they took plenty of time to react to commands. But during the brief moments they did reply, they did so with their usual speed as if everything was fine.

I attempted to log in remotely for maintenance, only to experience the same lag. Unable to do any meaningful troubleshooting this way, I decided to drive to the nearest of the company's offices to log into the server directly.

On-site analysis

On the way there, I racked my brain for an explanation. The afternoon before, all computers had received email client updates and each user then had to re-enter all of the configurations manually. At that moment, I didn't see a connection between the two events.

I arrived and logged in, and the server cheerfully replied it was in a fine state and everything was working well, and it was gladly munching on all the work it had to do. I checked the incoming and outgoing mail queues and discovered they were chock-full of messages coming from and going to the same mailbox that resided on the other office's mail server -- and all of the messages were gargantuan.

A brief analysis of the messages' contents revealed what had happened. An employee in Office A had emailed somebody in Office B. The recipient was out and had set up an automatic reply that got emailed to the senders' reply-to address. But the sender had made a typo while entering his reply-to address into the configuration -- there was no mailbox by that name.

The mail server in Office A dutifully replied with an error message, stating there was no such mailbox -- oh, and here's a copy of the message you had sent. This was transmitted to the initial recipient's reply-to address.

But the initial recipient, too, had made a typo while entering his reply-to address into the configuration. Again, the mail server in Office B dutifully replied with an error message, stating there was no such mailbox -- oh, and here's a copy of the message you had sent, by that point a copy of the copy.

The problem snowballs

This resulted in both mail servers engaging in a ping pong-like exchange of error messages with increasingly large attachments of copies of error messages.

By the time I arrived on scene, the attachment had already grown in size to several gigabytes, and copying it into a new message kept the mail servers fully occupied for more than a quarter of an hour. Only after was it available for a few minutes of calm to serve other requests before the ping to the pong (the error message reply from the other server) came crashing in and occupied it for the next quarter-hour.

I deleted the outgoing queue on both mail servers. One single command and the problem was solved, the day was saved, and everything was back to normal. Then I spent the remainder of the day checking with everyone to ensure their mail software was configured correctly.

From CIO: 8 Free Online Courses to Grow Your Tech Skills