Hack Tales: E-mail archiving the home-brew way

Custom code allowed one company to sidestep costly management solutions

Compliance is a painful reality for many IT administrators. Among the growing list of tasks, e-mail archiving is becoming a major requirement. Not only are various government audits interested in e-mail archiving data, but legal actions have begun to call on this information with increasing frequency. The demand for e-mail archive retrieval has become so great, in fact, that corporate e-mail hosting services -- Microsoft’s new Exchange Hosted Services, for example -- have begun adding high-availability archiving to their service menus.

The problem is that efficient e-mail archiving is often expensive. These hosted services can cost more than $20 per user, per month. Spread out across 2,000, 5,000, or 10,000 users, that can really start to add up.

This is the situation that Ray Martin, IS technical manager at Monrovia Nursery, found himself in not too long ago. Monrovia’s HR department had decreed that e-mail and IM chat data had to be archived for potential future legal wrangles. Pricing out commercial archiving solutions for his 2,000-user network left Martin looking at forking out approximately $30,000; that, in turn, made his manager look purple. So Martin and an in-house programmer, Brian Scott, hacked together an alternative solution.

First, they activated the built-in archiving features on their Lotus Domino and Sametime messaging servers to create semitransitory mail log files. Converting these to permanent archival records, however, was the real trick. They needed a free target server with smarts.

Fortunately, Martin already had a Linux server running Nagios and MRTG (Multi-Router Traffic Grapher) for network monitoring requirements. Because these tasks were fairly light, Martin said he thought this server was a perfect place to host an archiving solution based on homemade code. All he had to do was add a DVD-RW drive to the Linux box and give Scott the nod.

For his part, Scott took two weeks to write a small C program that would FTP the Domino and Sametime mail logs to the Linux server while monitoring the size of the captured log files on a cumulative basis. When that starts amounting to what can be easily fit onto a DVD, the program kicks off a DVD burn job on the Linux box. If the burn is successful, the program removes the local copies of the logs from the Linux box, as well as the working copies on the production Domino and Sametime machines, to conserve much-needed disk space. Finally, it sends an e-mail notification to the systems administrator reporting on the status of the burn, whether the job needs to be redone or it’s safe to drop a new blank disc into the drive.

Martin says he’s not too worried about the possibility of switching to a new e-mail server platform (in the event that ever happens), as long as that new server can be forced to copy all incoming mail to a separate archive file. He says his enterprise hack saved his company a few tens of thousands of buckaroos, while still giving the legal team the e-mail nutrition it craved.