Thank goodness for backups, because it saved our bacon. We reloaded the OS and Exchange from scratch, then restored the store. We were back in business. Fortunately, e-mail was not the ultrahigh priority it is currently, as it took us about two days to get everything back. Part of the slowness was the multiple reboots to patch NT and its special HAL to run on the Alpha. And the 8mm tapes weren't very fast, either.
Rick says the morals of the story were: a) you don't know what you don't know and b) backup, backup, backup!
Rick wasn't the only one with a chilling story. Moe Hoskins, a regional network manager, told a harrowing tale of loss:
When the telephone rings with a report of an error opening Outlook and its inability to locate the Exchange server, there is reason for concern. After attempts to remotely manage and ping the Exchange server fail, your concern level heightens. Upon contacting the technician on site, you learn he had discovered the server in an unresponsive state and forcibly cycled power. What you didn't know at the time was during the unattended reboot, the Windows 2003 Exchange server, running RAID 5 with a hot spare, was missing a hotfix(831374), which allowed chkdsk autochk to run and corrupt the Exchange database. The corruption soon became evident after your Exchange Admin spent several hours of failed attempts to recover the database.
You realize you are now facing significant downtime and while not optimal, there is no need for panic. Your nightly backup regime will finally pay off and you can simply roll back to the previous night's backup with a minimal loss of mail. But the battle has yet to begin. After inserting numerous backup tapes, you discover the data set is not complete and you cannot find a restorable database. You frantically begin examining the backup logs.
The harsh reality sets in when you discover the backup job was spanning two tapes. The morning ritual of inserting a new tape for the scheduled nightly backup without first reviewing the backup completion logs was allowing the spanning to go unnoticed. The second half of the backup was subsequently overwritten by the first half of the following backup. In a nutshell, this destructive cycle resulted in a two-week rotation of media that did not contain a restorable Exchange backup. You desperately move on to the monthly tapes and finally discover the two tape spanning began four months ago. As your heart sinks, you are now faced with more than 200 users losing their last four months of messaging information.