Sure enough, one night around 10 p.m. everything at the ISP stopped working: DNS, e-mail, the company's own Web sites, and the sites it hosted for its business customers -- all simply went poof.
The problem? The ISP had neglected to renew its own domains. The person in business support assumed Fred was also getting notified about the renewals (he wasn't) and Fred assumed that since he wasn't being notified, everything was hunky dory (it wasn't).
"By the time we diagnosed the problem -- because you rarely think to check whether your own domain has expired -- we had fallen out of the root servers and it took a full 24 hours before everything was restored," says Fred. He adds a variant on the old MasterCard commercials:
1) Domain renewal: $9.99
2) Late-night tech support call: $0.00
3) Breaking Internet connectivity and e-mail for 35,000 people in your hometown: Priceless.
Lessons learned? 1. Always have multiple people receiving import alerts. 2. Register your domains for 10 years and it will most likely be the next guy's problem.
True IT confession No. 7: Don't ask, don't tell, and don't let them make you take a polygraph
Four years ago "Paul" (not his real name), an independent data analyst in (yes) the Midwest, was working with a governmental client on a $20,000 analysis project. After two months of hard work, he delivered a preliminary draft to the client, then went off on a week-long business trip.
Before he left, Paul burned a disc with all the project data on it so that he could finish it up in the hotel during his trip. And as was his usual custom at the time, he deleted all 4GB of project data from his hard drive to free up space.