Think of scheduled downtime as disaster recovery on your terms

Disasters should, by definition, never be scheduled -- unless, of course, you're maintaining a data center

RELATED TOPICS
Page 2 of 2

But you need an emergency plan in the event that things go truly south and the UPS can't be stretched further. This usually means a rapid, orderly power-down of the data center. This dance should be scripted as much as possible so that you can act extremely quickly (and accurately) if the situation arises. Those last running servers and VMs need to be shut down ASAP, and in the right order. The storage needs to be next in line, but every check and guarantee must be made that nothing is attached to it when it bails. Then, you'd better be sure that all of your switch configurations have been saved and backed up; although they're going down last, they're still going down.

Of course, you need to make sure there's proper ventilation. When the air units shut off, you can expect to have hot gear that'll bake for a while.

When the power returns, usually without warning, all you can do is wait to see what havoc may ensue, then begin triage. Reverse the power-down order, and hope all the storage lights come back up. Then bring everything else back online, encountering more chicken-and-egg situations than you thought possible. If you're lucky, that four-hour power outage will only take 12 hours to repair, and the damage will not be permanent.

Most people don't really think about why data centers need to run 24/7/365. Even when they're largely dormant, actually powering down these otherwise critical systems only leads to mayhem. This is one of the major benefits that virtualization has brought us -- the fact that we can power down servers in times of low utilization, but we never, ever go all the way down if we can avoid it. When we can't, we take a deep breath and head into the breach, hoping for the best, yet planning for the worst.

This story, "Think of scheduled downtime as disaster recovery on your terms," was originally published at InfoWorld.com. Read more of Paul Venezia's The Deep End blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.

RELATED TOPICS
| 1 2 Page 2
From CIO: 8 Free Online Courses to Grow Your Tech Skills
View Comments
Join the discussion
Be the first to comment on this article. Our Commenting Policies