Typically, this kind of detailed testing is the thing that customers don't do, he says, and it's a bad decision to leave it out. "You've got to test the applications. You have to have the guts to do it."
Olds stresses that it is important to keep in mind the difference between redundancy and availability when it comes to your enterprise's data and applications in an emergency. "You want to have all of your data protected all of the time so that no matter what happens, you never lose it, other than the last half hour or so of data."
But at the same time, if disaster strikes, you don't need to access all of that data immediately. You have to have quick and sure availability only to the data that is mission critical for the business as you recover from the emergency, he says.
"You have to prioritize there," Olds says. "It would be silly and needlessly expensive to have your entire infrastructure mirrored so that you could instantly recover every application. The vast majority of businesses don't need that kind of availability."
One way to do this is to make a list of the applications and data that your business needs to have first in an emergency, followed by the apps and data that would be nice to have at that point. The remaining apps and data can return later, when the disaster is over.
Meanwhile, don't forget to get other input inside your company when making these kinds of decisions. "IT leaders need to be sure that they are bringing in the business side on this stuff," he says. "You want to make sure that what IT thinks should be brought back in first gets agreement from the people on the business side, too."
By testing your emergency plans regularly, you can ensure that no critical steps are left out, such as network topography details and your company's IP addresses. "These are all things that you won't necessarily think about in an emergency," Olds says. "You need to be able to move this all over and mirror it so it is all available outside your company infrastructure if it is out of service. If your data center is underwater for a while, you need to have a long range plan."
Once you start this kind of testing, you need to remember to keep it up, especially when you make new changes to applications and to your hardware infrastructure, he says. The reason is simple: You have to be sure that the system changes you make on a regular basis don't interfere with your existing disaster recovery systems and cause them to fail at the worst possible moment.
"You always have to make sure that you can recover an application with the right data and the right everything else," he says. "That's the key. I recommend that you make it a check box on your application update procedures, so you can determine if the application or system changes require any related changes to a backup recovery plan. Right after your plan is golden, you need to be able to document those changes and constantly check them to be sure they will still work."
If you leave out this step, then your disaster recovery plans are incomplete and likely won't work when they are needed, which defeats the whole purpose of having them.
Todd R. Weiss covers Enterprise Applications, SaaS, CRM, and Cloud Computing for CIO.com. Follow Todd on Twitter @TechManTalking. Follow everything from CIO.com on Twitter @CIOonline and on Facebook. Email Todd at email@example.com You can also join Todd in the "CIO Forum" group on LinkedIn.com to talk with CIOs and IT managers about the things that keep them up at night.
Read more about applications in CIO's Applications Drilldown.