"We knew the data center was in the storm's way, and we made a decision after that event to move the data center because we were holding our breath," says Jill Israel, Entergy's CIO. "We didn't have flooding in the immediate area of our data center," she explains. "But there was no power, our lines were down and we had to run on our generator and keep topping it off. "
By winter 2006, the company decided to create two mirror-image data centers in Jackson, Miss., and Little Rock, Ark. In Little Rock, the company retrofitted an old library with sturdy brick walls, moving hardware and critical applications from New Orleans, piece-by-piece, to the backup facility by 2008.
Finally, by 2010, Entergy had completed its brand-new $30 million Jackson, Miss., data center. The company load-balances several systems including email between the two centers, Israel says. "Moving applications from New Orleans involved quite a choreography plan. Subsequent to Katrina we've had [major] storm events, including ice storms in Little Rock. But I no longer have to hold my breath," she says.
The company holds drills for hurricanes and storms every year "to get better and better at responding," Israel says. "One of the things we quickly recognized was how effective a dispersed workforce can be. Our employees can do a lot more things from remote locations and that has served us very, very well."
Not everyone seems to have absorbed the "take heed" message. IT shops in both Europe and America's Northeast seem to cling to the idea that superstorms are non-repeatable freaks of nature. In some cases, even among those affected by major storms, vigilance plays a chess game with artful forgetfulness as managers choose other IT priorities.
Ignorance isn't bliss
"What was perceived as a safe area before may not be now," says Rakesh Kumar, a Gartner vice president who specializes in data center and infrastructure issues. In Europe, especially, he cites freezing temperatures, coastal flooding and other unpredictable weather events. In Asia, tsunamis are a concern. "Until we have a major data outage, though, most clients are not calculating for risk or change; they're turning a blind eye to it," Kumar says.
Many of his European and U.S. clients praise the idea of doing thorough risk assessments and thinking proactively, long-term, he says. At least in theory.
But even now, six months after Sandy, most East Coast-based companies aren't being proactive about repositioning their data centers, experts say. "They're expanding in the same locations; they're not even thinking about moving," says Neil Sheehan, a data center architect and principal of Chicago-based Sheehan Partners Ltd.
In fact, he says, "we are looking for expansion for our clients in New Jersey right near the coast, [near] sites that flooded." Sheehan says with proper surveys of 500-year-flood levels, data center architects can determine the ideal height of first floors, so that flooding, if it occurs, happens in parking lots and not in data centers.
Some are paying more than lip service. In downtown Manhattan, at 140 West Street, a Verizon switching center felt the full force of Sandy's stormwater. Five sub-basement levels, including a Verizon cable vault, went underwater. Technicians struggled to mount emergency generators and pump water out through elevator shafts.
Since then, Verizon has had to extract 150 tons of damaged copper cable from lower Manhattan streets, its central office, headquarters and customer sites, replacing virtually all of it with weather-proof fiber cable protected in conduit. "If you take a fiber optic cable and lay it in your bathtub it probably will still work; fiber is submersible," says Chris Kimm, vice president of global customer assurance for Verizon Enterprise Solutions.