Get creative in the basement. When basement floors flood, fuel tanks can float, but they also do damage to pipes and fuel pumps. That was a big problem for Internap during Sandy, and New York City regulations require that fuel tanks remain in city basements. "The challenge," says Internap's Steve Orchard, "is that fuel tanks may be underground, somewhat buried, or sitting above ground in basements on stands depending on when the building was built." Lessons: Make tanks submersible by strapping them into place to structural steel so they won't float, Orchard says. "We're also looking into submersible pumps sitting inside the fuel tank along with placing a redundant pump on the mezzanine level." For added safety, Internap could also install a basement pump at a height above six feet.
Consider triple power backup. New York's CitiServ data center in Brooklyn -- a two-building project that is part of Mayor Michael Bloomberg's efforts to consolidate 55 city data centers into one -- stayed up throughout Sandy while managing the emergency 911 and 311 services. "We placed the facility in Brooklyn outside the hurricane flood zone, and we had a triple power backup strategy," which includes utility power, emergency generators and battery, says Rahul N. Merchant, New York's chief information and innovation officer. At the height of the storm, lights flickered briefly, but generator power kicked in seamlessly. A tank holding 200,000 gallons of diesel located on-site at the Brooklyn facility -- and topped off regularly -- powered everything for 45.5 hours before regular utility power was restored.
Re-patch and restore. In addition, the city's own fiber optic network, CitiNet, was used to restore 311 service when Verizon's cable vaults flooded and failed at downtown Manhattan nodes. "We were able to re-patch the 311 network by using our own fiber backbone and connecting it [directly] into the Verizon Central Office in Brooklyn," Merchant says.
Use sandbags. The bullet just missed when Sandy hammered at QTS's data center in Jersey City, N.J. "We were impacted by rising waters and high winds, and the water pushed up right to the edges of our building," says Brian Johnston, QTS's chief technology officer. However, the building was storm-proofed; the actual data center had been located on the 16th through 20th floors, with all power generation and logic on the same level except for electrical vaults and lines coming from the streets. "In the event we lose the vaults, everything we need transfers to UPS; and our fuel oil in 1,000-gallon fuel tanks along with electric pumps in the garage parking lot are also water-proofed," Johnston says. Though QTS temporarily lost utility power, backup generators took over, and no basement flooding occurred. Lesson: "Our skilled staff redirected waters with sand bags. We had wraps around any openings in the building and our parking center, so water couldn't come into parking garage or flood our vaults." Future plans: more flood blockers -- borrowed from hurricane protocols in Miami -- and training for IT staff in emergency procedures.
This story, "Data centers under water: What, me worry?" was originally published by Computerworld.