Five lessons of a datacenter overhaul

A datacenter makeover and migration can go wrong in many ways. Do as we suggest, not as we did.

Finally, moving day arrived. To make our migration easier, we’d contacted the boisterous folks at Silverback Migration Solutions, a Walnut Creek, CA-based outfit that specializes in helping companies perform datacenter migrations and build-outs. Where a general IT staff might take days to put racks together, add shelving and other accessories, slide the servers in on new rails, and test functionality end-to-end, Silverback cranks through these tasks in record time, sometimes installing 30 or 40 full racks of servers in a single day. (In our case, it was 10 racks in a few hours; see "Pimp my datacenter: SilverBack Migration Solutions".)

But while Silverback’s on-site reps were willing, our planning was weak. Though we’d had months to do the prep work, we’d slipped into complacency and simply assumed certain things would work out as desired. Murphy gleefully proved us wrong.

Using APC’s datacenter planning software, we’d created the necessary blueprint of our new physical layout, but instead of fleshing that out we let it go and assumed that auto-generated floor maps were enough. A conversation with Silverback and Rackwise reps cured us of that notion, sending us back to the drawing board to fill in some important gaps.

APC’s rack maps were a good start, but they’re not designed to take into account customized weights of individual pieces of infrastructure — they use reference weights from a vendor database in order to provide ballpark figures. So the standard weight APC provides for a Dell PowerEdge 1650, for example, might reflect a configuration with two hard drives, whereas our servers might have four. Not a big difference for one server, but when you multiply by dozens per rack, and you're facing an 800-pound weight limit, the true weight becomes important. We were forced to make several rack reconfiguration decisions on the fly.

A second important omission was failing to gather full technical documentation for the equipment being migrated. Because the HIG 319 datacenter was to serve as a co-location facility for a number of SOEST departments, we would be re-racking, rewiring, and reassembling systems from all around campus. Detailed notes -- and aha, admin passwords! — would be needed to put them all back together again. Yes, we not only lacked detailed information on how some of that equipment was configured, we even failed to collect the admin passwords for six servers we moved from another building. That meant we couldn’t bring them up for testing until their research administrators could be found. Like most server migrations, ours was performed during off-hours, so not having the passwords wound up pushing the final equipment testing part of our plan well into production hours on the following day.

A migration day typically leaves little room for error or indecision. Before that day comes, you should have a punch list — a list of detailed, step by step instructions — that will guide everyone's actions from start to completion. We suggest that all team members keep notes on loose ends and to do's, and provide them to the team leader in time to create the punch list before any vendors go home. Don't let your solutions providers leave unfinished business behind. It will only be harder to finish without their help.

Finally, the project leader can’t expect to be everyone’s friend if you want your project to succeed. Guide your team and your vendors with a firm hand, and throw a completion party to smooth over any bruised egos. And be sure to schedule a wrap-up meeting to discuss what went right and what went wrong. Someday you might have to do this again.

| 1 2 Page 4