Avoid disaster with stretched clusters

So-called stretched clusters promise utopia of disaster avoidance and disaster recovery virtualization, but the technology is still new. Here's how to deal with the challenges

Because virtualization offers so much cost savings and agility, many organizations are going the whole hog and virtualizing mission-critical systems they wouldn't have dreamed of virtualizing a little while ago. Who would have thought, for example, that we'd see enterprises deploy Oracle virtually rather than on physical hardware?

It's no surprise there's increasing interest in wringing every last drop of disaster recovery and disaster avoidance out of virtualization infrastructures. Most commonly, this is manifest in a replicated, dual-site architecture where virtualization resources can be quickly failed over to a secondary data center in the event that catastrophic failure strikes the primary site.

For some organizations, however, this isn't enough. They want the capability to seamlessly migrate workloads from one site to another in order to utilize computing resources at both sites. Moreover, they want the infrastructure to heal itself automatically in the event of a site or hardware failure -- just as single-site virtualization clusters are able to do. Hence the new buzz phrase "stretched cluster."

The stretched cluster has its challenges. But advances in the capabilities of virtualization stacks and the underlying storage gear have made so-called stretched clusters an increasingly attainable and attractive option.

The challenges

There are three main challenges to implementing a stretched cluster:

  1. You must have serious connectivity between data centers.
  2. You need to be able to support live migration of virtual machines from one site to another (using VMware vMotion, for example).
  3. You need synchronously mirrored storage at both sites, incurring bandwidth and latency requirements that may be difficult to deliver.

Typically, you'll see stretched cluster implementations built on multiple gigabit-class metro Ethernet deployments or dark fiber pushing 10Gbps Ethernet and/or 8Gbps Fibre Channel. Whatever type of connectivity is deployed, it needs to be able to deliver no higher than 10-ms latency for VM migration and 5-ms latency for storage replication, but you'll probably want to shoot for much less.

Right off the bat, some organizations will simply lack the connectivity they need to implement this kind of design. Nonetheless, even in the sticks of northern New England, I've seen relatively cheap, fiber-based WAN alternatives appearing in places I would never have expected only a couple of years ago.

Provided you have the connectivity, the second requirement is a storage platform that can support a stretched cluster implementation. While synchronous site-to-site replication has been around for a long time, most synchronous replication implementations are active/passive configurations. This means one side is specifically designated as production, while the other is failover only. In such deployments, manual intervention is usually required to "promote" the inactive failover array to become active and offer up the storage at the recovery site. That's great for a hot/warm site configuration, but doesn't fulfill the goal of reacting to a failover automatically.

Only a few storage platforms allow for both sides of a synchronously replicated storage infrastructure to be active at the same time (or, more accurately, to appear as a single storage infrastructure). EMC's Vplex, HP's LeftHand P4000, and NetApp MetroCluster are a few good examples, though there are others. It's very difficult to gracefully handle the large number of failure scenarios that could occur without opening the storage environment to the real danger inherent in split-brain scenarios, where the two sites lose their synchronization.

Herein lies the third and largest challenge of implementing a stretched cluster, one that virtualization vendors and storage vendors are still grappling with: the wide range of split-brain and suboptimal workload placement scenarios that can result when you try to engineer geographically diverse, active-active storage infrastructures.

Imagine you've constructed a stretched virtualization cluster consisting of six virtualization hosts and a synchronously replicated storage infrastructure at each of two sites. The sites are attached via two low-latency fiber links: one to support inter-VM traffic and virtual machine migration and the other to provide the bandwidth for the two storage infrastructures to replicate.

1 2 Page 1
Page 1 of 2