Last week, I wrote about the huge benefits and associated challenges of implementing "stretched" virtualization clusters. These offer tangible benefits over traditional hot/warm site implementations in that they allow workloads to be seamlessly migrated from one site to the other. In some cases, they even allow site fail-over to take place automatically.
But the expense of clustered storage -- not to mention the complexity involved -- put true stretched clusters out of reach for many organizations that could benefit from them. In spite of that, even if you don't have the right kind of back-end storage to implement a true stretched cluster, you may still be able to reap some of the flexibility benefits they bring to the table and ease the heavy lifting of complex upgrades and stringent uptime requirements.
Recently, I found myself faced with the prospect of upgrading a sizable dual-site virtualization infrastructure. Though each upgrade taken individually wasn't that complicated, a strict zero-downtime requirement meant that careful planning would be required. In the end, the solution leveraged many of the same concepts present in stretched clusters, illustrating that you can enjoy some of the benefits without having all of the pieces in place.
The pre-upgrade infrastructure consisted of two fiber-attached sites each with their own synchronously replicated Fibre Channel-based storage arrays and blade-based virtualization clusters. During the course of the upgrade, new virtualization blades would be implemented at the production site, the older production virtualization blades would be migrated to the recovery site, and the recovery site virtualization blades would be retired. Along the way, the hypervisor -- VMware vSphere in this case -- would also undergo a major version upgrade, along with all of its dependent components: vCenter, Site Recovery Manager, View, and so on.
At the same time, a substantial data center networking upgrade was being undertaken that would see 10Gbps Ethernet supplant multiple 1Gbps Ethernet links used for top-of-rack to end-of-rack and for cross-site connectivity. Included in the course of this upgrade, the network interconnects in both blade chassis would be upgraded to 10Gbps-capable interconnects -- necessitating a substantial reconfiguration of the blades themselves.
After the VMware management components (namely vCenter, but also other applications that depended upon it) were upgraded, blades were removed from the production environment one at a time, physically reconfigured, installed in the newly reconfigured recovery site blade chassis, and upgraded to the newest version of the hypervisor. At that point, a commensurate chunk of the virtualization infrastructure was migrated (via intersite vMotion) to the recovery site -- essentially stretching the cluster between two sites. This process was iteratively repeated for each production virtualization blade until they had all been migrated, along with their workloads, to the recovery site.
At this point, the entire production environment was running at the recovery site while the primary storage it was accessing was running at the production site. Given that the infrastructure had been designed with sufficient intersite bandwidth to make this possible (just a pair of 8Gbps FC links in this case) and the implementation was relatively temporary, no effort was undertaken to actually migrate the storage to the recovery-site primary storage. However, it's important to note that, if it was desired, Storage vMotion could easily have been used to transition the back-end storage from the primary-site SAN to the recovery-site SAN -- effectively "walking" a VM from one site to another: first through vMotion to move the compute load, then through Storage vMotion for the storage load.
After the production-site blade chassis was upgraded with new interconnects, the new virtualization blades were installed and the process was reversed, except without the movement of physical hardware this time. All the VMs were migrated back to the production site virtualization hosts via vMotion, and the older recovery-site virtualization hosts were removed from the cluster one by one until only the new blades remained. Following that, the older recovery site blades were used to re-form the recovery-site cluster that had existed previously and Site Recovery Manager was reconfigured.
In a relatively short amount of time, literally every piece of networking, interconnect, and blade hardware was upgraded and physically reconfigured; workloads were migrated from one site to another; and the data center network infrastructure was upgraded -- without impacting access to a single VM for more than a second or two at a time. Pretty cool.
While the cluster was in fact stretched across two sites for a short period of time, it's important to realize this doesn't constitute a true "stretched cluster." If, at any point during the migration work, the production site primary storage or the intersite connectivity had failed for some reason, a relatively quick, but still very manual storage fail-over to the recovery site primary storage platform would have been necessary. This is the real difference between a "true" stretched cluster operating on top of clustered, geographically diverse storage and a cluster that just happens to exist at two sites, but is still dependent on storage at a single site.