On the x86 server side, Nationwide reached 5,000 servers two years ago, running all sorts of departmental applications, notes Scott Miggo, vice president of technology solutions. "We were growing at a steady pace, so we wanted to control the server space and the costs for new development," he says. "A lot of our servers had only a 10 percent utilization rate," he notes, wasting much of the server capacity. Roughly half are production application servers and half are test-and-development servers.
Today, there are 3,500 physical servers in operation but still 5,000 instances under management. The VMware servers typically host a dozen instances per physical server. "We assume every new server or application can be virtualized," Miggo says. He expects to have more than 1,000 virtual servers running on 89 physical servers by 2009. His team works to tweak I/O requirements or settings for virtual servers that may have trouble supporting I/O needs in their default state, and it works with vendors to support VMs where possible, before falling back to dedicated servers. Lotus Notes and database applications are examples of where the I/O requirements make virtualization not an option.
Although it's easier to deploy servers in a virtual environment by using predefined images, "our staffing has stayed pretty much the same," Miggo says, because the same number of servers must still be monitored. The slight reduction in management overhead is offset by the need for VMware experts to manage the VMware environment, in addition to the server instances. And Nationwide learned the hard way that it needs to monitor both its virtual and physical servers, then correlate the two to identify potential trouble spots, Miggo says. Not doing so can make diagnosis hard, since a VM issue may be caused by a physical issue that a server monitor application running in only the VM won't detect.
There's also more overhead needed for Miggo's shared-services group to coordinate all users (such as developers and application managers) of a physical server, as changes to the physical server could affect them all -- and because adjustments to one VM's application mix could affect I/O and other physical server performance characteristics. "We now have to coordinate with 12 users," Miggo says. "People were used to having their own boxes and doing their own thing. Now they're more tightly controlled, which they don't like. But they have to follow control processes like code management and release management since what they do could affect others," he notes.
The use of VMware’s VMotion software to load-balance and failover VMs across physical servers might reduce management, but the insurer is still exploring that technology. One barrier: Its requirement that all server instances run on physical servers with the same CPU family, requiring a consistent server pool to be in place.
Nationwide runs its Web servers in a zLinux environment on IBM zSeries mainframes, supporting 650 guest operating systems (VMs) today, with space for another 650 before additional hardware frames would be needed.
Often, using z/VM to support the virtualized Web servers requires a different strategy than using VMware to support virtualized Windows servers, Woeckener notes. For example, you might add memory, CPUs, or network cards to increase I/O on a VMware server. On a zSeries, the CPUs just handle executables, offloading physical I/O (such as network bandwidth and resource scheduling) to dedicated processors, so you don't need to worry about performance-sapping overhead incurred by server resource management.