"We're primarily an HP shop, and we had contracts on all those servers, which were from different generations, so if you lose a motherboard from one model, they'd overnight it and it was a big pain," Shipley says. "So we said, 'Look, we've got to get away from this. Virtualization is ready for prime time, and we need to get out of this traditional game.'"
Today, what Supplies Network has built in its data center is about as far from traditional as it gets. Rather than deploying Ethernet and Fibre Channel switches, the company turned to I/O Director from Xsigo, which sits on top of a rack of servers and directs traffic. All of the servers in that rack are plugged into the box, which dynamically establishes connectivity to all other data center resources. Unlike other data center fabrics, I/O Director offers InfiniBand, an open standards-based switched fabric communications link that provides high-performance computing.
"On all your servers you get rid of all those cables and Ethernet and Fibre switches and connect with one InfiniBand cable or two, for redundancy, which is what we did," says Shipley. The cables are plugged into I/O Director. "You say 'On servers one through 10, I want to connect all of those to this external Fibre Channel storage' and it creates a virtual Fibre Channel storage network. So in reality, this is all running across InfiniBand, but the server ... thinks it's still connecting via Fibre Channel."
The configuration means they now only have two cables instead of several, "and we have a ton of bandwidth."
Supplies Network is fully virtualized, and has seen its data center shrink from about 20 racks to about four, Shipley says. Power consumption and cooling have also been reduced.
Shipley says he likes that InfiniBand has been used in the supercomputer world for a decade, and is low-cost and open, whereas other vendors "are so invested in Ethernet, they don't want to see InfiniBand win." Today, I/O Director runs at 56 gigabits per second, compared with the fastest Ethernet connection, which is 10 gigabits per second, he says.
In terms of cost, Shipley says a single port 10-gigabit Ethernet card is probably around $600, and an Ethernet switch port is needed on the other side, which runs approximately $1,000 per port. "So for each Ethernet connection, you're looking at $1,600 for each one." A 40-gigabit, single-port InfiniBand adapter is probably about $450 to $500, he says, and a 36-port InfiniBand switch box is $6,000, which works out to $167 per port.
Shipley says the company has now gotten rid of all of its core Ethernet switches in favor of InfiniBand.
"I was afraid at first because ... I didn't know much about InfiniBand," he acknowledges, and most enterprise architectures run on Fibre Channel and Ethernet. "We brought [I/O Director] out here and did a bake-off with Cisco's [Unified Data Center]. It whooped their butt. It was way less cost, way faster, it was simple and easy to use and Xsigo's support has been fabulous," he says.
Previously, big database jobs would take 12 hours, Shipley says. Since the deployment of I/O Director, those same jobs run in less than three hours. Migrating a virtual machine from one host to another now takes seconds, as opposed to minutes, he says.
He says he was initially concerned that because Xsigo is a much smaller vendor, it might not be around over the long term. But, says Shipley, "we found out VMware uses these guys."
"What Xsigo is saying is, instead of having to use Ethernet and Fibre Channel, you can take all those out and put [their product] in and it creates a fabric," explains Bob Laliberte, senior analyst at Enterprise Strategy Group. "They're right, but when you're talking about data center networking and data center fabrics, Xsigo is helping to create two tiers. But the Junipers and Ciscos and Brocades are trying to create that flat fabric."
InfiniBand is a great protocol, Laliberte adds, but cautions that it's not necessarily becoming more widely used. "It's still primarily in the realm of supercomputing sites that need ultra-fast computing."
Esther Shein is a freelance writer and editor. She can be reached at email@example.com.
Read more about data center in Computerworld's Data Center Topic Center.