Fibre Channel was definitely not top of mind when Chris Brown hit the wall on disk space and, in mid-2005, decided to go shopping for a SAN. Brown is IT manager for DeltaValve, a division of Curtiss-Wright Flow Control. “I have an IT staff of two,” he explains, “and we do not have the resources to support Fibre Channel.”
Instead, Brown opted for the convenience and low cost of an IP-based network storage system. To that end, he bought five NSM 150s from LeftHand Networks as building blocks for a new iSCSI SAN. Each unit comes with four 250GB drives, so he had a cluster of five terabytes of new storage that, unlike a Fibre Channel SAN, would require little in the way of specialized skills to maintain. (See also “The trends that shape iSCSI’s trajectory” and “New choices in networked storage.”)
The value proposition of iSCSI storage has always been its simplicity and low cost compared to Fibre Channel. All you need is capacity on a Gigabit Ethernet network — no special training in an esoteric protocol, just a little education on top of basic networking skills. An iSCSI SAN can also smooth the path to data replication and disaster recovery, especially over long distances. And if speed is an issue, 10GbE (10 Gigabit Ethernet) is already here, if somewhat pricey.
Drue Reeves, research director with the Burton Group, calls the iSCSI SANs from such vendors as LeftHand, DataCore, and FalconStor “software-only targets. They are relatively cheap,” he explains, “because they run on standard hardware. They are powerful because you can cluster them and add storage virtualization on top, so a LUN [logical unit number] can fail over to another target, and the user never knows.”
It was clear that iSCSI had arrived when Microsoft put an initiator in Windows Server 2003. What was not so clear was where iSCSI would go. Fibre Channel still rules the SAN market, and Microsoft didn’t follow suit with an iSCSI target until last year.
But last fall iSCSI got a huge boost when VMware, the hottest name in virtualization, added iSCSI support. To get the most out of virtualization, you need a SAN — and now you can do it without Fibre Channel.
DeltaValve’s Brown, who isn’t afraid of getting under the hood, quickly saw the potential of his new SAN. “I moved everything — SQL Server, Navision, SharePoint, Exchange Server, and an Oracle database that runs our PLM (Product Lifecycle Management) System — onto them,” he says.
Brown’s willingness to tinker also took him deep into the world of virtualization. “About three months after we got the SAN running, I brought VMware into the mix. Virtual storage from the SAN and server virtualization from VMware go hand in hand.”
Brown has two host VMware servers (both homegrown, Quad AMD Opteron-powered boxes) and runs eight virtual servers on each. “The beauty,” he says, “is that if one host goes down, we can use the other host to mount the same volume and be up in a matter of minutes.”
Thane Morgan, director of information technology for the town of Fishers, a suburb of Indianapolis, had virtualization on his mind from the start. He also looked at LeftHand but did not like the way that software vendor made the hardware decision for you. “Being dependent on their [LeftHand’s] control of hardware drives my cost up,” he says.
So Morgan bought SANMelody software from DataCore that is hardware-agnostic. “For about $60,000, I got two brand-new, dual-core Dell 1950s to be my new app servers. Then I was able to load SANMelody onto the best two of my old servers to create my SAN.”
Hanging SATA drive cages off the converted SAN boxes gave him six terabytes of storage. “I am licensed for 16 terabytes with DataCore,” Morgan says. “With LeftHand, for the same amount of money, I would have been stuck at two four-terabyte nodes without my new app server boxes.”
He bought the hardware and software and did prototyping last summer, so when VMware announced support for iSCSI in September, he was ready. “We got it all running in December, and, finally, my server is completely decoupled from the hardware.”
Clearly, Morgan is excited by the power of combining server virtualization with SAN technology. “If I had virtual servers and no SAN, it would be easy to back up and restore the server on another machine if, for example, I needed to do maintenance. But this still takes time and probably means taking some applications offline. When you add the SAN, you can do the same thing with no interruption of service.”
Consultants such as Jamie Anderson, president of Emergent Networks, a consultancy and VAR in Minneapolis, are finding the combination of iSCSI and virtualization is enough to convince hesitant clients to make the leap to shared storage.
Anderson cites a recent engagement with a small bank. “This is a new bank in Savage, Minn.,” he says. “Without iSCSI they would not have considered a SAN. But we just put in an EMC Clariion and two virtual servers. Right now they only have 800GB of data, but they are in good shape to grow, and there was almost no learning curve since it is all Ethernet based.”
Anderson adds that virtualization was one of the drivers behind Chief Manufacturing’s selection of an iSCSI SAN. “We did this about 18 months ago,” he says. “We use iSCSI to mount local drives to Exchange and SQL Server.”
Easing the pain
Virtualization is a key iSCSI selling point, but it’s just one of several. As director of professional services for Intelenet Communications, a hosting firm, Jeff Stein is always searching for ways to streamline provisioning and maintenance. To that end, he started looking at iSCSI about three years ago. “We did not find it suitable for our managed server offering at the time,” he says.
Fast-forward to today, and Stein is an iSCSI fan. “The technology got flatter, and prices dropped,” he says. “iSCSI made it possible for us to go to a diskless managed server. Our OS and our customers’ data now reside on the SAN, and we deliver about 1,000 servers via our iSCSI environment. We have cut our provisioning and response times by a factor of six going to an iSCSI SAN.”
He had already provisioned Fibre Channel SAN infrastructure as a special service for contract customers, but Intelenet’s generic offering was all DAS. “We looked at extending Fibre Channel to all our customers,” he explains, “but it would have cost 10 times as much as iSCSI. Then there is the skill-set issue. No member of our 24/7 staff is afraid to get on these [iSCSI] devices. I don’t think this would be true for Fibre Channel.”
Intelenet built its SAN on data arrays from EqualLogic, an iSCSI storage hardware vendor. EqualLogic also ships with virtualization software running on clustered storage.
EqualLogic went head to head with the Clariion CX Series from storage giant EMC in a final, proof-of-concept test at Safeway Insurance Group. Mike Leather, network services manager there, says, “Even after sending out three teams, the EMC folks could not show me the performance stats I needed, so we went with the PS line from EqualLogic.”
Leather needed better performance on his primary business system, a Microsoft SQL Server cluster. The I/O to the system’s DAS was the bottleneck, which immediately suggested a SAN solution. “We looked at Fibre Channel, but iSCSI is so much cheaper,” Leather says, “and you don’t need to buy any HBAs or Fibre Channel switches.”
“We had it up and running by mid-October,” Leather says, “but did not go live until February 2006. We were extra-cautious since our whole business runs off this system.”
Since then he has migrated storage for other systems to the SAN, including secondary SQL databases and Microsoft Exchange. In summer 2006, he started a new project, replication, for which he bought a new PS 3600 and EqualLogic replication software. “We now have two equal architectures, and we are replicating our database every five minutes.”
Right now both of these are at the home site in Westmont, Ill., but in April, Leather says one will move more than 800 miles to a secondary site. “This will be iSCSI over 10Mb metro Ethernet,” he says. “Robust disaster recovery. By the second half of 2007, we plan to have full two-way replication, two fully redundant hot sites. By the end of the year, we plan to have load balancing as well.” In that same time frame, Leather also plans to start testing virtualization.
So far, Leather has had no major problems, but he recommends that before you consider iSCSI, you look carefully at one thing. “If you have not made major improvements to your network infrastructure in the last three years, you had better do it. We were OK since we had put in Gigabit Ethernet with a new switch a few years back.”
The right infrastructure helped sell Dan Brinegar, IT administrator for beer distributor House of LaRose, on iSCSI. “We had moved into a new building and so were able to design our own network, all Gigabit Ethernet,” Brinegar says. “We also bought Catalyst 4500 switches that allow routing.”
The routing let Brinegar segment his iSCSI traffic from his production network. This is an important consideration as iSCSI’s greatest strength, the fact that it can run over your standard IP lines, can also be a weakness if storage traffic drags down mission-critical systems.
Brinegar uses IPStor software from FalconStor, which transforms a high-end Linux box into a RAID array with four terabytes of storage. “Our main goal was CDP, continuous data protection, and I had a limited budget of about $40,000. iSCSI was really the only option given my performance requirements.”
He now has continuous backup of a data warehouse, ADP payroll, and a fleet-maintenance system. “We have a lot of Novell, but that is still on Fibre Channel,” Brinegar says. “NetWare 5.1 just does not like iSCSI. When we upgrade to 6.5, we will probably convert that to iSCSI as well.”
“This [iSCSI] really changed the way we do things. Before, we had a separate tape drive for every server. My life is a whole lot easier now,” Brinegar says.
The need for speed
At the high end, Fibre Channel is still SAN king, and speed is obviously one of the main reasons. Affordable Ethernet is still pretty much a one-gig horse, whereas Fibre Channel runs at four gig, with eight on the way.
There are ways to squeeze more performance out of iSCSI. Intelenet’s Stein, for example, plugs iSCSI HBAs from QLogic into his servers. This is one of the technologies that got “flatter,” in his opinion, making iSCSI both fast and cheap enough to move his datacenter from DAS to SAN.
“You can enhance iSCSI with HBAs,” Reeves says. “You can also add, on the target side, TOE (TCP offload engine) cards or DMA (direct memory access) cards. On the initiator side, you can get NICs with iSCSI initiators built in.”
Anderson of Emergent Networks chose to use QLogic iSCSI HBAs for the engagement with Chief Manufacturing. “We could have stuck with the Ethernet ports, and Microsoft’s iSCSI Initiator [the QLogic HBAs come with a separate iSCSI Initiator],” he says, “but we went this route in order to offload the TCP/IP from the processor onto the iSCSI cards.”
Now, Anderson is not sure the performance boost from that is worth the extra expense and hassle. “Some of the Ethernet cards from Intel and Broadcom now come with TCP/IP offload cards, so, in my opinion, the lines between iSCSI HBAs and standard Ethernet interfaces are blurring.”
Meanwhile there is no doubt that cheaper 10GbE (10 Gigabit Ethernet) is on the way, and iSCSI is one of the drivers. In January, Bell Micro, a distributor of storage and computing technologies, signed a distribution agreement with Chelsio, a provider of 10GbE Ethernet adapters and ASIC solutions. Also in January, Brocade Communications bought Silverback Systems, a company that makes network processors to help to accelerate the speed and performance of storage traffic in networked storage environments. Brocade cited Silverback’s technology and expertise in iSCSI as a main reason for the acquisition.
Stephanie Balaouras, analyst with Forrester Research, thinks all this 10GbE action will make the SAN market very interesting. “It [10GbE] is too expensive right now,” she says. “I expect the cost to come down to affordable levels in three to four years.”
Don’t, however, expect to see Fibre Channel beat a hasty retreat. “Storage buyers tend to be the most conservative,” Balaouras says, “so even if iSCSI is competitive in price and performance, you won’t see people ripping out their Fibre Channel.”
And Reeves says there are other issues. “Don’t forget that the high-end storage arrays are still built for Fibre Channel. Sure, EMC says they support iSCSI on their Symmetrix line, but this is essentially an add-on. I think these vendors will eventually embrace iSCSI, but they are going to protect their high profit margins on expensive Fibre Channel equipment for as long as they can.”