I've written a few times about the many unexpected complexities involved in getting an IP storage network to work optimally. Whether you're using NFS or iSCSI, it's not surprising that the switches you choose -- and how you configure them -- can make an enormous difference in both the performance and reliability you can expect to achieve.
Though 10Gbps Ethernet is becoming far less expensive than it once was, you'll still pay a substantial premium for it in comparison with 1Gbps Ethernet. This cost rears its head not only in the switches, but also in items as simple as cabling; even twin-ax direct-attached copper cables run around $150 each, a fair sight more expensive than a $5 CAT5e cable. For enterprises seeking the high bandwidths and simplicity that 10GbE can offer, it can be an excellent and entirely worthwhile investment, but most of us simply don't need it yet.
[ Also on InfoWorld: Matt Prigge helps you answer the eternal question: NFS or iSCSI? | Sign up for InfoWorld's Data Explosion newsletter for news and updates on how to deal with growing volumes of data in the enterprise. ]
That's why the vast majority of IP-based storage implementations use garden-variety 1Gbps Ethernet technology and rely upon multipath I/O or link aggregation to achieve performance in excess of 1Gbps. Though it's absolutely true you can use general-purpose networking gear for IP storage, not all 1Gbps Ethernet switches are well-suited to handling the loads associated with IP storage. Here are a few guidelines you can use to grade your options if you find yourself on the hunt.
Flow control and packet buffers
Another crucial switch tech is 802.3x flow control, a mechanism that allows an Ethernet interface to issue a packet called a "pause" frame that signals to the system on the other end it should stop sending traffic for a specified period of time (usually very, very small). This is extremely valuable in IP storage applications as a device, be it the client, SAN, or switch, can signal to the sending device that it is overwhelmed -- a process very similar to asking a coworker who has had too much coffee to speak just a bit more slowly.
Without flow control, when a link or device becomes overwhelmed with traffic, some traffic will inevitably be dropped. When that occurs, the sending and receiving hosts will detect it and retransmit the missing packets. That can take a relatively large amount of time and result in noticeably decreased performance.
Thus, a good IP storage switch will both honor and retransmit flow control pause frames issued by servers and storage devices attached to it, then issue them itself when it becomes overwhelmed. Now, you might think, if I have a switch with a nonblocking backplane, how could I ever have a situation where the switch itself is overwhelmed?
Imagine a situation where a number of hosts are writing a large amount of data to a SAN and that traffic is all trying to get down a single gigabit Ethernet port. In that scenario, there will undoubtedly be times (even if they are extremely short) when a port's available bandwidth will be less than the traffic that needs to be sent. In that case, the switch will typically use its output packet buffer to store the data waiting to be sent until there is adequate bandwidth available. However, if the packet buffer is filled, traffic will be unceremoniously dropped. By issuing a pause frame to the sending hosts, the switch can give itself time to empty its buffers.