Over the last several weeks, I've delved into forgotten aspects of building an IP storage network and how to best leverage it with both NFS and iSCSI -- the two dominant IP storage protocols used in virtualization. Throughout that time, I've received a bunch of queries from readers, all united by one question: Which is better, NFS or iSCSI?
As with many hotly debated IT subjects, the choice between any two popular competing technologies is less about which is better overall and more about which is best for solving the challenge at hand. NFS and iSCSI are no different. Both have strengths and weaknesses depending on the situation. But the future of storage -- which will make geographically diverse storage clustering a reality -- may significantically factor into your choice of protocol.
File vs. block
As I mentioned in last week's post, NFS and iSCSI couldn't be much more different, either in their implementation or history. NFS was developed by Sun Microsystems in the early 1980s as a general-purpose file sharing protocol that allowed network clients to read and write files to a server across a network. iSCSI came along much later, in the early 2000s, as an IP-based alternative to Fibre Channel -- which, like Fibre Channel, encapsulates block-level SCSI commands and ships them across a network.
The key difference is where the file system is implemented and managed. In file-level implementations such as NFS, the server or storage array hosts the file system, and clients read and write files into that file system. In block-level implementations such as iSCSI and Fibre Channel, the storage array offers up a collection of blocks to the client, which then formats that raw storage with whatever file system it decides to use.
Though this distinction has many ramifications, perhaps the most important is that in block-level protocols such as iSCSI (and Fibre Channel), the storage array generally isn't aware of what it is storing. All it knows is that it has allocated a collection of blocks and which iSCSI client(s) might have access to them. Conversely, in file-based protocols such as NFS, the storage array has full visibility to all of the application data stored on it -- whether that's general file sharing data or the files that might make up a collection of virtual machines.
From a practical perspective, this array-side knowledge of the data being stored in NFS-based deployments makes it easier for the storage array to track actual storage usage -- in thin provisioning, being able to take snapshots or backups of individual virtual machines and even array-side deduplication of primary storage data.
But recent SCSI T10 enhancements as implemented in VMware's VAAI (vSphere APIs for Array Integration) add similar functionality to block-based storage through support for the UNMAP SCSI primitive, which enhances the virtualization stack's ability to free unused blocks (thus allowing the array to reclaim them) and array-side copy offloads, for accelerating tasks such as virtual machine cloning. In a sense, some of the intelligent hypervisor-to-array integration already possible on file-level systems using NFS is being grafted onto block-level implementations through extensions to the SCSI protocol.
Yet I'm certain that's not where the file-vs.-block story will end. As much as adding those SCSI primitives allows SCSI-based storage protocols to perform some of the same tricks as NFS, in other situations a file-level protocol still has an edge -- stretched clusters being a good example. In these kinds of synchronously replicated, geographically diverse storage implementations, giving the storage layer the ability to treat each virtual machine as a separate storage resource -- to be moved and failed over individually rather than as huge opaque groups -- will be vitally important and potentially very difficult to accomplish using block-based storage protocols.