Who would have thought that storing bits could get so incredibly complicated? Storage has always contained a plethora of protocols, from Fibre Channel to iSCSI to SMB in all its variations, but the arrival of flash and the continual growth of virtualization have turned an already dense topic into a tangled jungle of acronyms, protocols, and abstractions.
The virtualization of the data center has prompted a virtualization wave in storage as well, gradually pulling storage away from physical protocols and toward logical, abstracted storage models like instance storage and volume storage. By providing abstractions, the data center has steadily decoupled virtual machines from storage protocols.
The rise of cloud data centers has also spawned a new class of storage called object storage, which sacrifices the strong consistency of traditional storage protocols in order to provide single namespaces at a global scale.
In this article I’ll provide some clarity by placing instance, volume, and object storage into the evolution of the data center, and show how these new abstractions fit on top of, or alongside, existing storage protocols.
The story of cloud storage is in many ways a story of virtualization. I’ll start with physical environments, move to virtualization, where virtual and physical models begin to diverge, and finish with the cloud, where the physical is almost completely abstracted by virtual models.
At the root of all storage is some set of physical storage protocols, so I’ll begin with a quick recap of physical storage. Three major classes of physical storage models are in use today: direct attached storage (DAS), the storage area network (SAN), and network attached storage (NAS).
DAS. Direct attached storage is the simplest storage model. We are all familiar with DAS; this is the model used by most laptops, phones, and desktop computers. The fundamental unit in DAS is the computer itself; the storage for a server is not separable from the server itself. In the case of a phone it is physically impossible to remove the storage from the compute, but even in the case of servers, where it is theoretically possible to pull disk drives, once a drive is separated from the server, it is generally wiped before reuse. SCSI and SATA are examples of DAS protocols.
SAN. Eventually the storage industry recognized the utility of separating storage from the compute. Rather than attaching disks to each individual computer, we placed all the disks on a single cluster of servers and accessed the disk over the network. This simplifies storage management tasks such as backup and failure repair. This division of storage and compute is often called shared storage, since multiple computers will use a single pool of storage.
It was most straightforward to communicate between the client and server over the network using the same (or very similar) block protocols that were used to communicate with locally attached disk drives. Storage exposed this way is called a storage area network. Fibre Channel and iSCSI are examples of SAN protocols.
In a SAN an administrator will group a set of disks (or a portion of a set of disks) into a LUN (logical unit), which then behaves like a single disk drive to outside computers. The LUN is the fundamental unit used to manage SAN storage.
NAS. While SANs allow us to move LUNs between one computer and another, the block protocols they use were not designed to concurrently share data in the same LUN between computers. To allow this kind of sharing we need a new kind of storage built for concurrent access. In this new kind of storage we communicate with the storage using file system protocols, which closely resemble the file systems run on local computers. This kind of storage is known as network attached storage. NFS and SMB are examples of NAS protocols.
The file system abstraction allows multiple servers to access the same data at the same time. Multiple servers can read the same file at the same time, and multiple servers can place new files into the file system at the same time. Thus, NAS is a very convenient model for shared user or application data.
NAS storage allows administrators to allocate portions of storage into individual file systems. Each file system is a single namespace, and the file system is the primary unit used to manage NAS.
Virtualization changed the landscape of the modern data center for storage as it did for compute. Just as physical machines were abstracted into virtual machines, physical storage was abstracted into virtual disks.
In virtualization, the hypervisor provides an emulated hardware environment for each virtual machine, including computer, memory, and storage. VMware, the initial modern hypervisor, chose to emulate local physical disk drives as a way to provide storage for each VM. Put another way, VMware chose the local disk drive (DAS) model as the way to expose storage to virtual machines.
Just as the fundamental unit of storage in DAS is the physical machine, the fundamental unit in virtual disk storage is the VM. Virtual disks are not exposed as independent objects, but as a part of a particular virtual machine, exactly as local disks are conceptually part of a physical computer. As with DAS, a virtual disk lives and dies with the VM itself; if the VM is deleted, then the virtual disk will be deleted as well.
Most conventional virtualization platforms use a virtual disk storage model. For example, storage in VMware vSphere, Microsoft Hyper-V, Red Hat Enterprise Virtualization, and Xen environments are all managed and attached in a similar way.
Implementing virtual disks
Because VMware wanted to continue to provide the benefits of shared storage to virtual machines, it couldn’t rely on a DAS protocol to implement virtual disks. The obvious next choice would be to use SAN, since a SAN LUN closely resembles a local disk drive.
However, physical LUNs have limitations that make for a challenging fit for virtual disks. Virtualized environments consolidate a number of logical computers onto a single physical server, which means the number of virtual disks on a given host will be much larger than the number of physical LUNs for a host in a physical environment. The maximum number of LUNs that could be attached to a given physical server was too low to support the necessary number of virtual disks.
Perhaps even more important, virtual disks, as with virtual CPUs, must be logical objects that can be created, destroyed, and moved programmatically, and these are not operations that SAN storage was designed to perform. For example, VMware needed to dynamically move VMs between physical hosts, which required shared storage access during the migration.
For these reasons, VMware chose to implement virtual disks as files in a file system (NFS) or in a distributed file system (VMFS) on SAN, rather than as raw LUNs.
From storage protocols to storage models
That VMware chose to implement virtual disks, a DAS-style block storage model, on top of NAS or SAN, illustrates one of the interesting characteristics of modern data center storage. Because the IO from a virtual machine is handed off to software in the hypervisor, rather than to hardware on a device bus, the protocol used by the VM to communicate with the hypervisor does not need to match the protocol the hypervisor uses to communicate with the storage itself.
This leads to a separation between the storage model that is exposed upward to the VM and administrator, and the storage protocol that is used by the hypervisor to actually store the data. In the case of virtual disks, VMware designed them according to a DAS storage model, then used a NAS storage protocol to implement them.
This is a powerful layer of indirection; it gives us the flexibility to mix and match storage models and storage protocols, and even dynamically change the storage protocol without impacting virtual machines. For example, virtual disks are implemented using files in NFS, files in VMFS stored on Fibre Channel LUNs, or even (in VVols, or Virtual Volumes) directly as iSCSI LUNs. The choice of implementation is completely transparent to the application, because eventually all of these protocols will look the same to the VM and administrator; they will look like local, physical disk drives attached to VMs.
Thus the application developer in most public cloud infrastructures cannot know what storage protocol is in use; in fact, the protocol may even change dynamically. We don’t know what storage protocol Amazon uses for Elastic Block Storage, nor is it important for us to know.
Because of the separation between storage model and storage protocol, the storage protocol becomes an infrastructure-facing issue, primarily important for cost and performance, rather than an application-facing decision that dictates functionality.
The landscape of the data center is shifting again as virtualized environments morph into cloud environments. Cloud environments embrace the virtual disk model pioneered in virtualization, and they provide additional models to enable a fully virtualized storage stack. Cloud environments attempt to virtualize the entire storage stack so that they can provide self-service and a clean separation between infrastructure and application.
Cloud environments come in many forms. They can be implemented by enterprises as private clouds using environments like OpenStack, CloudStack, and the VMware vRealize suite. They can also be implemented by service providers as public clouds such as Amazon Web Services, Microsoft Azure, and Rackspace.
Interestingly, the storage models used in cloud environments mirror those in use in physical environments. However, as with virtual disks, they are storage models abstracted away from the multiple storage protocols that can be used to implement them.
Instance storage: Virtual disks in the cloud
The virtual disk storage model is the primary (or only) model for storage in conventional virtualized environments. In cloud environments, however, this model is one of three. Therefore, the model is given a specific name in cloud environments: instance storage, meaning storage consumed like conventional virtual disks.
It is important to note that instance storage is a storage model, not a storage protocol, and can be implemented in multiple ways. For example, instance storage is sometimes implemented using DAS on the compute nodes themselves. Implemented this way, it is often called ephemeral storage because the storage is usually not highly reliable.
Instance storage can also be implemented as reliable storage using NAS or volume storage, a second storage model described next. For example, OpenStack allows users to implement instance storage as ephemeral storage on the hosts, as files on NFS mount points, or as Cinder volumes using boot-from-volume.
Volume storage: SAN sans the physical
Instance storage, however, has its limitations. Developers of cloud-native applications often explicitly differentiate configuration data, such as OS and application data, from user data, such as database tables or data files. By splitting the two, developers are then able to make configuration transient and rebuildable while still maintaining strong reliability for user data.
This distinction, in turn, leads to another type of storage: volume storage, a hybrid of instance storage and SAN. A volume is the primary unit of volume storage rather than a VM. A volume can be detached from one VM and attached to another. However, like a virtual disk, a volume more closely resembles a file than a LUN in scale and abstraction. In contrast to instance storage, volume storage is usually assumed to be highly reliable and is often used for user data.
OpenStack’s Cinder is an example of a volume store, as is Docker’s independent volume abstraction. Note again that volume storage is a storage model, not a storage protocol. Volume storage can be implemented on top of file protocols such as NFS or block protocols such as iSCSI transparently to the application.
Object storage: Web-scale NAS
Cloud native applications also need a home for data shared between VMs, but they often need namespaces that can scale to multiple data centers across geographic regions. Object storage provides exactly this kind of storage. For example, Amazon’s S3 provides a single logical namespace across an entire region and, arguably, across the entire world. In order to reach this scale, S3 needed to sacrifice the strong consistency and fine-grained updates of conventional NAS.
Object storage provides a file-like abstraction called an object, but it provides eventual consistency. This means that while all clients will eventually get the same answers to their requests, they may temporarily receive different answers. This consistency is similar to the consistency provided by Dropbox between two computers; clients may temporarily drift out of sync, but eventually everything will converge.
Traditional object stores also provide a simplified set of data operations tuned for use over high-latency WAN connections: listing the objects in a “bucket,” reading an object in its entirety, and replacing the data in an object with entirely new data. This model provides a more basic set of operations than NAS, which allows applications to read and write small blocks within a file, to truncate files to new sizes, to move files between directories, and so on.