Storage management in the application-centric cloud

New paradigms for storage in microservices and containers

With the advent of composable applications comes a need for software-defined architectures to enable these applications. A more simplistic model that brings this to reality is emerging.

The application-centric cloud depends on this design. Amazon has three trillion objects in their environment, and Google has billions of containers in theirs. Failure to create a simple architecture based on object storage and outside control to the application will cause the application-centric cloud to fail on its weight and complexity. I will describe the efforts in the industry such as object storage, the OCP (Open Container Project), and the evolution of control through plugins and orchestration.

In microservices or containers, there needs to be a data volume manager for the applications using an object storage system similar to Amazon S3 (Amazon's Storage Component). Microservices are services composed of small, independent processes communicating with each other using language-agnostic API. Data volume managers need to handle data migration and run micro-services or containers in a stateful storage service. Stateful storage records the state of the storage when anything is written or deleted in storage. An example is a database that has states defined in its storage.

Today, when containers and microservices are created, storage is statically bound to them. However, this is not plausible in the new application model. The new concept, the OCP, uses plugins; the data volume is portable and associated with any microservices or container in a cluster. When a microservice or container moves in the cluster, so does the volume. This is accomplished by using plugins and a REST API, which orchestrate the communication among agents running on each node in the cluster. ClusterHQ has been instrumental in creating the technology in a lightweight framework.

To make this scalable, Nexenta developed an object store, NexentaEdge, and OpenStack created Swift. NexentaEdge supports objects, files, and blocks, rather than having the volumes be files in a directory and requiring accessibility from any application, in which case performance and deployment become slow and arduous. An Object Storage system stores files in a flat organization of buckets accessed using keys. Storing the metadata with the object makes portability from system to the system easier. Recovery is established by replicating objects, and the recovery is rapid because the objects replicate on multiple volumes. Indeed, recovery is reconstruction since each object is stateful.

Controlling the movement of volumes uses metadata and buckets as "the objects" integrated through a controller with the orchestrator. The microservice or container can allow permissions based on the rules associated with each object and its application. The central repository is searched to find the data. The orchestration then moves the object with the container or clones the container (the objects being in a central repository and the control being on each node).

Using software-defined storage solutions for objects enables the use of commodity hardware. Using this approach, a user can enable their preferred storage array backend in this model.

Thus, the application-centric cloud can make networking and storage available to an application in the same architecture through the new paradigm, enabling a shift in development.

This article is published as part of the IDG Contributor Network. Want to Join?