Remember when modern infrastructure meant provisioning software in a few virtual machines with Chef? Or managing the lifecycle of a couple of VMs using Terraform? As an industry, we don’t live in that world anymore.
Today’s most successful development teams have moved beyond managing a dozen or a hundred cloud infrastructure components, and instead have to think about thousands of cloud resources. In the modern world of containers and Kubernetes, the environments are huge in scale and complexity, the rate of change is infinitely faster, and the division between application and infrastructure has become blurred.
In such a world, developers increasingly depend upon infrastructure-as-code (IaC) tools like Pulumi. I’ve written about declarative languages in this area like Polar and HCL, but Pulumi’s approach could give developers the ability to write code in their preferred language, like TypeScript, while calling APIs across a range of cloud and SaaS providers. It’s a promising approach, but will it work? It sort of has to.
Yes, you need IaC
Justin Etheredge does a good job of explaining why IaC is increasingly a must-have, not nice-to-have, in a cloud world. Among a variety of benefits (repeatability, auditability, portability, etc.), Etheredge’s first reason is central to all the rest: confidence. That is, “Infrastructure as code gives you the freedom to make changes without the fear that you’ll put things in an unrecoverable state. And it gives you a better understanding of how the environment came to be the way it is, which allows you to be more confident to make the changes you need.”
That’s the why, but the how to deliver on IaC’s promise is equally important. If we look back at the evolution of enterprise adoption of cloud, the first phase involved developers working within the confines of a single VM. A developer might spend their time laying down packages within a package manager inside their Ubuntu box. In the next phase, developers fell in love with Docker, running multiple VMs. It was progress, but much of the magic happened inside those VMs, with early iterations of IaC to connect VMs together. Today, by contrast, developers are inundated with thousands upon thousands of cloud resources and APIs.
This is where Pulumi steps in.
In my interview with Pulumi CTO Luke Hoban, he stressed the importance of enabling developers to define and deploy IaC using their preferred programming languages, including JavaScript, Python, and Go. Pulumi’s approach also gives developers access to every resource that a particular cloud provider might offer, including beta resources, made available directly in their IDE.
Every resource?
Abstracting away complexity
“We provide the raw API surface area of every cloud,” Hoban noted, giving developers and devops teams “one pane of glass” for AWS, Microsoft Azure, Google Cloud Platform, Kubernetes, and roughly 50 other providers (like SaaS providers). This sounds amazing... and terrifying. Amazing because of all the power to build, but terrifying because it sounds incredibly complex.
Pulumi and similar tools allow developers to embrace cloud services without having to manage them. Developers get to offload the operational complexity of, say, Kubernetes, by embracing a Kubernetes managed service, focusing instead on the “glue” that pulls together Azure Analytics and Amazon S3 and Google Cloud Machine Learning and New Relic and... you get the picture. Instead of having to build all this infrastructure, they “inherit” managed cloud services and end up writing less code overall.
How does Pulumi do this? “We build libraries and abstractions and things on top of [these resources], that are simpler,” said Hoban. “There’s a big opportunity to reliably build those reusable building blocks so folks can now enter and work at higher levels of abstraction that are different from what their cloud provider offers as managed services directly.”
Behind the scenes, Pulumi is basically building up a graph that includes all of the resources used in your infrastructure, as well as their dependencies on one another. As the developer writes IaC, Pulumi checks what the developer wants to build versus what’s actually running in the cloud, and figures out the changes needed to bring the developer’s infrastructure vision to life. So the developer can focus on writing a single resource, while Pulumi takes that instruction and builds 30 or more resources inside a cloud provider to create the subnets and the route tables and everything necessary to have a functioning VPC, or whatever the infrastructure may be.
This obviously helps developers manage cloud complexity. But there are other significant benefits as well.
Making friends, making clouds
For example, Pulumi helps to bridge the gap between infrastructure and application teams. According to Hoban, a tool like Pulumi allows the platform team, for example, to push more capability for managing infrastructure into the application development team so they can move faster. It’s a much better approach than telling the application team, “Write 10,000 lines of CloudFormation to get this thing to work.”
Does this make working across multiple clouds easier? “We do have some abstractions that try and abstract away differences across different clouds, but those are really a reasonable minority of what gets used in practice,” Hoban said. “Folks typically work more directly with individual capabilities and individual cloud providers.”
Much more typical, Hoban added, is for customers to focus on a single cloud provider, but then rely on a handful of other providers for various, more targeted pieces of infrastructure, like Cloudflare. One cloud or many, Pulumi’s bold approach promises to help enterprises embrace increasingly complex cloud deployment models with confidence.