How to keep container secrets secret

Keeping secrets secret in container-based applications is complex and challenging, but far from hopeless

How to keep container secrets secret
Pixabay (CC0)

In computing as in real life, a secret is information you want kept private, outside of the people and systems you want or need to share it with. In the application security realm, common examples of secrets are passwords, tokens, and private keys. While the concept of a secret is rather simple, keeping application secrets private (at least in technical terms) is anything but simple—especially in containerized environments.

The task of managing and securing application secrets is, of course, not new. Securing them has never been easy, but when applications had relatively few systems with static connections and low interconnectivity, it was manageable. Then came the cloud, virtualization, the connected enterprise, etc., which made secrets management much more difficult. In those environments, secrets needed to be accessible in many more places, and needed to remain accessible to the intended recipients in dynamic, changeable environments.

Container environment complexities

Containers increase that complexity by an order of magnitude. This is mainly due to the way containers scale out, which makes it even harder to ensure that secrets are exposed only to the container that needs them. Also, given the ephemeral nature of containers, it’s important to ensure that secrets don’t persist on the host once the container is no longer running.

In container environments, those responsible for secrets management usually opt to give up some control over the process to a container orchestrator (e.g. Kubernetes, OpenShift, Docker Swarm) in favor of simplicity and efficiency. The ideal scenario is to take advantage of the flexibility and simplicity that orchestrators provide while not conceding any security when it comes to secrets.

Most orchestration tools offer some level of secrets storage and management. However, orchestrators can create a new problem – secrets management duplication. When an organization decides to move to a containerized environment, it probably already has a Key Management System (KMS) for securing secrets (outside of container environments). Now it must either duplicate the secrets it needs in the orchestrator, or (depending on the orchestrator) change its entire secrets infrastructure, or (if possible) integrate the orchestrator’s key management feature with the KMS.

The lifecycle of a secret in a container-based app

Hopefully the complex nature of secrets management is now apparent, which is why having the security team’s input during this phase (which is what DevSecOps promotes) is so important. Equally important, the technical and functional requirements for effective secrets management must be understood by everyone involved in secrets management throughout the lifecycle of the secret. Before we define the actual requirements, let’s map the path that secrets should traverse within a container-based application:

  1. A secret is created by a user with sufficient privileges. The communication channel between the user and application should be secure and mutually authenticated.
  2. The application (or application component) is responsible of authenticating and authorizing users and services.
  3. Every administrative operation on the secret is logged.
  4. A secret may be created, updated, or revoked via a back-end application used by the application the user is engaging with). Communication with the back-end application should be encrypted and mutually authenticated.
  5. The secret is stored in sealed storage, which can only be decrypted using an “unseal” key. The unseal key is stored in a secure, separate location.
  6. For additional security, the secret itself may be encrypted for its consuming container.
  7. Nodes (container hosts) receive the secret (or encrypted secret) over an encrypted and mutually authenticated channel.
  8. Whenever a secret is transferred, updated, or removed from a container the event is logged.
  9. The secret is only available to the consuming container. For additional security, the secret can decrypted using the container’s private key.

Security requirements for effective secrets management

And finally, the requirements themselves:

  • Access control: Administrators should be able to granularly control who (users) and what (services) can access secrets, and what type of access they have (read, write, delete, list).
  • Write only: Except for the consumer of the secret, no one should be able to read it. Even the user that created the secret has no reason to read it.
  • Rotation: A secret should be rotated on a regular basis (e.g. every 90 days). When a secret is changed, the change should propagate to all containers that are currently using the secret.
  • Revocation: In case a secret is exposed, or is no longer needed, it should be possible not only to delete the secret from storage (and all containers using it), but also to revoke the access that a secret grants to a resource.
  • Channel protection: Secrets should only be transmitted over a secured communication channel that ensures the privacy, integrity, and identity of the communicating parties. Usually this is accomplished by using encrypted communication channels that are mutually authenticated.
  • Readable only in container: A secret should only be exposable in the context of its consuming container. That is, only processes running inside the container should have access to the secret. It should not be trivial for processes running outside the container to expose the secret (while it is impossible to fully protect the secret from the underlying host, it should not be “easy”).
  • Administration audit: All administrative access and attempted access to secrets should be logged (creation, revocation, rotation, reading, listing).
  • Usage audit: Log when a secret is injected, rotated, or removed from a container.

It’s hard enough to manage secrets in enterprise environments. Keeping secrets secret in container-based applications is exponentially more difficult, but the situation is far from hopeless. DevOps and DevSecOps are shaping a culture within enterprise development shops that welcomes the support of security teams and acknowledges the benefit of automating security controls into the CI/CD process. For many in security (including myself), this is an opportunity to correct longstanding, structural application security challenges – including secrets management.

So don’t just sit there… forward this post to anyone you think should see it!

This article is published as part of the IDG Contributor Network. Want to Join?