Hashicorp Vault integration wth Spectro Cloud kubernetes management

Kubernetes has gained exploding popularity over the last few years, and many organizations have started migrating their application workloads. If your application has sensitive information of any sort, then you’ll need a way to help manage those secrets in Kubernetes. HashiCorp’s Vault helps secure, store, and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets, and other sensitive data.

In this blog, we’ll talk about Vault components and the integration use cases with Spectro Cloud.

Let’s start with a brief overview of HashiCorp’s Vault components:

• Vault Server is the core component which provides an API through which clients can interact with and manage the interaction between all the secrets engines, ACL enforcement, and secret lease revocation
• Vault Agent is a client daemon that helps authenticate to the Vault server and perform token lifecycle management
• Agent injector is a Kubernetes Mutating Admission Webhook. This controller intercepts pod CREATE and UPDATE requests looking for the metadata annotation vault.hashicorp.com/agent-inject: true in the requests. When found, the controller will alter the pod specification based on additional vault specific annotations present. Two types of agent containers will be injected to the application deployments:

 Init container will pre-populate the shared memory volume with the requested secrets prior to the other containers starting  

Sidecar container will continue to authenticate and render secrets to the same location where the pod runs

Now, we’ll walk through the following Vault integration use cases supported in Spectro Cloud

1. Deploying Vault on a Kubernetes cluster,
2. Integrating your Kubernetes cluster to an external Vault,
3. Performing secrets injection via sidecar

Pre-requisites

You’ll need

• Access to a Spectro Cloud project
• Understanding of Vault components
• Kubernetes CLI access

Deploying Vault on a Kubernetes cluster

For deploying the Vault server on a Kubernetes cluster provisioned by Spectro Cloud, follow the steps below:

1. Spectro Cloud provides a declarative model of Kubernetes infrastructure layers, called a cluster profile. When creating your cluster profile, select Security layer > Vault integration
2. Choose the desired version of the Vault helm chart to deploy on the cluster
3. Make changes to the chart values to suit your needs
4. Finish the cluster profile
5. For new clusters, choose the profile which has Vault integration in the cluster provisioning wizard
6. For existing clusters, once you add Vault integration, your cluster will show an update notification. Apply the notification for Vault components to be deployed on the cluster

The default values.yaml is pre-configured to install the following components:

• Vault Server — Standalone mode
• Vault Agent

 Note: Above method is recommended for non-production use cases, where you want to deploy Vault on the Kubernetes cluster to understand the features. By default, this integration deploys a standalone vault server with a file storage backend. This is a less secure and less resilient installation that is NOT appropriate for a production setup. For production use cases, refer to Kubernetes reference architecture

Integrating to an external Vault server

Some enterprises might already be using the Vault server for secrets management. To integrate the Kubernetes clusters provisioned by Spectro Cloud to an external Vault, follow the steps below:

1. On the cluster profile, go to Security > Vault pack
2. Choose the desired version of the Vault helm chart to be deployed on the cluster
3. In the Chart values.yaml, set charts.vault.injector.externalVaultAddr to point to the external Vault server
4. Finish the cluster profile
5. For new clusters, choose the profile which has Vault integration in the cluster provisioning wizard
6. For existing clusters, once you add Vault integration, your cluster will show an update notification. Apply the notification for the Vault agent to be deployed on the cluster

 Note: This method will skip installing Vault server in the Kubernetes cluster and will deploy Vault agent and related components

Performing secrets injection via sidecar

To configure secret injection using annotations, following are the pre-requisites:

1. Kubernetes cluster is setup with Vault agent (using one of the two use cases described above)
2. Kubernetes auth method should be configured and enabled in Vault
3. Desired secrets exist in Vault
4. Application deployments should have a service account
5. Service account should be bound to a Vault role with a policy enabling access to desired secrets
6. Application deployments should supply one or more vault secret annotations

Steps on how to setup the above pre-requisites is described in detail in the Vault official blog

You can set Annotations on cluster profile for each of the packs. When the pack gets deployed on the cluster, here is what an annotation will look like

The above annotation, when applied to a deployment, instructs the vault agent to:

1. Connect to Vault server
2. Fetch the Kubernetes secret defined in secrets/wordpress
3. Populate the file /vault/secrets/config with the templated data above
4. Use the role named wordpress

At this point, a new secret file will be created in the application container under /vault/secrets/config. Behind the scenes, Vault’s mutating webhook controller intercepted the API request and changed the pod definition to include sidecar containers to help with Vault authentication and token lifecycle management.

References:

• https://www.vaultproject.io/docs/platform/k8s/helm/run
• https://www.vaultproject.io/docs/platform/k8s/injector
• https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar