Implement Kubernetes secrets management with HashiCorp Vault integration

Install HashiCorp Vault

First install Vault on your system to configure the initial setup and policies.

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y vault

sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y vault

Install kubectl and Helm

Install Kubernetes tools needed to deploy and manage the Vault operator.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Start Vault in development mode

For this tutorial, start Vault in dev mode to focus on the Kubernetes integration. In production, use a proper Vault cluster.

vault server -dev -dev-root-token-id=root -dev-listen-address=0.0.0.0:8200 &
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'
echo 'export VAULT_ADDR="http://127.0.0.1:8200"' >> ~/.bashrc
echo 'export VAULT_TOKEN="root"' >> ~/.bashrc

Enable Vault secrets engines

Enable the key-value secrets engine and database secrets engine for dynamic secret generation.

vault secrets enable -path=secret kv-v2
vault secrets enable -path=database database

Create example secrets

Create some test secrets that will be injected into Kubernetes pods.

vault kv put secret/myapp username="appuser" password="supersecret123"
vault kv put secret/database host="db.example.com" port="5432" database="myapp"

Enable Kubernetes authentication

Configure Vault to accept authentication from Kubernetes service accounts.

vault auth enable kubernetes

Get Kubernetes cluster information

Extract the information Vault needs to communicate with your Kubernetes cluster.

export VAULT_SA_NAME=$(kubectl get sa vault -o jsonpath="{.secrets[*]['name']}" -n default)
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" -n default | base64 --decode; echo)
export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" -n default | base64 --decode; echo)
export K8S_HOST=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')

Create Vault service account in Kubernetes

Create a service account that Vault will use to verify other service accounts.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
kind: ServiceAccount
  name: vault
  namespace: default

kubectl apply -f vault-sa.yaml

Configure Kubernetes authentication in Vault

Tell Vault how to connect to Kubernetes and verify service account tokens.

vault write auth/kubernetes/config \
    token_reviewer_jwt="$SA_JWT_TOKEN" \
    kubernetes_host="$K8S_HOST" \
    kubernetes_ca_cert="$SA_CA_CRT" \
    issuer="https://kubernetes.default.svc.cluster.local"

Create Vault policies

Define which secrets each application can access through specific policies.

path "secret/data/myapp" {
  capabilities = ["read"]
}

path "secret/data/database" {
  capabilities = ["read"]
}

path "database/creds/myapp-role" {
  capabilities = ["read"]
}

vault policy write myapp-policy myapp-policy.hcl

Create Kubernetes authentication role

Bind Kubernetes service accounts to Vault policies through authentication roles.

vault write auth/kubernetes/role/myapp \
    bound_service_account_names=myapp \
    bound_service_account_namespaces=default \
    policies=myapp-policy \
    ttl=24h

Add HashiCorp Helm repository

Add the official HashiCorp Helm repository to install the Vault Secrets Operator.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Create namespace for Vault operator

Create a dedicated namespace for the Vault Secrets Operator components.

kubectl create namespace vault-secrets-operator-system

Install Vault Secrets Operator

Deploy the operator using Helm with custom values for your environment.

controller:
  manager:
    image:
      repository: hashicorp/vault-secrets-operator
      tag: 0.4.3
    resources:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 10m
        memory: 64Mi

defaultVaultConnection:
  enabled: true
  address: "http://HOST_IP:8200"
  skipTLSVerify: true

Update values with your host IP

Replace HOST_IP with your actual machine IP address so pods can reach Vault.

export HOST_IP=$(ip route get 1 | awk '{print $7}' | head -1)
sed -i "s/HOST_IP/$HOST_IP/g" vso-values.yaml

Deploy the operator

Install the Vault Secrets Operator with your custom configuration.

helm install vault-secrets-operator hashicorp/vault-secrets-operator \
    -n vault-secrets-operator-system \
    -f vso-values.yaml \
    --version 0.4.3

Verify operator deployment

Check that all operator components are running correctly.

kubectl get pods -n vault-secrets-operator-system
kubectl logs -n vault-secrets-operator-system deployment/vault-secrets-operator-controller-manager

Create VaultConnection resource

Configure how the operator connects to your Vault instance.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: default
  name: vault-connection
spec:
  address: "http://HOST_IP:8200"
  skipTLSVerify: true

sed "s/HOST_IP/$HOST_IP/g" vault-connection.yaml | kubectl apply -f -

Create VaultAuth resource

Configure authentication between the operator and Vault using Kubernetes service accounts.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  namespace: default
  name: vault-auth
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: myapp
    serviceAccount: myapp

kubectl apply -f vault-auth.yaml

Create application service account

Create the service account that your application pods will use to authenticate with Vault.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp
  namespace: default

kubectl apply -f myapp-sa.yaml

Create VaultStaticSecret for KV secrets

Define which static secrets should be synced from Vault into Kubernetes secrets.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: default
  name: myapp-secret
spec:
  vaultAuthRef: vault-auth
  mount: secret
  type: kv-v2
  path: myapp
  destination:
    name: myapp-secret
    create: true
  refreshAfter: 30s

kubectl apply -f vault-static-secret.yaml

Create VaultDynamicSecret for database credentials

Configure dynamic database credentials that are generated on-demand by Vault.

#!/bin/bash
First configure a database connection in Vault
vault write database/config/postgresql \
    plugin_name=postgresql-database-plugin \
    connection_url="postgresql://{{username}}:{{password}}@localhost:5432/myapp?sslmode=disable" \
    allowed_roles="myapp-role" \
    username="postgres" \
    password="rootpassword"

Create a role for generating dynamic credentials
vault write database/roles/myapp-role \
    db_name=postgresql \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"\
    default_ttl="1h" \
    max_ttl="24h"

chmod +x database-config.sh
Run this only if you have a PostgreSQL database configured
./database-config.sh

Create example application deployment

Deploy a test application that uses the secrets injected by the Vault Secrets Operator.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      serviceAccountName: myapp
      containers:
      - name: myapp
        image: nginx:latest
        env:
        - name: APP_USERNAME
          valueFrom:
            secretKeyRef:
              name: myapp-secret
              key: username
        - name: APP_PASSWORD
          valueFrom:
            secretKeyRef:
              name: myapp-secret
              key: password
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: myapp-secret
              key: host
              optional: true
        volumeMounts:
        - name: secret-volume
          mountPath: /etc/secrets
          readOnly: true
      volumes:
      - name: secret-volume
        secret:
          secretName: myapp-secret

kubectl apply -f myapp-deployment.yaml

Create application with init container pattern

Use an init container to fetch secrets before the main application starts, useful for applications that need secrets at startup.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-init
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp-init
  template:
    metadata:
      labels:
        app: myapp-init
    spec:
      serviceAccountName: myapp
      initContainers:
      - name: vault-init
        image: hashicorp/vault:1.15.2
        command: ['/bin/sh']
        args:
        - -c
        - |
          export VAULT_ADDR=http://HOST_IP:8200
          export VAULT_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
          vault auth -method=kubernetes role=myapp
          vault kv get -field=username secret/myapp > /shared/username
          vault kv get -field=password secret/myapp > /shared/password
          echo "Secrets fetched successfully"
        volumeMounts:
        - name: shared-data
          mountPath: /shared
      containers:
      - name: myapp
        image: nginx:latest
        command: ['/bin/sh']
        args:
        - -c
        - |
          echo "Username: $(cat /shared/username)"
          echo "Password: $(cat /shared/password)"
          nginx -g 'daemon off;'
        volumeMounts:
        - name: shared-data
          mountPath: /shared
      volumes:
      - name: shared-data
        emptyDir: {}

sed "s/HOST_IP/$HOST_IP/g" myapp-init-deployment.yaml | kubectl apply -f -

Check Vault Secrets Operator status

Verify that the operator is successfully syncing secrets from Vault.

kubectl get vaultstaticsecrets -n default
kubectl describe vaultstaticsecret myapp-secret -n default

Verify Kubernetes secrets creation

Check that secrets have been created in Kubernetes with the expected data.

kubectl get secrets myapp-secret -o yaml
kubectl get secret myapp-secret -o jsonpath='{.data.username}' | base64 -d

Test application secret access

Verify that your application pods can access the injected secrets.

kubectl exec deployment/myapp -- env | grep APP_
kubectl exec deployment/myapp -- cat /etc/secrets/username
kubectl exec deployment/myapp -- cat /etc/secrets/password

Monitor operator logs

Check operator logs to ensure secrets are being synced without errors.

kubectl logs -n vault-secrets-operator-system deployment/vault-secrets-operator-controller-manager -f

Test secret rotation

Update a secret in Vault and verify it gets synchronized to Kubernetes.

vault kv put secret/myapp username="newuser" password="newsecret456"
Wait 30 seconds for refresh
sleep 30
kubectl get secret myapp-secret -o jsonpath='{.data.username}' | base64 -d