Configure Kubernetes secrets management with External Secrets Operator and HashiCorp Vault

Advanced 75 min Jul 27, 2026 289 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Learn how to deploy External Secrets Operator on Kubernetes, integrate it with HashiCorp Vault using the Kubernetes auth method, and sync secrets via SecretStore and ClusterSecretStore resources with production-grade RBAC and monitoring.

Prerequisites

  • Running Kubernetes cluster with kubectl access
  • Helm 3 installed
  • HashiCorp Vault server reachable from the cluster with an admin token
  • Cluster-admin access to configure RBAC and CRDs

What this solves

Storing secrets as plain Kubernetes Secret objects means they sit base64-encoded in etcd with no rotation, no audit trail, and no central policy control. External Secrets Operator (ESO) bridges Kubernetes to HashiCorp Vault, pulling secrets on a schedule and materializing them as native Secret objects that your pods consume normally.

This tutorial covers a production-ready setup: Vault as the backend, ESO deployed with Helm, Kubernetes auth method for authentication, namespace-scoped SecretStore and cluster-wide ClusterSecretStore resources, templating, rotation, RBAC isolation, and troubleshooting.

Prerequisites and architecture overview

You need a running Kubernetes cluster (kubeadm, EKS, GKE, or AKS all work) with kubectl configured, Helm 3 installed, and a Vault server reachable from the cluster network. If you have not installed Vault yet, see install and configure Vault for secrets management with high availability first.

The data flow is: ESO controller pods authenticate to Vault using a Kubernetes service account token, Vault validates that token against the Kubernetes API, Vault issues a scoped token tied to a policy, and ESO uses that token to read secrets and write them into Kubernetes Secret objects referenced by your workloads.

Note: This tutorial assumes Vault is already unsealed and reachable at a stable address. Production Vault should run with auto-unseal and TLS, not the dev server mode shown for the local backend example.

Step-by-step configuration

Install the Vault CLI and verify connectivity

The Vault CLI lets you configure policies and auth methods from your workstation or a bastion host.

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y vault
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y vault
export VAULT_ADDR="https://vault.example.com:8200"
vault status

Enable the KV v2 secrets engine

KV version 2 supports versioning, which lets you roll back accidental overwrites of application secrets.

vault login
vault secrets enable -path=kv-eso kv-v2
vault kv put kv-eso/production/api-service DB_PASSWORD="Str0ng-P@ss-2024!" API_KEY="ak_live_9f3c7d2a1b"

Enable the Kubernetes auth method in Vault

This lets Vault verify tokens presented by pods against your cluster's API server, without storing static credentials anywhere.

vault auth enable kubernetes

Retrieve the cluster CA and API endpoint, then configure Vault to talk to your cluster.

kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > /tmp/k8s-ca.crt
KUBE_HOST=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')

vault write auth/kubernetes/config \
  kubernetes_host="$KUBE_HOST" \
  kubernetes_ca_cert=@/tmp/k8s-ca.crt

Create a least-privilege Vault policy

Scope the policy to only the paths ESO needs to read. Never grant blanket access to the entire KV mount.

path "kv-eso/data/production/*" {
  capabilities = ["read"]
}

path "kv-eso/metadata/production/*" {
  capabilities = ["list", "read"]
}
vault policy write eso-production-read vault-eso-policy.hcl

Create a Kubernetes auth role bound to a service account

Binding the Vault role to a specific service account and namespace enforces that only ESO pods in that namespace can assume this role, which is the core of namespace isolation.

vault write auth/kubernetes/role/eso-production-role \
  bound_service_account_names=external-secrets-sa \
  bound_service_account_namespaces=external-secrets \
  policies=eso-production-read \
  ttl=15m

Deploy External Secrets Operator with Helm

ESO ships an official Helm chart that installs the controller, webhook, and CRDs. Deploy it into a dedicated namespace to keep it isolated from application workloads.

kubectl create namespace external-secrets
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --set installCRDs=true \
  --set replicaCount=2 \
  --set serviceAccount.name=external-secrets-sa

Two replicas give you a controller failover path if a pod is evicted or the node drains during maintenance.

Verify the operator is running

Confirm the controller, webhook, and cert-controller pods reach Running state before creating any secret store resources.

kubectl get pods -n external-secrets
kubectl get crd | grep external-secrets.io

Create a ClusterSecretStore for cluster-wide access

A ClusterSecretStore is not namespaced and can be referenced by ExternalSecret resources across the cluster, useful for shared credentials like registry pull secrets or platform-wide TLS certs.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "kv-eso"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-production-role"
          serviceAccountRef:
            name: "external-secrets-sa"
            namespace: "external-secrets"
kubectl apply -f manifests/cluster-secret-store.yaml
kubectl get clustersecretstore vault-backend -o wide

Create a namespace-scoped SecretStore

Use a SecretStore instead of a ClusterSecretStore when a team should only sync secrets within its own namespace, reinforcing the isolation established by the RBAC and Vault role bindings above.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend-app
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "kv-eso"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-production-role"
          serviceAccountRef:
            name: "external-secrets-sa"
            namespace: "production"
kubectl create namespace production
kubectl apply -f manifests/app-secret-store.yaml

Define an ExternalSecret to sync into a Kubernetes Secret

The ExternalSecret is the resource that actually maps Vault paths to keys inside a generated Kubernetes Secret. The refreshInterval controls how often ESO re-reads Vault.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-service-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend-app
    kind: SecretStore
  target:
    name: api-service-secrets
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: production/api-service
        property: DB_PASSWORD
    - secretKey: API_KEY
      remoteRef:
        key: production/api-service
        property: API_KEY
kubectl apply -f manifests/api-service-external-secret.yaml
kubectl get externalsecret api-service-secrets -n production
kubectl get secret api-service-secrets -n production -o jsonpath='{.data}'

Use templating to reshape synced secrets

Templating lets you build connection strings or config file formats directly from multiple Vault values instead of requiring the application to assemble them at runtime.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-service-connection
  namespace: production
spec:
  refreshInterval: 30m
  secretStoreRef:
    name: vault-backend-app
    kind: SecretStore
  target:
    name: api-service-connection
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgres://app_user:{{ .DB_PASSWORD }}@postgres-primary.production.svc.cluster.local:5432/appdb"
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: production/api-service
        property: DB_PASSWORD

Configure RBAC to restrict who can read synced Secrets and ExternalSecrets

ESO synchronizing Vault into a namespace does not automatically restrict which cluster users can read that Secret. Apply RBAC in the target namespace so only the application's service account and approved operators can view it.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["api-service-secrets", "api-service-connection"]
    verbs: ["get"]
kubectl apply -f manifests/secret-reader-role.yaml
kubectl create rolebinding api-service-secret-reader \
  --role=secret-reader \
  --serviceaccount=production:api-service-sa \
  -n production

For a deeper walkthrough of scoping cluster roles and service accounts, see configure Kubernetes RBAC with service accounts and cluster roles.

Verify your setup

Confirm ESO successfully authenticated to Vault, synced the secret, and that the resulting Kubernetes Secret has the expected keys.

kubectl describe externalsecret api-service-secrets -n production
kubectl get events -n production --field-selector involvedObject.name=api-service-secrets
kubectl logs -n external-secrets deployment/external-secrets --tail=50

The Status.Conditions field in the ExternalSecret describe output should show Ready: True with the reason SecretSynced. If it shows an error, the message usually names the exact Vault path or permission problem.

Warning: Never store raw Vault root tokens in CI/CD pipelines or ConfigMaps to bootstrap this integration. Use short-lived Kubernetes service account tokens through the auth method configured above, and rotate the Vault policy or role immediately if a token is ever exposed.

Secret rotation and refresh behavior

ESO does not push changes, it polls. When you update a value in Vault with vault kv put, the change appears in the Kubernetes Secret only after the next refreshInterval elapses. Set shorter intervals (5-15m) for credentials that rotate frequently, and longer intervals (1h or more) for stable static config to reduce load on Vault.

Secret typeSuggested refreshIntervalReason
Database passwords (static)1hRotated manually or via scheduled job, low churn
Dynamic database creds15mVault dynamic secrets expire on a TTL, needs frequent renewal
TLS certificates6hCerts typically valid for weeks or months
API keys for third-party services30mBalance between freshness and Vault request volume

Applications that read environment variables at startup only will not see rotated secrets until the pod restarts. Pair ESO with a rollout mechanism, such as the Reloader controller or a CI/CD hook, to restart deployments when the underlying Secret changes.

Production hardening and high availability considerations

Run Vault itself in HA mode with Raft integrated storage or Consul as the storage backend, and enable auto-unseal so Vault comes back online automatically after a node restart without manual intervention. See configure Vault auto-unseal with AWS KMS for a working setup.

Run at least two ESO controller replicas across separate nodes using pod anti-affinity, and set resource requests and limits so the controller is not evicted under memory pressure. Enable the ESO Prometheus metrics endpoint and scrape externalsecret_sync_calls_total and externalsecret_sync_calls_error to alert on sync failures before applications notice missing secrets.

kubectl get --raw /api/v1/namespaces/external-secrets/services/external-secrets-metrics:http-metrics/proxy/metrics | grep externalsecret_sync

Enforce network policies so only the ESO namespace can reach Vault's port over the network, reducing the blast radius if a workload pod is compromised. If you are also running Argo CD, review integrate ArgoCD with External Secrets Operator to keep SecretStore and ExternalSecret manifests under GitOps control without committing plaintext values.

Common issues

SymptomCauseFix
ExternalSecret stuck in SecretSyncedErrorVault role not bound to the correct service account or namespaceCheck vault read auth/kubernetes/role/eso-production-role matches the SecretStore's serviceAccountRef
permission denied reading secretVault policy path does not match KV mount path exactlyRemember KV v2 requires kv-eso/data/<path> in the policy, not just kv-eso/<path>
Secret never updates after Vault changerefreshInterval has not elapsed yetLower refreshInterval or manually force reconcile with kubectl annotate externalsecret api-service-secrets force-sync=$(date +%s) -n production --overwrite
x509 certificate signed by unknown authorityVault server uses a certificate not trusted by cluster nodesMount the CA bundle into the ESO pod via caBundle in the SecretStore provider config
ClusterSecretStore works but SecretStore fails in one namespacebound_service_account_namespaces in Vault role restricts to a different namespaceCreate a separate Vault role per namespace or widen the bound namespaces list deliberately

Next steps

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: Vault unseal drills, ESO controller upgrades, policy audits, and alerting when a sync silently fails. Talk to an engineer if you'd rather hand the ops side off.

不想自己管理这些吗?

我们为依赖稳定运行时间的企业管理基础设施。全托管服务,配备一位熟悉您系统架构的固定联系人。

您将拥有一位了解您整体架构的固定联系人

Rotterdam 02:40 · 一条消息即可联系我们,无需填写工单表单