Learn how to deploy External Secrets Operator on Kubernetes, integrate it with HashiCorp Vault using the Kubernetes auth method, and sync secrets via SecretStore and ClusterSecretStore resources with production-grade RBAC and monitoring.
Prerequisites
- Running Kubernetes cluster with kubectl access
- Helm 3 installed
- HashiCorp Vault server reachable from the cluster with an admin token
- Cluster-admin access to configure RBAC and CRDs
What this solves
Storing secrets as plain Kubernetes Secret objects means they sit base64-encoded in etcd with no rotation, no audit trail, and no central policy control. External Secrets Operator (ESO) bridges Kubernetes to HashiCorp Vault, pulling secrets on a schedule and materializing them as native Secret objects that your pods consume normally.
This tutorial covers a production-ready setup: Vault as the backend, ESO deployed with Helm, Kubernetes auth method for authentication, namespace-scoped SecretStore and cluster-wide ClusterSecretStore resources, templating, rotation, RBAC isolation, and troubleshooting.
Prerequisites and architecture overview
You need a running Kubernetes cluster (kubeadm, EKS, GKE, or AKS all work) with kubectl configured, Helm 3 installed, and a Vault server reachable from the cluster network. If you have not installed Vault yet, see install and configure Vault for secrets management with high availability first.
The data flow is: ESO controller pods authenticate to Vault using a Kubernetes service account token, Vault validates that token against the Kubernetes API, Vault issues a scoped token tied to a policy, and ESO uses that token to read secrets and write them into Kubernetes Secret objects referenced by your workloads.
Step-by-step configuration
Install the Vault CLI and verify connectivity
The Vault CLI lets you configure policies and auth methods from your workstation or a bastion host.
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y vaultsudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y vaultexport VAULT_ADDR="https://vault.example.com:8200"
vault statusEnable the KV v2 secrets engine
KV version 2 supports versioning, which lets you roll back accidental overwrites of application secrets.
vault login
vault secrets enable -path=kv-eso kv-v2
vault kv put kv-eso/production/api-service DB_PASSWORD="Str0ng-P@ss-2024!" API_KEY="ak_live_9f3c7d2a1b"Enable the Kubernetes auth method in Vault
This lets Vault verify tokens presented by pods against your cluster's API server, without storing static credentials anywhere.
vault auth enable kubernetesRetrieve the cluster CA and API endpoint, then configure Vault to talk to your cluster.
kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > /tmp/k8s-ca.crt
KUBE_HOST=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')
vault write auth/kubernetes/config \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert=@/tmp/k8s-ca.crtCreate a least-privilege Vault policy
Scope the policy to only the paths ESO needs to read. Never grant blanket access to the entire KV mount.
path "kv-eso/data/production/*" {
capabilities = ["read"]
}
path "kv-eso/metadata/production/*" {
capabilities = ["list", "read"]
}vault policy write eso-production-read vault-eso-policy.hclCreate a Kubernetes auth role bound to a service account
Binding the Vault role to a specific service account and namespace enforces that only ESO pods in that namespace can assume this role, which is the core of namespace isolation.
vault write auth/kubernetes/role/eso-production-role \
bound_service_account_names=external-secrets-sa \
bound_service_account_namespaces=external-secrets \
policies=eso-production-read \
ttl=15mDeploy External Secrets Operator with Helm
ESO ships an official Helm chart that installs the controller, webhook, and CRDs. Deploy it into a dedicated namespace to keep it isolated from application workloads.
kubectl create namespace external-secrets
helm repo add external-secrets https://charts.external-secrets.io
helm repo updatehelm install external-secrets external-secrets/external-secrets \
--namespace external-secrets \
--set installCRDs=true \
--set replicaCount=2 \
--set serviceAccount.name=external-secrets-saTwo replicas give you a controller failover path if a pod is evicted or the node drains during maintenance.
Verify the operator is running
Confirm the controller, webhook, and cert-controller pods reach Running state before creating any secret store resources.
kubectl get pods -n external-secrets
kubectl get crd | grep external-secrets.ioCreate a ClusterSecretStore for cluster-wide access
A ClusterSecretStore is not namespaced and can be referenced by ExternalSecret resources across the cluster, useful for shared credentials like registry pull secrets or platform-wide TLS certs.
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "kv-eso"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "eso-production-role"
serviceAccountRef:
name: "external-secrets-sa"
namespace: "external-secrets"kubectl apply -f manifests/cluster-secret-store.yaml
kubectl get clustersecretstore vault-backend -o wideCreate a namespace-scoped SecretStore
Use a SecretStore instead of a ClusterSecretStore when a team should only sync secrets within its own namespace, reinforcing the isolation established by the RBAC and Vault role bindings above.
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend-app
namespace: production
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "kv-eso"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "eso-production-role"
serviceAccountRef:
name: "external-secrets-sa"
namespace: "production"kubectl create namespace production
kubectl apply -f manifests/app-secret-store.yamlDefine an ExternalSecret to sync into a Kubernetes Secret
The ExternalSecret is the resource that actually maps Vault paths to keys inside a generated Kubernetes Secret. The refreshInterval controls how often ESO re-reads Vault.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-service-secrets
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend-app
kind: SecretStore
target:
name: api-service-secrets
creationPolicy: Owner
data:
- secretKey: DB_PASSWORD
remoteRef:
key: production/api-service
property: DB_PASSWORD
- secretKey: API_KEY
remoteRef:
key: production/api-service
property: API_KEYkubectl apply -f manifests/api-service-external-secret.yaml
kubectl get externalsecret api-service-secrets -n production
kubectl get secret api-service-secrets -n production -o jsonpath='{.data}'Use templating to reshape synced secrets
Templating lets you build connection strings or config file formats directly from multiple Vault values instead of requiring the application to assemble them at runtime.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-service-connection
namespace: production
spec:
refreshInterval: 30m
secretStoreRef:
name: vault-backend-app
kind: SecretStore
target:
name: api-service-connection
creationPolicy: Owner
template:
engineVersion: v2
data:
DATABASE_URL: "postgres://app_user:{{ .DB_PASSWORD }}@postgres-primary.production.svc.cluster.local:5432/appdb"
data:
- secretKey: DB_PASSWORD
remoteRef:
key: production/api-service
property: DB_PASSWORDConfigure RBAC to restrict who can read synced Secrets and ExternalSecrets
ESO synchronizing Vault into a namespace does not automatically restrict which cluster users can read that Secret. Apply RBAC in the target namespace so only the application's service account and approved operators can view it.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: production
name: secret-reader
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["api-service-secrets", "api-service-connection"]
verbs: ["get"]kubectl apply -f manifests/secret-reader-role.yaml
kubectl create rolebinding api-service-secret-reader \
--role=secret-reader \
--serviceaccount=production:api-service-sa \
-n productionFor a deeper walkthrough of scoping cluster roles and service accounts, see configure Kubernetes RBAC with service accounts and cluster roles.
Verify your setup
Confirm ESO successfully authenticated to Vault, synced the secret, and that the resulting Kubernetes Secret has the expected keys.
kubectl describe externalsecret api-service-secrets -n production
kubectl get events -n production --field-selector involvedObject.name=api-service-secrets
kubectl logs -n external-secrets deployment/external-secrets --tail=50The Status.Conditions field in the ExternalSecret describe output should show Ready: True with the reason SecretSynced. If it shows an error, the message usually names the exact Vault path or permission problem.
Secret rotation and refresh behavior
ESO does not push changes, it polls. When you update a value in Vault with vault kv put, the change appears in the Kubernetes Secret only after the next refreshInterval elapses. Set shorter intervals (5-15m) for credentials that rotate frequently, and longer intervals (1h or more) for stable static config to reduce load on Vault.
| Secret type | Suggested refreshInterval | Reason |
|---|---|---|
| Database passwords (static) | 1h | Rotated manually or via scheduled job, low churn |
| Dynamic database creds | 15m | Vault dynamic secrets expire on a TTL, needs frequent renewal |
| TLS certificates | 6h | Certs typically valid for weeks or months |
| API keys for third-party services | 30m | Balance between freshness and Vault request volume |
Applications that read environment variables at startup only will not see rotated secrets until the pod restarts. Pair ESO with a rollout mechanism, such as the Reloader controller or a CI/CD hook, to restart deployments when the underlying Secret changes.
Production hardening and high availability considerations
Run Vault itself in HA mode with Raft integrated storage or Consul as the storage backend, and enable auto-unseal so Vault comes back online automatically after a node restart without manual intervention. See configure Vault auto-unseal with AWS KMS for a working setup.
Run at least two ESO controller replicas across separate nodes using pod anti-affinity, and set resource requests and limits so the controller is not evicted under memory pressure. Enable the ESO Prometheus metrics endpoint and scrape externalsecret_sync_calls_total and externalsecret_sync_calls_error to alert on sync failures before applications notice missing secrets.
kubectl get --raw /api/v1/namespaces/external-secrets/services/external-secrets-metrics:http-metrics/proxy/metrics | grep externalsecret_syncEnforce network policies so only the ESO namespace can reach Vault's port over the network, reducing the blast radius if a workload pod is compromised. If you are also running Argo CD, review integrate ArgoCD with External Secrets Operator to keep SecretStore and ExternalSecret manifests under GitOps control without committing plaintext values.
Common issues
| Symptom | Cause | Fix |
|---|---|---|
| ExternalSecret stuck in SecretSyncedError | Vault role not bound to the correct service account or namespace | Check vault read auth/kubernetes/role/eso-production-role matches the SecretStore's serviceAccountRef |
| permission denied reading secret | Vault policy path does not match KV mount path exactly | Remember KV v2 requires kv-eso/data/<path> in the policy, not just kv-eso/<path> |
| Secret never updates after Vault change | refreshInterval has not elapsed yet | Lower refreshInterval or manually force reconcile with kubectl annotate externalsecret api-service-secrets force-sync=$(date +%s) -n production --overwrite |
| x509 certificate signed by unknown authority | Vault server uses a certificate not trusted by cluster nodes | Mount the CA bundle into the ESO pod via caBundle in the SecretStore provider config |
| ClusterSecretStore works but SecretStore fails in one namespace | bound_service_account_namespaces in Vault role restricts to a different namespace | Create a separate Vault role per namespace or widen the bound namespaces list deliberately |
Next steps
- Configure Vault dynamic secrets for databases with PostgreSQL and MySQL integration
- Set up Vault as a PKI certificate authority with SSL automation and intermediate CA
- Implement Kubernetes secrets management with HashiCorp Vault integration
- Configure Kubernetes secrets management with Sealed Secrets for secure Helm values
- Monitor Kubernetes network policies with Prometheus and Grafana for enhanced cluster security