Set up secure container secrets management by integrating Podman with HashiCorp Vault. Configure dynamic secret injection, automated rotation, and production-ready monitoring for containerized applications.
Prerequisites
- Root or sudo access
- Basic Podman knowledge
- Understanding of container security
- Network connectivity for package installation
What this solves
Container secrets management becomes critical when running production workloads that need database credentials, API keys, and certificates. This tutorial shows you how to integrate Podman with HashiCorp Vault to automatically inject secrets into containers, rotate credentials, and monitor access patterns without hardcoding sensitive data in images or environment variables.
Step-by-step configuration
Install HashiCorp Vault
Download and install the official Vault binary with GPG verification for secure package management.
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y vault
Configure Vault server
Create the Vault configuration file with file storage backend and API listener for development and testing environments.
sudo mkdir -p /opt/vault/data /etc/vault.d
sudo useradd --system --home /opt/vault --shell /bin/false vault
sudo chown -R vault:vault /opt/vault
ui = true
disable_mlock = true
storage "file" {
path = "/opt/vault/data"
}
listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = 1
}
api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
Create Vault systemd service
Set up Vault as a system service with proper security isolation and automatic startup configuration.
[Unit]
Description=HashiCorp Vault
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3
[Service]
Type=notify
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now vault
Initialize and unseal Vault
Initialize the Vault server and configure the root token for administrative access.
export VAULT_ADDR='http://127.0.0.1:8200'
vault operator init -key-shares=1 -key-threshold=1
vault operator unseal [UNSEAL_KEY_FROM_INIT]
export VAULT_TOKEN="[ROOT_TOKEN_FROM_INIT]"
Enable KV secrets engine
Enable the key-value secrets engine version 2 for storing application secrets with versioning support.
vault secrets enable -path=secret kv-v2
vault secrets list
Install Podman secrets provider
Install the vault-plugin-secrets-podman plugin to enable direct integration between Podman and Vault.
sudo apt install -y podman
wget https://github.com/containers/podman/releases/download/v4.8.0/podman-4.8.0-linux-amd64.tar.gz
tar -xzf podman-4.8.0-linux-amd64.tar.gz
Configure Podman secrets driver
Configure Podman to use Vault as the secrets driver for automatic secret injection into containers.
mkdir -p ~/.config/containers
[storage]
driver = "overlay"
runroot = "/run/user/1000/containers"
graphroot = "/home/user/.local/share/containers/storage"
[storage.options]
additionalimagestores = [
]
[storage.options.overlay]
mountopt = "nodev,metacopy=on"
Create Vault authentication policy
Set up an authentication policy and token for Podman to access secrets from Vault with minimal required permissions.
vault policy write podman-secrets - <
vault token create -policy=podman-secrets -ttl=24h
Store application secrets in Vault
Create sample application secrets in Vault that will be injected into containers at runtime.
vault kv put secret/app/database \n username="appuser" \n password="secure-db-password" \n host="db.example.com" \n port="5432"
vault kv put secret/app/api \n key="sk-1234567890abcdef" \n endpoint="https://api.example.com" \n timeout="30s"
Configure Vault agent for Podman
Set up Vault agent to automatically authenticate and provide secrets to Podman containers.
vault {
address = "http://127.0.0.1:8200"
}
auto_auth {
method "token_file" {
config = {
token_file_path = "/home/user/.vault-token"
}
}
sink "file" {
config = {
path = "/tmp/vault-token-via-agent"
}
}
}
template {
source = "/etc/vault-templates/database.tpl"
destination = "/tmp/database-config.json"
perms = 0644
}
template {
source = "/etc/vault-templates/api.tpl"
destination = "/tmp/api-config.json"
perms = 0644
}
Create secret templates
Define templates that format Vault secrets for consumption by your containerized applications.
sudo mkdir -p /etc/vault-templates
{{- with secret "secret/data/app/database" -}}
{
"username": "{{ .Data.data.username }}",
"password": "{{ .Data.data.password }}",
"host": "{{ .Data.data.host }}",
"port": "{{ .Data.data.port }}",
"connection_string": "postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@{{ .Data.data.host }}:{{ .Data.data.port }}/myapp"
}
{{- end -}}
{{- with secret "secret/data/app/api" -}}
{
"api_key": "{{ .Data.data.key }}",
"endpoint": "{{ .Data.data.endpoint }}",
"timeout": "{{ .Data.data.timeout }}"
}
{{- end -}}
Start Vault agent
Start the Vault agent to begin secret templating and automatic token renewal.
echo "[PODMAN_TOKEN_FROM_STEP_8]" > ~/.vault-token
vault agent -config=/etc/vault-agent.hcl &
Create container with Vault secrets
Run a Podman container that mounts the dynamically generated secret files from Vault.
podman run -d \n --name myapp \n --mount type=bind,source=/tmp/database-config.json,target=/app/database.json,readonly \n --mount type=bind,source=/tmp/api-config.json,target=/app/api.json,readonly \n --env-file /dev/null \n alpine:latest \n sh -c "while true; do cat /app/database.json; sleep 30; done"
Configure secret rotation
Set up automatic secret rotation using Vault's database secrets engine for dynamic credential generation.
vault secrets enable database
vault write database/config/postgresql \n plugin_name=postgresql-database-plugin \n connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/postgres?sslmode=disable" \n allowed_roles="readonly" \n username="vault-admin" \n password="admin-password"
vault write database/roles/readonly \n db_name=postgresql \n creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \n default_ttl="1h" \n max_ttl="24h"
Set up secret monitoring
Configure Vault audit logging and metrics collection to monitor secret access patterns and usage.
sudo mkdir -p /var/log/vault
sudo chown vault:vault /var/log/vault
vault audit enable file file_path=/var/log/vault/audit.log
vault write sys/metrics/config enabled=true
vault read sys/metrics/config
Create monitoring script
Set up a monitoring script to track secret usage and rotation status for operational visibility.
#!/bin/bash
set -euo pipefail
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN="$(cat ~/.vault-token)"
echo "=== Vault Status ==="
vault status
echo "=== Active Leases ==="
vault list sys/leases/lookup/database/creds/readonly || echo "No active database leases"
echo "=== Recent Audit Events ==="
tail -n 10 /var/log/vault/audit.log | jq -r '.time + " " + .request.operation + " " + .request.path'
echo "=== Token Info ==="
vault token lookup -format=json | jq -r '.data | "TTL: " + (.ttl|tostring) + "s, Policies: " + (.policies|join(","))'
echo "=== Secret Access Count (last hour) ==="
grep "$(date -d '1 hour ago' +'%Y-%m-%dT%H')" /var/log/vault/audit.log | grep '"secret/data/app"' | wc -l
sudo chmod +x /usr/local/bin/vault-monitor.sh
Configure systemd timer for monitoring
Set up automated monitoring checks using systemd timers for regular secret management health verification.
[Unit]
Description=Vault Monitoring Check
After=vault.service
Requires=vault.service
[Service]
Type=oneshot
User=vault
ExecStart=/usr/local/bin/vault-monitor.sh
StandardOutput=journal
StandardError=journal
[Unit]
Description=Run Vault monitoring every 15 minutes
Requires=vault-monitor.service
[Timer]
OnCalendar=*:0/15
Persistent=true
[Install]
WantedBy=timers.target
sudo systemctl daemon-reload
sudo systemctl enable --now vault-monitor.timer
Verify your setup
Test the complete secrets integration pipeline to ensure containers receive secrets from Vault correctly.
vault status
sudo systemctl status vault
vault kv get secret/app/database
podman logs myapp
cat /tmp/database-config.json
ls -la /tmp/vault-token-via-agent
/usr/local/bin/vault-monitor.sh
sudo systemctl status vault-monitor.timer
sudo journalctl -u vault-monitor.service -n 20
Common issues
| Symptom | Cause | Fix |
|---|---|---|
| Vault agent fails to start | Invalid token or permissions | Check token with vault token lookup and verify policy assignment |
| Templates not rendering | Secret path doesn't exist | Verify secret exists with vault kv get secret/app/database |
| Container can't read secrets | File permissions or mount issues | Check file ownership ls -la /tmp/*.json and mount syntax |
| Vault sealed after restart | Auto-unseal not configured | Run vault operator unseal [KEY] or configure auto-unseal |
| Audit log not writing | Directory permissions issue | Verify chown vault:vault /var/log/vault and log rotation setup |
| Token expired | TTL reached without renewal | Create new token with vault token create -policy=podman-secrets |
Next steps
- Configure Vault dynamic secrets for databases with PostgreSQL and MySQL integration
- Set up Vault as a PKI certificate authority with SSL automation and intermediate CA
- Secure Podman containers with SELinux and AppArmor mandatory access controls
- Configure Podman image scanning with Trivy security vulnerability detection
- Configure Vault high availability clustering for production environments
Running this in production?
Automated install script
Run this to automate the entire setup
#!/usr/bin/env bash
set -euo pipefail
# Colors for output
readonly RED='\033[0;31m'
readonly GREEN='\033[0;32m'
readonly YELLOW='\033[1;33m'
readonly NC='\033[0m'
# Script configuration
readonly SCRIPT_NAME="$(basename "$0")"
readonly VAULT_CONFIG_DIR="/etc/vault.d"
readonly VAULT_DATA_DIR="/opt/vault/data"
readonly VAULT_USER="vault"
readonly VAULT_ADDR="http://127.0.0.1:8200"
# Global variables
VAULT_UNSEAL_KEY=""
VAULT_ROOT_TOKEN=""
PKG_INSTALL=""
# Usage message
usage() {
cat << EOF
Usage: $SCRIPT_NAME [OPTIONS]
Configure Podman secrets management with HashiCorp Vault integration
OPTIONS:
-h, --help Show this help message
--vault-addr ADDR Vault server address (default: $VAULT_ADDR)
Examples:
$SCRIPT_NAME
$SCRIPT_NAME --vault-addr http://10.0.0.100:8200
EOF
}
# Logging functions
log_info() { echo -e "${GREEN}[INFO]${NC} $1" >&2; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1" >&2; }
log_error() { echo -e "${RED}[ERROR]${NC} $1" >&2; }
# Cleanup function
cleanup() {
local exit_code=$?
if [ $exit_code -ne 0 ]; then
log_error "Installation failed. Cleaning up..."
systemctl stop vault 2>/dev/null || true
systemctl disable vault 2>/dev/null || true
userdel "$VAULT_USER" 2>/dev/null || true
rm -rf "$VAULT_DATA_DIR" "$VAULT_CONFIG_DIR" 2>/dev/null || true
fi
exit $exit_code
}
trap cleanup EXIT ERR
# Parse arguments
parse_args() {
while [[ $# -gt 0 ]]; do
case $1 in
-h|--help)
usage
exit 0
;;
--vault-addr)
VAULT_ADDR="$2"
shift 2
;;
*)
log_error "Unknown option: $1"
usage
exit 1
;;
esac
done
}
# Check prerequisites
check_prerequisites() {
echo "[1/8] Checking prerequisites..."
if [ "$EUID" -ne 0 ]; then
log_error "This script must be run as root"
exit 1
fi
# Detect distribution
if [ ! -f /etc/os-release ]; then
log_error "/etc/os-release not found. Cannot detect distribution."
exit 1
fi
source /etc/os-release
case "$ID" in
ubuntu|debian)
PKG_INSTALL="apt-get install -y"
;;
almalinux|rocky|centos|rhel|ol|fedora)
PKG_INSTALL="dnf install -y"
command -v dnf >/dev/null 2>&1 || PKG_INSTALL="yum install -y"
;;
amzn)
PKG_INSTALL="yum install -y"
;;
*)
log_error "Unsupported distribution: $ID"
exit 1
;;
esac
log_info "Detected distribution: $ID"
}
# Install HashiCorp Vault
install_vault() {
echo "[2/8] Installing HashiCorp Vault..."
case "$ID" in
ubuntu|debian)
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" > /etc/apt/sources.list.d/hashicorp.list
apt-get update
$PKG_INSTALL vault
;;
almalinux|rocky|centos|rhel|ol|fedora|amzn)
if command -v dnf >/dev/null 2>&1; then
dnf install -y dnf-plugins-core
dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
else
yum install -y yum-utils
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
fi
$PKG_INSTALL vault
;;
esac
log_info "Vault installed successfully"
}
# Configure Vault server
configure_vault() {
echo "[3/8] Configuring Vault server..."
# Create directories and user
mkdir -p "$VAULT_DATA_DIR" "$VAULT_CONFIG_DIR"
useradd --system --home /opt/vault --shell /bin/false "$VAULT_USER" 2>/dev/null || true
chown -R "$VAULT_USER:$VAULT_USER" /opt/vault
# Create Vault configuration
cat > "$VAULT_CONFIG_DIR/vault.hcl" << 'EOF'
ui = true
disable_mlock = true
storage "file" {
path = "/opt/vault/data"
}
listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = 1
}
api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
EOF
chmod 644 "$VAULT_CONFIG_DIR/vault.hcl"
chown root:root "$VAULT_CONFIG_DIR/vault.hcl"
log_info "Vault configuration created"
}
# Create Vault systemd service
create_vault_service() {
echo "[4/8] Creating Vault systemd service..."
cat > /etc/systemd/system/vault.service << 'EOF'
[Unit]
Description=HashiCorp Vault
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3
[Service]
Type=notify
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable vault
systemctl start vault
# Wait for Vault to start
sleep 5
log_info "Vault service created and started"
}
# Initialize and unseal Vault
initialize_vault() {
echo "[5/8] Initializing and unsealing Vault..."
export VAULT_ADDR="$VAULT_ADDR"
# Initialize Vault
local init_output
init_output=$(vault operator init -key-shares=1 -key-threshold=1 -format=json)
VAULT_UNSEAL_KEY=$(echo "$init_output" | grep -o '"unseal_keys_b64":\["[^"]*"' | cut -d'"' -f4)
VAULT_ROOT_TOKEN=$(echo "$init_output" | grep -o '"root_token":"[^"]*"' | cut -d'"' -f4)
# Unseal Vault
vault operator unseal "$VAULT_UNSEAL_KEY"
export VAULT_TOKEN="$VAULT_ROOT_TOKEN"
log_info "Vault initialized and unsealed"
log_warn "IMPORTANT: Save these credentials securely!"
echo "Unseal Key: $VAULT_UNSEAL_KEY"
echo "Root Token: $VAULT_ROOT_TOKEN"
}
# Configure KV secrets engine
configure_kv_secrets() {
echo "[6/8] Configuring KV secrets engine..."
export VAULT_ADDR="$VAULT_ADDR"
export VAULT_TOKEN="$VAULT_ROOT_TOKEN"
vault secrets enable -path=secret kv-v2
log_info "KV secrets engine enabled"
}
# Install Podman
install_podman() {
echo "[7/8] Installing Podman..."
case "$ID" in
ubuntu|debian)
$PKG_INSTALL podman
;;
almalinux|rocky|centos|rhel|ol|fedora|amzn)
$PKG_INSTALL podman
;;
esac
log_info "Podman installed successfully"
}
# Verify installation
verify_installation() {
echo "[8/8] Verifying installation..."
# Check Vault status
if systemctl is-active --quiet vault; then
log_info "✓ Vault service is running"
else
log_error "✗ Vault service is not running"
return 1
fi
# Check Vault API
export VAULT_ADDR="$VAULT_ADDR"
export VAULT_TOKEN="$VAULT_ROOT_TOKEN"
if vault status >/dev/null 2>&1; then
log_info "✓ Vault API is accessible"
else
log_error "✗ Vault API is not accessible"
return 1
fi
# Check Podman
if command -v podman >/dev/null 2>&1; then
log_info "✓ Podman is installed"
else
log_error "✗ Podman is not installed"
return 1
fi
log_info "Installation completed successfully!"
echo
echo "Next steps:"
echo "1. Export VAULT_ADDR=$VAULT_ADDR"
echo "2. Export VAULT_TOKEN=$VAULT_ROOT_TOKEN"
echo "3. Create secrets with: vault kv put secret/myapp username=admin password=secret"
echo "4. Configure Podman containers to use Vault secrets"
}
# Main function
main() {
parse_args "$@"
check_prerequisites
install_vault
configure_vault
create_vault_service
initialize_vault
configure_kv_secrets
install_podman
verify_installation
}
main "$@"
Review the script before running. Execute with: bash install.sh