Set up mutual TLS authentication for Consul Connect using HashiCorp Vault's PKI backend to secure service-to-service communication with automatic certificate management and rotation.
Prerequisites
- Root access to Linux server
- Basic understanding of PKI concepts
- Familiarity with HashiCorp tools
- At least 2GB RAM available
What this solves
Consul Connect provides a secure service mesh with mutual TLS (mTLS) authentication between services, but managing certificates manually becomes complex at scale. This tutorial configures HashiCorp Vault as the PKI backend for Consul Connect, enabling automatic certificate provisioning, rotation, and centralized certificate authority management for your service mesh infrastructure.
Step-by-step implementation
Install Vault and Consul
Install both HashiCorp Vault and Consul on your system. We'll use the official HashiCorp repositories for the latest stable versions.
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y vault consul
Configure Vault server
Create the Vault configuration file with storage backend and listener settings. This configuration enables the Vault API and web interface.
storage "file" {
path = "/var/lib/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
}
api_addr = "http://127.0.0.1:8200"
cluster_addr = "http://127.0.0.1:8201"
ui = true
disable_mlock = true
Create Vault data directory and start service
Create the storage directory with proper permissions and start the Vault service.
sudo mkdir -p /var/lib/vault/data
sudo chown vault:vault /var/lib/vault/data
sudo chmod 755 /var/lib/vault/data
sudo systemctl enable --now vault
sudo systemctl status vault
Initialize and unseal Vault
Initialize Vault to create the master keys and root token, then unseal it for operation.
export VAULT_ADDR='http://127.0.0.1:8200'
vault operator init -key-shares=5 -key-threshold=3 > /tmp/vault-init.txt
cat /tmp/vault-init.txt
vault operator unseal [UNSEAL_KEY_1]
vault operator unseal [UNSEAL_KEY_2]
vault operator unseal [UNSEAL_KEY_3]
Authenticate and enable PKI engine
Login with the root token and enable the PKI secrets engine that will manage certificates for Consul Connect.
vault auth [ROOT_TOKEN]
vault secrets enable -path=connect-ca pki
vault secrets tune -max-lease-ttl=87600h connect-ca
Generate root certificate authority
Create the root CA certificate that will sign all service certificates in your Consul Connect mesh.
vault write connect-ca/root/generate/internal \
common_name="Consul Connect CA" \
ttl=87600h
Configure PKI role for Consul Connect
Create a role that defines the certificate parameters for Consul Connect services.
vault write connect-ca/roles/consul-connect \
allowed_domains="consul" \
allow_subdomains=true \
max_ttl=72h \
generate_lease=true
Create Vault policy for Consul
Define a policy that grants Consul the necessary permissions to manage certificates.
path "connect-ca/cert/ca" {
capabilities = ["read"]
}
path "connect-ca/root/sign-intermediate" {
capabilities = ["update"]
}
path "connect-ca/intermediate/set-signed" {
capabilities = ["update"]
}
vault policy write consul-connect /tmp/consul-connect-policy.hcl
Create Vault token for Consul
Generate a token that Consul will use to authenticate with Vault for certificate operations.
vault token create -policy=consul-connect -ttl=24h -renewable
Configure Consul server
Create the Consul configuration with Connect enabled and Vault PKI integration.
datacenter = "dc1"
data_dir = "/var/lib/consul"
log_level = "INFO"
server = true
bootstrap_expect = 1
bind_addr = "127.0.0.1"
client_addr = "0.0.0.0"
ui_config {
enabled = true
}
connect {
enabled = true
ca_provider = "vault"
ca_config {
address = "http://127.0.0.1:8200"
token = "[CONSUL_VAULT_TOKEN]"
root_pki_path = "connect-ca/"
intermediate_pki_path = "connect-intermediate/"
}
}
ports {
grpc = 8502
}
Enable intermediate PKI engine for Consul
Create a separate intermediate PKI path that Consul will use for certificate signing.
vault secrets enable -path=connect-intermediate pki
vault secrets tune -max-lease-ttl=43800h connect-intermediate
Update Consul policy with intermediate permissions
Extend the Consul policy to include permissions for the intermediate CA operations.
path "connect-ca/cert/ca" {
capabilities = ["read"]
}
path "connect-ca/root/sign-intermediate" {
capabilities = ["update"]
}
path "connect-intermediate/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
path "connect-intermediate/cert/ca" {
capabilities = ["read"]
}
path "connect-intermediate/issue/*" {
capabilities = ["create", "update"]
}
path "connect-intermediate/sign/*" {
capabilities = ["create", "update"]
}
vault policy write consul-connect /tmp/consul-connect-policy.hcl
Start Consul service
Create the Consul data directory and start the service with the new configuration.
sudo mkdir -p /var/lib/consul
sudo chown consul:consul /var/lib/consul
sudo chmod 755 /var/lib/consul
sudo systemctl enable --now consul
sudo systemctl status consul
Configure service registration
Create a sample service configuration to demonstrate mTLS functionality. This registers a web service with Connect enabled.
{
"service": {
"name": "web",
"id": "web-1",
"port": 8080,
"connect": {
"sidecar_service": {
"port": 20000,
"proxy": {
"upstreams": [
{
"destination_name": "api",
"local_bind_port": 9191
}
]
}
}
}
}
}
Register API service with Connect
Register a second service to demonstrate service-to-service mTLS communication.
{
"service": {
"name": "api",
"id": "api-1",
"port": 8090,
"connect": {
"sidecar_service": {
"port": 20001
}
}
}
}
Reload Consul configuration
Reload Consul to register the new services and apply the Connect configuration.
sudo systemctl reload consul
consul members
consul catalog services
Configure certificate rotation policy
Set up automatic certificate rotation by configuring the intermediate CA with shorter TTL values.
vault write connect-intermediate/roles/consul-connect \
allowed_domains="consul,spiffe" \
allow_subdomains=true \
allow_any_name=true \
max_ttl=24h \
ttl=1h \
generate_lease=true
Configure monitoring and alerting
Enable Consul metrics
Configure Consul to expose metrics for monitoring certificate operations and Connect health.
{
"telemetry": {
"prometheus_retention_time": "24h",
"disable_hostname": true
},
"ports": {
"http": 8500,
"grpc": 8502
}
}
sudo systemctl reload consul
Monitor certificate expiration
Create a script to monitor certificate expiration and send alerts when certificates are near expiry.
#!/bin/bash
Check certificate expiration for Consul Connect
CONSUL_ADDR="http://127.0.0.1:8500"
VAULT_ADDR="http://127.0.0.1:8200"
Get CA certificate info
ca_info=$(curl -s $CONSUL_ADDR/v1/connect/ca/roots)
echo "CA Certificate Status: $ca_info"
Check intermediate CA status
vault_ca_status=$(vault read -format=json connect-intermediate/cert/ca 2>/dev/null)
if [ $? -eq 0 ]; then
echo "Intermediate CA Status: Active"
else
echo "Intermediate CA Status: Error - Check Vault connection"
exit 1
fi
echo "Certificate monitoring check completed at $(date)"
sudo chmod +x /usr/local/bin/check-consul-certs.sh
sudo /usr/local/bin/check-consul-certs.sh
Verify your setup
# Check Vault status and CA
vault status
vault read connect-ca/cert/ca
Verify Consul Connect CA configuration
consul connect ca get-config
consul connect ca roots
Check service registrations
consul catalog services
consul catalog nodes
Test certificate generation
vault write connect-intermediate/issue/consul-connect common_name="test.service.consul" ttl=1h
Common issues
| Symptom | Cause | Fix |
|---|---|---|
| Consul fails to start with CA errors | Vault token expired or invalid | Generate new token: vault token create -policy=consul-connect |
| Services can't obtain certificates | Intermediate PKI path not configured | Enable intermediate engine: vault secrets enable -path=connect-intermediate pki |
| Certificate rotation fails | Insufficient Vault permissions | Update policy with intermediate path permissions |
| Connect proxy fails to start | GRPC port not enabled | Add "grpc": 8502 to Consul ports configuration |
| CA root verification errors | Root CA not properly generated | Regenerate root CA: vault write connect-ca/root/generate/internal |
Next steps
- Integrate Vault with Kubernetes for container-based service mesh deployments
- Set up Consul Template for dynamic configuration management
- Deploy Envoy proxies with Consul Connect for advanced traffic management
- Configure Vault clustering for production high availability
- Monitor your service mesh with comprehensive observability
Running this in production?
Automated install script
Run this to automate the entire setup
#!/usr/bin/env bash
set -euo pipefail
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
# Configuration variables
VAULT_TOKEN=""
CONSUL_TOKEN=""
DOMAIN="${1:-dc1.consul}"
# Cleanup function
cleanup() {
echo -e "${RED}[ERROR] Installation failed. Cleaning up...${NC}"
systemctl stop vault consul 2>/dev/null || true
rm -f /etc/vault.d/vault.hcl /etc/consul.d/consul.hcl 2>/dev/null || true
}
trap cleanup ERR
usage() {
echo "Usage: $0 [domain]"
echo " domain: Consul datacenter domain (default: dc1.consul)"
exit 1
}
# Check prerequisites
if [[ $EUID -ne 0 ]]; then
echo -e "${RED}This script must be run as root${NC}"
exit 1
fi
if ! command -v curl &> /dev/null; then
echo -e "${RED}curl is required but not installed${NC}"
exit 1
fi
# Auto-detect distribution
if [ -f /etc/os-release ]; then
. /etc/os-release
case "$ID" in
ubuntu|debian)
PKG_MGR="apt"
PKG_INSTALL="apt install -y"
PKG_UPDATE="apt update"
;;
almalinux|rocky|centos|rhel|ol|fedora)
PKG_MGR="dnf"
PKG_INSTALL="dnf install -y"
PKG_UPDATE="dnf update -y"
;;
amzn)
PKG_MGR="yum"
PKG_INSTALL="yum install -y"
PKG_UPDATE="yum update -y"
;;
*)
echo -e "${RED}Unsupported distribution: $ID${NC}"
exit 1
;;
esac
else
echo -e "${RED}Cannot detect distribution${NC}"
exit 1
fi
echo -e "${GREEN}[1/12] Installing HashiCorp repository...${NC}"
if [[ "$PKG_MGR" == "apt" ]]; then
curl -fsSL https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" > /etc/apt/sources.list.d/hashicorp.list
$PKG_UPDATE
else
$PKG_INSTALL dnf-plugins-core 2>/dev/null || $PKG_INSTALL yum-utils 2>/dev/null || true
if [[ "$PKG_MGR" == "dnf" ]]; then
dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
else
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
fi
fi
echo -e "${GREEN}[2/12] Installing Vault and Consul...${NC}"
$PKG_INSTALL vault consul
echo -e "${GREEN}[3/12] Creating directories and setting permissions...${NC}"
mkdir -p /var/lib/vault/data /var/lib/consul /etc/vault.d /etc/consul.d
chown vault:vault /var/lib/vault/data /etc/vault.d
chown consul:consul /var/lib/consul /etc/consul.d
chmod 755 /var/lib/vault/data /var/lib/consul /etc/vault.d /etc/consul.d
echo -e "${GREEN}[4/12] Configuring Vault...${NC}"
cat > /etc/vault.d/vault.hcl << 'EOF'
storage "file" {
path = "/var/lib/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
}
api_addr = "http://127.0.0.1:8200"
cluster_addr = "http://127.0.0.1:8201"
ui = true
disable_mlock = true
EOF
chown vault:vault /etc/vault.d/vault.hcl
chmod 644 /etc/vault.d/vault.hcl
echo -e "${GREEN}[5/12] Starting Vault service...${NC}"
systemctl enable vault
systemctl start vault
sleep 5
export VAULT_ADDR='http://127.0.0.1:8200'
echo -e "${GREEN}[6/12] Initializing Vault...${NC}"
vault operator init -key-shares=5 -key-threshold=3 > /tmp/vault-init.txt
chmod 600 /tmp/vault-init.txt
echo -e "${YELLOW}Vault initialization complete. Keys saved to /tmp/vault-init.txt${NC}"
echo -e "${YELLOW}IMPORTANT: Save these keys securely!${NC}"
UNSEAL_KEY_1=$(grep 'Unseal Key 1:' /tmp/vault-init.txt | awk '{print $4}')
UNSEAL_KEY_2=$(grep 'Unseal Key 2:' /tmp/vault-init.txt | awk '{print $4}')
UNSEAL_KEY_3=$(grep 'Unseal Key 3:' /tmp/vault-init.txt | awk '{print $4}')
ROOT_TOKEN=$(grep 'Initial Root Token:' /tmp/vault-init.txt | awk '{print $4}')
echo -e "${GREEN}[7/12] Unsealing Vault...${NC}"
vault operator unseal "$UNSEAL_KEY_1"
vault operator unseal "$UNSEAL_KEY_2"
vault operator unseal "$UNSEAL_KEY_3"
echo -e "${GREEN}[8/12] Configuring Vault PKI...${NC}"
vault auth "$ROOT_TOKEN"
vault secrets enable -path=connect-ca pki
vault secrets tune -max-lease-ttl=87600h connect-ca
vault write connect-ca/root/generate/internal \
common_name="Consul Connect CA" \
ttl=87600h
vault write connect-ca/roles/consul-connect \
allowed_domains="consul" \
allow_subdomains=true \
max_ttl=72h \
generate_lease=true
echo -e "${GREEN}[9/12] Creating Vault policy for Consul...${NC}"
cat > /tmp/consul-connect-policy.hcl << 'EOF'
path "connect-ca/cert/ca" {
capabilities = ["read"]
}
path "connect-ca/root/sign-intermediate" {
capabilities = ["update"]
}
path "connect-ca/intermediate/set-signed" {
capabilities = ["update"]
}
EOF
vault policy write consul-connect /tmp/consul-connect-policy.hcl
CONSUL_TOKEN=$(vault token create -policy=consul-connect -ttl=8760h -renewable -format=json | grep -o '"client_token":"[^"]*' | cut -d'"' -f4)
echo -e "${GREEN}[10/12] Configuring Consul...${NC}"
cat > /etc/consul.d/consul.hcl << EOF
datacenter = "dc1"
data_dir = "/var/lib/consul"
log_level = "INFO"
server = true
bootstrap_expect = 1
bind_addr = "127.0.0.1"
client_addr = "127.0.0.1"
connect {
enabled = true
ca_provider = "vault"
ca_config {
address = "http://127.0.0.1:8200"
token = "$CONSUL_TOKEN"
root_pki_path = "connect-ca"
intermediate_pki_path = "connect-intermediate"
}
}
ui_config {
enabled = true
}
ports {
grpc = 8502
}
EOF
chown consul:consul /etc/consul.d/consul.hcl
chmod 644 /etc/consul.d/consul.hcl
echo -e "${GREEN}[11/12] Starting Consul service...${NC}"
systemctl enable consul
systemctl start consul
sleep 10
echo -e "${GREEN}[12/12] Verifying installation...${NC}"
if ! systemctl is-active --quiet vault; then
echo -e "${RED}Vault service is not running${NC}"
exit 1
fi
if ! systemctl is-active --quiet consul; then
echo -e "${RED}Consul service is not running${NC}"
exit 1
fi
if ! vault status | grep -q "Sealed.*false"; then
echo -e "${RED}Vault is sealed${NC}"
exit 1
fi
if ! consul members | grep -q "alive"; then
echo -e "${RED}Consul cluster not healthy${NC}"
exit 1
fi
rm -f /tmp/consul-connect-policy.hcl
echo -e "${GREEN}Installation completed successfully!${NC}"
echo -e "${YELLOW}Important files:${NC}"
echo " - Vault keys: /tmp/vault-init.txt (secure this file!)"
echo " - Vault UI: http://localhost:8200"
echo " - Consul UI: http://localhost:8500"
echo ""
echo -e "${YELLOW}Next steps:${NC}"
echo " 1. Secure /tmp/vault-init.txt and remove it from this server"
echo " 2. Configure your services to use Consul Connect"
echo " 3. Set up proper TLS for Vault in production"
Review the script before running. Execute with: bash install.sh