Implement Consul Connect mTLS with Vault PKI backend for secure service mesh communication

Advanced 45 min Apr 22, 2026 13 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Set up mutual TLS authentication for Consul Connect using HashiCorp Vault's PKI backend to secure service-to-service communication with automatic certificate management and rotation.

Prerequisites

  • Root access to Linux server
  • Basic understanding of PKI concepts
  • Familiarity with HashiCorp tools
  • At least 2GB RAM available

What this solves

Consul Connect provides a secure service mesh with mutual TLS (mTLS) authentication between services, but managing certificates manually becomes complex at scale. This tutorial configures HashiCorp Vault as the PKI backend for Consul Connect, enabling automatic certificate provisioning, rotation, and centralized certificate authority management for your service mesh infrastructure.

Step-by-step implementation

Install Vault and Consul

Install both HashiCorp Vault and Consul on your system. We'll use the official HashiCorp repositories for the latest stable versions.

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y vault consul
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y vault consul

Configure Vault server

Create the Vault configuration file with storage backend and listener settings. This configuration enables the Vault API and web interface.

storage "file" {
  path = "/var/lib/vault/data"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = true
}

api_addr = "http://127.0.0.1:8200"
cluster_addr = "http://127.0.0.1:8201"
ui = true
disable_mlock = true

Create Vault data directory and start service

Create the storage directory with proper permissions and start the Vault service.

sudo mkdir -p /var/lib/vault/data
sudo chown vault:vault /var/lib/vault/data
sudo chmod 755 /var/lib/vault/data
sudo systemctl enable --now vault
sudo systemctl status vault

Initialize and unseal Vault

Initialize Vault to create the master keys and root token, then unseal it for operation.

export VAULT_ADDR='http://127.0.0.1:8200'
vault operator init -key-shares=5 -key-threshold=3 > /tmp/vault-init.txt
cat /tmp/vault-init.txt
Important: Save the unseal keys and root token securely. You'll need three unseal keys to unseal Vault after restarts.
vault operator unseal [UNSEAL_KEY_1]
vault operator unseal [UNSEAL_KEY_2]
vault operator unseal [UNSEAL_KEY_3]

Authenticate and enable PKI engine

Login with the root token and enable the PKI secrets engine that will manage certificates for Consul Connect.

vault auth [ROOT_TOKEN]
vault secrets enable -path=connect-ca pki
vault secrets tune -max-lease-ttl=87600h connect-ca

Generate root certificate authority

Create the root CA certificate that will sign all service certificates in your Consul Connect mesh.

vault write connect-ca/root/generate/internal \
  common_name="Consul Connect CA" \
  ttl=87600h

Configure PKI role for Consul Connect

Create a role that defines the certificate parameters for Consul Connect services.

vault write connect-ca/roles/consul-connect \
  allowed_domains="consul" \
  allow_subdomains=true \
  max_ttl=72h \
  generate_lease=true

Create Vault policy for Consul

Define a policy that grants Consul the necessary permissions to manage certificates.

path "connect-ca/cert/ca" {
  capabilities = ["read"]
}

path "connect-ca/root/sign-intermediate" {
  capabilities = ["update"]
}

path "connect-ca/intermediate/set-signed" {
  capabilities = ["update"]
}
vault policy write consul-connect /tmp/consul-connect-policy.hcl

Create Vault token for Consul

Generate a token that Consul will use to authenticate with Vault for certificate operations.

vault token create -policy=consul-connect -ttl=24h -renewable
Note: Save this token securely. Consul will use it to communicate with Vault for certificate management.

Configure Consul server

Create the Consul configuration with Connect enabled and Vault PKI integration.

datacenter = "dc1"
data_dir = "/var/lib/consul"
log_level = "INFO"
server = true
bootstrap_expect = 1
bind_addr = "127.0.0.1"
client_addr = "0.0.0.0"
ui_config {
  enabled = true
}

connect {
  enabled = true
  ca_provider = "vault"
  ca_config {
    address = "http://127.0.0.1:8200"
    token = "[CONSUL_VAULT_TOKEN]"
    root_pki_path = "connect-ca/"
    intermediate_pki_path = "connect-intermediate/"
  }
}

ports {
  grpc = 8502
}

Enable intermediate PKI engine for Consul

Create a separate intermediate PKI path that Consul will use for certificate signing.

vault secrets enable -path=connect-intermediate pki
vault secrets tune -max-lease-ttl=43800h connect-intermediate

Update Consul policy with intermediate permissions

Extend the Consul policy to include permissions for the intermediate CA operations.

path "connect-ca/cert/ca" {
  capabilities = ["read"]
}

path "connect-ca/root/sign-intermediate" {
  capabilities = ["update"]
}

path "connect-intermediate/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "connect-intermediate/cert/ca" {
  capabilities = ["read"]
}

path "connect-intermediate/issue/*" {
  capabilities = ["create", "update"]
}

path "connect-intermediate/sign/*" {
  capabilities = ["create", "update"]
}
vault policy write consul-connect /tmp/consul-connect-policy.hcl

Start Consul service

Create the Consul data directory and start the service with the new configuration.

sudo mkdir -p /var/lib/consul
sudo chown consul:consul /var/lib/consul
sudo chmod 755 /var/lib/consul
sudo systemctl enable --now consul
sudo systemctl status consul

Configure service registration

Create a sample service configuration to demonstrate mTLS functionality. This registers a web service with Connect enabled.

{
  "service": {
    "name": "web",
    "id": "web-1",
    "port": 8080,
    "connect": {
      "sidecar_service": {
        "port": 20000,
        "proxy": {
          "upstreams": [
            {
              "destination_name": "api",
              "local_bind_port": 9191
            }
          ]
        }
      }
    }
  }
}

Register API service with Connect

Register a second service to demonstrate service-to-service mTLS communication.

{
  "service": {
    "name": "api",
    "id": "api-1",
    "port": 8090,
    "connect": {
      "sidecar_service": {
        "port": 20001
      }
    }
  }
}

Reload Consul configuration

Reload Consul to register the new services and apply the Connect configuration.

sudo systemctl reload consul
consul members
consul catalog services

Configure certificate rotation policy

Set up automatic certificate rotation by configuring the intermediate CA with shorter TTL values.

vault write connect-intermediate/roles/consul-connect \
  allowed_domains="consul,spiffe" \
  allow_subdomains=true \
  allow_any_name=true \
  max_ttl=24h \
  ttl=1h \
  generate_lease=true

Configure monitoring and alerting

Enable Consul metrics

Configure Consul to expose metrics for monitoring certificate operations and Connect health.

{
  "telemetry": {
    "prometheus_retention_time": "24h",
    "disable_hostname": true
  },
  "ports": {
    "http": 8500,
    "grpc": 8502
  }
}
sudo systemctl reload consul

Monitor certificate expiration

Create a script to monitor certificate expiration and send alerts when certificates are near expiry.

#!/bin/bash

Check certificate expiration for Consul Connect

CONSUL_ADDR="http://127.0.0.1:8500" VAULT_ADDR="http://127.0.0.1:8200"

Get CA certificate info

ca_info=$(curl -s $CONSUL_ADDR/v1/connect/ca/roots) echo "CA Certificate Status: $ca_info"

Check intermediate CA status

vault_ca_status=$(vault read -format=json connect-intermediate/cert/ca 2>/dev/null) if [ $? -eq 0 ]; then echo "Intermediate CA Status: Active" else echo "Intermediate CA Status: Error - Check Vault connection" exit 1 fi echo "Certificate monitoring check completed at $(date)"
sudo chmod +x /usr/local/bin/check-consul-certs.sh
sudo /usr/local/bin/check-consul-certs.sh

Verify your setup

# Check Vault status and CA
vault status
vault read connect-ca/cert/ca

Verify Consul Connect CA configuration

consul connect ca get-config consul connect ca roots

Check service registrations

consul catalog services consul catalog nodes

Test certificate generation

vault write connect-intermediate/issue/consul-connect common_name="test.service.consul" ttl=1h
Integration tip: For comprehensive monitoring of your PKI infrastructure, consider setting up Prometheus monitoring to track certificate expiration and Vault health metrics.

Common issues

SymptomCauseFix
Consul fails to start with CA errorsVault token expired or invalidGenerate new token: vault token create -policy=consul-connect
Services can't obtain certificatesIntermediate PKI path not configuredEnable intermediate engine: vault secrets enable -path=connect-intermediate pki
Certificate rotation failsInsufficient Vault permissionsUpdate policy with intermediate path permissions
Connect proxy fails to startGRPC port not enabledAdd "grpc": 8502 to Consul ports configuration
CA root verification errorsRoot CA not properly generatedRegenerate root CA: vault write connect-ca/root/generate/internal

Next steps

Running this in production?

Need this managed for you? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. See how we run infrastructure like this for European teams.

Automated install script

Run this to automate the entire setup

Need help?

Don't want to manage this yourself?

We handle infrastructure security hardening for businesses that depend on uptime. From initial setup to ongoing operations.