Configure advanced Consul ACL policies for production security hardening

Advanced 35 min Apr 12, 2026 13 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Set up comprehensive Consul Access Control Lists with bootstrap tokens, role-based permissions, and secure inter-node communication for production environments. This tutorial covers ACL system initialization, policy creation, and agent token configuration.

Prerequisites

  • Root or sudo access
  • Basic understanding of Consul concepts
  • Network access for package installation
  • At least 2GB RAM for Consul server

What this solves

Consul's default configuration allows unrestricted access to all services and data, creating significant security vulnerabilities in production environments. This tutorial implements comprehensive Access Control Lists (ACLs) that restrict access based on roles and responsibilities, securing service discovery, key-value storage, and inter-node communication. You'll establish fine-grained permissions that follow the principle of least privilege while maintaining operational functionality.

Step-by-step configuration

Install and prepare Consul

Install Consul and create the necessary user and directory structure with proper permissions.

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y consul
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y consul
sudo useradd --system --home /etc/consul.d --shell /bin/false consul
sudo mkdir -p /opt/consul /etc/consul.d
sudo chown consul:consul /opt/consul /etc/consul.d
sudo chmod 755 /opt/consul /etc/consul.d

Generate encryption key and certificates

Create the encryption key for gossip protocol and generate TLS certificates for secure communication.

consul keygen

Save the output key for use in the configuration. Generate TLS certificates:

sudo mkdir -p /etc/consul.d/certs
cd /etc/consul.d/certs
sudo consul tls ca create
sudo consul tls cert create -server -dc dc1
sudo chown -R consul:consul /etc/consul.d/certs
sudo chmod 600 /etc/consul.d/certs/*.key
sudo chmod 644 /etc/consul.d/certs/*.pem

Configure Consul server with ACL support

Create the main Consul configuration file with ACL system enabled and TLS encryption.

datacenter = "dc1"
data_dir = "/opt/consul"
log_level = "INFO"
node_name = "consul-server-1"
bind_addr = "203.0.113.10"
client_addr = "0.0.0.0"

Server configuration

server = true bootstrap_expect = 1

UI configuration

ui_config { enabled = true }

ACL configuration

acl = { enabled = true default_policy = "deny" enable_token_persistence = true tokens = { initial_management = "BOOTSTRAP_TOKEN_PLACEHOLDER" } }

Encryption

encrypt = "YOUR_ENCRYPTION_KEY_HERE" encrypt_verify_incoming = true encrypt_verify_outgoing = true

TLS configuration

tls { defaults { ca_file = "/etc/consul.d/certs/consul-agent-ca.pem" cert_file = "/etc/consul.d/certs/dc1-server-consul-0.pem" key_file = "/etc/consul.d/certs/dc1-server-consul-0-key.pem" verify_incoming = true verify_outgoing = true } internal_rpc { verify_server_hostname = true } }

Performance

performance { raft_multiplier = 1 }

Logging

log_rotate_duration = "24h" log_rotate_max_files = 5

Replace YOUR_ENCRYPTION_KEY_HERE with the key generated in the previous step.

Set file permissions and start Consul

Secure the configuration file and start the Consul service.

sudo chown consul:consul /etc/consul.d/consul.hcl
sudo chmod 640 /etc/consul.d/consul.hcl
Never use chmod 777. It gives every user on the system full access to your files. Instead, fix ownership with chown and use minimal permissions.
sudo systemctl enable consul
sudo systemctl start consul
sudo systemctl status consul

Bootstrap the ACL system

Initialize the ACL system and create the initial management token.

consul acl bootstrap

Save the SecretID from the output as your management token. Update the configuration file with this token:

sudo sed -i 's/BOOTSTRAP_TOKEN_PLACEHOLDER/YOUR_ACTUAL_TOKEN_HERE/' /etc/consul.d/consul.hcl
sudo systemctl restart consul

Export the management token for subsequent commands:

export CONSUL_HTTP_TOKEN="YOUR_ACTUAL_TOKEN_HERE"
export CONSUL_HTTP_ADDR="https://203.0.113.10:8501"
export CONSUL_CACERT="/etc/consul.d/certs/consul-agent-ca.pem"

Create service discovery policies

Define policies for different service discovery roles with appropriate permissions.

service_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

key_prefix "services/" {
  policy = "read"
}
service_prefix "" {
  policy = "write"
}

node_prefix "" {
  policy = "write"
}

key_prefix "services/" {
  policy = "write"
}

session_prefix "" {
  policy = "write"
}
consul acl policy create -name "service-read" -description "Read-only access to services" -rules @/tmp/service-read-policy.hcl
consul acl policy create -name "service-write" -description "Write access to services" -rules @/tmp/service-write-policy.hcl

Create application-specific policies

Define granular policies for specific applications and environments.

service "webapp" {
  policy = "write"
}

service "webapp-sidecar-proxy" {
  policy = "write"
}

service_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

key_prefix "webapp/" {
  policy = "write"
}

key_prefix "config/webapp/" {
  policy = "read"
}
service "database" {
  policy = "write"
}

service "database-sidecar-proxy" {
  policy = "write"
}

node_prefix "" {
  policy = "read"
}

key_prefix "database/" {
  policy = "write"
}

key_prefix "config/database/" {
  policy = "read"
}

session_prefix "database/" {
  policy = "write"
}
consul acl policy create -name "webapp-access" -description "Webapp service access" -rules @/tmp/webapp-policy.hcl
consul acl policy create -name "database-access" -description "Database service access" -rules @/tmp/database-policy.hcl

Create node and agent policies

Define policies for Consul agents and node-level operations.

node_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "read"
}

agent_prefix "" {
  policy = "write"
}

key_prefix "_rexec" {
  policy = "write"
}
agent_prefix "" {
  policy = "write"
}

node_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "read"
}

key_prefix "agent/" {
  policy = "write"
}
consul acl policy create -name "node-access" -description "Node agent access" -rules @/tmp/node-policy.hcl
consul acl policy create -name "agent-access" -description "Agent operations" -rules @/tmp/agent-policy.hcl

Create ACL roles and tokens

Create roles that combine policies and generate tokens for different access levels.

consul acl role create -name "webapp-role" -description "Webapp service role" -policy-name "webapp-access" -policy-name "service-read"
consul acl role create -name "database-role" -description "Database service role" -policy-name "database-access" -policy-name "service-read"
consul acl role create -name "node-role" -description "Node agent role" -policy-name "node-access" -policy-name "agent-access"

Create tokens for each role:

consul acl token create -description "Webapp service token" -role-name "webapp-role"
consul acl token create -description "Database service token" -role-name "database-role"
consul acl token create -description "Node agent token" -role-name "node-role"

Configure agent tokens

Set up the default agent token and configure token persistence.

AGENT_TOKEN=$(consul acl token create -description "Default agent token" -role-name "node-role" -format="json" | jq -r '.SecretID')
consul acl set-agent-token default "$AGENT_TOKEN"

Update the Consul configuration to persist this token:

sudo cp /etc/consul.d/consul.hcl /etc/consul.d/consul.hcl.backup
sudo tee -a /etc/consul.d/consul.hcl > /dev/null <Agent tokens
acl {
  tokens {
    default = "$AGENT_TOKEN"
  }
}
EOF

Create monitoring and backup policies

Define policies for monitoring systems and backup operations.

service_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

key_prefix "" {
  policy = "read"
}

operator = "read"

session_prefix "" {
  policy = "read"
}
acl = "write"
key_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "read"
}
node_prefix "" {
  policy = "read"
}
operator = "read"
consul acl policy create -name "monitoring-access" -description "Monitoring system access" -rules @/tmp/monitoring-policy.hcl
consul acl policy create -name "backup-access" -description "Backup operations access" -rules @/tmp/backup-policy.hcl
consul acl token create -description "Monitoring token" -policy-name "monitoring-access"
consul acl token create -description "Backup token" -policy-name "backup-access"

Configure anonymous policy

Set up limited anonymous access for health checks and basic service discovery.

service_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

key_prefix "public/" {
  policy = "read"
}
consul acl policy create -name "anonymous-access" -description "Anonymous read access" -rules @/tmp/anonymous-policy.hcl
consul acl token update -id anonymous -policy-name "anonymous-access"

Enable audit logging

Configure audit logging to track ACL policy usage and access patterns.

sudo mkdir -p /var/log/consul
sudo chown consul:consul /var/log/consul
sudo chmod 755 /var/log/consul
sudo tee -a /etc/consul.d/consul.hcl > /dev/null <Audit logging
audit {
  enabled = true
  sink "file" {
    path = "/var/log/consul/audit.log"
    format = "json"
    mode = "blocking"
    rotate_duration = "24h"
    rotate_max_files = 7
    rotate_bytes = 104857600
  }
}
EOF
sudo systemctl restart consul

Verify your setup

Test the ACL configuration and verify that policies are working correctly.

consul acl policy list
consul acl role list
consul acl token list
consul members
consul catalog services

Test token-based access:

WEBAPP_TOKEN=$(consul acl token list | grep "Webapp service token" | awk '{print $1}')
CONSUL_HTTP_TOKEN="$WEBAPP_TOKEN" consul catalog services
CONSUL_HTTP_TOKEN="$WEBAPP_TOKEN" consul kv get services/webapp/config || echo "Access correctly denied"

Check audit logs:

sudo tail -f /var/log/consul/audit.log

Common issues

SymptomCauseFix
"Permission denied" errorsIncorrect or missing ACL tokenVerify token with consul acl token read -id TOKEN_ID
Services can't registerInsufficient service write permissionsAdd service write policy: service "name" { policy = "write" }
Consul UI access deniedManagement token not configuredSet management token in browser or use agent token
Nodes can't join clusterMissing node or agent permissionsEnsure agent token has node write permissions
Health checks failingAnonymous policy too restrictiveAdd service read permissions to anonymous policy
Backup operations failMissing ACL read permissions for backupUse token with acl = "write" permission

Next steps

Automated install script

Run this to automate the entire setup

Need help?

Don't want to manage this yourself?

We handle infrastructure security hardening for businesses that depend on uptime. From initial setup to ongoing operations.