Set up comprehensive Consul Access Control Lists with bootstrap tokens, role-based permissions, and secure inter-node communication for production environments. This tutorial covers ACL system initialization, policy creation, and agent token configuration.
Prerequisites
- Root or sudo access
- Basic understanding of Consul concepts
- Network access for package installation
- At least 2GB RAM for Consul server
What this solves
Consul's default configuration allows unrestricted access to all services and data, creating significant security vulnerabilities in production environments. This tutorial implements comprehensive Access Control Lists (ACLs) that restrict access based on roles and responsibilities, securing service discovery, key-value storage, and inter-node communication. You'll establish fine-grained permissions that follow the principle of least privilege while maintaining operational functionality.
Step-by-step configuration
Install and prepare Consul
Install Consul and create the necessary user and directory structure with proper permissions.
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install -y consul
sudo useradd --system --home /etc/consul.d --shell /bin/false consul
sudo mkdir -p /opt/consul /etc/consul.d
sudo chown consul:consul /opt/consul /etc/consul.d
sudo chmod 755 /opt/consul /etc/consul.d
Generate encryption key and certificates
Create the encryption key for gossip protocol and generate TLS certificates for secure communication.
consul keygen
Save the output key for use in the configuration. Generate TLS certificates:
sudo mkdir -p /etc/consul.d/certs
cd /etc/consul.d/certs
sudo consul tls ca create
sudo consul tls cert create -server -dc dc1
sudo chown -R consul:consul /etc/consul.d/certs
sudo chmod 600 /etc/consul.d/certs/*.key
sudo chmod 644 /etc/consul.d/certs/*.pem
Configure Consul server with ACL support
Create the main Consul configuration file with ACL system enabled and TLS encryption.
datacenter = "dc1"
data_dir = "/opt/consul"
log_level = "INFO"
node_name = "consul-server-1"
bind_addr = "203.0.113.10"
client_addr = "0.0.0.0"
Server configuration
server = true
bootstrap_expect = 1
UI configuration
ui_config {
enabled = true
}
ACL configuration
acl = {
enabled = true
default_policy = "deny"
enable_token_persistence = true
tokens = {
initial_management = "BOOTSTRAP_TOKEN_PLACEHOLDER"
}
}
Encryption
encrypt = "YOUR_ENCRYPTION_KEY_HERE"
encrypt_verify_incoming = true
encrypt_verify_outgoing = true
TLS configuration
tls {
defaults {
ca_file = "/etc/consul.d/certs/consul-agent-ca.pem"
cert_file = "/etc/consul.d/certs/dc1-server-consul-0.pem"
key_file = "/etc/consul.d/certs/dc1-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
}
internal_rpc {
verify_server_hostname = true
}
}
Performance
performance {
raft_multiplier = 1
}
Logging
log_rotate_duration = "24h"
log_rotate_max_files = 5
Replace YOUR_ENCRYPTION_KEY_HERE with the key generated in the previous step.
Set file permissions and start Consul
Secure the configuration file and start the Consul service.
sudo chown consul:consul /etc/consul.d/consul.hcl
sudo chmod 640 /etc/consul.d/consul.hcl
sudo systemctl enable consul
sudo systemctl start consul
sudo systemctl status consul
Bootstrap the ACL system
Initialize the ACL system and create the initial management token.
consul acl bootstrap
Save the SecretID from the output as your management token. Update the configuration file with this token:
sudo sed -i 's/BOOTSTRAP_TOKEN_PLACEHOLDER/YOUR_ACTUAL_TOKEN_HERE/' /etc/consul.d/consul.hcl
sudo systemctl restart consul
Export the management token for subsequent commands:
export CONSUL_HTTP_TOKEN="YOUR_ACTUAL_TOKEN_HERE"
export CONSUL_HTTP_ADDR="https://203.0.113.10:8501"
export CONSUL_CACERT="/etc/consul.d/certs/consul-agent-ca.pem"
Create service discovery policies
Define policies for different service discovery roles with appropriate permissions.
service_prefix "" {
policy = "read"
}
node_prefix "" {
policy = "read"
}
key_prefix "services/" {
policy = "read"
}
service_prefix "" {
policy = "write"
}
node_prefix "" {
policy = "write"
}
key_prefix "services/" {
policy = "write"
}
session_prefix "" {
policy = "write"
}
consul acl policy create -name "service-read" -description "Read-only access to services" -rules @/tmp/service-read-policy.hcl
consul acl policy create -name "service-write" -description "Write access to services" -rules @/tmp/service-write-policy.hcl
Create application-specific policies
Define granular policies for specific applications and environments.
service "webapp" {
policy = "write"
}
service "webapp-sidecar-proxy" {
policy = "write"
}
service_prefix "" {
policy = "read"
}
node_prefix "" {
policy = "read"
}
key_prefix "webapp/" {
policy = "write"
}
key_prefix "config/webapp/" {
policy = "read"
}
service "database" {
policy = "write"
}
service "database-sidecar-proxy" {
policy = "write"
}
node_prefix "" {
policy = "read"
}
key_prefix "database/" {
policy = "write"
}
key_prefix "config/database/" {
policy = "read"
}
session_prefix "database/" {
policy = "write"
}
consul acl policy create -name "webapp-access" -description "Webapp service access" -rules @/tmp/webapp-policy.hcl
consul acl policy create -name "database-access" -description "Database service access" -rules @/tmp/database-policy.hcl
Create node and agent policies
Define policies for Consul agents and node-level operations.
node_prefix "" {
policy = "write"
}
service_prefix "" {
policy = "read"
}
agent_prefix "" {
policy = "write"
}
key_prefix "_rexec" {
policy = "write"
}
agent_prefix "" {
policy = "write"
}
node_prefix "" {
policy = "write"
}
service_prefix "" {
policy = "read"
}
key_prefix "agent/" {
policy = "write"
}
consul acl policy create -name "node-access" -description "Node agent access" -rules @/tmp/node-policy.hcl
consul acl policy create -name "agent-access" -description "Agent operations" -rules @/tmp/agent-policy.hcl
Create ACL roles and tokens
Create roles that combine policies and generate tokens for different access levels.
consul acl role create -name "webapp-role" -description "Webapp service role" -policy-name "webapp-access" -policy-name "service-read"
consul acl role create -name "database-role" -description "Database service role" -policy-name "database-access" -policy-name "service-read"
consul acl role create -name "node-role" -description "Node agent role" -policy-name "node-access" -policy-name "agent-access"
Create tokens for each role:
consul acl token create -description "Webapp service token" -role-name "webapp-role"
consul acl token create -description "Database service token" -role-name "database-role"
consul acl token create -description "Node agent token" -role-name "node-role"
Configure agent tokens
Set up the default agent token and configure token persistence.
AGENT_TOKEN=$(consul acl token create -description "Default agent token" -role-name "node-role" -format="json" | jq -r '.SecretID')
consul acl set-agent-token default "$AGENT_TOKEN"
Update the Consul configuration to persist this token:
sudo cp /etc/consul.d/consul.hcl /etc/consul.d/consul.hcl.backup
sudo tee -a /etc/consul.d/consul.hcl > /dev/null <Agent tokens
acl {
tokens {
default = "$AGENT_TOKEN"
}
}
EOF
Create monitoring and backup policies
Define policies for monitoring systems and backup operations.
service_prefix "" {
policy = "read"
}
node_prefix "" {
policy = "read"
}
key_prefix "" {
policy = "read"
}
operator = "read"
session_prefix "" {
policy = "read"
}
acl = "write"
key_prefix "" {
policy = "read"
}
service_prefix "" {
policy = "read"
}
node_prefix "" {
policy = "read"
}
operator = "read"
consul acl policy create -name "monitoring-access" -description "Monitoring system access" -rules @/tmp/monitoring-policy.hcl
consul acl policy create -name "backup-access" -description "Backup operations access" -rules @/tmp/backup-policy.hcl
consul acl token create -description "Monitoring token" -policy-name "monitoring-access"
consul acl token create -description "Backup token" -policy-name "backup-access"
Configure anonymous policy
Set up limited anonymous access for health checks and basic service discovery.
service_prefix "" {
policy = "read"
}
node_prefix "" {
policy = "read"
}
key_prefix "public/" {
policy = "read"
}
consul acl policy create -name "anonymous-access" -description "Anonymous read access" -rules @/tmp/anonymous-policy.hcl
consul acl token update -id anonymous -policy-name "anonymous-access"
Enable audit logging
Configure audit logging to track ACL policy usage and access patterns.
sudo mkdir -p /var/log/consul
sudo chown consul:consul /var/log/consul
sudo chmod 755 /var/log/consul
sudo tee -a /etc/consul.d/consul.hcl > /dev/null <Audit logging
audit {
enabled = true
sink "file" {
path = "/var/log/consul/audit.log"
format = "json"
mode = "blocking"
rotate_duration = "24h"
rotate_max_files = 7
rotate_bytes = 104857600
}
}
EOF
sudo systemctl restart consul
Verify your setup
Test the ACL configuration and verify that policies are working correctly.
consul acl policy list
consul acl role list
consul acl token list
consul members
consul catalog services
Test token-based access:
WEBAPP_TOKEN=$(consul acl token list | grep "Webapp service token" | awk '{print $1}')
CONSUL_HTTP_TOKEN="$WEBAPP_TOKEN" consul catalog services
CONSUL_HTTP_TOKEN="$WEBAPP_TOKEN" consul kv get services/webapp/config || echo "Access correctly denied"
Check audit logs:
sudo tail -f /var/log/consul/audit.log
Common issues
| Symptom | Cause | Fix |
|---|---|---|
| "Permission denied" errors | Incorrect or missing ACL token | Verify token with consul acl token read -id TOKEN_ID |
| Services can't register | Insufficient service write permissions | Add service write policy: service "name" { policy = "write" } |
| Consul UI access denied | Management token not configured | Set management token in browser or use agent token |
| Nodes can't join cluster | Missing node or agent permissions | Ensure agent token has node write permissions |
| Health checks failing | Anonymous policy too restrictive | Add service read permissions to anonymous policy |
| Backup operations fail | Missing ACL read permissions for backup | Use token with acl = "write" permission |
Next steps
- Implement Consul ACL security and encryption for production deployments
- Monitor Consul with Prometheus and Grafana for service discovery observability
- Configure Consul backup and disaster recovery with automated snapshots and restoration
- Integrate Consul with Kubernetes service discovery and automatic configuration
- Set up Consul multi-datacenter replication with ACL token replication
Automated install script
Run this to automate the entire setup
#!/usr/bin/env bash
set -euo pipefail
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
# Script variables
SCRIPT_NAME=$(basename "$0")
CONSUL_DATA_DIR="/opt/consul"
CONSUL_CONFIG_DIR="/etc/consul.d"
CONSUL_CERT_DIR="/etc/consul.d/certs"
CONSUL_USER="consul"
DC_NAME="dc1"
NODE_NAME="consul-server-1"
BIND_ADDR=""
ENCRYPTION_KEY=""
BOOTSTRAP_TOKEN=""
TOTAL_STEPS=8
# Function to print colored output
print_status() {
echo -e "${GREEN}[INFO]${NC} $1"
}
print_warning() {
echo -e "${YELLOW}[WARNING]${NC} $1"
}
print_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
# Usage function
usage() {
cat << EOF
Usage: $SCRIPT_NAME [OPTIONS]
Configure Consul with advanced ACL policies for production security.
Options:
-b, --bind-addr ADDR Bind address for Consul (required)
-n, --node-name NAME Node name (default: consul-server-1)
-d, --datacenter DC Datacenter name (default: dc1)
-h, --help Show this help message
Example:
$SCRIPT_NAME --bind-addr 10.0.1.10
EOF
}
# Parse command line arguments
while [[ $# -gt 0 ]]; do
case $1 in
-b|--bind-addr)
BIND_ADDR="$2"
shift 2
;;
-n|--node-name)
NODE_NAME="$2"
shift 2
;;
-d|--datacenter)
DC_NAME="$2"
shift 2
;;
-h|--help)
usage
exit 0
;;
*)
print_error "Unknown option: $1"
usage
exit 1
;;
esac
done
# Validate required arguments
if [[ -z "$BIND_ADDR" ]]; then
print_error "Bind address is required"
usage
exit 1
fi
# Cleanup function for rollback
cleanup() {
print_error "Installation failed. Cleaning up..."
systemctl stop consul 2>/dev/null || true
systemctl disable consul 2>/dev/null || true
userdel -r consul 2>/dev/null || true
rm -rf $CONSUL_DATA_DIR $CONSUL_CONFIG_DIR 2>/dev/null || true
}
trap cleanup ERR
# Check if running as root
if [[ $EUID -ne 0 ]]; then
print_error "This script must be run as root"
exit 1
fi
# Detect distribution and set package manager
if [[ -f /etc/os-release ]]; then
. /etc/os-release
case "$ID" in
ubuntu|debian)
PKG_MGR="apt"
PKG_UPDATE="apt update"
PKG_INSTALL="apt install -y"
;;
almalinux|rocky|centos|rhel|ol|fedora)
PKG_MGR="dnf"
PKG_UPDATE="dnf makecache"
PKG_INSTALL="dnf install -y"
# Use yum for older CentOS versions
if ! command -v dnf &> /dev/null; then
PKG_MGR="yum"
PKG_INSTALL="yum install -y"
PKG_UPDATE="yum makecache fast"
fi
;;
amzn)
PKG_MGR="yum"
PKG_UPDATE="yum makecache fast"
PKG_INSTALL="yum install -y"
;;
*)
print_error "Unsupported distribution: $ID"
exit 1
;;
esac
else
print_error "Cannot detect distribution"
exit 1
fi
print_status "Detected distribution: $PRETTY_NAME"
print_status "Using package manager: $PKG_MGR"
# Step 1: Install prerequisites and Consul
echo -e "${GREEN}[1/$TOTAL_STEPS]${NC} Installing Consul..."
if [[ "$PKG_MGR" == "apt" ]]; then
curl -fsSL https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list
$PKG_UPDATE
$PKG_INSTALL consul
else
$PKG_INSTALL dnf-plugins-core 2>/dev/null || true
if command -v dnf &> /dev/null; then
dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
else
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
fi
$PKG_INSTALL consul
fi
# Step 2: Create consul user and directories
echo -e "${GREEN}[2/$TOTAL_STEPS]${NC} Creating consul user and directories..."
if ! id "$CONSUL_USER" &>/dev/null; then
useradd --system --home $CONSUL_CONFIG_DIR --shell /bin/false $CONSUL_USER
fi
mkdir -p $CONSUL_DATA_DIR $CONSUL_CONFIG_DIR $CONSUL_CERT_DIR
chown $CONSUL_USER:$CONSUL_USER $CONSUL_DATA_DIR $CONSUL_CONFIG_DIR $CONSUL_CERT_DIR
chmod 755 $CONSUL_DATA_DIR $CONSUL_CONFIG_DIR
chmod 750 $CONSUL_CERT_DIR
# Step 3: Generate encryption key
echo -e "${GREEN}[3/$TOTAL_STEPS]${NC} Generating encryption key..."
ENCRYPTION_KEY=$(consul keygen)
print_status "Generated encryption key: ${ENCRYPTION_KEY:0:8}..."
# Step 4: Generate TLS certificates
echo -e "${GREEN}[4/$TOTAL_STEPS]${NC} Generating TLS certificates..."
cd $CONSUL_CERT_DIR
consul tls ca create
consul tls cert create -server -dc $DC_NAME
chown -R $CONSUL_USER:$CONSUL_USER $CONSUL_CERT_DIR
chmod 600 $CONSUL_CERT_DIR/*.key
chmod 644 $CONSUL_CERT_DIR/*.pem
# Step 5: Create Consul configuration
echo -e "${GREEN}[5/$TOTAL_STEPS]${NC} Creating Consul configuration..."
cat > $CONSUL_CONFIG_DIR/consul.hcl << EOF
datacenter = "$DC_NAME"
data_dir = "$CONSUL_DATA_DIR"
log_level = "INFO"
node_name = "$NODE_NAME"
bind_addr = "$BIND_ADDR"
client_addr = "0.0.0.0"
# Server configuration
server = true
bootstrap_expect = 1
# UI configuration
ui_config {
enabled = true
}
# ACL configuration
acl = {
enabled = true
default_policy = "deny"
enable_token_persistence = true
tokens = {
initial_management = "BOOTSTRAP_TOKEN_PLACEHOLDER"
}
}
# Encryption
encrypt = "$ENCRYPTION_KEY"
encrypt_verify_incoming = true
encrypt_verify_outgoing = true
# TLS configuration
tls {
defaults {
ca_file = "$CONSUL_CERT_DIR/consul-agent-ca.pem"
cert_file = "$CONSUL_CERT_DIR/$DC_NAME-server-consul-0.pem"
key_file = "$CONSUL_CERT_DIR/$DC_NAME-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
}
internal_rpc {
verify_server_hostname = true
}
}
# Performance
performance {
raft_multiplier = 1
}
# Logging
log_rotate_duration = "24h"
log_rotate_max_files = 5
EOF
chown $CONSUL_USER:$CONSUL_USER $CONSUL_CONFIG_DIR/consul.hcl
chmod 640 $CONSUL_CONFIG_DIR/consul.hcl
# Step 6: Start Consul service
echo -e "${GREEN}[6/$TOTAL_STEPS]${NC} Starting Consul service..."
systemctl enable consul
systemctl start consul
sleep 5
# Step 7: Bootstrap ACL system
echo -e "${GREEN}[7/$TOTAL_STEPS]${NC} Bootstrapping ACL system..."
# Wait for Consul to be ready
for i in {1..30}; do
if consul members &>/dev/null; then
break
fi
sleep 2
done
# Bootstrap ACL
BOOTSTRAP_OUTPUT=$(consul acl bootstrap 2>/dev/null || echo "ALREADY_BOOTSTRAPPED")
if [[ "$BOOTSTRAP_OUTPUT" != "ALREADY_BOOTSTRAPPED" ]]; then
BOOTSTRAP_TOKEN=$(echo "$BOOTSTRAP_OUTPUT" | grep "SecretID" | awk '{print $2}')
# Update configuration with actual token
sed -i "s/BOOTSTRAP_TOKEN_PLACEHOLDER/$BOOTSTRAP_TOKEN/" $CONSUL_CONFIG_DIR/consul.hcl
systemctl restart consul
print_status "ACL Bootstrap Token: $BOOTSTRAP_TOKEN"
echo "$BOOTSTRAP_TOKEN" > /root/consul_bootstrap_token
chmod 600 /root/consul_bootstrap_token
print_warning "Bootstrap token saved to /root/consul_bootstrap_token - keep it secure!"
else
print_warning "ACL system already bootstrapped"
fi
# Step 8: Verification
echo -e "${GREEN}[8/$TOTAL_STEPS]${NC} Verifying installation..."
# Check service status
if systemctl is-active consul &>/dev/null; then
print_status "Consul service is running"
else
print_error "Consul service is not running"
exit 1
fi
# Check cluster status
if consul members &>/dev/null; then
print_status "Consul cluster is accessible"
else
print_error "Cannot access Consul cluster"
exit 1
fi
# Check TLS
if consul members -ca-file="$CONSUL_CERT_DIR/consul-agent-ca.pem" &>/dev/null; then
print_status "TLS certificates are working"
fi
print_status "Consul installation completed successfully!"
print_status "Web UI available at: https://$BIND_ADDR:8501"
print_status "Remember to:"
print_status " 1. Configure firewall to allow ports 8300-8302, 8500-8502"
print_status " 2. Set CONSUL_HTTP_TOKEN environment variable for CLI access"
print_status " 3. Create additional ACL policies for your services"
Review the script before running. Execute with: bash install.sh