Setup HAProxy with Docker container backends for dynamic load balancing

Install HAProxy 2.8

First, install HAProxy 2.8 which includes the Docker API integration features needed for dynamic backend discovery.

sudo apt update
sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:vbernat/haproxy-2.8
sudo apt update
sudo apt install -y haproxy=2.8.* docker.io

sudo dnf update -y
sudo dnf install -y haproxy docker
sudo systemctl enable --now docker

Configure Docker daemon for HAProxy access

HAProxy needs access to Docker's API to discover running containers. Configure Docker to expose its API socket with proper permissions.

sudo usermod -aG docker haproxy
sudo systemctl restart docker
sudo systemctl enable docker

Create HAProxy configuration with Docker backend discovery

This configuration enables HAProxy to automatically discover Docker containers labeled for load balancing and perform health checks.

global
    daemon
    user haproxy
    group haproxy
    log stdout local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    
    # SSL configuration
    ssl-default-bind-ciphers ECDHE+AESGCM:ECDHE+CHACHA20:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option httplog
    option dontlognull
    option http-server-close
    option forwardfor except 127.0.0.0/8
    option redispatch
    retries 3
    
    # Health check defaults
    option httpchk GET /health
    http-check expect status 200

frontend web_frontend
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/
    
    # Redirect HTTP to HTTPS
    redirect scheme https code 301 if !{ ssl_fc }
    
    # Security headers
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
    http-response set-header X-Content-Type-Options nosniff
    http-response set-header X-Frame-Options DENY
    http-response set-header X-XSS-Protection "1; mode=block"
    
    # Route based on Host header
    acl is_app_service hdr(host) -i app.example.com
    acl is_api_service hdr(host) -i api.example.com
    
    use_backend app_containers if is_app_service
    use_backend api_containers if is_api_service
    default_backend web_containers

Dynamic backend for web containers
backend web_containers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    
    # Docker service discovery template
    server-template web 10 _web._tcp.service.consul:80 check resolvers consul resolve-prefer ipv4

Dynamic backend for app containers
backend app_containers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    
backend api_containers
    balance roundrobin
    option httpchk GET /api/health
    http-check expect status 200

HAProxy statistics
listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 30s
    stats admin if TRUE

Create Docker service discovery script

This script monitors Docker containers and updates HAProxy configuration dynamically when containers are created or destroyed.

#!/usr/bin/env python3
import docker
import time
import subprocess
import logging
from pathlib import Path

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class HAProxyDockerDiscovery:
    def __init__(self):
        self.client = docker.from_env()
        self.config_path = '/etc/haproxy/haproxy.cfg'
        self.template_path = '/etc/haproxy/haproxy.cfg.template'
        self.last_config_hash = None
        
    def get_running_containers(self):
        containers = []
        for container in self.client.containers.list():
            labels = container.labels
            if 'haproxy.enable' in labels and labels['haproxy.enable'] == 'true':
                container_info = {
                    'name': container.name,
                    'ip': self.get_container_ip(container),
                    'port': labels.get('haproxy.port', '80'),
                    'backend': labels.get('haproxy.backend', 'web_containers'),
                    'health_check': labels.get('haproxy.health_check', '/health')
                }
                containers.append(container_info)
        return containers
    
    def get_container_ip(self, container):
        networks = container.attrs['NetworkSettings']['Networks']
        for network_name, network_info in networks.items():
            if network_info['IPAddress']:
                return network_info['IPAddress']
        return None
    
    def generate_backend_config(self, containers):
        backends = {}
        for container in containers:
            backend_name = container['backend']
            if backend_name not in backends:
                backends[backend_name] = []
            backends[backend_name].append(container)
        
        config_lines = []
        for backend_name, backend_containers in backends.items():
            config_lines.append(f"\nbackend {backend_name}")
            config_lines.append("    balance roundrobin")
            config_lines.append("    option httpchk GET /health")
            config_lines.append("    http-check expect status 200")
            
            for i, container in enumerate(backend_containers):
                server_line = f"    server {container['name']} {container['ip']}:{container['port']} check"
                config_lines.append(server_line)
        
        return '\n'.join(config_lines)
    
    def update_config(self):
        containers = self.get_running_containers()
        backend_config = self.generate_backend_config(containers)
        
        # Read template and replace backend sections
        with open('/etc/haproxy/haproxy.cfg.template', 'r') as f:
            template_content = f.read()
        
        # Replace dynamic backends marker with generated config
        new_config = template_content.replace('{{DYNAMIC_BACKENDS}}', backend_config)
        
        # Write new configuration
        with open(self.config_path, 'w') as f:
            f.write(new_config)
        
        # Reload HAProxy if configuration changed
        subprocess.run(['sudo', 'systemctl', 'reload', 'haproxy'], check=True)
        logger.info(f"Updated HAProxy config with {len(containers)} containers")
    
    def run(self):
        logger.info("Starting HAProxy Docker discovery service")
        while True:
            try:
                self.update_config()
                time.sleep(10)  # Check every 10 seconds
            except Exception as e:
                logger.error(f"Error updating configuration: {e}")
                time.sleep(5)

if __name__ == '__main__':
    discovery = HAProxyDockerDiscovery()
    discovery.run()

Make the discovery script executable

Set proper permissions for the discovery script and create a systemd service to run it automatically.

sudo chmod +x /usr/local/bin/haproxy-docker-discovery.py
sudo pip3 install docker

Create systemd service for discovery

This service ensures the Docker discovery script runs automatically and restarts if it fails.

[Unit]
Description=HAProxy Docker Discovery Service
After=docker.service haproxy.service
Requires=docker.service

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/haproxy-docker-discovery.py
Restart=always
RestartSec=5
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target

Create SSL certificate directory

Set up the directory structure for SSL certificates with proper permissions.

sudo mkdir -p /etc/haproxy/certs
sudo chown haproxy:haproxy /etc/haproxy/certs
sudo chmod 750 /etc/haproxy/certs

Generate self-signed certificate for testing

Create a test SSL certificate. Replace this with your real certificates in production.

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout /tmp/example.com.key \
  -out /tmp/example.com.crt \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=example.com"

Combine certificate and key for HAProxy
sudo cat /tmp/example.com.crt /tmp/example.com.key > /etc/haproxy/certs/example.com.pem
sudo chown haproxy:haproxy /etc/haproxy/certs/example.com.pem
sudo chmod 600 /etc/haproxy/certs/example.com.pem
sudo rm /tmp/example.com.key /tmp/example.com.crt

Create configuration template

Copy the current configuration as a template for the discovery script to use.

sudo cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.template

Enable and start services

Start HAProxy and the discovery service, enabling them to start automatically on boot.

sudo systemctl enable --now haproxy
sudo systemctl enable --now haproxy-docker-discovery
sudo systemctl status haproxy
sudo systemctl status haproxy-docker-discovery

Create test Docker containers

Launch example containers with the correct labels for HAProxy to discover them automatically.

# Create a simple web server container
docker run -d \
  --name web1 \
  --label haproxy.enable=true \
  --label haproxy.backend=web_containers \
  --label haproxy.port=80 \
  nginx:alpine

Create another web server for load balancing
docker run -d \
  --name web2 \
  --label haproxy.enable=true \
  --label haproxy.backend=web_containers \
  --label haproxy.port=80 \
  nginx:alpine

Create an API service container
docker run -d \
  --name api1 \
  --label haproxy.enable=true \
  --label haproxy.backend=api_containers \
  --label haproxy.port=3000 \
  --label haproxy.health_check=/api/health \
  node:alpine sh -c "npm init -y && npm install express && echo 'const express = require(\"express\"); const app = express(); app.get(\"/api/health\", (req, res) => res.json({status: \"ok\"})); app.listen(3000);' > server.js && node server.js"

Configure firewall rules

Open the necessary ports for HTTP, HTTPS, and HAProxy stats interface.

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 8404/tcp
sudo ufw reload

sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
sudo firewall-cmd --permanent --add-port=8404/tcp
sudo firewall-cmd --reload

Advanced health check configuration

Configure more sophisticated health checks that can detect application-level failures, not just connectivity.

# Add to backend sections for advanced health checks
backend web_containers
    balance roundrobin
    
    # Multi-layer health checks
    option httpchk GET /health HTTP/1.1\r\nHost:\ localhost
    http-check expect status 200
    http-check expect string "status":"ok"
    
    # Health check timing
    timeout check 5s
    
    # Mark server down after 3 failed checks
    default-server inter 10s downinter 5s rise 2 fall 3 slowstart 60s maxconn 256 maxqueue 128 weight 100

Implement service discovery with Consul

For production environments, integrate with Consul for more robust service discovery. First install Consul.

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt update
sudo apt install -y consul

sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y consul

Configure Consul for service registration

Set up Consul to automatically register Docker services and provide DNS resolution for HAProxy.

datacenter = "dc1"
data_dir = "/opt/consul"
log_level = "INFO"
node_name = "haproxy-node"
bind_addr = "127.0.0.1"
client_addr = "127.0.0.1"

ui_config {
  enabled = true
}

connect {
  enabled = true
}

server = true
bootstrap_expect = 1

Update HAProxy for Consul DNS resolution

Configure HAProxy to use Consul for DNS-based service discovery with automatic backend updates.

# Add resolver section to global configuration
resolvers consul
    nameserver consul 127.0.0.1:8600
    accepted_payload_size 8192
    hold valid 5s

Update backend to use Consul DNS
backend web_containers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    
    # Use Consul service discovery
    server-template web- 10 web.service.consul:80 check resolvers consul init-addr none

Configure SSL with Let's Encrypt

For production, replace the self-signed certificate with Let's Encrypt certificates. Install certbot first.

sudo apt install -y certbot
sudo systemctl stop haproxy

sudo dnf install -y certbot
sudo systemctl stop haproxy

Obtain SSL certificates

Generate Let's Encrypt certificates and convert them to HAProxy format. Replace example.com with your actual domain.

sudo certbot certonly --standalone -d example.com -d api.example.com -d app.example.com

Convert certificates to HAProxy format
sudo mkdir -p /etc/haproxy/certs
sudo cat /etc/letsencrypt/live/example.com/fullchain.pem \
     /etc/letsencrypt/live/example.com/privkey.pem \
     > /etc/haproxy/certs/example.com.pem
sudo chown haproxy:haproxy /etc/haproxy/certs/example.com.pem
sudo chmod 600 /etc/haproxy/certs/example.com.pem

Configure certificate renewal

Set up automatic certificate renewal with a script that updates HAProxy after renewal.

#!/bin/bash

Renew certificates
certbot renew --quiet

Update HAProxy certificates
for domain in $(certbot certificates 2>/dev/null | grep 'Certificate Name' | cut -d':' -f2 | tr -d ' '); do
    if [ -d "/etc/letsencrypt/live/$domain" ]; then
        cat /etc/letsencrypt/live/$domain/fullchain.pem \
            /etc/letsencrypt/live/$domain/privkey.pem \
            > /etc/haproxy/certs/$domain.pem
        chown haproxy:haproxy /etc/haproxy/certs/$domain.pem
        chmod 600 /etc/haproxy/certs/$domain.pem
    fi
done

Reload HAProxy to use new certificates
systemctl reload haproxy

Schedule certificate renewal

Add a cron job to automatically renew certificates before they expire.

sudo chmod +x /usr/local/bin/renew-haproxy-certs.sh
sudo crontab -e

Add this line to run renewal twice daily
0 2,14   * /usr/local/bin/renew-haproxy-certs.sh

Enhance security headers and rate limiting

Add comprehensive security headers and implement rate limiting to protect against common attacks.

frontend web_frontend
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
    
    # Rate limiting
    stick-table type ip size 100k expire 30s store http_req_rate(10s),http_err_rate(10s)
    http-request track-sc0 src
    http-request deny if { sc_http_req_rate(0) gt 20 }
    http-request deny if { sc_http_err_rate(0) gt 10 }
    
    # Security headers
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    http-response set-header X-Content-Type-Options nosniff
    http-response set-header X-Frame-Options DENY
    http-response set-header X-XSS-Protection "1; mode=block"
    http-response set-header Referrer-Policy "strict-origin-when-cross-origin"
    http-response set-header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'"
    
    # Remove server information
    http-response del-header Server
    http-response set-header Server "HAProxy"
    
    # Redirect HTTP to HTTPS
    redirect scheme https code 301 if !{ ssl_fc }
    
    # ACME challenge for Let's Encrypt
    acl letsencrypt-acl path_beg /.well-known/acme-challenge/
    use_backend letsencrypt-backend if letsencrypt-acl
    
    # Route based on Host header
    acl is_app_service hdr(host) -i app.example.com
    acl is_api_service hdr(host) -i api.example.com
    
    use_backend app_containers if is_app_service
    use_backend api_containers if is_api_service
    default_backend web_containers

Backend for Let's Encrypt challenges
backend letsencrypt-backend
    server letsencrypt 127.0.0.1:54321