Configure advanced gRPC load balancing with Envoy Proxy health checks and circuit breakers

Advanced 45 min Apr 23, 2026 657 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Set up Envoy Proxy with intelligent gRPC load balancing, health checks, and circuit breakers for production microservices. Includes SSL termination, monitoring integration, and security hardening.

Prerequisites

  • Root or sudo access
  • Basic understanding of gRPC and microservices
  • Network connectivity between Envoy and backend services
  • At least 2GB RAM for testing

What this solves

Envoy Proxy provides production-grade load balancing for gRPC services with built-in health checks, circuit breakers, and observability. This configuration handles automatic failover, prevents cascade failures, and gives you detailed metrics on service performance.

Step-by-step configuration

Install Envoy Proxy

Add the official Envoy repository and install the latest stable version.

sudo apt update
curl -sL 'https://deb.dl.getenvoy.io/public/gpg.8115BA8E629CC074.key' | sudo gpg --dearmor -o /usr/share/keyrings/getenvoy-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/getenvoy-keyring.gpg] https://deb.dl.getenvoy.io/public/deb/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/getenvoy.list
sudo apt update && sudo apt install -y getenvoy-envoy
sudo dnf install -y curl
curl -sL 'https://rpm.dl.getenvoy.io/public/gpg.CF716AF503183491.key' | sudo rpm --import -
echo -e "[getenvoy]\nname=GetEnvoy\nbaseurl=https://rpm.dl.getenvoy.io/public/rpm/el/8/\$basearch\nenabled=1\ngpgcheck=1\ngpgkey=https://rpm.dl.getenvoy.io/public/gpg.CF716AF503183491.key" | sudo tee /etc/yum.repos.d/getenvoy.repo
sudo dnf install -y getenvoy-envoy

Create Envoy user and directories

Set up a dedicated user and directory structure for security isolation.

sudo useradd --system --shell /bin/false --home-dir /var/lib/envoy --create-home envoy
sudo mkdir -p /etc/envoy /var/log/envoy
sudo chown -R envoy:envoy /var/lib/envoy /var/log/envoy
sudo chmod 755 /etc/envoy

Configure main Envoy configuration

Create the primary configuration file with admin interface, listeners, and cluster definitions.

admin:
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 9901
  access_log:
    - name: envoy.access_loggers.file
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
        path: "/var/log/envoy/admin.log"

static_resources:
  listeners:
  - name: grpc_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8080
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: grpc_proxy
          codec_type: HTTP2
          access_log:
            - name: envoy.access_loggers.file
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                path: "/var/log/envoy/access.log"
                format: |
                  [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"
                  %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT%
                  %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%"
                  "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"
          http_filters:
          - name: envoy.filters.http.grpc_stats
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig
              emit_filter_state: true
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: local_route
            virtual_hosts:
            - name: grpc_backend
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: grpc_cluster
                  timeout: 30s
                  retry_policy:
                    retry_on: "5xx,reset,connect-failure,refused-stream"
                    num_retries: 3
                    per_try_timeout: 10s
                    retry_back_off:
                      base_interval: 0.1s
                      max_interval: 1s

  clusters:
  - name: grpc_cluster
    type: ROUND_ROBIN
    lb_policy: ROUND_ROBIN
    http2_protocol_options: {}
    health_checks:
    - timeout: 5s
      interval: 10s
      unhealthy_threshold: 3
      healthy_threshold: 2
      grpc_health_check:
        service_name: "health"
        authority: "grpc-service"
    circuit_breakers:
      thresholds:
      - priority: DEFAULT
        max_connections: 100
        max_pending_requests: 50
        max_requests: 200
        max_retries: 3
        track_remaining: true
    outlier_detection:
      consecutive_5xx: 3
      consecutive_gateway_failure: 3
      interval: 30s
      base_ejection_time: 30s
      max_ejection_percent: 50
      split_external_local_origin_errors: true
    load_assignment:
      cluster_name: grpc_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 192.168.1.10
                port_value: 9000
          health_check_config:
            port_value: 9000
        - endpoint:
            address:
              socket_address:
                address: 192.168.1.11
                port_value: 9000
          health_check_config:
            port_value: 9000
        - endpoint:
            address:
              socket_address:
                address: 192.168.1.12
                port_value: 9000
          health_check_config:
            port_value: 9000

Set up gRPC backend services with health endpoints

Install and configure sample gRPC services that implement the health check protocol.

sudo apt install -y golang-go
export GOPATH=/opt/go
sudo mkdir -p $GOPATH
cd $GOPATH
sudo go mod init grpc-health-server
sudo go get google.golang.org/grpc
sudo go get google.golang.org/grpc/health
sudo go get google.golang.org/grpc/health/grpc_health_v1
sudo dnf install -y golang
export GOPATH=/opt/go
sudo mkdir -p $GOPATH
cd $GOPATH
sudo go mod init grpc-health-server
sudo go get google.golang.org/grpc
sudo go get google.golang.org/grpc/health
sudo go get google.golang.org/grpc/health/grpc_health_v1

Create sample gRPC health server

Build a simple gRPC server with health check implementation for testing.

package main

import (
    "context"
    "log"
    "net"
    "time"

    "google.golang.org/grpc"
    "google.golang.org/grpc/health"
    "google.golang.org/grpc/health/grpc_health_v1"
)

type server struct{}

func (s *server) SayHello(ctx context.Context, req *HelloRequest) (*HelloResponse, error) {
    return &HelloResponse{Message: "Hello from gRPC server"}, nil
}

type HelloRequest struct{}
type HelloResponse struct{
    Message string
}

func main() {
    port := ":9000"
    lis, err := net.Listen("tcp", port)
    if err != nil {
        log.Fatalf("Failed to listen: %v", err)
    }

    s := grpc.NewServer()
    
    // Register health service
    healthServer := health.NewServer()
    grpc_health_v1.RegisterHealthServer(s, healthServer)
    
    // Set service status to serving
    healthServer.SetServingStatus("health", grpc_health_v1.HealthCheckResponse_SERVING)
    
    log.Printf("gRPC server listening on %s", port)
    if err := s.Serve(lis); err != nil {
        log.Fatalf("Failed to serve: %v", err)
    }
}

Configure advanced load balancing algorithms

Update the cluster configuration to use weighted round robin and least request algorithms.

# Add this to replace the grpc_cluster section in envoy.yaml
  - name: grpc_cluster_weighted
    type: LEAST_REQUEST
    lb_policy: LEAST_REQUEST
    http2_protocol_options: {}
    health_checks:
    - timeout: 5s
      interval: 10s
      unhealthy_threshold: 3
      healthy_threshold: 2
      grpc_health_check:
        service_name: "health"
        authority: "grpc-service"
      event_log_path: "/var/log/envoy/health_check.log"
    circuit_breakers:
      thresholds:
      - priority: DEFAULT
        max_connections: 100
        max_pending_requests: 50
        max_requests: 200
        max_retries: 3
        track_remaining: true
      - priority: HIGH
        max_connections: 200
        max_pending_requests: 100
        max_requests: 400
        max_retries: 5
    outlier_detection:
      consecutive_5xx: 3
      consecutive_gateway_failure: 3
      interval: 30s
      base_ejection_time: 30s
      max_ejection_percent: 50
      min_health_percent: 30
      split_external_local_origin_errors: true
    common_lb_config:
      healthy_panic_threshold:
        value: 30.0
      zone_aware_lb_config:
        routing_enabled:
          value: 100.0
        min_cluster_size: 3
    load_assignment:
      cluster_name: grpc_cluster_weighted
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 192.168.1.10
                port_value: 9000
          load_balancing_weight: 100
          health_check_config:
            port_value: 9000
        - endpoint:
            address:
              socket_address:
                address: 192.168.1.11
                port_value: 9000
          load_balancing_weight: 150
          health_check_config:
            port_value: 9000
        - endpoint:
            address:
              socket_address:
                address: 192.168.1.12
                port_value: 9000
          load_balancing_weight: 80
          health_check_config:
            port_value: 9000

Enable Prometheus metrics collection

Configure Envoy to export detailed metrics for monitoring and alerting.

# Add this to the admin section in envoy.yaml
stats_config:
  stats_tags:
  - tag_name: "cluster_name"
    regex: "^cluster\\.((.+?)\\.)"
  - tag_name: "virtual_host_name" 
    regex: "^vhost\\.((.+?)\\.)"
  - tag_name: "listener_address"
    regex: "^listener\\.((.+?)\\.)"
  stats_matches:
  - name: "circuit_breaker"
    actions:
    - name: "circuit_breaker_stats"
      action:
        "@type": type.googleapis.com/envoy.config.core.v3.HeaderValueOption
        header:
          key: "x-circuit-breaker"
          value: "true"

stats_sinks:
- name: envoy.stat_sinks.metrics_service
  typed_config:
    "@type": type.googleapis.com/envoy.config.metrics.v3.MetricsServiceConfig
    transport_api_version: V3
    grpc_service:
      envoy_grpc:
        cluster_name: metrics_cluster
- name: envoy.stat_sinks.statsd
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.stat_sinks.statsd.v3.StatsdSink
    address:
      socket_address:
        address: 127.0.0.1
        port_value: 9125
    prefix: envoy

Configure SSL termination

Add TLS configuration for secure gRPC communication with certificate management.

sudo mkdir -p /etc/envoy/certs
sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/envoy/certs/server.key -out /etc/envoy/certs/server.crt -days 365 -nodes -subj "/C=US/ST=State/L=City/O=Organization/CN=grpc.example.com"
sudo chown -R envoy:envoy /etc/envoy/certs
sudo chmod 600 /etc/envoy/certs/server.key
sudo chmod 644 /etc/envoy/certs/server.crt

Update configuration for SSL

Modify the listener configuration to include TLS transport socket.

# Replace the filter_chains section in envoy.yaml
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: grpc_proxy_ssl
          codec_type: HTTP2
          access_log:
            - name: envoy.access_loggers.file
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                path: "/var/log/envoy/ssl_access.log"
          http_filters:
          - name: envoy.filters.http.grpc_stats
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig
              emit_filter_state: true
              stats_for_all_methods: true
          - name: envoy.filters.http.fault
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault
              delay:
                fixed_delay: 0.1s
                percentage:
                  numerator: 1
                  denominator: HUNDRED
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: ssl_local_route
            virtual_hosts:
            - name: grpc_ssl_backend
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                  grpc: {}
                route:
                  cluster: grpc_cluster
                  timeout: 30s
                  retry_policy:
                    retry_on: "5xx,reset,connect-failure,refused-stream"
                    num_retries: 3
                    per_try_timeout: 10s
                    retry_back_off:
                      base_interval: 0.1s
                      max_interval: 2s
                    retry_host_predicate:
                    - name: envoy.retry_host_predicates.previous_hosts
                      typed_config:
                        "@type": type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: "/etc/envoy/certs/server.crt"
              private_key:
                filename: "/etc/envoy/certs/server.key"
            alpn_protocols: ["h2"]

Create systemd service

Set up Envoy as a system service with proper security and restart policies.

[Unit]
Description=Envoy Proxy
After=network.target
Requires=network.target

[Service]
Type=simple
User=envoy
Group=envoy
ExecStart=/usr/bin/envoy -c /etc/envoy/envoy.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=5
LimitNOFILE=65536
StandardOutput=journal
StandardError=journal
SyslogIdentifier=envoy

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/log/envoy /var/lib/envoy
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Configure log rotation

Set up logrotate to manage Envoy log files and prevent disk space issues.

/var/log/envoy/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 644 envoy envoy
    postrotate
        /bin/systemctl reload envoy.service > /dev/null 2>&1 || true
    endscript
}

Start and enable Envoy

Enable the service to start automatically and verify it's running correctly.

sudo systemctl daemon-reload
sudo systemctl enable --now envoy
sudo systemctl status envoy

Configure firewall rules

Open necessary ports for gRPC traffic and admin interface access.

sudo ufw allow 8080/tcp comment 'Envoy gRPC proxy'
sudo ufw allow from 127.0.0.1 to any port 9901 comment 'Envoy admin interface'
sudo ufw reload
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=127.0.0.1 port protocol=tcp port=9901 accept'
sudo firewall-cmd --reload

Set up Prometheus monitoring integration

Configure Prometheus to scrape Envoy metrics for comprehensive observability.

# Add this job to your Prometheus configuration
- job_name: 'envoy-proxy'
  static_configs:
  - targets: ['localhost:9901']
  metrics_path: /stats/prometheus
  scrape_interval: 15s
  scrape_timeout: 10s
  honor_labels: true
  params:
    format: ['prometheus']
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'envoy_cluster_(.+)_circuit_breakers_(.+)_(.+)'
    target_label: 'circuit_breaker_type'
    replacement: '${2}'
  - source_labels: [__name__]
    regex: 'envoy_cluster_(.+)_health_check_(.+)'
    target_label: 'health_check_type' 
    replacement: '${2}'

Verify your setup

Test the Envoy configuration and verify all components are working correctly.

# Check Envoy service status
sudo systemctl status envoy

# Verify configuration syntax
envoy --mode validate -c /etc/envoy/envoy.yaml

# Test admin interface
curl -s http://localhost:9901/stats | grep cluster

# Check cluster health status
curl -s http://localhost:9901/clusters | grep health_flags

# Test gRPC endpoint (requires grpcurl)
grpcurl -plaintext localhost:8080 list

# Monitor circuit breaker status
curl -s http://localhost:9901/stats | grep circuit_breaker

# Check health check logs
sudo tail -f /var/log/envoy/health_check.log

# View access logs
sudo tail -f /var/log/envoy/access.log

Advanced circuit breaker configuration

Configure custom circuit breaker thresholds

Fine-tune circuit breaker settings based on your service capacity and requirements.

Automated install script

Run this to automate the entire setup

Vous ne voulez pas gérer cela vous-même ?

Nous gérons l'infrastructure des entreprises qui dépendent de leur disponibilité. Entièrement infogéré, avec un interlocuteur fixe qui connaît votre environnement.

Vous avez un interlocuteur fixe qui connaît votre installation

À son bureau à Rotterdam 10:07 · joignable par message, sans formulaire de ticket