Deploy Envoy-based service mesh in Kubernetes production environment with SSL and observability

Advanced 45 min Apr 24, 2026 540 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Set up a production-ready Envoy service mesh in Kubernetes with mutual TLS authentication, SSL certificate management, and comprehensive observability through Prometheus monitoring and distributed tracing.

Prerequisites

  • Kubernetes cluster with admin access
  • kubectl configured
  • Minimum 4GB RAM per node
  • SSL certificate management knowledge

What this solves

Envoy proxy provides a powerful service mesh solution for Kubernetes environments, handling east-west traffic management, security, and observability. This tutorial sets up a production-grade Envoy service mesh with SSL/TLS termination, mutual authentication between services, and comprehensive monitoring through Prometheus metrics and distributed tracing.

Step-by-step installation

Install required dependencies

Update your system and install the necessary tools for Envoy deployment.

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget gnupg software-properties-common
sudo dnf update -y
sudo dnf install -y curl wget gnupg

Install kubectl and Helm

Install the Kubernetes command-line tool and Helm package manager for deploying the Envoy service mesh components.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt update
sudo apt install -y helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Create Envoy service mesh namespace

Create a dedicated namespace for the Envoy service mesh components and enable automatic sidecar injection.

kubectl create namespace envoy-system
kubectl label namespace envoy-system envoy-injection=enabled

Generate SSL certificates for service mesh

Create a certificate authority and generate SSL certificates for mutual TLS authentication between services.

mkdir -p /tmp/envoy-certs
cd /tmp/envoy-certs

# Create CA private key
openssl genrsa -out ca-key.pem 2048

# Create CA certificate
openssl req -new -x509 -key ca-key.pem -out ca-cert.pem -days 365 \
  -subj "/C=US/ST=CA/L=San Francisco/O=Example/CN=Envoy CA"

# Create server private key
openssl genrsa -out server-key.pem 2048

# Create server certificate signing request
openssl req -new -key server-key.pem -out server.csr \
  -subj "/C=US/ST=CA/L=San Francisco/O=Example/CN=envoy-proxy"

# Sign server certificate
openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem \
  -out server-cert.pem -days 365 -CAcreateserial

Create Kubernetes secrets for SSL certificates

Store the generated certificates as Kubernetes secrets for use by Envoy proxies.

kubectl create secret tls envoy-certs \
  --cert=server-cert.pem \
  --key=server-key.pem \
  -n envoy-system

kubectl create secret generic ca-cert \
  --from-file=ca-cert.pem \
  -n envoy-system

Deploy Envoy control plane

Create the Envoy control plane configuration that manages proxy configurations and certificate distribution.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-control-plane
  namespace: envoy-system
  labels:
    app: envoy-control-plane
spec:
  replicas: 2
  selector:
    matchLabels:
      app: envoy-control-plane
  template:
    metadata:
      labels:
        app: envoy-control-plane
    spec:
      containers:
      - name: envoy-control-plane
        image: envoyproxy/go-control-plane:v0.12.0
        ports:
        - containerPort: 18000
          name: xds
        - containerPort: 19000
          name: admin
        env:
        - name: ENVOY_ADMIN_PORT
          value: "19000"
        volumeMounts:
        - name: ca-cert
          mountPath: /etc/ssl/certs
          readOnly: true
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
      volumes:
      - name: ca-cert
        secret:
          secretName: ca-cert
---
apiVersion: v1
kind: Service
metadata:
  name: envoy-control-plane
  namespace: envoy-system
spec:
  selector:
    app: envoy-control-plane
  ports:
  - name: xds
    port: 18000
    targetPort: 18000
  - name: admin
    port: 19000
    targetPort: 19000
kubectl apply -f envoy-control-plane.yaml

Create Envoy proxy DaemonSet

Deploy Envoy proxies as a DaemonSet to run on all nodes, providing service mesh capabilities.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: envoy-proxy
  namespace: envoy-system
  labels:
    app: envoy-proxy
spec:
  selector:
    matchLabels:
      app: envoy-proxy
  template:
    metadata:
      labels:
        app: envoy-proxy
    spec:
      hostNetwork: true
      containers:
      - name: envoy
        image: envoyproxy/envoy:v1.28.0
        command:
        - /usr/local/bin/envoy
        - --config-path
        - /etc/envoy/envoy.yaml
        - --service-cluster
        - envoy-proxy
        - --service-node
        - envoy-proxy
        - --log-level
        - info
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
        - containerPort: 443
          hostPort: 443
          name: https
        - containerPort: 15000
          hostPort: 15000
          name: admin
        - containerPort: 9901
          hostPort: 9901
          name: metrics
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
        - name: envoy-certs
          mountPath: /etc/ssl/envoy
          readOnly: true
        resources:
          requests:
            memory: "256Mi"
            cpu: "200m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /ready
            port: 15000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 15000
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config
      - name: envoy-certs
        secret:
          secretName: envoy-certs

Configure Envoy proxy with SSL and observability

Create the main Envoy configuration with SSL termination, load balancing, and metrics collection.

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-config
  namespace: envoy-system
data:
  envoy.yaml: |
    admin:
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 15000
    
    static_resources:
      listeners:
      - name: https_listener
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 443
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_https
              access_log:
              - name: envoy.access_loggers.stdout
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      cluster: backend_service
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                - certificate_chain:
                    filename: /etc/ssl/envoy/tls.crt
                  private_key:
                    filename: /etc/ssl/envoy/tls.key
      
      - name: http_listener
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 80
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              access_log:
              - name: envoy.access_loggers.stdout
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    redirect:
                      https_redirect: true
      
      clusters:
      - name: backend_service
        connect_timeout: 30s
        type: LOGICAL_DNS
        dns_lookup_family: V4_ONLY
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: backend_service
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: backend-service.default.svc.cluster.local
                    port_value: 8080
        health_checks:
        - timeout: 5s
          interval: 10s
          unhealthy_threshold: 2
          healthy_threshold: 2
          http_health_check:
            path: /health
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
            common_tls_context:
              validation_context:
                trusted_ca:
                  filename: /etc/ssl/certs/ca-cert.pem
    
    stats_config:
      stats_tags:
      - tag_name: cluster_name
        regex: "^cluster\\.((.+?)\\.).*"
      - tag_name: virtual_host_name
        regex: "^vhost\\.((.+?)\\.).*"
    
    tracing:
      http:
        name: envoy.tracers.zipkin
        typed_config:
          "@type": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig
          collector_cluster: jaeger
          collector_endpoint: "/api/v2/spans"
          shared_span_context: false
kubectl apply -f envoy-config.yaml

Deploy the Envoy DaemonSet

Apply the DaemonSet configuration to deploy Envoy proxies across all cluster nodes.

kubectl apply -f envoy-proxy-daemonset.yaml

Install Prometheus for metrics collection

Deploy Prometheus to collect metrics from Envoy proxies for monitoring and alerting.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false

Create ServiceMonitor for Envoy metrics

Configure Prometheus to scrape metrics from Envoy proxy instances.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: envoy-proxy-metrics
  namespace: envoy-system
  labels:
    app: envoy-proxy
spec:
  selector:
    matchLabels:
      app: envoy-proxy
  endpoints:
  - port: metrics
    interval: 30s
    path: /stats/prometheus
    honorLabels: true
---
apiVersion: v1
kind: Service
metadata:
  name: envoy-proxy-metrics
  namespace: envoy-system
  labels:
    app: envoy-proxy
spec:
  selector:
    app: envoy-proxy
  ports:
  - name: metrics
    port: 9901
    targetPort: 9901
  type: ClusterIP
kubectl apply -f envoy-servicemonitor.yaml

Install Jaeger for distributed tracing

Deploy Jaeger to collect and visualize distributed traces from the service mesh.

kubectl create namespace jaeger

kubectl apply -f - <

Update Envoy configuration for Jaeger integration

Add the Jaeger cluster to the Envoy configuration for distributed tracing.

kubectl patch configmap envoy-config -n envoy-system --type merge -p '
{
  "data": {
    "envoy.yaml": "admin:\n  address:\n    socket_address:\n      protocol: TCP\n      address: 0.0.0.0\n      port_value: 15000\n\nstatic_resources:\n  listeners:\n  - name: https_listener\n    address:\n      socket_address:\n        protocol: TCP\n        address: 0.0.0.0\n        port_value: 443\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: ingress_https\n          tracing:\n            provider:\n              name: envoy.tracers.zipkin\n              typed_config:\n                \"@type\": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\n                collector_cluster: jaeger\n                collector_endpoint: \"/api/v2/spans\"\n          access_log:\n          - name: envoy.access_loggers.stdout\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog\n          http_filters:\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: local_service\n              domains: [\"*\"]\n              routes:\n              - match:\n                  prefix: \"/\"\n                route:\n                  cluster: backend_service\n      transport_socket:\n        name: envoy.transport_sockets.tls\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext\n          common_tls_context:\n            tls_certificates:\n            - certificate_chain:\n                filename: /etc/ssl/envoy/tls.crt\n              private_key:\n                filename: /etc/ssl/envoy/tls.key\n  \n  - name: http_listener\n    address:\n      socket_address:\n        protocol: TCP\n        address: 0.0.0.0\n        port_value: 80\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: ingress_http\n          access_log:\n          - name: envoy.access_loggers.stdout\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog\n          http_filters:\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: local_service\n              domains: [\"*\"]\n              routes:\n              - match:\n                  prefix: \"/\"\n                redirect:\n                  https_redirect: true\n  \n  clusters:\n  - name: backend_service\n    connect_timeout: 30s\n    type: LOGICAL_DNS\n    dns_lookup_family: V4_ONLY\n    lb_policy: ROUND_ROBIN\n    load_assignment:\n      cluster_name: backend_service\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: backend-service.default.svc.cluster.local\n                port_value: 8080\n    health_checks:\n    - timeout: 5s\n      interval: 10s\n      unhealthy_threshold: 2\n      healthy_threshold: 2\n      http_health_check:\n        path: /health\n    transport_socket:\n      name: envoy.transport_sockets.tls\n      typed_config:\n        \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n        common_tls_context:\n          val

Automated install script

Run this to automate the entire setup

¿Prefiere no gestionarlo usted mismo?

Gestionamos la infraestructura de empresas que dependen del tiempo de actividad. Totalmente gestionada, con un contacto fijo que conoce su entorno.

Tiene un contacto fijo que conoce su entorno

Róterdam 04:07 · accesible por mensaje, sin formulario de tickets