Deploy Envoy-based service mesh in Kubernetes production environment with SSL and observability

Advanced 45 min Apr 24, 2026 86 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Set up a production-ready Envoy service mesh in Kubernetes with mutual TLS authentication, SSL certificate management, and comprehensive observability through Prometheus monitoring and distributed tracing.

Prerequisites

  • Kubernetes cluster with admin access
  • kubectl configured
  • Minimum 4GB RAM per node
  • SSL certificate management knowledge

What this solves

Envoy proxy provides a powerful service mesh solution for Kubernetes environments, handling east-west traffic management, security, and observability. This tutorial sets up a production-grade Envoy service mesh with SSL/TLS termination, mutual authentication between services, and comprehensive monitoring through Prometheus metrics and distributed tracing.

Step-by-step installation

Install required dependencies

Update your system and install the necessary tools for Envoy deployment.

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget gnupg software-properties-common
sudo dnf update -y
sudo dnf install -y curl wget gnupg

Install kubectl and Helm

Install the Kubernetes command-line tool and Helm package manager for deploying the Envoy service mesh components.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt update
sudo apt install -y helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Create Envoy service mesh namespace

Create a dedicated namespace for the Envoy service mesh components and enable automatic sidecar injection.

kubectl create namespace envoy-system
kubectl label namespace envoy-system envoy-injection=enabled

Generate SSL certificates for service mesh

Create a certificate authority and generate SSL certificates for mutual TLS authentication between services.

mkdir -p /tmp/envoy-certs
cd /tmp/envoy-certs

Create CA private key

openssl genrsa -out ca-key.pem 2048

Create CA certificate

openssl req -new -x509 -key ca-key.pem -out ca-cert.pem -days 365 \ -subj "/C=US/ST=CA/L=San Francisco/O=Example/CN=Envoy CA"

Create server private key

openssl genrsa -out server-key.pem 2048

Create server certificate signing request

openssl req -new -key server-key.pem -out server.csr \ -subj "/C=US/ST=CA/L=San Francisco/O=Example/CN=envoy-proxy"

Sign server certificate

openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem \ -out server-cert.pem -days 365 -CAcreateserial

Create Kubernetes secrets for SSL certificates

Store the generated certificates as Kubernetes secrets for use by Envoy proxies.

kubectl create secret tls envoy-certs \
  --cert=server-cert.pem \
  --key=server-key.pem \
  -n envoy-system

kubectl create secret generic ca-cert \
  --from-file=ca-cert.pem \
  -n envoy-system

Deploy Envoy control plane

Create the Envoy control plane configuration that manages proxy configurations and certificate distribution.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-control-plane
  namespace: envoy-system
  labels:
    app: envoy-control-plane
spec:
  replicas: 2
  selector:
    matchLabels:
      app: envoy-control-plane
  template:
    metadata:
      labels:
        app: envoy-control-plane
    spec:
      containers:
      - name: envoy-control-plane
        image: envoyproxy/go-control-plane:v0.12.0
        ports:
        - containerPort: 18000
          name: xds
        - containerPort: 19000
          name: admin
        env:
        - name: ENVOY_ADMIN_PORT
          value: "19000"
        volumeMounts:
        - name: ca-cert
          mountPath: /etc/ssl/certs
          readOnly: true
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
      volumes:
      - name: ca-cert
        secret:
          secretName: ca-cert
---
apiVersion: v1
kind: Service
metadata:
  name: envoy-control-plane
  namespace: envoy-system
spec:
  selector:
    app: envoy-control-plane
  ports:
  - name: xds
    port: 18000
    targetPort: 18000
  - name: admin
    port: 19000
    targetPort: 19000
kubectl apply -f envoy-control-plane.yaml

Create Envoy proxy DaemonSet

Deploy Envoy proxies as a DaemonSet to run on all nodes, providing service mesh capabilities.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: envoy-proxy
  namespace: envoy-system
  labels:
    app: envoy-proxy
spec:
  selector:
    matchLabels:
      app: envoy-proxy
  template:
    metadata:
      labels:
        app: envoy-proxy
    spec:
      hostNetwork: true
      containers:
      - name: envoy
        image: envoyproxy/envoy:v1.28.0
        command:
        - /usr/local/bin/envoy
        - --config-path
        - /etc/envoy/envoy.yaml
        - --service-cluster
        - envoy-proxy
        - --service-node
        - envoy-proxy
        - --log-level
        - info
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
        - containerPort: 443
          hostPort: 443
          name: https
        - containerPort: 15000
          hostPort: 15000
          name: admin
        - containerPort: 9901
          hostPort: 9901
          name: metrics
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
        - name: envoy-certs
          mountPath: /etc/ssl/envoy
          readOnly: true
        resources:
          requests:
            memory: "256Mi"
            cpu: "200m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /ready
            port: 15000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 15000
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config
      - name: envoy-certs
        secret:
          secretName: envoy-certs

Configure Envoy proxy with SSL and observability

Create the main Envoy configuration with SSL termination, load balancing, and metrics collection.

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-config
  namespace: envoy-system
data:
  envoy.yaml: |
    admin:
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 15000
    
    static_resources:
      listeners:
      - name: https_listener
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 443
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_https
              access_log:
              - name: envoy.access_loggers.stdout
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      cluster: backend_service
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                - certificate_chain:
                    filename: /etc/ssl/envoy/tls.crt
                  private_key:
                    filename: /etc/ssl/envoy/tls.key
      
      - name: http_listener
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 80
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              access_log:
              - name: envoy.access_loggers.stdout
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    redirect:
                      https_redirect: true
      
      clusters:
      - name: backend_service
        connect_timeout: 30s
        type: LOGICAL_DNS
        dns_lookup_family: V4_ONLY
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: backend_service
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: backend-service.default.svc.cluster.local
                    port_value: 8080
        health_checks:
        - timeout: 5s
          interval: 10s
          unhealthy_threshold: 2
          healthy_threshold: 2
          http_health_check:
            path: /health
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
            common_tls_context:
              validation_context:
                trusted_ca:
                  filename: /etc/ssl/certs/ca-cert.pem
    
    stats_config:
      stats_tags:
      - tag_name: cluster_name
        regex: "^cluster\\.((.+?)\\.).*"
      - tag_name: virtual_host_name
        regex: "^vhost\\.((.+?)\\.).*"
    
    tracing:
      http:
        name: envoy.tracers.zipkin
        typed_config:
          "@type": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig
          collector_cluster: jaeger
          collector_endpoint: "/api/v2/spans"
          shared_span_context: false
kubectl apply -f envoy-config.yaml

Deploy the Envoy DaemonSet

Apply the DaemonSet configuration to deploy Envoy proxies across all cluster nodes.

kubectl apply -f envoy-proxy-daemonset.yaml

Install Prometheus for metrics collection

Deploy Prometheus to collect metrics from Envoy proxies for monitoring and alerting.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false

Create ServiceMonitor for Envoy metrics

Configure Prometheus to scrape metrics from Envoy proxy instances.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: envoy-proxy-metrics
  namespace: envoy-system
  labels:
    app: envoy-proxy
spec:
  selector:
    matchLabels:
      app: envoy-proxy
  endpoints:
  - port: metrics
    interval: 30s
    path: /stats/prometheus
    honorLabels: true
---
apiVersion: v1
kind: Service
metadata:
  name: envoy-proxy-metrics
  namespace: envoy-system
  labels:
    app: envoy-proxy
spec:
  selector:
    app: envoy-proxy
  ports:
  - name: metrics
    port: 9901
    targetPort: 9901
  type: ClusterIP
kubectl apply -f envoy-servicemonitor.yaml

Install Jaeger for distributed tracing

Deploy Jaeger to collect and visualize distributed traces from the service mesh.

kubectl create namespace jaeger

kubectl apply -f - <

Update Envoy configuration for Jaeger integration

Add the Jaeger cluster to the Envoy configuration for distributed tracing.

kubectl patch configmap envoy-config -n envoy-system --type merge -p '
{
  "data": {
    "envoy.yaml": "admin:\n  address:\n    socket_address:\n      protocol: TCP\n      address: 0.0.0.0\n      port_value: 15000\n\nstatic_resources:\n  listeners:\n  - name: https_listener\n    address:\n      socket_address:\n        protocol: TCP\n        address: 0.0.0.0\n        port_value: 443\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: ingress_https\n          tracing:\n            provider:\n              name: envoy.tracers.zipkin\n              typed_config:\n                \"@type\": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\n                collector_cluster: jaeger\n                collector_endpoint: \"/api/v2/spans\"\n          access_log:\n          - name: envoy.access_loggers.stdout\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog\n          http_filters:\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: local_service\n              domains: [\"\"]\n              routes:\n              - match:\n                  prefix: \"/\"\n                route:\n                  cluster: backend_service\n      transport_socket:\n        name: envoy.transport_sockets.tls\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext\n          common_tls_context:\n            tls_certificates:\n            - certificate_chain:\n                filename: /etc/ssl/envoy/tls.crt\n              private_key:\n                filename: /etc/ssl/envoy/tls.key\n  \n  - name: http_listener\n    address:\n      socket_address:\n        protocol: TCP\n        address: 0.0.0.0\n        port_value: 80\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: ingress_http\n          access_log:\n          - name: envoy.access_loggers.stdout\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog\n          http_filters:\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: local_service\n              domains: [\"\"]\n              routes:\n              - match:\n                  prefix: \"/\"\n                redirect:\n                  https_redirect: true\n  \n  clusters:\n  - name: backend_service\n    connect_timeout: 30s\n    type: LOGICAL_DNS\n    dns_lookup_family: V4_ONLY\n    lb_policy: ROUND_ROBIN\n    load_assignment:\n      cluster_name: backend_service\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: backend-service.default.svc.cluster.local\n                port_value: 8080\n    health_checks:\n    - timeout: 5s\n      interval: 10s\n      unhealthy_threshold: 2\n      healthy_threshold: 2\n      http_health_check:\n        path: /health\n    transport_socket:\n      name: envoy.transport_sockets.tls\n      typed_config:\n        \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n        common_tls_context:\n          validation_context:\n            trusted_ca:\n              filename: /etc/ssl/certs/ca-cert.pem\n  \n  - name: jaeger\n    connect_timeout: 5s\n    type: LOGICAL_DNS\n    dns_lookup_family: V4_ONLY\n    lb_policy: ROUND_ROBIN\n    load_assignment:\n      cluster_name: jaeger\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: jaeger-service.jaeger.svc.cluster.local\n                port_value: 9411\n\nstats_config:\n  stats_tags:\n  - tag_name: cluster_name\n    regex: \"^cluster\\\\.((.+?)\\\\.).\"\n  - tag_name: virtual_host_name\n    regex: \"^vhost\\\\.((.+?)\\\\.).\""
  }
}'

kubectl rollout restart daemonset/envoy-proxy -n envoy-system

Deploy sample application for testing

Create a sample backend service to test the service mesh functionality.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-service
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: backend-service
  template:
    metadata:
      labels:
        app: backend-service
    spec:
      containers:
      - name: backend
        image: nginx:1.25
        ports:
        - containerPort: 80
        volumeMounts:
        - name: config
          mountPath: /etc/nginx/conf.d
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "100m"
      volumes:
      - name: config
        configMap:
          name: backend-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: backend-config
  namespace: default
data:
  default.conf: |
    server {
        listen 80;
        server_name localhost;
        
        location / {
            return 200 'Hello from backend service\n';
            add_header Content-Type text/plain;
        }
        
        location /health {
            return 200 'OK\n';
            add_header Content-Type text/plain;
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: backend-service
  namespace: default
spec:
  selector:
    app: backend-service
  ports:
  - name: http
    port: 8080
    targetPort: 80
  type: ClusterIP
kubectl apply -f sample-app.yaml

Verify your setup

Check that all components are running correctly and the service mesh is operational.

# Check Envoy proxy status
kubectl get pods -n envoy-system
kubectl get daemonset -n envoy-system

Verify Envoy admin interface

kubectl port-forward -n envoy-system ds/envoy-proxy 15000:15000 & curl http://localhost:15000/ready curl http://localhost:15000/stats | grep envoy

Check Prometheus metrics

kubectl get servicemonitor -n envoy-system kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &

Visit http://localhost:9090 and search for envoy_ metrics

Verify Jaeger tracing

kubectl port-forward -n jaeger svc/jaeger-service 16686:16686 &

Visit http://localhost:16686 for Jaeger UI

Test SSL termination

kubectl get nodes -o wide

Replace NODE_IP with actual node IP

curl -k https://NODE_IP/

Check certificate details

openssl s_client -connect NODE_IP:443 -servername example.com

Common issues

Symptom Cause Fix
Envoy pods not starting ConfigMap syntax error kubectl logs -n envoy-system ds/envoy-proxy and check YAML syntax
SSL handshake failures Certificate mismatch or expiration Regenerate certificates with correct CN and check expiration dates
No metrics in Prometheus ServiceMonitor not found kubectl get servicemonitor -A and verify labels match
Backend connection failures Service discovery issues Check service names and namespaces in Envoy cluster configuration
Tracing not working Jaeger cluster unreachable Verify Jaeger service is running and accessible from Envoy pods
High memory usage Default resource limits too low Adjust memory limits in DaemonSet based on traffic volume

Next steps

Running this in production?

Need enterprise-grade service mesh? Running this at scale adds a second layer of work: capacity planning, certificate rotation, failover drills, and on-call response. Our managed platform covers monitoring, backups and 24/7 response by default.

Automated install script

Run this to automate the entire setup

Need help?

Don't want to manage this yourself?

We handle managed devops services for businesses that depend on uptime. From initial setup to ongoing operations.