Deploy Envoy-based service mesh in Kubernetes production environment with SSL and observability

Advanced 45 min Apr 24, 2026 544 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Set up a production-ready Envoy service mesh in Kubernetes with mutual TLS authentication, SSL certificate management, and comprehensive observability through Prometheus monitoring and distributed tracing.

Prerequisites

  • Kubernetes cluster with admin access
  • kubectl configured
  • Minimum 4GB RAM per node
  • SSL certificate management knowledge

What this solves

Envoy proxy provides a powerful service mesh solution for Kubernetes environments, handling east-west traffic management, security, and observability. This tutorial sets up a production-grade Envoy service mesh with SSL/TLS termination, mutual authentication between services, and comprehensive monitoring through Prometheus metrics and distributed tracing.

Step-by-step installation

Install required dependencies

Update your system and install the necessary tools for Envoy deployment.

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget gnupg software-properties-common
sudo dnf update -y
sudo dnf install -y curl wget gnupg

Install kubectl and Helm

Install the Kubernetes command-line tool and Helm package manager for deploying the Envoy service mesh components.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt update
sudo apt install -y helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Create Envoy service mesh namespace

Create a dedicated namespace for the Envoy service mesh components and enable automatic sidecar injection.

kubectl create namespace envoy-system
kubectl label namespace envoy-system envoy-injection=enabled

Generate SSL certificates for service mesh

Create a certificate authority and generate SSL certificates for mutual TLS authentication between services.

mkdir -p /tmp/envoy-certs
cd /tmp/envoy-certs

# Create CA private key
openssl genrsa -out ca-key.pem 2048

# Create CA certificate
openssl req -new -x509 -key ca-key.pem -out ca-cert.pem -days 365 \
  -subj "/C=US/ST=CA/L=San Francisco/O=Example/CN=Envoy CA"

# Create server private key
openssl genrsa -out server-key.pem 2048

# Create server certificate signing request
openssl req -new -key server-key.pem -out server.csr \
  -subj "/C=US/ST=CA/L=San Francisco/O=Example/CN=envoy-proxy"

# Sign server certificate
openssl x509 -req -in server.csr -CA ca-cert.pem -CAkey ca-key.pem \
  -out server-cert.pem -days 365 -CAcreateserial

Create Kubernetes secrets for SSL certificates

Store the generated certificates as Kubernetes secrets for use by Envoy proxies.

kubectl create secret tls envoy-certs \
  --cert=server-cert.pem \
  --key=server-key.pem \
  -n envoy-system

kubectl create secret generic ca-cert \
  --from-file=ca-cert.pem \
  -n envoy-system

Deploy Envoy control plane

Create the Envoy control plane configuration that manages proxy configurations and certificate distribution.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-control-plane
  namespace: envoy-system
  labels:
    app: envoy-control-plane
spec:
  replicas: 2
  selector:
    matchLabels:
      app: envoy-control-plane
  template:
    metadata:
      labels:
        app: envoy-control-plane
    spec:
      containers:
      - name: envoy-control-plane
        image: envoyproxy/go-control-plane:v0.12.0
        ports:
        - containerPort: 18000
          name: xds
        - containerPort: 19000
          name: admin
        env:
        - name: ENVOY_ADMIN_PORT
          value: "19000"
        volumeMounts:
        - name: ca-cert
          mountPath: /etc/ssl/certs
          readOnly: true
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
      volumes:
      - name: ca-cert
        secret:
          secretName: ca-cert
---
apiVersion: v1
kind: Service
metadata:
  name: envoy-control-plane
  namespace: envoy-system
spec:
  selector:
    app: envoy-control-plane
  ports:
  - name: xds
    port: 18000
    targetPort: 18000
  - name: admin
    port: 19000
    targetPort: 19000
kubectl apply -f envoy-control-plane.yaml

Create Envoy proxy DaemonSet

Deploy Envoy proxies as a DaemonSet to run on all nodes, providing service mesh capabilities.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: envoy-proxy
  namespace: envoy-system
  labels:
    app: envoy-proxy
spec:
  selector:
    matchLabels:
      app: envoy-proxy
  template:
    metadata:
      labels:
        app: envoy-proxy
    spec:
      hostNetwork: true
      containers:
      - name: envoy
        image: envoyproxy/envoy:v1.28.0
        command:
        - /usr/local/bin/envoy
        - --config-path
        - /etc/envoy/envoy.yaml
        - --service-cluster
        - envoy-proxy
        - --service-node
        - envoy-proxy
        - --log-level
        - info
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
        - containerPort: 443
          hostPort: 443
          name: https
        - containerPort: 15000
          hostPort: 15000
          name: admin
        - containerPort: 9901
          hostPort: 9901
          name: metrics
        volumeMounts:
        - name: envoy-config
          mountPath: /etc/envoy
        - name: envoy-certs
          mountPath: /etc/ssl/envoy
          readOnly: true
        resources:
          requests:
            memory: "256Mi"
            cpu: "200m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /ready
            port: 15000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 15000
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: envoy-config
        configMap:
          name: envoy-config
      - name: envoy-certs
        secret:
          secretName: envoy-certs

Configure Envoy proxy with SSL and observability

Create the main Envoy configuration with SSL termination, load balancing, and metrics collection.

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-config
  namespace: envoy-system
data:
  envoy.yaml: |
    admin:
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 15000
    
    static_resources:
      listeners:
      - name: https_listener
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 443
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_https
              access_log:
              - name: envoy.access_loggers.stdout
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      cluster: backend_service
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                - certificate_chain:
                    filename: /etc/ssl/envoy/tls.crt
                  private_key:
                    filename: /etc/ssl/envoy/tls.key
      
      - name: http_listener
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 80
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              access_log:
              - name: envoy.access_loggers.stdout
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    redirect:
                      https_redirect: true
      
      clusters:
      - name: backend_service
        connect_timeout: 30s
        type: LOGICAL_DNS
        dns_lookup_family: V4_ONLY
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: backend_service
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: backend-service.default.svc.cluster.local
                    port_value: 8080
        health_checks:
        - timeout: 5s
          interval: 10s
          unhealthy_threshold: 2
          healthy_threshold: 2
          http_health_check:
            path: /health
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
            common_tls_context:
              validation_context:
                trusted_ca:
                  filename: /etc/ssl/certs/ca-cert.pem
    
    stats_config:
      stats_tags:
      - tag_name: cluster_name
        regex: "^cluster\\.((.+?)\\.).*"
      - tag_name: virtual_host_name
        regex: "^vhost\\.((.+?)\\.).*"
    
    tracing:
      http:
        name: envoy.tracers.zipkin
        typed_config:
          "@type": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig
          collector_cluster: jaeger
          collector_endpoint: "/api/v2/spans"
          shared_span_context: false
kubectl apply -f envoy-config.yaml

Deploy the Envoy DaemonSet

Apply the DaemonSet configuration to deploy Envoy proxies across all cluster nodes.

kubectl apply -f envoy-proxy-daemonset.yaml

Install Prometheus for metrics collection

Deploy Prometheus to collect metrics from Envoy proxies for monitoring and alerting.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false

Create ServiceMonitor for Envoy metrics

Configure Prometheus to scrape metrics from Envoy proxy instances.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: envoy-proxy-metrics
  namespace: envoy-system
  labels:
    app: envoy-proxy
spec:
  selector:
    matchLabels:
      app: envoy-proxy
  endpoints:
  - port: metrics
    interval: 30s
    path: /stats/prometheus
    honorLabels: true
---
apiVersion: v1
kind: Service
metadata:
  name: envoy-proxy-metrics
  namespace: envoy-system
  labels:
    app: envoy-proxy
spec:
  selector:
    app: envoy-proxy
  ports:
  - name: metrics
    port: 9901
    targetPort: 9901
  type: ClusterIP
kubectl apply -f envoy-servicemonitor.yaml

Install Jaeger for distributed tracing

Deploy Jaeger to collect and visualize distributed traces from the service mesh.

kubectl create namespace jaeger

kubectl apply -f - <

Update Envoy configuration for Jaeger integration

Add the Jaeger cluster to the Envoy configuration for distributed tracing.

kubectl patch configmap envoy-config -n envoy-system --type merge -p '
{
  "data": {
    "envoy.yaml": "admin:\n  address:\n    socket_address:\n      protocol: TCP\n      address: 0.0.0.0\n      port_value: 15000\n\nstatic_resources:\n  listeners:\n  - name: https_listener\n    address:\n      socket_address:\n        protocol: TCP\n        address: 0.0.0.0\n        port_value: 443\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: ingress_https\n          tracing:\n            provider:\n              name: envoy.tracers.zipkin\n              typed_config:\n                \"@type\": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\n                collector_cluster: jaeger\n                collector_endpoint: \"/api/v2/spans\"\n          access_log:\n          - name: envoy.access_loggers.stdout\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog\n          http_filters:\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: local_service\n              domains: [\"*\"]\n              routes:\n              - match:\n                  prefix: \"/\"\n                route:\n                  cluster: backend_service\n      transport_socket:\n        name: envoy.transport_sockets.tls\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext\n          common_tls_context:\n            tls_certificates:\n            - certificate_chain:\n                filename: /etc/ssl/envoy/tls.crt\n              private_key:\n                filename: /etc/ssl/envoy/tls.key\n  \n  - name: http_listener\n    address:\n      socket_address:\n        protocol: TCP\n        address: 0.0.0.0\n        port_value: 80\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: ingress_http\n          access_log:\n          - name: envoy.access_loggers.stdout\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog\n          http_filters:\n          - name: envoy.filters.http.router\n            typed_config:\n              \"@type\": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: local_service\n              domains: [\"*\"]\n              routes:\n              - match:\n                  prefix: \"/\"\n                redirect:\n                  https_redirect: true\n  \n  clusters:\n  - name: backend_service\n    connect_timeout: 30s\n    type: LOGICAL_DNS\n    dns_lookup_family: V4_ONLY\n    lb_policy: ROUND_ROBIN\n    load_assignment:\n      cluster_name: backend_service\n      endpoints:\n      - lb_endpoints:\n        - endpoint:\n            address:\n              socket_address:\n                address: backend-service.default.svc.cluster.local\n                port_value: 8080\n    health_checks:\n    - timeout: 5s\n      interval: 10s\n      unhealthy_threshold: 2\n      healthy_threshold: 2\n      http_health_check:\n        path: /health\n    transport_socket:\n      name: envoy.transport_sockets.tls\n      typed_config:\n        \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext\n        common_tls_context:\n          val

Automated install script

Run this to automate the entire setup

不想自己管理这些吗?

我们为依赖稳定运行时间的企业管理基础设施。全托管服务,配备一位熟悉您系统架构的固定联系人。

您将拥有一位了解您整体架构的固定联系人

Rotterdam 04:07 · 一条消息即可联系我们,无需填写工单表单