Implement Istio multi-cluster canary deployments and traffic splitting

Install istioctl

Download the Istio CLI, which manages installation, certificate generation and multi-cluster bootstrap operations.

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.23.2 sh -
cd istio-1.23.2
export PATH=$PWD/bin:$PATH
istioctl version --remote=false

Set kubectl contexts for both clusters

Name your contexts clearly so every command below is unambiguous about which cluster it targets.

kubectl config rename-context admin@cluster1 cluster1
kubectl config rename-context admin@cluster2 cluster2
kubectl get nodes --context=cluster1
kubectl get nodes --context=cluster2

Generate a shared root certificate authority

Both clusters must trust the same root CA so workload certificates issued by either istiod are mutually trusted. Generate intermediate CAs signed by one root for each cluster.

mkdir -p certs && cd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca
make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts
make -f ../tools/certs/Makefile.selfsigned.mk cluster2-cacerts

Install the shared CA secrets

Create the cacerts secret in the istio-system namespace on both clusters before installing Istio, so istiod picks up the shared root at bootstrap.

kubectl --context=cluster1 create namespace istio-system
kubectl --context=cluster1 create secret generic cacerts -n istio-system \
  --from-file=cluster1/ca-cert.pem \
  --from-file=cluster1/ca-key.pem \
  --from-file=cluster1/root-cert.pem \
  --from-file=cluster1/cert-chain.pem

kubectl --context=cluster2 create namespace istio-system
kubectl --context=cluster2 create secret generic cacerts -n istio-system \
  --from-file=cluster2/ca-cert.pem \
  --from-file=cluster2/ca-key.pem \
  --from-file=cluster2/root-cert.pem \
  --from-file=cluster2/cert-chain.pem

Install Istio on the primary cluster

Label the network and install the control plane with the multi-cluster mesh and network identifiers set explicitly.

kubectl --context=cluster1 label namespace istio-system topology.istio.io/network=network1

cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
EOF

istioctl install --context=cluster1 -f cluster1.yaml -y

Deploy the east-west gateway on cluster1

The east-west gateway exposes istiod and cross-cluster service traffic over mTLS so cluster2 workloads can discover and reach cluster1 services.

@istioctl x create-remote-secret --help > /dev/null
samples/multicluster/gen-eastwest-gateway.sh \
  --mesh mesh1 --cluster cluster1 --network network1 | \
  istioctl --context=cluster1 install -y -f -

kubectl --context=cluster1 get svc istio-eastwestgateway -n istio-system

Expose istiod and services through the gateway

Apply the exposure gateway configuration so remote clusters can reach the control plane and mesh services on port 15443.

kubectl --context=cluster1 apply -n istio-system -f \
  samples/multicluster/expose-istiod.yaml
kubectl --context=cluster1 apply -n istio-system -f \
  samples/multicluster/expose-services.yaml

Install Istio on the remote cluster

Cluster2 runs a lightweight config that points at the primary's discovery address instead of running its own istiod.

kubectl --context=cluster2 label namespace istio-system topology.istio.io/network=network2

export DISCOVERY_ADDRESS=$(kubectl --context=cluster1 \
  -n istio-system get svc istio-eastwestgateway \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    istiodRemote:
      injectionURL: https://$DISCOVERY_ADDRESS:15017/inject/cluster/cluster2/net/network2
    global:
      remotePilotAddress: $DISCOVERY_ADDRESS
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2
EOF

istioctl install --context=cluster2 -f cluster2.yaml -y

Link the clusters with remote secrets

Create a remote secret in each cluster so istiod can discover endpoints in the other cluster's API server.

istioctl create-remote-secret --context=cluster2 --name=cluster2 | \
  kubectl apply -f - --context=cluster1

istioctl create-remote-secret --context=cluster1 --name=cluster1 | \
  kubectl apply -f - --context=cluster2

Enable sidecar injection and deploy the stable version

Deploy version v1 (stable) to both clusters with a shared service name so Istio treats them as one logical service across the mesh.

kubectl --context=cluster1 create namespace checkout
kubectl --context=cluster2 create namespace checkout
kubectl --context=cluster1 label namespace checkout istio-injection=enabled
kubectl --context=cluster2 label namespace checkout istio-injection=enabled

kubectl --context=cluster1 apply -n checkout -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-v1
  labels: { app: checkout, version: v1 }
spec:
  replicas: 3
  selector:
    matchLabels: { app: checkout, version: v1 }
  template:
    metadata:
      labels: { app: checkout, version: v1 }
    spec:
      containers:
      - name: checkout
        image: example.com/checkout:1.4.0
        ports: [{ containerPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata:
  name: checkout
spec:
  selector: { app: checkout }
  ports: [{ port: 80, targetPort: 8080 }]
EOF

Deploy the canary version on cluster2

Deploy v2 only in cluster2 to validate the new build against a subset of regional traffic before promoting it mesh-wide.

kubectl --context=cluster2 apply -n checkout -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-v2
  labels: { app: checkout, version: v2 }
spec:
  replicas: 3
  selector:
    matchLabels: { app: checkout, version: v2 }
  template:
    metadata:
      labels: { app: checkout, version: v2 }
    spec:
      containers:
      - name: checkout
        image: example.com/checkout:1.5.0-rc1
        ports: [{ containerPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata:
  name: checkout
spec:
  selector: { app: checkout }
  ports: [{ port: 80, targetPort: 8080 }]
EOF

Define the DestinationRule subsets

DestinationRule declares the subsets v1 and v2 that VirtualService will route between. Apply it identically to both clusters so routing decisions are consistent mesh-wide.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: checkout
  namespace: checkout
spec:
  host: checkout.checkout.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  subsets:
  - name: v1
    labels: { version: v1 }
  - name: v2
    labels: { version: v2 }

kubectl --context=cluster1 apply -f /tmp/checkout-destinationrule.yaml
kubectl --context=cluster2 apply -f /tmp/checkout-destinationrule.yaml

Start the canary at 5 percent

Route the vast majority of traffic to v1 and a small slice to v2. Apply this identically on both clusters so the split is consistent regardless of which cluster receives the request.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: checkout
  namespace: checkout
spec:
  hosts:
  - checkout.checkout.svc.cluster.local
  http:
  - route:
    - destination:
        host: checkout.checkout.svc.cluster.local
        subset: v1
      weight: 95
    - destination:
        host: checkout.checkout.svc.cluster.local
        subset: v2
      weight: 5

kubectl --context=cluster1 apply -f /tmp/checkout-vs-5.yaml
kubectl --context=cluster2 apply -f /tmp/checkout-vs-5.yaml

Progress the weight in stages

Increase the weight in fixed increments, holding at each stage long enough to gather statistically meaningful error rate and latency data. A typical schedule is 5, 25, 50, 100 percent with 10-15 minutes per stage.

for weight in 25 50 100; do
  stable=$((100 - weight))
  sed -e "s/weight: 95/weight: $stable/" -e "s/weight: 5/weight: $weight/" \
    /tmp/checkout-vs-5.yaml > /tmp/checkout-vs-$weight.yaml
done
cat /tmp/checkout-vs-25.yaml

Install Prometheus and query error rate per subset

Istio's default telemetry exports istio_requests_total with a destination_version label, which lets you compute per-subset success rate directly in PromQL.

kubectl --context=cluster1 apply -f samples/addons/prometheus.yaml
kubectl --context=cluster1 -n istio-system port-forward svc/prometheus 9090:9090 &

sum(rate(istio_requests_total{destination_service_name="checkout",destination_version="v2",response_code!~"5.*"}[5m]))
/
sum(rate(istio_requests_total{destination_service_name="checkout",destination_version="v2"}[5m]))

Write a promotion/rollback script

This script polls the success rate for the canary subset and either advances the weight or reverts to the stable VirtualService if the threshold is breached.

#!/usr/bin/env bash
set -euo pipefail

PROM_URL="http://localhost:9090"
THRESHOLD="0.98"
QUERY='sum(rate(istio_requests_total{destination_service_name="checkout",destination_version="v2",response_code!~"5.*"}[5m]))/sum(rate(istio_requests_total{destination_service_name="checkout",destination_version="v2"}[5m]))'

success_rate=$(curl -s --data-urlencode "query=${QUERY}" "${PROM_URL}/api/v1/query" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["data"]["result"][0]["value"][1] if d["data"]["result"] else "1")')

result=$(python3 -c "print(1 if float('${success_rate}') >= ${THRESHOLD} else 0)")

if [ "$result" -eq 1 ]; then
  echo "Canary healthy at ${success_rate}, safe to promote"
  exit 0
else
  echo "Canary success rate ${success_rate} below threshold, rolling back"
  kubectl --context=cluster1 apply -f /tmp/checkout-vs-rollback.yaml
  kubectl --context=cluster2 apply -f /tmp/checkout-vs-rollback.yaml
  exit 1
fi

chmod 750 /usr/local/bin/canary-gate.sh
sudo chown root:root /usr/local/bin/canary-gate.sh

Prepare the rollback VirtualService

Keep a ready-to-apply manifest that routes 100 percent of traffic back to v1 so the rollback script has zero build time during an incident.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: checkout
  namespace: checkout
spec:
  hosts:
  - checkout.checkout.svc.cluster.local
  http:
  - route:
    - destination:
        host: checkout.checkout.svc.cluster.local
        subset: v1
      weight: 100

Schedule the gate as a systemd timer

Run the analysis every 5 minutes during an active rollout window instead of relying on manual checks.

[Unit]
Description=Istio canary metrics gate

[Service]
Type=oneshot
ExecStart=/usr/local/bin/canary-gate.sh
User=deploy

[Unit]
Description=Run canary gate every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now canary-gate.timer

Confirm mTLS is enforced mesh-wide

PeerAuthentication set to STRICT rejects any plaintext traffic between sidecars, which is required before you rely on cross-cluster identity for canary routing decisions.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

kubectl --context=cluster1 apply -f /tmp/peerauth-strict.yaml
kubectl --context=cluster2 apply -f /tmp/peerauth-strict.yaml
istioctl x describe pod checkout-v1-xxxx -n checkout --context=cluster1

Test cross-cluster failover

Scale checkout to zero in cluster1 and confirm cluster2 continues serving requests without client-visible errors, proving the mesh routes across the east-west gateway when local endpoints disappear.

kubectl --context=cluster1 -n checkout scale deployment checkout-v1 --replicas=0
for i in $(seq 1 20); do
  kubectl --context=cluster1 -n checkout exec deploy/sleep -- curl -s -o /dev/null -w "%{http_code}\n" http://checkout.checkout.svc.cluster.local
done
kubectl --context=cluster1 -n checkout scale deployment checkout-v1 --replicas=3

Install Kiali and Grafana add-ons

Kiali visualizes the live traffic split percentages between subsets and both clusters, which is faster to reason about than raw PromQL during an active rollout.

kubectl --context=cluster1 apply -f samples/addons/kiali.yaml
kubectl --context=cluster1 apply -f samples/addons/grafana.yaml
kubectl --context=cluster1 -n istio-system port-forward svc/kiali 20001:20001 &

Import the Istio service dashboards

Grafana's official Istio dashboards break down request volume, p99 latency and error rate per workload version, which is exactly the signal the canary gate script consumes.

kubectl --context=cluster1 -n istio-system port-forward svc/grafana 3000:3000 &

For a broader Prometheus and Grafana rollout dashboard setup covering the whole mesh, follow monitoring Istio service mesh with Prometheus and Grafana dashboards. For tracing individual canary requests end to end across clusters, pair this with integrating Jaeger with Istio service mesh for distributed tracing.

Confirm endpoint discovery across clusters

If cluster2 workloads never appear as viable endpoints from cluster1, the remote secret or network label is usually the cause.

istioctl proxy-config endpoints deploy/checkout-v1 -n checkout --context=cluster1 | grep checkout
kubectl --context=cluster1 -n istio-system get secret | grep istio-remote-secret