Istio Circuit Breaker & Retry Policies Tutorial

Configure Istio destination rules with circuit breaker patterns, implement retry policies with exponential backoff, and set up comprehensive fault injection testing for microservices resilience in Kubernetes service mesh environments.

Prerequisites

Kubernetes cluster running
Istio service mesh installed
kubectl configured
Basic understanding of Kubernetes networking

What this solves

Microservices architectures are vulnerable to cascading failures when one service becomes unresponsive or slow. Istio circuit breakers prevent these failures from propagating by temporarily stopping requests to unhealthy services, while retry policies automatically handle transient failures. This tutorial shows you how to implement production-grade resilience patterns that protect your service mesh from outages and performance degradation.

Step-by-step configuration

Verify Istio installation

Check that Istio is properly installed in your Kubernetes cluster and the control plane is running.

kubectl get pods -n istio-system
istioctl version

Deploy sample applications

Create a test environment with frontend and backend services to demonstrate circuit breaker and retry behavior.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: frontend
        image: nginx:alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: default
spec:
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: backend
        image: httpbin/httpbin
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: default
spec:
  selector:
    app: backend
  ports:
  - port: 80
    targetPort: 80

kubectl apply -f frontend-app.yaml

Configure circuit breaker destination rules

Create destination rules that define circuit breaker thresholds and connection pool settings for the backend service.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backend-circuit-breaker
  namespace: default
spec:
  host: backend.default.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 10
        connectTimeout: 30s
        tcpKeepalive:
          time: 7200s
          interval: 75s
      http:
        http1MaxPendingRequests: 10
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
        maxRetries: 3
        consecutiveGatewayErrors: 5
        interval: 30s
        baseEjectionTime: 30s
        maxEjectionPercent: 50
        minHealthPercent: 30
    outlierDetection:
      consecutive5xxErrors: 3
      consecutiveGatewayErrors: 3
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
      minHealthPercent: 30
      splitExternalLocalOriginErrors: false

kubectl apply -f backend-destination-rule.yaml

Implement retry policies with exponential backoff

Configure virtual services with retry policies that handle transient failures automatically using exponential backoff strategies.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backend-retry-policy
  namespace: default
spec:
  hosts:
  - backend.default.svc.cluster.local
  http:
  - match:
    - uri:
        prefix: "/"
    route:
    - destination:
        host: backend.default.svc.cluster.local
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: gateway-error,connect-failure,refused-stream,unavailable,cancelled,resource-exhausted
      retryRemoteLocalities: false
    timeout: 10s
    fault:
      delay:
        percentage:
          value: 0
        fixedDelay: 0s
      abort:
        percentage:
          value: 0
        httpStatus: 503

kubectl apply -f backend-virtual-service.yaml

Create advanced circuit breaker configuration

Configure more sophisticated circuit breaker patterns with different policies for different service endpoints.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backend-advanced-cb
  namespace: default
spec:
  host: backend.default.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 20
        connectTimeout: 10s
        tcpKeepalive:
          time: 7200s
          interval: 75s
          probes: 5
      http:
        http1MaxPendingRequests: 20
        http2MaxRequests: 100
        maxRequestsPerConnection: 50
        maxRetries: 5
        consecutiveGatewayErrors: 3
        h2UpgradePolicy: UPGRADE
        useClientProtocol: true
    outlierDetection:
      consecutive5xxErrors: 2
      consecutiveGatewayErrors: 2
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 80
      minHealthPercent: 20
      splitExternalLocalOriginErrors: true
  portLevelSettings:
  - port:
      number: 80
    connectionPool:
      tcp:
        maxConnections: 15
        connectTimeout: 5s
      http:
        http1MaxPendingRequests: 15
        maxRequestsPerConnection: 25

kubectl apply -f advanced-circuit-breaker.yaml

Configure fault injection for testing

Set up fault injection policies to test circuit breaker and retry behavior under simulated failure conditions.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backend-fault-injection
  namespace: default
spec:
  hosts:
  - backend.default.svc.cluster.local
  http:
  - match:
    - headers:
        test-fault:
          exact: "delay"
    fault:
      delay:
        percentage:
          value: 100
        fixedDelay: 5s
    route:
    - destination:
        host: backend.default.svc.cluster.local
  - match:
    - headers:
        test-fault:
          exact: "abort"
    fault:
      abort:
        percentage:
          value: 100
        httpStatus: 503
    route:
    - destination:
        host: backend.default.svc.cluster.local
  - match:
    - headers:
        test-fault:
          exact: "mixed"
    fault:
      delay:
        percentage:
          value: 30
        fixedDelay: 3s
      abort:
        percentage:
          value: 20
        httpStatus: 500
    route:
    - destination:
        host: backend.default.svc.cluster.local
  - route:
    - destination:
        host: backend.default.svc.cluster.local
    retries:
      attempts: 4
      perTryTimeout: 3s
      retryOn: 5xx,reset,connect-failure,refused-stream
    timeout: 15s

kubectl apply -f fault-injection-vs.yaml

Configure monitoring and observability

Enable Prometheus metrics collection for circuit breaker and retry policy monitoring. This requires Prometheus integration with Istio.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-control-plane
  namespace: istio-system
spec:
  values:
    telemetry:
      v2:
        prometheus:
          configOverride:
            metric_relabeling_configs:
            - source_labels: [__name__]
              regex: 'istio_(requests|request_duration_milliseconds|tcp).*'
              action: keep
            - source_labels: [__name__]
              regex: 'envoy_cluster_upstream_rq_(retry|timeout|pending).*'
              action: keep
            - source_labels: [__name__]
              regex: 'envoy_cluster_outlier_detection_.*'
              action: keep

kubectl apply -f telemetry-config.yaml

Create custom retry policy for specific endpoints

Implement endpoint-specific retry policies with different backoff strategies for various API routes.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backend-endpoint-retries
  namespace: default
spec:
  hosts:
  - backend.default.svc.cluster.local
  http:
  - match:
    - uri:
        prefix: "/api/v1/critical"
    route:
    - destination:
        host: backend.default.svc.cluster.local
    retries:
      attempts: 5
      perTryTimeout: 1s
      retryOn: 5xx,reset,connect-failure,refused-stream,cancelled
      retryRemoteLocalities: true
    timeout: 8s
  - match:
    - uri:
        prefix: "/api/v1/standard"
    route:
    - destination:
        host: backend.default.svc.cluster.local
    retries:
      attempts: 3
      perTryTimeout: 3s
      retryOn: gateway-error,connect-failure,refused-stream
      retryRemoteLocalities: false
    timeout: 12s
  - match:
    - uri:
        prefix: "/api/v1/bulk"
    route:
    - destination:
        host: backend.default.svc.cluster.local
    retries:
      attempts: 2
      perTryTimeout: 10s
      retryOn: connect-failure,refused-stream
    timeout: 30s
  - route:
    - destination:
        host: backend.default.svc.cluster.local

kubectl apply -f endpoint-specific-retries.yaml

Configure traffic shaping for load testing

Set up traffic policies that work with circuit breakers to control request distribution during load testing scenarios.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backend-load-test-policy
  namespace: default
spec:
  host: backend.default.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 50
        connectTimeout: 2s
      http:
        http1MaxPendingRequests: 100
        http2MaxRequests: 200
        maxRequestsPerConnection: 100
        maxRetries: 2
        consecutiveGatewayErrors: 2
    outlierDetection:
      consecutive5xxErrors: 3
      consecutiveGatewayErrors: 3
      interval: 5s
      baseEjectionTime: 15s
      maxEjectionPercent: 70
      minHealthPercent: 30
    loadBalancer:
      simple: LEAST_CONN
      consistentHash:
        httpHeaderName: "x-user-id"
      localityLbSetting:
        enabled: true
        distribute:
        - from: "region1/zone1/*"
          to:
            "region1/zone1/*": 80
            "region1/zone2/*": 20
        failover:
        - from: region1
          to: region2

kubectl apply -f load-test-traffic-policy.yaml

Test circuit breaker behavior

Verify that your circuit breaker configuration works correctly by simulating failures and monitoring the responses.

# Test normal traffic
kubectl exec -it deployment/frontend -- curl -H "Host: backend.default.svc.cluster.local" http://backend/get

Test with delay fault injection
kubectl exec -it deployment/frontend -- curl -H "Host: backend.default.svc.cluster.local" -H "test-fault: delay" http://backend/get

Test with abort fault injection
kubectl exec -it deployment/frontend -- curl -H "Host: backend.default.svc.cluster.local" -H "test-fault: abort" http://backend/get

Generate load to trigger circuit breaker
for i in {1..100}; do
  kubectl exec -it deployment/frontend -- curl -H "Host: backend.default.svc.cluster.local" -H "test-fault: mixed" http://backend/get &
done

Monitor circuit breaker metrics

Check Istio proxy metrics to verify circuit breaker and retry policy effectiveness using Prometheus queries.

# Check circuit breaker status
kubectl exec -it deployment/frontend -c istio-proxy -- curl localhost:15000/stats | grep circuit_breaker

Check retry metrics
kubectl exec -it deployment/frontend -c istio-proxy -- curl localhost:15000/stats | grep retry

Check outlier detection metrics
kubectl exec -it deployment/frontend -c istio-proxy -- curl localhost:15000/stats | grep outlier_detection

View cluster health status
kubectl exec -it deployment/frontend -c istio-proxy -- curl localhost:15000/clusters

Note: Circuit breaker metrics are available through Envoy's admin interface on port 15000. You can also access these through Grafana dashboards if you have Prometheus monitoring configured.

Verify your setup

# Check that destination rules are applied
kubectl get destinationrules -o wide
kubectl describe destinationrule backend-circuit-breaker

Verify virtual service configuration
kubectl get virtualservices -o wide
kubectl describe virtualservice backend-retry-policy

Check Istio proxy configuration
istioctl proxy-config cluster deployment/frontend
istioctl proxy-config route deployment/frontend

Validate circuit breaker settings
istioctl proxy-config cluster deployment/frontend --fqdn backend.default.svc.cluster.local -o json

Common issues

Symptom	Cause	Fix
Circuit breaker not triggering	Thresholds set too high	Lower `consecutive5xxErrors` and `consecutiveGatewayErrors` values
Retries not working	Wrong retry conditions	Check `retryOn` conditions match actual error types
Services ejected too quickly	Aggressive outlier detection	Increase `baseEjectionTime` and reduce `maxEjectionPercent`
Timeout errors during retries	`perTryTimeout` too low	Increase timeout values or reduce retry attempts
Fault injection not working	Header matching issues	Verify header values match exactly in virtual service rules
Metrics not showing up	Telemetry not configured	Enable Prometheus integration and restart Istio proxies

Next steps

Running this in production?

Need this managed? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. Our managed platform covers monitoring, backups and 24/7 response by default.

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash

set -euo pipefail

# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

# Configuration
NAMESPACE="${1:-default}"
CLUSTER_NAME="${2:-istio-demo}"

usage() {
    echo "Usage: $0 [NAMESPACE] [CLUSTER_NAME]"
    echo "  NAMESPACE: Kubernetes namespace (default: default)"
    echo "  CLUSTER_NAME: Cluster name for context (default: istio-demo)"
    exit 1
}

log() {
    echo -e "${GREEN}$1${NC}"
}

warn() {
    echo -e "${YELLOW}$1${NC}"
}

error() {
    echo -e "${RED}$1${NC}" >&2
}

cleanup() {
    warn "Installation failed. Cleaning up..."
    kubectl delete -f /tmp/istio-circuit-breaker/ --ignore-not-found=true 2>/dev/null || true
    rm -rf /tmp/istio-circuit-breaker
}

trap cleanup ERR

# Detect distro and set package manager
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian) PKG_MGR="apt"; PKG_INSTALL="apt install -y"; PKG_UPDATE="apt update" ;;
        almalinux|rocky|centos|rhel|ol|fedora) PKG_MGR="dnf"; PKG_INSTALL="dnf install -y"; PKG_UPDATE="dnf update -y" ;;
        amzn) PKG_MGR="yum"; PKG_INSTALL="yum install -y"; PKG_UPDATE="yum update -y" ;;
        *) error "Unsupported distro: $ID"; exit 1 ;;
    esac
else
    error "Cannot detect OS. /etc/os-release not found."
    exit 1
fi

# Check prerequisites
log "[1/8] Checking prerequisites..."
if [[ $EUID -eq 0 ]]; then
    error "Do not run this script as root. Use a user with sudo privileges."
    exit 1
fi

# Install required packages
log "[2/8] Installing required packages..."
sudo $PKG_UPDATE >/dev/null 2>&1
sudo $PKG_INSTALL curl wget unzip >/dev/null 2>&1

# Check for kubectl
if ! command -v kubectl &> /dev/null; then
    log "Installing kubectl..."
    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
    chmod +x kubectl
    sudo mv kubectl /usr/local/bin/
fi

# Check for istioctl
if ! command -v istioctl &> /dev/null; then
    log "Installing istioctl..."
    curl -L https://istio.io/downloadIstio | sh -
    sudo mv istio-*/bin/istioctl /usr/local/bin/
    rm -rf istio-*
fi

# Verify Kubernetes connection
log "[3/8] Verifying Kubernetes cluster connection..."
if ! kubectl cluster-info &> /dev/null; then
    error "Cannot connect to Kubernetes cluster. Check your kubeconfig."
    exit 1
fi

# Verify Istio installation
log "[4/8] Verifying Istio installation..."
if ! kubectl get namespace istio-system &> /dev/null; then
    error "Istio system namespace not found. Please install Istio first."
    exit 1
fi

if ! kubectl get pods -n istio-system | grep -q "Running"; then
    error "Istio pods are not running properly."
    exit 1
fi

istioctl version --remote=true &> /dev/null || {
    error "Istio control plane is not accessible."
    exit 1
}

# Create working directory
mkdir -p /tmp/istio-circuit-breaker
cd /tmp/istio-circuit-breaker

# Create namespace if it doesn't exist
kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -

# Deploy sample applications
log "[5/8] Deploying sample applications..."
cat << 'EOF' > frontend-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: frontend
        image: nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: backend
        image: kennethreitz/httpbin
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: backend
spec:
  selector:
    app: backend
  ports:
  - port: 80
    targetPort: 80
EOF

kubectl apply -f frontend-app.yaml -n "$NAMESPACE"

# Configure circuit breaker destination rules
log "[6/8] Configuring circuit breaker destination rules..."
cat << EOF > backend-destination-rule.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backend-circuit-breaker
spec:
  host: backend.$NAMESPACE.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 10
        connectTimeout: 30s
        tcpKeepalive:
          time: 7200s
          interval: 75s
      http:
        http1MaxPendingRequests: 10
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
        maxRetries: 3
    outlierDetection:
      consecutive5xxErrors: 3
      consecutiveGatewayErrors: 3
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
      minHealthPercent: 30
      splitExternalLocalOriginErrors: false
EOF

kubectl apply -f backend-destination-rule.yaml -n "$NAMESPACE"

# Implement retry policies
log "[7/8] Implementing retry policies..."
cat << EOF > backend-virtual-service.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backend-retry-policy
spec:
  hosts:
  - backend.$NAMESPACE.svc.cluster.local
  http:
  - match:
    - uri:
        prefix: "/"
    route:
    - destination:
        host: backend.$NAMESPACE.svc.cluster.local
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: gateway-error,connect-failure,refused-stream,unavailable,cancelled,resource-exhausted
      retryRemoteLocalities: false
    timeout: 10s
EOF

kubectl apply -f backend-virtual-service.yaml -n "$NAMESPACE"

# Wait for deployments to be ready
log "Waiting for deployments to be ready..."
kubectl wait --for=condition=available --timeout=300s deployment/frontend -n "$NAMESPACE"
kubectl wait --for=condition=available --timeout=300s deployment/backend -n "$NAMESPACE"

# Verification
log "[8/8] Verifying installation..."
echo "Checking pod status:"
kubectl get pods -n "$NAMESPACE" -l app=frontend
kubectl get pods -n "$NAMESPACE" -l app=backend

echo "Checking destination rules:"
kubectl get destinationrules -n "$NAMESPACE"

echo "Checking virtual services:"
kubectl get virtualservices -n "$NAMESPACE"

# Test circuit breaker functionality
log "Testing circuit breaker configuration..."
frontend_pod=$(kubectl get pod -l app=frontend -n "$NAMESPACE" -o jsonpath='{.items[0].metadata.name}')

if kubectl exec "$frontend_pod" -n "$NAMESPACE" -- curl -s backend/status/200 | grep -q "200"; then
    log "✓ Circuit breaker and retry policies configured successfully"
else
    warn "! Backend service may not be fully ready yet"
fi

log "Installation completed successfully!"
echo ""
echo "Next steps:"
echo "1. Test circuit breaker: kubectl exec -it $frontend_pod -n $NAMESPACE -- curl backend/status/503"
echo "2. Monitor with: kubectl logs -f deployment/frontend -n $NAMESPACE"
echo "3. View Istio config: istioctl proxy-config cluster $frontend_pod -n $NAMESPACE"

# Cleanup temp files
rm -rf /tmp/istio-circuit-breaker

Review the script before running. Execute with: bash install.sh

#istio #circuit-breaker #retry-policies #kubernetes #microservices

Implement Istio circuit breaker and retry policies for microservices resilience and fault tolerance

Prerequisites

What this solves

Step-by-step configuration

Verify Istio installation

Deploy sample applications

Configure circuit breaker destination rules

Implement retry policies with exponential backoff

Create advanced circuit breaker configuration

Configure fault injection for testing

Configure monitoring and observability

Create custom retry policy for specific endpoints

Configure traffic shaping for load testing

Test circuit breaker behavior

Test with delay fault injection

Test with abort fault injection

Generate load to trigger circuit breaker

Monitor circuit breaker metrics

Check retry metrics

Check outlier detection metrics

View cluster health status

Verify your setup

Verify virtual service configuration

Check Istio proxy configuration

Validate circuit breaker settings

Common issues

Next steps

Running this in production?

Related tutorials

Set up Kafka Streams testing framework with TopologyTestDriver for automated stream processing validation

Configure Consul multi-datacenter WAN federation for geographic redundancy

Configure centralized cron management with Ansible automation and systemd timers

Don't want to manage this yourself?