Monitor Istio service mesh with Prometheus and Grafana dashboards

Install Prometheus operator

Deploy the Prometheus operator to manage Prometheus instances and monitoring configurations in your Kubernetes cluster.

kubectl create namespace monitoring
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.71.0/bundle.yaml

Configure Prometheus for Istio metrics

Create a Prometheus instance specifically configured to scrape Istio control plane and data plane metrics.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: istio-prometheus
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      app: istio-proxy
  ruleSelector:
    matchLabels:
      app: istio
  resources:
    requests:
      memory: 400Mi
      cpu: 100m
    limits:
      memory: 2Gi
      cpu: 1000m
  retention: 7d
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: default
        resources:
          requests:
            storage: 50Gi
  additionalScrapeConfigs:
    name: istio-scrape-configs
    key: prometheus.yaml

Create Prometheus RBAC configuration

Set up service account and RBAC permissions for Prometheus to access Istio metrics endpoints across namespaces.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
kind: ServiceAccount
  name: prometheus
  namespace: monitoring

Configure Istio service monitors

Create ServiceMonitor resources to automatically discover and scrape Istio components including istiod, gateways, and sidecars.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-control-plane
  namespace: monitoring
  labels:
    app: istio-proxy
spec:
  selector:
    matchLabels:
      app: istiod
  namespaceSelector:
    matchNames:
    - istio-system
  endpoints:
  - port: http-monitoring
    interval: 15s
    path: /stats/prometheus
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-proxy
  namespace: monitoring
  labels:
    app: istio-proxy
spec:
  selector:
    matchExpressions:
    - key: security.istio.io/tlsMode
      operator: Exists
  namespaceSelector:
    any: true
  endpoints:
  - port: http-envoy-prom
    interval: 15s
    path: /stats/prometheus
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_name]
      targetLabel: pod_name
    - sourceLabels: [__meta_kubernetes_namespace]
      targetLabel: namespace

Apply monitoring configurations

Deploy all monitoring components to your cluster and verify they're running correctly.

kubectl apply -f prometheus-rbac.yaml
kubectl apply -f istio-prometheus.yaml
kubectl apply -f istio-service-monitors.yaml
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=prometheus -n monitoring --timeout=300s

Install Grafana

Deploy Grafana with persistent storage and configure it to use the Prometheus instance as a data source.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:10.2.2
        ports:
        - containerPort: 3000
        env:
        - name: GF_SECURITY_ADMIN_PASSWORD
          value: "admin123!@#"
        - name: GF_INSTALL_PLUGINS
          value: "grafana-piechart-panel"
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/lib/grafana
        - name: grafana-config
          mountPath: /etc/grafana/provisioning
        resources:
          requests:
            memory: 256Mi
            cpu: 100m
          limits:
            memory: 512Mi
            cpu: 500m
      volumes:
      - name: grafana-storage
        persistentVolumeClaim:
          claimName: grafana-pvc
      - name: grafana-config
        configMap:
          name: grafana-config
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: default

Configure Grafana data source

Create a ConfigMap to automatically provision Prometheus as a data source in Grafana.

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-config
  namespace: monitoring
data:
  datasources.yml: |
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      access: proxy
      url: http://prometheus-operated:9090
      isDefault: true
      editable: false
  dashboards.yml: |
    apiVersion: 1
    providers:
    - name: 'istio'
      orgId: 1
      folder: 'Istio'
      type: file
      disableDeletion: false
      updateIntervalSeconds: 10
      options:
        path: /var/lib/grafana/dashboards

Deploy Grafana service

Create a service to expose Grafana and apply all configurations to make it accessible.

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector:
    app: grafana
  ports:
  - name: http
    port: 3000
    targetPort: 3000
  type: LoadBalancer

kubectl apply -f grafana-config.yaml
kubectl apply -f grafana-deployment.yaml
kubectl apply -f grafana-service.yaml

Install Istio dashboard templates

Download and configure the official Istio Grafana dashboards for comprehensive service mesh monitoring.

mkdir -p /tmp/istio-dashboards
cd /tmp/istio-dashboards

Download official Istio dashboards
curl -L https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/grafana.yaml -o istio-grafana-dashboards.yaml

Extract dashboard JSON from the configmap
kubectl apply -f istio-grafana-dashboards.yaml
kubectl get configmap istio-grafana-dashboards -n istio-system -o yaml | grep -A 10000 'istio-mesh-dashboard.json:' | tail -n +2 > mesh-dashboard.json
kubectl get configmap istio-grafana-dashboards -n istio-system -o yaml | grep -A 10000 'istio-service-dashboard.json:' | tail -n +2 > service-dashboard.json
kubectl get configmap istio-grafana-dashboards -n istio-system -o yaml | grep -A 10000 'istio-workload-dashboard.json:' | tail -n +2 > workload-dashboard.json

Create dashboard ConfigMap

Create a ConfigMap containing the Istio dashboards so Grafana can automatically load them.

kubectl create configmap istio-dashboards \
  --from-file=mesh-dashboard.json \
  --from-file=service-dashboard.json \
  --from-file=workload-dashboard.json \
  -n monitoring

Mount the dashboards in Grafana
kubectl patch deployment grafana -n monitoring --patch '{
  "spec": {
    "template": {
      "spec": {
        "containers": [{
          "name": "grafana",
          "volumeMounts": [{
            "name": "istio-dashboards",
            "mountPath": "/var/lib/grafana/dashboards"
          }]
        }],
        "volumes": [{
          "name": "istio-dashboards",
          "configMap": {
            "name": "istio-dashboards"
          }
        }]
      }
    }
  }
}'

Configure alerting rules

Set up PrometheusRule resources to define alerting conditions for Istio service mesh health and performance issues.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: istio-alerts
  namespace: monitoring
  labels:
    app: istio
spec:
  groups:
  - name: istio.rules
    rules:
    - alert: IstioHighRequestRate
      expr: sum(rate(istio_requests_total[5m])) > 1000
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "High request rate detected"
        description: "Request rate is {{ $value }} requests per second"
    
    - alert: IstioHighErrorRate
      expr: sum(rate(istio_requests_total{response_code!~"2.."}[5m])) / sum(rate(istio_requests_total[5m])) > 0.1
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "High error rate detected"
        description: "Error rate is {{ $value | humanizePercentage }}"
    
    - alert: IstioPilotPushErrors
      expr: sum(rate(pilot_xds_push_errors[5m])) > 5
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: "Istio Pilot push errors"
        description: "Pilot is experiencing {{ $value }} push errors per second"
    
    - alert: IstioProxyConfigurationDrift
      expr: sum(pilot_k8s_cfg_events{type="warning"}) > 0
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: "Istio proxy configuration warnings"
        description: "{{ $value }} configuration warnings detected"

Apply alerting configuration

Deploy the alerting rules and restart Grafana to ensure all configurations are loaded.

kubectl apply -f istio-alerts.yaml
kubectl rollout restart deployment/grafana -n monitoring
kubectl wait --for=condition=available deployment/grafana -n monitoring --timeout=300s

Enable traffic generation for testing

Deploy a sample application with Istio injection to generate metrics for dashboard testing. This tutorial assumes you have an Istio-enabled cluster as covered in our Istio installation guide.

kubectl create namespace bookinfo
kubectl label namespace bookinfo istio-injection=enabled
kubectl apply -n bookinfo -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -n bookinfo -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/bookinfo/networking/bookinfo-gateway.yaml

Set up Grafana authentication

Configure proper authentication and access controls for your Grafana instance in production environments.

# Get Grafana external IP
GRAFANA_IP=$(kubectl get svc grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "Grafana URL: http://$GRAFANA_IP:3000"
echo "Username: admin"
echo "Password: admin123!@#"

Create additional Grafana users via API
curl -X POST http://admin:admin123!@#@$GRAFANA_IP:3000/api/admin/users \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Istio Viewer",
    "email": "viewer@example.com",
    "login": "istio-viewer",
    "password": "ViewerPass123!",
    "role": "Viewer"
  }'