Monitor Kubernetes network policies with Prometheus and Grafana for enhanced cluster security

Install Prometheus Operator

Deploy the Prometheus Operator to manage Prometheus instances and monitoring resources in your cluster.

kubectl create namespace monitoring

Add the Prometheus community Helm repository and install the operator:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus-operator prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false

Configure Cilium metrics (if using Cilium)

Enable Cilium metrics collection for network policy monitoring. Create a Cilium configuration to expose policy enforcement metrics:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
  enable-prometheus-serve: "true"
  prometheus-serve-addr: ":9962"
  operator-prometheus-serve-addr: ":9963"
  enable-policy-verdict: "true"
  monitor-aggregation: medium
  enable-hubble: "true"
  hubble-metrics-server: ":9965"
  hubble-metrics: "dns,drop,tcp,flow,icmp,http"

Apply the configuration and restart Cilium:

kubectl apply -f cilium-metrics.yaml
kubectl rollout restart daemonset/cilium -n kube-system

Configure Calico metrics (if using Calico)

For Calico deployments, enable Felix metrics to monitor network policy enforcement:

apiVersion: v1
kind: ConfigMap
metadata:
  name: calico-config
  namespace: kube-system
data:
  calico_backend: "bird"
  cluster_type: "k8s,bgp"
  felix_prometheusmetricsenabled: "true"
  felix_prometheusmetricsport: "9091"
  felix_reportingintervalsecs: "0"
  felix_reportingttlsecs: "0"

Apply the configuration:

kubectl apply -f calico-metrics.yaml
kubectl rollout restart daemonset/calico-node -n kube-system

Create ServiceMonitor for CNI metrics

Configure Prometheus to scrape CNI metrics by creating appropriate ServiceMonitor resources. For Cilium:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cilium-agent
  namespace: monitoring
spec:
  selector:
    matchLabels:
      k8s-app: cilium
  endpoints:
  - port: prometheus
    interval: 30s
    path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hubble-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      k8s-app: hubble-metrics
  endpoints:
  - port: hubble-metrics
    interval: 30s
    path: /metrics

For Calico, create this ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: calico-felix
  namespace: monitoring
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  endpoints:
  - port: calico-metrics
    interval: 30s
    path: /metrics

Apply the appropriate ServiceMonitor:

kubectl apply -f cilium-servicemonitor.yaml
OR for Calico:
kubectl apply -f calico-servicemonitor.yaml

Create Grafana dashboards

Import network policy monitoring dashboards to visualize CNI metrics. Create a ConfigMap with dashboard JSON:

apiVersion: v1
kind: ConfigMap
metadata:
  name: network-policy-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  network-policy.json: |
    {
      "dashboard": {
        "id": null,
        "title": "Network Policy Monitoring",
        "tags": ["kubernetes", "network", "security"],
        "timezone": "browser",
        "panels": [
          {
            "id": 1,
            "title": "Policy Verdicts",
            "type": "stat",
            "targets": [
              {
                "expr": "sum(rate(cilium_policy_verdict_total[5m])) by (verdict)",
                "legendFormat": "{{verdict}}"
              }
            ],
            "fieldConfig": {
              "defaults": {
                "color": {
                  "mode": "palette-classic"
                }
              }
            },
            "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}
          },
          {
            "id": 2,
            "title": "Denied Connections by Source",
            "type": "table",
            "targets": [
              {
                "expr": "topk(10, sum(rate(cilium_policy_verdict_total{verdict=\"DENIED\"}[5m])) by (source, destination))",
                "format": "table",
                "instant": true
              }
            ],
            "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
          }
        ],
        "time": {
          "from": "now-1h",
          "to": "now"
        },
        "refresh": "30s"
      }
    }

Apply the dashboard:

kubectl apply -f network-policy-dashboard.yaml

Configure alerting rules

Create Prometheus alerting rules for network policy violations and security events:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: network-policy-alerts
  namespace: monitoring
spec:
  groups:
  - name: network-policy.rules
    interval: 30s
    rules:
    - alert: HighPolicyViolationRate
      expr: rate(cilium_policy_verdict_total{verdict="DENIED"}[5m]) > 10
      for: 2m
      labels:
        severity: warning
        component: network-policy
      annotations:
        summary: "High network policy violation rate detected"
        description: "Network policy is denying {{ $value }} connections per second for {{ $labels.source }} to {{ $labels.destination }}"
    
    - alert: SuspiciousTrafficPattern
      expr: |
        (
          rate(cilium_policy_verdict_total{verdict="DENIED"}[10m]) > 
          rate(cilium_policy_verdict_total{verdict="DENIED"}[1h] offset 1h) * 3
        )
      for: 5m
      labels:
        severity: critical
        component: network-policy
      annotations:
        summary: "Suspicious traffic pattern detected"
        description: "Denied connection rate has increased 3x compared to the same time yesterday"
    
    - alert: NetworkPolicyEngineDown
      expr: up{job="cilium-agent"} == 0
      for: 1m
      labels:
        severity: critical
        component: network-policy
      annotations:
        summary: "CNI policy engine is down"
        description: "Cilium agent on {{ $labels.instance }} has been down for more than 1 minute"
    
    - alert: PolicyLoadErrors
      expr: rate(cilium_policy_regeneration_total{outcome="fail"}[5m]) > 0
      for: 2m
      labels:
        severity: warning
        component: network-policy
      annotations:
        summary: "Network policy load errors"
        description: "Policy regeneration failures detected on {{ $labels.instance }}"
    
    - alert: UnexpectedAllowedConnections
      expr: |
        rate(cilium_policy_verdict_total{verdict="ALLOWED"}[5m]) > 
        quantile_over_time(0.95, rate(cilium_policy_verdict_total{verdict="ALLOWED"}[5m])[7d:1h]) * 1.5
      for: 10m
      labels:
        severity: warning
        component: network-policy
      annotations:
        summary: "Unusual allowed connection volume"
        description: "Allowed connections are 50% higher than the 95th percentile over the last week"

Apply the alerting rules:

kubectl apply -f network-policy-alerts.yaml

Configure Alertmanager for notifications

Set up Alertmanager to send notifications for network policy alerts. You can integrate with Slack notifications or email:

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-main
  namespace: monitoring
stringData:
  alertmanager.yml: |
    global:
      smtp_smarthost: 'localhost:587'
      smtp_from: 'alerts@example.com'
    
    route:
      group_by: ['alertname', 'component']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'network-security-team'
      routes:
      - match:
          component: network-policy
        receiver: 'network-security-team'
        group_wait: 10s
        repeat_interval: 5m
    
    receivers:
    - name: 'network-security-team'
      email_configs:
      - to: 'security@example.com'
        subject: 'Network Policy Alert: {{ .GroupLabels.alertname }}'
        body: |
          {{ range .Alerts }}
          Alert: {{ .Annotations.summary }}
          Description: {{ .Annotations.description }}
          {{ end }}

Apply the Alertmanager configuration:

kubectl apply -f alertmanager-config.yaml

Deploy sample network policies for testing

Create test network policies to generate metrics and validate monitoring setup:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: nginx:alpine
        ports:
        - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: backend
        image: nginx:alpine
        ports:
        - containerPort: 8080

Deploy the test resources:

kubectl apply -f test-network-policies.yaml

Access Grafana dashboard

Port-forward to access Grafana and view your network policy dashboards:

kubectl port-forward -n monitoring svc/prometheus-operator-grafana 3000:80

Access Grafana at http://localhost:3000. The default credentials are admin/prom-operator. Navigate to the "Network Policy Monitoring" dashboard to view metrics.

Configure custom metrics retention

Set up longer retention for network policy metrics to track security trends over time:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: kube-prometheus-stack-kube-prom-prometheus
  namespace: monitoring
spec:
  retention: "90d"
  retentionSize: "50GB"
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi

Apply the retention configuration:

kubectl apply -f prometheus-retention.yaml

Create security compliance dashboard

Build a compliance-focused dashboard for security audits:

apiVersion: v1
kind: ConfigMap
metadata:
  name: compliance-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  compliance.json: |
    {
      "dashboard": {
        "title": "Network Security Compliance",
        "panels": [
          {
            "title": "Policy Coverage by Namespace",
            "type": "piechart",
            "targets": [{
              "expr": "count by (namespace) (kube_networkpolicy_info)"
            }]
          },
          {
            "title": "Monthly Violation Trends",
            "type": "graph",
            "targets": [{
              "expr": "increase(cilium_policy_verdict_total{verdict=\"DENIED\"}[30d])"
            }]
          }
        ]
      }
    }