Implement Thanos Ruler for distributed alerting and recording rules across Prometheus clusters

Create a dedicated system user

Run Thanos Ruler as an unprivileged user, never as root.

sudo groupadd --system thanos
sudo useradd --system --gid thanos --no-create-home --shell /usr/sbin/nologin thanos

Download and install the Thanos binary

Thanos ships as a single static binary containing all components, including Ruler.

curl -L -o thanos.tar.gz https://github.com/thanos-io/thanos/releases/download/v0.36.1/thanos-0.36.1.linux-amd64.tar.gz
tar -xzf thanos.tar.gz
sudo mv thanos-0.36.1.linux-amd64/thanos /usr/local/bin/thanos
sudo chown root:root /usr/local/bin/thanos
sudo chmod 755 /usr/local/bin/thanos
thanos --version

Create directories for rules, data, and config

Ruler needs a local data directory for its TSDB block staging area and a rules directory for rule files.

sudo mkdir -p /etc/thanos/rules /etc/thanos/tls /var/lib/thanos-ruler
sudo chown -R thanos:thanos /etc/thanos /var/lib/thanos-ruler
sudo chmod 750 /var/lib/thanos-ruler
sudo chmod 750 /etc/thanos/rules

Write a cross-cluster recording rule

Recording rules precompute expensive aggregations. This example rolls up HTTP error rate across every cluster labeled with a cluster label injected by each Prometheus's external_labels.

groups:
  - name: global-http-slo
    interval: 30s
    rules:
      - record: job:http_requests_error_rate:ratio_rate5m
        expr: |
          sum by (job, cluster) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job, cluster) (rate(http_requests_total[5m]))
      - record: global:http_requests_error_rate:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))

Write a cross-cluster alerting rule

This alert fires only when the global error rate across all clusters exceeds a threshold, something a single cluster's Prometheus cannot evaluate on its own.

groups:
  - name: global-alerts
    interval: 30s
    rules:
      - alert: GlobalHighErrorRate
        expr: global:http_requests_error_rate:ratio_rate5m > 0.05
        for: 10m
        labels:
          severity: critical
          scope: global
        annotations:
          summary: "Global HTTP error rate above 5% across all clusters"
          description: "Aggregate 5xx error rate is {{ $value | humanizePercentage }} across all connected Prometheus clusters."
      - alert: ClusterMissingFromQuery
        expr: absent(up{job="prometheus", cluster="eu-west-1"})
        for: 5m
        labels:
          severity: warning
          scope: global
        annotations:
          summary: "Cluster eu-west-1 has stopped reporting to Thanos Query"

Point Ruler at Query endpoints

Ruler evaluates rules by querying one or more Thanos Query instances over HTTP, and can also query multiple endpoints directly for failover.

THANOS_QUERY_ENDPOINTS="--query=203.0.113.10:10904 --query=203.0.113.11:10904"
THANOS_LABELS="--label=ruler_replica=\"ruler-1\" --label=cluster=\"global\""

Create the systemd unit

This unit runs Ruler with rules, TSDB block config, and Query discovery. Adjust the IP addresses in --query to match your Thanos Query instances.

[Unit]
Description=Thanos Ruler
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos rule \
  --data-dir=/var/lib/thanos-ruler \
  --rule-file=/etc/thanos/rules/*.yml \
  --eval-interval=30s \
  --query=203.0.113.10:10904 \
  --query=203.0.113.11:10904 \
  --alertmanagers.url=http://203.0.113.20:9093 \
  --alertmanagers.url=http://203.0.113.21:9093 \
  --alert.query-url=http://203.0.113.30:10902 \
  --label=ruler_replica="ruler-1" \
  --label=cluster="global" \
  --objstore.config-file=/etc/thanos/objstore.yml \
  --http-address=0.0.0.0:10904 \
  --grpc-address=0.0.0.0:10901 \
  --tsdb.retention=48h
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

The --alert.query-url flag sets the link embedded in alerts so recipients can jump straight to the query that triggered them in the Thanos Query UI.

Configure Alertmanager targets

Ruler pushes alerts to every Alertmanager listed via --alertmanagers.url. It automatically deduplicates alerts sent to multiple Alertmanager replicas because Alertmanager's own gossip protocol handles that layer.

Verify the alert route accepts global-scope alerts

Add a route in Alertmanager's config that matches the scope: global label used in the rule file above, so these alerts can be routed to a different receiver (e.g. a platform team channel) than per-cluster alerts.

route:
  receiver: default
  routes:
    - match:
        scope: global
      receiver: platform-team
      group_by: ['alertname']
      group_wait: 30s

receivers:
  - name: default
    slack_configs:
      - channel: '#alerts'
  - name: platform-team
    slack_configs:
      - channel: '#platform-global-alerts'

For webhook or PagerDuty routing patterns, see Configure Prometheus Alertmanager with custom webhook integrations for Slack, Microsoft Teams, and PagerDuty notifications.

Create an S3 or MinIO bucket for rule evaluation blocks

Ruler uploads completed TSDB blocks the same way Sidecar does, so it needs its own bucket path or a shared bucket with a distinct prefix.

mc alias set thanosminio https://minio.example.com minioadmin 'ChangeThisStrongSecret123!'
mc mb thanosminio/thanos-ruler-blocks
mc anonymous set none thanosminio/thanos-ruler-blocks

Write the object storage config file

This file is shared in structure with the one used by Thanos Query's Store Gateway and Compactor.

type: S3
config:
  bucket: thanos-ruler-blocks
  endpoint: minio.example.com:443
  access_key: minioadmin
  secret_key: ChangeThisStrongSecret123!
  insecure: false
  signature_version2: false
  http_config:
    tls_config:
      insecure_skip_verify: false

sudo chown thanos:thanos /etc/thanos/objstore.yml
sudo chmod 640 /etc/thanos/objstore.yml

If you're running MinIO for this backend, review Setup MinIO monitoring with Prometheus and Grafana dashboards to keep an eye on bucket usage and latency.

Start Ruler

sudo systemctl daemon-reload
sudo systemctl enable --now thanos-ruler
sudo systemctl status thanos-ruler

Run two identical Ruler replicas

Deploy a second Ruler instance on a separate host with the same rule files but a different ruler_replica label. Both replicas evaluate the same rules independently against the same Query endpoints.

rsync -avz --delete /etc/thanos/rules/ thanos-ruler-2:/etc/thanos/rules/

On the second host, change only the replica label in the systemd unit:

  --label=ruler_replica="ruler-2" \
  --label=cluster="global" \

Enable deduplication at Thanos Query

Thanos Query deduplicates series with matching labels except for the replica label you specify with --query.replica-label. This means alerts and recording rule outputs from both Ruler replicas collapse into a single series when read back through Query.

thanos query \
  --http-address=0.0.0.0:10902 \
  --grpc-address=0.0.0.0:10901 \
  --query.replica-label=ruler_replica \
  --query.replica-label=replica \
  --endpoint=203.0.113.30:10901 \
  --endpoint=203.0.113.31:10901

Both Ruler replicas will independently fire the same alert and push it to Alertmanager. Alertmanager's own deduplication and grouping, based on the alert's label set minus the replica label, collapses duplicate notifications into one.

Register Ruler as a StoreAPI in Query

Add each Ruler's gRPC address as an endpoint on Thanos Query so historical rule evaluation results become queryable through the same Query layer.

thanos query \
  --endpoint=203.0.113.30:10901 \
  --endpoint=203.0.113.31:10901 \
  --endpoint=203.0.113.40:10901 \
  --endpoint=203.0.113.41:10901

Here the first two endpoints are Rulers and the last two are Sidecars or Receivers from your existing clusters.

Scrape Ruler's own metrics

Ruler exposes Prometheus metrics on its HTTP address, including rule evaluation duration, failures, and alert send errors.

scrape_configs:
  - job_name: thanos-ruler
    static_configs:
      - targets:
          - 203.0.113.30:10904
          - 203.0.113.31:10904

Alert on rule evaluation failures

These meta-alerts catch cases where Ruler itself is unhealthy, which is critical since a broken Ruler means silent gaps in your global alerting.

groups:
  - name: thanos-ruler-health
    rules:
      - alert: ThanosRuleEvaluationFailures
        expr: rate(prometheus_rule_evaluation_failures_total[5m]) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Thanos Ruler is failing to evaluate rules"
      - alert: ThanosRuleAlertmanagerSendFailures
        expr: rate(thanos_alert_sender_alerts_dropped_total[5m]) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Thanos Ruler cannot deliver alerts to Alertmanager"
      - alert: ThanosRuleHighRuleEvaluationLatency
        expr: histogram_quantile(0.99, rate(prometheus_rule_evaluation_duration_seconds_bucket[5m])) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Rule evaluation p99 latency exceeds 5 seconds"

For dashboarding these metrics, reuse the panel patterns from Configure advanced Grafana dashboards and alerting with Prometheus integration.