Configure Prometheus Blackbox Exporter for endpoint monitoring with SSL and alerting

Intermediate 25 min Apr 22, 2026 18 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Set up Prometheus Blackbox Exporter to monitor HTTP, HTTPS, DNS, TCP, and ICMP endpoints with SSL certificate validation and automated alerting for comprehensive uptime monitoring.

Prerequisites

  • Root or sudo access
  • Prometheus server installed
  • Basic understanding of YAML configuration

What this solves

Prometheus Blackbox Exporter lets you monitor external services from the outside perspective, checking if your websites, APIs, and network services are actually reachable by users. It probes HTTP/HTTPS endpoints, validates SSL certificates, tests DNS resolution, and monitors TCP connections, giving you visibility into the end-user experience rather than just internal server metrics.

Step-by-step installation

Create blackbox exporter user

Create a dedicated system user to run the Blackbox Exporter service securely.

sudo useradd --system --no-create-home --shell /bin/false blackbox_exporter

Download and install Blackbox Exporter

Download the latest release of Blackbox Exporter from the official GitHub repository.

cd /tmp
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.24.0/blackbox_exporter-0.24.0.linux-amd64.tar.gz
tar xzf blackbox_exporter-0.24.0.linux-amd64.tar.gz
sudo cp blackbox_exporter-0.24.0.linux-amd64/blackbox_exporter /usr/local/bin/
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
sudo chmod 755 /usr/local/bin/blackbox_exporter

Create configuration directory

Set up the directory structure for Blackbox Exporter configuration files.

sudo mkdir -p /etc/blackbox_exporter
sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter
sudo chmod 755 /etc/blackbox_exporter

Configure Blackbox Exporter modules

Create the main configuration file that defines different probing modules for HTTP, HTTPS, DNS, and TCP monitoring.

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: true
      fail_if_ssl: false
      fail_if_not_ssl: false
      preferred_ip_protocol: "ip4"
  
  https_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: true
      fail_if_ssl: false
      fail_if_not_ssl: true
      preferred_ip_protocol: "ip4"
      tls_config:
        insecure_skip_verify: false
  
  ssl_expiry:
    prober: http
    timeout: 5s
    http:
      method: GET
      fail_if_not_ssl: true
      preferred_ip_protocol: "ip4"
      tls_config:
        insecure_skip_verify: false
  
  dns_query:
    prober: dns
    timeout: 5s
    dns:
      query_name: "example.com"
      query_type: "A"
      valid_rcodes:
      - NOERROR
      validate_answer_rrs:
        fail_if_matches_regexp:
        - ".*127.0.0.1"
        fail_if_not_matches_regexp:
        - ".*"
      preferred_ip_protocol: "ip4"
  
  tcp_connect:
    prober: tcp
    timeout: 5s
    tcp:
      preferred_ip_protocol: "ip4"
  
  icmp_ping:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"

Set configuration file permissions

Ensure the configuration file has correct ownership and permissions for security.

sudo chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/config.yml
sudo chmod 644 /etc/blackbox_exporter/config.yml

Create systemd service

Set up a systemd service to manage the Blackbox Exporter process with automatic startup and restart capabilities.

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=blackbox_exporter
Group=blackbox_exporter
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox_exporter/config.yml \
  --web.listen-address=:9115 \
  --log.level=info
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Enable and start Blackbox Exporter

Start the Blackbox Exporter service and enable it to start automatically on boot.

sudo systemctl daemon-reload
sudo systemctl enable blackbox_exporter
sudo systemctl start blackbox_exporter

Configure firewall access

Allow access to the Blackbox Exporter port from your Prometheus server.

sudo ufw allow from 203.0.113.10 to any port 9115 comment "Prometheus server access to Blackbox Exporter"
sudo ufw reload
sudo firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='203.0.113.10' port protocol='tcp' port='9115' accept"
sudo firewall-cmd --reload
Note: Replace 203.0.113.10 with your actual Prometheus server IP address.

Configure Prometheus integration

Add Blackbox Exporter job to Prometheus

Configure Prometheus to scrape metrics from Blackbox Exporter and define monitoring targets.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "/etc/prometheus/rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'blackbox_exporter'
    static_configs:
      - targets: ['localhost:9115']

  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - http://example.com
        - http://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  - job_name: 'blackbox_https'
    metrics_path: /probe
    params:
      module: [https_2xx]
    static_configs:
      - targets:
        - https://example.com
        - https://api.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  - job_name: 'blackbox_ssl_expiry'
    metrics_path: /probe
    params:
      module: [ssl_expiry]
    static_configs:
      - targets:
        - https://example.com
        - https://api.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115
    scrape_interval: 60s

  - job_name: 'blackbox_tcp'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
        - example.com:80
        - example.com:443
        - example.com:22
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Create alerting rules

Set up alerting rules for endpoint monitoring failures and SSL certificate expiration.

groups:
  - name: blackbox_alerts
    rules:
    - alert: EndpointDown
      expr: probe_success == 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Endpoint {{ $labels.instance }} is down"
        description: "The endpoint {{ $labels.instance }} has been down for more than 2 minutes."

    - alert: HttpStatusCodeNot200
      expr: probe_http_status_code != 200
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: "HTTP status code is not 200 for {{ $labels.instance }}"
        description: "HTTP status code is {{ $value }} for {{ $labels.instance }}."

    - alert: SSLCertExpiringSoon
      expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: "SSL certificate for {{ $labels.instance }} expires soon"
        description: "SSL certificate for {{ $labels.instance }} expires in {{ $value | humanizeDuration }}."

    - alert: SSLCertExpired
      expr: probe_ssl_earliest_cert_expiry - time() <= 0
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: "SSL certificate for {{ $labels.instance }} has expired"
        description: "SSL certificate for {{ $labels.instance }} has expired."

    - alert: SlowHttpResponse
      expr: probe_http_duration_seconds > 5
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "Slow HTTP response for {{ $labels.instance }}"
        description: "HTTP request took {{ $value }}s to complete for {{ $labels.instance }}."

    - alert: TcpConnectFailure
      expr: probe_success{job="blackbox_tcp"} == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "TCP connection failed for {{ $labels.instance }}"
        description: "TCP connection to {{ $labels.instance }} has been failing for more than 1 minute."

Set rule file permissions

Ensure the alerting rules file has proper permissions for Prometheus to read it.

sudo mkdir -p /etc/prometheus/rules
sudo chown -R prometheus:prometheus /etc/prometheus/rules
sudo chmod 644 /etc/prometheus/rules/blackbox.yml

Restart Prometheus

Reload Prometheus configuration to apply the new Blackbox Exporter integration and alerting rules.

sudo systemctl reload prometheus
sudo systemctl status prometheus

Set up Grafana dashboards

Import Blackbox Exporter dashboard

Add a comprehensive dashboard for visualizing endpoint monitoring metrics in Grafana.

curl -X POST \
  http://admin:admin@localhost:3000/api/dashboards/db \
  -H 'Content-Type: application/json' \
  -d '{
    "dashboard": {
      "id": null,
      "title": "Blackbox Exporter Overview",
      "tags": ["blackbox", "monitoring"],
      "timezone": "browser",
      "panels": [
        {
          "id": 1,
          "title": "Endpoint Status",
          "type": "stat",
          "targets": [
            {
              "expr": "probe_success",
              "legendFormat": "{{ instance }}"
            }
          ],
          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
          "options": {
            "colorMode": "background",
            "graphMode": "area",
            "justifyMode": "auto",
            "orientation": "auto"
          },
          "fieldConfig": {
            "defaults": {
              "mappings": [
                {
                  "options": {
                    "0": {"text": "DOWN", "color": "red"},
                    "1": {"text": "UP", "color": "green"}
                  },
                  "type": "value"
                }
              ],
              "thresholds": {
                "steps": [
                  {"color": "red", "value": 0},
                  {"color": "green", "value": 1}
                ]
              }
            }
          }
        },
        {
          "id": 2,
          "title": "HTTP Response Times",
          "type": "timeseries",
          "targets": [
            {
              "expr": "probe_http_duration_seconds",
              "legendFormat": "{{ instance }}"
            }
          ],
          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
          "fieldConfig": {
            "defaults": {
              "unit": "s"
            }
          }
        }
      ],
      "time": {
        "from": "now-1h",
        "to": "now"
      },
      "refresh": "5s"
    },
    "overwrite": true
  }'

Configure alerting notifications

Set up Alertmanager configuration

Configure Alertmanager to send notifications when endpoints fail or SSL certificates expire. First, ensure you have the Prometheus and Grafana monitoring stack running as described in our Prometheus and Grafana setup guide.

global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alerts@example.com'
  smtp_auth_username: 'alerts@example.com'
  smtp_auth_password: 'your-app-password'

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'web.hook'
  routes:
  - match:
      severity: critical
    receiver: 'critical-alerts'
  - match:
      severity: warning
    receiver: 'warning-alerts'

receivers:
  • name: 'web.hook'
webhook_configs: - url: 'http://127.0.0.1:5001/'
  • name: 'critical-alerts'
email_configs: - to: 'admin@example.com' subject: 'CRITICAL: {{ .GroupLabels.alertname }}' body: | {{ range .Alerts }} Alert: {{ .Annotations.summary }} Description: {{ .Annotations.description }} Labels: {{ .Labels }} {{ end }}
  • name: 'warning-alerts'
email_configs: - to: 'monitoring@example.com' subject: 'WARNING: {{ .GroupLabels.alertname }}' body: | {{ range .Alerts }} Alert: {{ .Annotations.summary }} Description: {{ .Annotations.description }} {{ end }} inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname', 'instance']

Restart Alertmanager

Apply the new Alertmanager configuration to enable email notifications for endpoint failures.

sudo systemctl restart alertmanager
sudo systemctl status alertmanager

Advanced monitoring configurations

Add custom HTTP headers and authentication

Configure modules for endpoints that require authentication or custom headers.

modules:
  http_with_auth:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      headers:
        Authorization: "Bearer your-api-token"
        User-Agent: "Blackbox-Exporter/1.0"
      follow_redirects: false
      preferred_ip_protocol: "ip4"

  http_post_json:
    prober: http
    timeout: 5s
    http:
      method: POST
      headers:
        Content-Type: "application/json"
      body: '{"status": "check"}'
      valid_status_codes: [200, 201]
      fail_if_body_not_matches_regexp:
        - "success"

  http_with_basic_auth:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      method: GET
      basic_auth:
        username: "monitoring"
        password: "secure-password"

Configure DNS monitoring with specific resolvers

Set up DNS monitoring to test specific DNS servers and query types.

  dns_google:
    prober: dns
    timeout: 5s
    dns:
      query_name: "example.com"
      query_type: "A"
      valid_rcodes:
        - NOERROR
      validate_answer_rrs:
        fail_if_matches_regexp:
          - "127.0.0.1"
      dns_over_tls: false
      transport_protocol: "udp"
      preferred_ip_protocol: "ip4"
      source_ip_address: "0.0.0.0"
      recursion_desired: true

Reload Blackbox Exporter configuration

Apply the updated configuration with new monitoring modules.

sudo systemctl reload blackbox_exporter
sudo systemctl status blackbox_exporter

Verify your setup

Check that Blackbox Exporter is running correctly and metrics are being collected:

sudo systemctl status blackbox_exporter
curl -s http://localhost:9115/probe?module=http_2xx&target=http://example.com | grep probe_success
curl -s http://localhost:9115/probe?module=ssl_expiry&target=https://example.com | grep probe_ssl_earliest_cert_expiry

Test specific endpoints and view the web interface:

curl -s "http://localhost:9115/probe?module=http_2xx&target=http://example.com"
ss -tlnp | grep :9115

Access the Blackbox Exporter web interface at http://your-server:9115 to test probes manually and verify configuration.

Common issues

SymptomCauseFix
Service fails to start Configuration syntax error sudo -u blackbox_exporter /usr/local/bin/blackbox_exporter --config.check --config.file=/etc/blackbox_exporter/config.yml
SSL probes always fail Incorrect TLS configuration Set insecure_skip_verify: true temporarily to test, then fix certificate issues
Permission denied errors Incorrect file ownership sudo chown -R blackbox_exporter:blackbox_exporter /etc/blackbox_exporter
Prometheus not scraping Firewall blocking port 9115 Check firewall rules and ensure Prometheus can reach port 9115
ICMP probes fail Insufficient privileges Add capability: sudo setcap cap_net_raw+ep /usr/local/bin/blackbox_exporter
DNS probes timeout Network connectivity issues Test manual DNS resolution: dig @8.8.8.8 example.com
Never use chmod 777. It gives every user on the system full access to your files. Instead, fix ownership with chown and use minimal permissions like 755 for directories and 644 for files.

Next steps

Running this in production?

Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European teams.

Automated install script

Run this to automate the entire setup

Need help?

Don't want to manage this yourself?

We handle managed devops services for businesses that depend on uptime. From initial setup to ongoing operations.