Configure Tailscale monitoring with Prometheus and Grafana dashboards

Update system packages

Start by updating your package manager to ensure you get the latest versions of all components.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install Tailscale mesh VPN

Install Tailscale using the official installation script which automatically configures the package repository and installs the client.

curl -fsSL https://tailscale.com/install.sh | sh

Start Tailscale and authenticate with your account. This command will display a URL for device authentication.

sudo tailscale up

Verify the Tailscale connection is active and note your assigned IP address.

tailscale status
tailscale ip

Install Prometheus

Install Prometheus for metrics collection and monitoring. Create a dedicated user for running Prometheus securely.

sudo apt install -y prometheus prometheus-node-exporter

sudo dnf install -y prometheus2 node_exporter

Create directories for Prometheus data and configuration with proper ownership.

sudo mkdir -p /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Install Tailscale Prometheus exporter

Download and install the Tailscale Prometheus exporter to collect mesh VPN metrics. This provides detailed statistics about node connections, traffic, and network health.

wget https://github.com/tailscale/tailscale/releases/latest/download/tailscale-prometheus-exporter_linux_amd64.tar.gz
tar -xzf tailscale-prometheus-exporter_linux_amd64.tar.gz
sudo mv tailscale-prometheus-exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/tailscale-prometheus-exporter

Create a systemd service file for the Tailscale exporter to run automatically.

[Unit]
Description=Tailscale Prometheus Exporter
After=network.target tailscaled.service
Requires=tailscaled.service

[Service]
Type=simple
User=nobody
ExecStart=/usr/local/bin/tailscale-prometheus-exporter --listen-addr=127.0.0.1:9090
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Configure Prometheus for Tailscale monitoring

Configure Prometheus to scrape metrics from the Tailscale exporter and system node exporter for comprehensive monitoring.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "tailscale_alerts.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'tailscale'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 30s
    metrics_path: '/metrics'

  - job_name: 'tailscale-nodes'
    tailscale_sd_configs:
      - url: "https://api.tailscale.com/api/v2/tailnet/"
        refresh_interval: 60s

Create alerting rules for Tailscale network monitoring to detect connectivity issues and performance problems.

groups:
name: tailscale
  rules:
  - alert: TailscaleNodeDown
    expr: up{job="tailscale"} == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Tailscale node is down"
      description: "Tailscale node {{ $labels.instance }} has been down for more than 2 minutes."

  - alert: TailscaleHighLatency
    expr: tailscale_node_latency_seconds > 0.5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High Tailscale latency detected"
      description: "Tailscale node {{ $labels.node }} has latency {{ $value }}s for more than 5 minutes."

  - alert: TailscaleConnectionLoss
    expr: increase(tailscale_connection_failures_total[10m]) > 3
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Frequent Tailscale connection failures"
      description: "Node {{ $labels.instance }} has {{ $value }} connection failures in the last 10 minutes."

Install and configure Grafana

Install Grafana for creating dashboards and visualizing Tailscale network metrics.

sudo apt install -y software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana

sudo tee /etc/yum.repos.d/grafana.repo <

Configure Grafana with security settings and enable anonymous access for dashboards if needed.

[server]
http_port = 3000
root_url = http://localhost:3000

[security]
admin_user = admin
admin_password = your_secure_password_here
secret_key = your_secret_key_here

[auth.anonymous]
enabled = false

[dashboards]
default_home_dashboard_path = /var/lib/grafana/dashboards/tailscale-overview.json

Enable and start services

Enable and start all monitoring services to begin collecting Tailscale metrics.

sudo systemctl daemon-reload
sudo systemctl enable --now tailscale-exporter
sudo systemctl enable --now prometheus
sudo systemctl enable --now grafana-server

Verify all services are running correctly.

sudo systemctl status tailscale-exporter
sudo systemctl status prometheus
sudo systemctl status grafana-server

Configure Grafana data source

Access Grafana web interface and configure Prometheus as a data source for Tailscale metrics.

curl -X POST http://admin:your_secure_password_here@localhost:3000/api/datasources \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Prometheus",
    "type": "prometheus",
    "url": "http://localhost:9090",
    "access": "proxy",
    "isDefault": true
  }'

Create Tailscale dashboard

Create a comprehensive Grafana dashboard for monitoring Tailscale mesh network performance and connectivity.

{
  "dashboard": {
    "id": null,
    "title": "Tailscale Network Overview",
    "tags": ["tailscale", "vpn", "networking"],
    "timezone": "browser",
    "panels": [
      {
        "id": 1,
        "title": "Connected Nodes",
        "type": "stat",
        "targets": [
          {
            "expr": "count(up{job=\"tailscale\"} == 1)",
            "legendFormat": "Active Nodes"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "color": {
              "mode": "thresholds"
            },
            "thresholds": {
              "steps": [
                {"color": "red", "value": 0},
                {"color": "yellow", "value": 1},
                {"color": "green", "value": 2}
              ]
            }
          }
        },
        "gridPos": {"h": 8, "w": 6, "x": 0, "y": 0}
      },
      {
        "id": 2,
        "title": "Network Traffic",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(tailscale_tx_bytes_total[5m])",
            "legendFormat": "TX - {{instance}}"
          },
          {
            "expr": "rate(tailscale_rx_bytes_total[5m])",
            "legendFormat": "RX - {{instance}}"
          }
        ],
        "yAxes": [
          {
            "unit": "Bps",
            "min": 0
          }
        ],
        "gridPos": {"h": 8, "w": 18, "x": 6, "y": 0}
      },
      {
        "id": 3,
        "title": "Node Latency",
        "type": "graph",
        "targets": [
          {
            "expr": "tailscale_node_latency_seconds * 1000",
            "legendFormat": "{{node}}"
          }
        ],
        "yAxes": [
          {
            "unit": "ms",
            "min": 0
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}
      },
      {
        "id": 4,
        "title": "Connection Status",
        "type": "table",
        "targets": [
          {
            "expr": "tailscale_node_info",
            "format": "table",
            "instant": true
          }
        ],
        "transformations": [
          {
            "id": "organize",
            "options": {
              "excludeByName": {},
              "indexByName": {},
              "renameByName": {
                "node": "Node",
                "online": "Status",
                "last_seen": "Last Seen"
              }
            }
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
      }
    ],
    "time": {
      "from": "now-1h",
      "to": "now"
    },
    "refresh": "30s"
  }
}

Import the dashboard into Grafana using the API.

curl -X POST http://admin:your_secure_password_here@localhost:3000/api/dashboards/db \
  -H "Content-Type: application/json" \
  -d @/var/lib/grafana/dashboards/tailscale-overview.json

Configure firewall rules

Configure firewall rules to allow access to Grafana and secure the monitoring endpoints. This tutorial shows how to use Linux firewall configuration for production environments.

sudo ufw allow 3000/tcp comment 'Grafana'
sudo ufw allow from 203.0.113.0/24 to any port 9090 comment 'Prometheus internal'
sudo ufw enable

sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="9090" accept'
sudo firewall-cmd --reload

Set up alerting notifications

Configure Alertmanager for sending notifications when Tailscale network issues occur. This complements the Prometheus alerting setup you might already have.

sudo apt install -y prometheus-alertmanager

sudo dnf install -y alertmanager

global:
  smtp_smarthost: 'localhost:587'
  smtp_from: 'alerts@example.com'

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'tailscale-alerts'

receivers:
name: 'tailscale-alerts'
  email_configs:
  - to: 'admin@example.com'
    subject: 'Tailscale Alert: {{ .GroupLabels.alertname }}'
    body: |
      {{ range .Alerts }}
      Alert: {{ .Annotations.summary }}
      Description: {{ .Annotations.description }}
      Instance: {{ .Labels.instance }}
      {{ end }}

Enable and start Alertmanager for notification handling.

sudo systemctl enable --now prometheus-alertmanager