Configure Prometheus Alertmanager Slack Integration

Set up Prometheus Alertmanager to route alerts to Slack channels with custom notification rules. Configure alerting rules, webhook integration, and team-based routing for production monitoring workflows.

Prerequisites

Prometheus server already installed
Slack workspace with admin access
Basic knowledge of YAML configuration

What this solves

Prometheus Alertmanager receives alerts from Prometheus and routes them to notification channels like Slack, email, or PagerDuty. This tutorial shows you how to configure Alertmanager with Slack integration, set up custom routing rules, and create alerting policies for different teams and severity levels.

Step-by-step installation

Update system packages

Start by updating your package manager to ensure you get the latest versions.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Create system user for Alertmanager

Create a dedicated system user to run Alertmanager securely without shell access.

sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
sudo chown alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager

Download and install Alertmanager

Download the latest Alertmanager binary and install it to the system PATH.

cd /tmp
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xzf alertmanager-0.27.0.linux-amd64.tar.gz
sudo cp alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo cp alertmanager-0.27.0.linux-amd64/amtool /usr/local/bin/
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager /usr/local/bin/amtool

Set up Slack webhook integration

Create a Slack webhook URL for your workspace. Go to your Slack workspace, navigate to Apps, search for "Incoming Webhooks", and create a new webhook for your target channel.

Note: Copy the webhook URL starting with https://hooks.slack.com/services/. You'll need this for the Alertmanager configuration.

Create Alertmanager configuration

Configure Alertmanager with Slack integration, routing rules, and notification templates.

global:
  smtp_smarthost: 'localhost:587'
  smtp_from: 'alertmanager@example.com'
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'web.hook'
  routes:
  - match:
      severity: critical
    receiver: 'critical-alerts'
    group_wait: 10s
    repeat_interval: 1h
  - match:
      severity: warning
    receiver: 'warning-alerts'
    group_wait: 30s
    repeat_interval: 6h
  - match:
      team: database
    receiver: 'database-team'
  - match:
      team: frontend
    receiver: 'frontend-team'

receivers:
name: 'web.hook'  slack_configs:
  - channel: '#alerts'
    title: 'Prometheus Alert'
    text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
    send_resolved: true

name: 'critical-alerts'  slack_configs:
  - channel: '#critical-alerts'
    title: 'CRITICAL: {{ .GroupLabels.alertname }}'
    text: |
      {{ range .Alerts }}
      Alert: {{ .Annotations.summary }}
      Description: {{ .Annotations.description }}
      Instance: {{ .Labels.instance }}
      Severity: {{ .Labels.severity }}
      {{ end }}
    color: 'danger'
    send_resolved: true

name: 'warning-alerts'  slack_configs:
  - channel: '#warnings'
    title: 'Warning: {{ .GroupLabels.alertname }}'
    text: |
      {{ range .Alerts }}
      Alert: {{ .Annotations.summary }}
      Instance: {{ .Labels.instance }}
      {{ end }}
    color: 'warning'
    send_resolved: true

name: 'database-team'  slack_configs:
  - channel: '#database-alerts'
    title: 'Database Alert: {{ .GroupLabels.alertname }}'
    text: |
      {{ range .Alerts }}
      Database Alert: {{ .Annotations.summary }}
      Instance: {{ .Labels.instance }}
      Team: {{ .Labels.team }}
      {{ end }}
    send_resolved: true

name: 'frontend-team'  slack_configs:
  - channel: '#frontend-alerts'
    title: 'Frontend Alert: {{ .GroupLabels.alertname }}'
    text: |
      {{ range .Alerts }}
      Frontend Alert: {{ .Annotations.summary }}
      Instance: {{ .Labels.instance }}
      Team: {{ .Labels.team }}
      {{ end }}
    send_resolved: true

inhibit_rules:
source_match:    severity: 'critical'
  target_match:
    severity: 'warning'
  equal: ['alertname', 'instance']

Configure Prometheus alerting rules

Create alerting rules that Prometheus will evaluate and send to Alertmanager.

groups:
name: system_alerts  rules:
  - alert: HighCPUUsage
    expr: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 5m
    labels:
      severity: warning
      team: infrastructure
    annotations:
      summary: "High CPU usage on {{ $labels.instance }}"
      description: "CPU usage is above 80% for more than 5 minutes on {{ $labels.instance }}"

  - alert: HighMemoryUsage
    expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85
    for: 5m
    labels:
      severity: warning
      team: infrastructure
    annotations:
      summary: "High memory usage on {{ $labels.instance }}"
      description: "Memory usage is above 85% on {{ $labels.instance }}"

  - alert: DiskSpaceLow
    expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
    for: 2m
    labels:
      severity: critical
      team: infrastructure
    annotations:
      summary: "Low disk space on {{ $labels.instance }}"
      description: "Disk space is below 10% on {{ $labels.instance }}"

  - alert: ServiceDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
      team: infrastructure
    annotations:
      summary: "Service down: {{ $labels.job }}"
      description: "Service {{ $labels.job }} on {{ $labels.instance }} is down"

name: database_alerts  rules:
  - alert: PostgreSQLDown
    expr: pg_up == 0
    for: 1m
    labels:
      severity: critical
      team: database
    annotations:
      summary: "PostgreSQL is down on {{ $labels.instance }}"
      description: "PostgreSQL database is not responding on {{ $labels.instance }}"

  - alert: PostgreSQLTooManyConnections
    expr: sum(pg_stat_activity_count) by (instance) > 80
    for: 5m
    labels:
      severity: warning
      team: database
    annotations:
      summary: "Too many PostgreSQL connections on {{ $labels.instance }}"
      description: "PostgreSQL has {{ $value }} active connections on {{ $labels.instance }}"

name: web_alerts  rules:
  - alert: WebsiteDown
    expr: probe_success == 0
    for: 1m
    labels:
      severity: critical
      team: frontend
    annotations:
      summary: "Website is down: {{ $labels.instance }}"
      description: "Website {{ $labels.instance }} is not responding to health checks"

  - alert: HighResponseTime
    expr: probe_duration_seconds > 5
    for: 5m
    labels:
      severity: warning
      team: frontend
    annotations:
      summary: "High response time for {{ $labels.instance }}"
      description: "Response time is {{ $value }}s for {{ $labels.instance }}"

  - alert: NginxDown
    expr: nginx_up == 0
    for: 1m
    labels:
      severity: critical
      team: frontend
    annotations:
      summary: "Nginx is down on {{ $labels.instance }}"
      description: "Nginx web server is not responding on {{ $labels.instance }}"

Update Prometheus configuration

Configure Prometheus to load the alerting rules and send alerts to Alertmanager.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert_rules.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'alertmanager'
    static_configs:
      - targets: ['localhost:9093']

Set correct file permissions

Ensure Alertmanager can read its configuration files with proper ownership and permissions.

sudo chown -R alertmanager:alertmanager /etc/alertmanager
sudo chmod 755 /etc/alertmanager
sudo chmod 644 /etc/alertmanager/alertmanager.yml
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chmod 644 /etc/prometheus/alert_rules.yml

Never use chmod 777. It gives every user on the system full access to your files. Instead, use specific ownership with chown and minimal permissions for security.

Create systemd service

Set up Alertmanager as a systemd service for automatic startup and process management.

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
WorkingDirectory=/var/lib/alertmanager
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.external-url=http://localhost:9093 \
  --web.route-prefix=/ \
  --cluster.listen-address=0.0.0.0:9094
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Start and enable services

Enable both Prometheus and Alertmanager services to start automatically on boot.

sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager
sudo systemctl restart prometheus
sudo systemctl status alertmanager
sudo systemctl status prometheus

Configure firewall access

Open the necessary ports for Alertmanager web interface and cluster communication.

sudo ufw allow 9093/tcp comment "Alertmanager web interface"
sudo ufw allow 9094/tcp comment "Alertmanager cluster"

sudo firewall-cmd --permanent --add-port=9093/tcp
sudo firewall-cmd --permanent --add-port=9094/tcp
sudo firewall-cmd --reload

Testing Slack integration

Send test alert

Use amtool to send a test alert and verify Slack integration is working.

amtool alert add test_alert alertname="TestAlert" severity="warning" instance="localhost" summary="Test alert from Alertmanager" --alertmanager.url=http://localhost:9093

Trigger CPU alert for testing

Create artificial CPU load to test your alerting rules and Slack notifications.

timeout 300s yes > /dev/null &
timeout 300s yes > /dev/null &
timeout 300s yes > /dev/null &
timeout 300s yes > /dev/null &

Note: This creates CPU load for 5 minutes to trigger the HighCPUUsage alert. Check your Slack channel for notifications.

Verify your setup

sudo systemctl status alertmanager
curl -s http://localhost:9093/-/healthy
curl -s http://localhost:9093/api/v1/status
amtool config show --alertmanager.url=http://localhost:9093
amtool alert query --alertmanager.url=http://localhost:9093

Access the Alertmanager web interface at http://your-server-ip:9093 to view active alerts and configuration status. The interface shows current alerts, silences, and routing information.

Advanced configuration

Configure alert silencing

Set up maintenance windows and alert silencing for planned downtime.

amtool silence add alertname="HighCPUUsage" instance="server1.example.com" --duration="2h" --comment="Planned maintenance" --alertmanager.url=http://localhost:9093
amtool silence query --alertmanager.url=http://localhost:9093

Add email notifications

Configure email notifications as a backup to Slack for critical alerts.

- name: 'critical-email'
  email_configs:
  - to: 'oncall@example.com'
    from: 'alertmanager@example.com'
    subject: 'CRITICAL: {{ .GroupLabels.alertname }}'
    body: |
      {{ range .Alerts }}
      Alert: {{ .Annotations.summary }}
      Description: {{ .Annotations.description }}
      Instance: {{ .Labels.instance }}
      Severity: {{ .Labels.severity }}
      {{ end }}
  slack_configs:
  - channel: '#critical-alerts'
    title: 'CRITICAL: {{ .GroupLabels.alertname }}'
    text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
    color: 'danger'
    send_resolved: true

Common issues

Symptom	Cause	Fix
Alertmanager won't start	Configuration syntax error	`amtool config check /etc/alertmanager/alertmanager.yml`
No Slack notifications received	Invalid webhook URL or channel	Test webhook URL manually with curl, verify channel exists
Alerts not firing	Prometheus not loading rules	`curl http://localhost:9090/api/v1/rules` to check loaded rules
Permission denied errors	Incorrect file ownership	`sudo chown -R alertmanager:alertmanager /etc/alertmanager`
Web interface shows "Unhealthy"	Storage path not accessible	Check `/var/lib/alertmanager` permissions and disk space
Too many duplicate notifications	Incorrect grouping configuration	Adjust `group_interval` and `repeat_interval` settings

Next steps

Configure advanced SNMP alerting with Prometheus Alertmanager for network monitoring
Setup Prometheus Blackbox Exporter for endpoint monitoring with SSL and alerting
Configure Prometheus Alertmanager webhook integrations for PagerDuty and Microsoft Teams
Implement Alertmanager high availability clustering for production environments

Running this in production?

Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European teams.

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash
set -euo pipefail

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

# Variables
ALERTMANAGER_VERSION="0.27.0"
SLACK_WEBHOOK_URL=""
SMTP_FROM="alertmanager@example.com"
SMTP_HOST="localhost:587"

# Print colored output
print_status() {
    echo -e "${GREEN}[INFO]${NC} $1"
}

print_warning() {
    echo -e "${YELLOW}[WARNING]${NC} $1"
}

print_error() {
    echo -e "${RED}[ERROR]${NC} $1"
}

# Usage message
usage() {
    echo "Usage: $0 [slack_webhook_url] [smtp_from] [smtp_host]"
    echo "  slack_webhook_url: Slack webhook URL (optional, can be configured later)"
    echo "  smtp_from: Email address for SMTP (default: alertmanager@example.com)"
    echo "  smtp_host: SMTP server host:port (default: localhost:587)"
    exit 1
}

# Cleanup on failure
cleanup() {
    print_error "Installation failed. Cleaning up..."
    systemctl stop alertmanager 2>/dev/null || true
    systemctl disable alertmanager 2>/dev/null || true
    rm -f /etc/systemd/system/alertmanager.service
    userdel alertmanager 2>/dev/null || true
    rm -rf /etc/alertmanager /var/lib/alertmanager
    rm -f /usr/local/bin/alertmanager /usr/local/bin/amtool
}

trap cleanup ERR

# Check if running as root
if [[ $EUID -ne 0 ]]; then
    print_error "This script must be run as root"
    exit 1
fi

# Parse arguments
if [[ $# -gt 0 ]]; then
    SLACK_WEBHOOK_URL="$1"
fi
if [[ $# -gt 1 ]]; then
    SMTP_FROM="$2"
fi
if [[ $# -gt 2 ]]; then
    SMTP_HOST="$3"
fi

# Auto-detect distribution
print_status "[1/8] Detecting distribution..."
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian) 
            PKG_MGR="apt"
            PKG_UPDATE="apt update && apt upgrade -y"
            PKG_INSTALL="apt install -y"
            FIREWALL_CMD="ufw"
            ;;
        almalinux|rocky|centos|rhel|ol|fedora) 
            PKG_MGR="dnf"
            PKG_UPDATE="dnf update -y"
            PKG_INSTALL="dnf install -y"
            FIREWALL_CMD="firewall-cmd"
            ;;
        amzn) 
            PKG_MGR="yum"
            PKG_UPDATE="yum update -y"
            PKG_INSTALL="yum install -y"
            FIREWALL_CMD="firewall-cmd"
            ;;
        *) 
            print_error "Unsupported distribution: $ID"
            exit 1
            ;;
    esac
else
    print_error "Cannot detect distribution"
    exit 1
fi

print_status "Detected distribution: $PRETTY_NAME"

# Update system packages
print_status "[2/8] Updating system packages..."
$PKG_UPDATE

# Install prerequisites
print_status "[3/8] Installing prerequisites..."
$PKG_INSTALL wget tar

# Create system user for Alertmanager
print_status "[4/8] Creating system user for Alertmanager..."
if ! id alertmanager &>/dev/null; then
    useradd --no-create-home --shell /bin/false alertmanager
fi

mkdir -p /etc/alertmanager /var/lib/alertmanager
chown alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager
chmod 755 /etc/alertmanager /var/lib/alertmanager

# Download and install Alertmanager
print_status "[5/8] Downloading and installing Alertmanager..."
cd /tmp
wget -q "https://github.com/prometheus/alertmanager/releases/download/v${ALERTMANAGER_VERSION}/alertmanager-${ALERTMANAGER_VERSION}.linux-amd64.tar.gz"
tar xzf "alertmanager-${ALERTMANAGER_VERSION}.linux-amd64.tar.gz"
cp "alertmanager-${ALERTMANAGER_VERSION}.linux-amd64/alertmanager" /usr/local/bin/
cp "alertmanager-${ALERTMANAGER_VERSION}.linux-amd64/amtool" /usr/local/bin/
chown alertmanager:alertmanager /usr/local/bin/alertmanager /usr/local/bin/amtool
chmod 755 /usr/local/bin/alertmanager /usr/local/bin/amtool
rm -rf "alertmanager-${ALERTMANAGER_VERSION}.linux-amd64"*

# Create Alertmanager configuration
print_status "[6/8] Creating Alertmanager configuration..."
cat > /etc/alertmanager/alertmanager.yml << EOF
global:
  smtp_smarthost: '${SMTP_HOST}'
  smtp_from: '${SMTP_FROM}'
  slack_api_url: '${SLACK_WEBHOOK_URL}'

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'web.hook'
  routes:
  - match:
      severity: critical
    receiver: 'critical-alerts'
    group_wait: 10s
    repeat_interval: 1h
  - match:
      severity: warning
    receiver: 'warning-alerts'
    group_wait: 30s
    repeat_interval: 6h
  - match:
      team: database
    receiver: 'database-team'
  - match:
      team: frontend
    receiver: 'frontend-team'

receivers:
- name: 'web.hook'
  slack_configs:
  - channel: '#alerts'
    title: 'Prometheus Alert'
    text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
    send_resolved: true

- name: 'critical-alerts'
  slack_configs:
  - channel: '#critical-alerts'
    title: 'CRITICAL: {{ .GroupLabels.alertname }}'
    text: |
      {{ range .Alerts }}
      Alert: {{ .Annotations.summary }}
      Description: {{ .Annotations.description }}
      Instance: {{ .Labels.instance }}
      Severity: {{ .Labels.severity }}
      {{ end }}
    color: 'danger'
    send_resolved: true

- name: 'warning-alerts'
  slack_configs:
  - channel: '#warnings'
    title: 'Warning: {{ .GroupLabels.alertname }}'
    text: |
      {{ range .Alerts }}
      Alert: {{ .Annotations.summary }}
      Instance: {{ .Labels.instance }}
      {{ end }}
    color: 'warning'
    send_resolved: true

- name: 'database-team'
  slack_configs:
  - channel: '#database-alerts'
    title: 'Database Alert: {{ .GroupLabels.alertname }}'
    text: |
      {{ range .Alerts }}
      Database Alert: {{ .Annotations.summary }}
      Instance: {{ .Labels.instance }}
      Team: {{ .Labels.team }}
      {{ end }}
    send_resolved: true

- name: 'frontend-team'
  slack_configs:
  - channel: '#frontend-alerts'
    title: 'Frontend Alert: {{ .GroupLabels.alertname }}'
    text: |
      {{ range .Alerts }}
      Frontend Alert: {{ .Annotations.summary }}
      Instance: {{ .Labels.instance }}
      Team: {{ .Labels.team }}
      {{ end }}
    send_resolved: true
EOF

chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml
chmod 644 /etc/alertmanager/alertmanager.yml

# Create systemd service
print_status "[7/8] Creating systemd service..."
cat > /etc/systemd/system/alertmanager.service << EOF
[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
    --config.file /etc/alertmanager/alertmanager.yml \
    --storage.path /var/lib/alertmanager/ \
    --web.external-url=http://localhost:9093

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable alertmanager
systemctl start alertmanager

# Configure firewall
print_status "[8/8] Configuring firewall..."
case "$PKG_MGR" in
    "apt")
        if command -v ufw &> /dev/null; then
            ufw allow 9093/tcp || true
        fi
        ;;
    "dnf"|"yum")
        if command -v firewall-cmd &> /dev/null; then
            firewall-cmd --permanent --add-port=9093/tcp || true
            firewall-cmd --reload || true
        fi
        ;;
esac

# Verification
print_status "Verifying installation..."
sleep 5
if systemctl is-active --quiet alertmanager; then
    print_status "✓ Alertmanager service is running"
else
    print_error "✗ Alertmanager service is not running"
    exit 1
fi

if curl -s http://localhost:9093 > /dev/null; then
    print_status "✓ Alertmanager web interface is accessible"
else
    print_warning "✗ Alertmanager web interface is not accessible"
fi

print_status "Alertmanager installation completed successfully!"
print_status "Access the web interface at: http://your-server-ip:9093"
if [[ -z "$SLACK_WEBHOOK_URL" ]]; then
    print_warning "No Slack webhook URL provided. Edit /etc/alertmanager/alertmanager.yml and restart the service."
fi
print_status "To restart Alertmanager: systemctl restart alertmanager"
print_status "To view logs: journalctl -u alertmanager -f"

Review the script before running. Execute with: bash install.sh

#prometheus #alertmanager #slack #monitoring #alerts

Configure Prometheus Alertmanager with Slack integration for team notifications