Set up Alertmanager with email and Slack notifications for monitoring alerts

Intermediate 25 min May 18, 2026 18 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Configure Prometheus Alertmanager with email SMTP and Slack webhook integrations for comprehensive monitoring notifications. Set up alert routing rules and test notification delivery.

Prerequisites

  • Root access to target server
  • Basic understanding of Prometheus monitoring
  • SMTP credentials for email notifications
  • Slack workspace access for webhook setup

What this solves

Alertmanager receives alerts from Prometheus and routes them to notification channels like email and Slack. This tutorial shows you how to install Alertmanager, configure SMTP email delivery, set up Slack webhook integration, and create routing rules to send different alerts to different channels based on severity or service.

Step-by-step installation

Update system packages

Start by updating your package manager to ensure you get the latest versions.

sudo apt update && sudo apt upgrade -y
sudo dnf update -y

Create alertmanager user

Create a dedicated system user to run Alertmanager securely.

sudo useradd --system --no-create-home --shell /bin/false alertmanager

Download and install Alertmanager

Download the latest Alertmanager binary from the official releases page.

cd /tmp
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xzf alertmanager-0.27.0.linux-amd64.tar.gz
sudo cp alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo cp alertmanager-0.27.0.linux-amd64/amtool /usr/local/bin/
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/amtool

Create directory structure

Set up the required directories for Alertmanager configuration and data storage.

sudo mkdir -p /etc/alertmanager
sudo mkdir -p /var/lib/alertmanager
sudo chown alertmanager:alertmanager /etc/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager

Configure basic Alertmanager settings

Create the main configuration file with email and Slack notification settings.

global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alerts@example.com'
  smtp_auth_username: 'alerts@example.com'
  smtp_auth_password: 'your-app-password'
  smtp_require_tls: true

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
  routes:
    - match:
        severity: critical
      receiver: 'critical-alerts'
    - match:
        service: database
      receiver: 'database-team'
    - match:
        severity: warning
      receiver: 'warning-alerts'

receivers:
  - name: 'web.hook'
    email_configs:
      - to: 'admin@example.com'
        subject: 'Alertmanager Notification'
        body: |
          {{ range .Alerts }}
          Alert: {{ .Annotations.summary }}
          Description: {{ .Annotations.description }}
          {{ end }}

  - name: 'critical-alerts'
    email_configs:
      - to: 'oncall@example.com'
        subject: 'CRITICAL: {{ .GroupLabels.alertname }}'
        body: |
          {{ range .Alerts }}
          CRITICAL ALERT
          
          Alert: {{ .Annotations.summary }}
          Description: {{ .Annotations.description }}
          Instance: {{ .Labels.instance }}
          Severity: {{ .Labels.severity }}
          {{ end }}
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
        channel: '#alerts'
        title: 'Critical Alert'
        text: |
          {{ range .Alerts }}
          Alert: {{ .Annotations.summary }}
          Description: {{ .Annotations.description }}
          Instance: {{ .Labels.instance }}
          {{ end }}
        color: danger

  - name: 'database-team'
    email_configs:
      - to: 'dba-team@example.com'
        subject: 'Database Alert: {{ .GroupLabels.alertname }}'
        body: |
          {{ range .Alerts }}
          Database Alert
          
          Alert: {{ .Annotations.summary }}
          Description: {{ .Annotations.description }}
          Database: {{ .Labels.database }}
          {{ end }}
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
        channel: '#database-alerts'
        title: 'Database Alert'
        text: |
          {{ range .Alerts }}
          Database Alert: {{ .Annotations.summary }}
          Description: {{ .Annotations.description }}
          Database: {{ .Labels.database }}
          {{ end }}
        color: warning

  - name: 'warning-alerts'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
        channel: '#monitoring'
        title: 'Warning Alert'
        text: |
          {{ range .Alerts }}
          Warning: {{ .Annotations.summary }}
          Description: {{ .Annotations.description }}
          Instance: {{ .Labels.instance }}
          {{ end }}
        color: warning

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Set correct file permissions

Secure the configuration file and set proper ownership.

sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml
sudo chmod 640 /etc/alertmanager/alertmanager.yml

Create systemd service

Set up a systemd service to manage Alertmanager automatically.

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
    --config.file=/etc/alertmanager/alertmanager.yml \
    --storage.path=/var/lib/alertmanager/ \
    --web.external-url=http://localhost:9093
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Configure Slack webhook

Set up a Slack incoming webhook for alert notifications.

Note: Go to your Slack workspace settings, create an incoming webhook, and replace the webhook URL in the configuration above with your actual webhook URL.

Enable and start Alertmanager

Start the Alertmanager service and enable it to start on boot.

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager

Configure firewall

Open port 9093 for Alertmanager web interface access.

sudo ufw allow 9093/tcp
sudo ufw reload
sudo firewall-cmd --add-port=9093/tcp --permanent
sudo firewall-cmd --reload

Configure email notifications

Set up Gmail SMTP

For Gmail, you need to create an app password instead of using your regular password.

Note: Enable 2-factor authentication on your Gmail account, then generate an app password at myaccount.google.com/apppasswords. Use this app password in the configuration.

Configure custom SMTP server

For other email providers, update the SMTP settings in the configuration file.

global:
  smtp_smarthost: 'mail.example.com:587'
  smtp_from: 'alerts@example.com'
  smtp_auth_username: 'alerts@example.com'
  smtp_auth_password: 'your-smtp-password'
  smtp_require_tls: true
  smtp_hello: 'alertmanager.example.com'

Test email configuration

Send a test alert to verify email delivery works correctly.

curl -XPOST http://localhost:9093/api/v1/alerts -H 'Content-Type: application/json' -d '[
  {
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning"
    },
    "annotations": {
      "summary": "Test alert from Alertmanager",
      "description": "This is a test alert to verify email configuration"
    },
    "startsAt": "'$(date -Iseconds)'",
    "endsAt": "'$(date -d '+1 hour' -Iseconds)'"
  }
]'

Set up routing rules

Create advanced routing configuration

Configure sophisticated routing based on labels, severity, and services.

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'critical-pager'
      group_wait: 5s
      repeat_interval: 5m
    - match_re:
        service: mysql|postgresql|redis
      receiver: 'database-team'
      group_by: ['alertname', 'database']
    - match:
        team: frontend
      receiver: 'frontend-team'
    - match:
        environment: production
      receiver: 'production-alerts'
      routes:
        - match:
            severity: critical
          receiver: 'production-critical'
    - match:
        alertname: DeadMansSwitch
      receiver: 'deadmansswitch'
      repeat_interval: 30s

Configure time-based routing

Set up different notification behavior during business hours versus off-hours.

route:
  routes:
    - match:
        severity: critical
      receiver: 'critical-business-hours'
      active_time_intervals:
        - business-hours
    - match:
        severity: critical
      receiver: 'critical-after-hours'
      active_time_intervals:
        - after-hours

time_intervals:
  - name: business-hours
    time_intervals:
      - times:
          - start_time: '09:00'
            end_time: '17:00'
        weekdays: ['monday:friday']
        location: 'Europe/London'
  - name: after-hours
    time_intervals:
      - times:
          - start_time: '17:00'
            end_time: '09:00'
        weekdays: ['monday:friday']
        location: 'Europe/London'
      - weekdays: ['saturday', 'sunday']
        location: 'Europe/London'

Apply configuration changes

Restart Alertmanager to load the new routing configuration.

sudo systemctl restart alertmanager
sudo systemctl status alertmanager

Test alert delivery

Send test alerts for different scenarios

Test critical alerts, database alerts, and warning notifications.

# Test critical alert
curl -XPOST http://localhost:9093/api/v1/alerts -H 'Content-Type: application/json' -d '[
  {
    "labels": {
      "alertname": "HighCPUUsage",
      "severity": "critical",
      "instance": "web-01"
    },
    "annotations": {
      "summary": "High CPU usage detected",
      "description": "CPU usage is above 90% for more than 5 minutes"
    },
    "startsAt": "'$(date -Iseconds)'"
  }
]'

Test database alert

curl -XPOST http://localhost:9093/api/v1/alerts -H 'Content-Type: application/json' -d '[ { "labels": { "alertname": "DatabaseConnectionHigh", "severity": "warning", "service": "database", "database": "mysql" }, "annotations": { "summary": "Database connection count is high", "description": "MySQL connection count is above threshold" }, "startsAt": "'$(date -Iseconds)'" } ]'

Verify alert delivery

Check the Alertmanager web interface and confirm notifications were sent.

# Check Alertmanager status
curl http://localhost:9093/api/v1/status

View active alerts

curl http://localhost:9093/api/v1/alerts

Check silences

curl http://localhost:9093/api/v1/silences

Connect with Prometheus

Configure Prometheus to send alerts

Add Alertmanager endpoint to your Prometheus configuration. This assumes you already have Prometheus installed.

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

rule_files:
  - "alert_rules.yml"

Create sample alert rules

Define alert rules that Prometheus will evaluate and send to Alertmanager.

groups:
  • name: system.rules
rules: - alert: HighCPUUsage expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 2m labels: severity: warning team: infrastructure annotations: summary: "High CPU usage on {{ $labels.instance }}" description: "CPU usage is {{ $value }}% on instance {{ $labels.instance }}" - alert: HighMemoryUsage expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85 for: 5m labels: severity: critical team: infrastructure annotations: summary: "High memory usage on {{ $labels.instance }}" description: "Memory usage is {{ $value }}% on instance {{ $labels.instance }}" - alert: DiskSpaceLow expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10 for: 1m labels: severity: critical team: infrastructure annotations: summary: "Low disk space on {{ $labels.instance }}" description: "Disk space is {{ $value }}% full on instance {{ $labels.instance }}"
  • name: database.rules
rules: - alert: MySQLDown expr: mysql_up == 0 for: 0m labels: severity: critical service: database database: mysql annotations: summary: "MySQL is down on {{ $labels.instance }}" description: "MySQL database server is not responding"

Reload Prometheus configuration

Apply the new configuration to start sending alerts to Alertmanager.

sudo systemctl restart prometheus
curl -X POST http://localhost:9090/-/reload

Verify your setup

# Check Alertmanager status
sudo systemctl status alertmanager

Verify Alertmanager is listening

ss -tlnp | grep 9093

Check configuration syntax

/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --config.check

View web interface

echo "Access Alertmanager at: http://your-server-ip:9093"

Test API endpoints

curl http://localhost:9093/api/v1/status curl http://localhost:9093/api/v1/alerts

Check logs for errors

sudo journalctl -u alertmanager -f

Common issues

SymptomCauseFix
Service won't startConfiguration syntax error/usr/local/bin/alertmanager --config.check to validate config
Emails not sendingSMTP authentication failureCheck credentials and use app password for Gmail
Slack notifications failingInvalid webhook URLVerify webhook URL in Slack settings and update config
Alerts not routing correctlyLabel matching issuesCheck alert labels match routing rules exactly
Permission denied errorsIncorrect file ownershipsudo chown -R alertmanager:alertmanager /etc/alertmanager
Port 9093 not accessibleFirewall blocking accessOpen port 9093 in firewall rules
Prometheus can't reach AlertmanagerNetwork connectivityTest with curl http://localhost:9093/api/v1/status
Duplicate notificationsGrouping configurationAdjust group_by and group_interval settings

Next steps

Running this in production?

Want this handled for you? Setting up Alertmanager once is straightforward. Keeping it patched, monitored, backed up and tuned across environments is the harder part. See how we run infrastructure like this for European SaaS and e-commerce teams.

Automated install script

Run this to automate the entire setup

Need help?

Don't want to manage this yourself?

We handle managed devops services for businesses that depend on uptime. From initial setup to ongoing operations.