Configure Prometheus Alertmanager with email SMTP and Slack webhook integrations for comprehensive monitoring notifications. Set up alert routing rules and test notification delivery.
Prerequisites
- Root access to target server
- Basic understanding of Prometheus monitoring
- SMTP credentials for email notifications
- Slack workspace access for webhook setup
What this solves
Alertmanager receives alerts from Prometheus and routes them to notification channels like email and Slack. This tutorial shows you how to install Alertmanager, configure SMTP email delivery, set up Slack webhook integration, and create routing rules to send different alerts to different channels based on severity or service.
Step-by-step installation
Update system packages
Start by updating your package manager to ensure you get the latest versions.
sudo apt update && sudo apt upgrade -y
Create alertmanager user
Create a dedicated system user to run Alertmanager securely.
sudo useradd --system --no-create-home --shell /bin/false alertmanager
Download and install Alertmanager
Download the latest Alertmanager binary from the official releases page.
cd /tmp
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xzf alertmanager-0.27.0.linux-amd64.tar.gz
sudo cp alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo cp alertmanager-0.27.0.linux-amd64/amtool /usr/local/bin/
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/amtool
Create directory structure
Set up the required directories for Alertmanager configuration and data storage.
sudo mkdir -p /etc/alertmanager
sudo mkdir -p /var/lib/alertmanager
sudo chown alertmanager:alertmanager /etc/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager
Configure basic Alertmanager settings
Create the main configuration file with email and Slack notification settings.
global:
smtp_smarthost: 'smtp.gmail.com:587'
smtp_from: 'alerts@example.com'
smtp_auth_username: 'alerts@example.com'
smtp_auth_password: 'your-app-password'
smtp_require_tls: true
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
receiver: 'web.hook'
routes:
- match:
severity: critical
receiver: 'critical-alerts'
- match:
service: database
receiver: 'database-team'
- match:
severity: warning
receiver: 'warning-alerts'
receivers:
- name: 'web.hook'
email_configs:
- to: 'admin@example.com'
subject: 'Alertmanager Notification'
body: |
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
{{ end }}
- name: 'critical-alerts'
email_configs:
- to: 'oncall@example.com'
subject: 'CRITICAL: {{ .GroupLabels.alertname }}'
body: |
{{ range .Alerts }}
CRITICAL ALERT
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
Severity: {{ .Labels.severity }}
{{ end }}
slack_configs:
- api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
channel: '#alerts'
title: 'Critical Alert'
text: |
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
{{ end }}
color: danger
- name: 'database-team'
email_configs:
- to: 'dba-team@example.com'
subject: 'Database Alert: {{ .GroupLabels.alertname }}'
body: |
{{ range .Alerts }}
Database Alert
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Database: {{ .Labels.database }}
{{ end }}
slack_configs:
- api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
channel: '#database-alerts'
title: 'Database Alert'
text: |
{{ range .Alerts }}
Database Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Database: {{ .Labels.database }}
{{ end }}
color: warning
- name: 'warning-alerts'
slack_configs:
- api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
channel: '#monitoring'
title: 'Warning Alert'
text: |
{{ range .Alerts }}
Warning: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
{{ end }}
color: warning
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
Set correct file permissions
Secure the configuration file and set proper ownership.
sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml
sudo chmod 640 /etc/alertmanager/alertmanager.yml
Create systemd service
Set up a systemd service to manage Alertmanager automatically.
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--storage.path=/var/lib/alertmanager/ \
--web.external-url=http://localhost:9093
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target
Configure Slack webhook
Set up a Slack incoming webhook for alert notifications.
Enable and start Alertmanager
Start the Alertmanager service and enable it to start on boot.
sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
Configure firewall
Open port 9093 for Alertmanager web interface access.
sudo ufw allow 9093/tcp
sudo ufw reload
Configure email notifications
Set up Gmail SMTP
For Gmail, you need to create an app password instead of using your regular password.
Configure custom SMTP server
For other email providers, update the SMTP settings in the configuration file.
global:
smtp_smarthost: 'mail.example.com:587'
smtp_from: 'alerts@example.com'
smtp_auth_username: 'alerts@example.com'
smtp_auth_password: 'your-smtp-password'
smtp_require_tls: true
smtp_hello: 'alertmanager.example.com'
Test email configuration
Send a test alert to verify email delivery works correctly.
curl -XPOST http://localhost:9093/api/v1/alerts -H 'Content-Type: application/json' -d '[
{
"labels": {
"alertname": "TestAlert",
"severity": "warning"
},
"annotations": {
"summary": "Test alert from Alertmanager",
"description": "This is a test alert to verify email configuration"
},
"startsAt": "'$(date -Iseconds)'",
"endsAt": "'$(date -d '+1 hour' -Iseconds)'"
}
]'
Set up routing rules
Create advanced routing configuration
Configure sophisticated routing based on labels, severity, and services.
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 10s
group_interval: 10s
repeat_interval: 12h
receiver: 'default'
routes:
- match:
severity: critical
receiver: 'critical-pager'
group_wait: 5s
repeat_interval: 5m
- match_re:
service: mysql|postgresql|redis
receiver: 'database-team'
group_by: ['alertname', 'database']
- match:
team: frontend
receiver: 'frontend-team'
- match:
environment: production
receiver: 'production-alerts'
routes:
- match:
severity: critical
receiver: 'production-critical'
- match:
alertname: DeadMansSwitch
receiver: 'deadmansswitch'
repeat_interval: 30s
Configure time-based routing
Set up different notification behavior during business hours versus off-hours.
route:
routes:
- match:
severity: critical
receiver: 'critical-business-hours'
active_time_intervals:
- business-hours
- match:
severity: critical
receiver: 'critical-after-hours'
active_time_intervals:
- after-hours
time_intervals:
- name: business-hours
time_intervals:
- times:
- start_time: '09:00'
end_time: '17:00'
weekdays: ['monday:friday']
location: 'Europe/London'
- name: after-hours
time_intervals:
- times:
- start_time: '17:00'
end_time: '09:00'
weekdays: ['monday:friday']
location: 'Europe/London'
- weekdays: ['saturday', 'sunday']
location: 'Europe/London'
Apply configuration changes
Restart Alertmanager to load the new routing configuration.
sudo systemctl restart alertmanager
sudo systemctl status alertmanager
Test alert delivery
Send test alerts for different scenarios
Test critical alerts, database alerts, and warning notifications.
# Test critical alert
curl -XPOST http://localhost:9093/api/v1/alerts -H 'Content-Type: application/json' -d '[
{
"labels": {
"alertname": "HighCPUUsage",
"severity": "critical",
"instance": "web-01"
},
"annotations": {
"summary": "High CPU usage detected",
"description": "CPU usage is above 90% for more than 5 minutes"
},
"startsAt": "'$(date -Iseconds)'"
}
]'
Test database alert
curl -XPOST http://localhost:9093/api/v1/alerts -H 'Content-Type: application/json' -d '[
{
"labels": {
"alertname": "DatabaseConnectionHigh",
"severity": "warning",
"service": "database",
"database": "mysql"
},
"annotations": {
"summary": "Database connection count is high",
"description": "MySQL connection count is above threshold"
},
"startsAt": "'$(date -Iseconds)'"
}
]'
Verify alert delivery
Check the Alertmanager web interface and confirm notifications were sent.
# Check Alertmanager status
curl http://localhost:9093/api/v1/status
View active alerts
curl http://localhost:9093/api/v1/alerts
Check silences
curl http://localhost:9093/api/v1/silences
Connect with Prometheus
Configure Prometheus to send alerts
Add Alertmanager endpoint to your Prometheus configuration. This assumes you already have Prometheus installed.
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
rule_files:
- "alert_rules.yml"
Create sample alert rules
Define alert rules that Prometheus will evaluate and send to Alertmanager.
groups:
- name: system.rules
rules:
- alert: HighCPUUsage
expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 2m
labels:
severity: warning
team: infrastructure
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
description: "CPU usage is {{ $value }}% on instance {{ $labels.instance }}"
- alert: HighMemoryUsage
expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85
for: 5m
labels:
severity: critical
team: infrastructure
annotations:
summary: "High memory usage on {{ $labels.instance }}"
description: "Memory usage is {{ $value }}% on instance {{ $labels.instance }}"
- alert: DiskSpaceLow
expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
for: 1m
labels:
severity: critical
team: infrastructure
annotations:
summary: "Low disk space on {{ $labels.instance }}"
description: "Disk space is {{ $value }}% full on instance {{ $labels.instance }}"
- name: database.rules
rules:
- alert: MySQLDown
expr: mysql_up == 0
for: 0m
labels:
severity: critical
service: database
database: mysql
annotations:
summary: "MySQL is down on {{ $labels.instance }}"
description: "MySQL database server is not responding"
Reload Prometheus configuration
Apply the new configuration to start sending alerts to Alertmanager.
sudo systemctl restart prometheus
curl -X POST http://localhost:9090/-/reload
Verify your setup
# Check Alertmanager status
sudo systemctl status alertmanager
Verify Alertmanager is listening
ss -tlnp | grep 9093
Check configuration syntax
/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --config.check
View web interface
echo "Access Alertmanager at: http://your-server-ip:9093"
Test API endpoints
curl http://localhost:9093/api/v1/status
curl http://localhost:9093/api/v1/alerts
Check logs for errors
sudo journalctl -u alertmanager -f
Common issues
| Symptom | Cause | Fix |
|---|---|---|
| Service won't start | Configuration syntax error | /usr/local/bin/alertmanager --config.check to validate config |
| Emails not sending | SMTP authentication failure | Check credentials and use app password for Gmail |
| Slack notifications failing | Invalid webhook URL | Verify webhook URL in Slack settings and update config |
| Alerts not routing correctly | Label matching issues | Check alert labels match routing rules exactly |
| Permission denied errors | Incorrect file ownership | sudo chown -R alertmanager:alertmanager /etc/alertmanager |
| Port 9093 not accessible | Firewall blocking access | Open port 9093 in firewall rules |
| Prometheus can't reach Alertmanager | Network connectivity | Test with curl http://localhost:9093/api/v1/status |
| Duplicate notifications | Grouping configuration | Adjust group_by and group_interval settings |
Next steps
- Set up MySQL backup monitoring with Prometheus alerts
- Configure Prometheus long-term storage with Thanos
- Set up Prometheus Alertmanager webhook notifications for Loki log alerts
- Configure advanced Grafana dashboards and alerting with Prometheus integration
- Implement Prometheus federation for multi-cluster monitoring
Running this in production?
Automated install script
Run this to automate the entire setup
#!/usr/bin/env bash
set -euo pipefail
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
# Default configuration
ALERTMANAGER_VERSION="0.27.0"
ALERTMANAGER_USER="alertmanager"
ALERTMANAGER_CONFIG_DIR="/etc/alertmanager"
ALERTMANAGER_DATA_DIR="/var/lib/alertmanager"
ALERTMANAGER_PORT="9093"
# Usage function
usage() {
echo "Usage: $0 [OPTIONS]"
echo "Options:"
echo " --smtp-host SMTP_HOST SMTP server (default: smtp.gmail.com:587)"
echo " --smtp-user SMTP_USER SMTP username"
echo " --smtp-pass SMTP_PASS SMTP password"
echo " --email EMAIL Alert email recipient"
echo " --slack-webhook URL Slack webhook URL"
echo " --help Show this help message"
exit 1
}
# Parse arguments
SMTP_HOST="smtp.gmail.com:587"
SMTP_USER=""
SMTP_PASS=""
ALERT_EMAIL=""
SLACK_WEBHOOK=""
while [[ $# -gt 0 ]]; do
case $1 in
--smtp-host) SMTP_HOST="$2"; shift 2 ;;
--smtp-user) SMTP_USER="$2"; shift 2 ;;
--smtp-pass) SMTP_PASS="$2"; shift 2 ;;
--email) ALERT_EMAIL="$2"; shift 2 ;;
--slack-webhook) SLACK_WEBHOOK="$2"; shift 2 ;;
--help) usage ;;
*) echo -e "${RED}Unknown option: $1${NC}"; usage ;;
esac
done
# Cleanup function
cleanup() {
echo -e "${RED}[ERROR] Installation failed. Cleaning up...${NC}"
systemctl stop alertmanager 2>/dev/null || true
systemctl disable alertmanager 2>/dev/null || true
rm -f /etc/systemd/system/alertmanager.service
rm -rf /tmp/alertmanager-*
echo -e "${YELLOW}Cleanup completed${NC}"
exit 1
}
trap cleanup ERR
# Check if running as root or with sudo
if [[ $EUID -ne 0 ]]; then
echo -e "${RED}This script must be run as root or with sudo${NC}"
exit 1
fi
echo -e "${GREEN}Starting Alertmanager installation...${NC}"
# Detect distribution and set package manager
echo -e "${YELLOW}[1/8] Detecting distribution...${NC}"
if [ -f /etc/os-release ]; then
. /etc/os-release
case "$ID" in
ubuntu|debian)
PKG_MGR="apt"
PKG_UPDATE="apt update"
PKG_INSTALL="apt install -y"
FIREWALL_CMD="ufw"
;;
almalinux|rocky|centos|rhel|ol|fedora)
PKG_MGR="dnf"
PKG_UPDATE="dnf check-update || true"
PKG_INSTALL="dnf install -y"
FIREWALL_CMD="firewall-cmd"
;;
amzn)
PKG_MGR="yum"
PKG_UPDATE="yum check-update || true"
PKG_INSTALL="yum install -y"
FIREWALL_CMD="firewall-cmd"
;;
*)
echo -e "${RED}Unsupported distribution: $ID${NC}"
exit 1
;;
esac
echo -e "${GREEN}Detected: $PRETTY_NAME${NC}"
else
echo -e "${RED}Cannot detect distribution${NC}"
exit 1
fi
# Update system packages
echo -e "${YELLOW}[2/8] Updating system packages...${NC}"
$PKG_UPDATE
if [ "$PKG_MGR" = "apt" ]; then
$PKG_INSTALL wget curl tar
else
$PKG_INSTALL wget curl tar which
fi
# Create alertmanager user
echo -e "${YELLOW}[3/8] Creating alertmanager user...${NC}"
if ! id "$ALERTMANAGER_USER" &>/dev/null; then
useradd --system --no-create-home --shell /bin/false "$ALERTMANAGER_USER"
echo -e "${GREEN}User $ALERTMANAGER_USER created${NC}"
else
echo -e "${YELLOW}User $ALERTMANAGER_USER already exists${NC}"
fi
# Download and install Alertmanager
echo -e "${YELLOW}[4/8] Downloading and installing Alertmanager...${NC}"
cd /tmp
wget -q "https://github.com/prometheus/alertmanager/releases/download/v${ALERTMANAGER_VERSION}/alertmanager-${ALERTMANAGER_VERSION}.linux-amd64.tar.gz"
tar xzf "alertmanager-${ALERTMANAGER_VERSION}.linux-amd64.tar.gz"
cp "alertmanager-${ALERTMANAGER_VERSION}.linux-amd64/alertmanager" /usr/local/bin/
cp "alertmanager-${ALERTMANAGER_VERSION}.linux-amd64/amtool" /usr/local/bin/
chmod 755 /usr/local/bin/alertmanager
chmod 755 /usr/local/bin/amtool
chown "$ALERTMANAGER_USER:$ALERTMANAGER_USER" /usr/local/bin/alertmanager
chown "$ALERTMANAGER_USER:$ALERTMANAGER_USER" /usr/local/bin/amtool
rm -rf "alertmanager-${ALERTMANAGER_VERSION}.linux-amd64"*
# Create directory structure
echo -e "${YELLOW}[5/8] Creating directory structure...${NC}"
mkdir -p "$ALERTMANAGER_CONFIG_DIR"
mkdir -p "$ALERTMANAGER_DATA_DIR"
chown "$ALERTMANAGER_USER:$ALERTMANAGER_USER" "$ALERTMANAGER_CONFIG_DIR"
chown "$ALERTMANAGER_USER:$ALERTMANAGER_USER" "$ALERTMANAGER_DATA_DIR"
chmod 755 "$ALERTMANAGER_CONFIG_DIR"
chmod 755 "$ALERTMANAGER_DATA_DIR"
# Create Alertmanager configuration
echo -e "${YELLOW}[6/8] Creating Alertmanager configuration...${NC}"
cat > "$ALERTMANAGER_CONFIG_DIR/alertmanager.yml" << EOF
global:
smtp_smarthost: '${SMTP_HOST}'
smtp_from: '${SMTP_USER}'
smtp_auth_username: '${SMTP_USER}'
smtp_auth_password: '${SMTP_PASS}'
smtp_require_tls: true
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
receiver: 'default'
routes:
- match:
severity: critical
receiver: 'critical-alerts'
- match:
severity: warning
receiver: 'warning-alerts'
receivers:
- name: 'default'
email_configs:
- to: '${ALERT_EMAIL:-admin@example.com}'
subject: 'Alertmanager Notification'
body: |
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
{{ end }}
- name: 'critical-alerts'
email_configs:
- to: '${ALERT_EMAIL:-admin@example.com}'
subject: 'CRITICAL: {{ .GroupLabels.alertname }}'
body: |
{{ range .Alerts }}
CRITICAL ALERT
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
Severity: {{ .Labels.severity }}
{{ end }}
EOF
# Add Slack configuration if webhook provided
if [[ -n "$SLACK_WEBHOOK" ]]; then
cat >> "$ALERTMANAGER_CONFIG_DIR/alertmanager.yml" << EOF
slack_configs:
- api_url: '${SLACK_WEBHOOK}'
channel: '#alerts'
title: 'Critical Alert'
text: |
{{ range .Alerts }}
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
{{ end }}
color: danger
EOF
fi
cat >> "$ALERTMANAGER_CONFIG_DIR/alertmanager.yml" << EOF
- name: 'warning-alerts'
email_configs:
- to: '${ALERT_EMAIL:-admin@example.com}'
subject: 'Warning: {{ .GroupLabels.alertname }}'
body: |
{{ range .Alerts }}
Warning Alert
Alert: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
Instance: {{ .Labels.instance }}
{{ end }}
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
EOF
chown "$ALERTMANAGER_USER:$ALERTMANAGER_USER" "$ALERTMANAGER_CONFIG_DIR/alertmanager.yml"
chmod 644 "$ALERTMANAGER_CONFIG_DIR/alertmanager.yml"
# Create systemd service
echo -e "${YELLOW}[7/8] Creating systemd service...${NC}"
cat > /etc/systemd/system/alertmanager.service << EOF
[Unit]
Description=Alertmanager
Documentation=https://prometheus.io/docs/alerting/latest/alertmanager/
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=$ALERTMANAGER_USER
Group=$ALERTMANAGER_USER
ExecStart=/usr/local/bin/alertmanager \\
--config.file=$ALERTMANAGER_CONFIG_DIR/alertmanager.yml \\
--storage.path=$ALERTMANAGER_DATA_DIR \\
--web.listen-address=0.0.0.0:$ALERTMANAGER_PORT \\
--web.external-url=http://localhost:$ALERTMANAGER_PORT \\
--cluster.listen-address=""
Restart=always
RestartSec=3
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable alertmanager
systemctl start alertmanager
# Configure firewall
echo -e "${YELLOW}[8/8] Configuring firewall...${NC}"
if [ "$PKG_MGR" = "apt" ]; then
if command -v ufw &> /dev/null; then
ufw allow "$ALERTMANAGER_PORT/tcp" comment "Alertmanager"
fi
else
if command -v firewall-cmd &> /dev/null && systemctl is-active --quiet firewalld; then
firewall-cmd --permanent --add-port="$ALERTMANAGER_PORT/tcp"
firewall-cmd --reload
fi
fi
# Verification
echo -e "${YELLOW}Verifying installation...${NC}"
sleep 5
if systemctl is-active --quiet alertmanager; then
echo -e "${GREEN}✓ Alertmanager service is running${NC}"
else
echo -e "${RED}✗ Alertmanager service is not running${NC}"
systemctl status alertmanager
exit 1
fi
if curl -s "http://localhost:$ALERTMANAGER_PORT/-/ready" | grep -q "Alertmanager is Ready"; then
echo -e "${GREEN}✓ Alertmanager is responding${NC}"
else
echo -e "${RED}✗ Alertmanager is not responding${NC}"
exit 1
fi
echo -e "${GREEN}Alertmanager installation completed successfully!${NC}"
echo -e "${YELLOW}Access Alertmanager at: http://your-server-ip:$ALERTMANAGER_PORT${NC}"
echo -e "${YELLOW}Configuration file: $ALERTMANAGER_CONFIG_DIR/alertmanager.yml${NC}"
echo -e "${YELLOW}Data directory: $ALERTMANAGER_DATA_DIR${NC}"
echo ""
echo -e "${YELLOW}Next steps:${NC}"
echo "1. Update your email and Slack settings in $ALERTMANAGER_CONFIG_DIR/alertmanager.yml"
echo "2. Configure Prometheus to send alerts to http://localhost:$ALERTMANAGER_PORT"
echo "3. Restart alertmanager after configuration changes: systemctl restart alertmanager"
Review the script before running. Execute with: bash install.sh