Implement SNMP trap monitoring and alerting system for proactive network management

Update system packages

Start by updating your package manager to ensure you get the latest versions of SNMP packages.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install SNMP packages and MIB tools

Install Net-SNMP daemon, utilities, and MIB management tools for comprehensive trap handling.

sudo apt install -y snmp snmp-mibs-downloader snmptrapd postfix mailutils curl jq

sudo dnf install -y net-snmp net-snmp-utils postfix mailx curl jq

Create trap handler directory structure

Set up directories for trap handlers, logs, and configuration files with proper permissions.

sudo mkdir -p /etc/snmp/traphandlers
sudo mkdir -p /var/log/snmp/traps
sudo mkdir -p /var/lib/snmp/trapdb
sudo chown -R snmp:snmp /var/log/snmp
sudo chown -R snmp:snmp /var/lib/snmp
sudo chmod 755 /etc/snmp/traphandlers

Configure snmptrapd daemon

Create the main snmptrapd configuration file with trap receiving settings and handler definitions.

# SNMP Trap Daemon Configuration
Listen on all interfaces, port 162 (standard SNMP trap port)
snmpTrapdAddr udp:162,udp6:162

Authentication settings
disableAuthorization yes

Logging configuration
logOption s 6
logTimestamp yes

Output format for traps
format1 %.4y-%.2m-%.2l %.2h:%.2j:%.2k [%B] %N: %W\n\t%V\n
format2 %.4y-%.2m-%.2l %.2h:%.2j:%.2k [%B] %N: Enterprise Specific Trap (%w) Uptime: %#T\n\t%v\n

Trap handlers for different trap types
traphandle default /etc/snmp/traphandlers/default_handler.sh
traphandle SNMPv2-MIB::coldStart /etc/snmp/traphandlers/system_handler.sh
traphandle SNMPv2-MIB::warmStart /etc/snmp/traphandlers/system_handler.sh
traphandle SNMPv2-MIB::linkDown /etc/snmp/traphandlers/interface_handler.sh
traphandle SNMPv2-MIB::linkUp /etc/snmp/traphandlers/interface_handler.sh
traphandle SNMPv2-MIB::authenticationFailure /etc/snmp/traphandlers/security_handler.sh

Create PID file
pidFile /var/run/snmptrapd.pid

Run as snmp user
agentuser snmp

Create default trap handler script

Build the main trap processing script that logs all traps and triggers alert mechanisms.

#!/bin/bash

Default SNMP Trap Handler
Processes all incoming SNMP traps and triggers alerts

TRAP_LOG="/var/log/snmp/traps/default.log"
ALERT_CONFIG="/etc/snmp/alert_config.conf"
TRAP_DB="/var/lib/snmp/trapdb/traps.db"

Source alerting configuration
if [ -f "$ALERT_CONFIG" ]; then
    source "$ALERT_CONFIG"
fi

Get trap information from environment variables
TRAP_TIME=$(date '+%Y-%m-%d %H:%M:%S')
SOURCE_IP="${SNMP_TRANSPORT_ADDRESS}"
OID="${SNMP_COMMAND}"
UPTIME="${SNMP_ARG2:-Unknown}"
TRAP_OID="${SNMP_ARG3:-Unknown}"

Read all trap variables
TRAP_DATA=""
while read line; do
    TRAP_DATA+="$line\n"
done

Log trap to file
echo "[$TRAP_TIME] Source: $SOURCE_IP, OID: $TRAP_OID" >> "$TRAP_LOG"
echo -e "$TRAP_DATA" >> "$TRAP_LOG"
echo "---" >> "$TRAP_LOG"

Store in simple database for tracking
echo "$TRAP_TIME|$SOURCE_IP|$TRAP_OID|$(echo -e "$TRAP_DATA" | tr '\n' ' ')" >> "$TRAP_DB"

Determine trap severity
SEVERITY="INFO"
case "$TRAP_OID" in
    coldStart|warmStart)
        SEVERITY="WARNING"
        ;;
    linkDown|authenticationFailure)
        SEVERITY="CRITICAL"
        ;;
    linkUp)
        SEVERITY="INFO"
        ;;
esac

Send alerts based on severity
if [ "$SEVERITY" = "CRITICAL" ] || [ "$SEVERITY" = "WARNING" ]; then
    /etc/snmp/traphandlers/send_alert.sh "$SEVERITY" "$SOURCE_IP" "$TRAP_OID" "$TRAP_DATA"
fi

exit 0

sudo chmod 755 /etc/snmp/traphandlers/default_handler.sh

Create specialized trap handlers

Create specific handlers for different types of network events with targeted processing logic.

#!/bin/bash

System Event Trap Handler (coldStart, warmStart)

TRAP_LOG="/var/log/snmp/traps/system.log"
SOURCE_IP="${SNMP_TRANSPORT_ADDRESS}"
TRAP_TIME=$(date '+%Y-%m-%d %H:%M:%S')

Read trap data
TRAP_DATA=""
while read line; do
    TRAP_DATA+="$line\n"
done

Determine event type
EVENT_TYPE="System Event"
if [[ "$SNMP_ARG3" == "coldStart" ]]; then
    EVENT_TYPE="System Cold Start"
elif [[ "$SNMP_ARG3" == "warmStart" ]]; then
    EVENT_TYPE="System Warm Start"
fi

Log system event
echo "[$TRAP_TIME] $EVENT_TYPE from $SOURCE_IP" >> "$TRAP_LOG"
echo -e "$TRAP_DATA" >> "$TRAP_LOG"
echo "---" >> "$TRAP_LOG"

Trigger system event alert
/etc/snmp/traphandlers/send_alert.sh "WARNING" "$SOURCE_IP" "$EVENT_TYPE" "Device restarted: $TRAP_DATA"

exit 0

#!/bin/bash

Interface Event Trap Handler (linkUp, linkDown)

TRAP_LOG="/var/log/snmp/traps/interface.log"
SOURCE_IP="${SNMP_TRANSPORT_ADDRESS}"
TRAP_TIME=$(date '+%Y-%m-%d %H:%M:%S')

Read trap data
TRAP_DATA=""
INTERFACE_ID="Unknown"
while read line; do
    TRAP_DATA+="$line\n"
    # Extract interface ID if present
    if [[ "$line" == "ifIndex" ]]; then
        INTERFACE_ID=$(echo "$line" | grep -o '[0-9]\+' | head -1)
    fi
done

Determine link status
LINK_STATUS="Unknown"
SEVERITY="INFO"
if [[ "$SNMP_ARG3" == "linkDown" ]]; then
    LINK_STATUS="Link Down"
    SEVERITY="CRITICAL"
elif [[ "$SNMP_ARG3" == "linkUp" ]]; then
    LINK_STATUS="Link Up"
    SEVERITY="INFO"
fi

Log interface event
echo "[$TRAP_TIME] $LINK_STATUS on interface $INTERFACE_ID from $SOURCE_IP" >> "$TRAP_LOG"
echo -e "$TRAP_DATA" >> "$TRAP_LOG"
echo "---" >> "$TRAP_LOG"

Send alert for link down events
if [ "$SEVERITY" = "CRITICAL" ]; then
    /etc/snmp/traphandlers/send_alert.sh "$SEVERITY" "$SOURCE_IP" "Interface Down" "Interface $INTERFACE_ID is down on device $SOURCE_IP"
fi

exit 0

#!/bin/bash

Security Event Trap Handler (authenticationFailure)

TRAP_LOG="/var/log/snmp/traps/security.log"
SOURCE_IP="${SNMP_TRANSPORT_ADDRESS}"
TRAP_TIME=$(date '+%Y-%m-%d %H:%M:%S')

Read trap data
TRAP_DATA=""
while read line; do
    TRAP_DATA+="$line\n"
done

Log security event
echo "[$TRAP_TIME] SECURITY ALERT: Authentication failure from $SOURCE_IP" >> "$TRAP_LOG"
echo -e "$TRAP_DATA" >> "$TRAP_LOG"
echo "---" >> "$TRAP_LOG"

Always send critical alert for security events
/etc/snmp/traphandlers/send_alert.sh "CRITICAL" "$SOURCE_IP" "Authentication Failure" "SNMP authentication failure detected from $SOURCE_IP"

exit 0

sudo chmod 755 /etc/snmp/traphandlers/*.sh

Create alert configuration and sender script

Configure email and webhook alert settings with the alert dispatch script.

# SNMP Trap Alert Configuration

Email settings
EMAIL_ENABLED=true
EMAIL_TO="admin@example.com"
EMAIL_FROM="snmp-monitor@example.com"
EMAIL_SUBJECT_PREFIX="[SNMP Alert]"

Webhook settings
WEBHOOK_ENABLED=true
WEBHOOK_URL="https://hooks.example.com/snmp-alerts"
WEBHOOK_SECRET="your-webhook-secret-here"

Slack webhook (optional)
SLACK_ENABLED=false
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"

Alert throttling (seconds between duplicate alerts)
ALERT_THROTTLE=300

Log settings
ALERT_LOG="/var/log/snmp/traps/alerts.log"

#!/bin/bash

SNMP Trap Alert Sender
Sends alerts via email, webhook, and other channels

ALERT_CONFIG="/etc/snmp/alert_config.conf"
THROTTLE_FILE="/var/lib/snmp/trapdb/alert_throttle"

Source configuration
if [ -f "$ALERT_CONFIG" ]; then
    source "$ALERT_CONFIG"
else
    echo "Alert configuration not found: $ALERT_CONFIG"
    exit 1
fi

SEVERITY="$1"
SOURCE_IP="$2"
EVENT_TYPE="$3"
MESSAGE="$4"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

Throttle duplicate alerts
ALERT_KEY="${SOURCE_IP}_${EVENT_TYPE}"
if [ -f "$THROTTLE_FILE" ]; then
    LAST_ALERT=$(grep "^$ALERT_KEY" "$THROTTLE_FILE" | cut -d'|' -f2)
    if [ -n "$LAST_ALERT" ]; then
        TIME_DIFF=$(($(date +%s) - $LAST_ALERT))
        if [ $TIME_DIFF -lt $ALERT_THROTTLE ]; then
            echo "[$TIMESTAMP] Alert throttled: $ALERT_KEY" >> "$ALERT_LOG"
            exit 0
        fi
    fi
fi

Update throttle file
grep -v "^$ALERT_KEY" "$THROTTLE_FILE" 2>/dev/null > "${THROTTLE_FILE}.tmp" || touch "${THROTTLE_FILE}.tmp"
echo "$ALERT_KEY|$(date +%s)" >> "${THROTTLE_FILE}.tmp"
mv "${THROTTLE_FILE}.tmp" "$THROTTLE_FILE"

Log alert
echo "[$TIMESTAMP] Sending $SEVERITY alert for $SOURCE_IP: $EVENT_TYPE" >> "$ALERT_LOG"

Send email alert
if [ "$EMAIL_ENABLED" = "true" ]; then
    EMAIL_BODY="SNMP Trap Alert\n\nSeverity: $SEVERITY\nSource: $SOURCE_IP\nEvent: $EVENT_TYPE\nTime: $TIMESTAMP\n\nDetails:\n$MESSAGE"
    echo -e "$EMAIL_BODY" | mail -s "$EMAIL_SUBJECT_PREFIX $SEVERITY - $EVENT_TYPE from $SOURCE_IP" "$EMAIL_TO"
fi

Send webhook alert
if [ "$WEBHOOK_ENABLED" = "true" ] && [ -n "$WEBHOOK_URL" ]; then
    WEBHOOK_PAYLOAD=$(cat <Send Slack alert
if [ "$SLACK_ENABLED" = "true" ] && [ -n "$SLACK_WEBHOOK_URL" ]; then
    SLACK_COLOR="good"
    case "$SEVERITY" in
        "CRITICAL") SLACK_COLOR="danger" ;;
        "WARNING") SLACK_COLOR="warning" ;;
    esac
    
    SLACK_PAYLOAD=$(cat <

sudo chmod 755 /etc/snmp/traphandlers/send_alert.sh

Configure Nagios integration

Create Nagios passive check integration for SNMP trap events.

#!/bin/bash

Nagios SNMP Trap Integration
Sends passive check results to Nagios based on trap events

NAGIOS_COMMAND_FILE="/var/lib/nagios3/rw/nagios.cmd"
NAGIOS_CONFIG="/etc/snmp/nagios_trap_config.conf"

Source Nagios configuration
if [ -f "$NAGIOS_CONFIG" ]; then
    source "$NAGIOS_CONFIG"
else
    # Default settings
    NAGIOS_ENABLED=false
    NAGIOS_SERVICE_PREFIX="SNMP_"
fi

if [ "$NAGIOS_ENABLED" != "true" ]; then
    exit 0
fi

SOURCE_IP="$1"
EVENT_TYPE="$2"
SEVERITY="$3"
MESSAGE="$4"

Map severity to Nagios states
NAGIOS_STATE=0
case "$SEVERITY" in
    "CRITICAL") NAGIOS_STATE=2 ;;
    "WARNING") NAGIOS_STATE=1 ;;
    "INFO") NAGIOS_STATE=0 ;;
esac

Create service name
SERVICE_NAME="${NAGIOS_SERVICE_PREFIX}${EVENT_TYPE}"

Map IP to hostname if configured
HOSTNAME="$SOURCE_IP"
if [ -f "/etc/snmp/ip_hostname_map.conf" ]; then
    MAPPED_HOST=$(grep "^$SOURCE_IP" /etc/snmp/ip_hostname_map.conf | cut -d' ' -f2)
    if [ -n "$MAPPED_HOST" ]; then
        HOSTNAME="$MAPPED_HOST"
    fi
fi

Submit passive check result
TIMESTAMP=$(date +%s)
echo "[$TIMESTAMP] PROCESS_SERVICE_CHECK_RESULT;$HOSTNAME;$SERVICE_NAME;$NAGIOS_STATE;$MESSAGE" >> "$NAGIOS_COMMAND_FILE"

exit 0

# Nagios SNMP Trap Integration Configuration

NAGIOS_ENABLED=true
NAGIOS_SERVICE_PREFIX="SNMP_"

Command file location (check your Nagios installation)
NAGIOS_COMMAND_FILE="/var/lib/nagios3/rw/nagios.cmd"

# IP to Hostname mapping for Nagios integration
Format: IP_ADDRESS HOSTNAME
203.0.113.10 router-01
203.0.113.11 switch-01
203.0.113.12 server-01

sudo chmod 755 /etc/snmp/traphandlers/nagios_integration.sh

Create Zabbix integration script

Build Zabbix integration for trap data using zabbix_sender utility.

#!/bin/bash

Zabbix SNMP Trap Integration
Sends trap data to Zabbix server using zabbix_sender

ZABBIX_CONFIG="/etc/snmp/zabbix_trap_config.conf"

Source Zabbix configuration
if [ -f "$ZABBIX_CONFIG" ]; then
    source "$ZABBIX_CONFIG"
else
    # Default settings
    ZABBIX_ENABLED=false
    ZABBIX_SERVER="127.0.0.1"
    ZABBIX_PORT=10051
fi

if [ "$ZABBIX_ENABLED" != "true" ]; then
    exit 0
fi

SOURCE_IP="$1"
EVENT_TYPE="$2"
SEVERITY="$3"
MESSAGE="$4"

Map IP to Zabbix hostname if configured
ZABBIX_HOST="$SOURCE_IP"
if [ -f "/etc/snmp/zabbix_host_map.conf" ]; then
    MAPPED_HOST=$(grep "^$SOURCE_IP" /etc/snmp/zabbix_host_map.conf | cut -d' ' -f2)
    if [ -n "$MAPPED_HOST" ]; then
        ZABBIX_HOST="$MAPPED_HOST"
    fi
fi

Create trap item key
ITEM_KEY="snmp.trap[${EVENT_TYPE}]"

Send data to Zabbix
echo "$ZABBIX_HOST $ITEM_KEY $MESSAGE" | zabbix_sender -z "$ZABBIX_SERVER" -p "$ZABBIX_PORT" -T -i - >/dev/null 2>&1

Send trap count metric
COUNT_KEY="snmp.trap.count[${EVENT_TYPE}]"
echo "$ZABBIX_HOST $COUNT_KEY 1" | zabbix_sender -z "$ZABBIX_SERVER" -p "$ZABBIX_PORT" -T -i - >/dev/null 2>&1

exit 0

# Zabbix SNMP Trap Integration Configuration

ZABBIX_ENABLED=true
ZABBIX_SERVER="203.0.113.100"
ZABBIX_PORT=10051

# IP to Zabbix Hostname mapping
Format: IP_ADDRESS ZABBIX_HOSTNAME
203.0.113.10 Router-Office-01
203.0.113.11 Switch-Floor-01
203.0.113.12 Server-Web-01

Install Zabbix sender utility

Install zabbix_sender for sending trap data to Zabbix server.

sudo apt install -y zabbix-sender

sudo dnf install -y zabbix-sender

Update trap handlers with integrations

Modify the main trap handlers to call monitoring system integrations.

sudo cp /etc/snmp/traphandlers/default_handler.sh /etc/snmp/traphandlers/default_handler.sh.backup

#!/bin/bash

Enhanced Default SNMP Trap Handler with Monitoring Integration

TRAP_LOG="/var/log/snmp/traps/default.log"
ALERT_CONFIG="/etc/snmp/alert_config.conf"
TRAP_DB="/var/lib/snmp/trapdb/traps.db"

Source alerting configuration
if [ -f "$ALERT_CONFIG" ]; then
    source "$ALERT_CONFIG"
fi

Get trap information from environment variables
TRAP_TIME=$(date '+%Y-%m-%d %H:%M:%S')
SOURCE_IP="${SNMP_TRANSPORT_ADDRESS}"
OID="${SNMP_COMMAND}"
UPTIME="${SNMP_ARG2:-Unknown}"
TRAP_OID="${SNMP_ARG3:-Unknown}"

Read all trap variables
TRAP_DATA=""
while read line; do
    TRAP_DATA+="$line\n"
done

Log trap to file
echo "[$TRAP_TIME] Source: $SOURCE_IP, OID: $TRAP_OID" >> "$TRAP_LOG"
echo -e "$TRAP_DATA" >> "$TRAP_LOG"
echo "---" >> "$TRAP_LOG"

Store in simple database for tracking
echo "$TRAP_TIME|$SOURCE_IP|$TRAP_OID|$(echo -e "$TRAP_DATA" | tr '\n' ' ')" >> "$TRAP_DB"

Determine trap severity and event type
SEVERITY="INFO"
EVENT_TYPE="Generic"
case "$TRAP_OID" in
    coldStart)
        SEVERITY="WARNING"
        EVENT_TYPE="ColdStart"
        ;;
    warmStart)
        SEVERITY="WARNING"
        EVENT_TYPE="WarmStart"
        ;;
    linkDown)
        SEVERITY="CRITICAL"
        EVENT_TYPE="LinkDown"
        ;;
    linkUp)
        SEVERITY="INFO"
        EVENT_TYPE="LinkUp"
        ;;
    authenticationFailure)
        SEVERITY="CRITICAL"
        EVENT_TYPE="AuthFailure"
        ;;
esac

Send alerts based on severity
if [ "$SEVERITY" = "CRITICAL" ] || [ "$SEVERITY" = "WARNING" ]; then
    /etc/snmp/traphandlers/send_alert.sh "$SEVERITY" "$SOURCE_IP" "$EVENT_TYPE" "$TRAP_DATA"
fi

Send to monitoring systems
/etc/snmp/traphandlers/nagios_integration.sh "$SOURCE_IP" "$EVENT_TYPE" "$SEVERITY" "$(echo -e "$TRAP_DATA" | head -1)" &
/etc/snmp/traphandlers/zabbix_integration.sh "$SOURCE_IP" "$EVENT_TYPE" "$SEVERITY" "$(echo -e "$TRAP_DATA" | head -1)" &

exit 0

Configure systemd service

Create and configure the systemd service for snmptrapd with proper security settings.

[Unit]
Description=Net-SNMP Trap Daemon
After=network.target
Wants=network.target

[Service]
Type=forking
PIDFile=/var/run/snmptrapd.pid
ExecStart=/usr/sbin/snmptrapd -Lsd -p /var/run/snmptrapd.pid
ExecReload=/bin/kill -HUP $MAINPID
User=snmp
Group=snmp
Restart=on-failure
RestartSec=5

Security settings
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/log/snmp /var/lib/snmp /var/run

[Install]
WantedBy=multi-user.target

Configure firewall for SNMP traps

Open UDP port 162 for incoming SNMP trap traffic.

sudo ufw allow 162/udp comment 'SNMP Traps'
sudo ufw reload

sudo firewall-cmd --permanent --add-port=162/udp
sudo firewall-cmd --reload

Create log rotation configuration

Set up log rotation to prevent trap logs from consuming excessive disk space.

/var/log/snmp/traps/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 snmp snmp
    postrotate
        if [ -f /var/run/snmptrapd.pid ]; then
            /bin/kill -HUP cat /var/run/snmptrapd.pid
        fi
    endscript
}

Enable and start the service

Enable snmptrapd to start automatically and start the service.

sudo systemctl daemon-reload
sudo systemctl enable snmptrapd
sudo systemctl start snmptrapd
sudo systemctl status snmptrapd

Update alert configuration

Customize the alert configuration file with your specific email and webhook settings.

sudo nano /etc/snmp/alert_config.conf

Update the EMAIL_TO, WEBHOOK_URL, and other settings according to your environment. The system needs a working mail server for email alerts. You can learn more about comprehensive monitoring setups in our Nagios SNMP monitoring guide.

Test email configuration

Verify that email alerts are working properly with the mail system.

echo "Test email from SNMP trap system" | mail -s "SNMP Test Alert" admin@example.com