Configure intrusion detection with OSSEC and fail2ban integration

Advanced 45 min Apr 30, 2026 524 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Set up comprehensive intrusion detection by integrating OSSEC HIDS with fail2ban for automated threat response. This advanced configuration provides real-time monitoring, log analysis, and automated IP blocking for enhanced server security.

Prerequisites

  • Root or sudo access
  • Basic knowledge of Linux security concepts
  • Understanding of log file formats
  • Familiarity with iptables firewall rules

What this solves

This tutorial shows you how to integrate OSSEC Host-based Intrusion Detection System (HIDS) with fail2ban to create a comprehensive security monitoring solution. OSSEC monitors system logs and file integrity while fail2ban automatically blocks suspicious IP addresses, creating a layered defense against intrusions and automated attacks.

Step-by-step configuration

Update system packages

Start by updating your package manager to ensure you get the latest security updates.

sudo apt update && sudo apt upgrade -y
sudo dnf update -y

Install OSSEC dependencies

Install the required packages for building OSSEC from source and running the system.

sudo apt install -y build-essential gcc make unzip sendmail inotify-tools wget libevent-dev libssl-dev libz-dev
sudo dnf install -y gcc gcc-c++ make unzip sendmail inotify-tools wget libevent-devel openssl-devel zlib-devel

Download and install OSSEC

Download the latest OSSEC release and compile it for your system.

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/v3.7.0.tar.gz
tar -xzf v3.7.0.tar.gz
cd ossec-hids-3.7.0

Configure OSSEC installation

Run the interactive installation script and select server installation for local monitoring.

sudo ./install.sh

When prompted, select these options:

  • Installation type: local
  • Email notification: y (enter your email)
  • SMTP server: localhost
  • Firewall response: y
  • System integrity check: y
  • Rootkit detection: y
  • Active response: y

Install fail2ban

Install fail2ban which will handle the automatic IP blocking based on OSSEC alerts.

sudo apt install -y fail2ban
sudo dnf install -y epel-release
sudo dnf install -y fail2ban

Configure OSSEC for fail2ban integration

Modify OSSEC configuration to enable log output that fail2ban can parse.

Create custom OSSEC rules for fail2ban integration

Add custom rules to detect specific attack patterns and generate alerts that fail2ban can process.

Configure fail2ban jail for OSSEC

Create a custom jail configuration that monitors OSSEC alerts and blocks offending IPs.

[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3
banaction = iptables-multiport
protocol = tcp
chain = INPUT
action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
action_mw = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
           %(mta)s-whois[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]
action_mwl = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
            %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
action = %(action_)s

[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600

[apache-auth]
enabled = true
port = http,https
filter = apache-auth
logpath = /var/log/apache2/*error.log
maxretry = 6
bantime = 3600

[apache-badbots]
enabled = true
port = http,https
filter = apache-badbots
logpath = /var/log/apache2/*error.log
maxretry = 2
bantime = 86400

[apache-noscript]
enabled = true
port = http,https
filter = apache-noscript
logpath = /var/log/apache2/*error.log
maxretry = 6
bantime = 3600

[apache-overflows]
enabled = true
port = http,https
filter = apache-overflows
logpath = /var/log/apache2/*error.log
maxretry = 2
bantime = 3600

[ossec]
enabled = true
filter = ossec
logpath = /var/ossec/logs/alerts/alerts.log
maxretry = 1
bantime = 86400
port = all
protocol = all
banaction = iptables-allports

Create OSSEC filter for fail2ban

Create a custom filter that parses OSSEC alert logs to extract IP addresses for blocking.

[INCLUDES]

[Definition]
failregex = .*\[\d+\]\s+\(\d+\).*Src IP:

Configure OSSEC active response integration

Create a custom active response script that interfaces with fail2ban for coordinated blocking.

#!/bin/bash
# fail2ban-ban.sh - OSSEC active response script for fail2ban integration
# Author: Binadit Infrastructure Team

LOGFILE="/var/ossec/logs/active-responses.log"
FAIL2BAN_CLIENT="/usr/bin/fail2ban-client"

# Function to log messages
log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - fail2ban-ban.sh: $1" >> $LOGFILE
}

# Read the action and IP from OSSEC
read ACTION
read USER
read IP
read ALERTID
read RULEID
read AGENT
read FILENAME

# Extract the IP address if it's in CIDR format
IP=$(echo $IP | cut -d'/' -f1)

# Validate IP address
if [[ ! $IP =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
    log "Invalid IP address: $IP"
    exit 1
fi

# Skip private IP addresses
if [[ $IP =~ ^10\. ]] || [[ $IP =~ ^172\.(1[6-9]|2[0-9]|3[0-1])\. ]] || [[ $IP =~ ^192\.168\. ]]; then
    log "Skipping private IP: $IP"
    exit 0
fi

case $ACTION in
    add)
        log "Banning IP $IP (Rule ID: $RULEID)"
        # Ban the IP using fail2ban
        $FAIL2BAN_CLIENT set ossec banip $IP
        if [ $? -eq 0 ]; then
            log "Successfully banned IP $IP"
        else
            log "Failed to ban IP $IP"
        fi
        ;;
    delete)
        log "Unbanning IP $IP (Rule ID: $RULEID)"
        # Unban the IP using fail2ban
        $FAIL2BAN_CLIENT set ossec unbanip $IP
        if [ $? -eq 0 ]; then
            log "Successfully unbanned IP $IP"
        else
            log "Failed to unban IP $IP"
        fi
        ;;
    *)
        log "Unknown action: $ACTION"
        ;;
esac

exit 0

Set permissions for active response script

Set proper permissions for the active response script so OSSEC can execute it.

sudo chmod 750 /var/ossec/active-response/bin/fail2ban-ban.sh
sudo chown root:ossec /var/ossec/active-response/bin/fail2ban-ban.sh
Never use chmod 777. It gives every user on the system full access to your files. Active response scripts should only be executable by root and the OSSEC group for security.

Add active response configuration to OSSEC

Configure OSSEC to use the custom active response script for high-level alerts.

Start and enable services

Enable and start both OSSEC and fail2ban services to begin monitoring.

sudo systemctl enable fail2ban
sudo systemctl start fail2ban
sudo /var/ossec/bin/ossec-control start
sudo systemctl status fail2ban

Configure log monitoring integration

Set up centralized logging to ensure both systems can access and monitor the same log files effectively. This configuration was referenced in our centralized logging tutorial.

# OSSEC log integration
# Send OSSEC alerts to a separate log file
if $programname startswith 'ossec' then /var/log/ossec-alerts.log
& stop

# Ensure fail2ban can read OSSEC logs
$FileCreateMode 0644
$DirCreateMode 0755

Restart rsyslog service

Restart rsyslog to apply the new logging configuration.

sudo systemctl restart rsyslog

Configure automated threat response

Set up email notifications

Configure email alerts for both OSSEC and fail2ban to notify administrators of security events.

[Definition]
actionstart = printf %%b "Subject: [Fail2ban]

Create monitoring dashboard script

Create a script to monitor the status of both systems and generate periodic reports.

#!/bin/bash
# security-monitor.sh - Monitor OSSEC and fail2ban status
# Author: Binadit Infrastructure Team

LOGFILE="/var/log/security-monitor.log"
EMAIL="admin@example.com"

# Function to log messages
log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a $LOGFILE
}

# Check OSSEC status
check_ossec() {
    if pgrep -f "ossec-syscheckd" > /dev/null; then
        log "OSSEC: Running"
        # Get recent alerts count
        RECENT_ALERTS=$(tail -n 100 /var/ossec/logs/alerts/alerts.log | grep "$(date '+%Y %b %d')" | wc -l)
        log "OSSEC: $RECENT_ALERTS alerts today"
    else
        log "OSSEC: Not running - ALERT!"
        echo "OSSEC service is down on $(hostname)" | mail -s "OSSEC Alert" $EMAIL
    fi
}

# Check fail2ban status
check_fail2ban() {
    if systemctl is-active --quiet fail2ban; then
        log "Fail2ban: Running"
        # Get banned IPs count
        BANNED_IPS=$(/usr/bin/fail2ban-client status | grep "Currently banned" | awk '{print $4}')
        log "Fail2ban: $BANNED_IPS IPs currently banned"
        
        # List active jails
        ACTIVE_JAILS=$(/usr/bin/fail2ban-client status | grep "Jail list" | cut -d: -f2 | xargs)
        log "Fail2ban: Active jails: $ACTIVE_JAILS"
    else
        log "Fail2ban: Not running - ALERT!"
        echo "Fail2ban service is down on $(hostname)" | mail -s "Fail2ban Alert" $EMAIL
    fi
}

# Check disk space for logs
check_disk_space() {
    DISK_USAGE=$(df /var/log | tail -1 | awk '{print $5}' | sed 's/%//')
    if [ $DISK_USAGE -gt 80 ]; then
        log "Disk space warning: /var/log is ${DISK_USAGE}% full"
        echo "Log partition is ${DISK_USAGE}% full on $(hostname)" | mail -s "Disk Spa

Automated install script

Run this to automate the entire setup

#OSSEC #fail2ban #intrusion detection #HIDS #automated threat response

Related tutorials

security Advanced

Implement Kubernetes security scanning with Falco and OPA Gatekeeper for runtime protection

Set up comprehensive Kubernetes security with Falco for runtime threat detection and OPA Gatekeeper for admission control policy enforcement. This tutorial covers installation, configuration, and custom security policies.

45 min 4 distros 489 views
security Advanced

Implement OpenResty rate limiting and DDoS protection with advanced Lua rules

Set up comprehensive rate limiting and DDoS protection for OpenResty using nginx directives, Redis-backed Lua middleware, and advanced security rules with monitoring and alerting.

45 min 4 distros 719 views
security Advanced

Configure Redis 7 cluster SSL encryption and authentication for production security

Secure your Redis 7 cluster with TLS encryption, client authentication, and inter-node SSL communication for production environments. Includes certificate generation, authentication setup, and security validation.

45 min 4 distros 701 views
Browse all tutorials in our complete directory →

Don't want to manage this yourself?

We handle infrastructure for businesses that depend on uptime. Fully managed, with one fixed contact who knows your setup.

You get one fixed contact who knows your setup

Rotterdam 01:28 · reachable in a message, no ticket form

+31 10 477 5362 WhatsApp