Set up intrusion detection with OSSEC HIDS and nftables integration for automated threat response

Advanced 45 min Apr 09, 2026 967 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Configure OSSEC Host-based Intrusion Detection System with nftables firewall integration for real-time threat detection and automated response. Monitor system activity, detect intrusions, and automatically block malicious IPs using active response mechanisms.

Prerequisites

  • Root or sudo access
  • Basic knowledge of Linux system administration
  • Understanding of firewall concepts
  • Email server configured (postfix)
  • Minimum 2GB RAM and 10GB disk space

What this solves

OSSEC HIDS provides comprehensive host-based intrusion detection by monitoring log files, system integrity, and network activity in real-time. Integrating OSSEC with nftables enables automated threat response by dynamically blocking suspicious IP addresses and implementing firewall rules based on detected security events. This setup creates a proactive security layer that responds to threats without manual intervention.

Step-by-step installation

Update system packages and install dependencies

Start by updating your package manager and installing required dependencies for OSSEC compilation and nftables management.

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential gcc make wget curl unzip
sudo apt install -y libevent-dev libssl-dev libpcre2-dev zlib1g-dev
sudo apt install -y nftables postfix mailutils
sudo dnf update -y
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y wget curl unzip libevent-devel openssl-devel
sudo dnf install -y pcre2-devel zlib-devel nftables postfix mailx

Create OSSEC user and directories

Create a dedicated user for OSSEC operations and set up the required directory structure with proper ownership.

sudo groupadd ossec
sudo useradd -d /var/ossec -s /bin/false -g ossec ossec
sudo useradd -d /var/ossec -s /bin/false -g ossec ossecm
sudo useradd -d /var/ossec -s /bin/false -g ossec ossecr

Download and compile OSSEC HIDS

Download the latest OSSEC source code and compile it with active response support enabled.

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar -xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0
sudo ./install.sh

During installation, select the following options:

  • Installation type: local
  • Email notification: yes
  • Email address: your-admin@example.com
  • SMTP server: 127.0.0.1
  • Run integrity check daemon: yes
  • Run rootkit detection engine: yes
  • Enable active response: yes

Configure OSSEC main settings

Configure the main OSSEC settings including email notifications, log monitoring, and active response thresholds.

<ossec_config>
  <global>
    <email_notification>yes</email_notification>
    <email_to>security@example.com</email_to>
    <smtp_server>127.0.0.1</smtp_server>
    <email_from>ossec@example.com</email_from>
    <email_maxperhour>12</email_maxperhour>
    <email_log_source>alerts.log</email_log_source>
    <agents_disconnection_time>600</agents_disconnection_time>
    <agents_disconnection_alert_time>1800</agents_disconnection_alert_time>
  </global>

  <rules>
    <include>rules_config.xml</include>
    <include>pam_rules.xml</include>
    <include>sshd_rules.xml</include>
    <include>telnetd_rules.xml</include>
    <include>syslog_rules.xml</include>
    <include>arpwatch_rules.xml</include>
    <include>symantec-av_rules.xml</include>
    <include>symantec-ws_rules.xml</include>
    <include>pix_rules.xml</include>
    <include>named_rules.xml</include>
    <include>smbd_rules.xml</include>
    <include>vsftpd_rules.xml</include>
    <include>pure-ftpd_rules.xml</include>
    <include>proftpd_rules.xml</include>
    <include>ms_ftpd_rules.xml</include>
    <include>ftpd_rules.xml</include>
    <include>hordeimp_rules.xml</include>
    <include>roundcube_rules.xml</include>
    <include>wordpress_rules.xml</include>
    <include>cimserver_rules.xml</include>
    <include>vpopmail_rules.xml</include>
    <include>vmpop3d_rules.xml</include>
    <include>courier_rules.xml</include>
    <include>web_rules.xml</include>
    <include>web_appsec_rules.xml</include>
    <include>apache_rules.xml</include>
    <include>nginx_rules.xml</include>
    <include>php_rules.xml</include>
    <include>mysql_rules.xml</include>
    <include>postgresql_rules.xml</include>
    <include>ids_rules.xml</include>
    <include>squid_rules.xml</include>
    <include>firewall_rules.xml</include>
    <include>cisco-ios_rules.xml</include>
    <include>netscreenfw_rules.xml</include>
    <include>sonicwall_rules.xml</include>
    <include>postfix_rules.xml</include>
    <include>sendmail_rules.xml</include>
    <include>imapd_rules.xml</include>
    <include>mailscanner_rules.xml</include>
    <include>dovecot_rules.xml</include>
    <include>ms-exchange_rules.xml</include>
    <include>racoon_rules.xml</include>
    <include>vpn_concentrator_rules.xml</include>
    <include>spamd_rules.xml</include>
    <include>msauth_rules.xml</include>
    <include>mcafee_av_rules.xml</include>
    <include>trend-osce_rules.xml</include>
    <include>ms-se_rules.xml</include>
    <include>zeus_rules.xml</include>
    <include>solaris_bsm_rules.xml</include>
    <include>vmware_rules.xml</include>
    <include>ms_dhcp_rules.xml</include>
    <include>asterisk_rules.xml</include>
    <include>ossec_rules.xml</include>
    <include>attack_rules.xml</include>
    <include>local_rules.xml</include>
  </rules>

  <syscheck>
    <frequency>7200</frequency>
    <alert_new_files>yes</alert_new_files>
    <auto_ignore>no</auto_ignore>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin,/boot</directories>
    <directories check_all="yes" realtime="yes">/var/www</directories>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
    <ignore>/etc/random-seed</ignore>
    <ignore>/etc/random.seed</ignore>
    <ignore>/etc/adjtime</ignore>
    <ignore>/etc/httpd/logs</ignore>
    <ignore>/etc/utmpx</ignore>
    <ignore>/etc/wtmpx</ignore>
    <ignore>/etc/cups/certs</ignore>
    <ignore>/etc/dumpdates</ignore>
    <ignore>/etc/svc/volatile</ignore>
  </syscheck>

  <rootcheck>
    <disabled>no</disabled>
    <check_files>yes</check_files>
    <check_trojans>yes</check_trojans>
    <check_dev>yes</check_dev>
    <check_sys>yes</check_sys>
    <check_pids>yes</check_pids>
    <check_ports>yes</check_ports>
    <check_if>yes</check_if>
    <frequency>7200</frequency>
    <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
    <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
  </rootcheck>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/messages</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/dpkg.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/apache2/error.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/apache2/access.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/nginx/error.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/nginx/access.log</location>
  </localfile>
</ossec_config>

Create nftables active response script

Create a custom active response script that integrates with nftables to automatically block malicious IP addresses.

#!/bin/bash
# OSSEC nftables active response script
# Author: OSSEC Project
# License: GPL

# Getting action and user
ACTION=$1
USER=$2
IP=$3

# Logging
LOG="logger -t ossec-nftables"

# Action function
if [ "x${ACTION}" = "xadd" ]; then
    # Check if IP is already blocked
    if ! nft list set ip filter ossec_blocked_ips 2>/dev/null | grep -q "$IP"; then
        # Add IP to nftables blocked set
        nft add element ip filter ossec_blocked_ips { $IP }
        if [ $? -eq 0 ]; then
            $LOG "OSSEC: Blocked IP $IP via nftables"
            echo "`date` + $IP" >> /var/ossec/logs/active-responses.log
        else
            $LOG "OSSEC: Error blocking IP $IP via nftables"
        fi
    fi
elif [ "x${ACTION}" = "xdelete" ]; then
    # Remove IP from nftables blocked set
    nft delete element ip filter ossec_blocked_ips { $IP } 2>/dev/null
    if [ $? -eq 0 ]; then
        $LOG "OSSEC: Unblocked IP $IP via nftables"
        echo "`date` - $IP" >> /var/ossec/logs/active-responses.log
    else
        $LOG "OSSEC: Error unblocking IP $IP via nftables (may not exist)"
    fi
else
    $LOG "OSSEC: Invalid action $ACTION for nftables-drop script"
fi

exit 0
sudo chmod 750 /var/ossec/active-response/bin/nftables-drop.sh
sudo chown root:ossec /var/ossec/active-response/bin/nftables-drop.sh

Configure nftables with OSSEC integration

Set up nftables rules with a dedicated set for OSSEC-blocked IP addresses and appropriate drop rules.

#!/usr/sbin/nft -f

# Clear existing rules
flush ruleset

# Define variables
define ALLOWED_SSH_IPS = { 203.0.113.0/24, 192.168.1.0/24 }
define WEB_PORTS = { 80, 443 }
define SSH_PORT = 22

# Main table for filtering
table ip filter {
    # Set for OSSEC blocked IPs
    set ossec_blocked_ips {
        type ipv4_addr
        flags dynamic,timeout
        timeout 1h
        gc-interval 5m
    }

    # Set for rate limiting
    set ssh_bruteforce {
        type ipv4_addr . inet_service
        flags dynamic,timeout
        timeout 10m
    }

    # Input chain
    chain input {
        type filter hook input priority filter; policy drop;
        
        # Allow loopback traffic
        iif lo accept
        
        # Drop packets from OSSEC blocked IPs immediately
        ip saddr @ossec_blocked_ips counter drop
        
        # Allow established and related connections
        ct state established,related accept
        
        # Allow ICMP
        icmp type { destination-unreachable, router-solicitation, router-advertisement, time-exceeded, parameter-problem } accept
        icmp type echo-request limit rate 5/second accept
        
        # SSH with rate limiting and IP restriction
        tcp dport $SSH_PORT ip saddr $ALLOWED_SSH_IPS ct state new limit rate 3/minute accept
        
        # SSH brute force protection
        tcp dport $SSH_PORT add @ssh_bruteforce { ip saddr . tcp dport limit rate 3/minute burst 3 packets }
        tcp dport $SSH_PORT ip saddr . tcp dport @ssh_bruteforce counter drop
        
        # Web services
        tcp dport $WEB_PORTS accept
        
        # Drop invalid packets
        ct state invalid counter drop
        
        # Log dropped packets (sample only to avoid log flooding)
        limit rate 10/minute counter log prefix "nftables-drop: "
        
        # Default drop
        counter drop
    }

    # Forward chain
    chain forward {
        type filter hook forward priority filter; policy drop;
        
        # Drop packets from OSSEC blocked IPs
        ip saddr @ossec_blocked_ips counter drop
        ip daddr @ossec_blocked_ips counter drop
    }

    # Output chain
    chain output {
        type filter hook output priority filter; policy accept;
    }
}

Enable and configure nftables service

Enable nftables to start on boot and apply the configuration rules.

sudo systemctl enable nftables
sudo systemctl start nftables
sudo nft list ruleset

Configure OSSEC active response

Add the active response configuration to OSSEC to use the nftables script for blocking malicious IPs.

  <command>
    <name>nftables-drop</name>
    <executable>nftables-drop.sh</executable>
    <expect>srcip</expect>
    <timeout_allowed>yes</timeout_allowed>
  </command>

  <active-response>
    <disabled>no</disabled>
    <command>nftables-drop</command>
    <location>local</location>
    <rules_id>5720,5721,5722,5723,5724,5725,5726</rules_id>
    <timeout>3600</timeout>
  </active-response>

  <active-response>
    <disabled>no</disabled>
    <command>nftables-drop</command>
    <location>local</location>
    <rules_id>40111,40121,40122</rules_id>
    <timeout>1800</timeout>
  </active-response>

  <active-response>
    <disabled>no</disabled>
    <command>nftables-drop</command>
    <location>local</location>
    <level>6</level>
    <timeout>600</timeout>
  </active-response>

Configure custom OSSEC rules

Create custom rules for enhanced threat detection including web application attacks and system intrusions.

<!-- Local rules -->

<group name="local,syslog,sshd,">
  <!-- SSH brute force attempts -->
  <rule id="100001" level="10" frequency="6" timeframe="120">
    <if_matched_sid>5716</if_matched_sid>
    <description>SSH brute force attack detected (6 attempts in 120 seconds).</description>
    <group>authentication_failures,attack,</group>
  </rule>

  <!-- Multiple SSH authentication failures -->
  <rule id="100002" level="8" frequency="4" timeframe="60">
    <if_matched_sid>5716</if_matched_sid>
    <description>Multiple SSH authentication failures from same source.</description>
    <group>authentication_failures,</group>
  </rule>

  <!-- Successful login after brute force -->
  <rule id="100003" level="12">
    <if_matched_sid>5715</if_matched_sid>
    <if_fts></if_fts>
    <options>no_email_alert</options>
    <description>Successful SSH login after brute force attempts.</description>
    <group>authentication_success,attack,</group>
  </rule>
</group>

<group name="local,web,apache,">
  <!-- Web application attacks -->
  <rule id="100010" level="8">
    <if_sid>31100</if_sid>
    <url>/admin|/wp-admin|/administrator|/phpmyadmin|/xmlrpc.php</url>
    <description>Web application admin interface access attempt.</description>
    <group>web_scan,attack,</group>
  </rule>

  <!-- SQL injection attempts -->
  <rule id="100011" level="10">
    <if_sid>31100,31101,31102,31103,31104,31105,31106,31107,31108</if_sid>
    <url>union|select|insert|delete|drop|create|alter|exec|script</url>
    <description>SQL injection attempt detected.</description>
    <group>web_scan,attack,sql_injection,</group>
  </rule>

  <!-- Web shell upload attempts -->
  <rule id="100012" level="12">
    <if_sid>31100,31101,31102,31103,31104,31105,31106,31107,31108</if_sid>
    <url>.php.|.asp.|.jsp.|eval(|system(|exec(|shell_exec</url>
    <description>Web shell upload or execution attempt.</description>
    <group>web_scan,attack,webshell,</group>
  </rule>
</group>

<group name="local,system,">
  <!-- System file modifications -->
  <rule id="100020" level="8">
    <if_sid>550</if_sid>
    <field name="file">^/etc/passwd$|^/etc/shadow$|^/etc/sudoers$</field>
    <description>Critical system file modified.</description>
    <group>syscheck,critical_file,</group>
  </rule>

  <!-- Suspicious process execution -->
  <rule id="100021" level="8">
    <if_sid>2833,2834</if_sid>
    <regex>nc |netcat |telnet |wget |curl |python -c|perl -e|bash -i</regex>
    <description>Suspicious process execution detected.</description>
    <group>process,attack,</group>
  </rule>
</group>

Set proper permissions and ownership

Configure correct ownership and permissions for OSSEC files to ensure secure operation.

Never use chmod 777. It gives every user on the system full access to your files. Instead, use minimal permissions and correct ownership for security.
sudo chown -R root:ossec /var/ossec
sudo chmod -R 750 /var/ossec
sudo chmod 640 /var/ossec/etc/ossec.conf
sudo chmod 750 /var/ossec/bin/*
sudo chmod 755 /var/ossec/logs
sudo chmod 644 /var/ossec/logs/*.log 2>/dev/null || true
sudo touch /var/ossec/logs/active-responses.log
sudo chown ossec:ossec /var/ossec/logs/active-responses.log
sudo chmod 644 /var/ossec/logs/active-responses.log

Start and enable OSSEC service

Start the OSSEC service and enable it to start automatically on boot.

sudo systemctl enable ossec
sudo systemctl start ossec
sudo systemctl status ossec

Configure email notifications

Set up email notifications for OSSEC alerts using postfix for immediate threat notifications.

Automated install script

Run this to automate the entire setup

इसे खुद मैनेज नहीं करना चाहते?

हम उन businesses के लिए infrastructure संभालते हैं जो uptime पर निर्भर हैं। Fully managed, एक fixed contact के साथ जो आपके setup को जानता है।

आपको एक निश्चित contact मिलता है जो आपके setup को जानता है

Rotterdam 05:05 · एक message में पहुंचें, कोई ticket form नहीं