Tutorials security

Configure OSSEC HIDS for intrusion detection with email alerts and real-time monitoring

Intermediate 45 min Apr 03, 2026 35 views
Ubuntu 24.04 Ubuntu 22.04 Debian 12 AlmaLinux 9 Rocky Linux 9 Fedora 41

Set up OSSEC Host-based Intrusion Detection System with server-agent architecture, custom detection rules, email notifications, and real-time monitoring for comprehensive security monitoring across multiple systems.

Prerequisites

  • Root or sudo access
  • Minimum 2GB RAM
  • SMTP server access for email alerts
  • Multiple systems for agent deployment
  • Basic knowledge of Linux system administration

What this solves

OSSEC HIDS provides comprehensive host-based intrusion detection by monitoring system logs, file integrity, registry changes, and rootkit detection. This tutorial sets up a centralized OSSEC server with distributed agents, email alerting, and custom detection rules for enterprise security monitoring.

Step-by-step installation

Update system packages

Update your package manager to ensure you have the latest security patches and dependencies.

sudo apt update && sudo apt upgrade -y
sudo dnf update -y

Install required dependencies

Install build tools and development libraries needed for OSSEC compilation.

sudo apt install -y build-essential gcc make libevent-dev zlib1g-dev libssl-dev libpcre2-dev wget curl postfix mailutils
sudo dnf install -y gcc gcc-c++ make libevent-devel zlib-devel openssl-devel pcre2-devel wget curl postfix mailx

Create OSSEC system user

Create dedicated system users for OSSEC processes with minimal privileges.

sudo groupadd ossec
sudo useradd -d /var/ossec -s /bin/false -g ossec ossec
sudo useradd -d /var/ossec -s /bin/false -g ossec ossecm
sudo useradd -d /var/ossec -s /bin/false -g ossec ossecr

Download and extract OSSEC

Download the latest OSSEC release and verify the installation package.

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar -xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0

Install OSSEC server

Run the interactive installation script for the OSSEC server configuration.

sudo ./install.sh

During installation, select these options:

  • What kind of installation: server
  • Installation path: /var/ossec
  • Email notification: y
  • SMTP server: 127.0.0.1
  • Email from: ossec@example.com
  • Email to: admin@example.com
  • Enable firewall response: y
  • Enable PCI DSS: n

Configure OSSEC server settings

Edit the main OSSEC configuration file to customize monitoring settings and email alerts.

<ossec_config>
  <global>
    <email_notification>yes</email_notification>
    <smtp_server>127.0.0.1</smtp_server>
    <email_from>ossec@example.com</email_from>
    <email_to>admin@example.com</email_to>
    <email_maxperhour>12</email_maxperhour>
    <email_log_source>alerts.log</email_log_source>
    <agents_disconnection_time>600</agents_disconnection_time>
    <agents_disconnection_alert_time>1800</agents_disconnection_alert_time>
  </global>

  <rules>
    <include>rules_config.xml</include>
    <include>pam_rules.xml</include>
    <include>sshd_rules.xml</include>
    <include>telnetd_rules.xml</include>
    <include>syslog_rules.xml</include>
    <include>arpwatch_rules.xml</include>
    <include>symantec-av_rules.xml</include>
    <include>symantec-ws_rules.xml</include>
    <include>pix_rules.xml</include>
    <include>named_rules.xml</include>
    <include>smbd_rules.xml</include>
    <include>vsftpd_rules.xml</include>
    <include>pure-ftpd_rules.xml</include>
    <include>proftpd_rules.xml</include>
    <include>ms_ftpd_rules.xml</include>
    <include>ftpd_rules.xml</include>
    <include>hordeimp_rules.xml</include>
    <include>roundcube_rules.xml</include>
    <include>wordpress_rules.xml</include>
    <include>cimserver_rules.xml</include>
    <include>vpopmail_rules.xml</include>
    <include>vmpop3d_rules.xml</include>
    <include>courier_rules.xml</include>
    <include>web_rules.xml</include>
    <include>web_appsec_rules.xml</include>
    <include>apache_rules.xml</include>
    <include>nginx_rules.xml</include>
    <include>php_rules.xml</include>
    <include>mysql_rules.xml</include>
    <include>postgresql_rules.xml</include>
    <include>ids_rules.xml</include>
    <include>squid_rules.xml</include>
    <include>firewall_rules.xml</include>
    <include>cisco-ios_rules.xml</include>
    <include>netscreenfw_rules.xml</include>
    <include>sonicwall_rules.xml</include>
    <include>postfix_rules.xml</include>
    <include>sendmail_rules.xml</include>
    <include>imapd_rules.xml</include>
    <include>mailscanner_rules.xml</include>
    <include>dovecot_rules.xml</include>
    <include>ms-exchange_rules.xml</include>
    <include>racoon_rules.xml</include>
    <include>vpn_concentrator_rules.xml</include>
    <include>spamd_rules.xml</include>
    <include>msauth_rules.xml</include>
    <include>mcafee_av_rules.xml</include>
    <include>trend-osce_rules.xml</include>
    <include>ms-se_rules.xml</include>
    <include>zeus_rules.xml</include>
    <include>solaris_bsm_rules.xml</include>
    <include>vmware_rules.xml</include>
    <include>ms_dhcp_rules.xml</include>
    <include>asterisk_rules.xml</include>
    <include>ossec_rules.xml</include>
    <include>attack_rules.xml</include>
    <include>local_rules.xml</include>
  </rules>

  <syscheck>
    <frequency>79200</frequency>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin,/boot</directories>
    <directories check_all="yes" realtime="yes">/var/www</directories>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
    <ignore>/etc/random-seed</ignore>
    <ignore>/etc/random.seed</ignore>
    <ignore>/etc/adjtime</ignore>
    <ignore>/etc/httpd/logs</ignore>
    <ignore>/etc/utmpx</ignore>
    <ignore>/etc/wtmpx</ignore>
    <ignore>/etc/cups/certs</ignore>
    <ignore>/etc/dumpdates</ignore>
    <ignore>/etc/svc/volatile</ignore>
    <ignore type="sregex">\.log$|logs/</ignore>
  </syscheck>

  <rootcheck>
    <disabled>no</disabled>
    <check_files>yes</check_files>
    <check_trojans>yes</check_trojans>
    <check_dev>yes</check_dev>
    <check_sys>yes</check_sys>
    <check_pids>yes</check_pids>
    <check_ports>yes</check_ports>
    <check_if>yes</check_if>
    <frequency>36000</frequency>
    <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
    <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
    <system_audit>/var/ossec/etc/shared/system_audit_rcl.txt</system_audit>
    <system_audit>/var/ossec/etc/shared/cis_debian_linux_rcl.txt</system_audit>
    <system_audit>/var/ossec/etc/shared/cis_rhel_linux_rcl.txt</system_audit>
    <system_audit>/var/ossec/etc/shared/cis_rhel5_linux_rcl.txt</system_audit>
  </rootcheck>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/messages</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>command</log_format>
    <command>df -P</command>
    <frequency>360</frequency>
  </localfile>

  <localfile>
    <log_format>full_command</log_format>
    <command>netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+\([[:digit:]]\+\)\ \+\([[:digit:]]\+\)\ \+\(.\):\([[:digit:]]\)\ \+\(.\):\([[:digit:]]\)\ \+\([[:alpha:]]\+\)\(.*\)/\1 \2 \3 \4:\5 \6:\7 \8 \9/' | sort -k 4 -g</command>
    <alias>netstat listening ports</alias>
    <frequency>360</frequency>
  </localfile>

  <localfile>
    <log_format>full_command</log_format>
    <command>last -n 20</command>
    <frequency>360</frequency>
  </localfile>

  <remote>
    <connection>secure</connection>
    <port>1514</port>
    <protocol>udp</protocol>
    <allowed-ips>0.0.0.0/0</allowed-ips>
  </remote>

  <alerts>
    <log_alert_level>1</log_alert_level>
    <email_alert_level>7</email_alert_level>
  </alerts>
</ossec_config>

Configure email alerts

Set up Postfix for email notifications and configure OSSEC email rules.

sudo dpkg-reconfigure postfix

Select Internet Site and configure with your domain. Then create email alert rules:

<?xml version="1.0" encoding="UTF-8"?>
<group name="local,syslog,">
  <rule id="100001" level="5">
    <if_sid>5715</if_sid>
    <srcip>!192.168.0.0/16</srcip>
    <description>sshd: authentication success from external IP.</description>
    <group>authentication_success,pci_dss_10.2.5,</group>
  </rule>

  <rule id="100002" level="10" frequency="6" timeframe="120">
    <if_matched_sid>5716</if_matched_sid>
    <description>sshd: brute force attack (multiple failed logins).</description>
    <group>authentication_failures,pci_dss_11.4,pci_dss_10.2.4,pci_dss_10.2.5,</group>
  </rule>

  <rule id="100003" level="12">
    <if_sid>550</if_sid>
    <match>ossec: File size reduced</match>
    <description>File size reduced. Possible data integrity violation.</description>
    <group>pci_dss_11.5,file_integrity,</group>
  </rule>

  <rule id="100004" level="7">
    <if_sid>1002</if_sid>
    <match>session opened for user root</match>
    <description>Root user login detected.</description>
    <group>authentication_success,pci_dss_10.2.5,</group>
  </rule>

  <rule id="100005" level="10" frequency="5" timeframe="300">
    <if_sid>31151</if_sid>
    <description>Multiple web server 4xx errors.</description>
    <group>web,attack,</group>
  </rule>
</group>

Configure firewall rules

Allow OSSEC agent connections and email traffic through the firewall.

sudo ufw allow 1514/udp comment "OSSEC Agent Communication"
sudo ufw allow 25/tcp comment "SMTP Email Alerts"
sudo ufw reload
sudo firewall-cmd --permanent --add-port=1514/udp --zone=public
sudo firewall-cmd --permanent --add-service=smtp --zone=public
sudo firewall-cmd --reload

Set proper permissions and ownership

Configure file permissions for OSSEC directories and files with proper security.

sudo chown -R ossec:ossec /var/ossec
sudo chmod -R 755 /var/ossec
sudo chmod 750 /var/ossec/etc
sudo chmod 640 /var/ossec/etc/ossec.conf
sudo chmod 750 /var/ossec/logs
sudo chmod 750 /var/ossec/queue
Never use chmod 777. It gives every user on the system full access to your files. Instead, fix ownership with chown and use minimal permissions.

Start OSSEC server

Enable and start the OSSEC server service with systemd integration.

sudo /var/ossec/bin/ossec-control start
sudo systemctl enable ossec

Install OSSEC agent on remote systems

Download and install OSSEC agent on systems you want to monitor.

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/3.7.0.tar.gz
tar -xzf 3.7.0.tar.gz
cd ossec-hids-3.7.0
sudo ./install.sh

During agent installation, select:

  • Installation type: agent
  • Server IP: 203.0.113.10 (your OSSEC server IP)
  • Agent ID: 001 (unique for each agent)
  • Agent name: web-server-01

Add agents to OSSEC server

Register agents on the OSSEC server and generate authentication keys.

sudo /var/ossec/bin/manage_agents

In the management interface:

  • Press A to add agent
  • Agent ID: 001
  • Agent name: web-server-01
  • Agent IP: 203.0.113.20
  • Press E to extract key for agent 001

Copy the generated key to import on the agent system:

sudo /var/ossec/bin/manage_agents

Press I to import key and paste the authentication key from server.

Configure agent monitoring

Customize agent configuration for specific log files and monitoring targets.

<ossec_config>
  <client>
    <server>
      <address>203.0.113.10</address>
      <port>1514</port>
      <protocol>udp</protocol>
    </server>
    <config-profile>ubuntu, ubuntu22, ubuntu22.04</config-profile>
    <notify_time>10</notify_time>
    <time-reconnect>60</time-reconnect>
    <auto_restart>yes</auto_restart>
    <crypto_method>aes</crypto_method>
  </client>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/auth.log</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/syslog</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/apache2/access.log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/apache2/error.log</location>
  </localfile>

  <localfile>
    <log_format>nginx</log_format>
    <location>/var/log/nginx/access.log</location>
  </localfile>

  <localfile>
    <log_format>nginx</log_format>
    <location>/var/log/nginx/error.log</location>
  </localfile>

  <syscheck>
    <frequency>79200</frequency>
    <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
    <directories check_all="yes">/bin,/sbin</directories>
    <directories check_all="yes" realtime="yes">/var/www</directories>
    <directories check_all="yes" realtime="yes">/etc/nginx</directories>
    <directories check_all="yes" realtime="yes">/etc/apache2</directories>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
    <ignore>/etc/random-seed</ignore>
    <ignore>/etc/adjtime</ignore>
    <ignore type="sregex">\.log$|logs/</ignore>
  </syscheck>

  <rootcheck>
    <disabled>no</disabled>
    <check_files>yes</check_files>
    <check_trojans>yes</check_trojans>
    <check_dev>yes</check_dev>
    <check_sys>yes</check_sys>
    <check_pids>yes</check_pids>
    <check_ports>yes</check_ports>
    <check_if>yes</check_if>
    <frequency>36000</frequency>
  </rootcheck>
</ossec_config>

Start OSSEC agents

Enable and start OSSEC agents on monitored systems.

sudo /var/ossec/bin/ossec-control start
sudo systemctl enable ossec

Configure log rotation

Set up automatic log rotation to prevent disk space issues with OSSEC logs.

/var/ossec/logs/alerts/alerts.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 640 ossec ossec
    postrotate
        /var/ossec/bin/ossec-control restart > /dev/null 2>/dev/null || true
    endscript
}

/var/ossec/logs/ossec.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 640 ossec ossec
    postrotate
        /var/ossec/bin/ossec-control restart > /dev/null 2>/dev/null || true
    endscript
}

/var/ossec/logs/archives/archives.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 640 ossec ossec
    postrotate
        /var/ossec/bin/ossec-control restart > /dev/null 2>/dev/null || true
    endscript
}

Configure real-time monitoring dashboard

Install and configure OSSEC web UI for real-time monitoring and alert management.

sudo apt install -y apache2 php php-cli php-common libapache2-mod-php
cd /tmp
wget https://github.com/ossec/ossec-wui/archive/0.9.tar.gz
tar -xzf 0.9.tar.gz
sudo cp -r ossec-wui-0.9/* /var/www/html/ossec/
sudo chown -R www-data:www-data /var/www/html/ossec
sudo chmod 755 /var/www/html/ossec
sudo dnf install -y httpd php php-cli php-common
cd /tmp
wget https://github.com/ossec/ossec-wui/archive/0.9.tar.gz
tar -xzf 0.9.tar.gz
sudo cp -r ossec-wui-0.9/* /var/www/html/ossec/
sudo chown -R apache:apache /var/www/html/ossec
sudo chmod 755 /var/www/html/ossec
sudo systemctl enable --now httpd

Configure web UI permissions

Set up proper permissions for OSSEC web interface to read logs and statistics.

sudo usermod -a -G ossec www-data
sudo chmod g+r /var/ossec/logs/alerts/alerts.log
sudo chmod g+r /var/ossec/logs/ossec.log
sudo chmod g+rx /var/ossec/stats
sudo chmod g+r /var/ossec/stats/

Create Apache virtual host for OSSEC

Configure Apache virtual host with SSL and basic authentication for secure access.

<VirtualHost :80>
    ServerName ossec.example.com
    DocumentRoot /var/www/html/ossec
    
    <Directory /var/www/html/ossec>
        Options -Indexes
        AllowOverride All
        Require all granted
        AuthType Basic
        AuthName "OSSEC Dashboard"
        AuthUserFile /etc/apache2/.htpasswd
        Require valid-user
    </Directory>
    
    ErrorLog ${APACHE_LOG_DIR}/ossec_error.log
    CustomLog ${APACHE_LOG_DIR}/ossec_access.log combined
</VirtualHost>
sudo htpasswd -c /etc/apache2/.htpasswd admin
sudo a2ensite ossec.conf
sudo systemctl reload apache2

Performance tuning configuration

Optimize OSSEC performance for high-volume environments and resource efficiency.

# Analysis options
analysisd.event_threads=4
analysisd.syscheck_threads=2
analysisd.syscollector_threads=1
analysisd.rootcheck_threads=1
analysisd.sca_threads=2
analysisd.hostinfo_threads=1
analysisd.winevt_threads=1
analysisd.other_threads=1
analysisd.min_rotate_interval=600


Remote daemon options

remoted.recv_counter_flush=128
remoted.comp_average_printout=19999
remoted.verify_msg_id=0
remoted.pass_empty_keyfile=1


Log collector options

logcollector.loop_timeout=2
logcollector.open_attempts=8
logcollector.vcheck_files=64
logcollector.max_lines=10000
logcollector.max_files=1000


Syscheck options

syscheck.sleep=2
syscheck.sleep_after=50
syscheck.rt_delay=30


Rootcheck options

rootcheck.sleep=2


Database options

dbd.reconnect_attempts=10

Verify your setup

Check OSSEC server status and verify agent connections are working properly.

sudo /var/ossec/bin/ossec-control status
sudo /var/ossec/bin/list_agents -c
sudo tail -f /var/ossec/logs/ossec.log

Test email alerts by generating a test event:

sudo /var/ossec/bin/ossec-logtest
sudo logger "OSSEC test alert - unauthorized access attempt"
sudo tail -f /var/ossec/logs/alerts/alerts.log

Verify web dashboard access:

curl -u admin:password http://localhost/ossec/
sudo systemctl status apache2

Common issues

SymptomCauseFix
Agent not connectingFirewall blocking port 1514sudo ufw allow 1514/udp or check authentication keys
Email alerts not sendingPostfix not configuredsudo dpkg-reconfigure postfix and check SMTP settings
High CPU usageToo many log files monitoredReduce monitored files in ossec.conf or increase analysis threads
Web UI permission deniedIncorrect file ownershipsudo chown -R www-data:www-data /var/www/html/ossec
Disk space filling upLog rotation not configuredConfigure logrotate rules and check /var/ossec/logs size
False positive alertsDefault rules too sensitiveTune alert levels in ossec.conf or create custom rules

Next steps

Automated install script

Run this to automate the entire setup

#ossec #hids #intrusion-detection #security-monitoring #log-analysis #real-time-alerts #ossec-server #ossec-agent

Related tutorials

security Beginner

Configure Linux file encryption with LUKS and cryptsetup for data protection

Learn how to encrypt files and partitions using LUKS encryption with cryptsetup tools. This tutorial covers creating encrypted volumes, managing encryption keys, and automating mount operations for secure data protection.

45 min 6 distros 4 views
security Beginner

Configure Linux audit system with auditd for security compliance and file monitoring

Set up the Linux audit system with auditd to monitor file access, user activities, and system calls for security compliance. Learn to create audit rules, analyze logs, and track privilege escalations.

25 min 6 distros 20 views
security Beginner

Configure Linux file permissions and access control with umask and chmod best practices

Learn how to configure secure Linux file permissions using chmod, set default permissions with umask, and implement proper access control to prevent security vulnerabilities while fixing permission denied errors.

25 min 6 distros 18 views

Need help?

Don't want to manage this yourself?

We handle infrastructure for businesses that depend on uptime. From initial setup to ongoing operations.

Talk to an engineer