Implement OpenLiteSpeed WAF and DDoS protection with ModSecurity 3 and rate limiting

Update system packages

Start by updating your package manager to ensure you have the latest security patches and package information.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install OpenLiteSpeed and dependencies

Install OpenLiteSpeed web server along with the development tools and libraries needed for ModSecurity compilation.

wget -O - https://repo.litespeed.sh | sudo bash
sudo apt install -y openlitespeed lsphp82 lsphp82-mysql
sudo apt install -y build-essential libpcre3-dev zlib1g-dev libssl-dev libxml2-dev libpcre++-dev libgd-dev

sudo rpm -Uvh http://rpms.litespeedtech.com/centos/litespeed-repo-1.3-1.el8.noarch.rpm
sudo dnf install -y openlitespeed lsphp82 lsphp82-mysql
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y pcre-devel openssl-devel libxml2-devel gd-devel

Install ModSecurity 3 library

Download and compile ModSecurity 3 from source to get the latest security features and performance improvements.

cd /tmp
git clone https://github.com/SpiderLabs/ModSecurity.git
cd ModSecurity
git checkout v3.0.12
git submodule init
git submodule update

Configure and compile ModSecurity

Build ModSecurity with optimizations enabled and install it to the system directories.

./build.sh
./configure --enable-pcre-study --enable-pcre-jit
make -j$(nproc)
sudo make install
sudo ldconfig

Install ModSecurity connector for OpenLiteSpeed

Clone and build the ModSecurity connector that integrates with OpenLiteSpeed's module system.

cd /tmp
git clone https://github.com/SpiderLabs/ModSecurity-nginx.git
cd ModSecurity-nginx
git checkout v1.0.3

Download OWASP Core Rule Set

Install the OWASP Core Rule Set (CRS) which provides comprehensive protection against common web application attacks.

sudo mkdir -p /usr/local/modsecurity
cd /usr/local/modsecurity
sudo git clone https://github.com/coreruleset/coreruleset.git
sudo mv coreruleset/crs-setup.conf.example coreruleset/crs-setup.conf
sudo mv coreruleset/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf.example coreruleset/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
sudo mv coreruleset/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf.example coreruleset/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf

Create ModSecurity main configuration

Configure ModSecurity with security rules, logging, and performance settings tailored for production use.

# ModSecurity Configuration
SecRuleEngine On
SecRequestBodyAccess On
SecResponseBodyAccess Off
SecRequestBodyLimit 13107200
SecRequestBodyNoFilesLimit 131072
SecRequestBodyLimitAction Reject
SecPcreMatchLimit 250000
SecPcreMatchLimitRecursion 250000

# Audit Log Configuration
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/openlitespeed/modsec_audit.log

# Debug Log
SecDebugLog /var/log/openlitespeed/modsec_debug.log
SecDebugLogLevel 3

# Temporary file configuration
SecTmpDir /tmp/
SecDataDir /tmp/

# Content-Type header configuration
SecRule REQUEST_HEADERS:Content-Type "^(?:application(?:/soap\+|/)|text/)xml" \
     "id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"

SecRule REQUEST_HEADERS:Content-Type "^application/json" \
     "id:'200001',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"

SecRule REQUEST_HEADERS:Content-Type "^application/x-www-form-urlencoded" \
     "id:'200002',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=URLENCODED"

SecRule REQUEST_HEADERS:Content-Type "^multipart/form-data" \
     "id:'200003',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=MULTIPART"

# Include OWASP Core Rule Set
Include /usr/local/modsecurity/coreruleset/crs-setup.conf
Include /usr/local/modsecurity/coreruleset/rules/*.conf

Configure OpenLiteSpeed virtual host security

Create a secure virtual host configuration with ModSecurity integration and rate limiting rules.

docRoot                   /usr/local/lsws/Example/html/
vhDomain                  example.com
allowSymbolLink           1
enableScript              1
restrained                0
setUIDMode                0

# Security Headers
extraHeaders {
  X-Frame-Options SAMEORIGIN
  X-Content-Type-Options nosniff
  X-XSS-Protection "1; mode=block"
  Referrer-Policy strict-origin-when-cross-origin
  Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline';"
}

# Rate Limiting Configuration
accessControl {
  allow                   *
}

# Request Rate Limiting
requestLimitCtrl {
  staticReqPerSec         20
  dynReqPerSec            10
  outBandwidth            1024
  inBandwidth             1024
}

# Connection Limits
connectionCtrl {
  staticReqPerSec         50
  dynReqPerSec            25
  reqRate                 0
  reqUnit                 1
  outBandwidth            0
  inBandwidth             0
}

# ModSecurity Configuration
moduleEnabled {
  cache                   0
  esi                     0
  expires                 1
  headers                 1
  uploadprogress          0
  modsecurity             1
}

moduleSecurityConf {
  modsec_rules            /usr/local/modsecurity/modsecurity.conf
  modsec_rules_remote     0
  modsec_rules_timeout    5
}

scripthandler {
  add                     lsphp82 php
}

phpIniOverride  {
  php_value upload_max_filesize 32M
  php_value post_max_size 32M
  php_value max_execution_time 300
  php_value max_input_vars 3000
}

Configure advanced rate limiting rules

Create comprehensive rate limiting and DDoS protection rules that block aggressive behavior while allowing legitimate traffic.

# Global Rate Limiting Configuration
requestLimitCtrl {
  staticReqPerSec         100
  dynReqPerSec            50
  outBandwidth            10240
  inBandwidth             10240
}

# Connection Control
connectionCtrl {
  maxConnections          2000
  maxSSLConnections       1000
  connTimeout             300
  maxKeepAliveReq         1000
  keepAliveTimeout        5
  sndBufSize              0
  rcvBufSize              0
}

# Anti-DDoS Configuration
perClientConnLimit {
  staticReqPerSec         20
  dynReqPerSec            10
  outBandwidth            1024
  inBandwidth             1024
  softLimit               500
  hardLimit               600
  gracePeriod             15
  banPeriod               300
}

# IP-based Throttling
accounting {
  staticReqPerSec         50
  dynReqPerSec            25
  outBandwidth            5120
  inBandwidth             2048
  reqRate                 20
  reqUnit                 1
  respBuffer              0
}

# Request Filtering
requestFilter {
  enableCtxControl        1
  ctxTimeout              60
  reqTimeOut              300
  respTimeOut             300
}

Create custom security rules

Add application-specific security rules that complement the OWASP Core Rule Set with custom threat detection.

# Custom ModSecurity Rules for Enhanced Protection

# Block common attack patterns
SecRule REQUEST_URI "@detectSQLi" \
    "id:100001,\
    phase:2,\
    block,\
    t:urlDecodeUni,\
    t:htmlEntityDecode,\
    t:normalisePathWin,\
    msg:'SQL Injection Attack Detected',\
    logdata:'Matched Data: %{MATCHED_VAR} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'attack-sqli',\
    tag:'paranoia-level/1',\
    ver:'OWASP_CRS/3.3.5',\
    severity:'CRITICAL',\
    setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

# Rate limiting for specific endpoints
SecRule REQUEST_URI "@beginsWith /wp-login.php" \
    "id:100002,\
    phase:2,\
    pass,\
    setvar:'ip.login_attempts=+1',\
    expirevar:'ip.login_attempts=300',\
    nolog"

SecRule IP:LOGIN_ATTEMPTS "@gt 5" \
    "id:100003,\
    phase:2,\
    deny,\
    status:429,\
    msg:'Rate limit exceeded for login attempts',\
    logdata:'IP: %{REMOTE_ADDR} exceeded login rate limit',\
    setvar:'ip.login_block=1',\
    expirevar:'ip.login_block=1800'"

# Block known malicious user agents
SecRule REQUEST_HEADERS:User-Agent "@pmFromFile /usr/local/modsecurity/malicious-ua.txt" \
    "id:100004,\
    phase:1,\
    block,\
    msg:'Malicious User Agent Detected',\
    logdata:'User-Agent: %{MATCHED_VAR}',\
    tag:'attack-reputation-scanner',\
    severity:'WARNING'"

# Geo-blocking (example for specific countries)
SecRule REMOTE_ADDR "@geoLookup" \
    "id:100005,\
    phase:1,\
    pass,\
    nolog,\
    setvar:'tx.country_code=%{geo.country_code}'"

SecRule TX:COUNTRY_CODE "@in CN RU" \
    "id:100006,\
    phase:1,\
    deny,\
    status:403,\
    msg:'Access denied from restricted country',\
    logdata:'Country: %{tx.country_code}, IP: %{REMOTE_ADDR}'"

# File upload restrictions
SecRule FILES_TMPNAMES "@inspectFile /usr/local/modsecurity/util/av.sh" \
    "id:100007,\
    phase:2,\
    block,\
    msg:'Malicious file upload detected',\
    logdata:'File: %{MATCHED_VAR}'"

# Slow HTTP attack protection
SecRule &REQUEST_HEADERS:Content-Length "@eq 0" \
    "id:100008,\
    phase:1,\
    pass,\
    nolog,\
    setvar:'ip.slow_dos_counter=+1',\
    expirevar:'ip.slow_dos_counter=60'"

SecRule IP:SLOW_DOS_COUNTER "@gt 30" \
    "id:100009,\
    phase:1,\
    deny,\
    status:429,\
    msg:'Slow DoS attack detected',\
    setvar:'ip.slow_dos_block=1',\
    expirevar:'ip.slow_dos_block=600'"

Create malicious user agent blocklist

Maintain a list of known malicious user agents and scanning tools to block automated attacks.

Nikto
Sqlmap
WPScan
DirBuster
ZmEu
LWP::Simple
BBBike
wget
curl/7.0
libwww-perl
python-requests
Masscan
Zmap
Nmap Scripting Engine
OpenVAS
w3af
acunetix
havij
fimap
commix
injectx
vega
burpsuite
owasp zap
Apache-HttpClient

Configure security event monitoring

Set up comprehensive logging and create monitoring scripts to track security events and potential threats.

sudo mkdir -p /var/log/openlitespeed
sudo touch /var/log/openlitespeed/modsec_audit.log
sudo touch /var/log/openlitespeed/modsec_debug.log
sudo touch /var/log/openlitespeed/security_events.log
sudo chown lsadm:lsadm /var/log/openlitespeed/modsec_*.log
sudo chmod 644 /var/log/openlitespeed/modsec_*.log

Create security monitoring script

Build an automated monitoring system that parses security logs and sends alerts when threats are detected.

#!/bin/bash

# ModSecurity Log Monitor Script
LOG_FILE="/var/log/openlitespeed/modsec_audit.log"
ALERT_EMAIL="admin@example.com"
TMP_FILE="/tmp/modsec_alerts.tmp"
LAST_CHECK="/var/lib/modsecurity/last_check"

# Create state directory
sudo mkdir -p /var/lib/modsecurity
sudo chown lsadm:lsadm /var/lib/modsecurity

# Function to send alert
send_alert() {
    local subject="$1"
    local message="$2"
    echo "$message" | mail -s "$subject" "$ALERT_EMAIL"
    logger "ModSecurity Alert: $subject"
}

# Check for new security events
if [ -f "$LAST_CHECK" ]; then
    LAST_TIME=$(cat "$LAST_CHECK")
else
    LAST_TIME="1 hour ago"
fi

# Find new critical events
find "$LOG_FILE" -newer "$LAST_CHECK" 2>/dev/null | while read -r line; do
    if echo "$line" | grep -q "severity.*CRITICAL\|severity.*ERROR"; then
        ATTACK_TYPE=$(echo "$line" | grep -oP 'msg:"\K[^"]*')
        SOURCE_IP=$(echo "$line" | grep -oP 'client: \K[^,]*')
        TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
        
        echo "[$TIMESTAMP] Critical security event detected" >> "$TMP_FILE"
        echo "Attack Type: $ATTACK_TYPE" >> "$TMP_FILE"
        echo "Source IP: $SOURCE_IP" >> "$TMP_FILE"
        echo "Details: $line" >> "$TMP_FILE"
        echo "---" >> "$TMP_FILE"
    fi
done

# Send alerts if events found
if [ -s "$TMP_FILE" ]; then
    send_alert "ModSecurity Critical Events Detected" "$(cat $TMP_FILE)"
    rm -f "$TMP_FILE"
fi

# Update last check timestamp
date > "$LAST_CHECK"

# Generate daily statistics
if [ "$(date +%H:%M)" = "09:00" ]; then
    YESTERDAY=$(date -d "yesterday" +%Y-%m-%d)
    STATS_FILE="/tmp/modsec_daily_stats.tmp"
    
    echo "ModSecurity Daily Report - $YESTERDAY" > "$STATS_FILE"
    echo "=====================================" >> "$STATS_FILE"
    echo "" >> "$STATS_FILE"
    
    # Count attacks by type
    echo "Top Attack Types:" >> "$STATS_FILE"
    grep "$YESTERDAY" "$LOG_FILE" | grep -oP 'attack-\K[^,]*' | sort | uniq -c | sort -nr | head -10 >> "$STATS_FILE"
    echo "" >> "$STATS_FILE"
    
    # Top source IPs
    echo "Top Source IPs:" >> "$STATS_FILE"
    grep "$YESTERDAY" "$LOG_FILE" | grep -oP 'client: \K[^,]*' | sort | uniq -c | sort -nr | head -10 >> "$STATS_FILE"
    echo "" >> "$STATS_FILE"
    
    # Blocked requests count
    BLOCKED_COUNT=$(grep -c "$YESTERDAY.*block" "$LOG_FILE")
    echo "Total Blocked Requests: $BLOCKED_COUNT" >> "$STATS_FILE"
    
    send_alert "ModSecurity Daily Statistics" "$(cat $STATS_FILE)"
    rm -f "$STATS_FILE"
fi

Make monitoring script executable and schedule

Set proper permissions and create a cron job to run the security monitoring script regularly.

sudo chmod +x /usr/local/bin/modsec-monitor.sh
sudo chown lsadm:lsadm /usr/local/bin/modsec-monitor.sh

# Add to crontab for regular monitoring
(crontab -l 2>/dev/null; echo "*/10 * * * * /usr/local/bin/modsec-monitor.sh >/dev/null 2>&1") | crontab -

Configure log rotation

Set up log rotation to prevent security logs from consuming excessive disk space while maintaining historical data.

/var/log/openlitespeed/modsec_*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 lsadm lsadm
    postrotate
        /usr/local/lsws/bin/lshttpd -k usr1 >/dev/null 2>&1 || true
    endscript
}

Restart OpenLiteSpeed with security configuration

Apply all security configurations by restarting OpenLiteSpeed and enabling the new WAF and rate limiting rules.

sudo systemctl restart lsws
sudo systemctl enable lsws
sudo systemctl status lsws

Install and configure mail system

Set up email notifications for security alerts using a lightweight mail transfer agent.

sudo apt install -y postfix mailutils
sudo dpkg-reconfigure postfix

sudo dnf install -y postfix mailx
sudo systemctl enable --now postfix

Create Slack webhook integration

Add Slack notifications for real-time security alerts to your team communication channel.

#!/bin/bash

# Slack notification script for ModSecurity alerts
SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
CHANNEL="#security-alerts"
USERNAME="ModSecurity-Bot"

send_slack_alert() {
    local message="$1"
    local color="$2"
    
    payload=$(cat <

Configure fail2ban integration

Integrate ModSecurity with fail2ban for automatic IP blocking based on attack patterns.