Set up ELK Stack for centralized ModSecurity log analysis and monitoring

Update system packages

Start by updating your package manager to ensure you get the latest versions of all components.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install Java 11

All ELK components require Java. Install OpenJDK 11 which provides the best compatibility with Elasticsearch 8.

sudo apt install -y openjdk-11-jdk

sudo dnf install -y java-11-openjdk java-11-openjdk-devel

Add Elastic repository

Add the official Elastic repository to install Elasticsearch, Logstash, and Kibana from the same source.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
sudo tee /etc/yum.repos.d/elasticsearch.repo << EOF
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF

Install Elasticsearch 8

Install Elasticsearch which will store and index your ModSecurity logs for fast searching and analysis.

sudo apt install -y elasticsearch

sudo dnf install -y --enablerepo=elasticsearch elasticsearch

Configure Elasticsearch

Configure Elasticsearch with appropriate memory settings and network binding for your ModSecurity log analysis setup.

cluster.name: modsecurity-logs
node.name: elk-node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/transport.p12
xpack.security.transport.ssl.truststore.path: certs/transport.p12

Set Elasticsearch heap size

Configure JVM heap size to use half of your available RAM. For ModSecurity logs, 2-4GB is typically sufficient for small to medium deployments.

-Xms2g
-Xmx2g

Start and enable Elasticsearch

Start Elasticsearch and enable it to start on boot. The first startup will generate security certificates and passwords.

sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch
sudo systemctl status elasticsearch

Configure Elasticsearch passwords

Set up the built-in user passwords for secure access. Save the elastic user password as you'll need it for Kibana configuration.

sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

Install Logstash 8

Install Logstash to parse and transform ModSecurity logs before sending them to Elasticsearch.

sudo apt install -y logstash

sudo dnf install -y --enablerepo=elasticsearch logstash

Configure Logstash for ModSecurity logs

Create a Logstash configuration that parses ModSecurity audit logs and extracts security events, attack patterns, and client information.

input {
  file {
    path => "/var/log/modsec_audit.log"
    start_position => "beginning"
    type => "modsecurity"
    codec => multiline {
      pattern => "^--[a-fA-F0-9]{8}-A--$"
      negate => true
      what => "previous"
    }
  }
}

filter {
  if [type] == "modsecurity" {
    grok {
      match => { 
        "message" => "(?--[a-fA-F0-9]{8}-A--\s\[%{HTTPDATE:timestamp}\]\s%{DATA:unique_id}\s%{IPORHOST:client_ip}\s%{INT:client_port}\s%{IPORHOST:server_ip}\s%{INT:server_port})" 
      }
    }
    
    grok {
      match => { 
        "message" => "--[a-fA-F0-9]{8}-B--\s(?[\s\S]?)(?=--[a-fA-F0-9]{8}-[C-Z]--|$)" 
      }
    }
    
    grok {
      match => { 
        "request_headers" => "%{WORD:http_method}\s+(?\S+)\s+HTTP/%{NUMBER:http_version}" 
      }
    }
    
    grok {
      match => { 
        "message" => "--[a-fA-F0-9]{8}-F--\s(?[\s\S]?)(?=--[a-fA-F0-9]{8}-[G-Z]--|$)" 
      }
    }
    
    grok {
      match => { 
        "response_headers" => "HTTP/%{NUMBER:response_http_version}\s+(?\d+)" 
      }
    }
    
    grok {
      match => { 
        "message" => "--[a-fA-F0-9]{8}-H--\s(?[\s\S]?)(?=--[a-fA-F0-9]{8}-[I-Z]--|$)" 
      }
    }
    
    if [audit_messages] {
      grok {
        match => { 
          "audit_messages" => "Message: (?.?)\. \[file \"(?.?)\"\] \[line \"(?\d+)\"\] \[id \"(?\d+)\"\] \[rev \"(?\d+)\"\] \[msg \"(?.?)\"\] \[data \"(?.?)\"\] \[severity \"(?.?)\"\] \[ver \"(?.?)\"\] \[maturity \"(?\d+)\"\] \[accuracy \"(?\d+)\"\] \[tag \"(?.*?)\"\]" 
        }
      }
    }
    
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    
    mutate {
      convert => { 
        "client_port" => "integer"
        "server_port" => "integer"
        "response_code" => "integer"
        "rule_id" => "integer"
        "rule_line" => "integer"
        "rule_revision" => "integer"
        "maturity" => "integer"
        "accuracy" => "integer"
      }
    }
    
    if [client_ip] {
      geoip {
        source => "client_ip"
        target => "geoip"
      }
    }
    
    if [rule_tags] {
      mutate {
        split => { "rule_tags" => "," }
      }
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "modsecurity-logs-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "YOUR_ELASTIC_PASSWORD"
    ssl => true
    ssl_certificate_verification => false
  }
  
  stdout {
    codec => rubydebug
  }
}

Update Logstash configuration with your password

Replace YOUR_ELASTIC_PASSWORD in the Logstash configuration with the actual elastic user password from the previous setup step.

sudo sed -i 's/YOUR_ELASTIC_PASSWORD/your_actual_password_here/g' /etc/logstash/conf.d/modsecurity.conf

Configure Logstash JVM settings

Set appropriate heap size for Logstash. For ModSecurity log processing, 1-2GB is usually sufficient.

-Xms1g
-Xmx1g

Start and enable Logstash

Start Logstash to begin processing ModSecurity logs and enable it to start on boot.

sudo systemctl enable --now logstash
sudo systemctl status logstash

Install Kibana 8

Install Kibana for creating dashboards and visualizations of your ModSecurity security events.

sudo apt install -y kibana

sudo dnf install -y --enablerepo=elasticsearch kibana

Configure Kibana

Configure Kibana to connect to Elasticsearch with SSL and proper authentication for secure access to your ModSecurity dashboards.

server.port: 5601
server.host: "0.0.0.0"
server.name: "modsecurity-kibana"
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "YOUR_KIBANA_SYSTEM_PASSWORD"
elasticsearch.ssl.certificateAuthorities: ["/etc/elasticsearch/certs/http_ca.crt"]
xpack.security.encryptionKey: "something_at_least_32_characters_long_for_session_encryption"
xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters_long_for_saved_objects"
xpack.reporting.encryptionKey: "something_at_least_32_characters_long_for_reporting"

Copy Elasticsearch CA certificate

Copy the Elasticsearch CA certificate so Kibana can verify the SSL connection to Elasticsearch.

sudo cp /etc/elasticsearch/certs/http_ca.crt /etc/kibana/
sudo chown kibana:kibana /etc/kibana/http_ca.crt
sudo chmod 644 /etc/kibana/http_ca.crt

Start and enable Kibana

Start Kibana and enable it to start on boot. Initial startup may take a few minutes as Kibana configures itself.

sudo systemctl enable --now kibana
sudo systemctl status kibana

Configure firewall access

Open the necessary ports for ELK Stack components. Elasticsearch (9200), Logstash (5044), and Kibana (5601).

sudo ufw allow 9200/tcp
sudo ufw allow 5601/tcp
sudo ufw allow 5044/tcp
sudo ufw reload

sudo firewall-cmd --permanent --add-port=9200/tcp
sudo firewall-cmd --permanent --add-port=5601/tcp
sudo firewall-cmd --permanent --add-port=5044/tcp
sudo firewall-cmd --reload

Configure ModSecurity log forwarding

Set up ModSecurity on your web servers to forward logs to your ELK Stack. This configuration assumes you already have ModSecurity configured with NGINX.

SecAuditEngine On
SecAuditLogType Concurrent
SecAuditLogStorageDir /var/log/modsec_audit
SecAuditLogFormat JSON
SecAuditLogParts ABCFHZ

Set up Filebeat for log shipping

Install Filebeat on your web servers to ship ModSecurity logs to the ELK Stack server.

sudo apt install -y filebeat

sudo dnf install -y --enablerepo=elasticsearch filebeat

Configure Filebeat for ModSecurity

Configure Filebeat to collect ModSecurity logs and send them to your ELK Stack server.

filebeat.inputs:
type: log
  enabled: true
  paths:
    - /var/log/modsec_audit.log
  fields:
    log_type: modsecurity
  fields_under_root: true
  multiline.pattern: '^--[a-fA-F0-9]{8}-A--$'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["YOUR_ELK_SERVER_IP:5044"]

processors:
add_host_metadata:
    when.not.contains.tags: forwarded

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

Update Logstash input for Beats

Modify the Logstash configuration to accept logs from Filebeat instead of reading local files.

input {
  beats {
    port => 5044
  }
}

filter {
  if [log_type] == "modsecurity" {
    # ... (keep the same filter configuration as before)
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "modsecurity-logs-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "YOUR_ELASTIC_PASSWORD"
    ssl => true
    ssl_certificate_verification => false
  }
}

Create Kibana index pattern

Access Kibana at http://your-server-ip:5601 and create an index pattern for ModSecurity logs. Use the pattern modsecurity-logs-* and select @timestamp as the time field.

curl -X POST "localhost:5601/api/saved_objects/index-pattern/modsecurity-logs-*" \
  -H "Content-Type: application/json" \
  -H "kbn-xsrf: true" \
  -u "elastic:YOUR_ELASTIC_PASSWORD" \
  -d '{
    "attributes": {
      "title": "modsecurity-logs-*",
      "timeFieldName": "@timestamp"
    }
  }'

Import ModSecurity dashboard

Create a comprehensive dashboard for ModSecurity monitoring with key security metrics and attack visualization.

{
  "version": "8.0.0",
  "objects": [
    {
      "id": "modsecurity-overview",
      "type": "dashboard",
      "attributes": {
        "title": "ModSecurity Security Overview",
        "hits": 0,
        "description": "Overview of ModSecurity security events and threats",
        "panelsJSON": "[{\"version\":\"8.0.0\",\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"1\"},\"panelIndex\":\"1\",\"embeddableConfig\":{},\"panelRefName\":\"panel_1\"}]",
        "timeRestore": false,
        "version": 1
      }
    }
  ]
}