Set up centralized security monitoring with ClamAV 1.4 and Elasticsearch 8

Update system packages and install Java

Start by updating your system and installing Java 11 or higher, which is required for Elasticsearch.

sudo apt update && sudo apt upgrade -y
sudo apt install -y openjdk-17-jdk curl gnupg2 software-properties-common

sudo dnf update -y
sudo dnf install -y java-17-openjdk curl gnupg2

Install Elasticsearch 8

Add the Elasticsearch repository and install the latest version with security features enabled by default.

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
sudo apt install -y elasticsearch

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
echo '[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md' | sudo tee /etc/yum.repos.d/elasticsearch.repo
sudo dnf install -y --enablerepo=elasticsearch elasticsearch

Configure Elasticsearch for security monitoring

Configure Elasticsearch with optimized settings for security log processing and adequate memory allocation.

cluster.name: security-monitoring
node.name: security-node-1
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/transport.p12
xpack.security.transport.ssl.truststore.path: certs/transport.p12
bootstrap.memory_lock: true
indices.query.bool.max_clause_count: 10000

Configure Elasticsearch JVM heap size

Set the JVM heap to half of your available RAM for optimal performance. This example uses 2GB for a 4GB system.

-Xms2g
-Xmx2g

Start and configure Elasticsearch

Start Elasticsearch and save the auto-generated elastic user password for later use.

sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch
sudo systemctl status elasticsearch

Reset the elastic user password and save it securely:

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i

Install ClamAV antivirus scanner

Install ClamAV daemon and utilities for real-time virus scanning capabilities.

sudo apt install -y clamav clamav-daemon clamav-freshclam clamav-unofficial-sigs

sudo dnf install -y epel-release
sudo dnf install -y clamav clamav-scanner clamav-scanner-systemd clamav-update

Configure ClamAV daemon

Configure ClamAV daemon for optimal scanning performance and logging integration with Elasticsearch.

LogFile /var/log/clamav/clamav.log
LogTime yes
LogClean yes
LogSyslog yes
LogFacility LOG_LOCAL6
LogVerbose yes
ExtendedDetectionInfo yes
PidFile /var/run/clamav/clamd.pid
LocalSocket /var/run/clamav/clamd.ctl
FixStaleSocket yes
LocalSocketGroup clamav
LocalSocketMode 666
User clamav
ScanMail yes
ScanArchive yes
ArchiveBlockEncrypted no
MaxDirectoryRecursion 15
FollowDirectorySymlinks no
FollowFileSymlinks no
ReadTimeout 180
MaxThreads 12
MaxConnectionQueueLength 15
StreamMaxLength 25M
MaxFiles 10000
MaxRecursion 16
MaxFileSize 25M
ScanPDF yes
ScanSWF yes
ScanXMLDOCS yes
ScanHWP3 yes
ScanOLE2 yes
AlertBrokenExecutables yes
AlertBrokenMedia no
AlertEncrypted no
AlertEncryptedArchive no
AlertEncryptedDoc no
AlertOLE2Macros yes
AlertPhishingSSLMismatch yes
AlertPhishingCloak yes
AlertPartitionIntersection no
HeuristicScanPrecedence no
StructuredDataDetection no
CommandReadTimeout 30
SendBufTimeout 200
MaxQueue 100
IdleTimeout 30
ExcludePath ^/proc/
ExcludePath ^/sys/

Configure ClamAV freshclam for updates

Set up automatic virus definition updates to ensure protection against the latest threats.

DatabaseDirectory /var/lib/clamav
UpdateLogFile /var/log/clamav/freshclam.log
LogVerbose yes
LogSyslog yes
LogFacility LOG_LOCAL6
LogFileMaxSize 0
LogRotate yes
LogTime yes
Foreground no
Debug no
MaxAttempts 5
DatabaseOwner clamav
AllowSupplementaryGroups yes
NotifyClamd /etc/clamav/clamd.conf
Checks 24
ConnectTimeout 30
ReceiveTimeout 0
TestDatabases yes
ScriptedUpdates yes
CompressLocalDatabase no
Bytecode yes
IncludePUA Packed
IncludePUA PWS
IncludePUA Casino
IncludePUA Suspect
ExtraDatabase /var/lib/clamav-unofficial-sigs

Install and configure Logstash

Install Logstash to process and forward ClamAV logs to Elasticsearch for analysis.

sudo apt install -y logstash

sudo dnf install -y --enablerepo=elasticsearch logstash

Create Logstash pipeline configuration

Configure Logstash to parse ClamAV logs and send structured data to Elasticsearch with proper field mapping.

input {
  file {
    path => "/var/log/clamav/clamav.log"
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/sincedb_clamav"
    tags => ["clamav"]
  }
  syslog {
    port => 5514
    tags => ["security-logs"]
  }
}

filter {
  if "clamav" in [tags] {
    grok {
      match => { 
        "message" => "%{TIMESTAMP_ISO8601:timestamp} -> %{GREEDYDATA:clamav_message}"
      }
    }
    
    if [clamav_message] =~ /FOUND/ {
      mutate {
        add_field => { "threat_detected" => "true" }
        add_field => { "severity" => "high" }
      }
      grok {
        match => {
          "clamav_message" => "%{PATH:infected_file}: %{DATA:virus_name} FOUND"
        }
      }
    }
    
    if [clamav_message] =~ /OK$/ {
      mutate {
        add_field => { "threat_detected" => "false" }
        add_field => { "severity" => "low" }
      }
    }
    
    date {
      match => [ "timestamp", "ISO8601" ]
    }
  }
  
  mutate {
    add_field => { "[@metadata][index]" => "security-monitoring-%{+YYYY.MM.dd}" }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    index => "%{[@metadata][index]}"
    user => "elastic"
    password => "${ELASTIC_PASSWORD}"
    ssl_certificate_verification => false
    ssl_enabled => true
  }
  
  if [threat_detected] == "true" {
    email {
      to => "security@example.com"
      subject => "Security Alert: Malware Detected - %{virus_name}"
      body => "Threat detected on %{host}: %{infected_file} contains %{virus_name}\n\nTimestamp: %{@timestamp}\nSeverity: %{severity}"
      from => "security-system@example.com"
      smtp_host => "localhost"
      smtp_port => 25
    }
  }
}

Set up environment variables for Logstash

Create environment file with Elasticsearch credentials for secure authentication.

ELASTIC_PASSWORD=your-elastic-password-here

sudo chown root:logstash /etc/logstash/startup.options
sudo chmod 640 /etc/logstash/startup.options

Configure rsyslog for centralized logging

Set up rsyslog to forward ClamAV syslog messages to Logstash for centralized processing.

# ClamAV logging configuration
local6.*    @@127.0.0.1:5514
Stop processing after sending to Logstash
& stop

sudo systemctl restart rsyslog

Create ClamAV scanning scripts

Create automated scanning scripts for different directories and file types with proper logging integration.

#!/bin/bash
Automated home directory scanning script

LOGFILE="/var/log/clamav/scheduled-scan.log"
SCAN_DIR="/home"
DATE=$(date '+%Y-%m-%d %H:%M:%S')

echo "[$DATE] Starting scheduled scan of $SCAN_DIR" >> $LOGFILE

/usr/bin/clamscan -r -i --bell --log=$LOGFILE $SCAN_DIR

SCAN_EXIT_CODE=$?

if [ $SCAN_EXIT_CODE -eq 0 ]; then
    echo "[$DATE] Scan completed successfully - No threats found" >> $LOGFILE
    logger -p local6.info "ClamAV: Scheduled scan of $SCAN_DIR completed successfully"
elif [ $SCAN_EXIT_CODE -eq 1 ]; then
    echo "[$DATE] THREAT DETECTED during scan of $SCAN_DIR" >> $LOGFILE
    logger -p local6.warn "ClamAV: THREAT DETECTED during scheduled scan of $SCAN_DIR"
    # Send immediate notification
    mail -s "URGENT: Malware detected on $(hostname)" security@example.com < $LOGFILE
else
    echo "[$DATE] Scan error occurred (exit code: $SCAN_EXIT_CODE)" >> $LOGFILE
    logger -p local6.error "ClamAV: Scan error occurred for $SCAN_DIR (exit code: $SCAN_EXIT_CODE)"
fi

sudo chmod +x /usr/local/bin/clamav-scan-home.sh
sudo chown clamav:clamav /usr/local/bin/clamav-scan-home.sh

Set up automated scanning schedule

Configure systemd timers for regular automated scanning with different intervals for various directories.

[Unit]
Description=ClamAV Home Directory Scan
After=network.target

[Service]
Type=oneshot
User=clamav
ExecStart=/usr/local/bin/clamav-scan-home.sh
StandardOutput=journal
StandardError=journal

[Unit]
Description=Run ClamAV home scan daily at 2 AM
Requires=clamav-home-scan.service

[Timer]
OnCalendar=daily
Persistent=true
AccuracySec=1h

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now clamav-home-scan.timer
sudo systemctl list-timers clamav-home-scan.timer

Start and enable all services

Start all components of the security monitoring stack and ensure they start automatically on boot.

# Update ClamAV virus definitions
sudo freshclam

Start ClamAV services
sudo systemctl enable --now clamav-freshclam
sudo systemctl enable --now clamav-daemon

Start Logstash with environment variables
sudo systemctl enable --now logstash

Verify all services are running
sudo systemctl status elasticsearch clamav-daemon clamav-freshclam logstash

Create Elasticsearch index template

Set up a proper index template for security monitoring data with optimized field mappings and retention policies.

curl -X PUT "https://localhost:9200/_index_template/security-monitoring" \
  -H "Content-Type: application/json" \
  -u elastic:your-elastic-password \
  -k -d '{
  "index_patterns": ["security-monitoring-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.lifecycle.name": "security-monitoring-policy",
      "index.lifecycle.rollover_alias": "security-monitoring"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "host": { "type": "keyword" },
        "threat_detected": { "type": "boolean" },
        "severity": { "type": "keyword" },
        "virus_name": { "type": "keyword" },
        "infected_file": { "type": "text", "fields": { "keyword": { "type": "keyword" } } },
        "clamav_message": { "type": "text" },
        "message": { "type": "text" }
      }
    }
  }
}'

Install Kibana for visualization

Install Kibana to create security dashboards and configure alerting rules for threat detection.

sudo apt install -y kibana

sudo dnf install -y --enablerepo=elasticsearch kibana

Configure Kibana for security monitoring

Set up Kibana with proper authentication and SSL configuration to access your security monitoring data.

server.port: 5601
server.host: "0.0.0.0"
server.name: "security-monitoring-kibana"
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "your-elastic-password"
elasticsearch.ssl.verificationMode: none
xpack.security.enabled: true
xpack.security.session.idleTimeout: "1h"
xpack.security.session.lifespan: "30d"
logging.appenders.file.fileName: /var/log/kibana/kibana.log
logging.root.level: info

sudo systemctl enable --now kibana
sudo systemctl status kibana

Create security monitoring dashboard

Access Kibana at http://your-server-ip:5601 and create index patterns and dashboards for security monitoring.