Configure centralized logging with rsyslog and logrotate for system monitoring and log management

Update system packages

Start by updating your package manager to ensure you get the latest versions of rsyslog and related tools.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install rsyslog and TLS support

Install rsyslog with TLS modules for secure log transmission and additional utilities for certificate management.

sudo apt install -y rsyslog rsyslog-gnutls gnutls-bin logrotate

sudo dnf install -y rsyslog rsyslog-gnutls gnutls-utils logrotate

Create certificate authority for TLS

Generate a certificate authority and server certificates for encrypted log transmission. This ensures logs cannot be intercepted or tampered with during transmission.

sudo mkdir -p /etc/rsyslog-certs
cd /etc/rsyslog-certs

sudo openssl genrsa -out ca-key.pem 4096
sudo openssl req -new -x509 -key ca-key.pem -out ca.pem -days 365 \  
  -subj "/C=US/ST=State/L=City/O=Organization/CN=rsyslog-ca"

Generate server certificates

Create server certificates signed by your CA for the rsyslog server. Replace example.com with your actual server hostname or IP address.

sudo openssl genrsa -out server-key.pem 4096
sudo openssl req -new -key server-key.pem -out server.csr \  
  -subj "/C=US/ST=State/L=City/O=Organization/CN=example.com"
sudo openssl x509 -req -in server.csr -CA ca.pem -CAkey ca-key.pem \  
  -CAcreateserial -out server-cert.pem -days 365

Set certificate permissions

Configure proper ownership and permissions for the certificates. The syslog user needs read access to certificates but private keys should be restricted.

sudo chown -R syslog:syslog /etc/rsyslog-certs
sudo chmod 755 /etc/rsyslog-certs
sudo chmod 644 /etc/rsyslog-certs/.pem /etc/rsyslog-certs/.csr
sudo chmod 600 /etc/rsyslog-certs/*-key.pem

Configure rsyslog server

Configure the rsyslog server to accept incoming logs over TCP with TLS encryption. This creates a secure centralized logging hub.

# Enable modules for TLS and TCP reception
$ModLoad imtcp
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/rsyslog-certs/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog-certs/server-cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog-certs/server-key.pem

TLS settings
$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverPermittedPeer *.example.com
$InputTCPServerStreamDriverMode 1
$InputTCPServerRun 6514

Template for remote logs with hostname
$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
. ?RemoteLogs
& stop

Local system logs
*.info;mail.none;authpriv.none;cron.none    /var/log/messages
authpriv.*                                  /var/log/secure
mail.*                                      /var/log/maillog
cron.*                                      /var/log/cron
.emerg                                     :omusrmsg:
uucp,news.crit                             /var/log/spooler
local7.*                                    /var/log/boot.log

Create remote log directory

Create the directory structure where remote logs will be stored, organized by hostname for easy identification.

sudo mkdir -p /var/log/remote
sudo chown syslog:syslog /var/log/remote
sudo chmod 755 /var/log/remote

Configure firewall for log server

Open port 6514 for secure rsyslog communication. This port is used for TLS-encrypted syslog transmission.

sudo ufw allow 6514/tcp
sudo ufw reload

sudo firewall-cmd --permanent --add-port=6514/tcp
sudo firewall-cmd --reload

Generate client certificates

Create certificates for each client that will send logs to the server. Run this on the server for each client machine.

cd /etc/rsyslog-certs
sudo openssl genrsa -out client-key.pem 4096
sudo openssl req -new -key client-key.pem -out client.csr \  
  -subj "/C=US/ST=State/L=City/O=Organization/CN=client.example.com"
sudo openssl x509 -req -in client.csr -CA ca.pem -CAkey ca-key.pem \  
  -CAcreateserial -out client-cert.pem -days 365

Configure rsyslog clients

On each client machine, configure rsyslog to forward logs to the central server. Copy the client certificates and CA to each client machine first.

sudo mkdir -p /etc/rsyslog-certs
sudo scp server:/etc/rsyslog-certs/ca.pem /etc/rsyslog-certs/
sudo scp server:/etc/rsyslog-certs/client-cert.pem /etc/rsyslog-certs/
sudo scp server:/etc/rsyslog-certs/client-key.pem /etc/rsyslog-certs/

# TLS configuration for client
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/rsyslog-certs/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog-certs/client-cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog-certs/client-key.pem
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer example.com
$ActionSendStreamDriverMode 1

Forward all logs to central server
. @@example.com:6514

Also keep local copies
*.info;mail.none;authpriv.none;cron.none    /var/log/messages
authpriv.*                                  /var/log/secure
mail.*                                      /var/log/maillog
cron.*                                      /var/log/cron

Configure log filtering rules

Add custom filtering rules to separate different types of logs into specific files for easier analysis.

# Separate SSH logs
if $programname == 'sshd' then /var/log/remote/%HOSTNAME%/ssh.log
& stop

Separate web server logs  
if $programname == 'nginx' or $programname == 'apache2' then /var/log/remote/%HOSTNAME%/web.log
& stop

Separate database logs
if $programname startswith 'mysql' or $programname startswith 'postgres' then /var/log/remote/%HOSTNAME%/database.log
& stop

Filter by severity - errors only
if $syslogseverity <= 3 then /var/log/remote/%HOSTNAME%/errors.log

Filter by facility - mail logs
if $syslogfacility-text == 'mail' then /var/log/remote/%HOSTNAME%/mail.log
& stop

Configure logrotate for remote logs

Set up automatic log rotation to prevent remote logs from filling up disk space. This configuration rotates logs weekly and compresses old files.

/var/log/remote//.log {
    weekly
    missingok
    rotate 8
    compress
    delaycompress
    notifempty
    create 640 syslog syslog
    postrotate
        /bin/kill -HUP cat /var/run/rsyslogd.pid 2> /dev/null 2> /dev/null || true
    endscript
}

/var/log/remote/*/errors.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 640 syslog syslog
    postrotate
        /bin/kill -HUP cat /var/run/rsyslogd.pid 2> /dev/null 2> /dev/null || true
    endscript
}

Configure standard log rotation

Update the standard rsyslog logrotate configuration to handle increased log volume from centralized logging.

/var/log/syslog {
    daily
    missingok
    rotate 15
    compress
    delaycompress
    notifempty
    create 640 syslog adm
    postrotate
        /bin/kill -HUP cat /var/run/rsyslogd.pid 2> /dev/null 2> /dev/null || true
    endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages {
    weekly
    missingok
    rotate 8
    compress
    delaycompress
    notifempty
    create 640 syslog adm
    postrotate
        /bin/kill -HUP cat /var/run/rsyslogd.pid 2> /dev/null 2> /dev/null || true
    endscript
}

Set up log monitoring script

Create a monitoring script that checks for log anomalies and sends alerts when error rates exceed thresholds.

#!/bin/bash

Configuration
LOG_DIR="/var/log/remote"
ERROR_THRESHOLD=10
EMAIL_ALERT="admin@example.com"
TMP_FILE="/tmp/log-errors.tmp"

Check for errors in the last hour
find $LOG_DIR -name "errors.log" -type f | while read logfile; do
    hostname=$(echo $logfile | cut -d'/' -f5)
    error_count=$(grep "$(date -d '1 hour ago' '+%b %d %H')" $logfile 2>/dev/null | wc -l)
    
    if [ $error_count -gt $ERROR_THRESHOLD ]; then
        echo "High error rate on $hostname: $error_count errors in last hour" >> $TMP_FILE
    fi
done

Send alert if errors found
if [ -s $TMP_FILE ]; then
    mail -s "Log Alert: High Error Rates Detected" $EMAIL_ALERT < $TMP_FILE
    rm -f $TMP_FILE
fi

Check disk usage
DISK_USAGE=$(df /var/log | tail -1 | awk '{print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt 80 ]; then
    echo "Warning: Log disk usage at ${DISK_USAGE}%" | mail -s "Log Alert: High Disk Usage" $EMAIL_ALERT
fi

sudo chmod +x /usr/local/bin/log-monitor.sh

Create monitoring cron job

Schedule the monitoring script to run every hour and check for issues automatically.

sudo crontab -e

# Run log monitoring every hour
0     /usr/local/bin/log-monitor.sh

Test logrotate daily
0 2   * /usr/sbin/logrotate -f /etc/logrotate.d/rsyslog-remote

Start and enable services

Enable rsyslog to start automatically and restart it to load the new configuration.

sudo systemctl enable rsyslog
sudo systemctl restart rsyslog
sudo systemctl status rsyslog