Centralized Logging with rsyslog and logrotate

Configure a centralized rsyslog server to collect security events from multiple systems, implement automated log rotation with logrotate, and set up filtering and alerting for critical security incidents across your infrastructure.

Prerequisites

Multiple Linux servers for centralized logging
Email system configured for alerts
Network connectivity between servers
Sufficient disk space for log storage

What this solves

Centralized logging consolidates security events from multiple servers into a single location, making it easier to monitor, analyze, and respond to security incidents. This tutorial shows you how to configure rsyslog for secure log forwarding, implement automated log rotation to manage disk space, and set up filtering and alerting for critical security events.

Step-by-step configuration

Update system packages

Start by updating your package manager to ensure you have the latest security patches and package versions.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install rsyslog and dependencies

Install rsyslog and necessary packages for centralized logging. Most systems have rsyslog pre-installed, but we'll ensure all required components are available.

sudo apt install -y rsyslog rsyslog-gnutls logrotate mailutils

sudo dnf install -y rsyslog rsyslog-gnutls logrotate mailx

Configure rsyslog server

Configure the central rsyslog server to receive logs from remote clients. This configuration enables UDP and TCP reception on port 514 with security filtering.

# Enable UDP and TCP reception
module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

# Create directories for remote logs
$CreateDirs on
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755

# Template for remote log file organization
$template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%.log"

# Template for security events
$template SecurityEvents,"/var/log/security/%HOSTNAME%/security-%$YEAR%-%$MONTH%-%$DAY%.log"

# Security event filtering
:msg, contains, "authentication failure" ?SecurityEvents
:msg, contains, "Failed password" ?SecurityEvents
:msg, contains, "Invalid user" ?SecurityEvents
:msg, contains, "sudo:" ?SecurityEvents
:msg, contains, "su:" ?SecurityEvents
:msg, contains, "COMMAND=" ?SecurityEvents
& stop

# General remote logging
if $fromhost-ip != '127.0.0.1' then ?RemoteHost
& stop

# Local logging continues normally
*.*;auth,authpriv.none          /var/log/syslog
authpriv.*                      /var/log/auth.log

Create log directories

Create the directory structure for centralized logs with proper ownership and permissions for security.

sudo mkdir -p /var/log/remote /var/log/security
sudo chown syslog:adm /var/log/remote /var/log/security
sudo chmod 755 /var/log/remote /var/log/security

Never use chmod 777. It gives every user on the system full access to your log files. Instead, use specific ownership and minimal permissions for security.

Configure firewall rules

Open the necessary firewall ports for rsyslog communication. Use specific rules instead of disabling the firewall.

sudo ufw allow from 203.0.113.0/24 to any port 514
sudo ufw reload

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="udp" port="514" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="514" accept'
sudo firewall-cmd --reload

Configure rsyslog client systems

Configure client systems to forward logs to the central server. Add this configuration to each client system you want to monitor.

# Forward all logs to central server
*.*                             @@203.0.113.10:514

# Forward security events with high priority
authpriv.*                      @@203.0.113.10:514
auth.*                          @@203.0.113.10:514

# Local logging continues
& /var/log/messages
& /var/log/auth.log

Restart rsyslog services

Restart rsyslog on both server and client systems to apply the new configuration.

sudo systemctl restart rsyslog
sudo systemctl enable rsyslog
sudo systemctl status rsyslog

Configure logrotate for centralized logs

Set up automated log rotation to manage disk space and maintain log history. This prevents log files from consuming all available storage.

/var/log/remote/*/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 0640 syslog adm
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

/var/log/security/*/*.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    create 0640 syslog adm
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
        # Send alert email for security log rotation
        echo "Security logs rotated on $(hostname) at $(date)" | mail -s "Security Log Rotation" admin@example.com
    endscript
}

Set up security event alerting

Create a script to monitor security events and send alerts when critical events are detected.

#!/bin/bash

# Security event monitoring script
LOG_DIR="/var/log/security"
ALERT_EMAIL="admin@example.com"
TEMP_FILE="/tmp/security-alerts.tmp"
LAST_CHECK_FILE="/var/lib/rsyslog/last-security-check"

# Create last check file if it doesn't exist
if [ ! -f "$LAST_CHECK_FILE" ]; then
    echo "$(date -d '1 minute ago' '+%Y-%m-%d %H:%M:%S')" > "$LAST_CHECK_FILE"
fi

LAST_CHECK=$(cat "$LAST_CHECK_FILE")
CURRENT_TIME=$(date '+%Y-%m-%d %H:%M:%S')

# Find recent security events
find "$LOG_DIR" -name "*.log" -newer "$LAST_CHECK_FILE" | while read -r logfile; do
    # Check for critical security events
    grep -E "(authentication failure|Failed password|Invalid user|sudo: .* : TTY=|COMMAND=)" "$logfile" | \
    while IFS= read -r line; do
        echo "$logfile: $line" >> "$TEMP_FILE"
    done
done

# Send alert if events found
if [ -s "$TEMP_FILE" ]; then
    {
        echo "Security events detected between $LAST_CHECK and $CURRENT_TIME:"
        echo ""
        cat "$TEMP_FILE"
        echo ""
        echo "Please review these events and take appropriate action."
    } | mail -s "[SECURITY ALERT] Events detected on $(hostname)" "$ALERT_EMAIL"
    
    # Log the alert
    logger -t security-monitor "Security alert sent to $ALERT_EMAIL"
fi

# Update last check time
echo "$CURRENT_TIME" > "$LAST_CHECK_FILE"

# Cleanup
rm -f "$TEMP_FILE"

Set script permissions and create directories

Make the security monitoring script executable and create required directories with proper ownership.

sudo chmod 755 /usr/local/bin/security-monitor.sh
sudo mkdir -p /var/lib/rsyslog
sudo chown syslog:adm /var/lib/rsyslog
sudo chmod 755 /var/lib/rsyslog

Configure automated monitoring with cron

Set up a cron job to run the security monitoring script every 5 minutes for near real-time alerting.

sudo crontab -e

Add this line to the crontab:

# Security event monitoring every 5 minutes
*/5 * * * * /usr/local/bin/security-monitor.sh >/dev/null 2>&1

Test logrotate configuration

Test the logrotate configuration to ensure it works correctly before relying on automated rotation.

sudo logrotate -d /etc/logrotate.d/rsyslog-remote
sudo logrotate -f /etc/logrotate.d/rsyslog-remote

Verify your setup

Test the centralized logging system to ensure logs are being received and processed correctly.

# Check rsyslog status
sudo systemctl status rsyslog

# Test log forwarding from client
logger -p auth.info "Test security log message from $(hostname)"

# Verify logs are received on server
sudo tail -f /var/log/remote/*/$(date +%Y-%m-%d).log

# Check security log filtering
sudo tail -f /var/log/security/*/security-$(date +%Y-%m-%d).log

# Test failed authentication (generates security event)
sudo su - nonexistentuser

# Verify logrotate is working
sudo logrotate -v /etc/logrotate.d/rsyslog-remote

# Check disk usage of log directories
sudo du -sh /var/log/remote /var/log/security

Common issues

Symptom	Cause	Fix
Logs not being received	Firewall blocking port 514	Configure firewall rules: `sudo ufw allow from trusted_network to any port 514`
Permission denied writing logs	Incorrect directory ownership	Fix ownership: `sudo chown syslog:adm /var/log/remote /var/log/security`
Logrotate not working	Missing or incorrect permissions	Check script permissions: `sudo chmod 755 /usr/lib/rsyslog/rsyslog-rotate`
Security alerts not sent	Mail system not configured	Install and configure mail system: `sudo dpkg-reconfigure exim4-config`
High disk usage	Logs not being rotated	Force rotation: `sudo logrotate -f /etc/logrotate.conf`
Client logs not forwarding	Network connectivity issues	Test connectivity: `telnet server_ip 514`

Next steps

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash
set -euo pipefail

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

# Default values
LOG_SERVER_IP="${1:-}"
CLIENT_SUBNET="${2:-}"

# Usage function
usage() {
    echo "Usage: $0 <log_server_ip> <client_subnet>"
    echo "Example: $0 192.168.1.100 192.168.1.0/24"
    echo "  log_server_ip: IP address of the central log server"
    echo "  client_subnet: Subnet to allow for log forwarding (CIDR notation)"
    exit 1
}

# Validate arguments
if [[ -z "$LOG_SERVER_IP" || -z "$CLIENT_SUBNET" ]]; then
    usage
fi

# Validate IP address format
if ! [[ $LOG_SERVER_IP =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    echo -e "${RED}Error: Invalid IP address format${NC}"
    exit 1
fi

# Error handling and cleanup
cleanup() {
    echo -e "${RED}Installation failed. Cleaning up...${NC}"
    systemctl restart rsyslog 2>/dev/null || true
}
trap cleanup ERR

# Check if running as root or with sudo
if [[ $EUID -ne 0 ]]; then
    echo -e "${RED}This script must be run as root or with sudo${NC}"
    exit 1
fi

echo -e "${GREEN}Setting up centralized logging with rsyslog${NC}"

# Detect distribution
echo -e "${YELLOW}[1/9] Detecting distribution...${NC}"
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian) 
            PKG_MGR="apt"
            PKG_UPDATE="apt update && apt upgrade -y"
            PKG_INSTALL="apt install -y"
            SYSLOG_USER="syslog"
            SYSLOG_GROUP="adm"
            FIREWALL_CMD="ufw"
            ;;
        almalinux|rocky|centos|rhel|ol|fedora)
            PKG_MGR="dnf"
            PKG_UPDATE="dnf update -y"
            PKG_INSTALL="dnf install -y"
            SYSLOG_USER="root"
            SYSLOG_GROUP="root"
            FIREWALL_CMD="firewalld"
            ;;
        amzn)
            PKG_MGR="yum"
            PKG_UPDATE="yum update -y"
            PKG_INSTALL="yum install -y"
            SYSLOG_USER="root"
            SYSLOG_GROUP="root"
            FIREWALL_CMD="firewalld"
            ;;
        *)
            echo -e "${RED}Unsupported distribution: $ID${NC}"
            exit 1
            ;;
    esac
else
    echo -e "${RED}Cannot detect distribution${NC}"
    exit 1
fi

echo -e "${GREEN}Detected: $PRETTY_NAME${NC}"

# Update system packages
echo -e "${YELLOW}[2/9] Updating system packages...${NC}"
$PKG_UPDATE

# Install required packages
echo -e "${YELLOW}[3/9] Installing rsyslog and dependencies...${NC}"
if [[ "$PKG_MGR" == "apt" ]]; then
    $PKG_INSTALL rsyslog rsyslog-gnutls logrotate mailutils
else
    $PKG_INSTALL rsyslog rsyslog-gnutls logrotate mailx
fi

# Backup original rsyslog configuration
echo -e "${YELLOW}[4/9] Backing up original configuration...${NC}"
cp /etc/rsyslog.conf /etc/rsyslog.conf.backup.$(date +%Y%m%d_%H%M%S)

# Create log directories
echo -e "${YELLOW}[5/9] Creating log directories...${NC}"
mkdir -p /var/log/remote /var/log/security
chown $SYSLOG_USER:$SYSLOG_GROUP /var/log/remote /var/log/security
chmod 755 /var/log/remote /var/log/security

# Configure rsyslog server
echo -e "${YELLOW}[6/9] Configuring rsyslog server...${NC}"
cat > /etc/rsyslog.d/10-centralized-logging.conf << EOF
# Enable UDP and TCP reception
module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

# Create directories for remote logs
\$CreateDirs on
\$FileOwner $SYSLOG_USER
\$FileGroup $SYSLOG_GROUP
\$FileCreateMode 0640
\$DirCreateMode 0755

# Template for remote log file organization
\$template RemoteHost,"/var/log/remote/%HOSTNAME%/%\$YEAR%-%\$MONTH%-%\$DAY%.log"

# Template for security events
\$template SecurityEvents,"/var/log/security/%HOSTNAME%/security-%\$YEAR%-%\$MONTH%-%\$DAY%.log"

# Security event filtering
:msg, contains, "authentication failure" ?SecurityEvents
:msg, contains, "Failed password" ?SecurityEvents
:msg, contains, "Invalid user" ?SecurityEvents
:msg, contains, "sudo:" ?SecurityEvents
:msg, contains, "su:" ?SecurityEvents
:msg, contains, "COMMAND=" ?SecurityEvents
& stop

# General remote logging
if \$fromhost-ip != '127.0.0.1' then ?RemoteHost
& stop
EOF

chmod 644 /etc/rsyslog.d/10-centralized-logging.conf

# Configure logrotate
echo -e "${YELLOW}[7/9] Configuring log rotation...${NC}"
cat > /etc/logrotate.d/centralized-logs << EOF
/var/log/remote/*/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 $SYSLOG_USER $SYSLOG_GROUP
    sharedscripts
    postrotate
        /bin/kill -HUP \$(cat /var/run/rsyslogd.pid 2> /dev/null) 2> /dev/null || true
    endscript
}

/var/log/security/*/*.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    create 640 $SYSLOG_USER $SYSLOG_GROUP
    sharedscripts
    postrotate
        /bin/kill -HUP \$(cat /var/run/rsyslogd.pid 2> /dev/null) 2> /dev/null || true
    endscript
}
EOF

chmod 644 /etc/logrotate.d/centralized-logs

# Configure firewall
echo -e "${YELLOW}[8/9] Configuring firewall rules...${NC}"
case "$FIREWALL_CMD" in
    ufw)
        if command -v ufw >/dev/null 2>&1; then
            ufw --force enable
            ufw allow from $CLIENT_SUBNET to any port 514
            ufw reload
        fi
        ;;
    firewalld)
        if systemctl is-active --quiet firewalld; then
            firewall-cmd --permanent --add-rich-rule="rule family=\"ipv4\" source address=\"$CLIENT_SUBNET\" port protocol=\"udp\" port=\"514\" accept"
            firewall-cmd --permanent --add-rich-rule="rule family=\"ipv4\" source address=\"$CLIENT_SUBNET\" port protocol=\"tcp\" port=\"514\" accept"
            firewall-cmd --reload
        fi
        ;;
esac

# Start and enable rsyslog service
echo -e "${YELLOW}[9/9] Starting rsyslog service...${NC}"
systemctl restart rsyslog
systemctl enable rsyslog

# Verification checks
echo -e "${YELLOW}Performing verification checks...${NC}"

# Check if rsyslog is running
if systemctl is-active --quiet rsyslog; then
    echo -e "${GREEN}✓ rsyslog service is running${NC}"
else
    echo -e "${RED}✗ rsyslog service is not running${NC}"
    exit 1
fi

# Check if ports are listening
if ss -tuln | grep -q ":514"; then
    echo -e "${GREEN}✓ rsyslog is listening on port 514${NC}"
else
    echo -e "${RED}✗ rsyslog is not listening on port 514${NC}"
fi

# Check log directories
if [[ -d "/var/log/remote" && -d "/var/log/security" ]]; then
    echo -e "${GREEN}✓ Log directories created successfully${NC}"
else
    echo -e "${RED}✗ Log directories not found${NC}"
fi

# Check configuration file
if [[ -f "/etc/rsyslog.d/10-centralized-logging.conf" ]]; then
    echo -e "${GREEN}✓ Rsyslog configuration file created${NC}"
else
    echo -e "${RED}✗ Rsyslog configuration file not found${NC}"
fi

echo -e "${GREEN}Centralized logging setup completed successfully!${NC}"
echo -e "${YELLOW}Server Configuration Summary:${NC}"
echo "  - Log server IP: $LOG_SERVER_IP"
echo "  - Allowed client subnet: $CLIENT_SUBNET"
echo "  - Remote logs: /var/log/remote/"
echo "  - Security logs: /var/log/security/"
echo -e "${YELLOW}To configure clients, add this line to their /etc/rsyslog.conf:${NC}"
echo "  *.* @@$LOG_SERVER_IP:514"
echo -e "${YELLOW}Then restart rsyslog on clients: systemctl restart rsyslog${NC}"

Review the script before running. Execute with: bash install.sh

#rsyslog #logrotate #centralized-logging #security-monitoring #log-forwarding

Set up centralized logging with rsyslog and logrotate for security events