Optimize systemd journal logging performance and storage

Check current journal status

First examine your current journal configuration and disk usage to understand what needs optimization.

journalctl --disk-usage
sudo journalctl --verify
systemctl status systemd-journald

Configure storage limits and compression

Set up journal storage limits to prevent excessive disk usage and enable compression to reduce space requirements.

# Journal storage configuration
[Journal]
Storage=persistent
Compress=yes
Seal=yes

# Storage limits
SystemMaxUse=1G
SystemKeepFree=2G
SystemMaxFileSize=128M
SystemMaxFiles=100

# Runtime journal limits
RuntimeMaxUse=200M
RuntimeKeepFree=500M
RuntimeMaxFileSize=32M
RuntimeMaxFiles=10

# Retention policies
MaxRetentionSec=30day
MaxFileSec=1week

Optimize performance settings

Configure journal performance parameters to reduce system impact and improve responsiveness.

# Performance optimization
[Journal]
SyncIntervalSec=5m
RateLimitIntervalSec=30s
RateLimitBurst=10000

# Forward to syslog for compatibility
ForwardToSyslog=no
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=yes

# TTY settings
TTYPath=/dev/console
MaxLevelStore=info
MaxLevelSyslog=info
MaxLevelKMsg=notice
MaxLevelConsole=info
MaxLevelWall=emerg

Set up log forwarding to rsyslog

Configure journal forwarding to rsyslog for centralized logging and external log management systems.

sudo apt update
sudo apt install -y rsyslog

sudo dnf install -y rsyslog

Configure rsyslog for journal forwarding

Set up rsyslog to receive and process journal messages with proper filtering and formatting.

# Enable systemd journal module
module(load="imjournal" StateFile="imjournal.state")

# Journal input configuration
input(type="imjournal" 
      PersistStateInterval="100"
      StateFile="/var/spool/rsyslog/imjournal.state")

# Forward high-priority messages to separate files
*.emerg                         /var/log/emergency.log
*.alert                         /var/log/alert.log
*.crit                          /var/log/critical.log
*.err                           /var/log/error.log
*.warning                       /var/log/warning.log
*.info                          /var/log/info.log

Enable journal forwarding to rsyslog

Update journald configuration to forward messages to rsyslog while maintaining journal storage.

# Add forwarding configuration
[Journal]
ForwardToSyslog=yes
Storage=persistent

Set up log filtering and routing

Configure advanced filtering to route different log types to appropriate destinations and reduce noise.

# Filter systemd messages by priority
:programname, isequal, "systemd" {
    if $syslogseverity <= 3 then {
        /var/log/systemd-critical.log
        stop
    }
    if $syslogseverity <= 6 then {
        /var/log/systemd-info.log
        stop
    }
}

# Filter application logs
:programname, startswith, "nginx" /var/log/nginx-systemd.log
:programname, startswith, "apache" /var/log/apache-systemd.log
:programname, startswith, "mysql" /var/log/mysql-systemd.log
:programname, startswith, "postgresql" /var/log/postgresql-systemd.log

# Rate limiting for noisy applications
:programname, isequal, "dbus" ~
:programname, isequal, "NetworkManager" ~

Configure log rotation for journal files

Set up proper log rotation to manage journal file growth and maintain system performance.

/var/log/journal/*/*.journal {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 0640 systemd-journal systemd-journal
    postrotate
        systemctl reload systemd-journald
    endscript
}

Apply configuration changes

Restart journald and rsyslog services to apply the new configuration settings.

sudo systemctl restart systemd-journald
sudo systemctl restart rsyslog
sudo systemctl enable rsyslog

Set up journal monitoring script

Create a monitoring script to track journal health and performance metrics.

#!/bin/bash

# Journal monitoring script
LOG_FILE="/var/log/journal-monitor.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')

echo "[$DATE] Journal Health Check" >> $LOG_FILE

# Check disk usage
USAGE=$(journalctl --disk-usage | grep -oP 'Archived and active journals take up \K[0-9.]+[A-Z]+')
echo "[$DATE] Disk usage: $USAGE" >> $LOG_FILE

# Check for corrupted journals
CORRUPT=$(journalctl --verify 2>&1 | grep -c "FAIL")
if [ $CORRUPT -gt 0 ]; then
    echo "[$DATE] WARNING: $CORRUPT corrupted journal files found" >> $LOG_FILE
fi

# Check service status
SERVICE_STATUS=$(systemctl is-active systemd-journald)
echo "[$DATE] Service status: $SERVICE_STATUS" >> $LOG_FILE

# Check recent error count
ERROR_COUNT=$(journalctl --since "1 hour ago" --priority=err --lines=0 | grep -c "^-- Logs")
echo "[$DATE] Errors in last hour: $ERROR_COUNT" >> $LOG_FILE

# Alert if disk usage exceeds 800MB
USAGE_MB=$(echo $USAGE | sed 's/G/*1024/g; s/M//g' | bc 2>/dev/null || echo 0)
if [ ${USAGE_MB%.*} -gt 800 ]; then
    logger -p user.warning "Journal disk usage high: $USAGE"
fi

Make monitoring script executable and schedule

Set proper permissions and schedule the monitoring script to run regularly via cron.

sudo chmod 755 /usr/local/bin/journal-monitor.sh

# Add to crontab for hourly monitoring
echo "0 * * * * /usr/local/bin/journal-monitor.sh" | sudo crontab -

Configure journal cleanup automation

Set up automated journal cleanup to maintain optimal performance and storage usage.

#!/bin/bash

# Journal cleanup script
LOG_FILE="/var/log/journal-cleanup.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')

echo "[$DATE] Starting journal cleanup" >> $LOG_FILE

# Remove journals older than 30 days
journalctl --vacuum-time=30d >> $LOG_FILE 2>&1

# Limit journal size to 800MB
journalctl --vacuum-size=800M >> $LOG_FILE 2>&1

# Keep only 50 journal files
journalctl --vacuum-files=50 >> $LOG_FILE 2>&1

# Verify journal integrity after cleanup
journalctl --verify >> $LOG_FILE 2>&1

echo "[$DATE] Journal cleanup completed" >> $LOG_FILE

Schedule automated cleanup

Configure the cleanup script to run weekly and set proper permissions.

sudo chmod 755 /usr/local/bin/journal-cleanup.sh

# Schedule weekly cleanup on Sundays at 2 AM
echo "0 2 * * 0 /usr/local/bin/journal-cleanup.sh" | sudo crontab -l | { cat; echo "0 2 * * 0 /usr/local/bin/journal-cleanup.sh"; } | sudo crontab -