Configure Linux disk usage monitoring and automated cleanup with systemd timers

Update system packages

Start by updating your package manager to ensure you have the latest system tools and utilities.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install monitoring and mail utilities

Install the necessary packages for disk monitoring, email notifications, and system utilities.

sudo apt install -y mailutils postfix logrotate ncdu tree

sudo dnf install -y mailx postfix logrotate ncdu tree

Create disk monitoring script

Create a script that checks disk usage and sends email alerts when thresholds are exceeded.

sudo mkdir -p /opt/disk-monitor
sudo tee /opt/disk-monitor/disk-check.sh > /dev/null << 'EOF'
#!/bin/bash

# Configuration
THRESHOLD_WARNING=80
THRESHOLD_CRITICAL=90
EMAIL_RECIPIENT="admin@example.com"
HOSTNAME=$(hostname -f)
LOG_FILE="/var/log/disk-monitor.log"

# Function to log messages
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}

# Function to send email alert
send_alert() {
    local severity=$1
    local filesystem=$2
    local usage=$3
    local available=$4
    
    local subject="[$severity] Disk Space Alert - $HOSTNAME"
    local body="Disk space alert for $HOSTNAME:

Filesystem: $filesystem
Usage: $usage%
Available: $available
Threshold: ${severity,,} at ${THRESHOLD_WARNING}%/${THRESHOLD_CRITICAL}%

Please investigate and free up disk space immediately."
    
    echo -e "$body" | mail -s "$subject" "$EMAIL_RECIPIENT"
    log_message "$severity alert sent for $filesystem ($usage% used)"
}

# Check disk usage for all mounted filesystems
df -h | awk 'NR>1 && !/tmpfs|devtmpfs|udev/ {print $5 " " $6 " " $4}' | while read output; do
    usage=$(echo $output | awk '{print $1}' | sed 's/%//')
    filesystem=$(echo $output | awk '{print $2}')
    available=$(echo $output | awk '{print $3}')
    
    if [ $usage -ge $THRESHOLD_CRITICAL ]; then
        send_alert "CRITICAL" "$filesystem" "$usage" "$available"
    elif [ $usage -ge $THRESHOLD_WARNING ]; then
        send_alert "WARNING" "$filesystem" "$usage" "$available"
    fi
done

# Log successful completion
log_message "Disk check completed successfully"
EOF

Make the monitoring script executable

Set proper permissions on the disk monitoring script to allow execution by the system.

sudo chmod 755 /opt/disk-monitor/disk-check.sh
sudo chown root:root /opt/disk-monitor/disk-check.sh

Create cleanup script for temporary files

Create a script to automatically clean up temporary files, old logs, and cache directories.

sudo tee /opt/disk-monitor/cleanup.sh > /dev/null << 'EOF'
#!/bin/bash

LOG_FILE="/var/log/disk-cleanup.log"
CLEANED_SPACE=0

# Function to log messages with space saved
log_cleanup() {
    local action=$1
    local space_before=$2
    local space_after=$3
    local saved=$((space_before - space_after))
    CLEANED_SPACE=$((CLEANED_SPACE + saved))
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $action: ${saved}KB freed" >> "$LOG_FILE"
}

# Function to get directory size in KB
get_size() {
    du -sk "$1" 2>/dev/null | cut -f1 || echo 0
}

echo "$(date '+%Y-%m-%d %H:%M:%S') - Starting cleanup process" >> "$LOG_FILE"

# Clean temporary directories
for temp_dir in "/tmp" "/var/tmp"; do
    if [ -d "$temp_dir" ]; then
        before=$(get_size "$temp_dir")
        find "$temp_dir" -type f -atime +7 -delete 2>/dev/null
        find "$temp_dir" -type d -empty -delete 2>/dev/null
        after=$(get_size "$temp_dir")
        log_cleanup "Cleaned $temp_dir" "$before" "$after"
    fi
done

# Clean old log files (older than 30 days)
if [ -d "/var/log" ]; then
    before=$(get_size "/var/log")
    find /var/log -name "*.log.*.gz" -mtime +30 -delete 2>/dev/null
    find /var/log -name "*.log.*" -mtime +30 -delete 2>/dev/null
    after=$(get_size "/var/log")
    log_cleanup "Cleaned old logs" "$before" "$after"
fi

# Clean package cache
if command -v apt-get >/dev/null 2>&1; then
    before=$(get_size "/var/cache/apt")
    apt-get clean >/dev/null 2>&1
    after=$(get_size "/var/cache/apt")
    log_cleanup "APT cache cleanup" "$before" "$after"
elif command -v dnf >/dev/null 2>&1; then
    before=$(get_size "/var/cache/dnf")
    dnf clean all >/dev/null 2>&1
    after=$(get_size "/var/cache/dnf")
    log_cleanup "DNF cache cleanup" "$before" "$after"
fi

# Clean journal logs older than 30 days
before=$(journalctl --disk-usage 2>/dev/null | grep -oE '[0-9.]+[KMGT]B' | head -1 | sed 's/[^0-9.]//g' || echo 0)
journalctl --vacuum-time=30d >/dev/null 2>&1
after=$(journalctl --disk-usage 2>/dev/null | grep -oE '[0-9.]+[KMGT]B' | head -1 | sed 's/[^0-9.]//g' || echo 0)
log_cleanup "Journal cleanup" "$before" "$after"

echo "$(date '+%Y-%m-%d %H:%M:%S') - Cleanup completed. Total space freed: ${CLEANED_SPACE}KB" >> "$LOG_FILE"
EOF

Make cleanup script executable

Set proper permissions on the cleanup script and ensure it's owned by root for security.

sudo chmod 755 /opt/disk-monitor/cleanup.sh
sudo chown root:root /opt/disk-monitor/cleanup.sh

Create systemd service for disk monitoring

Create a systemd service unit that will run the disk monitoring script.

[Unit]
Description=Disk Usage Monitor
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=root
Group=root
ExecStart=/opt/disk-monitor/disk-check.sh
StandardOutput=journal
StandardError=journal

Create systemd service for cleanup

Create a systemd service unit for the automated cleanup tasks.

[Unit]
Description=Disk Cleanup Service
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=root
Group=root
ExecStart=/opt/disk-monitor/cleanup.sh
StandardOutput=journal
StandardError=journal

Create systemd timers

Create timer units to schedule regular execution of the monitoring and cleanup services.

[Unit]
Description=Run disk monitor every 15 minutes
Requires=disk-monitor.service

[Timer]
OnBootSec=5min
OnUnitActiveSec=15min
Persistent=true

[Install]
WantedBy=timers.target

[Unit]
Description=Run disk cleanup daily at 2 AM
Requires=disk-cleanup.service

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Configure enhanced log rotation

Set up comprehensive log rotation to prevent log files from consuming excessive disk space.

# Disk monitor logs
/var/log/disk-monitor.log {
    weekly
    missingok
    rotate 12
    compress
    delaycompress
    notifempty
    create 644 root root
}

/var/log/disk-cleanup.log {
    weekly
    missingok
    rotate 12
    compress
    delaycompress
    notifempty
    create 644 root root
}

# Enhanced system log rotation
/var/log/syslog {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 644 syslog adm
    postrotate
        systemctl reload rsyslog
    endscript
}

/var/log/auth.log {
    weekly
    missingok
    rotate 8
    compress
    delaycompress
    notifempty
    create 644 syslog adm
    postrotate
        systemctl reload rsyslog
    endscript
}

Configure Postfix for email notifications

Set up basic Postfix configuration for sending email alerts. This configuration works for most cloud providers.

sudo debconf-set-selections <<< "postfix postfix/mailname string $(hostname -f)"
sudo debconf-set-selections <<< "postfix postfix/main_mailer_type string 'Internet Site'"

# Add or modify these settings in /etc/postfix/main.cf
sudo postconf -e "myhostname = $(hostname -f)"
sudo postconf -e "mydestination = \$myhostname, localhost.\$mydomain, localhost"
sudo postconf -e "relayhost = "
sudo postconf -e "mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128"
sudo postconf -e "inet_protocols = ipv4"

Enable and start services

Reload systemd configuration and enable the timer services to start automatically.

sudo systemctl daemon-reload
sudo systemctl enable --now disk-monitor.timer
sudo systemctl enable --now disk-cleanup.timer
sudo systemctl enable --now postfix

Create log directories and initial files

Ensure proper log directories exist with correct permissions for monitoring and cleanup scripts.

sudo touch /var/log/disk-monitor.log /var/log/disk-cleanup.log
sudo chmod 644 /var/log/disk-monitor.log /var/log/disk-cleanup.log
sudo chown root:root /var/log/disk-monitor.log /var/log/disk-cleanup.log

Add filesystem-specific monitoring

Create custom thresholds for specific filesystems that may need different monitoring levels.

# Custom thresholds per filesystem
# Format: filesystem:warning_threshold:critical_threshold
/:85:95
/var:80:90
/home:75:85
/tmp:90:95

Configure logrotate for application logs

Add rotation rules for common application log directories to prevent them from filling the disk.

# Nginx logs
/var/log/nginx/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 www-data adm
    sharedscripts
    postrotate
        systemctl reload nginx
    endscript
}

# Apache logs
/var/log/apache2/*.log {
    weekly
    missingok
    rotate 12
    compress
    delaycompress
    notifempty
    create 644 www-data adm
    sharedscripts
    postrotate
        systemctl reload apache2
    endscript
}