Configure Linux system performance monitoring with sar and sysstat for real-time metrics collection

Update system packages

Start by updating your package manager to ensure you get the latest versions of monitoring tools.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install sysstat package

Install the sysstat package which includes sar, sadc, iostat, mpstat, and other performance monitoring utilities.

sudo apt install -y sysstat

sudo dnf install -y sysstat

Enable sysstat data collection

By default, sysstat data collection is often disabled. Enable it by editing the configuration file to start automatic performance data gathering.

ENABLED="true"

For RHEL-based systems, the configuration is slightly different:

HISTORY=28
COMPRESS_AFTER=31

Configure data collection intervals

Configure how frequently performance data is collected by editing the crontab entries. The default collection runs every 10 minutes, but you can adjust this for more granular monitoring.

sudo crontab -e

Add or modify these cron entries for different collection intervals:

# Collect system activity data every 2 minutes
*/2 * * * * /usr/lib/sysstat/sa1

# Generate daily reports at 23:53
53 23 * * * /usr/lib/sysstat/sa2

Customize sysstat configuration

Configure advanced sysstat options including data retention, compression, and specific metrics to collect.

# Data retention period (days)
HISTORY=30

# Compress data files older than this many days
COMPRESS_AFTER=7

# Enable specific data collectors
SADC_OPTIONS="-S DISK -S INT -S IPV6"

# Directory for data files
SA_DIR=/var/log/sysstat

Start and enable sysstat services

Enable the sysstat service to ensure data collection starts automatically on boot and verify it's running correctly.

sudo systemctl enable sysstat
sudo systemctl start sysstat
sudo systemctl status sysstat

Configure log rotation

Set up log rotation to manage disk space usage from performance data files, which can grow large over time.

/var/log/sysstat/sa* {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 644 root root
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Create daily performance report script

Create an automated script that generates comprehensive daily performance reports and can email them to administrators.

#!/bin/bash

# Daily performance report generator
DATE=$(date -d "1 day ago" +%d)
HOSTNAME=$(hostname)
REPORT_FILE="/tmp/perf-report-$(date +%Y%m%d).txt"

echo "Daily Performance Report for $HOSTNAME - $(date -d '1 day ago' +%Y-%m-%d)" > $REPORT_FILE
echo "=========================================================" >> $REPORT_FILE

# CPU usage summary
echo "CPU Usage Summary:" >> $REPORT_FILE
sar -u -f /var/log/sysstat/sa$DATE | tail -1 >> $REPORT_FILE
echo "" >> $REPORT_FILE

# Memory usage summary
echo "Memory Usage Summary:" >> $REPORT_FILE
sar -r -f /var/log/sysstat/sa$DATE | tail -1 >> $REPORT_FILE
echo "" >> $REPORT_FILE

# Disk I/O summary
echo "Disk I/O Summary:" >> $REPORT_FILE
sar -b -f /var/log/sysstat/sa$DATE | tail -1 >> $REPORT_FILE
echo "" >> $REPORT_FILE

# Network traffic summary
echo "Network Traffic Summary:" >> $REPORT_FILE
sar -n DEV -f /var/log/sysstat/sa$DATE | grep -E "(IFACE|eth0|ens|eno)" | tail -2 >> $REPORT_FILE
echo "" >> $REPORT_FILE

# Load average summary
echo "Load Average Summary:" >> $REPORT_FILE
sar -q -f /var/log/sysstat/sa$DATE | tail -1 >> $REPORT_FILE

# Optional: Email the report
# mail -s "Daily Performance Report - $HOSTNAME" admin@example.com < $REPORT_FILE

# Keep reports for 7 days
find /tmp -name "perf-report-*.txt" -mtime +7 -delete

Make the script executable:

sudo chmod 755 /usr/local/bin/daily-perf-report.sh

Schedule automated reporting

Add the report script to cron to run automatically every morning, providing yesterday's performance summary.

sudo crontab -e

Add this entry to generate reports at 6 AM daily:

# Generate daily performance report at 6 AM
0 6 * * * /usr/local/bin/daily-perf-report.sh

Create real-time monitoring script

Build a script that monitors current system performance and triggers alerts when thresholds are exceeded.

#!/bin/bash

# Real-time performance monitoring with alerts
HOSTNAME=$(hostname)
ALERT_EMAIL="admin@example.com"
LOG_FILE="/var/log/perf-monitor.log"

# Thresholds
CPU_THRESHOLD=80
MEMORY_THRESHOLD=85
LOAD_THRESHOLD=5.0
DISK_IO_THRESHOLD=1000

# Get current metrics
CPU_USAGE=$(sar -u 1 1 | grep Average | awk '{print 100-$8}' | cut -d. -f1)
MEMORY_USAGE=$(free | grep Mem | awk '{printf "%.0f", $3/$2 * 100.0}')
LOAD_AVG=$(uptime | awk -F'load average:' '{print $2}' | awk -F, '{print $1}' | sed 's/ //g')
DISK_IO=$(iostat -x 1 2 | grep -E "(sd|nvme)" | tail -1 | awk '{print $10}' | cut -d. -f1)

echo "$(date): CPU=${CPU_USAGE}%, MEM=${MEMORY_USAGE}%, LOAD=${LOAD_AVG}, IO=${DISK_IO}" >> $LOG_FILE

# Check CPU usage
if [ "$CPU_USAGE" -gt "$CPU_THRESHOLD" ]; then
    echo "ALERT: High CPU usage: ${CPU_USAGE}% on $HOSTNAME" | logger -t perf-monitor
    # mail -s "ALERT: High CPU on $HOSTNAME" $ALERT_EMAIL <<< "CPU usage is ${CPU_USAGE}%"
fi

# Check memory usage
if [ "$MEMORY_USAGE" -gt "$MEMORY_THRESHOLD" ]; then
    echo "ALERT: High memory usage: ${MEMORY_USAGE}% on $HOSTNAME" | logger -t perf-monitor
    # mail -s "ALERT: High Memory on $HOSTNAME" $ALERT_EMAIL <<< "Memory usage is ${MEMORY_USAGE}%"
fi

# Check load average
if (( $(echo "$LOAD_AVG > $LOAD_THRESHOLD" | bc -l) )); then
    echo "ALERT: High load average: ${LOAD_AVG} on $HOSTNAME" | logger -t perf-monitor
    # mail -s "ALERT: High Load on $HOSTNAME" $ALERT_EMAIL <<< "Load average is ${LOAD_AVG}"
fi

# Check disk I/O
if [ ! -z "$DISK_IO" ] && [ "$DISK_IO" -gt "$DISK_IO_THRESHOLD" ]; then
    echo "ALERT: High disk I/O: ${DISK_IO} on $HOSTNAME" | logger -t perf-monitor
    # mail -s "ALERT: High Disk I/O on $HOSTNAME" $ALERT_EMAIL <<< "Disk I/O utilization is ${DISK_IO}%"
fi

Make the script executable and install required packages:

sudo chmod 755 /usr/local/bin/perf-monitor.sh

sudo apt install -y bc sysstat

sudo dnf install -y bc sysstat

Schedule performance monitoring

Set up the monitoring script to run every 5 minutes for continuous performance tracking and alerting.

sudo crontab -e

Add this entry for 5-minute monitoring intervals:

# Run performance monitoring every 5 minutes
*/5 * * * * /usr/local/bin/perf-monitor.sh

Create performance dashboard script

Build a script that displays current and historical performance data in an easy-to-read format for quick system assessment.

#!/bin/bash

# Performance dashboard for current system status
echo "====================================="
echo "System Performance Dashboard"
echo "Host: $(hostname)"
echo "Date: $(date)"
echo "====================================="
echo

# Current system load
echo "Current System Load:"
uptime
echo

# CPU usage (last 5 minutes)
echo "CPU Usage (last 5 minutes):"
sar -u 1 5 | grep Average
echo

# Memory usage
echo "Memory Usage:"
free -h
echo

# Disk usage
echo "Disk Usage:"
df -h | grep -E "(Filesystem|/dev/)"
echo

# Top processes by CPU
echo "Top 5 processes by CPU:"
ps aux --sort=-%cpu | head -6
echo

# Top processes by memory
echo "Top 5 processes by memory:"
ps aux --sort=-%mem | head -6
echo

# Network connections
echo "Network connections summary:"
netstat -tuln | grep LISTEN | wc -l
echo "Active listening ports: $(netstat -tuln | grep LISTEN | wc -l)"
echo

# Recent alerts from monitoring
echo "Recent performance alerts (last 24 hours):"
grep "$(date +%Y-%m-%d)" /var/log/perf-monitor.log 2>/dev/null | tail -10 || echo "No alerts found"

Make the dashboard script executable:

sudo chmod 755 /usr/local/bin/perf-dashboard.sh

Create comprehensive analysis script

Build an advanced script that analyzes historical data and identifies performance trends and anomalies.

#!/bin/bash

# Advanced performance analysis script
if [ $# -eq 0 ]; then
    echo "Usage: $0 [days_back] [report_type]"
    echo "Report types: cpu, memory, disk, network, all"
    echo "Example: $0 7 cpu"
    exit 1
fi

DAYS_BACK=${1:-1}
REPORT_TYPE=${2:-all}
DATE=$(date -d "$DAYS_BACK days ago" +%d)

echo "Performance Analysis Report - $(date -d "$DAYS_BACK days ago" +%Y-%m-%d)"
echo "================================================"

if [ "$REPORT_TYPE" = "cpu" ] || [ "$REPORT_TYPE" = "all" ]; then
    echo
    echo "CPU Analysis:"
    echo "Peak CPU usage:"
    sar -u -f /var/log/sysstat/sa$DATE | grep -v "Average\|RESTART\|Linux" | sort -k4 -nr | head -5
    echo
    echo "CPU idle time distribution:"
    sar -u -f /var/log/sysstat/sa$DATE | awk 'NR>3 && !/Average/ {idle+=$8; count++} END {if(count>0) printf "Average idle: %.2f%%\n", idle/count}'
fi

if [ "$REPORT_TYPE" = "memory" ] || [ "$REPORT_TYPE" = "all" ]; then
    echo
    echo "Memory Analysis:"
    echo "Peak memory usage:"
    sar -r -f /var/log/sysstat/sa$DATE | grep -v "Average\|RESTART\|Linux" | awk '{print $1, $4, $5}' | sort -k3 -nr | head -5
    echo
    echo "Memory utilization trend:"
    sar -r -f /var/log/sysstat/sa$DATE | awk 'NR>3 && !/Average/ {used+=$4; count++} END {if(count>0) printf "Average memory used: %.0f MB\n", used/count}'
fi

if [ "$REPORT_TYPE" = "disk" ] || [ "$REPORT_TYPE" = "all" ]; then
    echo
    echo "Disk I/O Analysis:"
    echo "Peak disk activity:"
    sar -b -f /var/log/sysstat/sa$DATE | grep -v "Average\|RESTART\|Linux" | sort -k2 -nr | head -5
fi

if [ "$REPORT_TYPE" = "network" ] || [ "$REPORT_TYPE" = "all" ]; then
    echo
    echo "Network Analysis:"
    echo "Peak network activity:"
    sar -n DEV -f /var/log/sysstat/sa$DATE | grep -E "eth0|ens|eno" | grep -v "Average" | sort -k5 -nr | head -5
fi

Make the analysis script executable:

sudo chmod 755 /usr/local/bin/perf-analysis.sh