Fix high load average with low CPU usage on web servers

Check current load average and CPU usage

View your system's current load and CPU statistics to confirm the issue.

uptime
top -n 1 | head -5
cat /proc/loadavg

Install diagnostic tools

Install system monitoring tools to analyze process behavior and resource usage.

sudo apt update
sudo apt install -y htop iotop sysstat lsof

sudo dnf install -y htop iotop sysstat lsof

Identify I/O wait processes

Check if processes are waiting for disk I/O operations, which commonly causes high load with low CPU usage.

iostat -x 1 5
iotop -o

Check for blocked processes

Examine processes in uninterruptible sleep state (D state) that contribute to load average.

ps auxf | grep " D "
ps -eo pid,ppid,state,comm --sort=-pid | grep " D "

Monitor disk utilization

Analyze disk performance metrics to identify storage bottlenecks.

df -h
du -sh /var/log/* | sort -rh | head -10
lsof +L1

Examine network I/O waiting

Check if processes are waiting for network operations that could increase load average.

netstat -i
ss -tuln
lsof -i

Check for zombie processes

Identify zombie processes that might contribute to system load issues.

ps aux | grep ""
ps -eo pid,ppid,state,comm | grep " Z "

Optimize disk I/O performance

Adjust I/O scheduler and mount options for better disk performance. Modern SSDs benefit from noop or mq-deadline schedulers.

cat /sys/block/sda/queue/scheduler
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

Clear large log files

Truncate large log files that might be causing disk I/O bottlenecks without stopping services.

sudo find /var/log -name "*.log" -size +100M -ls
sudo truncate -s 0 /var/log/large-application.log
sudo systemctl reload rsyslog

Increase file descriptor limits

Raise system limits for file descriptors that might cause processes to wait for resources.

* soft nofile 65536
* hard nofile 65536
root soft nofile 65536
root hard nofile 65536

Configure kernel parameters for high load scenarios

Adjust kernel parameters to handle high concurrent connections and I/O operations better.

vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
vm.vfs_cache_pressure = 50
fs.file-max = 2097152

Restart hung processes

Identify and restart processes stuck in uninterruptible sleep state that cannot be killed normally.

sudo systemctl restart apache2
sudo systemctl restart nginx
sudo systemctl restart mysql

Optimize Apache/Nginx worker processes

Adjust web server configuration to handle concurrent connections more efficiently.

worker_processes auto;
worker_connections 1024;
keepalive_timeout 30;
client_max_body_size 10m;

Configure database connection pooling

Reduce database connection overhead that can cause processes to wait for available connections.

max_connections = 200
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 64M

Enable web server caching

Implement caching to reduce disk I/O from repeated file access.

location ~* \.(jpg|jpeg|png|gif|css|js)$ {
    expires 30d;
    add_header Cache-Control "public, immutable";
}

Set up continuous monitoring

Install monitoring tools to track load average trends and get early warnings.

sudo crontab -e

/5    * uptime >> /var/log/load-average.log

Create load average alerts

Set up email alerts when load average exceeds acceptable thresholds.

#!/bin/bash
LOAD=$(uptime | awk '{print $12}' | cut -d',' -f1)
THRESHOLD=4.0
if (( $(echo "$LOAD > $THRESHOLD" | bc -l) )); then
    echo "High load average: $LOAD" | mail -s "Load Alert" admin@example.com
fi

Implement log rotation

Configure automatic log rotation to prevent large log files from causing I/O bottlenecks.

/var/log/myapp/.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 644 www-data www-data
    postrotate
        systemctl reload myapp
    endscript
}