Learn to optimize Linux process performance by configuring CPU scheduling priorities, CPU affinity, and systemd resource management for better workload distribution and reduced context switching.
Prerequisites
- Root or sudo access
- Basic understanding of Linux processes
- Multi-core CPU system
What this solves
Linux process scheduling and CPU affinity control how processes are assigned to CPU cores and their execution priority. By default, the Linux scheduler distributes processes across all available CPU cores, but this can lead to cache misses and suboptimal performance for specific workloads. Configuring CPU affinity pins processes to specific cores, while scheduling priorities ensure critical applications get the resources they need.
Understanding Linux process scheduling and CPU affinity
Process scheduling basics
The Linux kernel uses the Completely Fair Scheduler (CFS) to manage process execution. Each process has a nice value (-20 to 19) that affects its priority, with lower values giving higher priority. The scheduler also considers CPU affinity, which determines which CPU cores a process can use.
CPU affinity concepts
CPU affinity binds processes to specific CPU cores, reducing cache misses and improving performance for CPU-intensive applications. This is particularly useful for real-time applications, database servers, and high-frequency trading systems where consistent performance matters more than load balancing.
Check current system topology
First, understand your system's CPU layout and current process distribution.
lscpu
nproc
cat /proc/cpuinfo | grep processor | wc -l
View current process scheduling information
Check current nice values and CPU affinity for running processes.
ps -eo pid,ppid,cmd,nice,psr --sort=-pcpu | head -20
top -p 1 -n 1 -b | grep "Cpu(s)"
Configure CPU scheduling with systemd and nice priorities
Install scheduling tools
Install the necessary tools for CPU affinity and process scheduling management.
sudo apt update
sudo apt install -y util-linux schedtool htop numactl
Configure systemd service CPU scheduling
Create a systemd service configuration that demonstrates CPU affinity and nice value settings.
[Unit]
Description=High Priority Application
After=network.target
[Service]
Type=simple
User=appuser
Group=appgroup
ExecStart=/usr/local/bin/my-app
CPU Scheduling Configuration
Nice=-10
CPUAffinity=0 1
CPUSchedulingPolicy=1
CPUSchedulingPriority=50
Resource Limits
MemoryMax=2G
CPUQuota=150%
Security
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/log/my-app
[Install]
WantedBy=multi-user.target
Create systemd drop-in configuration for existing services
Configure CPU affinity for existing services like nginx without modifying the main service file.
sudo mkdir -p /etc/systemd/system/nginx.service.d
[Service]
Bind nginx to cores 2-7, leaving 0-1 for system tasks
CPUAffinity=2 3 4 5 6 7
Nice=-5
Limit CPU usage to prevent resource starvation
CPUQuota=400%
MemoryMax=1G
Configure cgroup CPU scheduling
Create custom cgroups for different application classes with specific CPU scheduling parameters.
sudo mkdir -p /sys/fs/cgroup/cpu/high-priority
sudo mkdir -p /sys/fs/cgroup/cpu/low-priority
# Set high priority group to use more CPU time
sudo echo 2048 > /sys/fs/cgroup/cpu/high-priority/cpu.shares
sudo echo 512 > /sys/fs/cgroup/cpu/low-priority/cpu.shares
Set CPU quota (100% = 1 core)
sudo echo 200000 > /sys/fs/cgroup/cpu/high-priority/cpu.cfs_quota_us
sudo echo 100000 > /sys/fs/cgroup/cpu/high-priority/cpu.cfs_period_us
Reload systemd and apply changes
Reload systemd configuration and restart services to apply the new CPU scheduling settings.
sudo systemctl daemon-reload
sudo systemctl restart nginx
sudo systemctl status nginx
Implement CPU affinity with taskset and cpuset
Use taskset for running processes
Apply CPU affinity to running processes using taskset command for immediate changes.
# Check current CPU affinity for a process (replace PID)
taskset -p 1234
Set CPU affinity to cores 0,1 for process 1234
sudo taskset -cp 0,1 1234
Start a new process with specific CPU affinity
taskset -c 2,3 /usr/bin/my-application
Configure persistent CPU affinity with cpuset
Create cpuset configurations for persistent CPU isolation across system reboots.
sudo mkdir -p /sys/fs/cgroup/cpuset/database
sudo mkdir -p /sys/fs/cgroup/cpuset/webserver
sudo mkdir -p /sys/fs/cgroup/cpuset/background
Configure cpuset parameters
Set up CPU and memory node assignments for each cpuset group.
# Configure database cpuset (cores 0-3)
sudo echo 0-3 > /sys/fs/cgroup/cpuset/database/cpuset.cpus
sudo echo 0 > /sys/fs/cgroup/cpuset/database/cpuset.mems
sudo echo 1 > /sys/fs/cgroup/cpuset/database/cpuset.cpu_exclusive
Configure webserver cpuset (cores 4-7)
sudo echo 4-7 > /sys/fs/cgroup/cpuset/webserver/cpuset.cpus
sudo echo 0 > /sys/fs/cgroup/cpuset/webserver/cpuset.mems
Configure background tasks (cores 6-7, shared)
sudo echo 6-7 > /sys/fs/cgroup/cpuset/background/cpuset.cpus
sudo echo 0 > /sys/fs/cgroup/cpuset/background/cpuset.mems
Create systemd slice configurations
Use systemd slices to manage cpuset assignments automatically.
[Unit]
Description=Database Services Slice
Before=slices.target
[Slice]
CPUAccounting=true
MemoryAccounting=true
AllowedCPUs=0-3
AllowedMemoryNodes=0
Create a script for dynamic CPU affinity management
Create a script to dynamically adjust CPU affinity based on system load.
#!/bin/bash
Dynamic CPU Affinity Manager
Adjusts process CPU affinity based on load
LOAD_THRESHOLD=2.0
HIGH_PRIORITY_PROCESSES=("mysql" "postgres" "nginx")
LOW_PRIORITY_CORES="6,7"
HIGH_PRIORITY_CORES="0-5"
Get current load average
LOAD_AVG=$(uptime | awk '{print $(NF-2)}' | sed 's/,//')
Function to set CPU affinity
set_affinity() {
local process_name=$1
local cores=$2
for pid in $(pgrep "$process_name"); do
if [ -n "$pid" ]; then
taskset -cp "$cores" "$pid" 2>/dev/null
echo "Set $process_name (PID: $pid) to cores $cores"
fi
done
}
Adjust based on load
if (( $(echo "$LOAD_AVG > $LOAD_THRESHOLD" | bc -l) )); then
echo "High load detected ($LOAD_AVG), isolating critical processes"
for process in "${HIGH_PRIORITY_PROCESSES[@]}"; do
set_affinity "$process" "$HIGH_PRIORITY_CORES"
done
else
echo "Normal load ($LOAD_AVG), allowing balanced scheduling"
for process in "${HIGH_PRIORITY_PROCESSES[@]}"; do
set_affinity "$process" "0-7"
done
fi
Make the script executable and create systemd timer
Set up the CPU affinity manager to run periodically.
sudo chmod +x /usr/local/bin/cpu-affinity-manager.sh
[Unit]
Description=CPU Affinity Manager
After=multi-user.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/cpu-affinity-manager.sh
User=root
[Unit]
Description=Run CPU Affinity Manager every 5 minutes
Requires=cpu-affinity-manager.service
[Timer]
OnBootSec=5min
OnUnitActiveSec=5min
Unit=cpu-affinity-manager.service
[Install]
WantedBy=timers.target
Enable and start the CPU affinity manager
Activate the automated CPU affinity management system.
sudo systemctl daemon-reload
sudo systemctl enable --now cpu-affinity-manager.timer
sudo systemctl status cpu-affinity-manager.timer
Monitor and optimize process performance
Install performance monitoring tools
Set up tools for monitoring CPU scheduling and affinity effectiveness.
sudo apt install -y sysstat iotop atop perf-tools-unstable
Create performance monitoring script
Build a comprehensive monitoring script to track CPU scheduling effectiveness.
#!/bin/bash
CPU Performance Monitor
Tracks process scheduling and CPU affinity effectiveness
LOG_FILE="/var/log/cpu-performance.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')
echo "=== CPU Performance Report - $DATE ===" >> $LOG_FILE
CPU utilization per core
echo "CPU Utilization per Core:" >> $LOG_FILE
mpstat -P ALL 1 1 | tail -n +4 >> $LOG_FILE
Process CPU affinity and usage
echo "\nTop CPU consumers with affinity:" >> $LOG_FILE
ps -eo pid,ppid,cmd,nice,psr,%cpu --sort=-%cpu | head -20 | while read line; do
pid=$(echo $line | awk '{print $1}')
if [[ $pid =~ ^[0-9]+$ ]]; then
affinity=$(taskset -p $pid 2>/dev/null | awk '{print $NF}')
echo "$line (Affinity: $affinity)" >> $LOG_FILE
else
echo "$line" >> $LOG_FILE
fi
done
Context switches and interrupts
echo "\nContext Switches and Interrupts:" >> $LOG_FILE
vmstat 1 5 | tail -1 >> $LOG_FILE
Load average
echo "\nLoad Average:" >> $LOG_FILE
uptime >> $LOG_FILE
echo "\n" >> $LOG_FILE
Set up performance monitoring with systemd
Configure automated performance monitoring and logging.
sudo chmod +x /usr/local/bin/cpu-performance-monitor.sh
sudo touch /var/log/cpu-performance.log
sudo chown root:adm /var/log/cpu-performance.log
sudo chmod 644 /var/log/cpu-performance.log
[Unit]
Description=CPU Performance Monitor
After=multi-user.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/cpu-performance-monitor.sh
User=root
[Unit]
Description=Run CPU Performance Monitor every 10 minutes
Requires=cpu-performance-monitor.service
[Timer]
OnBootSec=10min
OnUnitActiveSec=10min
Unit=cpu-performance-monitor.service
[Install]
WantedBy=timers.target
Enable performance monitoring
Start the automated performance monitoring system.
sudo systemctl enable --now cpu-performance-monitor.timer
sudo systemctl status cpu-performance-monitor.timer
Configure log rotation for performance logs
Set up log rotation to prevent performance logs from consuming too much disk space.
/var/log/cpu-performance.log {
daily
rotate 7
compress
delaycompress
missingok
create 644 root adm
postrotate
systemctl reload-or-restart rsyslog > /dev/null 2>&1 || true
endscript
}
Verify your setup
Check that CPU scheduling and affinity configurations are working correctly.
# Verify systemd service CPU affinity
systemctl show nginx | grep CPUAffinity
Check current process affinity
ps -eo pid,cmd,psr | grep nginx
taskset -p $(pgrep nginx | head -1)
Monitor CPU usage per core
watch -n 2 'mpstat -P ALL 1 1 | tail -n +4'
Check cgroup assignments
cat /proc/$(pgrep nginx | head -1)/cgroup
View performance monitoring results
sudo tail -f /var/log/cpu-performance.log
Test the CPU affinity manager
sudo /usr/local/bin/cpu-affinity-manager.sh
Common issues
| Symptom | Cause | Fix |
|---|---|---|
| Taskset command fails with "Invalid argument" | CPU core doesn't exist or wrong syntax | Check lscpu output and use correct core numbers |
| Systemd service ignores CPUAffinity | systemd version doesn't support CPU affinity | Use ExecStart with taskset wrapper or update systemd |
| High context switching despite CPU affinity | Processes competing for same resources | Review memory locality and adjust affinity groups |
| Performance degradation after setting affinity | Too restrictive CPU binding | Allow more cores or use CPU sets instead of strict binding |
| Nice values not taking effect | Process started before systemd configuration | Restart services after changing nice values |
| Cgroup cpuset not working | cpuset controller not enabled | Enable cpuset in kernel command line: cgroup_enable=cpuset |
Next steps
- Configure NUMA optimization for multi-socket servers
- Advanced memory cgroups with systemd
- Comprehensive Linux performance tuning
- Process resource monitoring with cgroups
- Kubernetes resource management for containerized workloads
Running this in production?
Automated install script
Run this to automate the entire setup
#!/usr/bin/env bash
set -euo pipefail
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
# Global variables
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BACKUP_DIR="/root/cpu-scheduling-backup-$(date +%Y%m%d-%H%M%S)"
# Usage function
usage() {
cat << EOF
Usage: $0 [OPTIONS]
Configure Linux process scheduling and CPU affinity for performance optimization
OPTIONS:
-c, --cores CORES CPU cores to assign for high-priority processes (default: 2-3)
-s, --system-cores CPU cores reserved for system processes (default: 0-1)
-h, --help Show this help message
Examples:
$0 # Use default CPU affinity settings
$0 -c "4-7" -s "0-3" # Custom core assignments
EOF
}
# Logging functions
log_info() { echo -e "${GREEN}[INFO]${NC} $1" >&2; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1" >&2; }
log_error() { echo -e "${RED}[ERROR]${NC} $1" >&2; }
# Cleanup function
cleanup() {
if [ -n "${BACKUP_DIR:-}" ] && [ -d "$BACKUP_DIR" ]; then
log_warn "Script failed. Backup files available in: $BACKUP_DIR"
log_warn "To restore: cp -r $BACKUP_DIR/* /"
fi
}
trap cleanup ERR
# Check if running as root
check_root() {
if [[ $EUID -ne 0 ]]; then
log_error "This script must be run as root"
exit 1
fi
}
# Detect distribution
detect_distro() {
if [ -f /etc/os-release ]; then
. /etc/os-release
case "$ID" in
ubuntu|debian)
PKG_MGR="apt"
PKG_INSTALL="apt install -y"
PKG_UPDATE="apt update"
;;
almalinux|rocky|centos|rhel|ol|fedora)
PKG_MGR="dnf"
PKG_INSTALL="dnf install -y"
PKG_UPDATE="dnf check-update || true"
;;
amzn)
PKG_MGR="yum"
PKG_INSTALL="yum install -y"
PKG_UPDATE="yum check-update || true"
;;
*)
log_error "Unsupported distribution: $ID"
exit 1
;;
esac
export PKG_MGR PKG_INSTALL PKG_UPDATE
log_info "Detected distribution: $ID"
else
log_error "Cannot detect distribution - /etc/os-release not found"
exit 1
fi
}
# Parse command line arguments
parse_args() {
HIGH_PRIORITY_CORES="2-3"
SYSTEM_CORES="0-1"
while [[ $# -gt 0 ]]; do
case $1 in
-c|--cores)
HIGH_PRIORITY_CORES="$2"
shift 2
;;
-s|--system-cores)
SYSTEM_CORES="$2"
shift 2
;;
-h|--help)
usage
exit 0
;;
*)
log_error "Unknown option: $1"
usage
exit 1
;;
esac
done
}
# Create backup directory
create_backup() {
mkdir -p "$BACKUP_DIR"
log_info "Created backup directory: $BACKUP_DIR"
}
# Install required packages
install_packages() {
echo "[1/8] Installing CPU scheduling and monitoring tools..."
$PKG_UPDATE > /dev/null 2>&1
case "$PKG_MGR" in
apt)
$PKG_INSTALL util-linux schedtool htop numactl cgroup-tools
;;
dnf|yum)
$PKG_INSTALL util-linux schedutils htop numactl libcgroup-tools
;;
esac
log_info "Required packages installed successfully"
}
# Display system topology
show_system_info() {
echo "[2/8] Analyzing system CPU topology..."
log_info "CPU Information:"
echo "Total CPU cores: $(nproc)"
echo "Architecture: $(lscpu | grep Architecture | awk '{print $2}')"
echo "CPU(s): $(lscpu | grep '^CPU(s):' | awk '{print $2}')"
echo "Thread(s) per core: $(lscpu | grep 'Thread(s) per core' | awk '{print $4}')"
echo "Core(s) per socket: $(lscpu | grep 'Core(s) per socket' | awk '{print $4}')"
}
# Configure cgroup CPU scheduling
setup_cgroups() {
echo "[3/8] Setting up CPU cgroups..."
# Backup existing cgroup configurations
if [ -d /sys/fs/cgroup/cpu ]; then
cp -r /sys/fs/cgroup/cpu "$BACKUP_DIR/" 2>/dev/null || true
fi
# Create cgroup directories
mkdir -p /sys/fs/cgroup/cpu/high-priority
mkdir -p /sys/fs/cgroup/cpu/low-priority
# Configure CPU shares (high priority gets more CPU time)
echo 2048 > /sys/fs/cgroup/cpu/high-priority/cpu.shares
echo 512 > /sys/fs/cgroup/cpu/low-priority/cpu.shares
# Set CPU quota for high priority group (200% = 2 cores)
echo 200000 > /sys/fs/cgroup/cpu/high-priority/cpu.cfs_quota_us
echo 100000 > /sys/fs/cgroup/cpu/high-priority/cpu.cfs_period_us
# Set CPU affinity for cgroups
echo "$HIGH_PRIORITY_CORES" > /sys/fs/cgroup/cpuset/high-priority/cpuset.cpus 2>/dev/null || true
echo "$SYSTEM_CORES" > /sys/fs/cgroup/cpuset/low-priority/cpuset.cpus 2>/dev/null || true
log_info "CPU cgroups configured successfully"
}
# Create systemd override configurations
configure_systemd_overrides() {
echo "[4/8] Configuring systemd CPU scheduling overrides..."
# Example configuration for common services
services=("sshd" "chronyd" "systemd-logind")
for service in "${services[@]}"; do
if systemctl list-unit-files "$service.service" > /dev/null 2>&1; then
override_dir="/etc/systemd/system/$service.service.d"
mkdir -p "$override_dir"
# Backup existing override if it exists
if [ -f "$override_dir/cpu-affinity.conf" ]; then
cp "$override_dir/cpu-affinity.conf" "$BACKUP_DIR/$service-cpu-affinity.conf"
fi
cat > "$override_dir/cpu-affinity.conf" << EOF
[Service]
# System service CPU affinity - use system cores
CPUAffinity=$SYSTEM_CORES
Nice=0
CPUSchedulingPolicy=0
EOF
chmod 644 "$override_dir/cpu-affinity.conf"
fi
done
log_info "Systemd CPU scheduling overrides configured"
}
# Create sample high-priority service template
create_sample_service() {
echo "[5/8] Creating sample high-priority service template..."
cat > "/etc/systemd/system/high-priority-app.service" << EOF
[Unit]
Description=High Priority Application Template
After=network.target
Documentation=man:systemd.service(5)
[Service]
Type=simple
User=root
Group=root
ExecStart=/bin/sleep 3600
# CPU Scheduling Configuration
Nice=-10
CPUAffinity=$HIGH_PRIORITY_CORES
CPUSchedulingPolicy=1
CPUSchedulingPriority=50
# Resource Limits
MemoryMax=1G
CPUQuota=150%
# Security
PrivateTmp=true
ProtectSystem=strict
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target
EOF
chmod 644 "/etc/systemd/system/high-priority-app.service"
log_info "Sample high-priority service template created"
}
# Create CPU affinity management scripts
create_management_scripts() {
echo "[6/8] Creating CPU affinity management scripts..."
# Create script directory
mkdir -p /usr/local/bin
# CPU affinity helper script
cat > "/usr/local/bin/cpu-affinity-tool" << 'EOF'
#!/bin/bash
# CPU Affinity Management Tool
case "$1" in
show)
echo "Current CPU topology:"
lscpu | grep -E "CPU\(s\)|Thread|Core|Socket"
echo
echo "Process CPU affinity (top 10 by CPU usage):"
ps -eo pid,ppid,cmd,nice,psr --sort=-pcpu | head -11
;;
set)
if [ $# -lt 3 ]; then
echo "Usage: $0 set <pid> <cpu_list>"
exit 1
fi
taskset -cp "$3" "$2"
;;
monitor)
watch -n 1 'ps -eo pid,ppid,cmd,nice,psr --sort=-pcpu | head -20'
;;
*)
echo "Usage: $0 {show|set|monitor}"
echo " show - Display current CPU topology and process affinity"
echo " set - Set CPU affinity for a process"
echo " monitor - Monitor process CPU usage and affinity"
exit 1
;;
esac
EOF
chmod 755 "/usr/local/bin/cpu-affinity-tool"
log_info "CPU affinity management scripts created"
}
# Apply configurations
apply_configurations() {
echo "[7/8] Applying CPU scheduling configurations..."
# Reload systemd daemon
systemctl daemon-reload
# Set kernel parameters for better scheduling
cat > "/etc/sysctl.d/99-cpu-scheduling.conf" << EOF
# CPU scheduling optimizations
kernel.sched_latency_ns = 6000000
kernel.sched_min_granularity_ns = 2000000
kernel.sched_wakeup_granularity_ns = 3000000
kernel.sched_migration_cost_ns = 500000
kernel.sched_nr_migrate = 32
# NUMA balancing
kernel.numa_balancing = 1
EOF
chmod 644 "/etc/sysctl.d/99-cpu-scheduling.conf"
sysctl -p "/etc/sysctl.d/99-cpu-scheduling.conf" > /dev/null
log_info "CPU scheduling configurations applied"
}
# Verify installation
verify_installation() {
echo "[8/8] Verifying CPU scheduling configuration..."
# Check if tools are installed
local tools=("taskset" "schedtool" "htop" "numactl")
for tool in "${tools[@]}"; do
if command -v "$tool" > /dev/null 2>&1; then
log_info "✓ $tool is installed"
else
log_warn "✗ $tool is not installed or not in PATH"
fi
done
# Check cgroup setup
if [ -d "/sys/fs/cgroup/cpu/high-priority" ]; then
log_info "✓ CPU cgroups are configured"
else
log_warn "✗ CPU cgroups not found"
fi
# Check systemd overrides
if [ -f "/etc/systemd/system/high-priority-app.service" ]; then
log_info "✓ Sample high-priority service created"
fi
# Check management script
if [ -x "/usr/local/bin/cpu-affinity-tool" ]; then
log_info "✓ CPU affinity management tool installed"
fi
echo
log_info "CPU scheduling and affinity configuration completed!"
echo
echo "Next steps:"
echo "1. Use 'cpu-affinity-tool show' to view current CPU topology"
echo "2. Use 'cpu-affinity-tool monitor' to monitor process CPU usage"
echo "3. Customize service CPU affinity in /etc/systemd/system/SERVICE.service.d/"
echo "4. Restart services to apply CPU affinity: systemctl restart SERVICE"
echo "5. Use 'taskset -cp CORES PID' to set CPU affinity for running processes"
}
# Main function
main() {
parse_args "$@"
check_root
detect_distro
create_backup
install_packages
show_system_info
setup_cgroups
configure_systemd_overrides
create_sample_service
create_management_scripts
apply_configurations
verify_installation
}
main "$@"
Review the script before running. Execute with: bash install.sh