Implement Linux memory cgroups for container workload isolation

Intermediate 45 min Apr 16, 2026 178 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Configure cgroups v2 memory subsystem to isolate container workloads with precise memory limits, monitoring, and automated enforcement for production environments.

Prerequisites

  • Root or sudo access
  • Linux kernel 4.5+ with cgroups v2 support
  • systemd-based distribution
  • Basic understanding of Linux process management

What this solves

Memory cgroups (control groups) provide essential resource isolation for containers by limiting and monitoring memory usage across different workloads. This prevents memory-intensive containers from consuming system resources and affecting other processes. You'll implement cgroups v2 memory controls to establish container boundaries, set memory limits, and monitor resource consumption in real-time.

Step-by-step configuration

Verify cgroups v2 support

Check if your system supports cgroups v2 and which version is currently active. Most modern distributions enable cgroups v2 by default.

mount | grep cgroup
cat /sys/fs/cgroup/cgroup.controllers
cat /proc/cgroups

Enable cgroups v2 memory controller

Enable the memory controller in the cgroup hierarchy to allow memory limit enforcement and monitoring.

echo '+memory' | sudo tee /sys/fs/cgroup/cgroup.subtree_control
cat /sys/fs/cgroup/cgroup.subtree_control

Install cgroup management tools

Install utilities for managing cgroups and monitoring resource usage across different distributions.

sudo apt update
sudo apt install -y cgroup-tools libcgroup-dev systemd-cgroup-utils
sudo dnf install -y libcgroup-tools systemd

Create container workload cgroups

Create dedicated cgroup hierarchies for different container workloads with appropriate directory structure and permissions.

sudo mkdir -p /sys/fs/cgroup/containers/web-tier
sudo mkdir -p /sys/fs/cgroup/containers/database-tier
sudo mkdir -p /sys/fs/cgroup/containers/cache-tier
sudo chown -R root:root /sys/fs/cgroup/containers/

Configure memory limits for web tier

Set memory limits and swap controls for web application containers with 512MB memory limit and monitoring enabled.

echo '+memory' | sudo tee /sys/fs/cgroup/containers/cgroup.subtree_control
echo '536870912' | sudo tee /sys/fs/cgroup/containers/web-tier/memory.max
echo '268435456' | sudo tee /sys/fs/cgroup/containers/web-tier/memory.high
echo '0' | sudo tee /sys/fs/cgroup/containers/web-tier/memory.swap.max

Configure memory limits for database tier

Set higher memory limits for database containers with 2GB maximum and appropriate swap configuration.

echo '2147483648' | sudo tee /sys/fs/cgroup/containers/database-tier/memory.max
echo '1073741824' | sudo tee /sys/fs/cgroup/containers/database-tier/memory.high
echo '536870912' | sudo tee /sys/fs/cgroup/containers/database-tier/memory.swap.max

Configure memory limits for cache tier

Set memory limits for cache containers with 1GB maximum and disabled swap for predictable performance.

echo '1073741824' | sudo tee /sys/fs/cgroup/containers/cache-tier/memory.max
echo '805306368' | sudo tee /sys/fs/cgroup/containers/cache-tier/memory.high
echo '0' | sudo tee /sys/fs/cgroup/containers/cache-tier/memory.swap.max

Create systemd service for cgroup management

Create a systemd service to automatically configure cgroups on system startup and maintain memory limits.

[Unit]
Description=Container Memory Cgroups Setup
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/setup-container-cgroups.sh
User=root

[Install]
WantedBy=multi-user.target

Create cgroup setup script

Create the automation script referenced by the systemd service for consistent cgroup configuration.

#!/bin/bash
set -euo pipefail

Enable memory controller

echo '+memory' > /sys/fs/cgroup/cgroup.subtree_control echo '+memory' > /sys/fs/cgroup/containers/cgroup.subtree_control

Web tier - 512MB max, 256MB high

echo '536870912' > /sys/fs/cgroup/containers/web-tier/memory.max echo '268435456' > /sys/fs/cgroup/containers/web-tier/memory.high echo '0' > /sys/fs/cgroup/containers/web-tier/memory.swap.max

Database tier - 2GB max, 1GB high, 512MB swap

echo '2147483648' > /sys/fs/cgroup/containers/database-tier/memory.max echo '1073741824' > /sys/fs/cgroup/containers/database-tier/memory.high echo '536870912' > /sys/fs/cgroup/containers/database-tier/memory.swap.max

Cache tier - 1GB max, 768MB high, no swap

echo '1073741824' > /sys/fs/cgroup/containers/cache-tier/memory.max echo '805306368' > /sys/fs/cgroup/containers/cache-tier/memory.high echo '0' > /sys/fs/cgroup/containers/cache-tier/memory.swap.max echo "Container cgroups configured successfully"
sudo chmod +x /usr/local/bin/setup-container-cgroups.sh

Enable and start the cgroup service

Enable the systemd service to automatically configure cgroups on system boot and start it immediately.

sudo systemctl daemon-reload
sudo systemctl enable container-cgroups.service
sudo systemctl start container-cgroups.service
sudo systemctl status container-cgroups.service

Create process assignment script

Create a utility script to assign processes to specific cgroups for container workload isolation.

#!/bin/bash
set -euo pipefail

if [ "$#" -ne 2 ]; then
    echo "Usage: $0  "
    echo "Available cgroups: web-tier, database-tier, cache-tier"
    exit 1
fi

CGROUP="$1"
PID="$2"
CGROUP_PATH="/sys/fs/cgroup/containers/${CGROUP}"

if [ ! -d "$CGROUP_PATH" ]; then
    echo "Error: Cgroup $CGROUP does not exist"
    exit 1
fi

if ! kill -0 "$PID" 2>/dev/null; then
    echo "Error: Process $PID does not exist"
    exit 1
fi

echo "$PID" > "${CGROUP_PATH}/cgroup.procs"
echo "Process $PID assigned to cgroup $CGROUP"
sudo chmod +x /usr/local/bin/assign-to-cgroup.sh

Configure memory pressure notifications

Set up memory pressure monitoring to receive notifications when memory usage approaches limits.

#!/bin/bash
set -euo pipefail

CGROUPS=("web-tier" "database-tier" "cache-tier")

for cgroup in "${CGROUPS[@]}"; do
    CGROUP_PATH="/sys/fs/cgroup/containers/${cgroup}"
    
    if [ -f "${CGROUP_PATH}/memory.current" ]; then
        CURRENT=$(cat "${CGROUP_PATH}/memory.current")
        MAX=$(cat "${CGROUP_PATH}/memory.max")
        HIGH=$(cat "${CGROUP_PATH}/memory.high")
        
        CURRENT_MB=$((CURRENT / 1024 / 1024))
        MAX_MB=$((MAX / 1024 / 1024))
        HIGH_MB=$((HIGH / 1024 / 1024))
        
        USAGE_PERCENT=$((CURRENT * 100 / MAX))
        
        echo "${cgroup}: ${CURRENT_MB}MB / ${MAX_MB}MB (${USAGE_PERCENT}% used, high threshold: ${HIGH_MB}MB)"
        
        if [ "$CURRENT" -gt "$HIGH" ]; then
            echo "WARNING: ${cgroup} memory usage exceeds high threshold!"
            logger -t cgroup-monitor "WARNING: ${cgroup} memory usage ${CURRENT_MB}MB exceeds high threshold ${HIGH_MB}MB"
        fi
    fi
done
sudo chmod +x /usr/local/bin/monitor-memory-pressure.sh

Set up automated monitoring with cron

Configure cron to run memory pressure monitoring every minute for proactive resource management.

echo "    * /usr/local/bin/monitor-memory-pressure.sh >> /var/log/cgroup-monitor.log 2>&1" | sudo crontab -

Verify your setup

Test the cgroup configuration and verify memory limits are properly enforced.

# Check cgroup hierarchy
find /sys/fs/cgroup/containers -name "memory.*" | head -10

Verify memory limits

cat /sys/fs/cgroup/containers/web-tier/memory.max cat /sys/fs/cgroup/containers/database-tier/memory.max cat /sys/fs/cgroup/containers/cache-tier/memory.max

Check current memory usage

cat /sys/fs/cgroup/containers/web-tier/memory.current cat /sys/fs/cgroup/containers/database-tier/memory.current cat /sys/fs/cgroup/containers/cache-tier/memory.current

Test process assignment

sudo /usr/local/bin/assign-to-cgroup.sh web-tier $$ cat /sys/fs/cgroup/containers/web-tier/cgroup.procs

Run memory pressure monitoring

sudo /usr/local/bin/monitor-memory-pressure.sh

Configure container runtime integration

Integrate with systemd-run

Use systemd-run to launch processes with automatic cgroup assignment and resource limits.

# Launch process in web-tier cgroup with 256MB limit
sudo systemd-run --slice=containers/web-tier --property=MemoryMax=256M --property=MemoryHigh=128M sleep 300

Launch process in database-tier cgroup with 1GB limit

sudo systemd-run --slice=containers/database-tier --property=MemoryMax=1G --property=MemoryHigh=512M sleep 300

Check running services

systemctl list-units --type=service --state=running | grep containers

Create container wrapper script

Create a wrapper script that automatically assigns containers to appropriate cgroups based on their role.

#!/bin/bash
set -euo pipefail

if [ "$#" -lt 2 ]; then
    echo "Usage: $0  "
    echo "Tiers: web-tier, database-tier, cache-tier"
    exit 1
fi

TIER="$1"
shift
COMMAND="$@"

case "$TIER" in
    web-tier)
        MEMORY_MAX="256M"
        MEMORY_HIGH="128M"
        ;;
    database-tier)
        MEMORY_MAX="1G"
        MEMORY_HIGH="512M"
        ;;
    cache-tier)
        MEMORY_MAX="512M"
        MEMORY_HIGH="256M"
        ;;
    *)
        echo "Error: Unknown tier $TIER"
        exit 1
        ;;
esac

echo "Starting container in $TIER with limits: max=$MEMORY_MAX, high=$MEMORY_HIGH"
systemd-run --slice="containers/$TIER" \
    --property="MemoryMax=$MEMORY_MAX" \
    --property="MemoryHigh=$MEMORY_HIGH" \
    --property="MemorySwapMax=0" \
    $COMMAND
sudo chmod +x /usr/local/bin/run-container.sh

Monitor and troubleshoot memory constraints

Create detailed monitoring script

Implement comprehensive monitoring that tracks memory statistics, pressure events, and OOM kills.

#!/bin/bash
set -euo pipefail

CGROUPS=("web-tier" "database-tier" "cache-tier")

echo "=== Container Memory Cgroup Status ==="
echo "Timestamp: $(date)"
echo

for cgroup in "${CGROUPS[@]}"; do
    CGROUP_PATH="/sys/fs/cgroup/containers/${cgroup}"
    
    if [ -d "$CGROUP_PATH" ]; then
        echo "=== $cgroup ==="
        
        # Memory usage
        CURRENT=$(cat "${CGROUP_PATH}/memory.current" 2>/dev/null || echo "0")
        MAX=$(cat "${CGROUP_PATH}/memory.max" 2>/dev/null || echo "0")
        HIGH=$(cat "${CGROUP_PATH}/memory.high" 2>/dev/null || echo "0")
        
        echo "Current: $((CURRENT / 1024 / 1024)) MB"
        echo "High threshold: $((HIGH / 1024 / 1024)) MB"
        echo "Maximum: $((MAX / 1024 / 1024)) MB"
        
        # Memory events
        if [ -f "${CGROUP_PATH}/memory.events" ]; then
            echo "Memory events:"
            cat "${CGROUP_PATH}/memory.events" | sed 's/^/  /'
        fi
        
        # Process count
        PROC_COUNT=$(cat "${CGROUP_PATH}/cgroup.procs" 2>/dev/null | wc -l)
        echo "Processes: $PROC_COUNT"
        
        # Memory pressure
        if [ -f "${CGROUP_PATH}/memory.pressure" ]; then
            echo "Memory pressure:"
            cat "${CGROUP_PATH}/memory.pressure" | sed 's/^/  /'
        fi
        
        echo
    fi
done
sudo chmod +x /usr/local/bin/detailed-memory-monitor.sh

Configure memory pressure alerts

Set up systemd service to monitor memory pressure and send alerts when thresholds are exceeded.

[Unit]
Description=Memory Pressure Alert Monitor
After=container-cgroups.service

[Service]
Type=simple
ExecStart=/usr/local/bin/memory-pressure-daemon.sh
Restart=always
RestartSec=10
User=root

[Install]
WantedBy=multi-user.target

Create memory pressure daemon

Implement a daemon that continuously monitors memory pressure and logs critical events.

#!/bin/bash
set -euo pipefail

LOG_FILE="/var/log/memory-pressure.log"
CGROUPS=("web-tier" "database-tier" "cache-tier")

log_event() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') $1" >> "$LOG_FILE"
    logger -t memory-pressure "$1"
}

log_event "Memory pressure monitoring daemon started"

while true; do
    for cgroup in "${CGROUPS[@]}"; do
        CGROUP_PATH="/sys/fs/cgroup/containers/${cgroup}"
        
        if [ -f "${CGROUP_PATH}/memory.current" ]; then
            CURRENT=$(cat "${CGROUP_PATH}/memory.current")
            HIGH=$(cat "${CGROUP_PATH}/memory.high")
            MAX=$(cat "${CGROUP_PATH}/memory.max")
            
            # Check if we're above high threshold
            if [ "$CURRENT" -gt "$HIGH" ]; then
                CURRENT_MB=$((CURRENT / 1024 / 1024))
                HIGH_MB=$((HIGH / 1024 / 1024))
                log_event "ALERT: ${cgroup} memory usage ${CURRENT_MB}MB exceeds high threshold ${HIGH_MB}MB"
            fi
            
            # Check memory events for OOM kills
            if [ -f "${CGROUP_PATH}/memory.events" ]; then
                OOM_KILL=$(grep "oom_kill" "${CGROUP_PATH}/memory.events" | cut -d' ' -f2)
                if [ "$OOM_KILL" -gt 0 ]; then
                    log_event "CRITICAL: ${cgroup} has experienced ${OOM_KILL} OOM kills"
                fi
            fi
        fi
    done
    
    sleep 30
done
sudo chmod +x /usr/local/bin/memory-pressure-daemon.sh
sudo systemctl enable memory-pressure-alert.service
sudo systemctl start memory-pressure-alert.service

Common issues

SymptomCauseFix
Cannot write to memory.maxcgroups v1 active or insufficient permissionsVerify cgroups v2: mount | grep cgroup2 and run as root
Memory controller not availableController not enabled in hierarchyecho '+memory' | sudo tee /sys/fs/cgroup/cgroup.subtree_control
Process assignment failsProcess doesn't exist or cgroup path wrongVerify PID: kill -0 PID and check cgroup path exists
Memory limits ignoredProcess not in correct cgroupCheck: cat /sys/fs/cgroup/containers/tier/cgroup.procs
OOM kills frequentMemory limits too restrictiveIncrease limits or optimize application memory usage
systemd slice errorsSlice hierarchy conflictUse consistent naming: --slice=containers/tier

Next steps

Automated install script

Run this to automate the entire setup

Need help?

Don't want to manage this yourself?

We handle managed cloud infrastructure for businesses that depend on uptime. From initial setup to ongoing operations.