Optimize Varnish 7 cache storage with memory tuning and persistence configuration

Intermediate 25 min May 24, 2026 551 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Configure Varnish 7 memory allocation and storage backends for optimal cache performance. Set up file-based persistence and tune memory parameters for high-traffic workloads with monitoring.

Prerequisites

  • Varnish 7 installed and running
  • Root or sudo access
  • Basic understanding of web caching concepts
  • At least 4GB available disk space for cache storage

What this solves

Varnish 7 ships with default memory settings that work for basic setups but need optimization for production workloads. This tutorial shows you how to configure memory allocation, implement file-based cache persistence, and tune storage backends for high-traffic websites. You'll optimize cache storage to survive restarts and handle memory pressure efficiently.

Step-by-step configuration

Check current Varnish configuration

First, examine your current Varnish setup to understand the baseline configuration and identify optimization opportunities.

sudo systemctl status varnish
sudo varnishstat -1 | grep -E "cache_hit|cache_miss|n_object|g_bytes"
varnishd -V

Stop Varnish for configuration changes

Stop the service to make storage backend changes that require a restart.

sudo systemctl stop varnish

Configure memory allocation and storage backends

Edit the Varnish systemd service file to configure storage backends and memory allocation. The malloc storage type stores cache in memory while file storage provides persistence.

sudo mkdir -p /etc/systemd/system/varnish.service.d
sudo tee /etc/systemd/system/varnish.service.d/override.conf > /dev/null << 'EOF'
[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd \
  -a :6081 \
  -T localhost:6082 \
  -f /etc/varnish/default.vcl \
  -s malloc,2G \
  -s file,/var/cache/varnish/storage.bin,4G \
  -p workspace_backend=128k \
  -p workspace_client=128k \
  -p http_max_hdr=128 \
  -p thread_pools=4 \
  -p thread_pool_min=100 \
  -p thread_pool_max=1000
EOF
sudo mkdir -p /etc/systemd/system/varnish.service.d
sudo tee /etc/systemd/system/varnish.service.d/override.conf > /dev/null << 'EOF'
[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd \
  -a :6081 \
  -T localhost:6082 \
  -f /etc/varnish/default.vcl \
  -s malloc,2G \
  -s file,/var/cache/varnish/storage.bin,4G \
  -p workspace_backend=128k \
  -p workspace_client=128k \
  -p http_max_hdr=128 \
  -p thread_pools=4 \
  -p thread_pool_min=100 \
  -p thread_pool_max=1000
EOF

Create cache storage directory

Create the directory for file-based cache storage with proper ownership and permissions.

sudo mkdir -p /var/cache/varnish
sudo chown varnish:varnish /var/cache/varnish
sudo chmod 755 /var/cache/varnish

Configure advanced memory tuning parameters

Create a VCL configuration file with optimized memory and cache settings for high-traffic workloads.

vcl 4.1;

import std;
import directors;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 5s;
    .first_byte_timeout = 10s;
    .between_bytes_timeout = 5s;
    .max_connections = 300;
}

sub vcl_recv {
    # Memory optimization: Remove cookies for static assets
    if (req.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$") {
        unset req.http.Cookie;
    }
    
    # Memory optimization: Normalize Accept-Encoding
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            unset req.http.Accept-Encoding;
        }
    }
}

sub vcl_backend_response {
    # Cache static assets for 1 day
    if (bereq.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$") {
        set beresp.ttl = 1d;
        set beresp.http.Cache-Control = "public, max-age=86400";
    }
    
    # Remove server headers that waste cache memory
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;
    unset beresp.http.X-Drupal-Dynamic-Cache;
    
    # Enable grace mode for better performance
    set beresp.grace = 6h;
}

sub vcl_deliver {
    # Add cache hit information
    if (obj.hits > 0) {
        set resp.http.X-Varnish-Cache = "HIT";
    } else {
        set resp.http.X-Varnish-Cache = "MISS";
    }
    
    # Remove internal headers
    unset resp.http.Via;
    unset resp.http.X-Varnish;
}

Update Varnish to use the optimized configuration

Modify the systemd override to use the memory-optimized VCL file.

sudo tee /etc/systemd/system/varnish.service.d/override.conf > /dev/null << 'EOF'
[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd \
  -a :6081 \
  -T localhost:6082 \
  -f /etc/varnish/memory-optimized.vcl \
  -s malloc,2G \
  -s file,/var/cache/varnish/storage.bin,4G \
  -p workspace_backend=128k \
  -p workspace_client=128k \
  -p http_max_hdr=128 \
  -p thread_pools=4 \
  -p thread_pool_min=100 \
  -p thread_pool_max=1000 \
  -p obj_readonly=on \
  -p vcc_allow_inline_c=on \
  -p shortlived=10 \
  -p lru_interval=20
EOF

Configure cache persistence and storage optimization

Create a script to manage cache persistence across restarts and optimize storage allocation.

#!/bin/bash

# Varnish Cache Management Script

CACHE_DIR="/var/cache/varnish"
STORAGE_FILE="${CACHE_DIR}/storage.bin"
LOG_FILE="/var/log/varnish/cache-manager.log"

# Create log directory
mkdir -p /var/log/varnish

log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}

# Function to check cache storage health
check_storage_health() {
    if [[ -f "$STORAGE_FILE" ]]; then
        local file_size=$(stat -c%s "$STORAGE_FILE" 2>/dev/null)
        if [[ $file_size -gt 0 ]]; then
            log_message "Cache storage file exists and is healthy ($file_size bytes)"
            return 0
        fi
    fi
    log_message "Cache storage file missing or corrupted"
    return 1
}

# Function to initialize storage
init_storage() {
    log_message "Initializing cache storage directory"
    mkdir -p "$CACHE_DIR"
    chown varnish:varnish "$CACHE_DIR"
    chmod 755 "$CACHE_DIR"
    
    # Pre-allocate storage file for better performance
    if [[ ! -f "$STORAGE_FILE" ]]; then
        log_message "Pre-allocating storage file"
        fallocate -l 4G "$STORAGE_FILE" 2>/dev/null || dd if=/dev/zero of="$STORAGE_FILE" bs=1M count=4096 2>/dev/null
        chown varnish:varnish "$STORAGE_FILE"
        chmod 644 "$STORAGE_FILE"
    fi
}

# Function to cleanup old cache files
cleanup_cache() {
    log_message "Cleaning up old cache files"
    find "$CACHE_DIR" -name "*.tmp" -mtime +1 -delete 2>/dev/null
    find "$CACHE_DIR" -name "varnish*" -mtime +7 -delete 2>/dev/null
}

# Main execution
case "$1" in
    "init")
        init_storage
        ;;
    "check")
        check_storage_health
        ;;
    "cleanup")
        cleanup_cache
        ;;
    "status")
        echo "Cache directory: $CACHE_DIR"
        echo "Storage file: $STORAGE_FILE"
        if [[ -f "$STORAGE_FILE" ]]; then
            echo "Storage size: $(du -h "$STORAGE_FILE" | cut -f1)"
        fi
        df -h "$CACHE_DIR"
        ;;
    *)
        echo "Usage: $0 {init|check|cleanup|status}"
        exit 1
        ;;
esac

Make the cache manager executable and initialize storage

Set proper permissions and initialize the cache storage system.

sudo chmod +x /usr/local/bin/varnish-cache-manager.sh
sudo /usr/local/bin/varnish-cache-manager.sh init

Create systemd timer for cache maintenance

Set up automated cache maintenance to keep storage optimized.

[Unit]
Description=Varnish Cache Maintenance
After=varnish.service

[Service]
Type=oneshot
User=root
ExecStart=/usr/local/bin/varnish-cache-manager.sh cleanup
ExecStart=/usr/local/bin/varnish-cache-manager.sh check
[Unit]
Description=Run Varnish Cache Maintenance
Requires=varnish-maintenance.service

[Timer]
OnCalendar=daily
RandomizedDelaySec=300
Persistent=true

[Install]
WantedBy=timers.target

Reload systemd and start services

Apply the configuration changes and start Varnish with the optimized settings.

sudo systemctl daemon-reload
sudo systemctl enable varnish-maintenance.timer
sudo systemctl start varnish-maintenance.timer
sudo systemctl start varnish
sudo systemctl enable varnish

Configure memory monitoring script

Create a monitoring script to track Varnish memory usage and cache performance.

#!/bin/bash

# Varnish Performance Monitoring Script

MONITOR_LOG="/var/log/varnish/performance.log"

# Create log directory
mkdir -p /var/log/varnish

# Function to log performance metrics
log_metrics() {
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    
    # Get Varnish statistics
    local cache_hits=$(varnishstat -1 -f cache_hit | awk '{print $2}')
    local cache_misses=$(varnishstat -1 -f cache_miss | awk '{print $2}')
    local objects=$(varnishstat -1 -f n_object | awk '{print $2}')
    local bytes=$(varnishstat -1 -f g_bytes | awk '{print $2}')
    local memory_usage=$(varnishstat -1 -f SMA.s0.g_bytes | awk '{print $2}')
    
    # Calculate hit ratio
    local total_requests=$((cache_hits + cache_misses))
    local hit_ratio=0
    if [[ $total_requests -gt 0 ]]; then
        hit_ratio=$(echo "scale=2; $cache_hits * 100 / $total_requests" | bc -l)
    fi
    
    # Log metrics
    echo "$timestamp,hits:$cache_hits,misses:$cache_misses,objects:$objects,bytes:$bytes,memory:$memory_usage,hit_ratio:${hit_ratio}%" >> "$MONITOR_LOG"
    
    # Check for memory pressure
    local memory_mb=$((memory_usage / 1024 / 1024))
    if [[ $memory_mb -gt 1800 ]]; then
        logger -t varnish-monitor "WARNING: Memory usage high: ${memory_mb}MB"
    fi
    
    # Check hit ratio
    if [[ $(echo "$hit_ratio < 80" | bc -l) -eq 1 ]] && [[ $total_requests -gt 1000 ]]; then
        logger -t varnish-monitor "WARNING: Low hit ratio: ${hit_ratio}%"
    fi
}

# Run monitoring
log_metrics

# Keep only last 7 days of logs
find /var/log/varnish -name "performance.log" -mtime +7 -delete 2>/dev/null

Make monitoring script executable and create cron job

Set up regular performance monitoring to track cache efficiency.

sudo chmod +x /usr/local/bin/varnish-monitor.sh

# Create cron job to run every 5 minutes
echo "*/5 * * * * /usr/local/bin/varnish-monitor.sh" | sudo crontab -

Verify your setup

Check that Varnish is running with the optimized configuration and monitoring the cache performance.

# Check service status
sudo systemctl status varnish

# Verify storage configuration
varnishd -C -f /etc/varnish/memory-optimized.vcl

# Check memory allocation
varnishstat -1 | grep -E "SMA|cache_hit|cache_miss|n_object"

# Test cache functionality
curl -I http://localhost:6081/

# Check storage files
sudo /usr/local/bin/varnish-cache-manager.sh status

# View performance log
tail -f /var/log/varnish/performance.log

You should see Varnish running with both malloc and file storage backends, cache hits/misses being tracked, and performance metrics being logged.

Storage configuration options

Different storage backends offer various trade-offs between performance and persistence:

Storage TypeUse CasePerformancePersistence
mallocHigh-performance cachingFastestLost on restart
filePersistent cache storageFastSurvives restarts
persistentLong-term storageMediumFull persistence

Memory tuning parameters

Key parameters for optimizing Varnish memory usage:

ParameterPurposeRecommended Value
workspace_backendBackend communication buffer128k for most workloads
workspace_clientClient communication buffer128k for most workloads
http_max_hdrMaximum HTTP headers128 for typical websites
thread_pool_minMinimum worker threads100 for high traffic
thread_pool_maxMaximum worker threads1000 for high traffic

Common issues

SymptomCauseFix
Varnish won't startStorage file permissionssudo chown varnish:varnish /var/cache/varnish
Cache not persistingFile storage not configuredCheck -s file parameter in service config
High memory usageMalloc storage too largeReduce malloc size, increase file storage
Low hit ratioTTL too short or cookies interferingCheck VCL cookie handling and TTL settings
"No space left" errorStorage file fills diskMove storage to larger partition or reduce size
Performance degradationToo many threads or insufficient workspaceTune thread pool and workspace parameters
Storage Size Planning: Ensure your file storage size doesn't exceed available disk space. Monitor disk usage regularly as cached content grows over time.

Next steps

Running this in production?

Want this handled for you? Setting up Varnish cache optimization once is straightforward. Keeping it tuned, monitored, and performing well under varying traffic loads is the harder part. See how we run infrastructure like this for European SaaS and e-commerce teams.

Automated install script

Run this to automate the entire setup

#varnish #cache #memory-tuning #storage-backends #performance

Related tutorials

performance Intermediate

Configure Cherokee caching and compression for improved performance

Set up Cherokee web server with advanced caching modules and gzip compression to dramatically improve page load times and reduce bandwidth usage for high-traffic websites.

25 min 4 distros 352 views
performance Advanced

Implement Spark SQL performance optimization with Catalyst optimizer and advanced tuning

Optimize Apache Spark 3.5 SQL performance using Catalyst optimizer with advanced query tuning, adaptive query execution, and production-grade configuration for high-throughput analytics workloads.

45 min 4 distros 472 views
performance Advanced

Configure Nginx Redis cluster caching for high availability and performance optimization

Set up Nginx with Redis cluster caching to achieve high availability and optimized performance. This advanced configuration includes cluster setup, failover testing, and performance tuning for production environments.

45 min 4 distros 405 views
Browse all tutorials in our complete directory →

Don't want to manage this yourself?

We handle infrastructure for businesses that depend on uptime. Fully managed, with one fixed contact who knows your setup.

You get one fixed contact who knows your setup

At their desk in Rotterdam 11:21 · reachable in a message, no ticket form

+31 10 477 5362 WhatsApp