Configure Varnish ESI (Edge Side Includes) for dynamic content optimization

Intermediate 25 min May 03, 2026 91 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Set up Varnish Cache 7 with Edge Side Includes to fragment and cache dynamic content separately. ESI allows you to cache static page parts while keeping dynamic sections fresh, improving performance for complex applications with mixed content types.

Prerequisites

  • A backend application running on port 8080
  • Basic understanding of HTTP caching
  • Root or sudo access

What this solves

Edge Side Includes (ESI) lets you cache different parts of a web page with different expiration times. Instead of marking an entire page as uncacheable because it contains one dynamic element, ESI fragments the page so static parts stay cached while dynamic sections refresh independently. This dramatically improves cache hit rates for applications with user-specific content, shopping carts, or real-time data sections.

Step-by-step installation

Update system packages

Start by updating your package manager to ensure you get the latest Varnish version.

sudo apt update && sudo apt upgrade -y
sudo dnf update -y

Install Varnish Cache 7

Install Varnish from the official repository to get version 7 with full ESI support.

curl -fsSL https://packagecloud.io/varnishcache/varnish70/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/varnish-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/varnish-archive-keyring.gpg] https://packagecloud.io/varnishcache/varnish70/ubuntu/ $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/varnish-cache.list
sudo apt update
sudo apt install -y varnish
curl -fsSL https://packagecloud.io/varnishcache/varnish70/gpgkey | sudo rpm --import -
echo '[varnish70]
name=Varnish 7.0 Repository
baseurl=https://packagecloud.io/varnishcache/varnish70/el/9/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://packagecloud.io/varnishcache/varnish70/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
' | sudo tee /etc/yum.repos.d/varnish70.repo
sudo dnf install -y varnish

Configure Varnish daemon settings

Configure Varnish to run on port 80 and use your backend application on port 8080.

[Unit]
Description=Varnish Cache HTTP reverse proxy
After=network.target

[Service]
Type=exec
ExecStart=/usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -s malloc,1G -p feature=+esi
ExecReload=/bin/kill -HUP $MAINPID
User=varnish
Group=varnish
PrivateDevices=yes
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/varnish
Restart=always

[Install]
WantedBy=multi-user.target
[Unit]
Description=Varnish Cache HTTP reverse proxy
After=network.target

[Service]
Type=exec
ExecStart=/usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -s malloc,1G -p feature=+esi
ExecReload=/bin/kill -HUP $MAINPID
User=varnish
Group=varnish
PrivateDevices=yes
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/varnish
Restart=always

[Install]
WantedBy=multi-user.target
Note: The -p feature=+esi parameter explicitly enables ESI processing. Without this, Varnish will ignore ESI tags.

Create ESI-enabled VCL configuration

Configure Varnish with a VCL that enables ESI processing and handles fragment caching.

vcl 4.1;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 10s;
    .first_byte_timeout = 30s;
    .between_bytes_timeout = 5s;
}

sub vcl_recv {
    # Enable ESI processing for specific paths
    if (req.url ~ "^/(page|content)/") {
        set req.http.X-ESI = "true";
    }
    
    # Pass through ESI fragment requests
    if (req.url ~ "^/fragments/") {
        return (pass);
    }
    
    # Remove cookies for static assets
    if (req.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg)$") {
        unset req.http.Cookie;
    }
}

sub vcl_backend_response {
    # Enable ESI processing for marked requests
    if (bereq.http.X-ESI == "true") {
        set beresp.do_esi = true;
    }
    
    # Cache fragments with specific TTL
    if (bereq.url ~ "^/fragments/") {
        set beresp.ttl = 300s;  # 5 minutes for fragments
        set beresp.http.Cache-Control = "max-age=300";
    }
    
    # Cache main pages with ESI enabled
    if (bereq.url ~ "^/(page|content)/" && beresp.http.Content-Type ~ "text/html") {
        set beresp.ttl = 1800s;  # 30 minutes for main pages
        set beresp.do_esi = true;
        # Remove backend cache headers that might interfere
        unset beresp.http.Set-Cookie;
    }
    
    # Cache static assets longer
    if (bereq.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg)$") {
        set beresp.ttl = 1d;
    }
}

sub vcl_deliver {
    # Add cache hit/miss headers for debugging
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
    } else {
        set resp.http.X-Cache = "MISS";
    }
    
    # Remove internal ESI header
    unset resp.http.X-ESI;
}

Configure your backend application for ESI

Modify your application to include ESI tags in HTML responses. Here's an example with different fragment types.




    ESI Example Page


    
    

My Website

Article Content

This is the main article content that changes rarely.

Create fragment endpoints in your application

Implement the fragment endpoints that ESI will call. Each fragment should return just the HTML snippet for that section.

# /fragments/user-info response:
Welcome, John Doe Logout

/fragments/recent-posts response:

/fragments/cart-summary response:

3 items $47.99 View Cart

Set fragment-specific cache headers

Configure your application to return appropriate cache headers for different fragment types.

# User-specific fragments (short cache)
Cache-Control: max-age=60, private
Vary: Cookie, Authorization

General dynamic content (medium cache)

Cache-Control: max-age=300, public Vary: Accept-Encoding

Semi-static content (long cache)

Cache-Control: max-age=1800, public Vary: Accept-Encoding

Real-time data (no cache)

Cache-Control: no-cache, must-revalidate Pragma: no-cache

Enable and start Varnish

Reload systemd configuration and start Varnish with the new ESI-enabled settings.

sudo systemctl daemon-reload
sudo systemctl enable --now varnish
sudo systemctl status varnish

Configure your backend application port

Ensure your application runs on port 8080 since Varnish will handle port 80. For example, if using Nginx as the backend:

server {
    listen 8080 default_server;
    listen [::]:8080 default_server;
    
    root /var/www/html;
    index index.html index.php;
    
    server_name example.com;
    
    location / {
        try_files $uri $uri/ =404;
    }
    
    # Fragment endpoints
    location /fragments/ {
        # Add cache headers for fragments
        add_header Cache-Control "max-age=300, public";
        try_files $uri $uri/ @app;
    }
    
    location @app {
        # Proxy to your application server
        proxy_pass http://127.0.0.1:9000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
server {
    listen 8080 default_server;
    listen [::]:8080 default_server;
    
    root /var/www/html;
    index index.html index.php;
    
    server_name example.com;
    
    location / {
        try_files $uri $uri/ =404;
    }
    
    # Fragment endpoints
    location /fragments/ {
        # Add cache headers for fragments
        add_header Cache-Control "max-age=300, public";
        try_files $uri $uri/ @app;
    }
    
    location @app {
        # Proxy to your application server
        proxy_pass http://127.0.0.1:9000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
sudo systemctl restart nginx

Verify your setup

Test that Varnish is properly processing ESI tags and caching fragments separately.

# Check Varnish is running and ESI is enabled
sudo systemctl status varnish
varnishd -V | grep -i esi

Test ESI processing with curl

curl -H "Host: example.com" http://localhost/ -v

Check cache headers

curl -H "Host: example.com" http://localhost/page/test -I

Test fragment caching separately

curl -H "Host: example.com" http://localhost/fragments/user-info -I

Monitor Varnish cache hits

sudo varnishstat -f MAIN.cache_hit,MAIN.cache_miss

Watch ESI processing in real-time

sudo varnishlog -q "VCL_call eq 'ESI'"
Note: You should see different cache headers for the main page versus fragments, and cache hit rates should improve as fragments are cached independently.

Optimize ESI caching strategies

Implement conditional ESI includes

Use ESI conditional statements to include different content based on request parameters.



    
        
    
    
        
    




    
        
    
    
        
    
    
        
    

Configure Vary headers for ESI fragments

Update your VCL to handle Vary headers properly for fragments that depend on specific request headers.

sub vcl_hash {
    hash_data(req.url);
    
    # Include specific cookies in hash for user-specific fragments
    if (req.url ~ "^/fragments/(user-|cart-|profile-)" && req.http.Cookie ~ "user_id=([^;]+)") {
        hash_data(regsub(req.http.Cookie, ".user_id=([^;]+).", "\1"));
    }
    
    # Include session ID for session-specific fragments
    if (req.url ~ "^/fragments/cart-" && req.http.Cookie ~ "session_id=([^;]+)") {
        hash_data(regsub(req.http.Cookie, ".session_id=([^;]+).", "\1"));
    }
    
    # Always include host
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    
    return (lookup);
}

sub vcl_backend_response {
    # Set appropriate Vary headers for different fragment types
    if (bereq.url ~ "^/fragments/user-") {
        set beresp.http.Vary = "Cookie, User-Agent";
        set beresp.ttl = 300s;
    }
    
    if (bereq.url ~ "^/fragments/public-") {
        set beresp.http.Vary = "Accept-Encoding";
        set beresp.ttl = 3600s;
    }
    
    if (bereq.url ~ "^/fragments/realtime-") {
        set beresp.ttl = 30s;
        set beresp.http.Cache-Control = "max-age=30";
    }
}

Add ESI error handling

Implement graceful fallbacks when ESI fragments fail to load.






    
        
    
    
        

Pricing information temporarily unavailable.

Performance optimization

Configure fragment cache warming

Create a script to pre-warm frequently accessed fragments.

#!/bin/bash

Common fragments to warm

fragments=( "/fragments/popular-products" "/fragments/latest-news" "/fragments/weather" "/fragments/stock-ticker" )

Base URL for your site

BASE_URL="http://localhost" echo "Warming ESI fragment cache..." for fragment in "${fragments[@]}"; do echo "Warming: $fragment" curl -s -o /dev/null "$BASE_URL$fragment" sleep 0.1 done echo "Cache warming completed"

Show cache statistics

varnishstat -1 -f MAIN.cache_hit,MAIN.cache_miss
sudo chmod +x /usr/local/bin/warm-esi-cache.sh

Add to crontab for regular warming

echo "/5 * /usr/local/bin/warm-esi-cache.sh > /dev/null 2>&1" | sudo crontab -

Monitor ESI performance

Set up monitoring to track ESI cache effectiveness and fragment performance.

#!/bin/bash

Monitor ESI-specific metrics

echo "=== ESI Cache Statistics ===" varnishstat -1 -f MAIN.esi_errors,MAIN.esi_warnings,MAIN.cache_hit,MAIN.cache_miss echo -e "\n=== Fragment Response Times ==="

Test fragment response times

for fragment in "/fragments/user-info" "/fragments/cart-summary" "/fragments/weather"; do time_ms=$(curl -s -w "%{time_total}" -o /dev/null "http://localhost$fragment" | awk '{print $1 * 1000}') echo "$fragment: ${time_ms}ms" done echo -e "\n=== Recent ESI Errors ===" varnishlog -d -q "VCL_Error ~ 'ESI'" | tail -10
sudo chmod +x /usr/local/bin/esi-monitor.sh
./usr/local/bin/esi-monitor.sh

Common issues

SymptomCauseFix
ESI tags appear in HTML outputESI processing disabledAdd -p feature=+esi to varnishd command and set beresp.do_esi = true
Fragments not caching separatelyMissing cache headersSet proper TTL in VCL: set beresp.ttl = 300s; for fragments
User-specific content cached globallyHash doesn't include user contextAdd user ID to hash in vcl_hash subroutine
High backend load from fragmentsFragments not cached effectivelyCheck fragment URLs match VCL patterns and have proper cache headers
ESI includes return 404Fragment endpoints not implementedImplement all fragment URLs referenced in ESI tags
Slow page renderingToo many serial ESI includesUse parallel processing and set timeouts: .first_byte_timeout = 10s
Cache not invalidatingMissing cache purge configurationImplement cache purging: ban req.url ~ "^/fragments/"
Warning: Always test ESI configuration in a staging environment first. Misconfigured ESI can cause infinite loops or expose sensitive fragment content.

Next steps

Running this in production?

Want this handled for you? Setting this up once is straightforward. Keeping it patched, monitored, backed up and performant across environments is the harder part. See how we run infrastructure like this for European teams.

Automated install script

Run this to automate the entire setup

Need help?

Don't want to manage this yourself?

We handle infrastructure performance optimization for businesses that depend on uptime. From initial setup to ongoing operations.