Set up Varnish Cache 7 with Edge Side Includes to fragment and cache dynamic content separately. ESI allows you to cache static page parts while keeping dynamic sections fresh, improving performance for complex applications with mixed content types.
Prerequisites
- A backend application running on port 8080
- Basic understanding of HTTP caching
- Root or sudo access
What this solves
Edge Side Includes (ESI) lets you cache different parts of a web page with different expiration times. Instead of marking an entire page as uncacheable because it contains one dynamic element, ESI fragments the page so static parts stay cached while dynamic sections refresh independently. This dramatically improves cache hit rates for applications with user-specific content, shopping carts, or real-time data sections.
Step-by-step installation
Update system packages
Start by updating your package manager to ensure you get the latest Varnish version.
sudo apt update && sudo apt upgrade -y
Install Varnish Cache 7
Install Varnish from the official repository to get version 7 with full ESI support.
curl -fsSL https://packagecloud.io/varnishcache/varnish70/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/varnish-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/varnish-archive-keyring.gpg] https://packagecloud.io/varnishcache/varnish70/ubuntu/ $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/varnish-cache.list
sudo apt update
sudo apt install -y varnish
Configure Varnish daemon settings
Configure Varnish to run on port 80 and use your backend application on port 8080.
[Unit]
Description=Varnish Cache HTTP reverse proxy
After=network.target
[Service]
Type=exec
ExecStart=/usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -s malloc,1G -p feature=+esi
ExecReload=/bin/kill -HUP $MAINPID
User=varnish
Group=varnish
PrivateDevices=yes
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/varnish
Restart=always
[Install]
WantedBy=multi-user.target
-p feature=+esi parameter explicitly enables ESI processing. Without this, Varnish will ignore ESI tags.Create ESI-enabled VCL configuration
Configure Varnish with a VCL that enables ESI processing and handles fragment caching.
vcl 4.1;
backend default {
.host = "127.0.0.1";
.port = "8080";
.connect_timeout = 10s;
.first_byte_timeout = 30s;
.between_bytes_timeout = 5s;
}
sub vcl_recv {
# Enable ESI processing for specific paths
if (req.url ~ "^/(page|content)/") {
set req.http.X-ESI = "true";
}
# Pass through ESI fragment requests
if (req.url ~ "^/fragments/") {
return (pass);
}
# Remove cookies for static assets
if (req.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg)$") {
unset req.http.Cookie;
}
}
sub vcl_backend_response {
# Enable ESI processing for marked requests
if (bereq.http.X-ESI == "true") {
set beresp.do_esi = true;
}
# Cache fragments with specific TTL
if (bereq.url ~ "^/fragments/") {
set beresp.ttl = 300s; # 5 minutes for fragments
set beresp.http.Cache-Control = "max-age=300";
}
# Cache main pages with ESI enabled
if (bereq.url ~ "^/(page|content)/" && beresp.http.Content-Type ~ "text/html") {
set beresp.ttl = 1800s; # 30 minutes for main pages
set beresp.do_esi = true;
# Remove backend cache headers that might interfere
unset beresp.http.Set-Cookie;
}
# Cache static assets longer
if (bereq.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg)$") {
set beresp.ttl = 1d;
}
}
sub vcl_deliver {
# Add cache hit/miss headers for debugging
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
set resp.http.X-Cache-Hits = obj.hits;
} else {
set resp.http.X-Cache = "MISS";
}
# Remove internal ESI header
unset resp.http.X-ESI;
}
Configure your backend application for ESI
Modify your application to include ESI tags in HTML responses. Here's an example with different fragment types.
ESI Example Page
My Website
Article Content
This is the main article content that changes rarely.
Create fragment endpoints in your application
Implement the fragment endpoints that ESI will call. Each fragment should return just the HTML snippet for that section.
# /fragments/user-info response:
Welcome, John Doe
Logout
/fragments/recent-posts response:
Recent Posts
/fragments/cart-summary response:
Set fragment-specific cache headers
Configure your application to return appropriate cache headers for different fragment types.
# User-specific fragments (short cache)
Cache-Control: max-age=60, private
Vary: Cookie, Authorization
General dynamic content (medium cache)
Cache-Control: max-age=300, public
Vary: Accept-Encoding
Semi-static content (long cache)
Cache-Control: max-age=1800, public
Vary: Accept-Encoding
Real-time data (no cache)
Cache-Control: no-cache, must-revalidate
Pragma: no-cache
Enable and start Varnish
Reload systemd configuration and start Varnish with the new ESI-enabled settings.
sudo systemctl daemon-reload
sudo systemctl enable --now varnish
sudo systemctl status varnish
Configure your backend application port
Ensure your application runs on port 8080 since Varnish will handle port 80. For example, if using Nginx as the backend:
server {
listen 8080 default_server;
listen [::]:8080 default_server;
root /var/www/html;
index index.html index.php;
server_name example.com;
location / {
try_files $uri $uri/ =404;
}
# Fragment endpoints
location /fragments/ {
# Add cache headers for fragments
add_header Cache-Control "max-age=300, public";
try_files $uri $uri/ @app;
}
location @app {
# Proxy to your application server
proxy_pass http://127.0.0.1:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
sudo systemctl restart nginx
Verify your setup
Test that Varnish is properly processing ESI tags and caching fragments separately.
# Check Varnish is running and ESI is enabled
sudo systemctl status varnish
varnishd -V | grep -i esi
Test ESI processing with curl
curl -H "Host: example.com" http://localhost/ -v
Check cache headers
curl -H "Host: example.com" http://localhost/page/test -I
Test fragment caching separately
curl -H "Host: example.com" http://localhost/fragments/user-info -I
Monitor Varnish cache hits
sudo varnishstat -f MAIN.cache_hit,MAIN.cache_miss
Watch ESI processing in real-time
sudo varnishlog -q "VCL_call eq 'ESI'"
Optimize ESI caching strategies
Implement conditional ESI includes
Use ESI conditional statements to include different content based on request parameters.
Configure Vary headers for ESI fragments
Update your VCL to handle Vary headers properly for fragments that depend on specific request headers.
sub vcl_hash {
hash_data(req.url);
# Include specific cookies in hash for user-specific fragments
if (req.url ~ "^/fragments/(user-|cart-|profile-)" && req.http.Cookie ~ "user_id=([^;]+)") {
hash_data(regsub(req.http.Cookie, ".user_id=([^;]+).", "\1"));
}
# Include session ID for session-specific fragments
if (req.url ~ "^/fragments/cart-" && req.http.Cookie ~ "session_id=([^;]+)") {
hash_data(regsub(req.http.Cookie, ".session_id=([^;]+).", "\1"));
}
# Always include host
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}
sub vcl_backend_response {
# Set appropriate Vary headers for different fragment types
if (bereq.url ~ "^/fragments/user-") {
set beresp.http.Vary = "Cookie, User-Agent";
set beresp.ttl = 300s;
}
if (bereq.url ~ "^/fragments/public-") {
set beresp.http.Vary = "Accept-Encoding";
set beresp.ttl = 3600s;
}
if (bereq.url ~ "^/fragments/realtime-") {
set beresp.ttl = 30s;
set beresp.http.Cache-Control = "max-age=30";
}
}
Add ESI error handling
Implement graceful fallbacks when ESI fragments fail to load.
Pricing information temporarily unavailable.
Performance optimization
Configure fragment cache warming
Create a script to pre-warm frequently accessed fragments.
#!/bin/bash
Common fragments to warm
fragments=(
"/fragments/popular-products"
"/fragments/latest-news"
"/fragments/weather"
"/fragments/stock-ticker"
)
Base URL for your site
BASE_URL="http://localhost"
echo "Warming ESI fragment cache..."
for fragment in "${fragments[@]}"; do
echo "Warming: $fragment"
curl -s -o /dev/null "$BASE_URL$fragment"
sleep 0.1
done
echo "Cache warming completed"
Show cache statistics
varnishstat -1 -f MAIN.cache_hit,MAIN.cache_miss
sudo chmod +x /usr/local/bin/warm-esi-cache.sh
Add to crontab for regular warming
echo "/5 * /usr/local/bin/warm-esi-cache.sh > /dev/null 2>&1" | sudo crontab -
Monitor ESI performance
Set up monitoring to track ESI cache effectiveness and fragment performance.
#!/bin/bash
Monitor ESI-specific metrics
echo "=== ESI Cache Statistics ==="
varnishstat -1 -f MAIN.esi_errors,MAIN.esi_warnings,MAIN.cache_hit,MAIN.cache_miss
echo -e "\n=== Fragment Response Times ==="
Test fragment response times
for fragment in "/fragments/user-info" "/fragments/cart-summary" "/fragments/weather"; do
time_ms=$(curl -s -w "%{time_total}" -o /dev/null "http://localhost$fragment" | awk '{print $1 * 1000}')
echo "$fragment: ${time_ms}ms"
done
echo -e "\n=== Recent ESI Errors ==="
varnishlog -d -q "VCL_Error ~ 'ESI'" | tail -10
sudo chmod +x /usr/local/bin/esi-monitor.sh
./usr/local/bin/esi-monitor.sh
Common issues
| Symptom | Cause | Fix |
|---|---|---|
| ESI tags appear in HTML output | ESI processing disabled | Add -p feature=+esi to varnishd command and set beresp.do_esi = true |
| Fragments not caching separately | Missing cache headers | Set proper TTL in VCL: set beresp.ttl = 300s; for fragments |
| User-specific content cached globally | Hash doesn't include user context | Add user ID to hash in vcl_hash subroutine |
| High backend load from fragments | Fragments not cached effectively | Check fragment URLs match VCL patterns and have proper cache headers |
| ESI includes return 404 | Fragment endpoints not implemented | Implement all fragment URLs referenced in ESI tags |
| Slow page rendering | Too many serial ESI includes | Use parallel processing and set timeouts: .first_byte_timeout = 10s |
| Cache not invalidating | Missing cache purge configuration | Implement cache purging: ban req.url ~ "^/fragments/" |
Next steps
- Set up Varnish 7 cluster with load balancing across multiple backends for high availability
- Configure NGINX reverse proxy with SSL termination and load balancing as your backend
- Implement Varnish cache warming with automated content preloading for better performance
- Configure Varnish monitoring with Prometheus and Grafana dashboards
- Setup Varnish SSL termination with NGINX backend
Running this in production?
Automated install script
Run this to automate the entire setup
#!/usr/bin/env bash
set -euo pipefail
# Production-quality Varnish ESI installation script
# Supports Ubuntu, Debian, AlmaLinux, Rocky Linux, CentOS
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
# Configuration
BACKEND_PORT="${1:-8080}"
VARNISH_PORT="${2:-80}"
CACHE_SIZE="${3:-1G}"
usage() {
echo "Usage: $0 [backend_port] [varnish_port] [cache_size]"
echo "Example: $0 8080 80 1G"
exit 1
}
log() {
echo -e "${GREEN}[INFO]${NC} $1"
}
warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
error() {
echo -e "${RED}[ERROR]${NC} $1"
exit 1
}
cleanup() {
if [ $? -ne 0 ]; then
warn "Installation failed. Cleaning up..."
systemctl stop varnish 2>/dev/null || true
systemctl disable varnish 2>/dev/null || true
fi
}
trap cleanup ERR
# Check prerequisites
if [[ $EUID -ne 0 ]]; then
error "This script must be run as root"
fi
# Validate arguments
if [[ "$BACKEND_PORT" =~ ^[0-9]+$ ]] && [ "$BACKEND_PORT" -ge 1 ] && [ "$BACKEND_PORT" -le 65535 ]; then
:
else
usage
fi
if [[ "$VARNISH_PORT" =~ ^[0-9]+$ ]] && [ "$VARNISH_PORT" -ge 1 ] && [ "$VARNISH_PORT" -le 65535 ]; then
:
else
usage
fi
# Detect distribution
if [ -f /etc/os-release ]; then
. /etc/os-release
case "$ID" in
ubuntu|debian)
PKG_MGR="apt"
PKG_INSTALL="apt install -y"
PKG_UPDATE="apt update && apt upgrade -y"
DISTRO="debian"
;;
almalinux|rocky|centos|rhel|ol|fedora)
PKG_MGR="dnf"
PKG_INSTALL="dnf install -y"
PKG_UPDATE="dnf update -y"
DISTRO="rhel"
;;
amzn)
PKG_MGR="yum"
PKG_INSTALL="yum install -y"
PKG_UPDATE="yum update -y"
DISTRO="rhel"
;;
*)
error "Unsupported distribution: $ID"
;;
esac
else
error "Cannot detect distribution. /etc/os-release not found"
fi
log "Detected distribution: $ID ($DISTRO family)"
# Step 1: Update system packages
echo "[1/6] Updating system packages..."
$PKG_UPDATE
# Step 2: Install prerequisites
echo "[2/6] Installing prerequisites..."
if [ "$DISTRO" = "debian" ]; then
$PKG_INSTALL curl gnupg2 lsb-release
else
$PKG_INSTALL curl gnupg2
fi
# Step 3: Install Varnish Cache 7
echo "[3/6] Installing Varnish Cache 7..."
if [ "$DISTRO" = "debian" ]; then
curl -fsSL https://packagecloud.io/varnishcache/varnish70/gpgkey | gpg --dearmor -o /usr/share/keyrings/varnish-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/varnish-archive-keyring.gpg] https://packagecloud.io/varnishcache/varnish70/ubuntu/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/varnish-cache.list
chmod 644 /etc/apt/sources.list.d/varnish-cache.list
apt update
$PKG_INSTALL varnish
else
curl -fsSL https://packagecloud.io/varnishcache/varnish70/gpgkey | rpm --import -
cat > /etc/yum.repos.d/varnish70.repo << 'EOF'
[varnish70]
name=Varnish 7.0 Repository
baseurl=https://packagecloud.io/varnishcache/varnish70/el/9/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://packagecloud.io/varnishcache/varnish70/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF
chmod 644 /etc/yum.repos.d/varnish70.repo
$PKG_INSTALL varnish
fi
# Step 4: Configure systemd service
echo "[4/6] Configuring Varnish daemon settings..."
cat > /etc/systemd/system/varnish.service << EOF
[Unit]
Description=Varnish Cache HTTP reverse proxy
After=network.target
[Service]
Type=exec
ExecStart=/usr/sbin/varnishd -a :${VARNISH_PORT} -f /etc/varnish/default.vcl -s malloc,${CACHE_SIZE} -p feature=+esi
ExecReload=/bin/kill -HUP \$MAINPID
User=varnish
Group=varnish
PrivateDevices=yes
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/varnish
Restart=always
[Install]
WantedBy=multi-user.target
EOF
chmod 644 /etc/systemd/system/varnish.service
# Step 5: Create ESI-enabled VCL configuration
echo "[5/6] Creating ESI-enabled VCL configuration..."
cat > /etc/varnish/default.vcl << EOF
vcl 4.1;
backend default {
.host = "127.0.0.1";
.port = "${BACKEND_PORT}";
.connect_timeout = 10s;
.first_byte_timeout = 30s;
.between_bytes_timeout = 5s;
}
sub vcl_recv {
# Enable ESI processing for specific paths
if (req.url ~ "^/(page|content)/") {
set req.http.X-ESI = "true";
}
# Pass through ESI fragment requests
if (req.url ~ "^/fragments/") {
return (pass);
}
# Remove cookies for static assets
if (req.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg)$") {
unset req.http.Cookie;
}
}
sub vcl_backend_response {
# Enable ESI processing for marked requests
if (bereq.http.X-ESI == "true") {
set beresp.do_esi = true;
}
# Cache fragments with specific TTL
if (bereq.url ~ "^/fragments/") {
set beresp.ttl = 300s;
set beresp.http.Cache-Control = "max-age=300";
}
# Cache main pages with ESI enabled
if (bereq.url ~ "^/(page|content)/" && beresp.http.Content-Type ~ "text/html") {
set beresp.ttl = 1800s;
set beresp.http.Cache-Control = "max-age=1800";
}
# Static assets caching
if (bereq.url ~ "\.(css|js|png|jpg|jpeg|gif|ico|svg)$") {
set beresp.ttl = 86400s;
set beresp.http.Cache-Control = "max-age=86400";
}
}
sub vcl_deliver {
# Add debug headers
if (obj.hits > 0) {
set resp.http.X-Varnish-Cache = "HIT";
} else {
set resp.http.X-Varnish-Cache = "MISS";
}
set resp.http.X-Varnish-Hits = obj.hits;
}
EOF
chown root:root /etc/varnish/default.vcl
chmod 644 /etc/varnish/default.vcl
# Configure firewall
if command -v firewall-cmd &> /dev/null && systemctl is-active firewalld &> /dev/null; then
log "Configuring firewall (firewalld)..."
firewall-cmd --permanent --add-port=${VARNISH_PORT}/tcp
firewall-cmd --reload
elif command -v ufw &> /dev/null; then
log "Configuring firewall (ufw)..."
ufw allow ${VARNISH_PORT}/tcp
fi
# Step 6: Enable and start services
echo "[6/6] Starting and enabling Varnish service..."
systemctl daemon-reload
systemctl enable varnish
systemctl start varnish
# Verification
log "Verifying installation..."
if systemctl is-active --quiet varnish; then
log "✓ Varnish service is running"
else
error "✗ Varnish service failed to start"
fi
if netstat -tlnp 2>/dev/null | grep -q ":${VARNISH_PORT}.*varnishd" || ss -tlnp 2>/dev/null | grep -q ":${VARNISH_PORT}.*varnishd"; then
log "✓ Varnish is listening on port ${VARNISH_PORT}"
else
error "✗ Varnish is not listening on port ${VARNISH_PORT}"
fi
log "Installation completed successfully!"
log "Configuration:"
log " - Varnish listening on port: ${VARNISH_PORT}"
log " - Backend port: ${BACKEND_PORT}"
log " - Cache size: ${CACHE_SIZE}"
log " - ESI processing: enabled"
log ""
log "Next steps:"
log "1. Configure your backend application to run on port ${BACKEND_PORT}"
log "2. Test ESI functionality with fragments"
log "3. Monitor cache performance with 'varnishstat'"
Review the script before running. Execute with: bash install.sh