Configure Jetty clustering and session replication for high availability

Install Java Development Kit

Jetty 12 requires Java 17 or later. Install OpenJDK which provides the runtime environment for Jetty.

sudo apt update
sudo apt install -y openjdk-17-jdk wget curl unzip

sudo dnf update -y
sudo dnf install -y java-17-openjdk-devel wget curl unzip

Create Jetty user and directories

Create a dedicated system user for Jetty with proper directory structure for security isolation.

sudo useradd --system --shell /bin/false --home /opt/jetty --create-home jetty
sudo mkdir -p /opt/jetty/{jetty-home,jetty-base1,jetty-base2}
sudo mkdir -p /var/log/jetty
sudo chown -R jetty:jetty /opt/jetty /var/log/jetty

Download and extract Jetty 12

Download the latest Jetty 12 release and extract it to the jetty-home directory.

cd /tmp
wget https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-home/12.0.5/jetty-home-12.0.5.tar.gz
tar -xzf jetty-home-12.0.5.tar.gz
sudo mv jetty-home-12.0.5/* /opt/jetty/jetty-home/
sudo chown -R jetty:jetty /opt/jetty/jetty-home

Install PostgreSQL for session storage

Set up PostgreSQL as the shared session store that all Jetty cluster nodes will use for session persistence.

sudo apt install -y postgresql postgresql-contrib

sudo dnf install -y postgresql-server postgresql-contrib
sudo postgresql-setup --initdb

Configure PostgreSQL for session storage

Create a database and user for Jetty session storage with appropriate permissions.

sudo systemctl enable --now postgresql
sudo -u postgres createdb jetty_sessions
sudo -u postgres psql -c "CREATE USER jetty_user WITH PASSWORD 'SecurePassword123!';"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE jetty_sessions TO jetty_user;"

Configure first Jetty instance

Set up the first Jetty base directory with clustering and session replication modules enabled.

sudo -u jetty bash -c 'cd /opt/jetty/jetty-base1 && java -jar ../jetty-home/start.jar --add-modules=server,http,deploy,jsp,jstl,websocket,session-store-jdbc'
sudo -u jetty bash -c 'cd /opt/jetty/jetty-base1 && java -jar ../jetty-home/start.jar --add-modules=jdbc-pool,postgresql'

Configure session storage for first instance

Configure JDBC session storage with PostgreSQL connection details for the first Jetty instance.

# Session store configuration
jetty.session.jdbc.datasourceName=sessions
jetty.session.jdbc.schema=jetty_sessions
jetty.session.jdbc.table=JettySessions
jetty.session.savePeriod=30
jetty.session.gracePeriod=3600
jetty.session.idleSavePeriod=300

Configure PostgreSQL connection pool

Set up the database connection pool configuration for session persistence.

# PostgreSQL connection configuration
db.url=jdbc:postgresql://localhost:5432/jetty_sessions
db.username=jetty_user
db.password=SecurePassword123!
db.driver=org.postgresql.Driver
db.datasource=sessions

Configure first instance ports

Set specific ports for the first Jetty instance to avoid conflicts when running multiple instances.

# HTTP configuration for instance 1
jetty.http.port=8080
jetty.http.host=0.0.0.0
jetty.http.acceptors=2
jetty.http.selectors=4

Configure second Jetty instance

Create the second Jetty instance with identical clustering configuration but different ports.

sudo -u jetty bash -c 'cd /opt/jetty/jetty-base2 && java -jar ../jetty-home/start.jar --add-modules=server,http,deploy,jsp,jstl,websocket,session-store-jdbc'
sudo -u jetty bash -c 'cd /opt/jetty/jetty-base2 && java -jar ../jetty-home/start.jar --add-modules=jdbc-pool,postgresql'

Configure session storage for second instance

Copy the session storage configuration to the second instance with identical settings.

# Session store configuration
jetty.session.jdbc.datasourceName=sessions
jetty.session.jdbc.schema=jetty_sessions
jetty.session.jdbc.table=JettySessions
jetty.session.savePeriod=30
jetty.session.gracePeriod=3600
jetty.session.idleSavePeriod=300

Configure PostgreSQL connection for second instance

Set up identical database connection configuration for the second instance.

# PostgreSQL connection configuration
db.url=jdbc:postgresql://localhost:5432/jetty_sessions
db.username=jetty_user
db.password=SecurePassword123!
db.driver=org.postgresql.Driver
db.datasource=sessions

Configure second instance ports

Set different ports for the second instance to run alongside the first instance.

# HTTP configuration for instance 2
jetty.http.port=8081
jetty.http.host=0.0.0.0
jetty.http.acceptors=2
jetty.http.selectors=4

Download PostgreSQL JDBC driver

Install the PostgreSQL JDBC driver in both Jetty instances for database connectivity.

cd /tmp
wget https://jdbc.postgresql.org/download/postgresql-42.7.1.jar
sudo cp postgresql-42.7.1.jar /opt/jetty/jetty-base1/lib/
sudo cp postgresql-42.7.1.jar /opt/jetty/jetty-base2/lib/
sudo chown jetty:jetty /opt/jetty/jetty-base*/lib/postgresql-42.7.1.jar

Create systemd service for first instance

Set up systemd service files to manage Jetty instances as system services with automatic startup.

[Unit]
Description=Jetty HTTP Server Instance 1
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=jetty
Group=jetty
WorkingDirectory=/opt/jetty/jetty-base1
ExecStart=/usr/bin/java -server -Xmx512m -Xms256m -XX:+UseG1GC -jar /opt/jetty/jetty-home/start.jar
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=jetty1
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Create systemd service for second instance

Create a similar systemd service for the second Jetty instance with different working directory.

[Unit]
Description=Jetty HTTP Server Instance 2
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=jetty
Group=jetty
WorkingDirectory=/opt/jetty/jetty-base2
ExecStart=/usr/bin/java -server -Xmx512m -Xms256m -XX:+UseG1GC -jar /opt/jetty/jetty-home/start.jar
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=jetty2
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Install and configure HAProxy load balancer

Set up HAProxy as a load balancer to distribute traffic between Jetty instances with health checking.

sudo apt install -y haproxy

sudo dnf install -y haproxy

Configure HAProxy for Jetty cluster

Configure HAProxy with sticky sessions and health checks to properly distribute traffic while maintaining session affinity.

global
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    
defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option httplog
    
frontend jetty_frontend
    bind *:80
    default_backend jetty_cluster
    
backend jetty_cluster
    balance roundrobin
    cookie JSESSIONID prefix nocache
    option httpchk GET /
    http-check expect status 200
    server jetty1 127.0.0.1:8080 check cookie jetty1
    server jetty2 127.0.0.1:8081 check cookie jetty2
    
frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 30s

Create sample web application

Create a simple web application that demonstrates session replication by showing session data and server information.

sudo mkdir -p /opt/jetty/jetty-base1/webapps/session-test
sudo mkdir -p /opt/jetty/jetty-base2/webapps/session-test

Create test application files

Create a JSP page that displays session information and allows testing session replication across cluster nodes.

<%@ page import="java.util.,java.net." %>
<%@ page session="true" %>



Jetty Cluster Session Test
Server: <%= InetAddress.getLocalHost().getHostName() %>
Port: <%= request.getLocalPort() %>
Session ID: <%= session.getId() %>
Creation Time: <%= new Date(session.getCreationTime()) %>
Last Access: <%= new Date(session.getLastAccessedTime()) %>
<%
    Integer count = (Integer) session.getAttribute("count");
    if (count == null) {
        count = 0;
    }
    count++;
    session.setAttribute("count", count);
%>
Visit Count: <%= count %>

    
    

<%
    String message = request.getParameter("message");
    if (message != null && !message.trim().isEmpty()) {
        session.setAttribute("userMessage", message);
    }
    String storedMessage = (String) session.getAttribute("userMessage");
    if (storedMessage != null) {
%>
Stored Message: <%= storedMessage %>
<% } %>
Refresh Page

Copy test application to second instance

Copy the test application to the second Jetty instance to ensure both nodes serve the same content.

sudo cp -r /opt/jetty/jetty-base1/webapps/session-test/* /opt/jetty/jetty-base2/webapps/session-test/
sudo chown -R jetty:jetty /opt/jetty/jetty-base*/webapps/

Enable and start all services

Start PostgreSQL, both Jetty instances, and HAProxy in the correct order with proper dependencies.

sudo systemctl daemon-reload
sudo systemctl enable --now postgresql
sudo systemctl enable --now jetty1 jetty2
sudo systemctl enable --now haproxy

Configure firewall rules

Open necessary firewall ports for HTTP traffic and HAProxy statistics while maintaining security.

sudo ufw allow 80/tcp
sudo ufw allow 8404/tcp

sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-port=8404/tcp
sudo firewall-cmd --reload

Set up log monitoring

Configure centralized logging for all cluster components to track performance and diagnose issues.

# Jetty cluster logging
:programname, isequal, "jetty1" /var/log/jetty/jetty1.log
:programname, isequal, "jetty2" /var/log/jetty/jetty2.log
& stop

Configure log rotation

Set up automatic log rotation to prevent disk space issues while maintaining log history for troubleshooting.

/var/log/jetty/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 jetty jetty
    postrotate
        systemctl reload rsyslog
    endscript
}

Create cluster health check script

Create a monitoring script that checks the health of all cluster components and session replication status.

#!/bin/bash
set -euo pipefail

echo "=== Jetty Cluster Health Check ==="
echo "Timestamp: $(date)"
echo

Check PostgreSQL
echo "PostgreSQL Status:"
if systemctl is-active --quiet postgresql; then
    echo "✓ PostgreSQL is running"
    sudo -u postgres psql -d jetty_sessions -c "SELECT COUNT(*) as session_count FROM JettySessions;" 2>/dev/null || echo "⚠ Session table not initialized"
else
    echo "✗ PostgreSQL is not running"
fi
echo

Check Jetty instances
for instance in jetty1 jetty2; do
    port=$([[ "$instance" == "jetty1" ]] && echo 8080 || echo 8081)
    echo "$instance Status:"
    if systemctl is-active --quiet $instance; then
        echo "✓ $instance is running"
        if curl -s -f http://localhost:$port/ > /dev/null; then
            echo "✓ $instance HTTP endpoint responsive"
        else
            echo "✗ $instance HTTP endpoint not responding"
        fi
    else
        echo "✗ $instance is not running"
    fi
    echo
done

Check HAProxy
echo "HAProxy Status:"
if systemctl is-active --quiet haproxy; then
    echo "✓ HAProxy is running"
    if curl -s -f http://localhost/ > /dev/null; then
        echo "✓ Load balancer endpoint responsive"
    else
        echo "✗ Load balancer endpoint not responding"
    fi
else
    echo "✗ HAProxy is not running"
fi
echo

Check backend health via HAProxy stats
echo "Backend Health (via HAProxy):"
curl -s http://localhost:8404/stats \;csv | grep jetty | while IFS=, read -r name queue cur max limit total_time; do
    echo "Backend $name: $cur/$max connections"
done 2>/dev/null || echo "Could not retrieve backend statistics"

echo
echo "=== Health Check Complete ==="

Make health check script executable

Set proper permissions for the health check script and create a systemd timer for regular monitoring.

sudo chmod +x /usr/local/bin/jetty-cluster-health.sh
sudo chown root:root /usr/local/bin/jetty-cluster-health.sh

Create automated monitoring timer

Set up systemd timer to run health checks automatically and log results for proactive monitoring.

[Unit]
Description=Jetty Cluster Health Check

[Service]
Type=oneshot
ExecStart=/usr/local/bin/jetty-cluster-health.sh
User=root
StandardOutput=journal
StandardError=journal

Create monitoring timer schedule

Configure the timer to run health checks every 5 minutes for continuous cluster monitoring.

[Unit]
Description=Run Jetty Cluster Health Check every 5 minutes
Requires=jetty-health-check.service

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

Enable monitoring timer

Start the automated health check timer to begin continuous monitoring of the Jetty cluster.

sudo systemctl daemon-reload
sudo systemctl enable --now jetty-health-check.timer
sudo systemctl restart rsyslog

Optimize JVM memory settings

Tune JVM memory allocation based on your application requirements and available system resources.

sudo systemctl edit jetty1

[Service]
ExecStart=
ExecStart=/usr/bin/java -server -Xmx1g -Xms512m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+UseStringDeduplication -jar /opt/jetty/jetty-home/start.jar

Configure connection pool tuning

Optimize PostgreSQL connection pooling for better performance under load with proper sizing.

# PostgreSQL connection configuration with pooling
db.url=jdbc:postgresql://localhost:5432/jetty_sessions
db.username=jetty_user
db.password=SecurePassword123!
db.driver=org.postgresql.Driver
db.datasource=sessions
db.pool.maxConnections=20
db.pool.minConnections=5
db.pool.maxIdleTime=300
db.pool.testQuery=SELECT 1

Configure session cleanup

Set up automatic cleanup of expired sessions to prevent database growth and maintain performance.

#!/bin/bash
set -euo pipefail

Clean up expired sessions (older than 4 hours)
sudo -u postgres psql -d jetty_sessions -c "DELETE FROM JettySessions WHERE accessTime < EXTRACT(EPOCH FROM NOW() - INTERVAL '4 hours') * 1000;"

echo "Session cleanup completed at $(date)"

Create session cleanup timer

Schedule automatic session cleanup to run every hour to maintain database performance.

sudo chmod +x /usr/local/bin/jetty-session-cleanup.sh

[Unit]
Description=Clean up expired Jetty sessions

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl enable --now jetty-session-cleanup.timer