Monitor Jetty performance with JMX and Grafana dashboards

Update system packages

Start by updating your package manager to ensure you get the latest versions of all components.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install Java and required packages

Jetty requires Java 11 or later. Install OpenJDK along with wget for downloading components.

sudo apt install -y openjdk-17-jdk wget curl unzip

sudo dnf install -y java-17-openjdk-devel wget curl unzip

Download and install Jetty 12

Download Jetty 12 from the official repository and extract it to a dedicated directory.

cd /opt
sudo wget https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-home/12.0.5/jetty-home-12.0.5.tar.gz
sudo tar -xzf jetty-home-12.0.5.tar.gz
sudo mv jetty-home-12.0.5 jetty-home
sudo chown -R jetty:jetty /opt/jetty-home

Create Jetty user and directories

Create a dedicated user for running Jetty and set up the required directory structure.

sudo useradd -r -s /bin/false jetty
sudo mkdir -p /opt/jetty-base
sudo mkdir -p /opt/jetty-base/webapps
sudo mkdir -p /opt/jetty-base/logs
sudo chown -R jetty:jetty /opt/jetty-base

Configure Jetty base directory

Initialize the Jetty base directory with required modules including JMX support.

cd /opt/jetty-base
sudo -u jetty java -jar /opt/jetty-home/start.jar --add-module=server,http,deploy,jsp,jstl,websocket,jmx

Download Prometheus JMX Exporter

Download the JMX Exporter JAR file that will collect Jetty metrics for Prometheus.

sudo mkdir -p /opt/jmx-exporter
cd /opt/jmx-exporter
sudo wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.20.0/jmx_prometheus_javaagent-0.20.0.jar
sudo chown jetty:jetty jmx_prometheus_javaagent-0.20.0.jar

Create JMX Exporter configuration

Configure the JMX Exporter to collect specific Jetty metrics including connection pools, threads, and request statistics.

rules:
  # Jetty Server metrics
  - pattern: 'org.eclipse.jetty.server<>(.+):(.+)'
    name: jetty_server_$3
    labels:
      type: "$1"
      id: "$2"
      attribute: "$4"
  
  # Jetty Handler metrics
  - pattern: 'org.eclipse.jetty.server.handler<>(.+)'
    name: jetty_handler_$3
    labels:
      type: "$1"
      id: "$2"
  
  # Jetty ThreadPool metrics
  - pattern: 'org.eclipse.jetty.util.thread<>(.+)'
    name: jetty_threadpool_$3
    labels:
      type: "$1"
      id: "$2"
  
  # Jetty Connector metrics
  - pattern: 'org.eclipse.jetty.server<>(.+)'
    name: jetty_connector_$4
    labels:
      type: "$1"
      context: "$2"
      id: "$3"
  
  # JVM metrics
  - pattern: 'java.lang(.+)'
    name: jvm_memory_heap_$1
  - pattern: 'java.lang(.+)'
    name: jvm_memory_nonheap_$1
  - pattern: 'java.lang<>CollectionCount'
    name: jvm_gc_collections_total
    labels:
      gc: "$1"
  - pattern: 'java.lang<>CollectionTime'
    name: jvm_gc_collection_time_seconds
    labels:
      gc: "$1"
    valueFactor: 0.001

Configure Jetty JVM options

Create a configuration file to enable JMX and attach the Prometheus JMX Exporter to Jetty.

# Enable JMX
--module=jmx

JMX configuration
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

JMX Exporter configuration
-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.20.0.jar=8081:/opt/jmx-exporter/jetty-config.yaml

Create systemd service for Jetty

Set up a systemd service to manage Jetty with proper user permissions and automatic startup.

[Unit]
Description=Jetty Web Server
After=network.target

[Service]
Type=forking
User=jetty
Group=jetty
Environment="JETTY_HOME=/opt/jetty-home"
Environment="JETTY_BASE=/opt/jetty-base"
Environment="JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64"
WorkingDirectory=/opt/jetty-base
ExecStart=/usr/bin/java -jar /opt/jetty-home/start.jar
ExecStop=/usr/bin/java -jar /opt/jetty-home/start.jar --stop
PIDFile=/opt/jetty-base/jetty.pid
Restart=on-failure
RestartSec=10

Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/jetty-base

[Install]
WantedBy=multi-user.target

Install Prometheus

Install Prometheus to collect metrics from the JMX Exporter endpoint.

sudo apt install -y prometheus

sudo dnf install -y prometheus2

Configure Prometheus to scrape Jetty metrics

Add the JMX Exporter endpoint to Prometheus configuration for metric collection.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'jetty'
    static_configs:
      - targets: ['localhost:8081']
    scrape_interval: 10s
    metrics_path: /metrics
    static_configs:
      - targets: ['localhost:8081']
        labels:
          instance: 'jetty-server'
          service: 'jetty'

Install and configure Grafana

Install Grafana for creating dashboards and visualization of Jetty metrics.

sudo apt install -y software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana

sudo dnf install -y https://dl.grafana.com/oss/release/grafana-10.2.3-1.x86_64.rpm

Enable and start all services

Start Jetty, Prometheus, and Grafana services and enable them for automatic startup.

sudo systemctl daemon-reload
sudo systemctl enable --now jetty
sudo systemctl enable --now prometheus
sudo systemctl enable --now grafana-server

Configure firewall rules

Open the necessary ports for Jetty web interface, JMX metrics, Prometheus, and Grafana.

sudo ufw allow 8080/tcp comment 'Jetty Web Server'
sudo ufw allow 8081/tcp comment 'JMX Exporter'
sudo ufw allow 9090/tcp comment 'Prometheus'
sudo ufw allow 3000/tcp comment 'Grafana'

sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --permanent --add-port=8081/tcp
sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --reload

Create Grafana data source

Configure Prometheus as a data source in Grafana for accessing Jetty metrics.

curl -X POST -H "Content-Type: application/json" -d '{
  "name": "Prometheus",
  "type": "prometheus",
  "url": "http://localhost:9090",
  "access": "proxy",
  "basicAuth": false
}' http://admin:admin@localhost:3000/api/datasources

Import Jetty dashboard

Create a comprehensive Grafana dashboard for monitoring Jetty performance metrics.

{
  "dashboard": {
    "id": null,
    "title": "Jetty Performance Monitoring",
    "tags": ["jetty", "jmx", "performance"],
    "timezone": "browser",
    "panels": [
      {
        "id": 1,
        "title": "HTTP Requests Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(jetty_server_requests_total[5m])",
            "legendFormat": "Requests/sec"
          }
        ],
        "yAxes": [
          {
            "label": "Requests per second"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0}
      },
      {
        "id": 2,
        "title": "Response Times",
        "type": "graph",
        "targets": [
          {
            "expr": "jetty_server_response_time_seconds",
            "legendFormat": "Response Time"
          }
        ],
        "yAxes": [
          {
            "label": "Seconds"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
      },
      {
        "id": 3,
        "title": "Thread Pool Status",
        "type": "graph",
        "targets": [
          {
            "expr": "jetty_threadpool_threads",
            "legendFormat": "Total Threads"
          },
          {
            "expr": "jetty_threadpool_idle_threads",
            "legendFormat": "Idle Threads"
          },
          {
            "expr": "jetty_threadpool_busy_threads",
            "legendFormat": "Busy Threads"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8}
      },
      {
        "id": 4,
        "title": "JVM Memory Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "jvm_memory_heap_used",
            "legendFormat": "Heap Used"
          },
          {
            "expr": "jvm_memory_heap_max",
            "legendFormat": "Heap Max"
          }
        ],
        "yAxes": [
          {
            "label": "Bytes",
            "logBase": 1
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
      }
    ],
    "time": {
      "from": "now-1h",
      "to": "now"
    },
    "refresh": "10s"
  }
}

Import the dashboard using the Grafana API:

curl -X POST -H "Content-Type: application/json" -d @/tmp/jetty-dashboard.json \
  http://admin:admin@localhost:3000/api/dashboards/db

Configure alerting rules

Set up Prometheus alerting rules for critical Jetty performance thresholds.

groups:
  - name: jetty.rules
    rules:
      - alert: JettyHighResponseTime
        expr: jetty_server_response_time_seconds > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Jetty response time is high"
          description: "Response time has been above 2 seconds for more than 5 minutes"
      
      - alert: JettyThreadPoolExhaustion
        expr: (jetty_threadpool_busy_threads / jetty_threadpool_threads) > 0.9
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Jetty thread pool near exhaustion"
          description: "Thread pool utilization is above 90%"
      
      - alert: JettyHighMemoryUsage
        expr: (jvm_memory_heap_used / jvm_memory_heap_max) > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Jetty JVM memory usage high"
          description: "JVM heap memory usage is above 85%"
      
      - alert: JettyDown
        expr: up{job="jetty"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Jetty server is down"
          description: "Jetty server has been down for more than 1 minute"

Update Prometheus configuration for alerts

Add the alerting rules to Prometheus configuration and restart the service.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "jetty-alerts.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'jetty'
    static_configs:
      - targets: ['localhost:8081']
    scrape_interval: 10s
    metrics_path: /metrics
    static_configs:
      - targets: ['localhost:8081']
        labels:
          instance: 'jetty-server'
          service: 'jetty'

Restart Prometheus to apply the new configuration:

sudo systemctl restart prometheus