Configure Jaeger with Elasticsearch backend security and encryption

Install required packages

Install Elasticsearch, Jaeger components, and certificate management tools needed for secure configuration.

sudo apt update
sudo apt install -y elasticsearch jaeger-collector jaeger-query jaeger-agent openssl curl

sudo dnf update -y
sudo dnf install -y elasticsearch jaeger-collector jaeger-query jaeger-agent openssl curl

Configure Elasticsearch X-Pack security

Enable X-Pack security features in Elasticsearch to provide authentication and authorization capabilities for Jaeger access.

cluster.name: jaeger-cluster
node.name: jaeger-node-1
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

Enable X-Pack security
xpack.security.enabled: true
xpack.security.enrollment.enabled: true

TLS configuration
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: certs/transport.p12
xpack.security.transport.ssl.truststore.path: certs/transport.p12

Audit logging for security events
xpack.security.audit.enabled: true

Generate TLS certificates

Create certificate authority and TLS certificates for secure communication between Jaeger and Elasticsearch components.

sudo mkdir -p /etc/elasticsearch/certs
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/certs
sudo chmod 750 /etc/elasticsearch/certs

Generate CA and certificates
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /etc/elasticsearch/certs/elastic-ca.p12 --pass ""
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-ca.p12 --ca-pass "" --out /etc/elasticsearch/certs/http.p12 --pass ""
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-ca.p12 --ca-pass "" --out /etc/elasticsearch/certs/transport.p12 --pass ""

Set appropriate permissions
sudo chown elasticsearch:elasticsearch /etc/elasticsearch/certs/*
sudo chmod 660 /etc/elasticsearch/certs/*

Create Jaeger service account

Set up dedicated Elasticsearch user account for Jaeger with minimal required permissions for tracing data access.

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

Wait for Elasticsearch to start
sleep 30

Reset built-in user passwords
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto > /tmp/es-passwords.txt

Extract elastic password for admin operations
ELASTIC_PASSWORD=$(grep "PASSWORD elastic" /tmp/es-passwords.txt | awk '{print $4}')
echo "Elastic password: $ELASTIC_PASSWORD"

Create Jaeger user and roles

Configure role-based access control with custom roles for Jaeger tracing operations and read-only access patterns.

# Create jaeger_writer role for collector
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/role/jaeger_writer" -H "Content-Type: application/json" -d '
{
  "indices": [
    {
      "names": ["jaeger-*"],
      "privileges": ["create", "index", "write", "delete", "manage"]
    }
  ]
}'

Create jaeger_reader role for query service
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/role/jaeger_reader" -H "Content-Type: application/json" -d '
{
  "indices": [
    {
      "names": ["jaeger-*"],
      "privileges": ["read"]
    }
  ]
}'

Create jaeger_admin role for index management
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/role/jaeger_admin" -H "Content-Type: application/json" -d '
{
  "indices": [
    {
      "names": ["jaeger-*"],
      "privileges": ["all"]
    }
  ]
}'

Create Jaeger service users

Set up individual user accounts for Jaeger collector, query service, and administrative operations with appropriate role assignments.

# Create user for Jaeger collector
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/user/jaeger_collector" -H "Content-Type: application/json" -d '
{
  "password": "JaegerCollector2024!",
  "roles": ["jaeger_writer"],
  "full_name": "Jaeger Collector Service"
}'

Create user for Jaeger query
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/user/jaeger_query" -H "Content-Type: application/json" -d '
{
  "password": "JaegerQuery2024!",
  "roles": ["jaeger_reader"],
  "full_name": "Jaeger Query Service"
}'

Create user for Jaeger admin operations
curl -k -u elastic:$ELASTIC_PASSWORD -X POST "https://localhost:9200/_security/user/jaeger_admin" -H "Content-Type: application/json" -d '
{
  "password": "JaegerAdmin2024!",
  "roles": ["jaeger_admin"],
  "full_name": "Jaeger Administrator"
}'

Extract certificates for Jaeger

Convert Elasticsearch certificates to PEM format for use by Jaeger components and configure certificate access permissions.

sudo mkdir -p /etc/jaeger/certs

Extract CA certificate
sudo openssl pkcs12 -in /etc/elasticsearch/certs/elastic-ca.p12 -cacerts -nokeys -out /etc/jaeger/certs/ca.crt -passin pass:""

Extract client certificate and key
sudo openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -clcerts -nokeys -out /etc/jaeger/certs/client.crt -passin pass:""
sudo openssl pkcs12 -in /etc/elasticsearch/certs/http.p12 -nocerts -out /etc/jaeger/certs/client.key -passin pass:"" -passout pass:""

Set appropriate permissions
sudo chown -R jaeger:jaeger /etc/jaeger/certs
sudo chmod 750 /etc/jaeger/certs
sudo chmod 644 /etc/jaeger/certs/ca.crt
sudo chmod 644 /etc/jaeger/certs/client.crt
sudo chmod 600 /etc/jaeger/certs/client.key

Configure Jaeger collector with security

Set up Jaeger collector with TLS encryption, authentication credentials, and secure connection parameters for Elasticsearch backend.

es:
  server-urls: https://localhost:9200
  username: jaeger_collector
  password: JaegerCollector2024!
  tls:
    enabled: true
    ca: /etc/jaeger/certs/ca.crt
    cert: /etc/jaeger/certs/client.crt
    key: /etc/jaeger/certs/client.key
    server-name: localhost
    insecure-skip-verify: false
  num-shards: 3
  num-replicas: 1
  index-prefix: jaeger
  create-index-templates: true
  timeout: 30s
  max-span-age: 72h
  
collector:
  grpc:
    host-port: 0.0.0.0:14250
    tls:
      enabled: true
      cert: /etc/jaeger/certs/client.crt
      key: /etc/jaeger/certs/client.key
  http:
    host-port: 0.0.0.0:14268
    tls:
      enabled: true
      cert: /etc/jaeger/certs/client.crt
      key: /etc/jaeger/certs/client.key
  zipkin:
    host-port: 0.0.0.0:9411

Configure Jaeger query service

Set up Jaeger query service with read-only Elasticsearch access, TLS encryption, and UI security configuration for web interface access.

es:
  server-urls: https://localhost:9200
  username: jaeger_query
  password: JaegerQuery2024!
  tls:
    enabled: true
    ca: /etc/jaeger/certs/ca.crt
    cert: /etc/jaeger/certs/client.crt
    key: /etc/jaeger/certs/client.key
    server-name: localhost
    insecure-skip-verify: false
  index-prefix: jaeger
  timeout: 30s
  max-lookback: 168h
  
query:
  http:
    host-port: 0.0.0.0:16686
    tls:
      enabled: true
      cert: /etc/jaeger/certs/client.crt
      key: /etc/jaeger/certs/client.key
  grpc:
    host-port: 0.0.0.0:16685
    tls:
      enabled: true
      cert: /etc/jaeger/certs/client.crt
      key: /etc/jaeger/certs/client.key
      
ui:
  config:
    archive:
      enabled: false
    dependencies:
      menuEnabled: true
    tracking:
      gaID: ""
    menu:
      - label: "About Jaeger"
        url: "https://jaegertracing.io"

Configure systemd services

Create systemd service files for Jaeger components with proper security context and resource limits for production deployment.

[Unit]
Description=Jaeger Collector
Documentation=https://jaegertracing.io
After=network.target elasticsearch.service
Requires=elasticsearch.service

[Service]
Type=simple
User=jaeger
Group=jaeger
ExecStart=/usr/bin/jaeger-collector --config-file=/etc/jaeger/collector.yaml
Restart=always
RestartSec=10
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30

Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/jaeger
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true

Resource limits
LimitNOFILE=65536
LimitNPROC=4096

[Install]
WantedBy=multi-user.target

Create Jaeger query service

Set up systemd service for Jaeger query with security hardening and proper dependency management on Elasticsearch availability.

[Unit]
Description=Jaeger Query Service
Documentation=https://jaegertracing.io
After=network.target elasticsearch.service
Requires=elasticsearch.service

[Service]
Type=simple
User=jaeger
Group=jaeger
ExecStart=/usr/bin/jaeger-query --config-file=/etc/jaeger/query.yaml
Restart=always
RestartSec=10
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=30

Security settings
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/jaeger
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true

Resource limits
LimitNOFILE=65536
LimitNPROC=4096

[Install]
WantedBy=multi-user.target

Configure log directories and permissions

Set up proper logging directory structure with appropriate permissions for Jaeger service user and log rotation configuration.

sudo mkdir -p /var/log/jaeger
sudo chown jaeger:jaeger /var/log/jaeger
sudo chmod 750 /var/log/jaeger

Create jaeger user if not exists
sudo useradd -r -s /bin/false jaeger 2>/dev/null || true

Ensure all certificate permissions are correct
sudo chown -R jaeger:jaeger /etc/jaeger
sudo chmod -R o-rwx /etc/jaeger/certs

Start and enable services

Start Jaeger services with systemd and verify they connect securely to Elasticsearch with proper authentication and encryption.

sudo systemctl daemon-reload
sudo systemctl enable jaeger-collector
sudo systemctl enable jaeger-query
sudo systemctl start jaeger-collector
sudo systemctl start jaeger-query

Check service status
sudo systemctl status jaeger-collector
sudo systemctl status jaeger-query

Configure firewall rules

Set up specific firewall rules for Jaeger ports with TLS encryption and restrict access to authorized networks only.

sudo ufw allow from 203.0.113.0/24 to any port 14250 comment 'Jaeger gRPC collector'
sudo ufw allow from 203.0.113.0/24 to any port 14268 comment 'Jaeger HTTP collector'
sudo ufw allow from 203.0.113.0/24 to any port 16686 comment 'Jaeger UI'
sudo ufw allow from 203.0.113.0/24 to any port 16685 comment 'Jaeger query gRPC'
sudo ufw reload

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="14250" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="14268" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="16686" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="203.0.113.0/24" port protocol="tcp" port="16685" accept'
sudo firewall-cmd --reload

Configure index lifecycle management

Set up automated index lifecycle policies for secure data retention and automated cleanup of old tracing data.

# Create ILM policy for Jaeger indices
curl -k -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_ilm/policy/jaeger_policy" -H "Content-Type: application/json" -d '
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "delete": {
        "min_age": "30d"
      }
    }
  }
}'

Apply policy to Jaeger index templates
curl -k -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_index_template/jaeger_template" -H "Content-Type: application/json" -d '
{
  "index_patterns": ["jaeger-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "jaeger_policy",
      "index.lifecycle.rollover_alias": "jaeger-write"
    }
  }
}'

Enable audit logging

Configure comprehensive audit logging for security monitoring and compliance tracking of Jaeger access patterns.

# Add to existing elasticsearch.yml
xpack.security.audit.enabled: true
xpack.security.audit.outputs: [index, logfile]
xpack.security.audit.logfile.events.include: [
  "access_denied",
  "access_granted",
  "anonymous_access_denied",
  "authentication_failed",
  "connection_denied",
  "tampered_request",
  "run_as_denied",
  "run_as_granted"
]
xpack.security.audit.logfile.events.exclude: ["access_granted"]
xpack.security.audit.index.rollover: "daily"
xpack.security.audit.index.events.include: ["authentication_failed", "access_denied", "tampered_request"]

Set up monitoring alerts

Configure security monitoring and alerting for authentication failures, unauthorized access attempts, and certificate expiration tracking.

# Create watcher for failed authentications
curl -k -u elastic:$ELASTIC_PASSWORD -X PUT "https://localhost:9200/_watcher/watch/jaeger_auth_failures" -H "Content-Type: application/json" -d '
{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [".security-audit-*"],
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "term": {
                    "event_type": "authentication_failed"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total.value": {
        "gt": 5
      }
    }
  },
  "actions": {
    "log_alert": {
      "logging": {
        "level": "warn",
        "text": "Multiple authentication failures detected for Jaeger users"
      }
    }
  }
}'