Implement OpenTelemetry distributed context propagation across microservices with automatic instrumentation

Update system packages

Start by updating your package manager to ensure you get the latest versions of dependencies.

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget gnupg2 software-properties-common

sudo dnf update -y
sudo dnf install -y curl wget gnupg2

Install OpenTelemetry Collector

Download and install the OpenTelemetry Collector which will receive, process, and export telemetry data from your applications.

curl -L https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.91.0/otelcol_0.91.0_linux_amd64.tar.gz -o otelcol.tar.gz
tar -xzf otelcol.tar.gz
sudo mv otelcol /usr/local/bin/
sudo chmod +x /usr/local/bin/otelcol

Create OpenTelemetry Collector configuration

Configure the collector to receive traces via OTLP and export to Jaeger. This setup enables context propagation across all instrumented services.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - "*"
          allowed_headers:
            - "*"

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  resource:
    attributes:
      - key: service.name
        action: upsert
        from_attribute: service_name
  
exporters:
  jaeger:
    endpoint: localhost:14250
    tls:
      insecure: true
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [jaeger, logging]
  telemetry:
    logs:
      level: info

Create collector directory and set permissions

Create the configuration directory and set proper ownership for the OpenTelemetry Collector service.

sudo mkdir -p /etc/otelcol
sudo chown -R root:root /etc/otelcol
sudo chmod 755 /etc/otelcol
sudo chmod 644 /etc/otelcol/config.yaml

Install Jaeger for trace visualization

Install Jaeger to store and visualize the distributed traces collected by OpenTelemetry.

curl -L https://github.com/jaegertracing/jaeger/releases/download/v1.52.0/jaeger-1.52.0-linux-amd64.tar.gz -o jaeger.tar.gz
tar -xzf jaeger.tar.gz
sudo mv jaeger-1.52.0-linux-amd64/jaeger-all-in-one /usr/local/bin/
sudo chmod +x /usr/local/bin/jaeger-all-in-one

Create systemd service for Jaeger

Set up Jaeger as a system service to ensure it starts automatically and runs continuously.

[Unit]
Description=Jaeger Tracing Platform
After=network.target

[Service]
Type=simple
User=jaeger
Group=jaeger
ExecStart=/usr/local/bin/jaeger-all-in-one --memory.max-traces=10000
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Create Jaeger user and start service

Create a dedicated user for Jaeger and enable the service to start on boot.

sudo useradd -r -s /bin/false jaeger
sudo systemctl daemon-reload
sudo systemctl enable --now jaeger

Create systemd service for OpenTelemetry Collector

Configure the OpenTelemetry Collector as a system service for reliable operation.

[Unit]
Description=OpenTelemetry Collector
After=network.target

[Service]
Type=simple
User=otelcol
Group=otelcol
ExecStart=/usr/local/bin/otelcol --config=/etc/otelcol/config.yaml
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Create collector user and start service

Create a dedicated user for the OpenTelemetry Collector and start the service.

sudo useradd -r -s /bin/false otelcol
sudo chown -R otelcol:otelcol /etc/otelcol
sudo systemctl daemon-reload
sudo systemctl enable --now otelcol

Install Python auto-instrumentation

Install OpenTelemetry auto-instrumentation for Python applications including Flask, Django, and FastAPI support.

pip install opentelemetry-distro[otlp]
opentelemetry-bootstrap -a install

Create Python service example with auto-instrumentation

Create a sample Flask application to demonstrate automatic context propagation between services.

from flask import Flask, request, jsonify
import requests
import os

app = Flask(__name__)

@app.route('/users/')
def get_user(user_id):
    # This request will automatically propagate trace context
    orders_response = requests.get(f'http://localhost:5001/orders/{user_id}')
    
    user_data = {
        'user_id': user_id,
        'name': f'User {user_id}',
        'orders': orders_response.json() if orders_response.status_code == 200 else []
    }
    return jsonify(user_data)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=True)

Create orders service with trace correlation

Create a second service that will receive and continue traces from the user service automatically.

from flask import Flask, jsonify
import time
import random

app = Flask(__name__)

@app.route('/orders/')
def get_orders(user_id):
    # Simulate database query time
    time.sleep(random.uniform(0.1, 0.5))
    
    orders = [
        {'order_id': f'ord_{user_id}_001', 'amount': 99.99},
        {'order_id': f'ord_{user_id}_002', 'amount': 149.50}
    ]
    return jsonify(orders)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5001, debug=True)

Create service directories and set permissions

Create application directories with proper permissions for the service files.

sudo mkdir -p /opt/app
sudo chown -R www-data:www-data /opt/app
sudo chmod 755 /opt/app
sudo chmod 644 /opt/app/*.py

Install Node.js auto-instrumentation

Set up automatic instrumentation for Node.js applications with comprehensive framework support.

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs

curl -fsSL https://rpm.nodesource.com/setup_20.x | sudo bash -
sudo dnf install -y nodejs npm

npm install -g @opentelemetry/auto-instrumentations-node

Create Node.js service with Express

Create an Express.js service that demonstrates automatic HTTP instrumentation and context propagation.

const express = require('express');
const axios = require('axios');

const app = express();
const port = 5002;

app.use(express.json());

app.get('/payments/:userId', async (req, res) => {
  const { userId } = req.params;
  
  try {
    // Simulate payment processing time
    await new Promise(resolve => setTimeout(resolve, Math.random() * 300));
    
    const payments = [
      { payment_id: pay_${userId}_001, amount: 99.99, status: 'completed' },
      { payment_id: pay_${userId}_002, amount: 149.50, status: 'pending' }
    ];
    
    res.json(payments);
  } catch (error) {
    res.status(500).json({ error: 'Payment service error' });
  }
});

app.listen(port, '0.0.0.0', () => {
  console.log(Payment service listening on port ${port});
});

Install Node.js dependencies

Install the required Node.js packages for the payment service.

cd /opt/app
npm init -y
npm install express axios

Create Java auto-instrumentation setup

Download the OpenTelemetry Java agent for automatic instrumentation of Spring Boot and other Java frameworks.

sudo apt install -y openjdk-17-jdk maven

sudo dnf install -y java-17-openjdk-devel maven

curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.32.0/opentelemetry-javaagent.jar -o /opt/app/opentelemetry-javaagent.jar

Configure context propagation headers

Create a configuration file that defines which trace context headers should be propagated across service boundaries.

# OpenTelemetry Configuration
otel.service.name=microservices-demo
otel.traces.exporter=otlp
otel.metrics.exporter=none
otel.logs.exporter=none
otel.exporter.otlp.endpoint=http://localhost:4318
otel.exporter.otlp.protocol=http/protobuf

Context Propagation Settings
otel.propagators=tracecontext,baggage,b3multi
otel.instrumentation.http.client.capture-request-headers=traceparent,tracestate,baggage,b3,x-trace-id
otel.instrumentation.http.server.capture-request-headers=traceparent,tracestate,baggage,b3,x-trace-id

Resource attributes
otel.resource.attributes=service.version=1.0.0,deployment.environment=production

Create service startup scripts

Create wrapper scripts that automatically apply OpenTelemetry instrumentation when starting services.

#!/bin/bash
export OTEL_SERVICE_NAME="user-service"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_PROPAGATORS="tracecontext,baggage,b3multi"

opentelemetry-instrument python /opt/app/user-service.py

#!/bin/bash
export OTEL_SERVICE_NAME="orders-service"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_PROPAGATORS="tracecontext,baggage,b3multi"

opentelemetry-instrument python /opt/app/orders-service.py

#!/bin/bash
export NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
export OTEL_SERVICE_NAME="payment-service"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_PROPAGATORS="tracecontext,baggage,b3multi"

node /opt/app/payment-service.js

Make startup scripts executable

Set execute permissions on the startup scripts and ensure proper ownership.

sudo chmod +x /opt/app/start-*.sh
sudo chown www-data:www-data /opt/app/start-*.sh

Configure firewall for OpenTelemetry ports

Open the necessary ports for OpenTelemetry Collector and Jaeger UI access.

sudo ufw allow 4317/tcp comment 'OpenTelemetry OTLP gRPC'
sudo ufw allow 4318/tcp comment 'OpenTelemetry OTLP HTTP'
sudo ufw allow 16686/tcp comment 'Jaeger UI'
sudo ufw allow 14250/tcp comment 'Jaeger gRPC'
sudo ufw reload

sudo firewall-cmd --permanent --add-port=4317/tcp --add-port=4318/tcp --add-port=16686/tcp --add-port=14250/tcp
sudo firewall-cmd --reload