Monitor FastAPI applications with Prometheus and Grafana for production observability

Update system packages

Start by updating your package manager to ensure you get the latest versions.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Create a monitoring user

Create a dedicated user for monitoring services to enhance security.

sudo useradd --no-create-home --shell /bin/false monitoring
sudo mkdir -p /etc/prometheus /etc/grafana /var/lib/prometheus /var/lib/grafana
sudo chown monitoring:monitoring /etc/prometheus /var/lib/prometheus
sudo chown monitoring:monitoring /etc/grafana /var/lib/grafana

Install Python dependencies for FastAPI metrics

Install the required Python packages to add Prometheus metrics to your FastAPI application.

pip3 install prometheus-client prometheus-fastapi-instrumentator uvicorn[standard]

Configure FastAPI application with metrics

Create a sample FastAPI application with built-in Prometheus metrics endpoint.

from fastapi import FastAPI, HTTPException
from prometheus_fastapi_instrumentator import Instrumentator
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import time
import random

app = FastAPI(title="FastAPI Monitoring Demo")

Initialize Prometheus instrumentator
instrumentator = Instrumentator(
    should_group_status_codes=False,
    should_ignore_untemplated=True,
    should_instrument_requests_inprogress=True,
    excluded_handlers=["/metrics"],
    env_var_name="ENABLE_METRICS",
    inprogress_name="fastapi_inprogress",
    inprogress_labels=True,
)

Custom metrics
REQUEST_COUNT = Counter(
    'fastapi_requests_total',
    'Total requests',
    ['method', 'endpoint', 'status_code']
)

REQUEST_DURATION = Histogram(
    'fastapi_request_duration_seconds',
    'Request duration',
    ['method', 'endpoint']
)

ERROR_COUNT = Counter(
    'fastapi_errors_total',
    'Total errors',
    ['endpoint', 'error_type']
)

Instrument the app
instrumentator.instrument(app).expose(app)

@app.get("/")
async def root():
    REQUEST_COUNT.labels(method="GET", endpoint="/", status_code="200").inc()
    return {"message": "Hello World", "status": "healthy"}

@app.get("/health")
async def health_check():
    REQUEST_COUNT.labels(method="GET", endpoint="/health", status_code="200").inc()
    return {"status": "ok", "timestamp": time.time()}

@app.get("/slow")
async def slow_endpoint():
    start_time = time.time()
    # Simulate slow processing
    await asyncio.sleep(random.uniform(1, 3))
    duration = time.time() - start_time
    REQUEST_DURATION.labels(method="GET", endpoint="/slow").observe(duration)
    REQUEST_COUNT.labels(method="GET", endpoint="/slow", status_code="200").inc()
    return {"message": "Slow response", "duration": duration}

@app.get("/error")
async def error_endpoint():
    if random.choice([True, False]):
        ERROR_COUNT.labels(endpoint="/error", error_type="random_error").inc()
        REQUEST_COUNT.labels(method="GET", endpoint="/error", status_code="500").inc()
        raise HTTPException(status_code=500, detail="Random error occurred")
    REQUEST_COUNT.labels(method="GET", endpoint="/error", status_code="200").inc()
    return {"message": "No error this time"}

Create FastAPI systemd service

Set up the FastAPI application as a systemd service for automatic startup and management.

[Unit]
Description=FastAPI Application
After=network.target

[Service]
Type=exec
User=monitoring
Group=monitoring
WorkingDirectory=/opt/fastapi-app
ExecStart=/usr/bin/python3 -m uvicorn main:app --host 0.0.0.0 --port 8000
Restart=always
RestartSec=3
Environment=ENABLE_METRICS=1

[Install]
WantedBy=multi-user.target

Download and install Prometheus

Download the latest Prometheus binary and set up the installation.

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz
tar xzf prometheus-2.48.0.linux-amd64.tar.gz
sudo cp prometheus-2.48.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.48.0.linux-amd64/promtool /usr/local/bin/
sudo chown monitoring:monitoring /usr/local/bin/prometheus /usr/local/bin/promtool
rm -rf prometheus-2.48.0.linux-amd64*

Configure Prometheus

Create the Prometheus configuration file with scraping targets for your FastAPI application.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "/etc/prometheus/fastapi_rules.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'fastapi'
    static_configs:
      - targets: ['localhost:8000']
    metrics_path: '/metrics'
    scrape_interval: 5s
    scrape_timeout: 5s

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']
    scrape_interval: 15s

Create Prometheus alerting rules

Set up alerting rules for FastAPI application monitoring.

groups:
  - name: fastapi_alerts
    rules:
      - alert: FastAPIHighErrorRate
        expr: rate(fastapi_errors_total[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected in FastAPI application"
          description: "Error rate is {{ $value }} errors per second"

      - alert: FastAPIHighLatency
        expr: histogram_quantile(0.95, rate(fastapi_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency detected in FastAPI application"
          description: "95th percentile latency is {{ $value }} seconds"

      - alert: FastAPIDown
        expr: up{job="fastapi"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "FastAPI application is down"
          description: "FastAPI application has been down for more than 1 minute"

      - alert: FastAPIHighRequestRate
        expr: rate(fastapi_requests_total[5m]) > 100
        for: 2m
        labels:
          severity: info
        annotations:
          summary: "High request rate detected"
          description: "Request rate is {{ $value }} requests per second"

Install Node Exporter

Install Node Exporter to collect system metrics alongside application metrics.

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xzf node_exporter-1.7.0.linux-amd64.tar.gz
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
sudo chown monitoring:monitoring /usr/local/bin/node_exporter
rm -rf node_exporter-1.7.0.linux-amd64*

Create Node Exporter systemd service

Set up Node Exporter as a systemd service.

[Unit]
Description=Node Exporter
After=network.target

[Service]
Type=simple
User=monitoring
Group=monitoring
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Create Prometheus systemd service

Set up Prometheus as a systemd service with proper configuration.

[Unit]
Description=Prometheus
After=network.target

[Service]
Type=simple
User=monitoring
Group=monitoring
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Install Grafana

Install Grafana for creating monitoring dashboards and visualizations.

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafana

sudo dnf install -y https://dl.grafana.com/oss/release/grafana-10.2.0-1.x86_64.rpm

Configure Grafana

Set up Grafana configuration with security settings and data source configuration.

[server]
http_port = 3000
domain = example.com
root_url = http://example.com:3000

[security]
admin_user = admin
admin_password = secure_password_here
secret_key = change_this_secret_key
disable_gravatar = true

[users]
allow_sign_up = false
allow_org_create = false

[auth.anonymous]
enabled = false

[database]
type = sqlite3
path = grafana.db

[alerting]
enabled = true

Set proper file permissions

Ensure all files have correct ownership and permissions for security.

sudo chown -R monitoring:monitoring /etc/prometheus /var/lib/prometheus
sudo chown -R grafana:grafana /etc/grafana /var/lib/grafana
sudo chmod -R 755 /etc/prometheus
sudo chmod 644 /etc/prometheus/prometheus.yml /etc/prometheus/fastapi_rules.yml
sudo chmod 755 /opt/fastapi-app
sudo chmod 644 /opt/fastapi-app/main.py

Start and enable all services

Start all monitoring services and enable them to start automatically on boot.

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo systemctl enable --now prometheus
sudo systemctl enable --now grafana-server
sudo systemctl enable --now fastapi-app

Configure firewall rules

Open the necessary ports for Prometheus, Grafana, and your FastAPI application.

sudo ufw allow 3000/tcp comment 'Grafana'
sudo ufw allow 9090/tcp comment 'Prometheus'
sudo ufw allow 8000/tcp comment 'FastAPI'
sudo ufw --force enable

sudo firewall-cmd --permanent --add-port=3000/tcp
sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --permanent --add-port=8000/tcp
sudo firewall-cmd --reload

Create Grafana dashboard JSON

Create a comprehensive dashboard configuration for FastAPI monitoring.

{
  "dashboard": {
    "id": null,
    "title": "FastAPI Monitoring",
    "tags": ["fastapi", "monitoring"],
    "timezone": "browser",
    "panels": [
      {
        "id": 1,
        "title": "Request Rate",
        "type": "stat",
        "targets": [
          {
            "expr": "rate(fastapi_requests_total[5m])",
            "legendFormat": "Requests/sec"
          }
        ],
        "gridPos": {"h": 8, "w": 6, "x": 0, "y": 0}
      },
      {
        "id": 2,
        "title": "Error Rate",
        "type": "stat",
        "targets": [
          {
            "expr": "rate(fastapi_errors_total[5m])",
            "legendFormat": "Errors/sec"
          }
        ],
        "gridPos": {"h": 8, "w": 6, "x": 6, "y": 0}
      },
      {
        "id": 3,
        "title": "Response Time (95th percentile)",
        "type": "stat",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(fastapi_request_duration_seconds_bucket[5m]))",
            "legendFormat": "95th percentile"
          }
        ],
        "gridPos": {"h": 8, "w": 6, "x": 12, "y": 0}
      },
      {
        "id": 4,
        "title": "Active Requests",
        "type": "stat",
        "targets": [
          {
            "expr": "fastapi_inprogress",
            "legendFormat": "In Progress"
          }
        ],
        "gridPos": {"h": 8, "w": 6, "x": 18, "y": 0}
      }
    ],
    "time": {
      "from": "now-1h",
      "to": "now"
    },
    "refresh": "5s"
  }
}

Access Grafana web interface

Open your browser and navigate to Grafana to complete the setup.

http://your_server_ip:3000

Login with admin/secure_password_here (or the password you set in the configuration).

Add Prometheus data source

Configure Grafana to use Prometheus as a data source for metrics collection.

1. Go to Configuration > Data Sources
2. Click "Add data source"
3. Select "Prometheus"
4. Set URL to http://localhost:9090
5. Click "Save & Test"

Import FastAPI dashboard

Import the custom dashboard for comprehensive FastAPI monitoring.

curl -X POST \
  http://admin:secure_password_here@localhost:3000/api/dashboards/db \
  -H 'Content-Type: application/json' \
  -d @/tmp/fastapi-dashboard.json

Configure Grafana alerting

Set up alert notifications for critical FastAPI metrics.

In Grafana:

1. Go to Alerting > Notification channels
2. Add an email notification channel
3. Create alert rules for high error rates and latency
4. Test the notification system

Test alerting rules

Verify that Prometheus alerting rules are working correctly.

# Check if rules are loaded
curl http://localhost:9090/api/v1/rules

Trigger error alerts by calling error endpoint
for i in {1..20}; do curl http://localhost:8000/error; done

Check active alerts
curl http://localhost:9090/api/v1/alerts