Deploy Thanos components with Prometheus to achieve unlimited data retention using object storage. This advanced setup enables querying years of historical metrics while maintaining high availability and reducing local storage costs.

Prerequisites

Root or sudo access
Minimum 4GB RAM
Object storage backend (S3, MinIO, or GCS)
Basic Prometheus knowledge

What this solves

Prometheus has a built-in 15-day data retention limit that creates gaps in long-term monitoring and capacity planning. Thanos solves this by shipping metric data to object storage (S3, MinIO, GCS) while maintaining the same PromQL query interface. This setup enables unlimited historical data retention, horizontal scaling across multiple Prometheus instances, and significant cost reduction compared to local disk storage.

Step-by-step installation

Update system packages

Start by updating your package manager to ensure you get the latest versions and security patches.

sudo apt update && sudo apt upgrade -y
sudo apt install -y wget curl tar systemd

sudo dnf update -y
sudo dnf install -y wget curl tar systemd

Create dedicated users

Create separate system users for Prometheus and Thanos components to follow security best practices.

sudo useradd --no-create-home --shell /bin/false prometheus
sudo useradd --no-create-home --shell /bin/false thanos
sudo mkdir -p /etc/prometheus /var/lib/prometheus /etc/thanos /var/lib/thanos
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus
sudo chown thanos:thanos /etc/thanos /var/lib/thanos

Download and install Prometheus

Download the latest Prometheus binary and install it with proper permissions.

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.49.1/prometheus-2.49.1.linux-amd64.tar.gz
tar xzf prometheus-2.49.1.linux-amd64.tar.gz
sudo cp prometheus-2.49.1.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.49.1.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool
sudo chmod 755 /usr/local/bin/prometheus /usr/local/bin/promtool

Download and install Thanos

Download Thanos binary which includes all components (sidecar, query, store, compactor).

cd /tmp
wget https://github.com/thanos-io/thanos/releases/download/v0.33.0/thanos-0.33.0.linux-amd64.tar.gz
tar xzf thanos-0.33.0.linux-amd64.tar.gz
sudo cp thanos-0.33.0.linux-amd64/thanos /usr/local/bin/
sudo chown thanos:thanos /usr/local/bin/thanos
sudo chmod 755 /usr/local/bin/thanos

Configure object storage

Create the object storage configuration file for Thanos. This example uses MinIO, but you can adapt it for S3 or GCS.

type: s3
config:
  bucket: "thanos-storage"
  endpoint: "203.0.113.10:9000"
  access_key: "thanos-access-key"
  secret_key: "thanos-secret-key-change-this"
  insecure: false
  signature_version2: false
  encrypt_sse: false
  put_user_metadata: {}
  http_config:
    idle_timeout: 90s
    response_header_timeout: 2m
    insecure_skip_verify: false
  trace:
    enable: false
  part_size: 67108864

For AWS S3, use this configuration instead:

type: s3
config:
  bucket: "your-thanos-bucket"
  region: "us-east-1"
  access_key: "AKIAIOSFODNN7EXAMPLE"
  secret_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  encrypt_sse: true

sudo chown thanos:thanos /etc/thanos/bucket.yml
sudo chmod 640 /etc/thanos/bucket.yml

Security Notice: Store object storage credentials securely. Consider using IAM roles, environment variables, or secret management systems instead of plain text files in production.

Configure Prometheus for Thanos integration

Configure Prometheus to enable external labels and remote read capabilities for Thanos sidecar.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    replica: 'prometheus-01'
    region: 'us-east-1'

rule_files:
  - "/etc/prometheus/rules/*.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'thanos-sidecar'
    static_configs:
      - targets: ['localhost:10902']

  - job_name: 'thanos-query'
    static_configs:
      - targets: ['localhost:10904']

  - job_name: 'thanos-store'
    static_configs:
      - targets: ['localhost:10905']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
sudo chmod 644 /etc/prometheus/prometheus.yml

Create Prometheus systemd service

Create a systemd service file for Prometheus with Thanos integration enabled.

[Unit]
Description=Prometheus Time Series Database
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --storage.tsdb.retention.time=2h \
  --storage.tsdb.min-block-duration=2h \
  --storage.tsdb.max-block-duration=2h \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle \
  --web.enable-admin-api
ExecReload=/bin/kill -HUP $MAINPID
KillMode=mixed
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

Note: The retention time is set to 2 hours because Thanos sidecar will handle long-term storage. The min/max block duration ensures compatibility with Thanos.

Create Thanos sidecar service

The sidecar runs alongside Prometheus and uploads data blocks to object storage.

[Unit]
Description=Thanos Sidecar
After=network.target prometheus.service
Requires=prometheus.service

[Service]
User=thanos
Group=thanos
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/thanos sidecar \
  --tsdb.path=/var/lib/prometheus \
  --prometheus.url=http://localhost:9090 \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902 \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --log.level=info
KillMode=mixed
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

Create Thanos query service

Query component provides the unified query interface across all Prometheus instances and object storage.

[Unit]
Description=Thanos Query
After=network.target

[Service]
User=thanos
Group=thanos
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/thanos query \
  --grpc-address=0.0.0.0:10903 \
  --http-address=0.0.0.0:10904 \
  --store=localhost:10901 \
  --store=localhost:10905 \
  --query.replica-label=replica \
  --query.auto-downsampling \
  --log.level=info
KillMode=mixed
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

Create Thanos store gateway service

Store gateway serves historical data directly from object storage for queries beyond Prometheus retention.

[Unit]
Description=Thanos Store Gateway
After=network.target

[Service]
User=thanos
Group=thanos
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/thanos store \
  --grpc-address=0.0.0.0:10905 \
  --http-address=0.0.0.0:10906 \
  --data-dir=/var/lib/thanos/store \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --log.level=info
KillMode=mixed
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

Create Thanos compactor service

Compactor downsamples and compacts data in object storage to optimize query performance and reduce storage costs.

[Unit]
Description=Thanos Compactor
After=network.target

[Service]
User=thanos
Group=thanos
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/thanos compact \
  --data-dir=/var/lib/thanos/compactor \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --retention.resolution-raw=30d \
  --retention.resolution-5m=90d \
  --retention.resolution-1h=365d \
  --wait \
  --log.level=info
KillMode=mixed
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

sudo mkdir -p /var/lib/thanos/store /var/lib/thanos/compactor
sudo chown -R thanos:thanos /var/lib/thanos

Configure firewall rules

Open the necessary ports for Thanos components communication.

sudo ufw allow 9090/tcp comment 'Prometheus'
sudo ufw allow 10901/tcp comment 'Thanos Sidecar gRPC'
sudo ufw allow 10902/tcp comment 'Thanos Sidecar HTTP'
sudo ufw allow 10903/tcp comment 'Thanos Query gRPC'
sudo ufw allow 10904/tcp comment 'Thanos Query HTTP'
sudo ufw allow 10905/tcp comment 'Thanos Store gRPC'
sudo ufw allow 10906/tcp comment 'Thanos Store HTTP'

sudo firewall-cmd --permanent --add-port=9090/tcp
sudo firewall-cmd --permanent --add-port=10901/tcp
sudo firewall-cmd --permanent --add-port=10902/tcp
sudo firewall-cmd --permanent --add-port=10903/tcp
sudo firewall-cmd --permanent --add-port=10904/tcp
sudo firewall-cmd --permanent --add-port=10905/tcp
sudo firewall-cmd --permanent --add-port=10906/tcp
sudo firewall-cmd --reload

Enable and start all services

Start all services in the correct order and enable them for automatic startup.

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl enable --now thanos-sidecar
sudo systemctl enable --now thanos-store
sudo systemctl enable --now thanos-query
sudo systemctl enable --now thanos-compactor

Configure Grafana data source

Configure Grafana to use Thanos Query as the data source instead of direct Prometheus connection.

Name: Thanos
Type: Prometheus
URL: http://localhost:10904
Access: Server (default)
Scrape interval: 15s
Query timeout: 60s
HTTP Method: GET

Verify your setup

Check that all services are running correctly and can communicate with each other.

sudo systemctl status prometheus thanos-sidecar thanos-query thanos-store thanos-compactor
curl http://localhost:9090/-/healthy
curl http://localhost:10902/-/healthy
curl http://localhost:10904/-/healthy
curl http://localhost:10905/-/healthy

Test Thanos Query interface and verify it can access both live and historical data:

curl "http://localhost:10904/api/v1/stores"
curl "http://localhost:10904/api/v1/query?query=up"
thanos tools bucket ls --objstore.config-file=/etc/thanos/bucket.yml

Access the web interfaces to confirm everything is working:

Prometheus: http://your-server:9090
Thanos Query: http://your-server:10904
Thanos Sidecar: http://your-server:10902
Thanos Store: http://your-server:10906

Common issues

Symptom	Cause	Fix
Sidecar can't upload blocks	Object storage credentials or connectivity	Verify bucket.yml config and network access: `thanos tools bucket ls --objstore.config-file=/etc/thanos/bucket.yml`
Query shows no historical data	Store gateway not connected or no uploaded blocks	Check store gateway logs: `sudo journalctl -u thanos-store -f`
Prometheus blocks not uploading	Block duration mismatch or insufficient retention	Ensure min/max block duration is 2h in prometheus.service
Permission denied errors	Incorrect file ownership	Fix ownership: `sudo chown -R thanos:thanos /var/lib/thanos /etc/thanos`
gRPC communication failures	Firewall blocking or wrong addresses	Verify ports are open and services bind to correct interfaces
Compactor not processing data	Insufficient retention settings	Wait 2+ hours for first blocks or check compactor logs

Next steps

Automated install script

Run this to automate the entire setup

install.sh

#!/usr/bin/env bash

set -euo pipefail

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

# Configuration
PROMETHEUS_VERSION="2.49.1"
THANOS_VERSION="0.33.0"
STORAGE_TYPE="${1:-minio}"
STORAGE_ENDPOINT="${2:-localhost:9000}"
BUCKET_NAME="${3:-thanos-storage}"

usage() {
    echo "Usage: $0 [storage_type] [endpoint] [bucket_name]"
    echo "  storage_type: minio (default) or s3"
    echo "  endpoint: Storage endpoint (default: localhost:9000)"
    echo "  bucket_name: Storage bucket name (default: thanos-storage)"
    exit 1
}

log_info() { echo -e "${GREEN}$1${NC}"; }
log_warn() { echo -e "${YELLOW}$1${NC}"; }
log_error() { echo -e "${RED}$1${NC}"; }

cleanup() {
    log_error "[ERROR] Installation failed. Cleaning up..."
    systemctl stop prometheus thanos-sidecar thanos-query thanos-store thanos-compactor 2>/dev/null || true
    systemctl disable prometheus thanos-sidecar thanos-query thanos-store thanos-compactor 2>/dev/null || true
    rm -f /etc/systemd/system/{prometheus,thanos-*}.service
    systemctl daemon-reload
}

trap cleanup ERR

if [[ $EUID -ne 0 ]]; then
    log_error "This script must be run as root"
    exit 1
fi

# Detect distribution
if [ -f /etc/os-release ]; then
    . /etc/os-release
    case "$ID" in
        ubuntu|debian) 
            PKG_MGR="apt"
            PKG_UPDATE="apt update && apt upgrade -y"
            PKG_INSTALL="apt install -y"
            ;;
        almalinux|rocky|centos|rhel|ol|fedora) 
            PKG_MGR="dnf"
            PKG_UPDATE="dnf update -y"
            PKG_INSTALL="dnf install -y"
            ;;
        amzn) 
            PKG_MGR="yum"
            PKG_UPDATE="yum update -y"
            PKG_INSTALL="yum install -y"
            ;;
        *) 
            log_error "Unsupported distribution: $ID"
            exit 1
            ;;
    esac
else
    log_error "Cannot detect distribution"
    exit 1
fi

log_info "[1/10] Updating system packages..."
$PKG_UPDATE
$PKG_INSTALL wget curl tar systemd

log_info "[2/10] Creating dedicated users..."
if ! id prometheus >/dev/null 2>&1; then
    useradd --no-create-home --shell /bin/false prometheus
fi
if ! id thanos >/dev/null 2>&1; then
    useradd --no-create-home --shell /bin/false thanos
fi

mkdir -p /etc/prometheus /var/lib/prometheus /etc/thanos /var/lib/thanos
chown prometheus:prometheus /etc/prometheus /var/lib/prometheus
chown thanos:thanos /etc/thanos /var/lib/thanos
chmod 755 /etc/prometheus /var/lib/prometheus /etc/thanos /var/lib/thanos

log_info "[3/10] Downloading and installing Prometheus..."
cd /tmp
wget -q "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz"
tar xzf "prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz"
cp "prometheus-${PROMETHEUS_VERSION}.linux-amd64/prometheus" /usr/local/bin/
cp "prometheus-${PROMETHEUS_VERSION}.linux-amd64/promtool" /usr/local/bin/
chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool
chmod 755 /usr/local/bin/prometheus /usr/local/bin/promtool
rm -rf "prometheus-${PROMETHEUS_VERSION}.linux-amd64"*

log_info "[4/10] Downloading and installing Thanos..."
wget -q "https://github.com/thanos-io/thanos/releases/download/v${THANOS_VERSION}/thanos-${THANOS_VERSION}.linux-amd64.tar.gz"
tar xzf "thanos-${THANOS_VERSION}.linux-amd64.tar.gz"
cp "thanos-${THANOS_VERSION}.linux-amd64/thanos" /usr/local/bin/
chown thanos:thanos /usr/local/bin/thanos
chmod 755 /usr/local/bin/thanos
rm -rf "thanos-${THANOS_VERSION}.linux-amd64"*

log_info "[5/10] Configuring object storage..."
if [[ "$STORAGE_TYPE" == "s3" ]]; then
    cat > /etc/thanos/bucket.yml << 'EOF'
type: s3
config:
  bucket: "BUCKET_NAME"
  region: "us-east-1"
  access_key: "AKIAIOSFODNN7EXAMPLE"
  secret_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  encrypt_sse: true
EOF
else
    cat > /etc/thanos/bucket.yml << 'EOF'
type: s3
config:
  bucket: "BUCKET_NAME"
  endpoint: "STORAGE_ENDPOINT"
  access_key: "thanos-access-key"
  secret_key: "thanos-secret-key-change-this"
  insecure: false
  signature_version2: false
  encrypt_sse: false
  put_user_metadata: {}
  http_config:
    idle_timeout: 90s
    response_header_timeout: 2m
  part_size: 67108864
EOF
fi

sed -i "s/BUCKET_NAME/$BUCKET_NAME/g" /etc/thanos/bucket.yml
sed -i "s/STORAGE_ENDPOINT/$STORAGE_ENDPOINT/g" /etc/thanos/bucket.yml
chown thanos:thanos /etc/thanos/bucket.yml
chmod 640 /etc/thanos/bucket.yml

log_info "[6/10] Configuring Prometheus..."
cat > /etc/prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    replica: 'prometheus-01'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'thanos-sidecar'
    static_configs:
      - targets: ['localhost:10902']

  - job_name: 'thanos-query'
    static_configs:
      - targets: ['localhost:10904']

  - job_name: 'thanos-store'
    static_configs:
      - targets: ['localhost:10905']
EOF

chown prometheus:prometheus /etc/prometheus/prometheus.yml
chmod 644 /etc/prometheus/prometheus.yml

log_info "[7/10] Creating systemd services..."

# Prometheus service
cat > /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=always
RestartSec=5s
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle \
  --storage.tsdb.min-block-duration=2h \
  --storage.tsdb.max-block-duration=2h

[Install]
WantedBy=multi-user.target
EOF

# Thanos Sidecar service
cat > /etc/systemd/system/thanos-sidecar.service << 'EOF'
[Unit]
Description=Thanos Sidecar
After=network.target prometheus.service
Requires=prometheus.service

[Service]
Type=simple
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos sidecar \
  --tsdb.path=/var/lib/prometheus \
  --prometheus.url=http://localhost:9090 \
  --grpc-address=0.0.0.0:10901 \
  --http-address=0.0.0.0:10902 \
  --objstore.config-file=/etc/thanos/bucket.yml

Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

# Thanos Query service
cat > /etc/systemd/system/thanos-query.service << 'EOF'
[Unit]
Description=Thanos Query
After=network.target

[Service]
Type=simple
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos query \
  --http-address=0.0.0.0:10904 \
  --grpc-address=0.0.0.0:10903 \
  --store=localhost:10901 \
  --store=localhost:10906

Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

# Thanos Store service
cat > /etc/systemd/system/thanos-store.service << 'EOF'
[Unit]
Description=Thanos Store Gateway
After=network.target

[Service]
Type=simple
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos store \
  --data-dir=/var/lib/thanos \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --http-address=0.0.0.0:10905 \
  --grpc-address=0.0.0.0:10906

Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

# Thanos Compactor service
cat > /etc/systemd/system/thanos-compactor.service << 'EOF'
[Unit]
Description=Thanos Compactor
After=network.target

[Service]
Type=simple
User=thanos
Group=thanos
ExecStart=/usr/local/bin/thanos compact \
  --data-dir=/var/lib/thanos \
  --objstore.config-file=/etc/thanos/bucket.yml \
  --http-address=0.0.0.0:10907

Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

log_info "[8/10] Starting services..."
systemctl daemon-reload
systemctl enable prometheus thanos-sidecar thanos-query thanos-store thanos-compactor
systemctl start prometheus
sleep 10
systemctl start thanos-sidecar thanos-query thanos-store thanos-compactor

log_info "[9/10] Configuring firewall..."
if command -v ufw >/dev/null 2>&1; then
    ufw allow 9090/tcp  # Prometheus
    ufw allow 10902/tcp # Thanos Sidecar
    ufw allow 10904/tcp # Thanos Query
    ufw allow 10905/tcp # Thanos Store
elif command -v firewall-cmd >/dev/null 2>&1; then
    firewall-cmd --permanent --add-port=9090/tcp
    firewall-cmd --permanent --add-port=10902/tcp
    firewall-cmd --permanent --add-port=10904/tcp
    firewall-cmd --permanent --add-port=10905/tcp
    firewall-cmd --reload
fi

log_info "[10/10] Verifying installation..."
sleep 5
for service in prometheus thanos-sidecar thanos-query thanos-store thanos-compactor; do
    if systemctl is-active --quiet $service; then
        log_info "✓ $service is running"
    else
        log_error "✗ $service failed to start"
        systemctl status $service --no-pager
        exit 1
    fi
done

log_info ""
log_info "Installation completed successfully!"
log_info "Services:"
log_info "  Prometheus: http://localhost:9090"
log_info "  Thanos Query: http://localhost:10904"
log_info "  Thanos Sidecar: http://localhost:10902"
log_info "  Thanos Store: http://localhost:10905"
log_info ""
log_warn "Remember to:"
log_warn "1. Update object storage credentials in /etc/thanos/bucket.yml"
log_warn "2. Configure your object storage bucket"
log_warn "3. Adjust firewall rules for external access"

Review the script before running. Execute with: bash install.sh

#thanos #prometheus #long-term-storage #object-storage #monitoring

Configure Prometheus long-term storage with Thanos for unlimited data retention