Implement backup encryption key rotation and secure management with GPG and automated scripts

Advanced 45 min Jun 09, 2026 414 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Build a production-grade backup encryption system with automated GPG key rotation, secure key distribution, and monitoring. Learn to implement enterprise-level key management policies with systemd timers and secure storage practices.

Prerequisites

  • Root access to backup servers
  • Basic GPG knowledge
  • SSH access between servers
  • Mail system for alerts (optional)

What this solves

Manual backup encryption creates security gaps when keys never rotate or become compromised. This tutorial implements automated GPG key rotation for backup systems with secure key distribution, monitoring alerts, and compliance-ready audit logs.

Prerequisites

Before starting: You need root access to your backup servers and basic familiarity with GPG concepts. This builds on GPG backup encryption fundamentals.

Step-by-step implementation

Install required packages

Install GPG tools and secure random number generation utilities for key creation.

sudo apt update
sudo apt install -y gnupg2 rng-tools pwgen secure-delete rsync
sudo dnf install -y gnupg2 rng-tools pwgen rsync
sudo dnf install -y epel-release
sudo dnf install -y secure-delete

Create dedicated backup encryption user

Create a system user to isolate backup encryption operations from other system processes.

sudo useradd -r -s /bin/bash -d /opt/backup-encryption -m backup-crypt
sudo mkdir -p /opt/backup-encryption/{keys,scripts,logs,archive}
sudo chown -R backup-crypt:backup-crypt /opt/backup-encryption
sudo chmod 700 /opt/backup-encryption

Configure GPG environment

Set up GPG configuration with strong defaults for backup encryption keys.

keyserver hkps://keys.openpgp.org
keyserver-options auto-key-retrieve
personal-cipher-preferences AES256 AES192 AES
personal-digest-preferences SHA256 SHA384 SHA512 SHA224
default-preference-list SHA256 SHA384 SHA512 SHA224 AES256 AES192 AES ZLIB BZIP2 ZIP Uncompressed
cipher-algo AES256
digest-algo SHA256
cert-digest-algo SHA256
compress-algo 1
s2k-digest-algo SHA256
s2k-cipher-algo AES256
throw-keyids
no-emit-version
no-comments
keyid-format 0xlong
with-fingerprint
sudo -u backup-crypt mkdir -p /opt/backup-encryption/.gnupg
sudo -u backup-crypt chmod 700 /opt/backup-encryption/.gnupg

Create master key generation script

Build the script that generates new encryption keys with proper metadata and expiration dates.

#!/bin/bash
set -euo pipefail

# Configuration
KEY_DIR="/opt/backup-encryption/keys"
ARCHIVE_DIR="/opt/backup-encryption/archive"
LOG_FILE="/opt/backup-encryption/logs/key-rotation.log"
KEY_SIZE=4096
KEY_VALIDITY="2y"  # 2 years validity
BACKUP_IDENTITY="backup-encryption@$(hostname -f)"

# Logging function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

# Generate secure passphrase
generate_passphrase() {
    pwgen -s -B -1 32
}

# Create new backup key
create_backup_key() {
    local key_id="backup-$(date +%Y%m%d-%H%M%S)"
    local passphrase_file="$KEY_DIR/${key_id}.passphrase"
    local key_file="$KEY_DIR/${key_id}.key"
    
    log "Generating new backup encryption key: $key_id"
    
    # Generate secure passphrase
    generate_passphrase > "$passphrase_file"
    chmod 600 "$passphrase_file"
    
    # Create key generation batch file
    cat > "/tmp/key-gen-$key_id" << EOF
%echo Generating backup encryption key
Key-Type: RSA
Key-Length: $KEY_SIZE
Subkey-Type: RSA
Subkey-Length: $KEY_SIZE
Name-Real: Backup Encryption Key
Name-Comment: $key_id
Name-Email: $BACKUP_IDENTITY
Expire-Date: $KEY_VALIDITY
Passphrase: $(cat "$passphrase_file")
%commit
%echo Key generation complete
EOF
    
    # Generate the key
    gpg --batch --generate-key "/tmp/key-gen-$key_id"
    
    # Export the key
    local fingerprint=$(gpg --list-secret-keys --with-colons "$BACKUP_IDENTITY" | grep fpr | head -1 | cut -d: -f10)
    
    gpg --armor --export "$fingerprint" > "${key_file}.pub"
    gpg --armor --export-secret-keys "$fingerprint" > "${key_file}.sec"
    
    # Set proper permissions
    chmod 600 "${key_file}.sec"
    chmod 644 "${key_file}.pub"
    
    # Store key metadata
    cat > "${key_file}.info" << EOF
KEY_ID=$key_id
FINGERPRINT=$fingerprint
CREATED=$(date -Iseconds)
EXPIRES=$(date -d "+$KEY_VALIDITY" -Iseconds)
HOSTNAME=$(hostname -f)
EOF
    
    # Cleanup
    srm -f "/tmp/key-gen-$key_id"
    
    log "Key $key_id created successfully with fingerprint $fingerprint"
    echo "$key_file"
}

# Main execution
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    create_backup_key
fi
sudo chown backup-crypt:backup-crypt /opt/backup-encryption/scripts/generate-backup-key.sh
sudo chmod 750 /opt/backup-encryption/scripts/generate-backup-key.sh

Implement key rotation automation

Create the main rotation script that manages key lifecycle, archival, and distribution.

#!/bin/bash
set -euo pipefail

# Configuration
KEY_DIR="/opt/backup-encryption/keys"
ARCHIVE_DIR="/opt/backup-encryption/archive"
LOG_FILE="/opt/backup-encryption/logs/key-rotation.log"
CURRENT_KEY_LINK="$KEY_DIR/current"
OLD_KEY_RETENTION_DAYS=90
DISTRIBUTION_HOSTS=(
    "backup-server-1.example.com"
    "backup-server-2.example.com"
)
SSH_USER="backup-sync"

# Source the key generation functions
source "/opt/backup-encryption/scripts/generate-backup-key.sh"

# Logging function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

# Check if current key needs rotation
needs_rotation() {
    if [[ ! -L "$CURRENT_KEY_LINK" ]] || [[ ! -e "$CURRENT_KEY_LINK" ]]; then
        log "No current key found, rotation needed"
        return 0
    fi
    
    local current_key_info="$(readlink "$CURRENT_KEY_LINK").info"
    
    if [[ ! -f "$current_key_info" ]]; then
        log "Current key info missing, rotation needed"
        return 0
    fi
    
    # Check expiration
    local expires=$(grep '^EXPIRES=' "$current_key_info" | cut -d'=' -f2)
    local expires_epoch=$(date -d "$expires" +%s)
    local warning_threshold=$((expires_epoch - 604800))  # 7 days before expiration
    local current_epoch=$(date +%s)
    
    if [[ $current_epoch -ge $warning_threshold ]]; then
        log "Current key expires soon or has expired, rotation needed"
        return 0
    fi
    
    return 1
}

# Archive old key
archive_old_key() {
    if [[ -L "$CURRENT_KEY_LINK" ]]; then
        local old_key=$(readlink "$CURRENT_KEY_LINK")
        local archive_date=$(date +%Y%m%d-%H%M%S)
        local archive_path="$ARCHIVE_DIR/archived-$archive_date"
        
        log "Archiving old key: $old_key"
        
        mkdir -p "$archive_path"
        cp "${old_key}"* "$archive_path/"
        
        # Create archive manifest
        cat > "$archive_path/ARCHIVE_INFO" << EOF
ARCHIVED_DATE=$(date -Iseconds)
ORIGINAL_PATH=$old_key
ARCHIVE_REASON=Key rotation
HOSTNAME=$(hostname -f)
EOF
        
        log "Old key archived to $archive_path"
    fi
}

# Distribute new key to backup servers
distribute_key() {
    local key_file="$1"
    local key_id=$(basename "$key_file" .key)
    
    log "Distributing key $key_id to backup servers"
    
    for host in "${DISTRIBUTION_HOSTS[@]}"; do
        log "Distributing to $host"
        
        # Create remote directory
        ssh "$SSH_USER@$host" "mkdir -p /opt/backup-keys/"
        
        # Copy public key
        scp "${key_file}.pub" "$SSH_USER@$host:/opt/backup-keys/${key_id}.pub"
        
        # Copy key info
        scp "${key_file}.info" "$SSH_USER@$host:/opt/backup-keys/${key_id}.info"
        
        # Update current link on remote host
        ssh "$SSH_USER@$host" "ln -sfn /opt/backup-keys/${key_id} /opt/backup-keys/current"
        
        log "Key distributed to $host successfully"
    done
}

# Clean up old archived keys
cleanup_old_archives() {
    log "Cleaning up old archived keys"
    
    find "$ARCHIVE_DIR" -type d -name "archived-*" -mtime "+$OLD_KEY_RETENTION_DAYS" | while read -r old_archive; do
        log "Removing old archive: $old_archive"
        srm -rf "$old_archive"
    done
}

# Send rotation notification
send_notification() {
    local key_id="$1"
    local action="$2"
    
    # Log to system journal
    logger -t backup-key-rotation "$action: $key_id on $(hostname -f)"
    
    # Send to monitoring system (customize as needed)
    if command -v curl >/dev/null 2>&1; then
        curl -X POST "http://monitoring.example.com/api/events" \
            -H "Content-Type: application/json" \
            -d "{
                \"event\": \"backup_key_rotation\",
                \"action\": \"$action\",
                \"key_id\": \"$key_id\",
                \"hostname\": \"$(hostname -f)\",
                \"timestamp\": \"$(date -Iseconds)\"
            }" 2>/dev/null || true
    fi
}

# Main rotation process
perform_rotation() {
    log "Starting backup key rotation process"
    
    # Check if rotation is needed
    if ! needs_rotation; then
        log "Current key is still valid, no rotation needed"
        return 0
    fi
    
    # Archive current key if it exists
    archive_old_key
    
    # Generate new key
    local new_key_file
    new_key_file=$(create_backup_key)
    
    # Update current key symlink
    ln -sfn "$new_key_file" "$CURRENT_KEY_LINK"
    
    # Distribute to backup servers
    distribute_key "$new_key_file"
    
    # Clean up old archives
    cleanup_old_archives
    
    # Send notifications
    local key_id=$(basename "$new_key_file" .key)
    send_notification "$key_id" "rotated"
    
    log "Backup key rotation completed successfully"
}

# Main execution
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    perform_rotation
fi
sudo chown backup-crypt:backup-crypt /opt/backup-encryption/scripts/rotate-backup-keys.sh
sudo chmod 750 /opt/backup-encryption/scripts/rotate-backup-keys.sh

Create backup encryption wrapper

Build a wrapper script that uses the current key for backup operations with proper error handling.

#!/bin/bash
set -euo pipefail

# Configuration
KEY_DIR="/opt/backup-encryption/keys"
CURRENT_KEY_LINK="$KEY_DIR/current"
LOG_FILE="/opt/backup-encryption/logs/backup-encryption.log"

# Logging function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

# Usage information
usage() {
    echo "Usage: $0 [options]  
sudo chown backup-crypt:backup-crypt /opt/backup-encryption/scripts/encrypt-backup.sh
sudo chmod 755 /opt/backup-encryption/scripts/encrypt-backup.sh

Configure SSH key distribution

Set up SSH keys for secure distribution to backup servers without password prompts.

sudo -u backup-crypt ssh-keygen -t ed25519 -f /opt/backup-encryption/.ssh/id_ed25519 -N ""
sudo -u backup-crypt chmod 600 /opt/backup-encryption/.ssh/id_ed25519
sudo -u backup-crypt chmod 644 /opt/backup-encryption/.ssh/id_ed25519.pub
Note: Copy the public key content from /opt/backup-encryption/.ssh/id_ed25519.pub and add it to the ~backup-sync/.ssh/authorized_keys file on each backup server listed in the DISTRIBUTION_HOSTS array.

Set up systemd timer for automated rotation

Create systemd service and timer for automated key rotation scheduling.

[Unit]
Description=Backup Encryption Key Rotation
Wants=backup-key-rotation.timer

[Service]
Type=oneshot
User=backup-crypt
Group=backup-crypt
WorkingDirectory=/opt/backup-encryption
Environment="GNUPGHOME=/opt/backup-encryption/.gnupg"
ExecStart=/opt/backup-encryption/scripts/rotate-backup-keys.sh
PrivateTmp=true
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/backup-encryption

[Install]
WantedBy=multi-user.target
[Unit]
Description=Run backup key rotation weekly
Requires=backup-key-rotation.service

[Timer]
# Run every Sunday at 2:00 AM
OnCalendar=Sun *-*-* 02:00:00
# Run 5 minutes after boot if we missed the scheduled time
OnBootSec=5min
# Add randomization to prevent thundering herd
RandomizedDelaySec=30min
Persistent=true

[Install]
WantedBy=timers.target
sudo systemctl daemon-reload
sudo systemctl enable backup-key-rotation.timer
sudo systemctl start backup-key-rotation.timer

Create monitoring and alerting

Implement monitoring script to check key health and send alerts for issues.

#!/bin/bash
set -euo pipefail

# Configuration
KEY_DIR="/opt/backup-encryption/keys"
ARCHIVE_DIR="/opt/backup-encryption/archive"
CURRENT_KEY_LINK="$KEY_DIR/current"
LOG_FILE="/opt/backup-encryption/logs/key-monitor.log"
ALERT_EMAIL="admin@example.com"
EXPIRY_WARNING_DAYS=14

# Logging function
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

# Send alert
send_alert() {
    local subject="$1"
    local message="$2"
    
    log "ALERT: $subject"
    
    # Log to system
    logger -p user.warning -t backup-key-monitor "$subject: $message"
    
    # Send email if mail is configured
    if command -v mail >/dev/null 2>&1; then
        echo "$message" | mail -s "Backup Key Alert: $subject" "$ALERT_EMAIL"
    fi
    
    # Send to monitoring system
    if command -v curl >/dev/null 2>&1; then
        curl -X POST "http://monitoring.example.com/api/alerts" \
            -H "Content-Type: application/json" \
            -d "{
                \"alert\": \"backup_key_issue\",
                \"subject\": \"$subject\",
                \"message\": \"$message\",
                \"hostname\": \"$(hostname -f)\",
                \"timestamp\": \"$(date -Iseconds)\",
                \"severity\": \"warning\"
            }" 2>/dev/null || true
    fi
}

# Check current key status
check_current_key() {
    log "Checking current key status"
    
    # Check if current key link exists
    if [[ ! -L "$CURRENT_KEY_LINK" ]]; then
        send_alert "No Current Key" "Current key symlink missing on $(hostname -f)"
        return 1
    fi
    
    # Check if target exists
    if [[ ! -e "$CURRENT_KEY_LINK" ]]; then
        send_alert "Broken Key Link" "Current key symlink points to non-existent file on $(hostname -f)"
        return 1
    fi
    
    # Check key info file
    local key_info="$(readlink "$CURRENT_KEY_LINK").info"
    if [[ ! -f "$key_info" ]]; then
        send_alert "Missing Key Info" "Key info file missing for current key on $(hostname -f)"
        return 1
    fi
    
    # Check expiration
    local expires=$(grep '^EXPIRES=' "$key_info" | cut -d'=' -f2)
    local expires_epoch=$(date -d "$expires" +%s)
    local current_epoch=$(date +%s)
    local days_until_expiry=$(( (expires_epoch - current_epoch) / 86400 ))
    
    if [[ $days_until_expiry -lt 0 ]]; then
        send_alert "Key Expired" "Current backup key expired $((-days_until_expiry)) days ago on $(hostn

Automated install script

Run this to automate the entire setup

Don't want to manage this yourself?

We handle infrastructure for businesses that depend on uptime. Fully managed, with one fixed contact who knows your setup.

You get one fixed contact who knows your setup

Rotterdam 05:15 · reachable in a message, no ticket form