Implement backup encryption key rotation and secure management with GPG and automated scripts
Advanced
45 min
Jun 09, 2026
Ubuntu 24.04Debian 12AlmaLinux 9Rocky Linux 9
Build a production-grade backup encryption system with automated GPG key rotation, secure key distribution, and monitoring. Learn to implement enterprise-level key management policies with systemd timers and secure storage practices.
Prerequisites
Root access to backup servers
Basic GPG knowledge
SSH access between servers
Mail system for alerts (optional)
What this solves
Manual backup encryption creates security gaps when keys never rotate or become compromised. This tutorial implements automated GPG key rotation for backup systems with secure key distribution, monitoring alerts, and compliance-ready audit logs.
Prerequisites
Before starting: You need root access to your backup servers and basic familiarity with GPG concepts. This builds on GPG backup encryption fundamentals.
Step-by-step implementation
Install required packages
Install GPG tools and secure random number generation utilities for key creation.
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}
Check if current key needs rotation
needs_rotation() {
if [[ ! -L "$CURRENT_KEY_LINK" ]] || [[ ! -e "$CURRENT_KEY_LINK" ]]; then
log "No current key found, rotation needed"
return 0
fi
local current_key_info="$(readlink "$CURRENT_KEY_LINK").info"
if [[ ! -f "$current_key_info" ]]; then
log "Current key info missing, rotation needed"
return 0
fi
# Check expiration
local expires=$(grep '^EXPIRES=' "$current_key_info" | cut -d'=' -f2)
local expires_epoch=$(date -d "$expires" +%s)
local warning_threshold=$((expires_epoch - 604800)) # 7 days before expiration
local current_epoch=$(date +%s)
if [[ $current_epoch -ge $warning_threshold ]]; then
log "Current key expires soon or has expired, rotation needed"
return 0
fi
return 1
}
Archive old key
archive_old_key() {
if [[ -L "$CURRENT_KEY_LINK" ]]; then
local old_key=$(readlink "$CURRENT_KEY_LINK")
local archive_date=$(date +%Y%m%d-%H%M%S)
local archive_path="$ARCHIVE_DIR/archived-$archive_date"
log "Archiving old key: $old_key"
mkdir -p "$archive_path"
cp "${old_key}"* "$archive_path/"
# Create archive manifest
cat > "$archive_path/ARCHIVE_INFO" << EOF
ARCHIVED_DATE=$(date -Iseconds)
ORIGINAL_PATH=$old_key
ARCHIVE_REASON=Key rotation
HOSTNAME=$(hostname -f)
EOF
log "Old key archived to $archive_path"
fi
}
Distribute new key to backup servers
distribute_key() {
local key_file="$1"
local key_id=$(basename "$key_file" .key)
log "Distributing key $key_id to backup servers"
for host in "${DISTRIBUTION_HOSTS[@]}"; do
log "Distributing to $host"
# Create remote directory
ssh "$SSH_USER@$host" "mkdir -p /opt/backup-keys/"
# Copy public key
scp "${key_file}.pub" "$SSH_USER@$host:/opt/backup-keys/${key_id}.pub"
# Copy key info
scp "${key_file}.info" "$SSH_USER@$host:/opt/backup-keys/${key_id}.info"
# Update current link on remote host
ssh "$SSH_USER@$host" "ln -sfn /opt/backup-keys/${key_id} /opt/backup-keys/current"
log "Key distributed to $host successfully"
done
}
Clean up old archived keys
cleanup_old_archives() {
log "Cleaning up old archived keys"
find "$ARCHIVE_DIR" -type d -name "archived-*" -mtime "+$OLD_KEY_RETENTION_DAYS" | while read -r old_archive; do
log "Removing old archive: $old_archive"
srm -rf "$old_archive"
done
}
Send rotation notification
send_notification() {
local key_id="$1"
local action="$2"
# Log to system journal
logger -t backup-key-rotation "$action: $key_id on $(hostname -f)"
# Send to monitoring system (customize as needed)
if command -v curl >/dev/null 2>&1; then
curl -X POST "http://monitoring.example.com/api/events" \
-H "Content-Type: application/json" \
-d "{
\"event\": \"backup_key_rotation\",
\"action\": \"$action\",
\"key_id\": \"$key_id\",
\"hostname\": \"$(hostname -f)\",
\"timestamp\": \"$(date -Iseconds)\"
}" 2>/dev/null || true
fi
}
Main rotation process
perform_rotation() {
log "Starting backup key rotation process"
# Check if rotation is needed
if ! needs_rotation; then
log "Current key is still valid, no rotation needed"
return 0
fi
# Archive current key if it exists
archive_old_key
# Generate new key
local new_key_file
new_key_file=$(create_backup_key)
# Update current key symlink
ln -sfn "$new_key_file" "$CURRENT_KEY_LINK"
# Distribute to backup servers
distribute_key "$new_key_file"
# Clean up old archives
cleanup_old_archives
# Send notifications
local key_id=$(basename "$new_key_file" .key)
send_notification "$key_id" "rotated"
log "Backup key rotation completed successfully"
}
Main execution
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
perform_rotation
fi
Note: Copy the public key content from /opt/backup-encryption/.ssh/id_ed25519.pub and add it to the ~backup-sync/.ssh/authorized_keys file on each backup server listed in the DISTRIBUTION_HOSTS array.
Set up systemd timer for automated rotation
Create systemd service and timer for automated key rotation scheduling.
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}
Send alert
send_alert() {
local subject="$1"
local message="$2"
log "ALERT: $subject"
# Log to system
logger -p user.warning -t backup-key-monitor "$subject: $message"
# Send email if mail is configured
if command -v mail >/dev/null 2>&1; then
echo "$message" | mail -s "Backup Key Alert: $subject" "$ALERT_EMAIL"
fi
# Send to monitoring system
if command -v curl >/dev/null 2>&1; then
curl -X POST "http://monitoring.example.com/api/alerts" \
-H "Content-Type: application/json" \
-d "{
\"alert\": \"backup_key_issue\",
\"subject\": \"$subject\",
\"message\": \"$message\",
\"hostname\": \"$(hostname -f)\",
\"timestamp\": \"$(date -Iseconds)\",
\"severity\": \"warning\"
}" 2>/dev/null || true
fi
}
Check current key status
check_current_key() {
log "Checking current key status"
# Check if current key link exists
if [[ ! -L "$CURRENT_KEY_LINK" ]]; then
send_alert "No Current Key" "Current key symlink missing on $(hostname -f)"
return 1
fi
# Check if target exists
if [[ ! -e "$CURRENT_KEY_LINK" ]]; then
send_alert "Broken Key Link" "Current key symlink points to non-existent file on $(hostname -f)"
return 1
fi
# Check key info file
local key_info="$(readlink "$CURRENT_KEY_LINK").info"
if [[ ! -f "$key_info" ]]; then
send_alert "Missing Key Info" "Key info file missing for current key on $(hostname -f)"
return 1
fi
# Check expiration
local expires=$(grep '^EXPIRES=' "$key_info" | cut -d'=' -f2)
local expires_epoch=$(date -d "$expires" +%s)
local current_epoch=$(date +%s)
local days_until_expiry=$(( (expires_epoch - current_epoch) / 86400 ))
if [[ $days_until_expiry -lt 0 ]]; then
send_alert "Key Expired" "Current backup key expired $((-days_until_expiry)) days ago on $(hostname -f)"
return 1
elif [[ $days_until_expiry -le $EXPIRY_WARNING_DAYS ]]; then
send_alert "Key Expiring Soon" "Current backup key expires in $days_until_expiry days on $(hostname -f)"
fi
log "Current key is healthy, expires in $days_until_expiry days"
return 0
}
Check key permissions
check_permissions() {
log "Checking file permissions"
local issues=0
# Check directory permissions
if [[ "$(stat -c %a "$KEY_DIR")" != "700" ]]; then
send_alert "Permission Issue" "Key directory has incorrect permissions on $(hostname -f)"
issues=$((issues + 1))
fi
# Check secret key files
find "$KEY_DIR" -name "*.sec" | while read -r key_file; do
if [[ "$(stat -c %a "$key_file")" != "600" ]]; then
send_alert "Permission Issue" "Secret key file $key_file has incorrect permissions on $(hostname -f)"
issues=$((issues + 1))
fi
done
if [[ $issues -eq 0 ]]; then
log "File permissions are correct"
fi
return $issues
}
Check disk space
check_disk_space() {
log "Checking disk space"
local usage=$(df "$KEY_DIR" | awk 'NR==2 {print $5}' | sed 's/%//')
if [[ $usage -gt 90 ]]; then
send_alert "Low Disk Space" "Key storage disk is ${usage}% full on $(hostname -f)"
return 1
elif [[ $usage -gt 80 ]]; then
log "WARNING: Disk usage is ${usage}%"
fi
log "Disk usage is ${usage}%"
return 0
}
Check archive cleanup
check_archives() {
log "Checking archived keys"
local archive_count=$(find "$ARCHIVE_DIR" -type d -name "archived-*" | wc -l)
if [[ $archive_count -gt 20 ]]; then
send_alert "Too Many Archives" "Found $archive_count archived keys, cleanup may be needed on $(hostname -f)"
fi
log "Found $archive_count archived keys"
}
Generate health report
generate_report() {
log "Generating key health report"
local current_key_info="$(readlink "$CURRENT_KEY_LINK").info"
local key_id=$(grep '^KEY_ID=' "$current_key_info" | cut -d'=' -f2)
local fingerprint=$(grep '^FINGERPRINT=' "$current_key_info" | cut -d'=' -f2)
local created=$(grep '^CREATED=' "$current_key_info" | cut -d'=' -f2)
local expires=$(grep '^EXPIRES=' "$current_key_info" | cut -d'=' -f2)
cat > "/tmp/key-health-report" << EOF
Backup Key Health Report for $(hostname -f)
Generated: $(date -Iseconds)
Current Key Information:
Key ID: $key_id
Fingerprint: $fingerprint
Created: $created
Expires: $expires
Key Statistics:
Total archived keys: $(find "$ARCHIVE_DIR" -type d -name "archived-*" | wc -l)
Disk usage: $(df "$KEY_DIR" | awk 'NR==2 {print $5}')
Last Rotation: $(find "$KEY_DIR" -name "*.info" -printf '%T@ %p\n' | sort -n | tail -1 | cut -d' ' -f2- | xargs stat -c %y)
EOF
log "Health report generated"
}
Main monitoring function
monitor_keys() {
log "Starting key monitoring check"
local exit_code=0
check_current_key || exit_code=1
check_permissions || exit_code=1
check_disk_space || exit_code=1
check_archives
generate_report
if [[ $exit_code -eq 0 ]]; then
log "Key monitoring completed successfully - all checks passed"
else
log "Key monitoring completed with issues - check alerts"
fi
return $exit_code
}
Main execution
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
monitor_keys
fi
Security note: The system maintains detailed audit logs of all key operations. Review /opt/backup-encryption/logs/ regularly and integrate with your existing compliance automation if required.
Want this handled for you? Running this at scale adds a second layer of work: capacity planning, failover drills, cost control, and on-call. Our managed platform covers monitoring, backups and 24/7 response by default.
Automated install script
Run this to automate the entire setup
install.sh
#!/usr/bin/env bash
set -euo pipefail
# Backup Encryption Key Rotation System Installer
# Production-ready installation script for GPG-based backup encryption with automated key rotation
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Configuration
BACKUP_USER="backup-crypt"
BACKUP_HOME="/opt/backup-encryption"
SERVICE_NAME="backup-key-rotation"
# Logging function
log() {
echo -e "${GREEN}[$(date '+%Y-%m-%d %H:%M:%S')] $*${NC}"
}
warn() {
echo -e "${YELLOW}[WARNING] $*${NC}"
}
error() {
echo -e "${RED}[ERROR] $*${NC}" >&2
}
# Cleanup function
cleanup() {
if [ $? -ne 0 ]; then
error "Installation failed. Cleaning up..."
systemctl stop $SERVICE_NAME 2>/dev/null || true
systemctl disable $SERVICE_NAME 2>/dev/null || true
rm -f /etc/systemd/system/${SERVICE_NAME}.{service,timer}
userdel -r $BACKUP_USER 2>/dev/null || true
rm -rf $BACKUP_HOME 2>/dev/null || true
fi
}
trap cleanup ERR
usage() {
echo "Usage: $0 [OPTIONS]"
echo "Options:"
echo " --monitoring-url URL Monitoring endpoint for key rotation events"
echo " --key-validity PERIOD Key validity period (default: 2y)"
echo " --help Show this help message"
exit 1
}
# Parse arguments
MONITORING_URL=""
KEY_VALIDITY="2y"
while [[ $# -gt 0 ]]; do
case $1 in
--monitoring-url)
MONITORING_URL="$2"
shift 2
;;
--key-validity)
KEY_VALIDITY="$2"
shift 2
;;
--help)
usage
;;
*)
error "Unknown option: $1"
usage
;;
esac
done
# Check prerequisites
echo "[1/8] Checking prerequisites..."
if [ "$EUID" -ne 0 ]; then
error "This script must be run as root"
exit 1
fi
# Detect distribution
if [ -f /etc/os-release ]; then
. /etc/os-release
case "$ID" in
ubuntu|debian)
PKG_MGR="apt"
PKG_INSTALL="apt install -y"
PKG_UPDATE="apt update"
EPEL_PKG=""
;;
almalinux|rocky|centos|rhel|ol|fedora)
PKG_MGR="dnf"
PKG_INSTALL="dnf install -y"
PKG_UPDATE="dnf check-update || true"
EPEL_PKG="epel-release"
;;
amzn)
PKG_MGR="yum"
PKG_INSTALL="yum install -y"
PKG_UPDATE="yum check-update || true"
EPEL_PKG="epel-release"
;;
*)
error "Unsupported distribution: $ID"
exit 1
;;
esac
else
error "Cannot detect distribution"
exit 1
fi
log "Detected distribution: $ID using $PKG_MGR"
# Install packages
echo "[2/8] Installing required packages..."
$PKG_UPDATE
if [ -n "$EPEL_PKG" ]; then
$PKG_INSTALL $EPEL_PKG
fi
# Package names vary by distro
if [ "$ID" = "ubuntu" ] || [ "$ID" = "debian" ]; then
PACKAGES="gnupg2 rng-tools pwgen secure-delete rsync curl"
else
PACKAGES="gnupg2 rng-tools pwgen rsync curl"
# secure-delete might not be available, try wipe instead
$PKG_INSTALL wipe 2>/dev/null || true
fi
$PKG_INSTALL $PACKAGES
# Create backup encryption user
echo "[3/8] Creating backup encryption user..."
if ! id "$BACKUP_USER" &>/dev/null; then
useradd -r -s /bin/bash -d "$BACKUP_HOME" -m "$BACKUP_USER"
fi
# Create directory structure
mkdir -p "$BACKUP_HOME"/{keys,scripts,logs,archive,.gnupg}
chown -R "$BACKUP_USER:$BACKUP_USER" "$BACKUP_HOME"
chmod 700 "$BACKUP_HOME"
chmod 700 "$BACKUP_HOME/.gnupg"
# Configure GPG environment
echo "[4/8] Configuring GPG environment..."
cat > "$BACKUP_HOME/.gnupg/gpg.conf" << 'EOF'
keyserver hkps://keys.openpgp.org
keyserver-options auto-key-retrieve
personal-cipher-preferences AES256 AES192 AES
personal-digest-preferences SHA256 SHA384 SHA512 SHA224
default-preference-list SHA256 SHA384 SHA512 SHA224 AES256 AES192 AES ZLIB BZIP2 ZIP Uncompressed
cipher-algo AES256
digest-algo SHA256
cert-digest-algo SHA256
compress-algo 1
s2k-digest-algo SHA256
s2k-cipher-algo AES256
throw-keyids
no-emit-version
no-comments
keyid-format 0xlong
with-fingerprint
EOF
chmod 644 "$BACKUP_HOME/.gnupg/gpg.conf"
chown "$BACKUP_USER:$BACKUP_USER" "$BACKUP_HOME/.gnupg/gpg.conf"
# Create key rotation script
echo "[5/8] Creating key rotation script..."
cat > "$BACKUP_HOME/scripts/rotate-keys.sh" << EOF
#!/bin/bash
set -euo pipefail
# Configuration
KEY_DIR="$BACKUP_HOME/keys"
ARCHIVE_DIR="$BACKUP_HOME/archive"
LOG_FILE="$BACKUP_HOME/logs/key-rotation.log"
KEY_SIZE=4096
KEY_VALIDITY="$KEY_VALIDITY"
BACKUP_IDENTITY="backup-encryption@\$(hostname -f)"
MONITORING_URL="$MONITORING_URL"
# Logging function
log() {
echo "[\$(date '+%Y-%m-%d %H:%M:%S')] \$*" | tee -a "\$LOG_FILE"
}
# Generate secure passphrase
generate_passphrase() {
pwgen -s -B -1 32
}
# Check if rotation is needed
needs_rotation() {
local current_key="\$KEY_DIR/current.key"
if [ ! -f "\$current_key" ]; then
return 0 # No current key, need rotation
fi
local key_date=\$(stat -c %Y "\$current_key" 2>/dev/null || echo 0)
local current_date=\$(date +%s)
local days_old=\$(( (current_date - key_date) / 86400 ))
# Rotate if key is older than 60 days (configurable)
[ \$days_old -gt 60 ]
}
# Send monitoring alert
send_alert() {
local action="\$1"
local key_id="\$2"
if [ -n "\$MONITORING_URL" ]; then
curl -X POST "\$MONITORING_URL" \\
-H "Content-Type: application/json" \\
-d "{
\\"event\\": \\"backup_key_rotation\\",
\\"action\\": \"\$action\\",
\\"key_id\\": \"\$key_id\\",
\\"hostname\\": \\"\$(hostname -f)\\",
\\"timestamp\\": \\"\$(date -Iseconds)\\"
}" 2>/dev/null || true
fi
}
# Main rotation function
perform_rotation() {
if ! needs_rotation; then
log "Current key is still valid, no rotation needed"
return 0
fi
local key_id="backup-\$(date +%Y%m%d-%H%M%S)"
local passphrase_file="\$KEY_DIR/\${key_id}.passphrase"
log "Starting key rotation: \$key_id"
# Generate passphrase
generate_passphrase > "\$passphrase_file"
chmod 600 "\$passphrase_file"
# Create key
export GNUPGHOME="\$HOME/.gnupg"
gpg --batch --gen-key << KEYEOF
Key-Type: RSA
Key-Length: \$KEY_SIZE
Subkey-Type: RSA
Subkey-Length: \$KEY_SIZE
Name-Real: Backup Encryption Key
Name-Email: \$BACKUP_IDENTITY
Expire-Date: \$KEY_VALIDITY
Passphrase: \$(cat "\$passphrase_file")
%commit
KEYEOF
# Export keys
local fingerprint=\$(gpg --list-secret-keys --with-colons | awk -F: '/fpr:/ { print \$10 }' | tail -1)
gpg --armor --export "\$fingerprint" > "\$KEY_DIR/current.key"
gpg --armor --export-secret-keys "\$fingerprint" > "\$KEY_DIR/current.sec"
chmod 644 "\$KEY_DIR/current.key"
chmod 600 "\$KEY_DIR/current.sec"
log "Key rotation completed: \$key_id"
send_alert "rotated" "\$key_id"
}
# Initialize logging
mkdir -p "\$(dirname "\$LOG_FILE")"
perform_rotation
EOF
chmod 755 "$BACKUP_HOME/scripts/rotate-keys.sh"
chown "$BACKUP_USER:$BACKUP_USER" "$BACKUP_HOME/scripts/rotate-keys.sh"
# Create systemd service
echo "[6/8] Creating systemd service..."
cat > "/etc/systemd/system/${SERVICE_NAME}.service" << EOF
[Unit]
Description=Backup Key Rotation Service
After=network.target
[Service]
Type=oneshot
User=$BACKUP_USER
Group=$BACKUP_USER
WorkingDirectory=$BACKUP_HOME
ExecStart=$BACKUP_HOME/scripts/rotate-keys.sh
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
EOF
cat > "/etc/systemd/system/${SERVICE_NAME}.timer" << EOF
[Unit]
Description=Run backup key rotation weekly
Requires=${SERVICE_NAME}.service
[Timer]
OnCalendar=weekly
Persistent=true
RandomizedDelaySec=3600
[Install]
WantedBy=timers.target
EOF
# Enable and start services
echo "[7/8] Enabling systemd timer..."
systemctl daemon-reload
systemctl enable "${SERVICE_NAME}.timer"
systemctl start "${SERVICE_NAME}.timer"
# Initial key generation
echo "[8/8] Generating initial encryption key..."
sudo -u "$BACKUP_USER" bash -c "cd $BACKUP_HOME && ./scripts/rotate-keys.sh"
# Verification
log "Verifying installation..."
if systemctl is-active --quiet "${SERVICE_NAME}.timer"; then
log "Service timer is active"
else
error "Service timer is not active"
exit 1
fi
if [ -f "$BACKUP_HOME/keys/current.key" ]; then
log "Initial key generated successfully"
else
error "Initial key generation failed"
exit 1
fi
# Display summary
echo -e "\n${GREEN}=== Installation Complete ===${NC}"
echo "Backup encryption system installed successfully!"
echo ""
echo "Key details:"
echo " - Backup user: $BACKUP_USER"
echo " - Installation directory: $BACKUP_HOME"
echo " - Service: ${SERVICE_NAME}.timer"
echo " - Key validity: $KEY_VALIDITY"
echo " - Rotation schedule: Weekly"
echo ""
echo "Management commands:"
echo " - Check status: systemctl status ${SERVICE_NAME}.timer"
echo " - Manual rotation: sudo -u $BACKUP_USER $BACKUP_HOME/scripts/rotate-keys.sh"
echo " - View logs: journalctl -u ${SERVICE_NAME}.service"
echo ""
warn "Remember to:"
warn " 1. Backup your keys securely to an offline location"
warn " 2. Test key rotation and backup processes"
warn " 3. Configure monitoring alerts if not already done"
warn " 4. Update your backup scripts to use the new keys"
Review the script before running. Execute with: bash install.sh