Implement MinIO data lifecycle management and automated archival with S3 retention policies

Update system packages

Start by updating your package manager to ensure you have the latest security patches.

sudo apt update && sudo apt upgrade -y

sudo dnf update -y

Install MinIO client

Download and install the MinIO client to manage lifecycle policies and interact with your MinIO deployment.

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/
mc --version

Configure MinIO client alias

Set up an alias for your MinIO server to simplify command execution. Replace the endpoint and credentials with your actual MinIO configuration.

mc alias set myminio http://203.0.113.10:9000 minioadmin minioadmin
mc admin info myminio

Create test buckets for lifecycle policies

Create buckets to demonstrate different lifecycle management scenarios including archival and expiration.

mc mb myminio/production-data
mc mb myminio/archive-bucket
mc mb myminio/temp-logs
mc ls myminio

Create basic expiration lifecycle policy

Create a lifecycle policy JSON file that expires objects after 90 days for temporary data storage.

{
  "Rules": [
    {
      "ID": "expire-temp-logs",
      "Status": "Enabled",
      "Expiration": {
        "Days": 90
      }
    }
  ]
}

Apply expiration policy to temp logs bucket

Apply the lifecycle policy to automatically delete log files older than 90 days.

mc ilm add --expiry-days 90 myminio/temp-logs
mc ilm ls myminio/temp-logs

Create tiered storage lifecycle policy

Configure a policy that transitions objects to different storage classes based on age for cost optimization.

{
  "Rules": [
    {
      "ID": "production-tiering",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        },
        {
          "Days": 365,
          "StorageClass": "DEEP_ARCHIVE"
        }
      ],
      "Expiration": {
        "Days": 2555
      }
    }
  ]
}

Configure remote storage tier for archival

Set up a remote storage tier for archival. This example uses AWS S3, but you can configure other compatible backends.

mc admin tier add s3 myminio ARCHIVE \
  --endpoint https://s3.amazonaws.com \
  --access-key AKIAIOSFODNN7EXAMPLE \
  --secret-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
  --bucket archive-storage \
  --region us-east-1

Apply tiered lifecycle policy

Apply the comprehensive lifecycle policy with multiple storage tiers to the production data bucket.

mc ilm add --transition-days 30 --storage-class STANDARD_IA myminio/production-data
mc ilm add --transition-days 90 --storage-class ARCHIVE myminio/production-data
mc ilm add --expiry-days 2555 myminio/production-data
mc ilm ls myminio/production-data

Create prefix-based lifecycle rules

Configure different lifecycle policies for specific object prefixes within the same bucket for granular data management.

{
  "Rules": [
    {
      "ID": "logs-retention",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "logs/"
      },
      "Expiration": {
        "Days": 30
      }
    },
    {
      "ID": "backup-archival",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "backups/"
      },
      "Transitions": [
        {
          "Days": 7,
          "StorageClass": "ARCHIVE"
        }
      ],
      "Expiration": {
        "Days": 365
      }
    }
  ]
}

Apply prefix-based policies

Use the MinIO client to apply different lifecycle rules based on object prefixes for fine-grained control.

mc ilm add --prefix "logs/" --expiry-days 30 myminio/production-data
mc ilm add --prefix "backups/" --transition-days 7 --storage-class ARCHIVE myminio/production-data
mc ilm add --prefix "backups/" --expiry-days 365 myminio/production-data

Configure tag-based lifecycle rules

Set up lifecycle policies that apply to objects with specific tags for advanced data classification.

{
  "Rules": [
    {
      "ID": "sensitive-data-retention",
      "Status": "Enabled",
      "Filter": {
        "Tag": {
          "Key": "DataClass",
          "Value": "Sensitive"
        }
      },
      "Expiration": {
        "Days": 2555
      }
    },
    {
      "ID": "temporary-data",
      "Status": "Enabled",
      "Filter": {
        "Tag": {
          "Key": "Retention",
          "Value": "Temporary"
        }
      },
      "Expiration": {
        "Days": 7
      }
    }
  ]
}

Enable lifecycle monitoring and alerting

Configure MinIO to track lifecycle transitions and send notifications for monitoring purposes.

mc admin config set myminio notify_webhook:1 endpoint="http://203.0.113.10:9001/webhook" \
  auth_token="your-webhook-token"
mc admin service restart myminio
mc event add myminio/production-data arn:minio:sqs::1:webhook --event put,delete

Create monitoring script for lifecycle status

Create a script to monitor lifecycle policy execution and generate reports on storage optimization.

#!/bin/bash

MinIO Lifecycle Monitoring Script
DATE=$(date '+%Y-%m-%d %H:%M:%S')
LOG_FILE="/var/log/minio-lifecycle.log"

echo "[$DATE] Checking lifecycle policies..." >> $LOG_FILE

Check lifecycle policies for all buckets
for bucket in $(mc ls myminio | awk '{print $5}'); do
    echo "[$DATE] Bucket: $bucket" >> $LOG_FILE
    mc ilm ls myminio/$bucket >> $LOG_FILE 2>&1
    
    # Get bucket usage statistics
    mc du myminio/$bucket >> $LOG_FILE 2>&1
done

Check tier status
echo "[$DATE] Storage tier status:" >> $LOG_FILE
mc admin tier ls myminio >> $LOG_FILE 2>&1

echo "[$DATE] Lifecycle monitoring completed" >> $LOG_FILE

Make monitoring script executable and schedule

Set proper permissions and create a cron job to run lifecycle monitoring automatically.

sudo chmod +x /usr/local/bin/minio-lifecycle-monitor.sh
sudo crontab -e

Add this line to run monitoring every 6 hours:

0 /6    /usr/local/bin/minio-lifecycle-monitor.sh

Test lifecycle policy execution

Upload test files and verify that lifecycle policies are applied correctly with different scenarios.

# Create test files with different prefixes
echo "Test log data" > /tmp/test-log.txt
echo "Backup data" > /tmp/backup-file.txt
echo "Production data" > /tmp/prod-data.txt

Upload to different prefixes
mc cp /tmp/test-log.txt myminio/production-data/logs/
mc cp /tmp/backup-file.txt myminio/production-data/backups/
mc cp /tmp/prod-data.txt myminio/production-data/data/

Add tags to objects for tag-based policies
mc tag set myminio/production-data/data/prod-data.txt "DataClass=Sensitive"
mc tag set myminio/temp-logs/test-log.txt "Retention=Temporary"

Create compliance reporting script

Generate automated reports on data retention compliance and lifecycle policy effectiveness.

#!/bin/bash

REPORT_DATE=$(date '+%Y-%m-%d')
REPORT_FILE="/var/log/minio-compliance-$REPORT_DATE.json"

Initialize JSON report
cat > $REPORT_FILE << EOF
{
  "report_date": "$REPORT_DATE",
  "buckets": [
EOF

Generate compliance report for each bucket
BUCKET_COUNT=0
for bucket in $(mc ls myminio | awk '{print $5}'); do
    if [ $BUCKET_COUNT -gt 0 ]; then
        echo "," >> $REPORT_FILE
    fi
    
    OBJECT_COUNT=$(mc ls --recursive myminio/$bucket | wc -l)
    BUCKET_SIZE=$(mc du myminio/$bucket | awk '{print $1}')
    
    cat >> $REPORT_FILE << EOF
    {
      "name": "$bucket",
      "object_count": $OBJECT_COUNT,
      "size_bytes": "$BUCKET_SIZE",
      "lifecycle_policies": [
EOF
    
    mc ilm ls myminio/$bucket --json >> $REPORT_FILE 2>/dev/null
    
    echo "      ]" >> $REPORT_FILE
    echo "    }" >> $REPORT_FILE
    
    BUCKET_COUNT=$((BUCKET_COUNT + 1))
done

cat >> $REPORT_FILE << EOF
  ],
  "storage_tiers": [
EOF

mc admin tier ls myminio --json >> $REPORT_FILE 2>/dev/null

cat >> $REPORT_FILE << EOF
  ]
}
EOF

echo "Compliance report generated: $REPORT_FILE"

Set up automated compliance reporting

Configure the compliance report to run weekly and optionally email results to administrators.

sudo chmod +x /usr/local/bin/minio-compliance-report.sh

Add to crontab for weekly execution
sudo crontab -e

Add this line for weekly reports on Sundays at 2 AM:

0 2   0 /usr/local/bin/minio-compliance-report.sh

Configure multipart upload cleanup

Set up automatic cleanup of incomplete multipart uploads to free up storage space.

mc ilm add --incomplete-upload-days 7 myminio/production-data
mc ilm add --incomplete-upload-days 1 myminio/temp-logs

Set up version-aware lifecycle policies

Configure policies that handle object versioning for comprehensive data lifecycle management.

{
  "Rules": [
    {
      "ID": "version-management",
      "Status": "Enabled",
      "NoncurrentVersionTransitions": [
        {
          "NoncurrentDays": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "NoncurrentDays": 90,
          "StorageClass": "ARCHIVE"
        }
      ],
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 365
      }
    }
  ]
}

Enable bucket versioning and apply version policies

Enable versioning on buckets and apply lifecycle policies that manage both current and non-current object versions.

# Enable versioning
mc version enable myminio/production-data

Apply version-aware lifecycle policies
mc ilm add --noncurrent-version-transition-days 30 --noncurrent-version-transition-storage-class STANDARD_IA myminio/production-data
mc ilm add --noncurrent-version-expiration-days 365 myminio/production-data