Learn how to build a cluster-safe backup strategy for MariaDB Galera using Mariabackup, compression, systemd timers and remote storage offloading, with verification and node restore procedures.
Prerequisites
- A running MariaDB Galera cluster with at least one accessible node
- Root or sudo access on the backup node
- Basic familiarity with systemd units and shell scripting
- An S3-compatible storage bucket for offsite backups (optional)
What this solves
Galera cluster nodes share the same dataset through synchronous replication, but that does not mean backups are optional. A bad schema migration, an accidental DELETE, or a corrupted table replicates to every node instantly. This tutorial builds a repeatable, compressed, automated backup strategy using Mariabackup, with rotation, integrity checks and offsite storage.
Choosing a backup strategy for Galera clusters
mysqldump produces logical SQL dumps. It is portable and human-readable, but slow to restore on large datasets and it locks tables during the dump unless you use single-transaction mode, which does not fully guarantee consistency on a busy Galera node.
Mariabackup performs physical, hot backups of InnoDB data files without blocking writes. It understands Galera's gtid_binlog_state and wsrep_position, which lets a restored node rejoin the cluster cleanly via State Snapshot Transfer (SST) or by seeding a fresh node. For clusters beyond a few gigabytes, Mariabackup is the right default.
| Criteria | Mariabackup | mysqldump |
|---|---|---|
| Backup speed on large datasets | Fast, file-level copy | Slow, row-by-row export |
| Restore speed | Fast, direct file copy | Slow, replays SQL statements |
| Locking impact on cluster | None, uses InnoDB redo log tracking | Can lock tables briefly |
| Selective table/database restore | Harder, requires partial backup mode | Easy, just re-import specific tables |
| Best use case | Full cluster or node backups | Schema migrations, small exports, dev seeding |
In practice, most production Galera setups run Mariabackup as the primary backup mechanism and keep mysqldump available for occasional logical exports of specific schemas.
Step-by-step configuration
Install Mariabackup
Mariabackup ships as part of the MariaDB backup package. Install it on the node you plan to run backups from, ideally a non-primary node to avoid extra load on the write path.
sudo apt update
sudo apt install -y mariadb-backup gzip zstdsudo dnf install -y MariaDB-backup gzip zstdCreate a dedicated backup user
Avoid using the root MariaDB account in scripts. Create a user scoped to only the privileges Mariabackup needs.
sudo mysql -u root -pCREATE USER 'backupuser'@'localhost' IDENTIFIED BY 'Kx9#mPz2Trq!Lw7v';
GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT, BACKUP_ADMIN ON *.* TO 'backupuser'@'localhost';
FLUSH PRIVILEGES;
EXIT;Store credentials securely
Create a MariaDB option file readable only by the backup script's user, instead of embedding the password in the script or command history.
sudo mkdir -p /etc/mysql/backup
sudo tee /etc/mysql/backup/.mycnf > /dev/null <<'EOF'
[client]
user=backupuser
password=Kx9#mPz2Trq!Lw7v
EOF
sudo chown root:root /etc/mysql/backup/.mycnf
sudo chmod 600 /etc/mysql/backup/.mycnfSetting permissions to 600 means only the file owner, root in this case, can read or write it. This prevents other local users from reading the database password.
Create the backup directory structure
Separate raw backups from compressed archives, and keep a dedicated log directory.
sudo mkdir -p /var/backups/mariadb/{staging,archives,logs}
sudo chown -R mysql:mysql /var/backups/mariadb
sudo chmod 750 /var/backups/mariadb
sudo chmod 750 /var/backups/mariadb/staging /var/backups/mariadb/archives /var/backups/mariadb/logsThe mysql system user needs to read and write here since Mariabackup runs under that account during the backup. 750 permissions allow the owner full access and the group read/execute, while blocking all other users.
Write the backup script with compression and rotation
This script takes a full Mariabackup snapshot, streams it through zstd for fast compression, timestamps the archive, and deletes archives older than the retention window.
#!/bin/bash
set -euo pipefail
BACKUP_USER_CNF="/etc/mysql/backup/.mycnf"
STAGING_DIR="/var/backups/mariadb/staging"
ARCHIVE_DIR="/var/backups/mariadb/archives"
LOG_DIR="/var/backups/mariadb/logs"
RETENTION_DAYS=7
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_NAME="galera-backup-${TIMESTAMP}"
LOG_FILE="${LOG_DIR}/${BACKUP_NAME}.log"
mkdir -p "${STAGING_DIR}/${BACKUP_NAME}"
echo "[$(date)] Starting Mariabackup for ${BACKUP_NAME}" >> "${LOG_FILE}"
mariabackup --defaults-extra-file="${BACKUP_USER_CNF}" \
--backup \
--target-dir="${STAGING_DIR}/${BACKUP_NAME}" \
>> "${LOG_FILE}" 2>&1
mariabackup --defaults-extra-file="${BACKUP_USER_CNF}" \
--prepare \
--target-dir="${STAGING_DIR}/${BACKUP_NAME}" \
>> "${LOG_FILE}" 2>&1
echo "[$(date)] Compressing backup with zstd" >> "${LOG_FILE}"
tar -cf - -C "${STAGING_DIR}" "${BACKUP_NAME}" | zstd -T0 -19 -o "${ARCHIVE_DIR}/${BACKUP_NAME}.tar.zst"
rm -rf "${STAGING_DIR}/${BACKUP_NAME}"
echo "[$(date)] Rotating archives older than ${RETENTION_DAYS} days" >> "${LOG_FILE}"
find "${ARCHIVE_DIR}" -name "galera-backup-*.tar.zst" -mtime +${RETENTION_DAYS} -delete
find "${LOG_DIR}" -name "galera-backup-*.log" -mtime +${RETENTION_DAYS} -delete
echo "[$(date)] Backup ${BACKUP_NAME} completed successfully" >> "${LOG_FILE}"zstd at level 19 with multi-threading (-T0) gives strong compression without the long runtimes of gzip -9 on multi-gigabyte datasets. If you prefer gzip for compatibility with older tooling, replace the compression line with tar -czf "${ARCHIVE_DIR}/${BACKUP_NAME}.tar.gz" -C "${STAGING_DIR}" "${BACKUP_NAME}".
Set correct ownership and permissions on the script
The script needs to run as the mysql user since that account owns the data directory and staging path. It should not be world-writable.
sudo chown root:mysql /usr/local/bin/galera-backup.sh
sudo chmod 750 /usr/local/bin/galera-backup.shTest the script manually
Run it once by hand before automating it, so you can confirm output and catch permission errors early.
sudo -u mysql /usr/local/bin/galera-backup.sh
ls -lh /var/backups/mariadb/archivesAutomating backups with systemd timers
systemd timers give you better logging, dependency management and failure visibility than plain cron, and they integrate cleanly with journalctl. If you already manage scheduled jobs with cron elsewhere, the same rotation principles apply, see Implement backup rotation policies with automated cleanup.
Create the systemd service unit
This defines what runs, and as which user.
[Unit]
Description=Galera cluster backup with Mariabackup
After=mariadb.service
Requires=mariadb.service
[Service]
Type=oneshot
User=mysql
Group=mysql
ExecStart=/usr/local/bin/galera-backup.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7Nice and IOSchedulingClass keep the backup from starving the mysqld process of CPU and disk I/O during peak hours.
Create the systemd timer unit
This schedules the service to run nightly at 02:15, with a randomized delay to avoid every cluster node hitting disk I/O at the exact same second.
[Unit]
Description=Run Galera backup nightly
[Timer]
OnCalendar=*-*-* 02:15:00
RandomizedDelaySec=600
Persistent=true
[Install]
WantedBy=timers.targetEnable and start the timer
sudo systemctl daemon-reload
sudo systemctl enable --now galera-backup.timer
systemctl list-timers galera-backup.timerStagger schedules across cluster nodes
Running full backups on every node simultaneously wastes disk I/O and network bandwidth for no benefit, since the data is identical. Pick one designated backup node per cluster, and only enable the timer there. Keep the package installed on other nodes as a manual fallback.
sudo systemctl disable --now galera-backup.timerRun this on the nodes that should not perform scheduled backups. If your designated backup node goes down, you can enable the timer on a secondary node temporarily.
Verifying backup integrity and restoring a node
Verify archive integrity
Confirm the compressed archive is not truncated or corrupted before you rely on it.
zstd -t /var/backups/mariadb/archives/galera-backup-20250601-021500.tar.zst
echo $?An exit code of 0 means the archive passed integrity checks. Any other value means the file is corrupted and you should investigate the backup job's logs.
Extract and prepare for restore
Decompress into a scratch location, separate from the live data directory.
mkdir -p /tmp/restore-test
zstd -d /var/backups/mariadb/archives/galera-backup-20250601-021500.tar.zst -o /tmp/restore-test/backup.tar
tar -xf /tmp/restore-test/backup.tar -C /tmp/restore-testRestore a failed or new node
Stop MariaDB, clear the existing data directory and copy the prepared backup into place. Do this only on a node you are rebuilding, never on a healthy node still serving traffic.
sudo systemctl stop mariadb
sudo mv /var/lib/mysql /var/lib/mysql.bak
sudo mkdir /var/lib/mysql
sudo mariabackup --copy-back --target-dir=/tmp/restore-test/galera-backup-20250601-021500
sudo chown -R mysql:mysql /var/lib/mysql
sudo chmod 750 /var/lib/mysqlStart the restored node and rejoin the cluster
Start MariaDB and confirm it rejoins via SST or reports a healthy wsrep state.
sudo systemctl start mariadb
sudo mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
sudo mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_local_state_comment';"You want wsrep_local_state_comment to report Synced. For full cluster setup details, see Configure MariaDB Galera cluster 10.11 for multi-master replication.
Offloading backups to remote or S3-compatible storage
Local backups protect against data corruption, but not against disk failure or a compromised host. Offload archives to a remote location as part of the same automation.
Install the S3-compatible client
sudo apt install -y awscliConfigure remote credentials
Store credentials in a restricted profile file, not inline in scripts.
sudo -u mysql aws configure --profile galera-backupProvide your access key, secret key, and region when prompted. This writes to ~mysql/.aws/credentials, which should already inherit restrictive permissions from the mysql home directory.
Add the upload step to the backup script
Append this block to the end of the backup script, after the rotation step.
echo "[$(date)] Uploading archive to remote storage" >> "${LOG_FILE}"
aws s3 cp "${ARCHIVE_DIR}/${BACKUP_NAME}.tar.zst" \
s3://example-galera-backups/nightly/ \
--profile galera-backup \
--endpoint-url https://s3.eu-central-1.example.com \
>> "${LOG_FILE}" 2>&1
echo "[$(date)] Remote upload complete" >> "${LOG_FILE}"Point --endpoint-url at your actual S3-compatible provider. For a self-hosted alternative, see Setup S3-compatible disaster recovery with cross-region replication using MinIO.
Encrypt archives before upload for sensitive workloads
If your data falls under GDPR or contains customer PII, encrypt the archive before it leaves the host, rather than relying solely on transport encryption.
gpg --batch --yes --passphrase-file /etc/mysql/backup/.gpgpass \
--cipher-algo AES256 \
--symmetric \
--output "${ARCHIVE_DIR}/${BACKUP_NAME}.tar.zst.gpg" \
"${ARCHIVE_DIR}/${BACKUP_NAME}.tar.zst"For a full walkthrough of key rotation and secure passphrase handling, see Implement MariaDB backup encryption with Mariabackup and automated restoration.
Verify your setup
systemctl list-timers galera-backup.timer
journalctl -u galera-backup.service --since "1 day ago"
ls -lh /var/backups/mariadb/archives
aws s3 ls s3://example-galera-backups/nightly/ --profile galera-backup --endpoint-url https://s3.eu-central-1.example.comConfirm at least one successful run exists in the journal, the archive directory contains recent files matching your retention window, and the remote bucket lists the same file count.
Common issues
| Symptom | Cause | Fix |
|---|---|---|
| Mariabackup fails with access denied | backupuser missing BACKUP_ADMIN or RELOAD privilege | Re-run the GRANT statement and FLUSH PRIVILEGES |
| Backup succeeds but restore fails to start mariadb | Prepare step was skipped or failed silently | Re-run mariabackup --prepare against the extracted backup before copy-back |
| Timer never triggers | Timer enabled but not started, or unit file typo | systemctl status galera-backup.timer and check for load errors |
| Archive directory fills up disk | Retention find command not matching filenames | Check filename pattern matches exactly what the script produces |
| S3 upload fails with 403 | Wrong endpoint URL or expired credentials | Re-run aws configure and verify bucket policy allows PutObject |
| Restored node stuck at Joining state | Corrupted backup or mismatched Galera version | Verify archive with zstd -t before restore, confirm mariadb-server version matches cluster |
Next steps
- Configure MariaDB Galera cluster for multi-master replication with automatic failover
- Implement MariaDB backup encryption with Mariabackup and automated restoration
- Configure MariaDB 11.6 performance monitoring with Prometheus and Grafana dashboards
- Set up MariaDB backup alerting with Prometheus and Grafana
- Set up Galera cluster disaster recovery with cross-region replication
Running this in production?
Automated install script
Run this to automate the entire setup
#!/usr/bin/env bash
set -euo pipefail
# ============================================================
# MariaDB Galera cluster backup setup script
# Installs Mariabackup, creates a scoped backup user, secures
# credentials, sets up directory structure, and installs a
# compressed/rotated backup script with a cron schedule.
# ============================================================
USAGE="Usage: $0 -p <backup_user_password> [-u backupuser] [-r retention_days] [-c cron_schedule]
-p Password for the dedicated backup MySQL user (required)
-u Backup MySQL username (default: backupuser)
-r Retention in days for archives (default: 7)
-c Cron schedule string (default: '0 2 * * *')"
BACKUP_USER="backupuser"
RETENTION_DAYS=7
CRON_SCHEDULE="0 2 * * *"
BACKUP_PASSWORD=""
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m'
log_info() { echo -e "${GREEN}$1${NC}"; }
log_warn() { echo -e "${YELLOW}$1${NC}"; }
log_error() { echo -e "${RED}$1${NC}" >&2; }
while getopts ":p:u:r:c:h" opt; do
case "$opt" in
p) BACKUP_PASSWORD="$OPTARG" ;;
u) BACKUP_USER="$OPTARG" ;;
r) RETENTION_DAYS="$OPTARG" ;;
c) CRON_SCHEDULE="$OPTARG" ;;
h) echo "$USAGE"; exit 0 ;;
*) log_error "$USAGE"; exit 1 ;;
esac
done
if [ -z "$BACKUP_PASSWORD" ]; then
log_error "Missing required password."
echo "$USAGE"
exit 1
fi
if [ "$(id -u)" -ne 0 ]; then
log_error "This script must be run as root (use sudo)."
exit 1
fi
# Cleanup on failure: remove partially created credential file only
cleanup_on_error() {
log_error "[ERROR] Setup failed. Rolling back sensitive artifacts..."
rm -f /etc/mysql/backup/.mycnf 2>/dev/null || true
}
trap cleanup_on_error ERR
# ---------------------------------------------------------
# [1/8] Detect distro and package manager
# ---------------------------------------------------------
echo "[1/8] Detecting distribution..."
if [ -f /etc/os-release ]; then
. /etc/os-release
case "$ID" in
ubuntu|debian)
PKG_MGR="apt"; PKG_INSTALL="apt install -y"
BACKUP_PKG="mariadb-backup"
MYCNF_BASE="/etc/mysql"
;;
almalinux|rocky|centos|rhel|ol|fedora)
PKG_MGR="dnf"; PKG_INSTALL="dnf install -y"
BACKUP_PKG="MariaDB-backup"
MYCNF_BASE="/etc/my.cnf.d"
;;
amzn)
PKG_MGR="yum"; PKG_INSTALL="yum install -y"
BACKUP_PKG="MariaDB-backup"
MYCNF_BASE="/etc/my.cnf.d"
;;
*)
log_error "Unsupported distro: $ID"
exit 1
;;
esac
else
log_error "/etc/os-release not found. Cannot detect distro."
exit 1
fi
log_info "Detected distro: $ID (using $PKG_MGR)"
# ---------------------------------------------------------
# [2/8] Verify MariaDB client is present
# ---------------------------------------------------------
echo "[2/8] Checking for MariaDB/MySQL client..."
if ! command -v mysql >/dev/null 2>&1; then
log_error "mysql client not found. Install/configure MariaDB Galera cluster first."
exit 1
fi
log_info "mysql client found."
# ---------------------------------------------------------
# [3/8] Install Mariabackup and compression tools
# ---------------------------------------------------------
echo "[3/8] Installing Mariabackup, gzip, and zstd..."
if [ "$PKG_MGR" = "apt" ]; then
apt update
fi
$PKG_INSTALL "$BACKUP_PKG" gzip zstd
log_info "Packages installed."
# ---------------------------------------------------------
# [4/8] Create dedicated backup MySQL user
# ---------------------------------------------------------
echo "[4/8] Creating dedicated backup MySQL user '$BACKUP_USER'..."
mysql -u root <<SQL
CREATE USER IF NOT EXISTS '${BACKUP_USER}'@'localhost' IDENTIFIED BY '${BACKUP_PASSWORD}';
GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT, BACKUP_ADMIN ON *.* TO '${BACKUP_USER}'@'localhost';
FLUSH PRIVILEGES;
SQL
log_info "Backup MySQL user ready."
# ---------------------------------------------------------
# [5/8] Store credentials securely
# ---------------------------------------------------------
echo "[5/8] Storing credentials in restricted option file..."
mkdir -p /etc/mysql/backup
cat > /etc/mysql/backup/.mycnf <<EOF
[client]
user=${BACKUP_USER}
password=${BACKUP_PASSWORD}
EOF
chown root:root /etc/mysql/backup/.mycnf
chmod 600 /etc/mysql/backup/.mycnf
log_info "Credentials file secured at /etc/mysql/backup/.mycnf"
# ---------------------------------------------------------
# [6/8] Create backup directory structure
# ---------------------------------------------------------
echo "[6/8] Creating backup directory structure..."
mkdir -p /var/backups/mariadb/{staging,archives,logs}
chown -R mysql:mysql /var/backups/mariadb
chmod 750 /var/backups/mariadb
chmod 750 /var/backups/mariadb/staging /var/backups/mariadb/archives /var/backups/mariadb/logs
log_info "Directory structure created under /var/backups/mariadb"
# ---------------------------------------------------------
# [7/8] Write the backup script with compression and rotation
# ---------------------------------------------------------
echo "[7/8] Writing backup script..."
BACKUP_SCRIPT="/usr/local/bin/mariadb-galera-backup.sh"
cat > "$BACKUP_SCRIPT" <<'SCRIPT_EOF'
#!/usr/bin/env bash
set -euo pipefail
CRED_FILE="/etc/mysql/backup/.mycnf"
STAGING="/var/backups/mariadb/staging"
ARCHIVES="/var/backups/mariadb/archives"
LOGS="/var/backups/mariadb/logs"
RETENTION_DAYS="__RETENTION_DAYS__"
TIMESTAMP="$(date +%Y%m%d_%H%M%S)"
SNAPSHOT_DIR="${STAGING}/${TIMESTAMP}"
ARCHIVE_FILE="${ARCHIVES}/galera_backup_${TIMESTAMP}.tar.zst"
LOG_FILE="${LOGS}/backup_${TIMESTAMP}.log"
cleanup() {
rm -rf "${SNAPSHOT_DIR}" 2>/dev/null || true
}
trap cleanup EXIT
{
echo "Starting Mariabackup snapshot at ${TIMESTAMP}"
mkdir -p "${SNAPSHOT_DIR}"
# Take a physical hot backup using Mariabackup, defaults-file keeps
# credentials out of process listing and shell history.
mariabackup --defaults-file="${CRED_FILE}" \
--backup --target-dir="${SNAPSHOT_DIR}" \
2>&1
# Prepare the backup so it is consistent and restorable.
mariabackup --defaults-file="${CRED_FILE}" \
--prepare --target-dir="${SNAPSHOT_DIR}" \
2>&1
echo "Compressing snapshot with zstd..."
tar -cf - -C "${STAGING}" "${TIMESTAMP}" | zstd -T0 -19 -o "${ARCHIVE_FILE}"
# Basic integrity check on the produced archive.
if ! zstd -t "${ARCHIVE_FILE}"; then
echo "ERROR: archive integrity check failed for ${ARCHIVE_FILE}"
exit 1
fi
echo "Backup complete: ${ARCHIVE_FILE}"
# Rotate old archives beyond the retention window.
find "${ARCHIVES}" -name 'galera_backup_*.tar.zst' -mtime "+${RETENTION_DAYS}" -delete
echo "Rotation complete. Retention: ${RETENTION_DAYS} days"
} >> "${LOG_FILE}" 2>&1
SCRIPT_EOF
# Inject retention value into the script
sed -i "s/__RETENTION_DAYS__/${RETENTION_DAYS}/" "$BACKUP_SCRIPT"
chown root:root "$BACKUP_SCRIPT"
chmod 750 "$BACKUP_SCRIPT"
log_info "Backup script written to $BACKUP_SCRIPT"
# Run backup script as mysql user via cron to match dir ownership
CRON_FILE="/etc/cron.d/mariadb-galera-backup"
echo "${CRON_SCHEDULE} mysql ${BACKUP_SCRIPT}" > "$CRON_FILE"
chown root:root "$CRON_FILE"
chmod 644 "$CRON_FILE"
log_info "Cron job installed: ${CRON_SCHEDULE}"
# ---------------------------------------------------------
# [8/8] Verification
# ---------------------------------------------------------
echo "[8/8] Verifying installation..."
FAIL=0
if ! command -v mariabackup >/dev/null 2>&1; then
log_error "mariabackup binary not found."
FAIL=1
fi
if [ ! -f /etc/mysql/backup/.mycnf ]; then
log_error "Credentials file missing."
FAIL=1
fi
if [ "$(stat -c '%a' /etc/mysql/backup/.mycnf)" != "600" ]; then
log_error "Credentials file permissions incorrect."
FAIL=1
fi
if [ ! -d /var/backups/mariadb/archives ]; then
log_error "Archives directory missing."
FAIL=1
fi
if [ ! -x "$BACKUP_SCRIPT" ]; then
log_error "Backup script missing or not executable."
FAIL=1
fi
if ! mysql --defaults-file=/etc/mysql/backup/.mycnf -e "SELECT 1;" >/dev/null 2>&
Review the script before running. Execute with: bash install.sh