Build a highly available Linux NFS cluster with Pacemaker and DRBD

Advanced 95 min Nov 17, 2026 19 views
Ubuntu 24.04 Debian 12 AlmaLinux 9 Rocky Linux 9

Build a two-node NFS high availability cluster using DRBD block replication, Pacemaker and Corosync resource management, and STONITH fencing to eliminate split-brain and survive node failures with automatic failover.

Prerequisites

  • Two Linux servers with root or sudo access and a dedicated unused block device on each
  • Out-of-band management (IPMI, iLO, iDRAC, or a hypervisor fencing agent) for STONITH
  • A dedicated or low-latency network link between nodes for DRBD replication
  • A floating virtual IP address available on the same subnet as both nodes
  • Basic familiarity with Pacemaker and Corosync concepts

What this solves

A single NFS server is a hard dependency for every client mounting it. If it goes down, every application reading or writing to that share stalls. This tutorial builds a two-node active-passive NFS cluster where DRBD replicates the underlying block device in real time, and Pacemaker with Corosync manages automatic failover of the filesystem, virtual IP and NFS export.

We cover partitioning and DRBD setup, Pacemaker resource configuration, STONITH fencing to prevent split-brain, and failover testing with data consistency checks. This builds on concepts from Build a highly available Linux cluster with Pacemaker and Corosync, applied specifically to shared storage and NFS.

Note: This guide assumes two nodes, node1 (203.0.113.10) and node2 (203.0.113.11), each with a dedicated unused block device /dev/sdb of equal size, plus a floating virtual IP of 203.0.113.20 for clients to mount.

Step-by-step configuration

Prepare hostnames and networking

Both nodes need to resolve each other by hostname, and cluster traffic should ideally use a dedicated interface or VLAN separate from client traffic.

203.0.113.10 node1 node1.example.com
203.0.113.11 node2 node2.example.com

Apply this file identically on both nodes.

Partition the replication disk

DRBD replicates a raw block device, not a filesystem. Create a dedicated partition on /dev/sdb on both nodes, leaving it unformatted since DRBD manages the raw blocks.

sudo parted /dev/sdb --script mklabel gpt mkpart primary 0% 100%
lsblk /dev/sdb
Warning: Never run mkfs on /dev/sdb1 directly. DRBD sits below the filesystem layer; formatting happens on top of the DRBD device (/dev/drbd0) later, not on the raw partition.

Install DRBD and Pacemaker packages

Install the DRBD kernel utilities along with Pacemaker, Corosync and the resource agents package that includes the NFS server resource script.

sudo apt update
sudo apt install -y drbd-utils pacemaker corosync pcs fence-agents nfs-kernel-server resource-agents-extra
sudo dnf install -y epel-release
sudo dnf install -y drbd90-utils kmod-drbd90 pacemaker corosync pcs fence-agents-all nfs-utils resource-agents

Run this on both nodes. On AlmaLinux and Rocky, the kmod-drbd90 package provides a prebuilt kernel module matching the current kernel.

Configure the DRBD resource

Define a DRBD resource named nfsdata that pairs /dev/sdb1 on each node. This config must be identical on both hosts.

resource nfsdata {
  protocol C;
  device /dev/drbd0;
  disk /dev/sdb1;
  meta-disk internal;

  net {
    verify-alg sha256;
  }

  on node1 {
    address 203.0.113.10:7789;
  }

  on node2 {
    address 203.0.113.11:7789;
  }
}

Protocol C provides synchronous replication: a write is only acknowledged once it has been confirmed on both nodes, which is required for a true HA setup.

Initialize and bring up the DRBD device

Create the metadata, start the DRBD service, and designate node1 as the initial primary so it can sync the full dataset to node2.

sudo drbdadm create-md nfsdata
sudo systemctl enable --now drbd
sudo drbdadm up nfsdata

Run the three commands above on both nodes, then force initial sync from node1 only.

sudo drbdadm primary --force nfsdata
watch cat /proc/drbd

Wait until the sync percentage reaches 100 before proceeding. This can take a while depending on disk size and network speed.

Format and mount manually to verify

With node1 as primary, create the filesystem on the DRBD device and mount it to confirm it works before handing control to Pacemaker.

sudo mkfs.xfs /dev/drbd0
sudo mkdir -p /srv/nfs/export
sudo mount /dev/drbd0 /srv/nfs/export
sudo mkdir -p /srv/nfs/export/share
sudo umount /srv/nfs/export

Unmount it afterward. Pacemaker will own mounting and unmounting from this point forward, so manual mounts must not persist.

Install and configure Corosync cluster communication

Use pcs to authenticate both nodes and generate the Corosync cluster configuration with a dedicated cluster name.

sudo systemctl enable --now pcsd
sudo passwd hacluster

Set the same hacluster password on both nodes, then authenticate from node1.

sudo pcs host auth node1 node2 -u hacluster -p 'ChangeThisStrongPassw0rd!'
sudo pcs cluster setup nfs_cluster node1 node2
sudo pcs cluster start --all
sudo pcs cluster enable --all

Configure STONITH fencing

Without fencing, a network partition can leave both nodes believing they are primary, corrupting the DRBD device. STONITH forcibly powers off or reboots the unresponsive node before any resource is promoted elsewhere.

If your nodes are on a platform supporting IPMI, use fence_ipmilan. For KVM/libvirt test labs, fence_xvm works well. The example below uses fence_ipmilan against out-of-band management interfaces.

sudo pcs stonith create fence_node1 fence_ipmilan \
  pcmk_host_list="node1" ipaddr="203.0.113.50" login="admin" \
  passwd="StrongIpmiPassw0rd!" lanplus=1 action="reboot"

sudo pcs stonith create fence_node2 fence_ipmilan \
  pcmk_host_list="node2" ipaddr="203.0.113.51" login="admin" \
  passwd="StrongIpmiPassw0rd!" lanplus=1 action="reboot"
Warning: Never disable STONITH in production, even temporarily. A cluster without working fencing will eventually split-brain under a network partition, and DRBD split-brain requires manual, often destructive, resolution.
sudo pcs property set stonith-enabled=true
sudo pcs property set no-quorum-policy=stop

Create the DRBD Pacemaker resource

Define a master/promotable resource so Pacemaker can manage which node holds the DRBD primary role, and constrain it so the filesystem only mounts where DRBD is primary.

sudo pcs resource create drbd_nfsdata ocf:linbit:drbd \
  drbd_resource=nfsdata op monitor interval=30s role=Promoted \
  op monitor interval=31s role=Unpromoted

sudo pcs resource promotable drbd_nfsdata \
  promoted-max=1 promoted-node-max=1 clone-max=2 clone-node-max=1 notify=true

Create the filesystem, virtual IP and NFS export resources

Each of these becomes a managed cluster resource, grouped together so they always move as a unit to whichever node is active.

sudo pcs resource create nfs_fs ocf:heartbeat:Filesystem \
  device=/dev/drbd0 directory=/srv/nfs/export fstype=xfs

sudo pcs resource create nfs_server ocf:heartbeat:nfsserver \
  nfs_shared_infodir=/srv/nfs/export/nfsinfo nfs_no_notify=true

sudo pcs resource create nfs_export ocf:heartbeat:exportfs \
  clientspec="203.0.113.0/24" options="rw,sync,no_root_squash" \
  directory=/srv/nfs/export/share fsid=0

sudo pcs resource create nfs_vip ocf:heartbeat:IPaddr2 \
  ip=203.0.113.20 cidr_netmask=24 op monitor interval=10s

Group the filesystem, NFS server, export and VIP so Pacemaker starts and stops them in order on the same node.

sudo pcs resource group add nfs_group nfs_fs nfs_server nfs_export nfs_vip

Constrain the group to follow the DRBD primary

The NFS group must only run on whichever node currently holds the promoted (primary) DRBD role, and must start only after DRBD promotion completes.

sudo pcs constraint colocation add nfs_group with Promoted drbd_nfsdata-clone INFINITY
sudo pcs constraint order promote drbd_nfsdata-clone then start nfs_group

Verify your setup

Check overall cluster health and resource placement with pcs status, which gives a condensed summary of nodes, resources and recent failures.

sudo pcs status

For a live, continuously updating view, use crm_mon, which is useful while actively testing failover.

sudo crm_mon -1 --include=all

Confirm the NFS export is reachable from a third client machine before relying on it.

showmount -e 203.0.113.20
sudo mount -t nfs 203.0.113.20:/srv/nfs/export/share /mnt
df -h /mnt

Write a test file, then simulate failover by putting the active node in standby and confirming the resource group migrates.

echo "failover test $(date)" | sudo tee /srv/nfs/export/share/test.txt
sudo pcs node standby node1
sudo pcs status

From the client, the mount should briefly pause, then resume once the VIP moves to node2. Confirm the test file is intact on node2 after the move, which validates DRBD replication is consistent.

cat /mnt/test.txt

Bring node1 back into the cluster once testing is complete.

sudo pcs node unstandby node1

Common issues

SymptomCauseFix
DRBD shows StandAlone or Unknown connection stateNodes cannot reach each other on the replication port, or a split-brain was detectedCheck connectivity on TCP 7789 with telnet node2 7789, then resolve split-brain with drbdadm connect --discard-my-data nfsdata on the node to discard
pcs status shows resource failed and not restartingResource hit its migration-threshold after repeated failuresClear the failcount with sudo pcs resource cleanup nfs_group after fixing the root cause
STONITH action never fires during a real failureFencing device credentials or IPMI network path misconfiguredTest manually with sudo pcs stonith fence node2 and verify out-of-band management connectivity
NFS clients hang during failover longer than expectedNFS grace period or stale file handles after VIP migrationConfirm nfs_no_notify=true is set and clients mount with hard,intr options, not soft
Filesystem resource refuses to mount on the new primaryPrevious node did not cleanly unmount, leaving the DRBD device inconsistentCheck cat /proc/drbd for UpToDate status on both nodes before promotion completes

Next steps

Running this in production?

Want this handled for you? Running this at scale adds a second layer of work: capacity planning, failover drills, STONITH testing, and being on call when a node actually dies at 3am. See how we run infrastructure like this for European teams.
#nfs #high availability #pacemaker #corosync #drbd #stonith #failover #cluster #linux #storage replication

Don't want to manage this yourself?

We handle infrastructure for businesses that depend on uptime. Fully managed, with one fixed contact who knows your setup.

You get one fixed contact who knows your setup

Rotterdam 20:45 · reachable in a message, no ticket form

Learn more: managed cloud infrastructure