Build a two-node NFS high availability cluster using DRBD block replication, Pacemaker and Corosync resource management, and STONITH fencing to eliminate split-brain and survive node failures with automatic failover.
Prerequisites
- Two Linux servers with root or sudo access and a dedicated unused block device on each
- Out-of-band management (IPMI, iLO, iDRAC, or a hypervisor fencing agent) for STONITH
- A dedicated or low-latency network link between nodes for DRBD replication
- A floating virtual IP address available on the same subnet as both nodes
- Basic familiarity with Pacemaker and Corosync concepts
What this solves
A single NFS server is a hard dependency for every client mounting it. If it goes down, every application reading or writing to that share stalls. This tutorial builds a two-node active-passive NFS cluster where DRBD replicates the underlying block device in real time, and Pacemaker with Corosync manages automatic failover of the filesystem, virtual IP and NFS export.
We cover partitioning and DRBD setup, Pacemaker resource configuration, STONITH fencing to prevent split-brain, and failover testing with data consistency checks. This builds on concepts from Build a highly available Linux cluster with Pacemaker and Corosync, applied specifically to shared storage and NFS.
Step-by-step configuration
Prepare hostnames and networking
Both nodes need to resolve each other by hostname, and cluster traffic should ideally use a dedicated interface or VLAN separate from client traffic.
203.0.113.10 node1 node1.example.com
203.0.113.11 node2 node2.example.comApply this file identically on both nodes.
Partition the replication disk
DRBD replicates a raw block device, not a filesystem. Create a dedicated partition on /dev/sdb on both nodes, leaving it unformatted since DRBD manages the raw blocks.
sudo parted /dev/sdb --script mklabel gpt mkpart primary 0% 100%
lsblk /dev/sdbInstall DRBD and Pacemaker packages
Install the DRBD kernel utilities along with Pacemaker, Corosync and the resource agents package that includes the NFS server resource script.
sudo apt update
sudo apt install -y drbd-utils pacemaker corosync pcs fence-agents nfs-kernel-server resource-agents-extrasudo dnf install -y epel-release
sudo dnf install -y drbd90-utils kmod-drbd90 pacemaker corosync pcs fence-agents-all nfs-utils resource-agentsRun this on both nodes. On AlmaLinux and Rocky, the kmod-drbd90 package provides a prebuilt kernel module matching the current kernel.
Configure the DRBD resource
Define a DRBD resource named nfsdata that pairs /dev/sdb1 on each node. This config must be identical on both hosts.
resource nfsdata {
protocol C;
device /dev/drbd0;
disk /dev/sdb1;
meta-disk internal;
net {
verify-alg sha256;
}
on node1 {
address 203.0.113.10:7789;
}
on node2 {
address 203.0.113.11:7789;
}
}Protocol C provides synchronous replication: a write is only acknowledged once it has been confirmed on both nodes, which is required for a true HA setup.
Initialize and bring up the DRBD device
Create the metadata, start the DRBD service, and designate node1 as the initial primary so it can sync the full dataset to node2.
sudo drbdadm create-md nfsdata
sudo systemctl enable --now drbd
sudo drbdadm up nfsdataRun the three commands above on both nodes, then force initial sync from node1 only.
sudo drbdadm primary --force nfsdata
watch cat /proc/drbdWait until the sync percentage reaches 100 before proceeding. This can take a while depending on disk size and network speed.
Format and mount manually to verify
With node1 as primary, create the filesystem on the DRBD device and mount it to confirm it works before handing control to Pacemaker.
sudo mkfs.xfs /dev/drbd0
sudo mkdir -p /srv/nfs/export
sudo mount /dev/drbd0 /srv/nfs/export
sudo mkdir -p /srv/nfs/export/share
sudo umount /srv/nfs/exportUnmount it afterward. Pacemaker will own mounting and unmounting from this point forward, so manual mounts must not persist.
Install and configure Corosync cluster communication
Use pcs to authenticate both nodes and generate the Corosync cluster configuration with a dedicated cluster name.
sudo systemctl enable --now pcsd
sudo passwd haclusterSet the same hacluster password on both nodes, then authenticate from node1.
sudo pcs host auth node1 node2 -u hacluster -p 'ChangeThisStrongPassw0rd!'
sudo pcs cluster setup nfs_cluster node1 node2
sudo pcs cluster start --all
sudo pcs cluster enable --allConfigure STONITH fencing
Without fencing, a network partition can leave both nodes believing they are primary, corrupting the DRBD device. STONITH forcibly powers off or reboots the unresponsive node before any resource is promoted elsewhere.
If your nodes are on a platform supporting IPMI, use fence_ipmilan. For KVM/libvirt test labs, fence_xvm works well. The example below uses fence_ipmilan against out-of-band management interfaces.
sudo pcs stonith create fence_node1 fence_ipmilan \
pcmk_host_list="node1" ipaddr="203.0.113.50" login="admin" \
passwd="StrongIpmiPassw0rd!" lanplus=1 action="reboot"
sudo pcs stonith create fence_node2 fence_ipmilan \
pcmk_host_list="node2" ipaddr="203.0.113.51" login="admin" \
passwd="StrongIpmiPassw0rd!" lanplus=1 action="reboot"sudo pcs property set stonith-enabled=true
sudo pcs property set no-quorum-policy=stopCreate the DRBD Pacemaker resource
Define a master/promotable resource so Pacemaker can manage which node holds the DRBD primary role, and constrain it so the filesystem only mounts where DRBD is primary.
sudo pcs resource create drbd_nfsdata ocf:linbit:drbd \
drbd_resource=nfsdata op monitor interval=30s role=Promoted \
op monitor interval=31s role=Unpromoted
sudo pcs resource promotable drbd_nfsdata \
promoted-max=1 promoted-node-max=1 clone-max=2 clone-node-max=1 notify=trueCreate the filesystem, virtual IP and NFS export resources
Each of these becomes a managed cluster resource, grouped together so they always move as a unit to whichever node is active.
sudo pcs resource create nfs_fs ocf:heartbeat:Filesystem \
device=/dev/drbd0 directory=/srv/nfs/export fstype=xfs
sudo pcs resource create nfs_server ocf:heartbeat:nfsserver \
nfs_shared_infodir=/srv/nfs/export/nfsinfo nfs_no_notify=true
sudo pcs resource create nfs_export ocf:heartbeat:exportfs \
clientspec="203.0.113.0/24" options="rw,sync,no_root_squash" \
directory=/srv/nfs/export/share fsid=0
sudo pcs resource create nfs_vip ocf:heartbeat:IPaddr2 \
ip=203.0.113.20 cidr_netmask=24 op monitor interval=10sGroup the filesystem, NFS server, export and VIP so Pacemaker starts and stops them in order on the same node.
sudo pcs resource group add nfs_group nfs_fs nfs_server nfs_export nfs_vipConstrain the group to follow the DRBD primary
The NFS group must only run on whichever node currently holds the promoted (primary) DRBD role, and must start only after DRBD promotion completes.
sudo pcs constraint colocation add nfs_group with Promoted drbd_nfsdata-clone INFINITY
sudo pcs constraint order promote drbd_nfsdata-clone then start nfs_groupVerify your setup
Check overall cluster health and resource placement with pcs status, which gives a condensed summary of nodes, resources and recent failures.
sudo pcs statusFor a live, continuously updating view, use crm_mon, which is useful while actively testing failover.
sudo crm_mon -1 --include=allConfirm the NFS export is reachable from a third client machine before relying on it.
showmount -e 203.0.113.20
sudo mount -t nfs 203.0.113.20:/srv/nfs/export/share /mnt
df -h /mntWrite a test file, then simulate failover by putting the active node in standby and confirming the resource group migrates.
echo "failover test $(date)" | sudo tee /srv/nfs/export/share/test.txt
sudo pcs node standby node1
sudo pcs statusFrom the client, the mount should briefly pause, then resume once the VIP moves to node2. Confirm the test file is intact on node2 after the move, which validates DRBD replication is consistent.
cat /mnt/test.txtBring node1 back into the cluster once testing is complete.
sudo pcs node unstandby node1Common issues
| Symptom | Cause | Fix |
|---|---|---|
| DRBD shows StandAlone or Unknown connection state | Nodes cannot reach each other on the replication port, or a split-brain was detected | Check connectivity on TCP 7789 with telnet node2 7789, then resolve split-brain with drbdadm connect --discard-my-data nfsdata on the node to discard |
| pcs status shows resource failed and not restarting | Resource hit its migration-threshold after repeated failures | Clear the failcount with sudo pcs resource cleanup nfs_group after fixing the root cause |
| STONITH action never fires during a real failure | Fencing device credentials or IPMI network path misconfigured | Test manually with sudo pcs stonith fence node2 and verify out-of-band management connectivity |
| NFS clients hang during failover longer than expected | NFS grace period or stale file handles after VIP migration | Confirm nfs_no_notify=true is set and clients mount with hard,intr options, not soft |
| Filesystem resource refuses to mount on the new primary | Previous node did not cleanly unmount, leaving the DRBD device inconsistent | Check cat /proc/drbd for UpToDate status on both nodes before promotion completes |
Next steps
- Build a highly available Linux cluster with Pacemaker and Corosync
- Configure encrypted network storage with LUKS and NFS
- Set up RAID arrays with mdadm for performance and redundancy
- Monitor DRBD replication lag with Prometheus and Grafana
- Configure Pacemaker cluster backup and disaster recovery