Offline copy of a PBS datastore with zfs send

Two ways to get a PBS datastore out

PBS offers its own route: a removable datastore fed by a sync job. ZFS replication is another. They do not copy the same thing.

The sync job is no second-best: it does not require ZFS, and transfer-last divides the volume when the offline copy does not need the whole history. We chose ZFS because our datastores already live on it, and the copy is then the datastore itself: nothing to rebuild on the day it has to be read.

The principle: the copy pulls

The machine holding the disk fetches the data, over SSH. Production never pushes. This is not an implementation detail, it is what holds everything else together:

• production holds no credentials for the copy, and has no route to it;
• if production is compromised, an attacker who gets root there can see that a copy comes to collect data — an authorised key, connections in the logs — but has neither credentials nor a route to open a connection to it;
• it is exactly the model of media you unplug — it comes to help itself, then disappears.

On the firewall, a single opening: from the copy to production, over SSH. Snapshots, and their deletion, remain decided by production itself.

1. On production: an SSH key for the copy

The copy connects to production with an SSH key authorised on production. That is all the setup needs, and production, for its part, has no key to the copy. Restrict that key as far as your operations allow (restrict, from=, forced command): without restrictions, whoever takes over the copy machine takes over production too.

How we manage our own access is part of our operations: it is not detailed here.

Who takes the snapshots? Production, with its own timer: it keeps control of its own snapshots. The same timer places a bookmark on each snapshot: it costs nothing, and it is what later sends will use as their origin.

S=offline-$(date -u +%Y%m%dT%H%M)
zfs snapshot rpool/datastores/clientX@$S
zfs bookmark rpool/datastores/clientX@$S rpool/datastores/clientX#$S

2. On the copy: preparing the disk

ls -l /dev/disk/by-id/ | grep -v part           # the disk, by its stable identifier

zpool create -o ashift=12 -o compatibility=openzfs-2.2 -O atime=on airgap-A1 /dev/disk/by-id/<disk>
zfs create -o mountpoint=/mnt/datastore airgap-A1/copies

• /dev/disk/by-id, never /dev/sdX, which changes from one plug-in to the next;
• compatibility=openzfs-2.2 freezes the pool's features at those of OpenZFS 2.2 (or the level of your choice, see /usr/share/zfs/compatibility.d/). A pool created on a newer OpenZFS can enable features an older OpenZFS does not know, and zpool import then refuses the disk: freezing them in advance guarantees it will also import on a less up-to-date server;
• ashift=12 set explicitly: a USB bridge or a LUN sometimes reports 512-byte sectors on a disk that works in 4 KiB;
• atime=on: the PBS garbage collector relies on chunk atime, and PBS checks it when the datastore is declared;
• the mountpoint on the parent: received datasets land directly under /mnt/datastore/<name>.

A disk that leaves the room can get lost: encrypt the media, and never store the key with the disk. Client-side encryption does not make this unnecessary — it protects the content of the backups, not the media, and it remains optional in PBS.

3. The first copy

set -o pipefail
SRC=rpool/datastores/clientX
PROD="-i /root/.ssh/id_ed25519_copy pbs-production"

ssh $PROD zfs list -t snapshot,bookmark -o name,creation -r $SRC    # what is available
ssh $PROD zfs send -nP -L -c $SRC@offline-20260914T0700                # estimated size, nothing is transferred

ssh $PROD zfs send -L -c $SRC@offline-20260914T0700 \
  | pv -L 20m \
  | zfs receive -s -u -x mountpoint airgap-A1/copies/clientX
echo "PIPESTATUS=${PIPESTATUS[*]}"                                  # three 0s, otherwise nothing is settled

• send -L -c sends blocks as they are compressed on disk, without decompressing them or re-splitting those larger than 128 KiB;
• receive -s makes the receive resumable: if the link drops, you pick up where you left off instead of starting over;
• -u: the received dataset is not mounted;
• -x mountpoint: the received dataset takes the mountpoint of its local parent, even if the stream carries properties (a send with -p or -R);
• pv -L caps the throughput, so the link is not eaten up during backups;
• PIPESTATUS: three commands, three exit codes. Looking only at the last one is not enough.

TOKEN=$(zfs get -H -o value receive_resume_token airgap-A1/copies/clientX)
ssh $PROD zfs send -t $TOKEN | pv -L 20m | zfs receive -s -u airgap-A1/copies/clientX

Do not judge by the first few minutes: the send starts by walking the metadata of millions of chunks, with a cold cache, and starts slowly before settling into its pace.

4. Checking the copy before it goes to the safe

Never declare the received dataset itself in PBS for a routine check: a datastore in service writes (locks, atime), the dataset diverges, and the next incremental is rejected. Read a throwaway clone instead — which means PBS has to be installed on the machine that carries the disks; ZFS alone is not enough:

zfs clone -o readonly=off -o mountpoint=/mnt/controle \
  airgap-A1/copies/clientX@offline-20260914T0700 airgap-A1/controle

proxmox-backup-manager datastore create controle /mnt/controle --reuse-datastore true
proxmox-backup-manager verify controle --ignore-verified false

proxmox-backup-manager datastore remove controle   # removes the declaration, not the data
zfs destroy airgap-A1/controle
zpool status airgap-A1                             # 0 errors

5. When a disk comes back: the incremental

Each disk comes back with a different lag, so it restarts from its own last send. The last snapshot on the disk has the same name as a bookmark on production: that is the origin of the incremental.

zpool import airgap-A1
zfs list -t snapshot -o name airgap-A1/copies/clientX       # last snapshot received by THIS disk

ssh $PROD zfs send -nP -L -c -i $SRC#offline-20260829T0700 $SRC@offline-20260914T0700   # size of the delta
ssh $PROD zfs send -L -c -i $SRC#offline-20260829T0700 $SRC@offline-20260914T0700 \
  | pv -L 20m | zfs receive -s -u airgap-A1/copies/clientX
echo "PIPESTATUS=${PIPESTATUS[*]}"

The options that matter for an incremental

• if zfs receive refuses because the destination has been modified, someone wrote to the received dataset — most often by mounting it;
• if the bookmark has disappeared on production, there is no common point left: start over with a full copy;
• -x mountpoint only matters on the first send, which creates the dataset;
• on the disk, only the last received snapshot is needed for the next incremental. Earlier ones hold blocks that PBS has freed since.

6. Unplugging

zpool export airgap-A1        # always before unplugging, never after

If the disk is on USB, four checks, once per enclosure model:

• UAS or BOT (lsusb -t, look for Driver=uas). In usb-storage mode, one command at a time: throughput hits a ceiling. If the kernel itself disabled UAS for that bridge, it has its reasons — change enclosures rather than forcing it;
• SMART through the bridge: smartctl -a -d sat. If no -d works, this enclosure does not let you monitor the disk;
• the serial number seen in /dev/disk/by-id is sometimes the bridge's, identical for every disk. Rely on the pool name, which ZFS reads from the disk itself;
• a powered enclosure for a 3.5-inch drive. A disconnection mid-write suspends the pool: zpool clear, then resume with the token.

What this copy gives you, and what it does not

• Unplugged, the disk is out of reach: ransomware can neither encrypt nor erase media connected to nothing;
• a compromised production does not lead to the copy: the copy pulls, and production has neither credentials nor a route to it;
• a lost or stolen disk gives nothing away without its encryption key, which never travels with it;
• with client-side encryption enabled, the content stays unreadable to anyone without your key — including us;
• what has been backed up since the last send is not on the disk: the offline copy is as old as its last update;
• a deletion on production gets copied: if the backups are wiped there, the next incremental carries the deletion onto the refreshed disk. That is by design — the copy is the datastore itself. What protects you then are the other media, still in the safe with the state from before;
• a copy never read back is not a backup: the forced verify, then an actual restore from time to time, are the only proof.

Frequently asked questions