Verify jobs in Proxmox Backup Server: what they check, what they don't prove

1. What verify reads

For each snapshot to verify, PBS takes the list of its files and checks each of them (src/backup/verify.rs):

• Small files (blobs: VM configuration, client log): size and checksum.
• Indexes (.fidx for a disk, .didx for a file archive): size and checksum of the index itself.
• Every chunk referenced by the index: the chunk is read from disk. If it is unencrypted, PBS decompresses it, recomputes its SHA-256 digest and compares it with its name, then checks its size. If it is encrypted, the server does not have the key: it only checks the CRC-32 checksum of the stored chunk.

Within a single task, a chunk shared by several snapshots is read only once. The result (ok or failed, with the date and task ID) is stored with the snapshot; it is what the Verify State column of the Content tab shows, and that column also flags a verification older than 30 days as outdated.

When a chunk is bad

A chunk that is unreadable, missing or whose digest does not match counts as an error. If it exists on disk, it is renamed to <digest>.0.bad: it is no longer considered present. The snapshot becomes failed. What happens next is worth knowing (src/api2/backup/mod.rs):

• if the latest snapshot of a machine (VM, container or host) is failed, the next backup does not build on it: the client sends every chunk again, and those missing on the server, including the discarded one, are written back;
• older snapshots that referenced that chunk stay failed until they are verified again. With Skip Verified, that only happens after the Re-Verify After delay: run it by hand once the next backup has completed, with the V. icon on the snapshot's row in the Content tab (on a single snapshot it always re-verifies; on a group it skips verifications less than 29 days old).

2. The cost: reads, lots of them

Verifying a snapshot means reading back all of its chunks, not only those it added. Deduplication saves space, not reads: a chunk verified yesterday, in another task, is read again today if it belongs to the new snapshot.

Example: a VM whose disk holds 400 GB of data, backed up every night. That night's verify job reads back the roughly 400 GB of chunks of the new snapshot, even if the backup only sent 3 GB. Verify reads far more than the backup writes; it is often the longest task on a PBS.

• Parallelism: the read-threads (default 1) and verify-threads (default 4) options, from 1 to 32, set the number of readers and verifiers (docs). On spinning disks, more readers help until the disks are saturated, not beyond.
• ZFS special device: the documentation recommends it with spinning disks (System Requirements). It mainly speeds up metadata work: walking the .chunks/ directory, garbage collection marking. Verify reads chunk contents, which stay on the data disks; it benefits much less.

3. Skip Verified and Re-Verify After

Two options decide which snapshots a job examines (ignore-verified and outdated-after on the command line):

The trap is in the last row. The interface suggests 30 days by default, but on the command line outdated-after has no default: a job created without that option never re-verifies. The computation uses whole days, and a snapshot is re-verified when its last verification is strictly more than 30 days old. Note also that the result barely matters: a failed snapshot counts as already verified and waits for the delay too.

proxmox-backup-manager verify-job create verify-<datastore> --store <datastore> \
  --schedule daily --ignore-verified true --outdated-after 30

# one-off verification of the whole datastore, skipping nothing
proxmox-backup-manager verify <datastore> --ignore-verified false

The documentation suggests two jobs: a frequent one for new backups, and a weekly or monthly one that re-verifies everything. A daily job with 30 days of Re-Verify After achieves the same result with a single job, and spreads the reads out instead of piling them into one night.

Verifying as soon as the backup ends

The datastore option verify-new starts a verification of each snapshot as soon as the backup finishes (« all new backups will be verified right after completion »). It shortens the time to detection, at the cost of a full read-back right after every write.

4. Verify and sync

A copy from one PBS to another (sync job) does not carry the verification state. The code says so plainly: it is dropped « to be reverified independent from the sync » (src/server/pull.rs). A synced snapshot therefore arrives unverified, and it is the verify job of the target PBS that will check it, on its own disks.

• verified-only: only copies snapshots whose last verification at the source is ok.
• resync-corrupt: if a local snapshot failed verification, the next sync copies it again from the source.

The two complement each other: you avoid copying a snapshot that is already damaged at the source, and you repair the local copy from a healthy source. Both sides need their own verify job.

5. What a successful verify does not prove

• That the VM boots. The chunks can be intact and the guest unusable: backup of an already corrupted disk, missing driver, configuration absent on the new machine.
• That application data is consistent. A database backed up live without a filesystem freeze (guest agent) is faithfully stored, and may still need recovery at startup.
• That you have the key. For an encrypted backup, the server only checks the CRC-32 of the encrypted chunk; without the key, nothing can be restored. See the PBS encryption key.
• Restore time. It depends on network and disk throughput; you measure it by restoring.

Only a tested restore answers these questions. At NimbusBackup, restore tests are part of our three managed-service plans, for customers whose Proxmox VE we administer, with the key we already hold as administrator. For backup hosting only, the key is not requested: the restore test stays on your side.

6. What about garbage collection?

Garbage collection is the other scheduled task of a datastore. It verifies nothing: it frees the space of chunks no snapshot references any more, with a grace period of a little over 24 hours. Like verify, it walks every index, and the two share the disks. The details, and why the space does not come back straight away, are in the garbage collection section of the deduplication article.

7. Our default settings

On every NimbusBackup datastore, verify and garbage collection run with the defaults suggested by the PBS interface: daily verification, Skip Verified, re-verification after 30 days, daily garbage collection. These are the PBS defaults, and we keep them. The table is on our managed PBS page.

Frequently asked questions