<!-- Copyright 2024 Wolfgang Hoschek AT mac DOT com Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.--> <h1 align="left"> <img src="bzfs_docs/logo.svg" alt="bzfs logo" width="640" /> </h1>

Why bzfs? bzfs is a CLI built for highly reliable and scalable near real-time ZFS snapshot replication with minimal operational complexity. It reliably replicates ZFS datasets in parallel using zfs send/receive and ssh, and can operate at sub-second intervals across large fleets of hosts. It safely supports disaster recovery and high availability (DR/HA), scale-out deployments, and protection against data loss or ransomware. It handles the many edge cases that you will eventually run into over the course of your deployment.

• Introduction
• Periodic Jobs with bzfs_jobrunner
• Quickstart
• Installation
• Design Aspects
• Continuous Integration Testing
• End-to-End Testing on the Testbed
• Testing Locally
• Testing all Supported Platforms via GitHub Hosted Runners
• Man Page

Introduction

<!-- DO NOT EDIT (This section was auto-generated from ArgumentParser help text as the source of "truth", via update_readme.sh) --> <!-- BEGIN DESCRIPTION SECTION -->

On the first run, bzfs replicates the source dataset and all its snapshots to the destination. On subsequent runs, it sends only changes since the previous run by incrementally replicating snapshots created on the source after that run. Source snapshots older than the most recent common snapshot on the destination are skipped automatically.

Unless bzfs is told to create snapshots on the source, it treats the source as read-only. With --dryrun, it also treats the destination as read-only. In normal operation, the destination is append-only. Optional flags can delete destination snapshots and datasets if you want to manage storage space consumption, reconcile divergence, restore production from backup, or resync backup from production.

bzfs supports include/exclude filters that can be combined to choose which datasets, snapshots, and properties to create, replicate, delete, or compare.

A common setup uses scheduled cron jobs: one to create and prune source snapshots, one to prune destination snapshots, and one to replicate recently created snapshots from source to destination. Schedules can run every N milliseconds, seconds, minutes, hours, days, weeks, months, or years.

Snapshot creation, replication, pruning, monitoring, and comparison work with snapshots in any naming format, including snapshots created by third-party tools or by manual zfs snapshot commands. These functions can also be used independently.

The source can push to the destination, and the destination can pull from the source. bzfs runs on the initiator host, which can be the source host (push mode), destination host (pull mode), same host (local mode, no network, no ssh), or a third-party host that can SSH into source and destination (pull-push mode). In pull-push mode, the source zfs send stream is forwarded by the initiator directly to the destination zfs receive, without storing anything locally. In this mode, bzfs does not need to be installed on source or destination; only the zfs CLI is required there. bzfs can run as root or as a non-root user via sudo or delegated zfs allow permissions.

bzfs is written in Python and continuously tested with unit and integration tests on old and new ZFS versions, on multiple Linux and FreeBSD versions, and on all Python versions >= 3.9 (including latest stable, currently python-3.14).

bzfs is a stand-alone program with zero required dependencies. It is meant to run in restricted barebones server environments. No external Python packages are required; indeed no Python package management at all is required. You can symlink the program wherever you like, such as /usr/local/bin, and run it like a shell script or binary executable.

bzfs replicates snapshots for multiple datasets in parallel. It also deletes (or monitors or compares) snapshots of multiple datasets in parallel. Atomic snapshots can be created as often as every N milliseconds.

Replication progress (e.g. throughput and ETA) is shown aggregated across parallel replication tasks. Example console status line:

2025-01-17 01:23:04 [I] zfs sent 41.7 GiB 0:00:46 [963 MiB/s] [907 MiB/s] 80% ETA 0:00:04 ETA 01:23:08

bzfs uses streaming algorithms to process millions of datasets with low memory usage and low latency. It handles replication policies with multiple sources and multiple destinations per source.

Optionally, bzfs applies bandwidth rate limiting and progress monitoring (via pv) during zfs send/receive transfers. Over the network, it can insert lightweight compression (via zstd) and buffering (via mbuffer) between endpoints. If one of these tools is not installed, bzfs auto-detects that and continues without that auxiliary feature.

Periodic Jobs with bzfs_jobrunner

The project also ships with bzfs_jobrunner, a companion program that wraps bzfs for periodic snapshot creation, replication, pruning, and monitoring across N source hosts and M destination hosts, using one shared fleet-wide jobconfig script.

Typical use cases include geo-replicated backup where each destination host is in a different region and receives replicas from the same set of source hosts, low-latency replication from a primary to a secondary or to M read replicas, and backups to removable drives.

Quickstart

• Create adhoc atomic snapshots without a schedule:

$ bzfs tank1/foo/bar dummy --recursive --skip-replication --create-src-snapshots \
--create-src-snapshots-plan "{'test':{'':{'adhoc':1}}}"

$ zfs list -t snapshot tank1/foo/bar
tank1/foo/bar@test_2024-11-06_08:30:05_adhoc

• Create periodic atomic snapshots on a schedule, every minute, every hour and every day, by launching this from a periodic cron job:

$ bzfs tank1/foo/bar dummy --recursive --skip-replication --create-src-snapshots \
--create-src-snapshots-plan \
"{'prod':{'us-west':{'minutely':40,'hourly':36,'daily':31}}}"

$ zfs list -t snapshot tank1/foo/bar
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_minutely

Note: A periodic snapshot is created if it is due per the schedule indicated by its suffix (e.g. _daily or _hourly or _minutely or _2secondly or _100millisecondly), or if the --create-src-snapshots-even-if-not-due flag is specified, or if the most recent scheduled snapshot is somehow missing. In the latter case bzfs immediately creates a snapshot (named with the current time, not backdated to the missed time), and then resumes the original schedule. If the suffix is _adhoc or not a known period then a snapshot is considered non-periodic and is thus created immediately regardless of the creation time of any existing snapshot.

• Replication example in local mode (no network, no ssh), to replicate ZFS dataset tank1/foo/bar to tank2/boo/bar:

$ bzfs tank1/foo/bar tank2/boo/bar

$ zfs list -t snapshot tank1/foo/bar
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank1/foo/bar@prod_us-west_2024-11-06_08:30:05_minutely

$ zfs list -t snapshot tank2/boo/bar
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_daily
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_hourly
tank2/boo/bar@prod_us-west_2024-11-06_08:30:05_minutely

• Same example in pull mode:

$ bzfs root@host1.example.com:tank1/foo/bar tank2/boo/bar

• Same example in push mode:

$ bzfs tank1/foo/bar root@host2.example.com:tank2/boo/bar

• Same example in pull-push mode:

$ bzfs root@host1:tank1/foo/bar root@host2:tank2/boo/bar

• Example in local mode (no network, no ssh) to recursively replicate ZFS dataset tank1/foo/bar and its descendant datasets to tank2/boo/bar:

$ bzfs --recursive tank1/foo/bar tank2/boo/bar

$ zfs list -t snapshot -r tank1/foo/bar
tank1/fo