GitHub Release GitHub License <a href="https://discord.gg/5PxcuUeaaT"> Discord </a>

Documentation | QuickStart | ZeroFS vs AWS EFS

</div>

ZeroFS - The Filesystem That Makes S3 your Primary Storage

File systems AND block devices on S3 storage

ZeroFS makes S3 storage feel like a real filesystem. It provides file-level access via NFS and 9P and block-level access via NBD.

Key Features:

NFS Server - Mount as a network filesystem on any OS
9P Server - High-performance alternative with better POSIX semantics
NBD Server - Access as raw block devices for ZFS, databases, or any filesystem
Always Encrypted - XChaCha20-Poly1305 encryption with LZ4 or Zstd compression
High Performance - Multi-layered caching with microsecond latencies
S3 Compatible - Works with any S3-compatible storage

Testing

ZeroFS passes all tests in the pjdfstest_nfs test suite - 8,662 tests covering POSIX filesystem operations including file operations, permissions, ownership, and more.

We use ZFS as an end-to-end test in our CI. We create ZFS pools on ZeroFS, extract the Linux kernel source tree, and run scrub operations to verify data integrity. All operations complete without errors.

We also compile the Linux kernel on ZeroFS as part of our CI, using parallel compilation (make -j$(nproc)) to stress-test concurrent operations.

Demo

Compiling the linux kernel on top of S3

Compiling the Linux Kernel in 16 seconds with a ZeroFS NBD volume + ZFS

ZFS on S3 via NBD

ZeroFS provides NBD block devices that ZFS can use directly - no intermediate filesystem needed. Here's ZFS running on S3 storage:

Ubuntu Running on ZeroFS

Watch Ubuntu boot from ZeroFS:

Self-Hosting ZeroFS

ZeroFS can self-host! Here's a demo showing Rust's toolchain building ZeroFS while running on ZeroFS:

Architecture


graph TB
    subgraph "Client Layer"
        NFS[NFS Client]
        P9[9P Client]
        NBD[NBD Client]
    end
    
    subgraph "ZeroFS Core"
        NFSD[NFS Server]
        P9D[9P Server]
        NBDD[NBD Server]
        VFS[Virtual Filesystem]
        ENC[Encryption Manager]
        CACHE[Cache Manager]
        
        NFSD --> VFS
        P9D --> VFS
        NBDD --> VFS
        VFS --> ENC
        ENC --> CACHE
    end
    
    subgraph "Storage Backend"
        SLATE[SlateDB]
        LSM[LSM Tree]
        S3[S3 Object Store]
        
        CACHE --> SLATE
        SLATE --> LSM
        LSM --> S3
    end
    
    NFS --> NFSD
    P9 --> P9D
    NBD --> NBDD

Quick Start

Installation

Install script (Recommended)

curl -sSfL https://sh.zerofs.net | sh

Via Cargo

cargo install zerofs

Via Docker

docker pull ghcr.io/barre/zerofs:latest

Getting Started

# Generate a configuration file
zerofs init

# Edit the configuration with your S3 credentials
$EDITOR zerofs.toml

# Run ZeroFS
zerofs run -c zerofs.toml

Configuration

ZeroFS uses a TOML configuration file that supports environment variable substitution. This makes it easy to manage secrets and customize paths.

Creating a Configuration

Generate a default configuration file:

zerofs init  # Creates zerofs.toml

The configuration file has sections for:

Cache - Local cache settings for performance
Storage - S3/Azure/local backend configuration and encryption
Servers - Enable/disable NFS, 9P, and NBD servers
LSM tuning - Write-ahead log, compaction, and flush settings
Cloud credentials - AWS or Azure authentication

Example Configuration

[cache]
dir = "${HOME}/.cache/zerofs"
disk_size_gb = 10.0
memory_size_gb = 1.0  # Optional, defaults to 0.25

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "${ZEROFS_PASSWORD}"

[filesystem]
max_size_gb = 100.0  # Optional: limit filesystem to 100 GB (defaults to 16 EiB)
compression = "lz4"  # Optional: "lz4" (default) or "zstd-{1-22}"

[servers.nfs]
addresses = ["127.0.0.1:2049"]  # Can specify multiple addresses

[servers.ninep]
addresses = ["127.0.0.1:5564"]
unix_socket = "/tmp/zerofs.9p.sock"  # Optional

[servers.nbd]
addresses = ["127.0.0.1:10809"]
unix_socket = "/tmp/zerofs.nbd.sock"  # Optional

[lsm]
wal_enabled = true  # WAL reduces compaction churn from frequent fsyncs (default: true)
                    # Disable for bulk data loading where fsyncs are rare

[aws]
access_key_id = "${AWS_ACCESS_KEY_ID}"
secret_access_key = "${AWS_SECRET_ACCESS_KEY}"
# endpoint = "https://s3.us-east-1.amazonaws.com"  # For S3-compatible services
# default_region = "us-east-1"
# allow_http = "true"  # For non-HTTPS endpoints
# conditional_put = "redis://localhost:6379"  # For S3-compatible stores without conditional put support

# [azure]
# storage_account_name = "${AZURE_STORAGE_ACCOUNT_NAME}"
# storage_account_key = "${AZURE_STORAGE_ACCOUNT_KEY}"

Environment Variable Substitution

The configuration supports ${VAR} syntax for environment variables. This is useful for:

Keeping secrets out of configuration files
Using different settings per environment
Sharing configurations across systems

All referenced environment variables must be set when running ZeroFS.

Storage Backends

ZeroFS supports multiple storage backends through the url field in [storage]:

Amazon S3

[storage]
url = "s3://my-bucket/path"

[aws]
access_key_id = "${AWS_ACCESS_KEY_ID}"
secret_access_key = "${AWS_SECRET_ACCESS_KEY}"
# endpoint = "https://s3.us-east-1.amazonaws.com"  # For S3-compatible services
# default_region = "us-east-1"
# allow_http = "true"  # For non-HTTPS endpoints (e.g., MinIO)
# conditional_put = "redis://localhost:6379"  # For S3-compatible stores without conditional put support

Note: ZeroFS requires conditional write (put-if-not-exists) support for fencing. AWS S3 supports this natively. For S3-compatible object stores that do not support conditional puts, set conditional_put to a Redis URL. ZeroFS will use Redis to coordinate conditional write operations.

Microsoft Azure

[storage]
url = "azure://container/path"

[azure]
storage_account_name = "${AZURE_STORAGE_ACCOUNT_NAME}"
storage_account_key = "${AZURE_STORAGE_ACCOUNT_KEY}"

Google Cloud Storage (GCS)

Option 1: Application Default Credentials (recommended for GCP VMs/GKE)

If running on a GCP VM or GKE pod with an attached service account, no configuration is needed:

[storage]
url = "gs://my-bucket/path"
# No [gcp] section needed - uses VM/pod service account automatically

Option 2: Service Account Key File

[storage]
url = "gs://my-bucket/path"

[gcp]
service_account = "${GCS_SERVICE_ACCOUNT}"  # Path to service account JSON file

Or set the GOOGLE_APPLICATION_CREDENTIALS environment variable:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
zerofs server -c zerofs.toml

Local Filesystem

[storage]
url = "file:///path/to/storage"
# No additional configuration needed

Server Configuration

You can enable or disable individual servers by including or commenting out their sections:

# To disable a server, comment out or remove its entire section
[servers.nfs]
addresses = ["0.0.0.0:2049"]  # Bind to all IPv4 interfaces
# addresses = ["[::]:2049"]  # Bind to all IPv6 interfaces
# addresses = ["127.0.0.1:2049", "[::1]:2049"]  # Dual-stack localhost

[servers.ninep]
addresses = ["127.0.0.1:5564"]
unix_socket = "/tmp/zerofs.9p.sock"  # Optional: adds Unix socket support

[servers.nbd]
addresses = ["127.0.0.1:10809"]
unix_socket = "/tmp/zerofs.nbd.sock"  # Optional: adds Unix socket support

Filesystem Quotas

ZeroFS supports configurable filesystem size limits:

[filesystem]
max_size_gb = 100.0  # Limit filesystem to 100 GB

When the quota is reached, write operations return ENOSPC (No space left on device). Delete and truncate operations continue to work, allowing you to free space. If not specified, the filesystem defaults to 16 EiB (effectively unlimited).

Compression

ZeroFS compresses file data before encryption. Choose between fast or high-ratio compression:

[filesystem]
compression = "lz4"      # Fast compression (default)
# or
compression = "zstd-3"   # Zstd with level 1-22

lz4 (default): Very fast, moderate compression ratio
zstd-{level}: Configurable compression (1=fast, 22=maximum compression)

You can change compression at any time without migration.

Multiple Instances

ZeroFS supports running multiple instances on the same storage backend: one read-write instance and multiple read-only instances.

# Read-write instance (default)
zerofs run -c zerofs.toml

# Read-only instances
ze

ZeroFS

Install / Use

README