<!-- SPDX-FileCopyrightText: Copyright (c) Marcus Holland-Moritz SPDX-License-Identifier: MIT -->

DwarFS

The Deduplicating Warp-speed Advanced Read-only File System.

A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows.

Table of contents

<a href="https://repology.org/project/dwarfs/versions"> <img src="https://repology.org/badge/vertical-allrepos/dwarfs.svg" alt="Packaging status" align="right"> </a>

• What is DwarFS (in plain words)?
• Why not just use .zip or .tar.gz?
• Performance comparison overview
• Quick Start
• Overview
• History
• Building and Installing
  • Note to Package Maintainers
  • Prebuilt Binaries
  • Universal Binaries
  • Dependencies
  • Building
  • Installing
  • Static Builds
• Usage
• Using the Libraries
• Windows Support
  • Building on Windows
• macOS Support
• Use Cases
  • Astrophotography
• Dealing with Bit Rot
• Extended Attributes
• Comparison
  • With SquashFS
  • With SquashFS & xz
  • With lrzip
  • With zpaq
  • With zpaqfranz
  • With wimlib
  • With Cromfs
  • With EROFS
  • With fuse-archive
• Performance Monitoring
• Other Obscure Features
• Related projects
• Notable users
• Stargazers over Time

What is DwarFS (in plain words)?

DwarFS is a mountable archive: you pack a directory into one file (an "image"), then open it instantly like a normal folder — no full extraction, no temporary files. It’s read-only when mounted, so you can browse, open, and run files safely in place. This works particularly well for big collections with lots of similar or repeated content — think many versions of a project or dataset, folders of documents and text files, backups/snapshots that mostly overlap, or libraries with many near-duplicates. The image can be mounted in fractions of a second, you can use the contents immediately with no delay, and you extract only if you actually need to.

Why not just use .zip or .tar.gz?

Traditional archives are fine for storage, but they’re usually slow to open and awkward for random access (jumping around inside large files or across many files). DwarFS is built for fast random reads and space savings. It groups similar files and removes duplication, so images are often smaller than a simple tar/zip, while reads stay snappy — even when accessing lots of files simultaneously. In practice, you keep huge directories compressed, mount them in milliseconds, and work as if they were already unpacked.

Performance comparison overview

This comparison uses DwarFS 0.14.1, 7-Zip 25.00 (x64), and tar + pigz for all .tar.gz tests since plain gzip would have been incredibly slow. Both .tar.gz and 7zip archives were mounted using fuse-archive v1.16 with the -olazycache option since the default of caching the entire archive in memory is infeasible for large archives.

1139 complete Perl installations

| Perl (47.49 GiB, 1.9M files) | .tar.gz [^pl1] | .tar.zst [^pl2] | 7zip (-mx=7) | SquashFS [^pl7] | DwarFS (lzma) | DwarFS (zstd) | |---------------------------------------|---------------:|----------------:|---------------:|----------------:|--------------:|--------------:| | Compression time | 4m 59s | 8m 06s | 23m 27s | 5m 37s | 2m 13s | 5m 3s | | Compression CPU time | 1h 47m | 2h 18m | 5h 5m | 2h 29m | 31m 17s | 49m 51s | | Compressed size | 12.17 GiB | 0.387 GiB | 1.219 GiB | 3.245 GiB | 0.310 GiB | 0.352 GiB | | Compression ratio | 3.902 | 122.7 | 38.96 | 14.63 | 153.2 | 134.9 | | Decompression time | 2m 19s | 57.2s | 1m 14s | 39.3s | 48.7s | 46.5s | | Decompression CPU time | 3m 44s | 1m 21s | 2m 28s | 1m 25s | 2m 23s | 1m 59s | | Mount time | 2m 07s | ❌ [^pl3] | 3.638s | 0.011s | 0.420s | 0.009s | | Find all 1.9M files [^pl8] | 5.670s | ❌ [^pl3] | 5.695s | 5.311s | 2.800s | 2.821s | | Checksum 1139 files (2.58 GiB) [^pl4] | ❌ [^pl5] | ❌ [^pl3] | ~5h [^pl6] | 1.541s | 4.330s | 1.134s |

[^pl1]: using pigz -9 [^pl2]: using zstd --long=31 --ultra -22 -T0 [^pl3]: not supported by fuse-archive [^pl4]: $ ls -1 mnt/*/perl*/bin/perl5* | xargs -d $'\n' -n1 -P16 sha256sum [^pl5]: killed after making no progress for 15 minutes [^pl6]: killed when only 78 files were finished after about 20 minutes [^pl7]: using -comp zstd -Xcompression-level 22 -b 1M -tailends; using squashfuse_ll 0.6.0 [^pl8]: $ fd -t f . mnt | wc -l

All artifacts from 205 DwarFS CI builds

| DwarFS CI (465.2 GiB, 3.6M files) | .tar.gz (pigz -9) | 7zip (-mx=7) | DwarFS (zstd) | |---------------------------------------|--------------------:|---------------:|--------------:| | Compression time | 40m 28s | 141m 46s | 68m 6s | | Compression CPU time | 13h 55m | 42h 47m | 31h 12m | | Compressed size | 142.9 GiB | 60.04 GiB | 28.63 GiB | | Compression ratio | 3.255 | 7.748 | 16.25 | | Decompression time | 26m 17s | 8m 15s | 7m 21s | | Decompression CPU time | 35m 16s | 41m 29s | 8m 59s | | Mount time | 22m 18s | 10.5s | 0.024s | | Run 441 binaries (2.72 GiB) [^ci1] | ❌ [^ci2] | ❌ [^ci3] | 0.774s |

[^ci1]: $ ls -1 mnt/*/dwarfs-*-Linux-x86_64-*-minsize*/bin/mkdwarfs | xargs -d $'\n' -n1 -P16 sh -c '$0 -h' [^ci2]: killed after making no progress for 15 minutes [^ci3]: killed after making no progress and consuming more than 32 GiB of RAM

Game audio assets from sonniss.com

| Sonniss (3.072 GiB, 171 files) [^wav1] | .tar.gz (pigz -9) | 7zip (-mx=7) | SquashFS [^wav3] | DwarFS (categorize) | |--------------------------------------------|--------------------:|---------------:|-----------------:|--------------------:| | Compression time | 5.34s | 6m 21s | 19.52s |