Subduction

[!CAUTION] This is an early release preview. It has a very unstable API. No guarantees are given. DO NOT use for production use cases at this time. USE AT YOUR OWN RISK.

Subduction is a peer-to-peer synchronization protocol and implementation for [CRDTs]. It enables efficient synchronization of encrypted, partitioned data between peers without requiring a central server.

Features

• Efficient Sync Protocol: Uses [Sedimentree] for history sharding, diffing, and efficient incremental synchronization
• Encryption-Friendly: Designed to work with encrypted data partitions without requiring decryption during sync
• No Central Server: True peer-to-peer synchronization via pluggable transports (WebSocket, HTTP long-poll, Iroh/QUIC)
• Multi-Platform: Runs on native Rust, WebAssembly (browser & Node.js), and provides a CLI tool
• Automerge Integration: While not the only data that can be synced via Subduction, [Automerge] was the original target.

Documentation

The design/ directory contains protocol design documents including message formats, threat model, and so on.

Architecture

Subduction is organized as a Rust workspace with multiple crates:

graph TD
    subgraph Core
        sedimentree_core
        subduction_crypto
        subduction_core
    end

    subgraph Storage
        sedimentree_fs_storage
    end

    subgraph Transport
        subduction_http_longpoll
        subduction_iroh
        subduction_websocket
    end

    subgraph Integrations
        automerge_sedimentree
        subduction_keyhive
        subduction_keyhive_policy
    end

    subgraph Wasm
        sedimentree_wasm
        subduction_wasm
        automerge_sedimentree_wasm
        automerge_subduction_wasm
    end

    subduction_cli

    sedimentree_core --> subduction_crypto
    subduction_crypto --> subduction_core

    sedimentree_core --> automerge_sedimentree

    subduction_core --> sedimentree_fs_storage
    subduction_core --> subduction_http_longpoll
    subduction_core --> subduction_iroh
    subduction_core --> subduction_websocket
    subduction_core --> subduction_keyhive_policy

    subduction_core --> sedimentree_wasm
    subduction_core --> subduction_wasm
    sedimentree_wasm --> subduction_wasm

    subduction_wasm --> automerge_sedimentree_wasm
    sedimentree_wasm --> automerge_sedimentree_wasm

    automerge_sedimentree_wasm --> automerge_subduction_wasm
    subduction_wasm --> automerge_subduction_wasm

    subduction_http_longpoll --> subduction_cli
    subduction_iroh --> subduction_cli
    subduction_websocket --> subduction_cli
    sedimentree_fs_storage --> subduction_cli

| Crate | Description | |----------------------------|----------------------------------------------------------------------------------------| | sedimentree_core | The core data partitioning scheme that enables efficient metadata-based synchronization | | sedimentree_fs_storage | Filesystem-based persistent storage for Sedimentree | | subduction_crypto | Cryptographic types: signed payloads and verification witnesses | | subduction_core | The main synchronization protocol implementation | | subduction_http_longpoll | HTTP long-poll transport for restricted network environments | | subduction_iroh | Iroh (QUIC) transport for direct peer-to-peer connections | | subduction_websocket | WebSocket transport layer for peer-to-peer connections |

Platform Bindings

| Crate | Description | |------------------------------|------------------------------------------------------------| | sedimentree_wasm | WebAssembly bindings for Sedimentree | | subduction_wasm | WebAssembly bindings for browser and Node.js environments | | automerge_sedimentree | Sedimentree adapter for Automerge documents | | automerge_sedimentree_wasm | Wasm bindings for Automerge + Sedimentree | | automerge_subduction_wasm | Wasm bindings for Automerge + Subduction (full sync stack) |

Integrations

| Crate | Description | |-----------------------------|--------------------------------------------------------| | subduction_keyhive | Keyhive integration types | | subduction_keyhive_policy | Keyhive-based authorization policy for connections |

Tools

| Crate | Description | |------------------|------------------------------------------------------------------| | subduction_cli | Command-line tool for running Subduction servers and purging data |

Sedimentree

[Sedimentree][sedimentree-design] is a novel data structure for organizing encrypted data into hierarchical layers (strata). Each layer contains metadata (hashes) that represent fragments of a larger file or log. This enables:

1. Efficient Diffing: Compare metadata to determine which fragments need synchronization
2. Privacy: Sync without exposing plaintext data
3. Incremental Updates: Only transfer changed fragments

Sedimentree uses a depth-based partitioning scheme where fragments are organized by the number of leading zero bytes in their hashes, creating natural hierarchical layers.

Quick Start

Using Nix ❄️

If you use Nix, the Subduction server is wrapped in a flake:

# Run the server directly
nix run github:inkandswitch/subduction -- server --socket 0.0.0.0:8080

# Install to your profile
nix profile install github:inkandswitch/subduction

The flake also provides NixOS and Home Manager modules for running Subduction as a system service:

{
  inputs.subduction.url = "github:inkandswitch/subduction";

  outputs = { nixpkgs, subduction, ... }: {
    nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
      modules = [
        subduction.nixosModules.default
        {
          services.subduction.server.enable = true;
        }
      ];
    };
  };
}

See the [CLI README] for full configuration options including Home Manager and reverse proxy setup.

Prerequisites

• Rust 1.90 or later
• For Wasm development: [wasm-pack]
• For browser testing: Node.js 22+ and pnpm

Building

Build all workspace crates:

cargo build --release

Build Wasm packages:

cd subduction_wasm
pnpm install
pnpm build

Running Tests

Run all Rust tests:

cargo test

Run Wasm browser tests:

cd subduction_wasm
pnpm install
npx playwright install
npx playwright test

Using the CLI

Start a WebSocket server:

cargo run --release -p subduction_cli -- server --socket 127.0.0.1:8080

Usage Examples

Rust

use subduction_core::Subduction;
use subduction_core::storage::MemoryStorage;

// Create a Subduction instance with in-memory storage
let storage = MemoryStorage::new();
let subduction = Subduction::new(storage);

// Connect to peers, sync data...

WebAssembly (Browser)

import * as subduction from '@automerge/subduction';

// Create a Subduction instance
const storage = new subduction.MemoryStorage();
const syncer = new subduction.Subduction(storage);

// Connect to a WebSocket server
const ws = new WebSocket('ws://localhost:8080');
const peerId = new subduction.PeerId(new Uint8Array(32)); // Your peer ID
const subductionWs = await subduction.SubductionWebSocket.connect(
  new URL('ws://localhost:8080'),
  peerId,
  5000
);

// Register the connection
await syncer.register(subductionWs);

// Start syncing...

Development

Project Structure

subduction/
├── sedimentree_core/           # Core Sedimentree data structure
├── sedimentree_fs_storage/     # Filesystem storage for Sedimentree
├── sedimentree_wasm/           # Wasm bindings for Sedimentree
├── subduction_crypto/          # Signed payloads and verification witnesses
├── subduction_core/            # Sync protocol implementation
├── subduction_http_longpoll/   # HTTP long-poll transport
├── subduction_iroh/            # Iroh (QUIC) transport
├── subduction_websocket/       # WebSocket transport
├── subduction_wasm/            # Wasm bindings for Subduction
├── subduction_cli/             # CLI server and data management
├── subduction_keyhive/         # Keyhive integration types
├── subduction_keyhive_policy/  # Keyhive authorization policy
├── automerge_sedimentree/      # Automerge integration
├── automerge_sedimentree_wasm/ # Wasm wrapper for automerge_sedimentree
└── automerge_subduction_wasm/  # Wasm wrapper for automerge_sedimentree + subduction

Testing

The project uses several testing strategies:

| Strategy | Description | |----------------------|-----------------------------------------------------------------| | Unit Tests | Standard cargo test for Rust code