Earthstar

v11 using Willow is now available in beta!

Development is happening on the willow branch.

Earthstar is a small and resilient distributed storage protocol designed with a strong focus on simplicity and versatility, with the social realities of peer-to-peer computing kept in mind.

This is a reference implementation written in Typescript. You can use it to add Earthstar functionality to applications running on servers, browsers, the command line, or anywhere else JavaScript can be run.

Detailed API documentation for this module can be found here.

This document is concerned with the usage of this module's APIs. To learn more about what Earthstar is, please see these links:

• What is Earthstar?
• How does Earthstar work?

To learn more about running Earthstar servers, see README_SERVERS

To learn more about this codebase, please see ARCHITECTURE.

To learn about contributing to this codebase, please see CONTRIBUTING.

Table of contents

Importing the module

It can be imported via URL into a browser:

<script type="module">
  import * as Earthstar from "https://cdn.earthstar-project.org/js/earthstar.web.v10.0.1.js";
</script>

Or Deno:

import * as Earthstar from "https://deno.land/x/earthstar/mod.ts";

Earthstar's web syncing does not work with version of Deno between 1.28.0 - 1.29.3 (inclusive) due to a regression in these versions' WebSocket implementation. Use Deno 1.30.0. or later.

or installed with NPM:

{`npm install earthstar`}

import * as Earthstar from "earthstar";

// Node and browser APIs are namespaced in the NPM distribution:
import { ReplicaDriverWeb } from "earthstar/browser";
import { ReplicaDriverSqlite } from "earthstar/node";

We recommend the browser and Deno versions. This module has been built with many standard web APIs that have need to be polyfilled to work in Node.

Instantiating a replica

Replica is the central API of this module. It is used to write and read data to a locally persisted copy of a share's data, and much more besides.

To instantiate a replica, you will need knowledge of a share's public address.

import { Replica, ReplicaDriverMemory } from "earthstar";

const replica = new Replica({
  driver: ReplicaDriverMemory(YOUR_SHARE_ADDRESS),
  shareSecret: YOUR_SHARE_SECRET,
});

The shareSecret property is optional. If we omit it, the replica will be read-only.

Generating share keypairs

You can create new shares whenever you want.

import { Crypto } from "earthstar";

const shareKeypair = await Crypto.generateShareKeypair("gardening");

The result of this operation will either be a ShareKeypair object with shareAddress and secret properties, or a ValidationError.

Persisting data with drivers

Replica must always be instantiated with a driver. These drivers tell the Replica how to store and retrieve data, with different drivers using different storage mechanisms.

Here are the available drivers:

• ReplicaDriverMemory (works in all environments, but only stores data in memory)
• ReplicaDriverWeb (works in the browser, stores data with IndexedDB)
• ReplicaDriverFs (works on runtimes with filesystem access, stores data with Sqlite and the filesystem)

Drivers are made of two sub-drivers: one for documents, and one for attachments (arbitrary binary data).

There are some extra document drivers not used in the default drivers:

• DocDriverLocalStorage (works in runtimes supporting the WebStorage API)
• DocDriverSqliteFFI (works in Deno, stores data with a FFI implementation of Sqlite, requires using the --unstable flag, and is faster than the default driver in ReplicaDriverFs)

These document drivers can be used like this:

const driver: IReplicaDriver = {
  docDriver: new DocDriverSqliteFfi(SHARE_ADDR, FS_PATH),
  attachmentDriver: new AttachmentDriverFs(FS_ATTACHMENTS_PATH),
};

const replica = new Replica({ driver });

Writing data

Writing data requires two things:

• A replica configured with a valid share secret
• An author keypair

Author keypairs can be generated like this:

import { Crypto } from "earthstar";

const authorKeypair = await Crypto.generateAuthorKeypair("suzy");

The result will be a new AuthorKeypair object with address and secret properties, or a ValidationError.

With a valid author keypair you can write data using Replica.set:

const setResult = await replica.set(authorKeypair, {
  path: "/my-note",
  text: "Saw seven magpies today",
});

The result of this operation is either an IngestEvent describing the operation's success (or failure, if one of the parameters was invalid in some way).

Wiping data

Once written, data can be removed by overwriting it:

await replica.set(authorKeypair, {
  path: "/my-note",
  text: "",
});

Or with the convenience method:

await replica.wipeDocAtPath(authorKeypair, "/my-note");

Creating ephemeral documents

There is another way to remove written data without leaving any trace of it.

Ephemeral documents are held by replicas until a specified time, until at which point they are deleted.

await replica.set(authorKeypair, {
  path: "/my-temporary-note!",
  text: "I accidentally stepped on the strawberries.",
  deleteAfter: TIME_IN_MICROSECONDS,
});

To set an ephemeral document, the path must contain a !, and the deleteAfter property must be set with a timestamp in microseconds.

Querying data

There are many ways to get data back out of a Replica. The simplest one is Replica.getAllDocs:

const everything = await replica.getAllDocs();

The most powerful is Replica.queryDocs:

const mostRecentlyEditedWikiPageDocs = await replica.queryDocs({
  historyMode: "latest",
  filter: {
    pathStartsWith: "/wiki",
  },
  limit: 10,
});

Here are all the querying methods on Replica:

• getAllDocs
• getLatestDocs
• getAllDocsAtPath
• getLatestDocAtPath
• queryDocs
• queryPaths
• queryAuthors

Detailed API documentation for all of them can be found here.

Using document contents

The documents returned by queries are plain objects with the following shape:

type Doc = {
  /** Which document format the doc adheres to, e.g. `es.5`. */
  format: "es.5";
  author: AuthorAddress;
  text: string;
  textHash: string;
  /** When the document should be deleted, as a UNIX timestamp in microseconds. */
  deleteAfter?: number;
  path: Path;
  /** Used to verify the authorship of the document. */
  signature: Signature;
  /** Used to verify the author knows the share's secret */
  shareSignature: Signature;
  /** When the document was written, as a UNIX timestamp in microseconds (millionths of a second, e.g. `Date.now() * 1000`).*/
  timestamp: Timestamp;
  /** The share this document is from. */
  share: ShareAddress;
  /** The size of the associated attachment in bytes, if any. */
  attachmentSize?: number;
  /** The sha256 hash of the associated attachment, if any. */
  attachmentHash?: string;
};

Though most applications will probably only use the author, text, and timestamp properties.

Syncing with other peers

Syncing data with other peers requires adding your replica(s) to an instance of Peer:

import { Peer } from "earthstar";

const peer = new Peer();

// Pretend myReplica is an instance of `Replica`
peer.addReplica(myReplica);

peer.sync("https://my.server");

Peer.sync can be passed another instance of Peer or a valid URL of an Earthstar server to sync with.

The two peers will only sync the replicas with shares they have in common.

The result of Peer.sync can be assigned and used to monitor the progress of the sync operation:

const syncer = peer.sync("https://my.server");

syncer.onStatusChange((newStatus) => {
  console.log(newStatus);
});

syncer.isDone().then(() => {
  console.log("Sync complete");
}).catch((err) => {
  console.error("Sync failed", err);
});

Using document attachments

Documents can be written along with some arbitrary data which is persisted as an 'attachment'. Whereas a document's text field can hold a UTF-8 string of 8kb, attachments can be of any kind of data and of any size.

// Here we use Deno.readFile to get a file's contents as a Uint8Array
const imageData = await Deno.readFile("/Desktop/leaf.jpg");

await replica.set(authorKeypair, {
  path: "/images/pear-leaf.jpg",
  text: "A close-up of a leaf of a pear tree",
  attachment: imageData,
});

The path must have a file extension e.g. .jpg, .mp3 if it also has an attachment.

If we were attaching a large amount of data, we would use a ReadableStream instead:

// Here we use Deno.readFile to get a file's contents as a ReadableStream<Uint8Array>
const videoFile = await Deno.open("/Desktop/little-mole.mp4");

await replica.set(authorKeypair, {
  path: "/videos/little-mole.mp4",
  text: "A close-up of a leaf of a pear tree",
  attachment: videoFile.readable,
});

Retrieving attachments

If you already have a document with an attachment, you can use Replica.getAttachment:

const attachment = await replica.getAttachment(docWithAttachment);

The result of this operation will be a DocAttachment with getBytes and getStream methods, undefined (if our replica has not received a copy of this attachment from other peers), or a ValidationError in case getAttachment was passed a document which can't have an attachmen