Structurae

A collection of data structures for high-performance JavaScript applications that includes:

• Binary Protocol - simple binary protocol based on DataView and defined with JSONSchema
• Bit Structures:
  • BitField & BigBitField - stores and operates on data in Numbers and BigInts treating them as bitfields.
  • BitArray - an array of bits implemented with Uint32Array.
  • Pool - manages availability of objects in object pools.
  • RankedBitArray - extends BitArray with O(1) time rank and O(logN) select methods.
• Graphs:
  • Adjacency Structures - implement adjacency list & matrix data structures.
  • Graph - extends an adjacency list/matrix structure and provides methods for traversal (BFS, DFS), pathfinding (Dijkstra, Bellman-Ford), spanning tree construction (BFS, Prim), etc.
• Grids:
  • BinaryGrid - creates a grid or 2D matrix of bits.
  • Grid - extends built-in indexed collections to handle 2 dimensional data (e.g. nested arrays).
  • SymmetricGrid - a grid to handle symmetric or triangular matrices using half the space required for a normal grid.
• Sorted Structures:
  • BinaryHeap - extends Array to implement the Binary Heap data structure.
  • SortedArray - extends Array to handle sorted data.

Usage

Node.js:

npm i structurae

import {...} from "structurae";

Deno:

import {...} from "https://deno.land/x/structurae/index.ts"

Documentation

• Articles:
  • Structurae: Data Structures for Heigh-Performance JavaScript
  • Structurae 1.0: Graphs, Strings, and WebAssembly
  • Binary Protocol for JavaScript

Overview

Binary Protocol

Binary data in JavaScript is represented by ArrayBuffer and accessed through TypedArrays and DataView. However, both of those interfaces are limited to working with numbers. Structurae offers a set of classes that extend the DataView interface to support using ArrayBuffers for strings, objects, and arrays. These classes ("views") form the basis for a simple binary protocol with the following features:

• smaller and faster than schema-less binary formats (e.g. BSON, MessagePack);
• supports zero-copy operations, e.g. reading and changing object fields without decoding the whole object;
• supports static typing through TypeScript;
• uses JSON Schema for schema definitions;
• does not require compilation unlike most other schema-based formats (e.g. FlatBuffers).

The protocol is operated through the View class that handles creation and caching of necessary structures for a given JSON Schema as well as simplifying serialization of tagged objects.

import { View } from "structurae";

// instantiate a view protocol
const view = new View();

// define interface for out animal objects
interface Animal {
  name: string;
  age: number;
}
// create and return a view class (extension of DataView) that handles our Animal objects
const AnimalView = view.create<Animal>({
  $id: "Pet",
  type: "object",
  properties: {
    name: { type: "string", maxLength: 10 },
    // by default, type `number` is treated as int32, but can be further specified usin `btype`
    age: { type: "number", btype: "uint8" },
  },
});
// encode our animal object
const animal = AnimalView.from({ name: "Gaspode", age: 10 });
animal instanceof DataView; //=> true
animal.byteLength; //=> 14
animal.get("age"); //=> 10
animal.set("age", 20);
animal.toJSON(); //=> { name: "Gaspode", age: 20 }

Objects and Maps

Objects by default are treated as C-like structs, the data is laid out sequentially with fixed sizes, all standard JavaScript values are supported, inluding other objects and arrays of fixed size:

interface Friend {
  name: string;
}
interface Person {
  name: string;
  fullName: Array<string>;
  bestFriend: Friend;
  friends: Array<Friend>;
}
const PersonView = view.create<Person>({
  // each object requires a unique id
  $id: "Person",
  type: "object",
  properties: {
    // the size of a string field is required and defined by maxLength
    name: { type: "string", maxLength: 10 },
    fullName: {
      type: "array",
      // the size of an array is required and defined by maxItems
      maxItems: 2,
      // all items have to be the same type
      items: { type: "string", maxLength: 20 },
    },
    // objects can be referenced with $ref using their $id
    bestFriend: { $ref: "#Friend" },
    friends: {
      type: "array",
      maxItems: 3,
      items: {
        $id: "Friend",
        type: "object",
        properties: {
          name: { type: "string", maxLength: 20 },
        },
      },
    },
  },
});
const person = Person.from({
  name: "Carrot",
  fullName: ["Carrot", "Ironfoundersson"],
  bestFriend: { name: "Sam Vimes" },
  friends: [{ name: "Sam Vimes" }],
});
person.get("name"); //=> Carrot
person.getView("name"); //=> StringView [10]
person.get("fullName"); //=> ["Carrot", "Ironfoundersson"]
person.toJSON();
//=> {
//     name: "Carrot",
//     fullName: ["Carrot", "Ironfoundersson"],
//     bestFriend: { name: "Sam Vimes" },
//     friends: [{ name: "Sam Vimes" }]
//    }

Objects that support optional fields and fields of variable size ("maps") should additionally specify btype map and list non-optional (fixed sized) fields as required:

interface Town {
  name: string;
  railstation: boolean;
  clacks?: number;
}
const TownView = view.create<Town>({
    $id: "Town",
    type: "object",
    btype: "map",
    properties: {
      // notice that maxLength is not required for optional fields in maps
      // however, if set, map with truncate longer strings to fit the maxLength
      name: { type: "string" },
      railstation: { type: "boolean" },
      // optional, nullable field
      clacks: { type: "integer" },
    }
    required: ["railstation"],
  });
const lancre = TownView.from({ name: "Lancre", railstation: false });
lancre.get("name") //=> Lancre
lancre.get("clacks") //=> undefined
lancre.byteLength //=> 19
const stoLat = TownView.from({ name: "Sto Lat", railstation: true, clacks: 1 });
stoLat.get("clacks") //=> 1
stoLat.byteLength //=> 24

The size and layout of each map instance is calculated upon creation and stored within the instance (unlike fixed sized objects, where each instance have the same size and layout). Maps are useful for densely packing objects and arrays whose size my vary greatly. There is a limitation, though, since ArrayBuffers cannot be resized, optional fields that were absent upon creation of a map view cannot be set later, and those set cannot be resized, that is, assigned a value that is greater than their current size.

For performance sake, all variable size views are encoded using single global ArrayBuffer that is 8192 bytes long, if you expect to handle bigger views, supply a bigger DataView when instantiating a view protocol:

import { View } from "structurae";

// instantiate a view protocol
const view = new View(new DataView(new ArrayBuffer(65536)));

There are certain requirements for a JSON Schema used for fixed sized objects:

• Each object should have a unique id defined with $id field. Upon initialization, the view class is stored in View.Views and accessed with the id used as the key. References made with $ref are also resolved against the id.
• For fixed sized objects, sizes of strings and arrays should be defined using maxLength and maxItems properties respectfully.
• $ref can be used to reference objects by their $id. The referenced object should be defined either in the same schema or in a schema initialized previously.
• Type number by default resolves to float64 and type integer to int32, you can use any other type by specifying it in btype property.

Objects and maps support setting default values of required fields. Default values are applied upon creation of a view:

interface House {
  size: number;
}
const House = view.create<House>({
  $id: "House",
  type: "object",
  properties: {
    size: { type: "integer", btype: "uint32", default: 100 },
  },
});
const house = House.from({} as House);
house.get("size"); //=> 100

Default values of an object can be overridden when it is nested inside another object:

interface Neighborhood {
  house: House;
  biggerHouse: House;
}
const Neighborhood = view.create<Neighborhood>({
  $id: "Neighborhood",
  type: "object",
  properties: {
    house: { $ref: "#House" },
    biggerHouse: { $ref: "#House", default: { size: 200 } },
  },
});
const neighborhood = Neighborhood.from({} as Neighborhood);
neighborhood.get("house"); //=> { size: 100 }
neighborhood.get("biggerHouse"); //=> { size: 200 }

Dictionaries

Objects and maps described above assume that all properties of encoded objects are known and defined beforehand, however, if the properties are not known, and we are dealing with an object used as a lookup table (also called map, hash map, or records in TypeScript) with varying amount of properties and known type of values, we can use a dictionary view:

const NumberDict = view.create<Record<number, string | undefined>>({
  $id: "NumberDict",