Storex is a minimal storage layer as a foundation for easing common problems around storing and moving data around. Allowing you to describe your data layout as a graph and providing different plugins, it helps you interact with (No)SQL databases, data migration, offline first applications architecture, creating and consuming REST/GraphQL APIs, permission management, finding optimization opportunaties and more. The aim is to provide a minimalistic common ground/language for working with your data, providing packages for solving the most common problems around data, while giving you easy access to the underlying machinery to do the things that are specific to your application. Everything together that means that every problem you encounter while rapidly iterating towards a serious product, from choosing a suitable DB to suddenly realizing you need to migrate your data model, or even switch DBs, will get a ton easier because you don't have to solve them for the 109th time yourself.

This project started as the storage layer for Memex, a tool to organize your web-research for yourself and collaboratively. The design of this storage layer was done by Vincent den Boer and you can read about the original thoughts behind it here.

Status: This has been powering Memex in production for over 3 years, facilitating our experiments with peer-to-peer sync and our eventual transition to the cloud. It allows us to share a lot of infrastructure and business logic across the browser, our React Native app and our Firestore backend. However, we don't have the resources to properly launch this as a quality open source project with awesome docs and fixes for some of the design issues we discovered over the years. The recommended way to use it for now is to fork it and include it as submodules in your own project. It's easy to understand, so it's encouraged that you take the time to understand what's happening so you can modify it to fit your exact needs. With this in mind, the instructions below are outdated because we don't actively publish new versions to NPM. That being said, the API is stable and will be until Storex 1.0 in which we can finally fix a bunch of small things!

Installation

Storex is a collection of Node.js modules (written in TypeScript) meant to be used both client- and server-side. To start, you need the core and a backend:

$ npm install @worldbrain/storex --save

$ # For a client-side DB
$ npm install @worldbrain/storex-backend-dexie --save # IndexedDB through Dexie library

$ # For a server-side SQL DB
$ npm install @worldbrain/storex-backend-sequelize --save # MySQL, PostgreSQL, SQLite, MSSQL through Sequelize

$ # For a Firestore mBaaS DB
$ npm install @worldbrain/storex-backend-firestore --save

Basic usage

First, configure a StorageBackend and set up the StorageManager, which will be the main point of access to define, query and manipulate your data. For more in-depth information on how to do all of this, please refer to the docs.

import StorageManager from 'storex'
import { DexieStorageBackend } from 'storex-backend-dexie'

const storageBackend = new DexieStorageBackend({dbName: 'my-awesome-product'})
const storageManager = new StorageManager({ backend: storageBackend })
storageManager.registry.registerCollections({
    user: {
        version: new Date(2018, 11, 11),
        fields: {
            identifier: { type: 'string' },
            isActive: { type: 'boolean' },
        },
        indices: [
            { field: 'identifier' },
        ]
    },
    todoList: {
        version: new Date(2018, 7, 11),
        fields: {
            title: { type: 'string' },
        },
        relationships: [
            {childOf: 'user'} # creates one-to-many relationship
        ],
        indices: []
    },
    todoListEntry: {
        version: new Date(2018, 7, 11),
        fields: {
            content: {type: 'text'},
            done: {type: 'boolean'}
        },
        relationships: [
            {childOf: 'todoList', reverseAlias: 'entries'}
        ]
    }
})
await storageManager.finishInitialization()

const user = await storageManager.collection('user').createObject({
    identifier: 'email:boo@example.com',
    isActive: true,
    todoLists: [{
        title: 'Procrastinate this as much as possible',
        entries: [
            {content: 'Write intro article', done: true},
            {content: 'Write docs', done: false},
            {content: 'Publish article', done: false},
        ]
    }]
})
# user now contains things generated by underlying backend, like ids and random keys if you have such fields
console.log(user.id)

await storageManager.collection('todoList').findObjects({user: user.id}) # You can also use MongoDB-like queries

Further documentation

You can find the docs here. Also, we'll be writing more and more automated tests which also serve as documentation.

Storex ecosystem

The power of Storex comes from having modular packages that can be recombined in different contexts based on the core 'language' or patterns Storex provides to talk about data. Officially supported packages will be included in the @worldbrain npm namespace. This helps us to endorse patterns that emerge throughout the ecosystem in a controlled, collectively governed way. These are the currently supported packages:

• Storage modules: An pattern of organizing your storage logic modules so other parts of your application and tooling can have more meta-information about the application. This includes standard ways of defining your data structure and schema changes, describing how you access your data, describing higher-level methods, and describing access rules. This allows for really cool stuff like automatically detecting operations on fields that are not indexed, automatically generating client-server APIs for moving your data around, generating rules for other systems (like Firebase) to manage user rights, and a lot more.
• Schema migrations: The functionality you need, provided in a modular and adaptable way, to describe and execute schema migration as your application evolves. Written with a diverse context in mind, like generating SQL scripts for DBAs, running migration in Serverless environments, converting exported data on import, etc.
• Client-server communication usign GraphQL: This allows you to move your storage layer, expressed in storage modules, server-side transparently by using GraphQL as the transport layer with your back-end. You consume this API with the client so the business logic consuming your storage logic can stay exactly the same, making GraphQL an implementation detail, not something running throughout your code.
• Data tools: A collection of data tools allowing you to apply schema migration on live data (coming in through the network or read from files), generate large sets of fake, complex data for testing purposes, and tools for dumping and loading test fixtures.
• Schema visualization using Graphviz: Still in its infancy, this creates a GraphViz DOT files showing you the relationships between your collections.

Also, current officially supported back-ends are:

• Dexie: Manages interaction with IndexedDB for you, so your application can run fully client-side. Use for your daily development workflow, for fully client-side applications, or offline-first applications.
• Sequelize: Interact with any SQL database supported by Sequelize, such as MySQL or PostgreSQL. Meant to be used server-side.
• Firestore: Store your data in Firestore, so you don't have to build a whole back-end yourself. Can be used directly from the client-side, or inside Cloud Functions. Includes a Security Rule generator that removes a lot of the pain of writing secure and maintainable security rules.

Status and future development

In addtion to the functionality described above, these are some features to highlight that are implemented and used in production:

• One DB abstraction layer for client-side, server-side and mBaaS code: Use the Dexie backend for IndexedDB client-side applications, the Sequelize backend for server-side SQL-based databases, or the Firestore backend for mBaaS-based applications. This allows you to write storage-related business logic portable between front- and back-end, while easily switching to non-SQL storage back-ends later if you so desire, so you can flexible adjust your architecture as your application evolves.
• Defining data in a DB-agnostic way as a graph of collections: By registering your data collections with the StorageManager, you can have an easily introspectable representation of your data model
• Automatic creation of relationships in DB-agnostic way: One-to-one, one-to-many and many-to-many relationships declared in DB-agnostic ways are automatically being taken care of by the underlying StorageBackend on creation.
• MongoDB-style querying: Basic CRUD operations take MongoDB-style queries, which will then be translated by the underlying StorageBackend.
• Client-side full-text search using Dexie backend: By passing a stemmer into the Dexie storage backend, you can full-text search text fields using the fastest cli