EJDB 2.0

NOTE: Issues tracker is disabled. You are welcome to contribute, pull requests accepted.

maintained

EJDB2 is an embeddable JSON database engine published under MIT license.

The Story of the IT-depression, birds and EJDB 2.0

C11 API
Single file database
Online backups support
Simple but powerful query language
Support of collection joins
Powered by IOWOW - The persistent key/value storage engine
HTTP REST/Websockets endpoints powered by IWNET

Native language bindings
Supported platforms
- macOS
- iOS
- Linux
- Windows
JQL query language
Indexes and performance
Network API
- HTTP API
- Websockets API
C API
License

Status

EJDB 2.0 core engine is well tested and used in various heavily loaded deployments
Tested on Linux, macOS and FreeBSD.

Building from sources

Prerequisites

Linux, macOS or FreeBSD
gcc or clang compiler
pkgconf or pkg-config

Build by Autark

./build.sh

Installation

./build.sh --prefix=$HOME/.local

JQL

EJDB query language (JQL) syntax inspired by ideas behind XPath and Unix shell pipes. It designed for easy querying and updating sets of JSON documents.

JQL grammar

JQL parser created created by peg/leg — recursive-descent parser generators for C Here is the formal parser grammar: https://github.com/Softmotions/ejdb/blob/master/src/jql/jqp.leg

Non formal JQL grammar adapted for brief overview

Notation used below is based on SQL syntax description:

Rule | Description --- | --- ' ' | String in single quotes denotes unquoted string literal as part of query. <code>{ a | b }</code> | Curly brackets enclose two or more required alternative choices, separated by vertical bars. <code>[ ]</code> | Square brackets indicate an optional element or clause. Multiple elements or clauses are separated by vertical bars. <code>|</code> | Vertical bars separate two or more alternative syntax elements. <code>...</code> | Ellipses indicate that the preceding element can be repeated. The repetition is unlimited unless otherwise indicated. <code>( )</code> | Parentheses are grouping symbols. Unquoted word in lower case| Denotes semantic of some query part. For example: placeholder_name - name of any placeholder.

QUERY = FILTERS [ '|' APPLY ] [ '|' PROJECTIONS ] [ '|' OPTS ];

STR = { quoted_string | unquoted_string };

JSONVAL = json_value;

PLACEHOLDER = { ':'placeholder_name | '?' }

FILTERS = FILTER [{ and | or } [ not ] FILTER];

  FILTER = [@collection_name]/NODE[/NODE]...;

  NODE = { '*' | '**' | NODE_EXPRESSION | STR };

  NODE_EXPRESSION = '[' NODE_EXPR_LEFT OP NODE_EXPR_RIGHT ']'
                        [{ and | or } [ not ] NODE_EXPRESSION]...;

  OP =   [ '!' ] { '=' | '>=' | '<=' | '>' | '<' | ~ }
      | [ '!' ] { 'eq' | 'gte' | 'lte' | 'gt' | 'lt' }
      | [ not ] { 'in' | 'ni' | 're' };

  NODE_EXPR_LEFT = { '*' | '**' | STR | NODE_KEY_EXPR };

  NODE_KEY_EXPR = '[' '*' OP NODE_EXPR_RIGHT ']'

  NODE_EXPR_RIGHT =  JSONVAL | STR | PLACEHOLDER

APPLY = { 'apply' | 'upsert' } { PLACEHOLDER | json_object | json_array  } | 'del'

OPTS = { 'skip' n | 'limit' n | 'count' | 'noidx' | 'inverse' | ORDERBY }...

  ORDERBY = { 'asc' | 'desc' } PLACEHOLDER | json_path

PROJECTIONS = PROJECTION [ {'+' | '-'} PROJECTION ]

  PROJECTION = 'all' | json_path

json_value: Any valid JSON value: object, array, string, bool, number.
json_path: Simplified JSON pointer. Eg.: /foo/bar or /foo/"bar with spaces"/
* in context of NODE: Any JSON object key name at particular nesting level.
** in context of NODE: Any JSON object key name at arbitrary nesting level.
* in context of NODE_EXPR_LEFT: Key name at specific level.
** in context of NODE_EXPR_LEFT: Nested array value of array element under specific key.

JQL quick introduction

Lets play with some very basic data and queries. For simplicity we will use ejdb websocket network API which provides us a kind of interactive CLI. The same job can be done using pure C API too (ejdb2.h jql.h).

NOTE: Take a look into JQL test cases for more examples.

{
  "firstName": "John",
  "lastName": "Doe",
  "age": 28,
  "pets": [
    {"name": "Rexy rex", "kind": "dog", "likes": ["bones", "jumping", "toys"]},
    {"name": "Grenny", "kind": "parrot", "likes": ["green color", "night", "toys"]}
  ]
}

Save json as sample.json then upload it the family collection:

# Start HTTP/WS server protected by some access token
./jbs -a 'myaccess01'
8 Mar 16:15:58.601 INFO: HTTP/WS endpoint at localhost:9191

Server can be accessed using HTTP or Websocket endpoint. More info

curl -d '@sample.json' -H'X-Access-Token:myaccess01' -X POST http://localhost:9191/family

We can play around using interactive wscat websocket client.

wscat  -H 'X-Access-Token:myaccess01' -c http://localhost:9191
connected (press CTRL+C to quit)
> k info
< k     {
 "version": "2.0.0",
 "file": "db.jb",
 "size": 8192,
 "collections": [
  {
   "name": "family",
   "dbid": 3,
   "rnum": 1,
   "indexes": []
  }
 ]
}

> k get family 1
< k     1       {
 "firstName": "John",
 "lastName": "Doe",
 "age": 28,
 "pets": [
  {
   "name": "Rexy rex",
   "kind": "dog",
   "likes": [
    "bones",
    "jumping",
    "toys"
   ]
  },
  {
   "name": "Grenny",
   "kind": "parrot",
   "likes": [
    "green color",
    "night",
    "toys"
   ]
  }
 ]
}

Note about the k prefix before every command; It is an arbitrary key chosen by client and designated to identify particular websocket request, this key will be returned with response to request and allows client to identify that response for his particular request. More info

Query command over websocket has the following format:

<key> query <collection> <query>

So we will consider only <query> part in this document.

Get all elements in collection

k query family /*

k query family /**

or specify collection name in query explicitly

k @family/*

We can execute query by HTTP POST request

curl --data-raw '@family/[firstName = John]' -H'X-Access-Token:myaccess01' -X POST http://localhost:9191

1	{"firstName":"John","lastName":"Doe","age":28,"pets":[{"name":"Rexy rex","kind":"dog","likes":["bones","jumping","toys"]},{"name":"Grenny","kind":"parrot","likes":["green color","night","toys"]}]}

Set the maximum number of elements in result set

k @family/* | limit 10

Get documents where specified json path exists

Element at index 1 exists in likes array within a pets sub-object

> k query family /pets/*/likes/1
< k     1       {"firstName":"John"...

Element at index 1 exists in likes array at any likes nesting level

> k query family /**/likes/1
< k     1       {"firstName":"John"...

From this point and below I will omit websocket specific prefix k query family and consider only JQL queries.

Get documents by primary key

In order to get documents by primary key the following options are available:

Use API call ejdb_get()

 const doc = await db.get('users', 112);

Use the special query construction: /=:? or @collection/=:?

Get document from users collection with primary key 112

> k @users/=112

Update tags array for document in jobs collection (TypeScript):

 await db.createQuery('@jobs/ = :? | apply :? | count')
    .setNumber(0, id)
    .setJSON(1, { tags })
    .completionPromise();

Array of primary keys can also be used for matching:

 await db.createQuery('@jobs/ = :?| apply :? | count')
    .setJSON(0, [23, 1, 2])
    .setJSON(1, { tags })
    .completionPromise();

Matching JSON entry values

Below is a set of self explaining queries:

/pets/*/[name = "Rexy rex"]

/pets/*/[name eq "Rexy rex"]

/pets/*/[name = "Rexy rex" or name = Grenny]

Note about quotes around words with spaces.

Get all documents where owner age greater than 20 and have some pet who like bones or toys

/[age > 20] and /pets/*/likes/[** in ["bones", "toys"]]

Here ** denotes some element in likes array.

ni is the inverse operator to in. Get documents where bones somewhere in likes array.

/pets/*/[likes ni "bones"]

We can create more complicated filters

( /[age <= 20] or /[lastName re "Do.*"] )
  and /pets/*/likes/[** in ["bones", "toys"]]

Note about grouping parentheses and regular expression matching using re operator.

~ is a prefix matching operator (Since ejdb v2.0.53). Prefix matching can benefit from using indexes.

Get documents where /lastName starts with "Do".

/[lastName ~ Do]

Arrays and maps can be matched as is

Filter documents with likes ar

Ejdb

Install / Use

README