rtdb

Overview

rtdb is a real-time JSON document database. Data is made available via map/reduce queries.
Queries are updated in real-time and subscribers are notified instantly as data is added.

The open source version of rtdb takes big data principles and extends into the practical domain. The map/reduce construct is applied for real-time data to provide a unique, live analytical view of data.

Use rtdb and leave polling behind!

Installing

Clone the git repository (use --recursive!)

git clone --recursive https://github.com/chicagozer/rtdb.git
Then run npm install

cd rtdb
npm install
(New!) rtdb is avaliable through docker.

docker pull chicagozer/rtdb

Usage

Launch rtdb with node.js. Application settings are supplied via json.

node rtdb.js --settings settings/settings.json

PaaS sites

Online demo versions of rtdb are available.

https://rtdb.herokuapp.com

https://rtdb-rheosoft.rhcloud.com

https://rtdb.mybluemix.net

Acknowledgements

rtdb was inspired by and indebted to several projects.

Especially couchdb and node.js.

Thanks to npm and several excellent libraries. See package.json for more details.

Some samples contain a freely licensed template from MediaLoot.

Web interface

When running locally, the main web interface is reachable at

http://localhost:9001/web/

The web interface provides the ability to manage collections, views and subscriptions.

Architecture

Documents

rtdb is a document-oriented database. A document may be any JSON object.

Collections

Documents are organized into "Collections". A collection is a specific type of document. There is no requirement that all documents be the same JSON format but it does help somewhat that all documents in a given collection have similar structures when designing queries.

Views

Each collection contains "Views". A View represents a particular query into the Collection. All queries must be "pre-registered". There is no "ad-hoc" query facility. However, as the data changes, registered queries are run continuously and subscribers are updated in real-time.

Subscriber

A "Subscriber" represents a party (usually a browser) interested in receiving updates when the query results change. A query may have many subscribers. Subscribers are registered via HTML EventSources.

The hierarchy is straightforward.

Collections → Views → Subscribers
↳ Documents

Map/Reduce

The query within each view is implemented via map/reduce. Each query requires a map function and a reduce function.
Optionally, a finalize function and a personalize function can be added. For incoming documents, the pipeline follows this pattern.

Map → Reduce → Finalize → Personalize

Map

The map function is called once for each incoming document. The purpose is to initially categorize the data and it takes the following arguments.

item - item is the document to process.
emit - emit is a function that is called to map the item.
database - a reference to the database for accessing other collections/views.

emit takes two arguments.

* key - the hash to store
* value - the value to store

emit may be called one or more times for each call to map.

Reduce

Reduce is called once for each key emitted by map. It takes the following arguments.

values - an array of values to reduce
rereduce - a boolean indicating if we are rereducing.
emit - a function that is called to emit the result.
database - a reference to the database for accessing other collections/views.

emit takes one argument, the result of the reduction. emit should be called no more than once per reduce.

rtdb uses the re-reduce approach to accomplish incremental map/reduce. The reduce function you supply must honor this flag.

"[A] Reduce function has the requirement that not only must it be referentially transparent, but it must also be commutative and associative for the array value input, to be able reduce on its own output and get the same answer."

Damien Katz

Therefore, again citing Damien, for a reduce function,

 f(Key, Values) == f(Key, [ f(Key, Values) ] )

More discussion here.

Finalize

The finalize function may be used to sort and/or cull the result set before sending to subscribers. A common usage would be to sort and trim to a "top 10" list.

finalize takes the following arguments.

reduction - an array of reduced values to finalize.
emit - a function that is called to emit the result. Pass the finalized reduction.
database - a reference to the database for accessing other collections/views.

Personalize.

The personalize function is similar to finalize however is it called individually for each subscriber. The intent is to allow the function to use HTTP header values to identify the subscriber and personalize the result set specifically for the subscriber. A common usage pattern would be to filter the data based on the user.

personalize takes the following arguments.

reduction - an array of reduced values to finalize.
headers - the HTTP headers of the current subscriber. See node.js documentation.
emit - a function that is called to emit the result. Pass the personalized reduction.
database - a reference to the database for accessing other collections/views.

Since personalize is called for each subscriber, consider the scalability of this feature as your user base grows.

Security

rtdb has limited inherent security. This is by design.

rtdb is intended to be run behind a secure web server such as Apache. Apache and other web servers provide secure facilities applying granular, URL based security to rtdb. See advanced topic, securing rtdb.

"Out of the box", rtdb implements Basic Authentication security for the administration functions. The user/password are set via environmental variables RTDBADMIN_USER and RTDBADMIN_PWD. In a production environment where security has been delegated, the Basic Authentication may be disabled by changing the settings json parameter disableBasicAuth to true.

If rtdb is used over the internet, be sure to secure the traffic with https if you intend to rely on basic authentication.

REST API

rtdb uses a simple REST API to manage and interact with the database. (The web interface even uses this REST API behind the scenes. Feel free to check out the HTML!)

The REST API speaks JSON and uses standard verbs.

GET - return a JSON object.
PUT - update passing a JSON object.
DELETE - delete object according to url.
POST - insert JSON object or execute command according to URL.

Each collection, view and subscriber is given a Globally Unique Identifier (GUID). Using a GUID guarantees uniqueness of the individual objects and allows separate databases to be combined if needed. The GUID is used to reference specific objects via the REST API.

The web interface is an easy way to familiarize yourself with the GUIDs.

Operations on Collections

POST /db/collections - Add a new Collection.
PUT /db/collections/[col_guid] - Update Collection.
DELETE /db/collections/[col_guid] - Delete Collection.
GET /db/collections - List all collections.
GET /db/collections/[col_guid] - List specific collection.
POST /db/collections[col_guid]/documents - Add a Document or Array of Documents.
DELETE /db/collections/[col_guid]/documents - Delete all Documents.

Operations on Views

POST /db/collections[col_guid]/views - Add a new View.
PUT /db/collections/[col_guid]/views/[view_guid] - Update View.
DELETE /db/collections/[col_guid]/views/[view_guid] - Delete View.
GET /db/collections/[col_guid]/views - List all Views.
GET /db/collections/[col_guid]/views/[view_guid] - List View.
GET /db/collections/[col_guid]/views/[view_guid]/ticket - fetch security ticket.
GET /db/collections/[col_guid]/views/[view_guid]/subscribers - List subscribers.
GET /db/collections/[col_guid]/views/[view_guid]/reduction - List query result.

Miscellaneous Operations

GET /db/stream?view=[guid] - Used by EventSource. Multiple view params are supported.
GET /db/admin/stats - Show database stats in JSON.
POST /db/admin/stop - Shutdown the database.

Using rtdb

Steps for using rtdb are:

Use the web interface or REST API to create a collection.
Create one or more views for the collection.
Subscribers register via WebSocket or Server Sent Event API.
Insert new JSON documents via the REST API.
Subscribers receive updates as documents are inserted.

Use the REST API to insert documents. An example CURL syntax to load a file mydoc.json would be

curl -X POST -H 'Content-Type: application/json' -d @mydoc.json http://localhost:9001/db/collections/[col_guid]/documents

Note that for inserts, the rtdb REST API expects either a single JSON document or array of documents. To maximize performance, the map/reduce is run once f

Rtdb

Install / Use

README