AnuDB

AnuDB is a lightweight, serverless document database designed for C++ applications, offering efficient storage of JSON documents through MessagePack serialization. It provides a serverless, schema-less solution for applications requiring flexible data management with robust query capabilities.

Since AnuDB written on top of RocksDB, it ensures atomicity, durability and consistency of your documents. You can adjust memory/CPU usage of AnuDB based on RocksDB options mentioned in StorageEngine.cpp and StorageEngine.h.Based on these configurations, you can get your desired performance results tailored to your specific platform requirements.

Get Started with AnuDB!

Ready to try AnuDB for your project? Follow these simple steps to get started quickly:

1. Install AnuDB
   See the installation guide to set up AnuDB on your machine.

2. Explore Example Code
   Check out our usage examples to understand how to interact with AnuDB.

3. Start Building!
   Dive right in by using AnuDB in your application and start storing your data with lightning-fast speed.

4. Docker Support
   AnuDB supports Docker! Run AnuDB in a containerized environment with ease. See our Docker setup guide to get started.

5. MQTT Support
   AnuDB integrates seamlessly with MQTT, enabling real-time data communication between devices and servers. If you're working on IoT or real-time messaging systems, AnuDB's MQTT support will make your job easier. Check out our MQTT integration guide to get started with MQTT and AnuDB.

6. Benchmark

Features

• Embedded & Serverless: Run directly within your application with no separate server process required
• Embedded Platform Support: Specifically designed for both standard Linux/Windows systems and embedded platforms
• Faster Writes – RocksDB (LSM-tree) backend is optimized for high-speed inserts/updates.
• JSON Document Storage: Store and query complex JSON documents
• RocksDB Backend: Built on RocksDB for high performance and durability
• Transactional Properties: Inherits atomic operations and configurable durability from RocksDB
• Compression Options: Support for ZSTD compression to reduce storage footprint
• Flexible Querying: Support for equality, comparison, logical operators and sorting
• Indexing: Create indexes to speed up queries on frequently accessed fields
• Update Operations: Rich set of document modification operations
• Import/Export: Easy JSON import and export for data migration. Exported JSON file can be also used to migrate data to Postgresql/SQL sever/MySQL/Oracle DB
• C++11 Compatible: Designed to support a wide range of embedded devices efficiently
• Windows/Linux Support: Designed for Windows/Linux environments and embedded Linux platforms
• WAL Tracking: Real-time monitoring of database operations through customizable callback functions
• MQTT Interface: Connect and operate via MQTT protocol from various platforms
• High Concurrency: Supports 32 concurrent nng worker threads(configurable using CONCURRENT_THREADS) for handling MQTT requests
• TLS Security: Secure communications using mbedTLS for encrypted MQTT connections
• Cloud MQTT Support: Compatible with major cloud MQTT brokers
• ACID Compliance: AnuDB inherits atomicity, consistency, isolation, and durability from RocksDB

MQTT Interface

AnuDB now supports interaction via MQTT protocol, allowing you to connect and operate the database from various platforms without direct C++ integration. The implementation uses nlohmann::json for JSON handling. AnuDB subscribe to MQTT anudb/request topic then if request comes with request id on mentioned topic, then it does that operation then send reponse back on anudb/response/requestid, we can get all responses on anudb/response/+ topic, below diagram to illustrate it.

In below demo, showing AnuDBMqttBridge and mosquitto broker server started, then using client.bash script, ran all supported MQTT commands

MQTT supported commands

Docker Support

To run AnuDB using Docker:

1. Pull the Docker Image

docker pull kanadaanu/anudb-mqtt:v1

2. Ensure MQTT Broker is Running

You need an MQTT broker (e.g., EMQX, Mosquitto) running before starting AnuDB.

3. Run the Docker Container

Use one of the following commands based on your setup:

# With TLS (e.g., EMQX public broker, mosquitto broker, etc)
docker run -v <disk folder>:/data kanadaanu/anudb-mqtt:v1 \
  --broker_url tls+mqtt-tcp://<broker_url>:8883 \
  --database_name AnuDB \
  --tls_cacert <> \
  --tls_cert <>
   ...

# Without TLS
docker run -v <disk folder>:/data kanadaanu/anudb-mqtt:v1 \
  --broker_url mqtt-tcp://<broker_url>:1883 \
  --database_name AnuDB

Notes

• The /data folder is where you should mount a disk volume to persist the database.
• If you're using a TLS connection, place the required certificate files (e.g., CA cert) inside the mounted /data directory so that the binary can access them.

Prerequisites

• C++11 >= compatible compiler
• CMake 3.10 or higher
• ZSTD development libraries (optional, for compression support)
• Mosquitto MQTT broker (for MQTT interface, supported version v3.1.1 but 5.0.0 version can be supported with simple API modification)
• mbedTLS libraries (for TLS support)

Building from Source

There is no need to install any other third party libraries. Below commands will generate AnuDB.exe bin which executes all operations supported by AnuDB, also it will generate liblibanu.a static library, you can use it in your application with required header files.

# Clone the repository
git clone https://github.com/hash-anu/AnuDB.git
cd AnuDB/third_party/nanomq/
git submodule update --init --recursive
cd ../..
mkdir build
cd build
cmake ..
make

or
# Configure with ZSTD compression support
cmake -DZSTD_INCLUDE_DIR=/path/to/zstd/include -DZSTD_LIB_DIR=/path/to/zstd/lib ..
make

Embedded Platform Build

For embedded platforms, you may need to use a cross-compiler toolchain:

# Configure with cross-compiler
cmake .. \
  -DCMAKE_SYSTEM_NAME=Linux \
  -DCMAKE_SYSTEM_PROCESSOR=arm \
  -DCMAKE_C_COMPILER=<path-to-arm-gcc>/arm-linux-gnueabihf-gcc \
  -DCMAKE_CXX_COMPILER=<path-to-arm-g++>/arm-linux-gnueabihf-g++ \
  -DCMAKE_C_FLAGS="-march=armv7-a -mfpu=neon -mfloat-abi=hard -DNIOSTATS_CONTEXT=1 -std=c99" \
  -DCMAKE_CXX_FLAGS="-march=armv7-a -mfpu=neon -mfloat-abi=hard -DNIOSTATS_CONTEXT=1 -DNPERF_CONTEXT -Wno-error --std=c++11" \
  -DWITH_ALL_TESTS=OFF \
  -DWITH_TESTS=OFF \
  -DWITH_TOOLS=OFF \
  -DWITH_GFLAGS=OFF \
  -DPORTABLE=ON \
  -DBUILD_TESTS=OFF

Examples

Examples of AnuDB database have been added to the examples folder. API Overview

Quick Start

#include "Database.h"
#include <iostream>

int main() {
    anudb::Database db("./my_database");
    anudb::Status status = db.open();

    if (!status.ok()) {
        std::cerr << "Failed to open database: " << status.message() << std::endl;
        return 1;
    }

    // Create a collection
    status = db.createCollection("users");

    // Get the collection
    anudb::Collection* users = db.getCollection("users");

    // Create a document
    nlohmann::json userData = {
        {"name", "Hash"},
        {"email", "hash@example.com"},
        {"age", 33}
    };
    anudb::Document doc("user001", userData);

    // Insert the document
    status = users->createDocument(doc);

    anudb::Document doc1;
    status = users->readDocument("user001", doc1);
    std::cout << doc1.data().dump(4) << std::endl;
    // Close the database
    db.close();

    return 0;
}

Sample for normal write -> read operation

Sample for get list of collections

Sample for Export then Import operation

High-Performance MQTT Worker Architecture

AnuDB implements a high-performance concurrent worker architecture to handle MQTT requests efficiently:

• 32 Concurrent Worker Threads: The MQTT bridge spawns 32 nng (nanomsg-next-generation) worker threads to process incoming requests in parallel
• Load Balancing: Requests are automatically distributed across all worker threads for optimal performance
• Thread Safety: All database operations are thread-safe, enabling true concurrent processing
• Asynchronous Processing: Each worker operates independently, preventing slow operations from blocking the entire system
• Scalable Design: The worker architecture scales efficiently on multi-core systems

Worker Thread Architecture Diagram

                           ┌─────────────────┐
                           │  MQTT Broker    │
                           │  (Mosquitto)    │
                           └────────┬────────┘
                                    │
                                    ▼
                           ┌─────────────────┐
                           │  MQTT Bridge    │
                           │   Connector     │
                           └────────┬────────┘