Lite³: A JSON-Compatible Zero-Copy Serialization Format

Parse no more—the wire format is the memory format.

<h2><a href="https://lite3.io/">Official Documentation (with examples): lite3.io</a></h2> <h4><a href="https://lite3.io/design_and_limitations.html">Read more about the design of Lite³</a></h4>

Introduction

Lite³ is a zero-copy binary serialization format encoding data as a B-tree inside a single contiguous buffer, allowing access and mutation on any arbitrary field in O(log n) time. Essentially, it functions as a serialized dictionary.

As a result, the serialization boundary has been broken: 'parsing' or 'serializing' in the traditional sense is no longer necessary. Lite³ structures can be read and mutated directly similar to hashmaps or binary trees, and since they exist in a single contiguous buffer, they always remain ready to send.

Compared to other binary formats, Lite³ is schemaless, self-describing (no IDL or schema definitions required) and supports conversion to/from JSON, enabling compatibility with existing datasets/APIs and allowing for easy debugging/inspecting of messages.

Thanks to the cache-friendly properties of the B-tree and the very minimalistic C implementation (9.3 kB), Lite³ outperforms the fastest JSON libraries (that make use of SIMD) by up to 120x depending on the benchmark. It also outperforms schema-only formats, such as Google Flatbuffers (242x). Lite³ is possibly the fastest schemaless data format in the world.

Example to illustrate:

1. A Lite³ message is received from a socket
2. Without doing any parsing, the user can immediately:
   • Lookup keys and read values via zero-copy pointers
   • Insert/overwrite arbitrary key/value entries
3. After all operations are done, the structure can be transmitted 'as-is' (no serialization required, just memcpy())
4. The receiver then has access to all the same operations

Typically, in such a scenario a distinct 'serializing' and 'deserializing' step would be required. However Lite³ blurs the line between memory and wire formats, allowing direct access, traversal and mutation of a serialized buffer.

Features

• Schemaless & self-describing, no IDL or schema definitions required
• Zero-copy reads + writes of any data size
• Lives on OSI layer 6 (transport/protocol agnostic)
• O(log n) amortized time complexity for all IOPS
• Built-in pointer validation
• Low memory profile
• Predictable latency
• No malloc() API, caller provides buffer
• Library size 9.3 kB (core) and dependency-free
• Written in C11 using GNU C syntax
• Optional subdependency (yyjson) to support conversion to/from JSON
• MIT license

Code Example

(error handling omitted for brevity)

#include <stdio.h>
#include <stdbool.h>

#include "lite3.h"


uint8_t buf[1024];


int main() {
    size_t buflen = 0;
    size_t bufsz = sizeof(buf);

    lite3_init_obj(buf, &buflen, bufsz);
    lite3_set_str(buf, &buflen, 0, bufsz, "app_name", "demo_app");
    lite3_set_i64(buf, &buflen, 0, bufsz, "max_retries", 3);
    lite3_set_bool(buf, &buflen, 0, bufsz, "debug_mode", false);

    int64_t max_retries;
    lite3_get_i64(buf, buflen, 0, "max_retries", &max_retries);
    printf("max retries: %li\n", max_retries);

    return 0;
}

Output:

max retries: 3

Lite³ provides an alternative API called the 'Context API' where memory management is abstracted away from the user.

This example is taken from examples/context_api/04-nesting.c. Again, with error handling omitted for brevity:

#include <stdio.h>
#include <string.h>

#include "lite3_context_api.h"


int main() {
        lite3_ctx *ctx = lite3_ctx_create();
        
        // Build message
        lite3_ctx_init_obj(ctx);
        lite3_ctx_set_str(ctx, 0, "event", "http_request");
        lite3_ctx_set_str(ctx, 0, "method", "POST");
        lite3_ctx_set_i64(ctx, 0, "duration_ms", 47);

        // Set headers
        size_t headers_ofs;
        lite3_ctx_set_obj(ctx, 0, "headers", &headers_ofs);
        lite3_ctx_set_str(ctx, headers_ofs, "content-type", "application/json");
        lite3_ctx_set_str(ctx, headers_ofs, "x-request-id", "req_9f8e2a");
        lite3_ctx_set_str(ctx, headers_ofs, "user-agent", "curl/8.1.2");

        lite3_ctx_json_print(ctx, 0); // Print Lite³ as JSON

        // Get user-agent
        lite3_str user_agent;
        size_t ofs;
        lite3_ctx_get_obj(ctx, 0, "headers", &ofs);
        lite3_ctx_get_str(ctx, ofs, "user-agent", &user_agent);
        printf("User agent: %s\n", LITE3_STR(ctx->buf, user_agent));

        lite3_ctx_destroy(ctx);
        return 0;
}

Output:

{
    "method": "POST",
    "event": "http_request",
    "duration_ms": 47,
    "headers": {
        "user-agent": "curl/8.1.2",
        "x-request-id": "req_9f8e2a",
        "content-type": "application/json"
    }
}
User agent: curl/8.1.2

For a complete How-to Guide with examples, see the documentation.

Getting Started

Make Commands

| Command | Description | |-------------------|-----------------------------------------------------------| | make all | Build the static library with -O2 optimizations (default) | | make tests | Build and run all tests (use VERBOSE=1 for stdout output) | | make examples | Build all examples | | make install | Install library in /usr/local (for pkg-config) | | make uninstall | Uninstall library | | make clean | Remove all build artifacts | | make help | Show this help message |

Installation

A gcc or clang compiler is required due to the use of various builtins.

First clone the repository:

git clone https://github.com/fastserial/lite3.git
cd lite3/

Then choose between installation via pkg-config or manual linking.

Installation via pkg-config (easiest)

Inside the project root, run:

sudo make install -j
sudo ldconfig

This will build the static library, then install it to /usr/local and refresh the pkg-config cache. If installation was successful, you should be able to check the library version like so:

pkg-config --modversion lite3

You can now compile using these flags:

$(pkg-config --libs --cflags --static lite3)

For example, to compile a single file main.c:

gcc -o main main.c $(pkg-config --libs --cflags --static lite3)

Installation via manual linking

First build the library inside project root:

make -j

Then in your main program:

1. Link against build/liblite3.a
2. And include: include/lite3.h + include/lite3_context_api.h

For example, to compile a single file main.c:

gcc -o main main.c -I/path/to/lite3/include /path/to/lite3/build/liblite3.a

Using the library

Choose your API

The Buffer API provides the most control, utilizing caller-supplied buffers to support environments with custom allocation patterns, avoiding the use of malloc().

The Context API is a wrapper around the Buffer API where memory allocations are hidden from the user, presenting a more accessible interface. If you are using Lite³ for the first time, it is recommended to start with the Context API.

#include "lite3.h"              // Buffer API
#include "lite3_context_api.h"  // Context API

There is no need to include both headers, only the API you intend to use.

Library error messages

By default, library error messages are disabled. However it is recommended to enable them to receive feedback during development. To do this, either:

1. uncomment the line // #define LITE3_ERROR_MESSAGES inside the header file: include/lite3.h
2. build the library using compilation flag -DLITE3_ERROR_MESSAGES

If you installed using pkg-config, you may need to reinstall the library to apply the changes. To do this, run:

sudo make uninstall
sudo make clean
sudo make install
sudo ldconfig

Building Examples

Examples can be found in separate directories for each API:

• examples/buffer_api/*
• examples/context_api/*

To build the examples, inside the project root run:

make examples -j

To run an example:

./build/examples/context_api/01-building-messages

For learning how to use Lite³, it is recommended to follow the How-to Guide series.

Feature Matrix

| Format name | Schemaless | Zero-copy reads[^1] | Zero-copy writes[^2] | Human-readable[^3] | | ----------------------------- | ------------- | --------------------- | --------------------- | ------------------------ | | Lite³ | ✅ | ✅ O(log n) | ✅ O(log n) | ⚠️ (convertable to JSON) | | JSON | ✅ | ❌ | ❌ | ✅ | | BSON | ✅ | ❌ | ❌ | ⚠️ (convertable to JSON) | | MessagePack | ✅ | ❌ | ❌ | ⚠️ (convertable to JSON) | | CBOR | ✅ | ❌ | ❌ | ⚠️ (convertable to JSON) | | Smile | ✅ | ❌ | ❌ | ⚠️ (convertable to JSON) | | Ion (Amazon) | ✅ | ❌ | ❌ | ⚠️ (convertable to JSON) | | Protobuf (Google) | ❌ | ❌ | ❌