VennDB

An append-only in-memory database in Rust for rows queried using bit (flag) columns. This database is designed for a very specific use case where you have mostly static data that you typically load at startup and have to query constantly using very simple filters. Datasets like these can be large and should be both fast and compact.

For the limited usecases where venndb can be applied to, it has less dependencies and is faster then traditional choices, such as a naive implementation or a more heavy lifted dependency such as Sqlite.

See the benchmarks for more information on this topic.

This project was developed originally in function of rama, where you can see it being used for example to provide an in-memory (upstream) proxy database. Do let us know in case you use it as well in your project, such that we can assemble a showcase list.

💬 Come join us at Discord on the #venndb public channel. To ask questions, discuss ideas and ask how venndb may be useful for you.

Index

venndb manual:

Usage: quick introduction on how to use venndb;
Benchmarks: benchmark results to give you a rough idea how venndb peforms for the use case it is made for (write once, read constantly, using binary filters mostly);
Q&A: Frequently Asked Questions (FAQ);
Example: the full example (expanded version from Usage), tested and documented;
Generated Code Summary: a documented overview of the API that venndb will generate for you when using #[derive(VennDB)] on your named field struct;

technical information:

Safety
Compatibility
MSRV (older versions might work but we make no guarantees);
Roadmap
License: [MIT license][mit-license] and [Apache 2.0 License][apache-license]

misc:

Contributing
Sponsors

Usage

Add venndb as a dependency:

cargo add venndb

and import the derive macro in the module where you want to use it:

use venndb::VennDB

#[derive(Debug, VennDB)]
pub struct Employee {
    #[venndb(key)]
    id: u32,
    name: String,
    is_manager: Option<bool>,
    is_admin: bool,
    #[venndb(skip)]
    foo: bool,
    #[venndb(filter, any)]
    department: Department,
    #[venndb(filter)]
    country: Option<String>,
}

fn main() {
    let db = EmployeeDB::from_iter(/* .. */);

    let mut query = db.query();
    let employee = query
        .is_admin(true)
        .is_manager(false)
        .department(Department::Engineering)
        .execute()
        .expect("to have found at least one")
        .any();

    println!("non-manager admin engineer: {:?}", employee);
}

See the full example or the "Generated Code Summary" chapter below to learn how to use the VennDB and its generated code.

Benchmarks

Benchmarks displayed here are taken on a dev machine with following specs:

Macbook Pro — 16 inch (2023)
Chip: Apple M2 Pro
Memory: 16 GB
OS: Sonoma 14.2

The benchmarks tests 3 different implementations of a proxy database

venndb version (very similar to the example below)
a naive version, which is just a Vec<Proxy>, over which is iterated
an sqlite version (using the sqlite crate (version: 0.34.0))

The benchmarks are created by:

running just bench;
copying the output into ./scripts/plot_bench_charts and running it.

Snippet that is ran for each 3 implementations:

fn test_db(db: &impl ProxyDB) {
    let i = next_round();

    let pool = POOLS[i % POOLS.len()];
    let country = COUNTRIES[i % COUNTRIES.len()];

    let result = db.get(i as u64);
    divan::black_box(result);

    let result = db.any_tcp(pool, country);
    divan::black_box(result);

    let result = db.any_socks5_isp(pool, country);
    divan::black_box(result);
}

Benchmark Performance Results

Performance for Database with 100 records:

| Proxy DB | Fastest | Slowest | Median | | --- | --- | --- | --- | | naive_proxy_db_100 | 1.276 µs | 1.922 µs | 1.328 µs | | sql_lite_proxy_db_100 | 22.96 µs | 40.84 µs | 26.52 µs | | venn_proxy_db_100 | 156.6 ns | 243.8 ns | 183.9 ns |

Performance for Database with 12_500 records:

| Proxy DB | Fastest | Slowest | Median | | --- | --- | --- | --- | | naive_proxy_db_12_500 | 202.7 µs | 260 µs | 217.1 µs | | sql_lite_proxy_db_12_500 | 22 µs | 903.4 µs | 792.2 µs | | venn_proxy_db_12_500 | 1.767 µs | 3.558 µs | 2.1 µs |

Performance for Database with 100_000 records:

| Proxy DB | Fastest | Slowest | Median | | --- | --- | --- | --- | | naive_proxy_db_100_000 | 2.196 ms | 2.563 ms | 2.288 ms | | sql_lite_proxy_db_100_000 | 5.85 ms | 6.634 ms | 6.223 ms | | venn_proxy_db_100_000 | 13.01 µs | 30.47 µs | 17.85 µs |

We are not database nor hardware experts though. Please do open an issue if you think these benchmarks are incorrect or if related improvements can be made. Contributions in the form of Pull requests are welcomed as well.

See the Contribution guidelines for more information.

Benchmark Allocations Results

Allocations for Database with 100 records:

| Proxy DB | Fastest | Median | Slowest | | --- | --- | --- | --- | | naive_proxy_db_100 | 1.25 B | 1.25 B | 1.25 B | | sql_lite_proxy_db_100 | 1.378 KB | 1.378 KB | 1.378 KB | | venn_proxy_db_100 | 430 B | 430 B | 430 B |

Allocations for Database with 12_500 records:

| Proxy DB | Fastest | Median | Slowest | | --- | --- | --- | --- | | naive_proxy_db_12_500 | 40.73 KB | 40.73 KB | 40.73 KB | | sql_lite_proxy_db_12_500 | 5.149 KB | 5.141 KB | 5.144 KB | | venn_proxy_db_12_500 | 3.534 KB | 3.534 KB | 3.534 KB |

Allocations for Database with 100_000 records:

| Proxy DB | Fastest | Median | Slowest | | --- | --- | --- | --- | | naive_proxy_db_100_000 | 323.3 KB | 323.3 KB | 323.3 KB | | sql_lite_proxy_db_100_000 | 5.141 KB | 5.147 KB | 5.143 KB | | venn_proxy_db_100_000 | 25.4 KB | 25.4 KB | 25.4 KB |

See the Contribution guidelines for more information.

Q&A

❓ Why use this over Database X?

venndb is not a database, but is close enough for some specific purposes. It shines for long-lived read-only use cases where you need to filter on plenty of binary properties and get a rando matching result.

Do not try to replace your usual database needs with it.

❓ Where can I propose a new feature X or some other improvement?

Please open an issue and also read the Contribution guidelines. We look forward to hear from you.

Alternatively you can also join our Discord and start a conversation / discussion over there.

❓ Can I use whatever type for a #[venndb(filter)] property?

Yes, as long as it implements PartialEq + Eq + Hash + Clone. That said, we do recommend that you use enum values if you can, or some other highly restricted form.

Using for example a String directly is a bad idea as that would mean that bE != Be != BE != Belgium != Belgique != België. Even though these are really referring all to the same country. In such cases a much better idea is to at the very least create a wrapper type such as struct Country(String), to allow you to enforce sanitization/validation when creating the value and ensuring the ha

Venndb

Install / Use

README