UStore
Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️
Install / Use
/learn @unum-cloud/UStoreREADME
<div align="center"> <b>drivers</b>: Python • C • C++ • GoLang • Java <br/> <b>packages</b>: <a href="https://pypi.org/project/ukv/">PyPI</a> • <a href="#cmake">CMake</a> • <a href="https://hub.docker.com/repository/docker/unum/ustore">Docker Hub</a>
<a href="https://www.youtube.com/watch?v=ybWeUf_hC7o">Youtube</a> intro • <a href="https://discord.gg/4mxGrenbNt">Discord</a> chat • Full <a href="https://unum-cloud.github.io/ustore">documentation</a>
<a href="https://discord.gg/4mxGrenbNt"><img src="https://img.shields.io/discord/1063947616615923875?label=discord"></a> <a href="https://www.linkedin.com/company/unum-cloud/"><img src="https://img.shields.io/badge/linkedin-connect_with_us-0a66c2.svg?"/></a> <a href="https://twitter.com/unum_cloud"><img src="https://img.shields.io/badge/twitter-follow_us-1d9bf0.svg?"/></a> <a href="https://zenodo.org/badge/latestdoi/502647695"><img src="https://zenodo.org/badge/502647695.svg" alt="DOI"></a> <a href="https://www.github.com/unum-cloud/"><img src="https://img.shields.io/github/issues-closed-raw/unum-cloud/ustore?"/></a> <a href="https://www.github.com/unum-cloud/"><img src="https://img.shields.io/github/stars/unum-cloud/ustore?"/></a> <a href="#"><img src="https://img.shields.io/github/workflow/status/unum-cloud/ustore/Build"/></a>
</div>Quickstart
Installing UStore is a breeze, and the usage is about as simple as a Python dict.
$ pip install ukv
$ python
from ukv import umem
db = umem.DataBase()
db.main[42] = 'Hi'
We have just create an in-memory embedded transactional database and added one entry in its main collection.
Would you prefer that data on disk?
Change one line.
from ukv import rocksdb
db = rocksdb.DataBase('/some-folder/')
Would you prefer to connect to a remote UStore server? UStore comes with an Apache Arrow Flight RPC interface!
from ukv import flight_client
db = flight_client.DataBase('grpc://0.0.0.0:38709')
Are you storing [NetworkX][networkx]-like MultiDiGraph?
Or [Pandas][pandas]-like DataFrame?
db = rocksdb.DataBase()
users_table = db['users'].table
users_table.merge(pd.DataFrame([
{'id': 1, 'name': 'Lex', 'lastname': 'Fridman'},
{'id': 2, 'name': 'Joe', 'lastname': 'Rogan'},
]))
friends_graph = db['friends'].graph
friends_graph.add_edge(1, 2)
assert friends_graph.has_edge(1, 2) and \
friends_graph.has_node(1) and \
friends_graph.number_of_edges(1, 2) == 1
Function calls may look identical, but the underlying implementation can be addressing hundreds of terabytes of data placed somewhere in persistent memory on a remote machine.
Is someone else concurrently updating those collections? Bundle your operations to guarantee consistency!
db = rocksdb.DataBase()
with db.transact() as txn:
txn['users'].table.merge(...)
txn['friends'].graph.add_edge(1, 2)
So far we have only covered the tip of the UStore. You may use it to...
- Get C99, Python, GoLang, or Java wrappers for RocksDB or LevelDB.
- Serve them via Apache Arrow Flight RPC to Spark, Kafka, or PyTorch.
- Store Document and Graphs in embedded DB, avoiding networking overheads.
- Tier DBMS between in-memory and persistent backends under one API.
But UStore can more. Here is the map:
- Basic Usage:
- Advanced Usage for production, performance tuning, and administration:
- For contributors and advanced users looking to fork, extend, wrap, or distribute and, potentially, monetize alternative builds of UStore:
## Basic Usage
UStore is intended not just as database, but as "build your database" toolkit and an open standard for NoSQL potentially-transactional databases, defining zero-copy binary interfaces for "Create, Read, Update, Delete" operations, or CRUD for short.
A few simple C99 headers can link almost any underlying storage engine to numerous high-level language drivers, extending their support for binary string values to graphs, flexible-schema documents, and other modalities, aiming to replace MongoDB, Neo4J, Pinecone, and ElasticSearch with a single ACID-transactional system.
[Redis][redis], for example, provides RediSearch, RedisJSON, and RedisGraph with similar objectives. UStore does it better, allowing you to add your favorite Key-Value Stores (KVS), embedded, standalone, or sharded, such as [FoundationDB][foundationdb], multiplying its functionality.
Modalities
Blobs
Binary Large Objects can be placed inside UStore. The performance will vastly vary depending on the used underlying technology. The in-memory UCSet will be the fastest, but the least suited for larger objects. The persistent UDisk, when properly configured, can entirely bypass the the Linux kernel, including the filesystem layer, directly addressing block devices.
Modern persistent IO on high-end servers can exceed 100 GB/s per socket when built on user-space drivers like [SPDK][spdk]. This is close to the real-world throughput of high-end RAM and unlocks new, uncommon to databases use cases. One may now put a Gigabyte-sized video file in an ACID-transactional database, right next to its metadata, instead of using a separate object store, like MinIO.
Documents
JSON is the most commonly used document format these days. UStore document collections support JSON, as well as MessagePack, and BSON, used by MongoDB.
