konserve :PROPERTIES: :CUSTOM_ID: h:6f85a7f4-3694-4703-8c0b-ffcc34f2e5c9 :END:

[[https://clojurians.slack.com/archives/CB7GJAN0L][https://img.shields.io/badge/slack-join_chat-brightgreen.svg]] [[https://clojars.org/org.replikativ/konserve][https://img.shields.io/clojars/v/org.replikativ/konserve.svg]] [[https://circleci.com/gh/replikativ/konserve][https://circleci.com/gh/replikativ/konserve.svg?style=shield]] [[https://github.com/replikativ/konserve/tree/development][https://img.shields.io/github/last-commit/replikativ/konserve/main.svg]]

[[https://whilo.github.io/old/articles/16/unified-storage-io][Simple durability, made flexible.]]

Heads-up: Breaking change konserve 0.9. now requires a UUID under :id in the configuration in general. See below.*

** What is konserve?

A simple document store protocol defined with synchronous and [[https://github.com/clojure/core.async][core.async]] semantics to allow Clojuresque collection operations on associative key-value stores, both from Clojure and ClojureScript for different backends. Data is generally serialized with [[https://github.com/edn-format/edn][edn]] semantics or, if supported, as native binary blobs and can be accessed similarly to =clojure.core= functions =get-in=, =assoc-in= and =update-in=. =update-in= especially allows to run functions atomically and returns old and new value. Each operation is run atomically and must be consistent (in fact ACID), but further consistency across keys is, depending on the backend, only optionally supported.

*** Key Features

/cross-platform/ between Clojure and ClojureScript
/lowest-common denominator interface/ for an associative datastructure with =edn= semantics
/thread-safety with atomicity over key operations/
/fast serialization/ options (fressian, transit, ...), independent of the underlying kv-store
/very low overhead/ protocol, including direct binary access for high throughput
/no additional dependencies and setup/ required for IndexedDB in the browser and the file backend on the JVM and Node.js
/avoids blocking io/, the filestore for instance will not block any thread on reading

** Quick Start

Add to your dependencies: [[https://clojars.org/org.replikativ/konserve][https://img.shields.io/clojars/v/org.replikativ/konserve.svg]]

#+BEGIN_SRC clojure (require '[konserve.core :as k])

;; All stores require a UUID :id for global identification ;; Generate once: (java.util.UUID/randomUUID) or (random-uuid) ;; Then use the literal in your config (def config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440000"})

;; Create new store, pass opts as separate argument (def store (k/create-store config {:sync? true}))

;; Use the store (k/assoc-in store [:user] {:name "Alice"} {:sync? true}) (k/get-in store [:user] nil {:sync? true}) ;; => {:name "Alice"}

(k/update-in store [:user :age] (fnil inc 0) {:sync? true}) ;; => [nil 1]

;; Clean up (k/delete-store config) #+END_SRC

** Store Identification (UUID Requirement) :PROPERTIES: :CUSTOM_ID: h:uuid-requirement :END:

All konserve stores require a globally unique =:id= field containing a UUID type. This ensures stores can be uniquely identified and matched across different backends, machines, and synchronization contexts.

Why UUID IDs are required:

Global identifiability: Match stores regardless of backend type or file path
Cross-machine sync: Identify the same logical store across different systems
High entropy: 128-bit UUIDs prevent collisions
Backend-agnostic: Same ID works for memory, file, S3, Redis, etc.

How to use UUIDs:

#+BEGIN_SRC clojure ;; 1. Generate a UUID once (in your REPL or terminal) (java.util.UUID/randomUUID) ;; Clojure (random-uuid) ;; ClojureScript ;; => #uuid "550e8400-e29b-41d4-a716-446655440000"

;; 2. Copy the UUID and use it as a literal in your config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440000"}

;; 3. Pass opts as separate argument to store functions (k/create-store config {:sync? true})

;; 4. Use the SAME UUID every time for the same logical store ;; 5. Use DIFFERENT UUIDs for different stores (dev, test, prod) #+END_SRC

Important:

Generate a UUID once and use it consistently for the same store
Store the UUID in your application config (EDN files support =#uuid= literals)
Different stores (dev, test, prod) should have different UUIDs
Never generate UUIDs dynamically in your code - use fixed literals

** Installation

Add to your =deps.edn=:

#+BEGIN_SRC clojure {:deps {org.replikativ/konserve {:mvn/version "LATEST"}}} #+END_SRC

Or to your =project.clj=:

#+BEGIN_SRC clojure [org.replikativ/konserve "LATEST"] #+END_SRC

** Core Concepts

*** Synchronous vs Asynchronous

Konserve supports both synchronous and asynchronous execution modes via =core.async=.

Synchronous mode (=:sync? true=):

#+BEGIN_SRC clojure (def config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440000"})

(def store (k/create-store config {:sync? true}))

(k/assoc-in store [:key] "value" {:sync? true}) (k/get-in store [:key] nil {:sync? true}) ;; => "value" #+END_SRC

Asynchronous mode (=:sync? false=):

#+BEGIN_SRC clojure (require '[clojure.core.async :refer [go <!]])

(def config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440001"})

(go (def store (<! (k/create-store config {:sync? false})))

(<! (k/assoc-in store [:key] "value"))
(println (<! (k/get-in store [:key]))))

;; => "value" #+END_SRC

*** Store Lifecycle

Konserve provides five key lifecycle functions:

=create-store= - Create a new store, errors if already exists
=connect-store= - Connect to existing store, errors if doesn't exist
=store-exists?= - Check if store exists at the given configuration
=release-store= - Release connections and resources held by a store
=delete-store= - Delete underlying storage

#+BEGIN_SRC clojure (def config {:backend :file :id #uuid "550e8400-e29b-41d4-a716-446655440002" :path "/tmp/my-store"})

;; Check if store exists (k/store-exists? config {:sync? true}) ;; => false

;; Create new store (errors if already exists) (def store (k/create-store config {:sync? true}))

;; Use the store... (k/assoc-in store [:data] {:value 42} {:sync? true})

;; Later, connect to existing store (errors if doesn't exist) ;; (def store (k/connect-store config {:sync? true}))

;; Clean up resources (k/release-store config store {:sync? true})

;; Delete underlying storage (k/delete-store config {:sync? true})

;; Verify deletion (k/store-exists? config {:sync? true}) ;; => false #+END_SRC

*** Create vs Connect Semantics

All backends follow consistent strict semantics:

Strict semantics (All backends: File, S3, DynamoDB, Redis, LMDB, RocksDB, IndexedDB, Memory with :id):

=create-store= - Creates new store, errors if already exists
=connect-store= - Connects to existing store, errors if doesn't exist
=store-exists?= - Checks for existence before create/connect

** Built-in Backends

*** Memory Store

An in-memory store wrapping an Atom, available for both Clojure and ClojureScript.

#+BEGIN_SRC clojure (require '[konserve.core :as k])

;; Persistent registry-based store with ID (def config {:backend :memory :id #uuid "550e8400-e29b-41d4-a716-446655440003"})

(def my-db (k/create-store config {:sync? true}))

;; Later sessions can reconnect: ;; (def my-db (k/connect-store config {:sync? true})) #+END_SRC

*** File Store (JVM)

A file-system store using [[https://github.com/clojure/data.fressian][fressian]] serialization. No setup or additional dependencies needed.

#+BEGIN_SRC clojure (require '[konserve.core :as k])

(def config {:backend :file :id #uuid "550e8400-e29b-41d4-a716-446655440004" :path "/tmp/konserve-store"})

;; Create new store (def my-db (k/create-store config {:sync? true}))

;; Or connect to existing ;; (def my-db (k/connect-store config {:sync? true})) #+END_SRC

The file store supports:

Optional fsync control via =:sync-blob? false= for better performance
Custom =java.nio.file.FileSystem= instances via =:filesystem= parameter
Thoroughly tested using [[https://github.com/google/jimfs][Jimfs]] (Google's in-memory NIO filesystem)

*** File Store (Node.js)

For Node.js environments, require the Node.js-specific file store:

#+BEGIN_SRC clojure (require '[konserve.core :as k] '[konserve.node-filestore] ;; Registers :file backend for Node.js '[clojure.core.async :refer [go <!]])

(go (def config {:backend :file :id #uuid "550e8400-e29b-41d4-a716-446655440005" :path "/tmp/konserve-store"})

(def my-db (<! (k/create-store config {:sync? false}))))

#+END_SRC

*** IndexedDB (Browser)

[[https://developer.mozilla.org/en-US/docs/IndexedDB][IndexedDB]] backend for ClojureScript browser applications. Async-only, must be explicitly required.

#+BEGIN_SRC clojure (require '[konserve.core :as k] '[konserve.indexeddb] ;; Register :indexeddb backend '[clojure.core.async :refer [go <!]])

(go (def config {:backend :indexeddb :id #uuid "550e8400-e29b-41d4-a716-446655440006" :name "my-app-db"})

;; Create new store
(def my-idb-store (<! (k/create-store config {:sync? false})))

;; Use the store
(<! (k/assoc-in my-idb-store [:user] {:name "Alice" :age 30}))
(<! (k/get-in my-idb-store [:user]))

;; Multi-key atomic operations
(<! (k/multi-assoc my-idb-store {:user1 {:name "Alice"}
                                 :user2 {:name "Bob"}}))

;; Efficient bulk retrieval - returns sparse map of found keys
(<! (k/multi-get my-idb-store [:user1 :user2 :nonexistent]))
;; => {:user1 {:name "Alice"} :us

Konserve

Install / Use

README