gokv

Simple key-value store abstraction and implementations for Go

Contents

1. Features
   1. Simple interface
   2. Implementations
   3. Value types
   4. Marshal formats
   5. Roadmap
2. Usage
   1. Examples
3. Project status
4. Motivation
5. Design decisions
6. Related projects
7. License

Features

Simple interface

Note: The interface is not final yet! See Project status for details.

type Store interface {
    Set(k string, v any) error
    Get(k string, v any) (found bool, err error)
    Delete(k string) error
    Close() error
}

There are detailed descriptions of the methods in the docs and in the code. You should read them if you plan to write your own gokv.Store implementation or if you create a Go package with a method that takes a gokv.Store as parameter, so you know exactly what happens in the background.

Implementations

Some of the following databases aren't specifically engineered for storing key-value pairs, but if someone's running them already for other purposes and doesn't want to set up one of the proper key-value stores due to administrative overhead etc., they can of course be used as well. In those cases let's focus on a few of the most popular though. This mostly goes for the SQL, NoSQL and NewSQL categories.

Feel free to suggest more stores by creating an issue or even add an actual implementation - .

For differences between the implementations, see Choosing an implementation.
For the Godoc of specific implementations, see https://pkg.go.dev/github.com/philippgille/gokv#section-directories.

• Local in-memory
  • [X] Go sync.Map
  • [X] Go map (with sync.RWMutex)
  • [X] FreeCache
  • [X] BigCache
• Embedded
  • [X] bbolt (formerly known as Bolt / Bolt DB)
  • [X] BadgerDB
  • [X] LevelDB / goleveldb
  • [X] Local files (one file per key-value pair, with the key being the filename and the value being the file content)
• Distributed store
  • [X] Redis
  • [X] Consul
  • [X] etcd
  • [X] Apache ZooKeeper
  • [ ] TiKV
• Distributed cache (no presistence by default)
  • [X] Memcached
  • [X] Hazelcast
• Cloud
  • [X] Amazon DynamoDB
  • [X] Amazon S3 / Google Cloud Storage / Alibaba Cloud Object Storage Service (OSS) / DigitalOcean Spaces / Scaleway Object Storage / OpenStack Swift / Ceph / Minio / ...
  • [ ] Azure Cosmos DB
  • [X] Azure Table Storage
  • [X] Google Cloud Datastore
  • [ ] Google Cloud Firestore
  • [X] Alibaba Cloud Table Store
• SQL
  • [X] MySQL
  • [X] PostgreSQL
  • [X] pgx
• NoSQL
  • [X] MongoDB
  • [ ] Apache Cassandra
• "NewSQL"
  • [X] CockroachDB
  • [ ] TiDB
• Multi-model
  • [X] Apache Ignite
  • [ ] ArangoDB
  • [ ] OrientDB
• Misc
  • [X] Go noop does nothing except validate the inputs, if applicable.

Again:
For differences between the implementations, see Choosing an implementation.
For the Godoc of specific implementations, see https://pkg.go.dev/github.com/philippgille/gokv#section-directories.

Value types

Most Go packages for key-value stores just accept a []byte as value, which requires developers for example to marshal (and later unmarshal) their structs. gokv is meant to be simple and make developers' lifes easier, so it accepts any type (with using any/interface{} as parameter), including structs, and automatically (un-)marshals the value.

The kind of (un-)marshalling is left to the implementation. All implementations in this repository currently support JSON and gob by using the encoding subpackage in this repository, which wraps the core functionality of the standard library's encoding/json and encoding/gob packages. See Marshal formats for details.

For unexported struct fields to be (un-)marshalled to/from JSON/gob, the respective custom (un-)marshalling methods need to be implemented as methods of the struct (e.g. MarshalJSON() ([]byte, error) for custom marshalling into JSON). See Marshaler and Unmarshaler for JSON, and GobEncoder and GobDecoder for gob.

To improve performance you can also implement the custom (un-)marshalling methods so that no reflection is used by the encoding/json / encoding/gob packages. This is not a disadvantage of using a generic key-value store package, it's the same as if you would use a concrete key-value store package which only accepts []byte, requiring you to (un-)marshal your structs.

Marshal formats

This repository contains the subpackage encoding, which is an abstraction and wrapper for the core functionality of packages like encoding/json and encoding/gob. The currently supported marshal formats are:

• [X] JSON
• [X] gob
• [X] protobuf

More formats will be supported in the future (e.g. XML).

The stores use this encoding package to marshal and unmarshal the values when storing / retrieving them. The default format is JSON, but all gokv.Store implementations in this repository also support gob as alternative, configurable via their Options.

The marshal format is up to the implementations though, so package creators using the gokv.Store interface as parameter of a function should not make any assumptions about this. If they require any specific format they should inform the package user about this in the GoDoc of the function taking the store interface as parameter.

Differences between the formats:

• Depending on the struct, one of the formats might be faster
• Depending on the struct, one of the formats might lead to a lower storage size
• Depending on the use case, the custom (un-)marshal methods of one of the formats might be easier to implement
  • JSON: MarshalJSON() ([]byte, error) and UnmarshalJSON([]byte) error
  • gob: GobEncode() ([]byte, error) and GobDecode([]byte) error
  • protobuf: Marshal(proto.Message) ([]byte, error) and Unmarshal([]byte, proto.Message) error

Roadmap

• Benchmarks!
• CLI: A simple command line interface tool that allows you create, read, update and delete key-value pairs in all of the gokv storages
• A combiner package that allows you to create a gokv.Store which forwards its call to multiple implementations at the same time. So for example you can use memcached and s3 simultaneously to have 1) super fast access but also 2) durable redundant persistent storage.
• A way to directly configure the clients via the options of the underlying used Go package (e.g. not the redis.Options struct in github.com/philippgille/gokv, but instead the redis.Options