Jering.KeyValueStore

Table of Contents

Overview
Target Frameworks
Platforms
Installation
Usage
API
Performance
Building and Testing
Alternatives
Related Concepts
Contributing
About

Overview

Jering.KeyValueStore enables you to store key-value data across memory and disk.

Usage:

var mixedStorageKVStore = new MixedStorageKVStore<int, string>(); // Stores data across memory (primary storage) and disk (secondary storage)

// Insert
await mixedStorageKVStore.UpsertAsync(0, "dummyString1").ConfigureAwait(false); // Insert a key-value pair (record)

// Verify inserted
(Status status, string? result) = await mixedStorageKVStore.ReadAsync(0).ConfigureAwait(false);
Assert.Equal(Status.OK, status); // Status.NOTFOUND if no record with key 0
Assert.Equal("dummyString1", result);

// Update
await mixedStorageKVStore.UpsertAsync(0, "dummyString2").ConfigureAwait(false);

// Verify updated
(status, result) = await mixedStorageKVStore.ReadAsync(0).ConfigureAwait(false);
Assert.Equal(Status.OK, status);
Assert.Equal("dummyString2", result);

// Delete
await mixedStorageKVStore.DeleteAsync(0).ConfigureAwait(false);

// Verify deleted
(status, result) = await mixedStorageKVStore.ReadAsync(0).ConfigureAwait(false);
Assert.Equal(Status.NOTFOUND, status);
Assert.Null(result);

This library is a wrapper of Microsoft's Faster key-value store. Faster is a low-level key-value store that introduces a novel, lock-free concurrency system. You'll need a basic understanding of Faster to use this library. Refer to Faster Basics for a quick primer and an overview of features this library provides on top of Faster.

Target Frameworks

• .NET Standard 2.1

Platforms

• Windows
• macOS
• Linux

Installation

Using Package Manager:

PM> Install-Package Jering.KeyValueStore

Using .Net CLI:

> dotnet add package Jering.KeyValueStore

Usage

This section explains how to use this library. Topics:

Choosing Key and Value Types
Using This Library in Highly Concurrent Logic
Configuring
Creating and Managing On-Disk Data

Choosing Key and Value Types

MessagePack C# must be able to serialize your MixedStorageKVStore key and value types.

The list of types MessagePack C# can serialize includes built-in types and custom types annotated according to MessagePack C# conventions.

Common Key and Value Types

The following are examples of common key and value types.

Reference Types

The following custom reference type is annotated according to MessagePack C# conventions:

[MessagePackObject] // MessagePack C# attribute
public class DummyClass
{
    [Key(0)] // MessagePack C# attribute
    public string? DummyString { get; set; }

    [Key(1)]
    public string[]? DummyStringArray { get; set; }

    [Key(2)]
    public int DummyInt { get; set; }

    [Key(3)]
    public int[]? DummyIntArray { get; set; }
}

We can use it, together with the built-in reference type string as key and value types:

var mixedStorageKVStore = new MixedStorageKVStore<string, DummyClass>(); // string key, DummyClass value
var dummyClassInstance = new DummyClass()
{
    DummyString = "dummyString",
    DummyStringArray = new[] { "dummyString1", "dummyString2", "dummyString3", "dummyString4", "dummyString5" },
    DummyInt = 10,
    DummyIntArray = new[] { 10, 100, 1000, 10000, 100000, 1000000, 10000000 }
};

// Insert
await mixedStorageKVStore.UpsertAsync("dummyKey", dummyClassInstance).ConfigureAwait(false);

// Read
(Status status, DummyClass? result) = await mixedStorageKVStore.ReadAsync("dummyKey").ConfigureAwait(false);

// Verify
Assert.Equal(Status.OK, status);
Assert.Equal(dummyClassInstance.DummyString, result!.DummyString); // result is only null if status is Status.NOTFOUND
Assert.Equal(dummyClassInstance.DummyStringArray, result!.DummyStringArray);
Assert.Equal(dummyClassInstance.DummyInt, result!.DummyInt);
Assert.Equal(dummyClassInstance.DummyIntArray, result!.DummyIntArray);

Value Types

The following custom value-type is annotated according to MessagePack C# conventions:

[MessagePackObject]
public struct DummyStruct
{
    [Key(0)]
    public byte DummyByte { get; set; }

    [Key(1)]
    public short DummyShort { get; set; }

    [Key(2)]
    public int DummyInt { get; set; }

    [Key(3)]
    public long DummyLong { get; set; }
}

We can use it, together with the built-in value type int as key and value types:

var mixedStorageKVStore = new MixedStorageKVStore<int, DummyStruct>(); // int key, DummyStruct value
var dummyStructInstance = new DummyStruct()
{
    // Populate with dummy values
    DummyByte = byte.MaxValue,
    DummyShort = short.MaxValue,
    DummyInt = int.MaxValue,
    DummyLong = long.MaxValue
};

// Insert
await mixedStorageKVStore.UpsertAsync(0, dummyStructInstance).ConfigureAwait(false);

// Read
(Status status, DummyStruct result) = await mixedStorageKVStore.ReadAsync(0).ConfigureAwait(false);

// Verify
Assert.Equal(Status.OK, status);
Assert.Equal(dummyStructInstance.DummyByte, result.DummyByte);
Assert.Equal(dummyStructInstance.DummyShort, result.DummyShort);
Assert.Equal(dummyStructInstance.DummyInt, result.DummyInt);
Assert.Equal(dummyStructInstance.DummyLong, result.DummyLong);

Mutable Type as Key Type

Before we conclude this section on key and value types, a word of caution on using mutable types (type with members you can modify after creation) as key types:

Under-the-hood, the binary serialized form of what you pass as keys are the actual keys. This means that if you pass an instance of a mutable type as a key, then modify a member, you can no longer use it retrieve the original record.

For example, consider the situation where you insert a value using a DummyClass instance (defined above) as key, and then change a member of the instance. When you try to read the value using the same instance, you either read nothing or a different value:

var mixedStorageKVStore = new MixedStorageKVStore<DummyClass, string>();
var dummyClassInstance = new DummyClass()
{
    DummyString = "dummyString",
    DummyStringArray = new[] { "dummyString1", "dummyString2", "dummyString3", "dummyString4", "dummyString5" },
    DummyInt = 10,
    DummyIntArray = new[] { 10, 100, 1000, 10000, 100000, 1000000, 10000000 }
};

// Insert
await mixedStorageKVStore.UpsertAsync(dummyClassInstance, "dummyKey").ConfigureAwait(false);

// Read
dummyClassInstance.DummyInt = 11; // Change a member
(Status status, string? result) = await mixedStorageKVStore.ReadAsync(dummyClassInstance).ConfigureAwait(false);

// Verify
Assert.Equal(Status.NOTFOUND, status); // No value for given key
Assert.Null(result);

We suggest avoiding mutable object types as key types.

Using This Library in Highly Concurrent Logic

MixedStorageKVStore.UpsertAsync, MixedStorageKVStore.DeleteAsync and MixedStorageKVStore.ReadAsync are thread-safe and suitable for highly concurrent situations situations. Some example usage:

var mixedStorageKVStore = new MixedStorageKVStore<int, string>();
int numRecords = 100_000;

// Concurrent inserts
ConcurrentQueue<Task> upsertTasks = new();
Parallel.For(0, numRecords, key => upsertTasks.Enqueue(mixedStorageKVStore.UpsertAsync(key, "dummyString1")));
await Task.WhenAll(upsertTasks).ConfigureAwait(false);

// Concurrent reads
ConcurrentQueue<ValueTask<(Status, string?)>> readTasks = new();
Parallel.For(0, numRecords, key => readTasks.Enqueue(mixedStorageKVStore.ReadAsync(key)));
foreach (ValueTask<(Status, string?)> task in readTasks)
{
    // Verify
    Assert.Equal((Status.OK, "dummyString1"), await task.ConfigureAwait(false));
}

// Concurrent updates
upsertTasks.Clear();
Parallel.For(0, numRecords, key => upsertTasks.Enqueue(mixedStorageKVStore.UpsertAsync(key, "dummyString2")));
await Task.WhenAll(upsertTasks).ConfigureAwait(false);

// Read again so we can verify updates
readTasks.Clear();
Parallel.For(0, numRecords, key => readTasks.Enqueue(mixedStorageKVStore.ReadAsync(key)));
foreach (ValueTask<(Status, string?)> task in readTasks)
{
    // Verify
    Assert.Equal((Status.OK, "dummyString2"), await task.ConfigureAwait(false));
}

// Concurrent deletes
ConcurrentQueue<ValueTask<Status>> deleteTasks = new();
Parallel.For(0, numRecords, key => deleteTasks.Enqueue(mixedStorageKVStore.DeleteAsync(key)));
foreach (ValueTask<Status> task in deleteTasks)
{
    Status result = await task.ConfigureAwait(false);

    // Verify
    Assert.Equal(Status.OK, result);
}

// Read again so we can verify deletes
readTasks.Clear();
Parall