🌰  Chestnut

Chestnut is encrypted storage for Go. The goal was an easy to use encrypted store with helpful features that was quick to set up, but highly flexible.

Chestnut is written in pure go and designed not to have strong opinions about things like storage, compression, hashing, secrets, or encryption. Chestnut is a storage chest, and not a datastore itself. As such, Chestnut must be backed by a storage solution.

Currently, Chestnut supports BBolt and NutsDB as backing storage.

Table of Contents

• Getting Started
  • Installing
  • Importing Chestnut
    • Requirements
• Storage
  • Built-in
    • BBolt
    • NutsDB
  • Planned
• Encryption
  • AES256-CTR
  • Custom Encryption
  • Chained Encryption
  • Sparse Encryption
    • What is "sparse" encryption?
    • Enabling Sparse Encryption
    • Using Sparse Encryption
      • Sparse Loading
      • Decryption
• Secrets
  • TextSecret
  • ManagedSecret
  • SecureSecret
• Compression
  • Zstandard
  • Custom Compression
  • Compression + Sparse Encryption
• Operations
  • Basic Operations
    • Put
    • Get
    • Delete
  • Struct Operations
    • Save
    • Load
    • Sparse
  • Keyed Operations
    • SaveKeyed
    • LoadKeyed
    • SparseKeyed
  • Extra Operations
    • Has
    • List
    • Export
• Struct Field Tags
  • Secure
  • Hash
    • SHA256
    • Hash Prefix
  • Multiple Tags
• Disable Overwrites
• Keystore
  • Importing Keystore
  • Important Note
• Logging
  • Logrus Logger
  • Zap Logger
  • Standard Logger
  • Storage
• Examples
• Known Issues
• Misc
  • JSON encoding

Getting Started

Installing

To start using Chestnut, install Go (version 1.11+) and run go get:

$ go get -u github.com/jrapoport/chestnut

Importing Chestnut

To use Chestnut as an encrypted store, import as:

import (
  "github.com/jrapoport/chestnut"
  "github.com/jrapoport/chestnut/encryptor/aes"
  "github.com/jrapoport/chestnut/encryptor/crypto"
  "github.com/jrapoport/chestnut/storage/nuts"
)

// use nutsdb for storage
store := nuts.NewStore(path)

// use AES256-CFB for encryption
opt := chestnut.WithAES(crypto.Key256, aes.CFB, mySecret)

cn := chestnut.NewChestnut(store, opt)
if err := cn.Open(); err != nil {
    return err
}

defer cn.Close()

Requirements

Chestnut has two requirements:

1. Storage that supports the storage.Storage interface (with a lightweight adapter).
2. Encryption which supports the crypto.Encryptor interface.

Storage

Chestnut will work seamlessly with any storage solution (or adapter) that supports thestorage.Storage interface.

Built-in

Currently, Chestnut has built-in support for BBolt and NutsDB.

BBolt

https://github.com/etcd-io/bbolt Chestnut has built-in support for using BBolt as a backing store.

To use bbolt for a backing store you can import Chestnut's bolt package and call bolt.NewStore():

import "github.com/jrapoport/chestnut/storage/bolt"

//use or create a bbolt backing store at path
store := bolt.NewStore(path)

// use bbolt for the storage chest
cn := chestnut.NewChestnut(store, ...)

NutsDB

https://github.com/nutsdb/nutsdb
Chestnut has built-in support for using NutsDB as a backing store.

To use nutsDB for a backing store you can import Chestnut's nuts package and call nuts.NewStore():

import "github.com/jrapoport/chestnut/storage/nuts"

//use or create a nutsdb backing store at path
store := nuts.NewStore(path)

// use nutsdb for the storage chest
cn := chestnut.NewChestnut(store, ...)

Planned

Other K/V stores like LevelDB.

GORM (probably not) Gorm is an ORM, so while it's not a datastore per se, it could be adapted to support sparse encryption and would mean automatic support for databases like mysql, sqlite, etc. However, most (if not all) of those DBs already support built-in encryption, so w/o some compelling use-case that's not already covered I don't see a lot of value-add.

Encryption

Chestnut supports several flavors of AES out of the box:

• AES128-CFB, AES192-CFB, and AES256-CFB
• AES128-CTR, AES192-CTR, and AES256-CTR
• AES128-GCM, AES192-GCM, and AES256-GCM

You can add AES encryption to Chestnut by passing the chestnut.WithAES() option:

opt := chestnut.WithAES(crypto.Key256, aes.CFB, mySecret)

AES256-CTR

For encryption we recommend using AES256-CTR. We chose AES256-CTR based in part on this helpful analysis from Shawn Wang, PostgreSQL Database Core.

Custom Encryption

Chestnut supports drop-in custom encryption. A struct that supports the crypto.Encryptor interface can be used with the chestnut.WithEncryptor() option.

Supporting crypto.Encryptor interface is straightforward and mainly consists of vending the following two methods:

// Encrypt returns data encrypted with the secret.
Encrypt(plaintext []byte) (ciphertext []byte, err error)

// Decrypt returns data decrypted with the secret.
Decrypt(ciphertext []byte) (plaintext []byte, err error)

Chained Encryption

Chestnut also supports chained encryption which allows data to be arbitrarily transformed by a chain of Encryptors in a FIFO order.

A chain of crypto.Encryptors can be passed to Chestnut with the chestnut.WithEncryptorChain option:

opt := chestnut.WithEncryptorChain(
    encryptor.NewAESEncryptor(crypto.Key128, aes.CFB, secret1),
    encryptor.NewAESEncryptor(crypto.Key192, aes.CTR, secret2),
    encryptor.NewAESEncryptor(crypto.Key256, aes.GCM, secret3),
)

or by using a crypto.ChainEncryptor with the chestnut.WithEncryptor option:

encryptors := []crypto.Encryptor{
    encryptor.NewAESEncryptor(crypto.Key128, aes.CFB, secret1),
    encryptor.NewAESEncryptor(crypto.Key192, aes.CTR, secret2),
    encryptor.NewAESEncryptor(crypto.Key256, aes.GCM, secret3),
}
chain := crypto.NewChainEncryptor(encryptors...)
opt := chestnut.WithEncryptor(chain)

If you use both the chestnut.WithEncryptor and the chestnut.WithEncryptorChain options, the crypto.Encryptor from chestnut.WithEncryptor will be prepended* to the chain.

Sparse Encryption

Chestnut supports the sparse encryption of structs.

Sparse encryption is a transparent feature of saving structs with Chestnut.Save(), Chestnut.Load(), and Chestnut.Sparse(); or structs that support the value.Keyed interface with Chestnut.SaveKeyed(), Chestnut.LoadKeyed(), and Chestnut.SparseKeyed().

What is "sparse" encryption?

With sparse encryption, only struct fields marked as secure will be encrypted. The remaining "plaintext" fields are encoded and stored separately.

This allows you to load a "sparse" copy of the struct by calling Chestnut.Sparse() or Chestnut.SparseKeyed() (if you have a value.Keyed value) and examine the plaintext fields without the overhead of decryption. When a sparse struct is loaded, the contents of struct fields marked as secure are replaced by empty values.

Enabling Sparse Encryption

Chestnut uses struct tags to indicate which specific struct fields should be encrypted. To enable sparse encryption for a struct, add the secure tag option to the JSON tag of at least one struct field:

SecretKey string `json:",secure"` // 'secure' option (bare minimum)

like so:

type MySparseStruct struct {
    SecretValue string `json:"secret_value,secure"` // <-- add 'secure' here
    PublicValue string `json:"public_value"`
}

Using Sparse Encryption

Structs can be sparsely encrypted by calling `Chestnut.