dcdr (decider)

Distributed Feature Flags

Pre-Release

This is pre-release software. The Consul backend support has been used in production at VSCO for a year.

As of version 0.4.0-rc0, the etcd backend is no longer supported.

Overview

Decider is a feature flag system with adaptable backends. It supports both percentile and boolean flags for controlled infrastructure rollouts and kill switches. Decider is built to be adaptable to any backing datastore. At the moment, Consul and Redis are supported.

Decider has four major components.

• Client for use within your Go applications
• Server for accessing features over HTTP
• Watcher observes change in the datastore and writes them to disk
• CLI for managing features, watches, and starting the server

Each of these components are comprised of lower level libraries that you can use to suit your system's specific needs.

About Feature Flags

Feature flags have many use cases and there are many implementations. With Decider, the three supported types of flags are boolean, percentile, and scalar. For our purposes at VSCO, these have been enough to handle our needs.

Boolean Flags

An example use case for a boolean flag would be an API kill switch that could alleviate load for a backing database.

disable-load-heavy-api-endpoint => true

Your code would look something like this.

if dcdr.IsAvailable("disable-load-heavy-api-endpoint") {
	// DB is having a bad day, please check back later
} else {
	// All good, go about your day
}

Percentile Flags

Percentiles work much the same way but allow you to stress features or new infrastructure with a percentage of the request volume.

A common use case for percentile features would be testing out a new backend store with dual write percentage.

rollout-new-fancy-db-dual-write => 0.1

// Handle the write to the existing store

if dcdr.IsAvailableForID("rollout-new-fancy-db-dual-write", user.Id) {
	// If the `user.Id` falls into 10% of requests do the dual write
}

Scalars

Percentiles have an added bonus: you may use their float64 values as scalars in certain cases.

Here, we'll use the float value to scale the wait time for DB inserts between 0-1000ms.

 daemon-db-insert-wait-ms => 0.1

// waitMS would be 1000*0.1 => 100
waitMS := dcdr.ScaleValue("daemon-db-insert-wait-ms", 0, 1000)
time.Sleep(waitMS * time.Millisecond)

Read more on how to use the Client.

Caveat

Feature flags and remote configuration are hard problems to solve in the general sense. Most organizations will have many corner cases unique to their own infrastructure and policies that are cumbersome to solve in an abstract way. Decider is an extracted set of flexible libraries that we at VSCO have developed over the past year that have worked well for us in solving these issues.

This package does not set out to provide features like authentication or ACLs, but it does aim to provide enough of the tooling and libraries so that you can do so yourself.

Scopes

In order to allow for expanding use cases and to avoid naming collisions, Decider provides arbitrary scoping of feature flags. An example use case would be providing separate features sets according to country code or mobile platform. Additionally, multiple Decider instances can be run within a cluster with separate namespaces and key sets by configuring config.hcl.

Audit Trail

Due to the sensitive nature of configuration management, knowing the who, what, and when of changes can be very important. Decider uses git to handle this responsibility. By easily specifying a git repository and its origin in config.hcl, Decider will export your keyspace as a JSON file and then commit and push the changeset to the specified origin. Of course, this is all optional if you enjoy living dangerously.

Observabilty

It's nice to know when changes are happening. Decider can be configured to emit Statsd events when changes occur. Custom event tags can be sent as well if your collector supports them. Included in this package is a DataDog adapter with Event and Tag support. Custom stats can also be configured by supplying a custom stats.IFace implementation.

Installation

• Install via go get
  • go get -a github.com/vsco/dcdr
• Install via release
  • https://github.com/vsco/dcdr/releases
• Build from source

git clone git@github.com:vsco/dcdr.git
cd dcdr
script/bootstrap
script/install

Getting Started

CLI

The dcdr CLI has comprehensive help system for all commands.

dcdr help [command]" for more information about a command.

Setting Features

Features have several fields that are accessible via set command.

	-n, --name="flag_name"
		the name of the flag to set
	-v, --value=0.0-1.0 or true|false
		the value of the flag
	-c, --comment="flag description"
		an optional comment or description
	-s, --scope="users/beta"
		an optional scope to nest the flag within

Example

dcdr set -n new-feature -v 0.1 -c "some new feature" -s user-groups/beta

The above command sets the key dcdr/features/user-groups/beta/new-feature equal to 0.1 and commits the update to the audit repo.

Listing Features

Listing features can be filtered by a given scope and prefix. Any further fanciness can be handled by piping the output to grep or less.

	-p, --prefix="<flag-prefix>"
		List only flags with matching prefix.
	-s, --scope="<flag-scope>"
		List only flags within a scope.

Example

dcdr list -p new -s user-groups/beta

Deleting Features

Features are removed using the dcdr delete command and take a name and scope parameters. If no scope is provided the default scope is assumed. Once deleted and if you have a repository configured, Decider will commit the changeset and push it to origin.

	-p, --prefix="<flag-prefix>"
		Name of the flag to delete
	-s, --scope="<flag-scope>"
		Optional scope to delete the flag from

Example

dcdr delete -n another-feature -s user-groups/beta

Starting the Watcher

The watch command is central to how Decider features are distributed to nodes in a cluster. It observes the configured namespace and writes a JSON file containing the exported structure to the Server:OutputPath.

By default the Decider configuration and watch path are located in /etc/dcdr. If this path does not exist you will need to create it.

 sudo mkdir /etc/dcdr
 sudo chown `whoami` /etc/dcdr

You can override this location by setting the DCDR_CONFIG_DIR environment variable. More on configuration can be found here.

Tying the room together

TL;DR Using Docker Compose

Decider has many moving parts that require orchestration in order to do a proper demonstration of how they all work together. So thanks to Docker Compose we can bundle this up quite easily. This example uses a Consul backend and starts a dcdr watch and dcdr server for you.

Building & running the images

./script/compose

This starts a Consul agent for the backend, a Decider server, and a Decider Watcher.

With these services running you can now interact with the dcdr CLI via docker-compose exec from another terminal window through the dcdr_server container.

command to add/modify a feature flag:

$ docker-compose exec dcdr_server dcdr set -n feat-foo -v true
[dcdr] set flag 'dcdr/features/default/feat-foo'

result logs from running docker-compose log:

dcdr_watch_1   | [dcdr] 2018/03/13 18:25:48 wrote changes to: /etc/dcdr/decider.json

command to fetch feature flags from Decider server API:

$ docker-compose exec dcdr_server curl 127.0.0.1:8000/dcdr.json
{
  "dcdr": {
    "info": {},
    "features": {
      "feat-foo": true
    }
  }
}

result logs from running docker-compose log:

dcdr_server_1  | 127.0.0.1 - - [13/Mar/2018:18:36:51 +0000] "GET /dcdr.json HTTP/1.1" 200 102 "" "curl/7.52.1"

Now you can update features, CURL the results from the server, and see the changes update in realtime.

From Scratch

This example uses the Consul backend. If you need instructions for getting Consul installed, check their Getting Started page.

Let's start a development consul agent with an empty feature set and see how this all works together. For simplicity we can use the default Decider configuration without a git repository or stats.

consul agent -bind "127.0.0.1" -dev

This will start a local Consul agent ready to accept connections on http://127.0.0.1:8500. Decider should now be able to connect to this instance and set features.

Set some features

# check that we can talk to the local agent
~  → dcdr list
[dcdr] no feature flags found in namespace: dcdr

# set a feature into the 'default' scope.
~  → dcdr set -n example-feature -v false
[dcdr] set flag 'dcdr/features/default/example-feature'

# set a feature into the 'user-groups/beta' scope.
~  → dcdr set -n example-feature -v true -s user-groups/beta
[dcdr] set flag 'dcdr/features/user-groups/beta/example-f