Konfig

Composable, observable and performant config handling for Go. Written for larger distributed systems where you may have plenty of configuration sources - it allows you to compose configurations from multiple sources with reload hooks making it simple to build apps that live in a highly dynamic environment.

What's up with the name?

The name is Swedish for "config". We have a lot of nationalities here at Lalamove and to celebrate cultural diversity most of our open source packages will carry a name derived from a non-English language that is perhaps spoken by at least one of our employees(?).

Why another config package?

Most config packages for Golang are not very extensible and rarely expose interfaces. This makes it complex to build apps which can reload their state dynamically and difficult to mock. Fewer still come with sources such as Vault, Etcd and multiple encoding formats. In short, we didn't find a package that satisfied all of our requirements when we started out.

konfig is built around 4 small interfaces:

• Loader
• Watcher
• Parser
• Closer

Konfig features include:

• Dynamic configuration loading
• Composable load configs from multiple sources, such as vault, files and etcd
• Polyglot load configs from multiple format. Konfig supports JSON, YAML, TOML, Key=Value
• Fast, Lock-free, Thread safe Read Reads are up to 10x faster than Viper
• Observable config - Hot Reload mechanism and tooling to manage state
• Typed Read get typed values from config or bind a struct
• Metrics exposed prometheus metrics telling you how many times a config is reloaded, if it failed, and how long it takes to reload

Get started

go get github.com/lalamove/konfig

Load and watch a json formatted config file.

var configFiles = []klfile.File{
	{
		Path:   "./config.json",
		Parser: kpjson.Parser,
	},
}

func init() {
	konfig.Init(konfig.DefaultConfig())
}

func main() {
	// load from json file
	konfig.RegisterLoaderWatcher(
		klfile.New(&klfile.Config{
			Files: configFiles,
			Watch: true,
		}),
		// optionally you can pass config hooks to run when a file is changed
		func(c konfig.Store) error {
			return nil
		},
	)

	if err := konfig.LoadWatch(); err != nil {
		log.Fatal(err)
	}

	// retrieve value from config file
	konfig.Bool("debug")
}

Store

The Store is the base of the config package. It holds and gives access to values stored by keys.

Creating a Store

You can create a global Store by calling konfig.Init(*konfig.Config):

konfig.Init(konfig.DefaultConfig())

The global store is accessible directly from the package:

konfig.Get("foo") // calls store.Get("foo")

You can create a new store by calling konfig.New(*konfig.Config):

s := konfig.New(konfig.DefaultConfig())

Loading and Watching a Store

After registering Loaders and Watchers in the konfig.Store, you must load and watch the store.

You can do both by calling LoadWatch:

if err := konfig.LoadWatch(); err != nil {
	log.Fatal(err)
}

You can call Load only, it will load all loaders and return:

if err := konfig.Load(); err != nil {
	log.Fatal(err)
}

And finally you can call Watch only, it will start all watchers and return:

if err := konfig.Watch(); err != nil {
	log.Fatal(err)
}

Loaders

Loaders load config values into the store. A loader is an implementation of the loader interface.

type Loader interface {
	// Name return the name of the load, it is used to create labeled vectors metrics per loader
	Name() string
	// StopOnFailure indicates if a failure of the loader should trigger a stop
	StopOnFailure() bool
	// Loads the config and add it to the Store
	Load(Store) error
	// MaxRetry returns the maximum number of times to allow retrying on load failure
	MaxRetry() int
	// RetryDelay returns the delay to wait before retrying
	RetryDelay() time.Duration
}

You can register loaders in the config individually or with a watcher.

Register a loader by itself:

configLoader := konfig.RegisterLoader(
	klfile.New(
		&klfile.Config{
			Files: []klfile.File{
				{
					Parser: kpjson.Parser,
					Path:   "./konfig.json",
				},
			},
		},
	),
)

Register a loader with a watcher:

To register a loader and a watcher together, you must register a LoaderWatcher which is an interface that implements both the Loader and the Watcher interface.

configLoader := konfig.RegisterLoaderWatcher(
	klfile.New(
		&klfile.Config{
			Files: []klfile.File{
				{
					Parser: kpjson.Parser,
					Path:   "./konfig.json",
				},
			},
			Watch: true,
		},
	),
)

You can also compose a loader and a watcher to create a LoaderWatcher:

configLoader := konfig.RegisterLoaderWatcher(
	// it creates a LoaderWatcher from a loader and a watcher
	konfig.NewLoaderWatcher(
		someLoader,
		someWatcher,
	),
)

Built in loaders

Konfig already has the following loaders, they all have a built in watcher:

• File Loader

Loads configs from files which can be watched. Files can have different parsers to load different formats. It has a built in file watcher which triggers a config reload (running hooks) when files are modified.

• Vault Loader

Loads configs from vault secrets. It has a built in Poll Watcher which triggers a config reload (running hooks) before the secret and the token from the auth provider expires.

• HTTP Loader

Loads configs from HTTP sources. Sources can have different parsers to load different formats. It has a built in Poll Diff Watcher which triggers a config reload (running hooks) if data is different.

• Etcd Loader

Loads configs from Etcd keys. Keys can have different parser to load different formats. It has a built in Poll Diff Watcher which triggers a config reload (running hooks) if data is different.

• Consul loader

Loads configs from Consul KV. Keys can have different parser to load different formats. It has built in Poll Diff Watcher which triggers a config reload (running hooks) if data is different.

• ENV Loader

Loads configs from environment variables.

• Flag Loader

Loads configs from command line flags.

• io.Reader Loader

Loads configs from an io.Reader.

Parsers

Parsers parse an io.Reader into a konfig.Store. These are used by some loaders to parse the data they fetch into the config store. The File Loader, Etcd Loader and HTTP Loader use Parsers.

Config already has the following parsers:

• JSON Parser
• TOML Parser
• YAML Parser
• KV Parser
• Map Parser

Watchers

Watchers trigger a call on a Loader on events. A watcher is an implementation of the Watcher interface.

type Watcher interface {
	// Start starts the watcher, it must not be blocking.
	Start() error
	// Done indicate whether the watcher is done or not
	Done() <-chan struct{}
	// Watch should block until an event unlocks it
	Watch() <-chan struct{}
	// Close closes the watcher, it returns a non nil error if it is already closed
	// or something prevents it from closing properly.
	Close() error
	// Err returns the error attached to the watcher
	Err() error
}

Built in watchers

Konfig already has the following watchers:

• File Watcher

Watches files for changes.

• Poll Watcher

Sends events at a given rate, or if diff is enabled. It takes a Getter and fetches the data at a given rate. If data is different, it sends an event.

Hooks

Hooks are functions ran after a successful loader Load() call. They are used to reload the state of the application on a config change.

Registering a loader with some hooks

You can register a loader or a loader watcher with hooks.

configLoader := konfig.RegisterLoaderWatcher(
	klfile.New(
		&klfile.Config{
			Files: []klfile.File{
				{
					Parser: kpyaml.Parser,
					Path:   "./konfig.yaml",
				},
			},
			Watch: true,
		},
	),
	func(s konfig.Store) error {
		// Here you should reload the state of your app
		return nil
	},
)

Adding hooks to an existing loader

You can register a Loader or a LoaderWatcher with hooks.

configLoader.AddHooks(
	func(s konfig.Store) error {
		// Here you should reload the state of your app
		return nil
	},
	func(s konfig.Store) error {
		// Here you should reload the state of your app
		return nil
	},
)

Adding hooks on keys

Alternatively, you can add hooks on keys. Hooks on keys will match for prefix in order to run a hook when any key with a given prefix is updated. A hook can only be run once per load event, even if multiple keys match that hook.

konfig.RegisterKeyHook(
	"db.",
	func(s konfig.Store) error {
		return nil
	},
)

Closers

Closers can be added to konfig so that if konfig fails to load, it will execute Close() on the registered Closers.

type Closer interface {
	Close() error
}

Register a Closer

konfig.RegisterCloser(closer)

Config Groups

You can namespace your configs using config Groups.

konfig.Group("db").RegisterLoaderWatcher(
	klfile.New(
		&klfile.Config{
			Files: []klfile.File{
				{
					Parser: kpyaml.Par