koanf

koanf is a library for reading configuration from different sources in different formats in Go applications. It is a cleaner, lighter alternative to spf13/viper with better abstractions and extensibility and far fewer dependencies.

koanf v2 has modules (Providers) for reading configuration from a variety of sources such as files, command line flags, environment variables, Vault, and S3 and for parsing (Parsers) formats such as JSON, YAML, TOML, HUML, Hashicorp HCL. It is easy to plug in custom parsers and providers.

All external dependencies in providers and parsers are detached from the core and can be installed separately as necessary.

Installation

# Install the core.
go get -u github.com/knadh/koanf/v2

# Install the necessary Provider(s).
# Available: file, env/v2, posflag, basicflag, confmap, rawbytes,
#            structs, fs, s3, appconfig/v2, consul/v2, etcd/v2, vault/v2, parameterstore/v2
# eg: go get -u github.com/knadh/koanf/providers/s3
# eg: go get -u github.com/knadh/koanf/providers/consul/v2

go get -u github.com/knadh/koanf/providers/file


# Install the necessary Parser(s).
# Available: toml, toml/v2, json, yaml, huml, dotenv, hcl, hjson, nestedtext
# go get -u github.com/knadh/koanf/parsers/$parser

go get -u github.com/knadh/koanf/parsers/toml

See the list of all bundled Providers and Parsers.

Concepts
Reading config from files
Watching file for changes
Reading from command line
Reading environment variables
Reading from an S3 bucket
Reading raw bytes
Reading from maps and structs
Unmarshalling and marshalling
Order of merge and key case sensitivity
Custom Providers and Parsers
Custom merge strategies
List of installable Providers and Parsers

Concepts

koanf.Provider is a generic interface that provides configuration, for example, from files, environment variables, HTTP sources, or anywhere. The configuration can either be raw bytes that a parser can parse, or it can be a nested map[string]any that can be directly loaded.
koanf.Parser is a generic interface that takes raw bytes, parses, and returns a nested map[string]any. For example, JSON and YAML parsers.
Once loaded into koanf, configuration are values queried by a delimited key path syntax. eg: app.server.port. Any delimiter can be chosen.
Configuration from multiple sources can be loaded and merged into a koanf instance, for example, load from a file first and override certain values with flags from the command line.

With these two interface implementations, koanf can obtain configuration in any format from any source, parse it, and make it available to an application.

Reading config from files

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf/v2"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/parsers/yaml"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load YAML config and merge into the previously loaded config (because we can).
	k.Load(file.Provider("mock/mock.yml"), yaml.Parser())

	fmt.Println("parent's name is = ", k.String("parent1.name"))
	fmt.Println("parent's ID is = ", k.Int("parent1.id"))
}

Watching file for changes

Some providers expose a Watch() method that makes the provider watch for changes in configuration and trigger a callback to reload the configuration. This is not goroutine safe if there are concurrent *Get() calls happening on the koanf object while it is doing a Load(). Such scenarios will need mutex locking.

file, appconfig, vault, consul providers have a Watch() method.

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf/v2"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/parsers/yaml"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	f := file.Provider("mock/mock.json")
	if err := k.Load(f, json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load YAML config and merge into the previously loaded config (because we can).
	k.Load(file.Provider("mock/mock.yml"), yaml.Parser())

	fmt.Println("parent's name is = ", k.String("parent1.name"))
	fmt.Println("parent's ID is = ", k.Int("parent1.id"))

	// Watch the file and get a callback on change. The callback can do whatever,
	// like re-load the configuration.
	// File provider always returns a nil `event`.
	f.Watch(func(event any, err error) {
		if err != nil {
			log.Printf("watch error: %v", err)
			return
		}

		// Throw away the old config and load a fresh copy.
		log.Println("config changed. Reloading ...")
		k = koanf.New(".")
		k.Load(f, json.Parser())
		k.Print()
	})

	// To stop a file watcher, call:
	// f.Unwatch()

	// Block forever (and manually make a change to mock/mock.json) to
	// reload the config.
	log.Println("waiting forever. Try making a change to mock/mock.json to live reload")
	<-make(chan bool)
}

Reading from command line

The following example shows the use of posflag.Provider, a wrapper over the spf13/pflag library, an advanced commandline lib. For Go's built in flag package, use basicflag.Provider.

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/knadh/koanf/v2"
	"github.com/knadh/koanf/parsers/toml"

	// TOML version 2 is available at:
	// "github.com/knadh/koanf/parsers/toml/v2"

	"github.com/knadh/koanf/providers/file"
	"github.com/knadh/koanf/providers/posflag"
	flag "github.com/spf13/pflag"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Use the POSIX compliant pflag lib instead of Go's flag lib.
	f := flag.NewFlagSet("config", flag.ContinueOnError)
	f.Usage = func() {
		fmt.Println(f.FlagUsages())
		os.Exit(0)
	}
	// Path to one or more config files to load into koanf along with some config params.
	f.StringSlice("conf", []string{"mock/mock.toml"}, "path to one or more .toml config files")
	f.String("time", "2020-01-01", "a time string")
	f.String("type", "xxx", "type of the app")
	f.Parse(os.Args[1:])

	// Load the config files provided in the commandline.
	cFiles, _ := f.GetStringSlice("conf")
	for _, c := range cFiles {
		if err := k.Load(file.Provider(c), toml.Parser()); err != nil {
			log.Fatalf("error loading file: %v", err)
		}
	}

	// "time" and "type" may have been loaded from the config file, but
	// they can still be overridden with the values from the command line.
	// The bundled posflag.Provider takes a flagset from the spf13/pflag lib.
	// Passing the Koanf instance to posflag helps it deal with default command
	// line flag values that are not present in conf maps from previously loaded
	// providers.
	if err := k.Load(posflag.Provider(f, ".", k), nil); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	fmt.Println("time is = ", k.String("time"))
}

Reading environment variables

package main

import (
	"fmt"
	"log"
	"strings"

	"github.com/knadh/koanf/v2"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/env/v2"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use . as the key path delimiter. This can be / or anything.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load only environment variables with prefix "MYVAR_" and merge into config.
	// Transform var names by:
	// 1. Converting to lowercase
	// 2. Removing "MYVAR_" prefix  
	// 3. Replacing "_" with "." to representing nesting using the . delimiter.
	// Example: MYVAR_PARENT1_CHILD1_NAME becomes "parent1.child1.name"
	k.Load(env.Provider(".", env.Opt{
		Prefix: "MYVAR_",
		TransformFunc: func(k, v string) (string, any) {
			// Transform the key.
			k = strings.ReplaceAll(strings.ToLower(strings.TrimPrefix(k, "MYVAR_")), "_", ".")

			// Transform the value into slices, if they contain spaces.
			// Eg: MYVAR_TAGS="foo bar baz" -> tags: ["foo", "bar", "baz"]
			// This is to demonstrate that string values can be transformed to any type
			// where necessary.
			if strings.Contains(v, " ") {
				return k, strings.Split(v, " ")
			}

			return k, v
		},
	}), nil)

	fmt.Println("name is =", k.String("parent1.child1.name"))
	fmt.Println("time is =", k.Time("time", time.DateOnly))
	fmt.Println("ids are =", k.Strings("parent1.child1.grandchild1.ids"))
}

Reading from an S3 bucket

// Load JSON config from s3.
if err := k.Load(s3.Provider(s3.Config{
	AccessKey: os.Getenv("AWS_S3_ACCESS_KEY"),
	SecretKey: os.Getenv("AWS_S3_SECRET_KEY"),
	Region:    os.Getenv("AWS_S3

Koanf

Install / Use

README