Quick start

We recommend to start with Microconfig Features guide and then continue reading this documentation.

Microconfig overview and features

Microconfig is intended to make it easy and convenient to manage configuration for microservices (or just for a big amount of services) and reuse the common part.

If your project consists of tens or hundreds of services you have to:

Keep the configuration for each service ideally separate from the code.
The configuration for different services can have common and specific parts. Also, the configuration for the same service in different environments can have common and specific parts.
A common part for different services (or for one service in different environments) should not be copied and pasted and must be easy to reuse.
It must be easy to understand how the resulting configuration is generated and based on which values the config placeholders are resolved.
Some configuration properties must be dynamic, calculated using an expression language.

Microconfig is written in Java, but it's designed to be used with systems written in any language. Microconfig just describes a format of configuration sources, syntax for placeholders, includes, excludes, overrides, an expression language for dynamic properties and an engine that can transform the config sources into simple *.yaml or *.properties files. Also it can resolve placeholders in arbitrary template files and show differences between config releases.

The difference between Microconfig and popular DevOps tools

Compared to config servers (like Consul, Zookeeper, Spring Cloud Config):

Config servers solve the problem of dynamic distribution of configuration in runtime (can use http endpoints), but to distribute configuration you have to store it, ideally with changes in history and without duplication of common parts.

Compared to Ansible:

Ansible is a powerful, but much too general, engine for deployment management and doesn't provide a common and clean way to store configuration for microservices, and a lot of teams have to invent their own solutions based on Ansible.

Microconfig does one thing and does it well. It provides an approach, best practices for how to keep configuration for a big amount of services, and an engine to build config sources into result files.

You can use Microconfig together with any config servers and deployment frameworks. Configuration can be built during the deployment phase and the resulting plain config files can be copied to the filesystem, where your services can access them directly (for instance, Spring Boot can read configuration from *.yaml or *.properties), or you can distribute the resulting configuration using any config servers. Also, you can store not only application configuration but configuration used to run your services, and your deployment framework can read that configuration from Microconfig to start your services with the right parameters and settings.

Where to store the configuration

It’s a good practice to keep service configuration separate from code. It doesn't require you to rebuild services every time the configuration is changed and allows you to use the same service artefacts (for instance, *.jar) for all environments because it doesn’t contain any environment specific configuration. The configuration can be updated even in runtime without service source code changes.

The best way to follow this principle is to have a dedicated repository for configuration in your favorite version control system. You can store configuration for all microservices in the same repository to make it easy to reuse a common part and be sure the common part is consistent for all your services.

Service configuration types

It's convenient to have different kinds of configuration and keep it in different files:

Deploy configuration (the configuration used by deployment tools that describes where/how to deploy your service, like artifact repository settings, container params).
Process configuration (the configuration used by deployment tools to start your service with right params, like memory limits, VM params, etc.).
Application configuration (the configuration that your service reads after start-up and uses in runtime).
Environment variables.
Secret configuration (note, you should not store in a VCS any sensitive information, like passwords. In a VCS you can store references(keys) to passwords, and keep passwords in special secured stores(like Vault) or at least in encrypted files on environment machines).
Library specific templates (for instance, Dockerfile, kafka.conf, cassandra.yaml, some scripts to run before/after your service start-up, etc.)

Microconfig detects the configuration type by the config file extension. The default configuration for config types:

*.yaml or *.properties for application configuration.
*.deploy for deployment configuration.
*.k8s for k8s configuration.
*.proc or *.process for process configuration.
*.helm for helm values.
*.env for environment variables.
*.secret for secret configuration.
For static files - see the 'Templates files' section.

You can use all the configuration types or only some of them. Also you can override the default extensions or define your own config types.

Basic folder layout

Let’s take a look at a basic folder layout that you can keep in a dedicated repository.

For every service, you have to create a folder with a unique name(the name of the service). In the service directory, we will keep common and environment specific configurations.

So, let’s imagine we have 4 microservices: 'orders', 'payments', 'service-discovery', and 'api-gateway'. For convenience, we can group services by layers: 'infra' for infrastructure services and 'core' for our business domain services. The resulting layout will look like:

repo
└───core  
│   └───orders
│   └───payments
│	
└───infra
    └───service-discovery
    └───api-gateway

Configuration sources

Inside the service folder, you can create a configuration in key=value format. For the following examples, we will prefer using *.yaml, but Microconfig also supports *.properties.

Let’s create the basic application and process configuration files for each service. You can split configuration among several files, but for simplicity, we will create application.yaml and process.proc for each service. No matter how many base files are used, after the configuration build for each service and each config type, a single result file will be generated.

repo
└───core  
│    └───orders
│    │   └───application.yaml
│    │   └───process.proc
│    └───payments
│        └───application.yaml
│        └───process.proc
│	
└───infra
    └───service-discovery
    │   └───application.yaml
    │   └───process.proc
    └───api-gateway
        └───application.yaml
        └───process.proc

Inside process.proc we will store configuration that describes what your service is, and how to run it (your config files can have other properties, so don't pay attention to concrete values).

orders/process.proc

artifact=org.example:orders:19.4.2 # partial duplication
java.main=org.example.orders.OrdersStarter
java.opts.mem=-Xms1024M -Xmx2048M -XX:+UseG1GC -XX:+PrintGCDetails -Xloggc:logs/gc.log # duplication

payments/process.proc

artifact=org.example:payments:19.4.2 # partial duplication
java.main=org.example.payments.PaymentStarter
java.opts.mem=-Xms1024M -Xmx2048M -XX:+UseG1GC -XX:+PrintGCDetails -Xloggc:logs/gc.log # duplication
instance.count=2

service-discovery/process.proc

artifact=org.example.discovery:eureka:19.4.2 # partial duplication 
java.main=org.example.discovery.EurekaStarter
java.opts.mem=-Xms1024M -Xmx2048M # partial duplication

As you can see we already have some small duplication (all services have '19.4.2' version, and two of them have the same java.ops params). Configuration duplication is as bad as code duplication. We will see later how to do this in a better way.

Let's see what application properties look like. In the comments we note what can be improved.

orders/application.yaml

server.port: 9000
application.name: orders # better to get name from folder
orders.personalRecommendation: true
statistics.enableExtendedStatistics: true
service-discovery.url: http://10.12.172.11:6781 # are you sure url is consistent with SD configuration?
eureka.instance.prefer-ip-address: true  # duplication
datasource:
  minimum-pool-size: 2  # duplication
  maximum-pool-size: 10
  url: jdbc:oracle:thin:@172.30.162.31:1521:ARMSDEV  # partial duplication
jpa.properties.hibernate.id.optimizer.pooled.prefer_lo: true  # duplication

payments/application.yaml

server.port: 8080
application.name: payments # better to get name from folder
payments:
  bookTimeoutInMs: 900000 # difficult to read. How long in minutes?
  system.retries: 3
consistency.validateConsistencyIntervalInMs: 420000 # difficult to read. How long in minutes?
service-discovery.url: http://10.12.172.11:6781 # are you sure url is consistent with eureka configuration?
eureka.instance.prefer-ip-address: true  # duplication
datasource:
  minimum-pool-size: 2  # duplication
  maximum-pool-size: 5
datasource.url: jdbc:oracle:thin:@172.30.162.127:1521:ARMSDEV  # partial duplication
jpa.properties.hibernate.id.optimizer.pooled.prefer_lo: true # duplicati

Microconfig

Install / Use

README