confidential (working title)

Export parameters from AWS Systems Manager Parameters as environment variables.

See some examples of common use cases.

Why I wrote this?

Configuration management, specifically secrets management tends to get complicated. After having been through several projects, both in my spare time and at work, using solutions such as Ansible Vault, private AWS CodeCommit repositories or Amazon KMS encrypted configuration files in Amazon S3, I was looking for something simpler, while still maintaining a high level of security.

I deemed self-hosted solution, such as Hashicorp Vault (and the other solutions listed on this Hashicorp Vault vs. Other Software page) too time consuming to set up and maintain.

Luckily, Amazon Web Services have been busy improving their Amazon EC2 Systems Manager Parameter Store in 2017 and it now supports both seamless Amazon KMS encryption and versioning of parameters.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

Confidential is written in Go and can be run as a single binary, no language specific requirements are needed.

It is designed to run on the GNU/Linux, macOS, and Windows operating systems. Other operating systems will probably work as long as you can compile a Go binary on them.

Installing

Make sure you have Go installed and that the $GOPATH is set correctly.

Build binary

go get github.com/nlindblad/confidential/apps/confidential
cd $GOPATH/src/github.com/nlindblad/confidential/apps/confidential
go build

Or if you have cloned this repository:

make PLATFORM

where PLATFORM is one of linux, darwin or windows.

Run

./confidential --help

And you should see:

NAME:
   confidential - Export parameters from AWS Systems Manager Parameters as environment variables

USAGE:
   confidential [global options] command [command options] [arguments...]

VERSION:
   0.0.0

AUTHOR:
   Niklas Lindblad <niklas@lindblad.info>

COMMANDS:
     exec, e    retrieve environment variables and execute command with an updated environment
     output, o  retrieve and atomically output environment variables to a file
     help, h    Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --forwarded-profile value  AWS profile to forward credentials for in the created environment [$AWS_FORWARDED_PROFILE]
   --prefix value             Amazon SSM parameter prefix
   --profile value            AWS profile to use when calling Amazon SSM [$AWS_PROFILE]
   --region value             AWS region e.g. eu-west-1 [$AWS_REGION]
   --help, -h                 show help
   --version, -v              print the version

Running the tests

Tests use the excellent Go stretchr/testify package.

go test -v ./...

Deployment

Ideally, check out the source code (either via Git or go get github.com/nlindblad/confidential/apps/confidential) and make yourself familiar with it and make sure you trust it to manage the most sensitive aspects of your application.

For each release, there are also packages for Debian/Ubuntu and CentOS/Amazon Linux/Red Hat Linux on the releases page.

The CentOS/Amazon Linux/Red Hat Linux RPM package is signed with the GPG key with ID A4847C36, the same key used by the original author for signing each Git commit in this repository.

Self built binary

Simply copy the confidential binary to somewhere on your server, e.g. /usr/local/bin.

Debian/Ubuntu

# wget https://github.com/nlindblad/confidential/releases/download/v0.1.0/confidential_0.1.0_amd64.deb
...
# dpkg -i confidential_0.1.0_amd64.deb
...
# which confidential
/usr/local/bin/confidential

CentOS/Amazon Linux/Red Hat Linux

# curl -L https://github.com/nlindblad/confidential/releases/download/v0.1.0/confidential-0.1.0-1.x86_64.rpm --output confidential-0.1.0-1.x86_64.rpm
...
# rpm -i confidential-0.1.0-1.x86_64.rpm
warning: confidential-0.1.0-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID a4847c36: NOKEY
# which confidential
/usr/local/bin/confidential

Examples

The machine needs to have the following AWS IAM permissions:

kms:Decrypt on the relevant Amazon KMS key used to encrypt sensitive parameters.
ssm:GetParametersByPath on the relevant resource: arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/<PREFIX> (note there should be no trailing slash or wildcards)

AWS Systems Manager Parameters names are translated into environment variable compatible names using the following logic:

All disallowed characters are omitted. Allowed characters are -, ., _, a-z, A-Z and 0-9.
Any non-alphanumerical character (not a-z, A-Z or 0-9) is converted to an underscore (_).

For example:

/my-prefix/database.password becomes DATABASE_PASSWORD (as would /my-prefix/database/password).

/my-prefix/bar simply becomes BAR

:whale2: Use with Docker and systemd services:

A handy way of running Docker containers supervised by systemd is to create a unit (service) using the systemd-docker wrapper:

[Unit]
Description=My service
Requires=docker.service
After=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=/usr/local/bin/confidential --region eu-west-1 --prefix /my-service/prod output --env-file /etc/my-service/prod.env
ExecStartPre=/usr/bin/docker pull username/image-name:latest
ExecStart=/usr/local/bin/systemd-docker --cgroups name=systemd run \
    --name %n \
    --env-file /etc/my-service/prod.env \
    ... Add other Docker run flags here ...
    -d username/image-name:latest
ExecStop=/usr/bin/docker stop %n
ExecStopPost=/usr/bin/docker rm -f %n
Restart=always
RestartSec=10s
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target

The following service will run username/image-name as a service which will get restarted if it falls over.

Every time the service is started/restarted, it runs the two ExecStartPre steps:

Uses confidential to get the latest environment variables from AWS Systems Manager Parameters in the eu-west-1 AWS Region and writes them to the file /etc/my-service/prod.env in a format that Docker understands
Pulls down the latest version of the username/image-name Docker image

This ensures that the service is always running using the latest published Docker image and that any configuration changes are picked up automatically.

Managing the environment variables for the service is now done within the /my-service/prod namespace in AWS Systems Manager Parameters.

:horse: Use with generic systemd services:

The EnvironmentFile directive can be used to expose the retrieved environment variable to any kind of executable running as a systemd service:

[Unit]
Description=Service
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/my-service.pid
ExecStartPre=/usr/local/bin/confidential --region eu-west-1 --prefix /my-service/prod output --env-file /etc/my-service/prod.env
EnvironmentFile=-/etc/my-service/prod.env
ExecStart=/usr/local/bin/my-service --flag=something --foo=bar
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Every time the service is started/restarted, it runs the ExecStartPre steps and populates /etc/my-service/prod.env and includes it using the EnvironmentFile directive (note the - before the filename).

Managing the environment variables for the service is now done within the /my-service/prod namespace in AWS Systems Manager Parameters in the eu-west-1 AWS Region.

:cloud: Give EC2 hosts permissions to access specific parameters:

See full CloudFormation template: examples/cloudformation/example-3-cloudformation.yml

:telescope: Create an IAM role with permissions to access specific parameters:

See full CloudFormation template: examples/cloudformation/example-4-cloudformation.yml

Creates a dedicated IAM user and access keys that is allowed to decrypt and retrieve parameters with a specific prefix.

Note: Some other tools using [AWS Systems Manager Parameters](http://docs.a

Confidential

Install / Use

README