jdelic's Saltshaker

This is a collection of saltstack formulae designed to bring up an small hosting environment for multiple applications and services. The hosting environment is reusable, the services are primarily there to fulfill my needs and allow experimentation with different DevOps setups.

Cloning this repository is a good basis for your own Salt setup as it implements a number of best practices I discovered and includes a fully fledged SmartStack implementation for internal, external and cross-datacenter services.

It also builds on the principles I have documented in my GoPythonGo build and deployment process.

It has full support for Vagrant, allowing easy testing of new functionality and different setups on your local machine before deploying them. Personally, I'm deploying this configuration on my laptop using Vagrant, on Digital Ocean and my own server on Hetzner which I configure with a XEN Hypervisor running VMs for all my development needs.

Everything in here is based around Debian 13.0 Trixie (i.e. requires systemd, nftables, and uses Debian package naming).

Using these salt formulae you can bring up:

a primarily Python/Django based application environment
a Hashicorp Nomad based Docker/application cluster that integrates with the Consul-Smartstack implementation and provides easy application deployment.
including a consul/consul-template/haproxy based smartstack implementation for service discovery (you can find an extracted stand- alone version of that in my consul-smartstack repository.
a PostgreSQL database configuration for a "fast" database and a separate tablespace on an encrypted partition
a Concourse.CI build server environment for your projects
an HAProxy based reverse proxying load balancer for applications based around the same smartstack implementation

It also contains configuration for

a fully fledged PIM+Mail server with encrypted storage (based on Radicale, Dovecot, OpenSMTPD), Vaultwarden, and StandardNotes
single-sign-on for Radicale, Dovecot and OpenSMTPD, other web applications and even PAM using a single database for authentication and providing OAuth2, CAS and SAML (through Dovecot). This is provided through authserver, a separate Django application written by me.

The salt configuration is pretty modular, so you can easily just use this repository to bring up a GoPythonGo build and deployment environment without any of the other stuff.

Configuration deployment

Deploying this salt configuration requires you to:

create a bootstrap server (for example a Amazon EC2 instance, a Dom0 VM on your own Xen server or a Digital Ooean droplet)

Assign that server the saltmaster and consulserver roles

mkdir -p /etc/roles.d
touch /etc/roles.d/saltmaster
touch /etc/roles.d/consulserver

check out the saltshaker repository

cd /opt
git clone https://github.com/jdelic/saltshaker
ln -sv /opt/saltshaker/srv/salt /srv/salt
ln -sv /opt/saltshaker/srv/pillar /srv/pillar
ln -sv /opt/saltshaker/srv/reactor /srv/reactor
ln -sv /opt/saltshaker/srv/salt-modules /srv/salt-modules
mkdir -p /etc/salt/master.d /etc/salt/minion.d
ln -sv /opt/saltshaker/etc/salt-master/master.d/saltshaker.conf /etc/salt/master.d/saltshaker.conf
ln -sv /opt/saltshaker/etc/salt-minion/minion.d/saltshaker.conf /etc/salt/minion.d/saltshaker.conf

Install Salt

wget -O /tmp/install_salt.sh https://bootstrap.saltstack.com
chmod 700 /tmp/install_salt.sh
/tmp/install_salt.sh -M -P

Edit the Pillar data in /srv/pillar. You must create a network configuration for your environment (see Networking below) and assign it to your systems in top.sls. It's especially important to select a count of consul server instances (3 are recommended for a production environment). You also must provide a secrets pillar that contains SSL certificates and such things.
Run salt-call state.highstate -l debug on your master to bring it up.
Bring up additional nodes (at least the count of consul server instances)
Assign them roles, install the salt minion using install_salt.sh -P and call state.highstate. It's obviously much better to automate this step. I did so for the XEN Hypervisor for example using the scripts in role.d together with xen-create-image.

The secrets pillar

You should clone the saltshaker repository and then as a first step, replace the git submodule in srv/pillar/shared/secrets with your own private Git repository.

You can find an example pillar that you can modify for your own use in the saltshaker-secrets-example repository.

For my salt states to work, you must provide your ownshared.secrets pillar in srv/pillar/shared/secrets that must contain the following pillars, unless you rework the salt states to use different ones. I use a wildcard certificate for my domains and the configuration pillars for individual servers allow you to specify pillar references to change what certificates are used, but in the default configuration most services refer to the maincert, which is the wildcard certificate:

In shared.secrets.ssl:

ssl:maincert:cert - The public X.509 SSL certificate for your domain. You can replace these with cert pillars for your individual domain.
ssl:maincert:key: - The private key for your SSL certificate without a passphrase. You can replace this with key pillars for your individual domain.
ssl:maincert:certchain - The X.509 certificates tying your CA to a browser-accredited CA, if necessary.
ssl:maincert:combined - A concatenation of :cert and :certchain.
ssl:maincert:combined-key - A concatenation of :cert, :certchain and :key.

In shared.secrets.vault:

ssl:vault:cert - the public X.509 SSL certificate used by the vault role/server. Should contain SANs for vault.local resolving to 127.0.0.1 (see notes on the .local and .internal domains under "Networking" below).
ssl:vault:key - its private key
ssl:vault:certchain - its CA chain
ssl:vault:combined - A concatenation of :cert and :certchain
ssl:vault:combined-key - A concatenation of :cert and :certchain and :key.
vault:s3:aws-accesskey - The access key for an IAM role that can be used for the Vault S3 backend (if you want to use that). Must have read/write access to a S3 bucket.
vault:s3:aws-secretkey - the secret key corresponding to the access key above.

In shared.secrets.concourse:

ssh:concourse:public - A SSH2 public RSA key for the concourse.ci TSA SSH host.
ssh:concourse:key - The private key for the public host key.

In shared.secrets.postgresql:

ssl:postgresql:cert,key,certchain,combined,combined-key in the same structure as the SSL certs mentioned above, containing a SSL cert used to encrypt database traffic through postgresql.local.

I manage these pillars in a private Git repository that I clone to srv/pillar/shared/secrets as a Git submodule. To keep the PEM encoded certificates and keys in the pillar file, I use the following trick:

# pillar file
{% set mycert = "
-----BEGIN CERTIFICATE-----
MIIFBDCCAuwCAQEwDQYJKoZIhvcNAQELBQAwgdgxCzAJBgNVBAYTAkRFMQ8wDQYD
...
-----END CERTIFICATE-----"|indent(12) %}

# because we're using the indent(12) template filter we can then do:
ssl:
    maincert:
        cert: | {{mycert}}

shared.secrets: The managed GPG keyring

The salt config also contains states which manage a shared GPG keyring. All keys added to the dict pillar gpg:keys are iterated by the crypto.gpg state and put into a GPG keyring accessible only by root and the user groupgpg-access. There is a parallel pillar called gpg:fingerprints that is used to check whether a key has already been added. The file system location of the shared keyring is set by the gpg:shared-keyring-location pillar, which by default is /etc/gpg-managed-keyring.

Server configuration

Pillar overrides

Disks

/secure

The roledir grain

Available roles

Salt modules

The dynamicsecrets Pillar

This is a Pillar module that can be configured on the master to provide dynamically generated passwords and RSA keys across an environment managed by a Salt master. The secrets are stored in a single unencrypted sqlite database file that can (and should) be easily backed up (default path: /etc/salt/dynamicsecrets.sqlite). This should be used for non-critical secrets that must be shared between minions and/or where a more secure solution like Hashicorp's Vault is not yet applicable.

To be clear: The purpose of dynamicsecrets is to help bootstrap a cluster of servers where a number of initial secrets must be set and "remembered" across multiple nodes that are configured with Salt. This solution is better than putting hardcoded secrets in your configuration management, but you should really only rely on it to bootstrap your way to an installation of Hashicorp Vault or another solution that solves the secret introduction problem comprehensively!

Secrets from the pillar are assigned to the "roles" grain or minion ids and are therefor only rendered to assigne

Saltshaker

Install / Use

README