Terrasalt

Terrasalt (Terraform + Salt) integrates Hashicorp's Terraform cloud orchestration with Salt configuration management. This enables users to create infrastructure (e.g. virtual machines) using Terraform and then configure and manage the infrastructure using Salt.

The inspiration for this project came when we switched from using Salt cloud to Terraform. We missed the close integration between orchestration (Terraform) and configuration management (Salt) that salt cloud had provided out of the box. We wanted a newly spun up VM to install salt-minion software and then to automatically register with our salt-master. We also wanted the VM to be de-registered from the salt-master when it was terminated. In addition we wanted to set (salt) grains in these new VMs from our terraform scripts. The terrasalt project was born.

Getting Started

The terrasalt idea is to create infrastructure using terraform and install salt-minion software on the spun up VMs. These minions would be connected to a salt-master so that all server configuration and remote execution post-minion installation can be executed via salt.

Pre-requisites

Before proceeding further we assume you know the basics of Terraform and Salt. You can brush-up your knowledge via these links

• Terraform Introduction
• Terraform Modules
• Salt Documentation, in particular look at
  • Installing salt

You will be adding the Terrasalt module to your terraform infrastructure-as-code scripts and then connecting the VM(s) that terraform creates to your pre-existing salt-master. If you want to directly jump into some examples then browse the terraform/{aws_/azure_/gcp_}example folders of this repository.

Stack and Assumptions

1. A configured salt-master capable of being connected from the new VMs that terraform spins up and capable of running salt API.
2. Ability for the terraform machine to send an HTTP(s) request to the salt-master (including firewall rules to allow connecting from the Terraform machine to the salt-master via HTTPS on a user-configurable port).
3. Python is installed on the terraform machine; also please install the python packages in the requirements.txt file included in this repository.

# In the shell you use to run terraform commands run

# *nix
python -m pip install -r requirements.txt

4. We have assumed that the new minion can reach the Internet - it runs salt-bootstrap to install minion software. If you are behind a corporate network and your VMs don't have internet accesss then this can easily be tweaked by adjusting the salt minion installation script of the module here.

5. Similarly, if you need to install a specific software version of the salt minion then please tweak the same script and change the installation options (refer to salt bootstrap documentation or use your package repository for example).

Architecture

Our aim is to add terraform-created VMs as minions of our salt-master. For this the terraform machine needs to

1. Install salt-minion software on a terraform newly created VM.
2. Configure the salt-minion VM to send its minion key to the salt-master.
3. Tell the salt-master to accept the key of the salt-minion VM, so that you don't have to manually go to the salt-master to accept the minion key of the newly created VM.
4. Tell the salt-master when terraform destroys the salt-minion VM, so that it's key can be removed from the list of accepted minion keys.

Fig. 1 shows the timing diagram of a VM lifecycle. The important phases are marked with numbers 1..6 in the figure:

1. The user calls terraform apply on the terraform scripts to create a VM and make it a minion; all configuration is within the terraform code being executed. The cloud provider boots up the VM, brings up the OS, installs salt-minion software and signals the terraform client that the VM is ready.

2. The new VM sends its minion key to the salt-master; the salt-master will only accept the key when it gets a API request from the terraform machine (or when the user manually accepts the key using the salt-key -a <minion_id> command on the salt-master).

3. The terraform machine sends an (authenticated) API request to the salt-master to accept the minion key of the newly spun up VM. The salt-master accepts the minion key.

4. The virtual machine is ready to be managed by the salt-master (e.g. run highstate, remote execution commands etc.)

   If the user wishes to terminate the VM then:

5. The user issues the terraform destroy command; the terraform machine sends a salt API request to the salt-master to remove the VM from its list of accepted minions.

6. Terraform completes the termination of the VM.

Usage

Configuring the salt-master for salt-api

We need to install the salt-api server into the salt-master and setup a "terrasalt" user whose credentials can be used to authenticate requests being sent to the salt-api server running on the salt-master. We have included a simple script along with descriptions of each command below. Use the script as a starting point and adapt it to your setup.

#!/bin/bash

#Usage: terrasalt_configure_salt-master.sh <terrasalt_user_password>  <http(s) port>
# e.g. (as root user)
# ./terrasalt_configure_salt-master.sh sUp3r53cr3T 8000

# Install cherrypy and salt-api (former necessary for ubuntu 1604); 
# assuming that the salt-master is also a salt-minion
salt-call pip.install cherrypy 
salt-call pkg.install salt-api

# Generate certificate for HTTPS, assuming salt-master is also a salt-minion
salt-call tls.create_self_signed_cert

# Write out saltapi configuration 
cat <<EOF > /etc/salt/master.d/saltapi_terrasalt.conf
# We highly recommend using HTTPS since we don't want to send credentials 
# over an unencrypted HTTP channel; refer to the saltapi documentation on how to 
# disable HTTPS (not recommended).
rest_cherrypy:
  port: $2
  ssl_crt: /etc/pki/tls/certs/localhost.crt
  ssl_key: /etc/pki/tls/certs/localhost.key

external_auth:
  pam:
    terrasalt:
      - '@wheel'
EOF

# Create a non-shell/non-homedir restricted terrasalt user; 
# the user name doesn't have to be terrasalt.
useradd -M terrasalt --shell /bin/false
usermod --password $1 terrasalt

# Restart salt-master to apply changes
service salt-master restart

# Restart the salt-api 
service salt-api restart

You can test the setup from your terraform machine. curl the following commands to check if the outputs are similar to the ones pasted below:

$ curl -ski https://saltmaster.com:$1/login \
 -H 'Accept: application/json' \
 -d username='terrasalt' \
 -d password='saltterra' \
 -d eauth='pam' \
 | python -m json.tool

Piping the output to the python json tool is optional; it pretty prints the json output.

{
  "return": [
    {
      "perms": [
        "@wheel"
      ],
      "start": 153327418.78768,
      "token": "408f5b43b4ac36a5c5ffa6e1502027000528",
      "expire": 153290618.787681,
      "user": "terrasalt",
      "eauth": "pam"
    }
  ]
}

If you see an error, consider these potential issues:

1. Is there a firewall blocking the saltapi port?
2. Has the salt-api server successfully started (check the process and the logs at /var/log/salt/api on the salt-master)?
3. Have you entered the salt-api URL, username and password correctly?
4. Check the SSL configuration; as well as check if salt-api was previously configured with different settings in the /etc/salt configuration files; you may want to disable HTTPS for testing the requests and responses to debug the problem but please enable HTTPS in production.

The token can now be used to access salt-api (the terrasalt user is limited to salt wheel commands as per our salt-master configuration above). For example:

# Note - we are asking curl to ignore self-issued SSL certificates; a better way would be to add the self-issued SSL certificate to our terraform machine. 
$ curl -k  https://saltmaster.com:8000/ \
 -H "Accept: application/json" 
 -H "X-Auth-Token: 408f5b43b4ac36a5c5ffa6e1502027000528"   
 -d client='wheel' 
 -d fun='key.list_all' \
  | python -m json.tool

Returned data:

{
  "return": [
    {
      "tag": "salt/wheel/201808020425106733",
      "data": {
        "jid": "201808022204206733",
        "return": {
          "local": [
            "master.pem",
            "master.pub"
          ],
          "minions_rejected": [],
          "minions_denied": [],
          "minions_pre": [],
          "minions": [
            "salt-master",
            "some-other-minion"
          ]
        },
        "success": true,
        "_stamp": "2018-08-02T22:04:25.120663",
        "tag": "salt/wheel/201808022204256733",
        "user": "terrasalt",
        "fun": "wheel.key.list_all"
      }
    }
  ]
}

Similar key.accept and the key.delete API request commands are issued by the terrasalt module to control adding and deleting minions on the salt-master.

The Terrasalt module

We have packaged up the salt-minion software installation and key acceptance/deletion operations in the terrasalt module in the terraform/modules/tf_salt_mod folder. This module is us