Lumper

Distributed building system for docker. Will pull repo from the github by tag-webhook and build it by Dockerfile.

Features

#. Distributed system. Any part might be working on different hosts. #. Email notifications (success and errors). #. Pushing into public or private docker registry. #. Building in queue. #. Emailing reports to administrator about exceptions. #. TLS client auth for docker daemon. #. SMTP authentication #. Building by webhook tag from github (You might be author of the extension for other services through pull-request ;-). #. Multiple installations (thanks to RMQ vhosts)

Requirements

• Python >=2.7 (>3.4 need testing).
• RabbitMQ server. Provide communication for components.

Parts

The system consists of 3 parts

• WEB Server. Based on tornado http server for accepting webhooks
• Worker. Building daemon listen AMQP
• Mailer. Mailing daemon. Provides notifying about build results.

Installation

#. Install Rabbitmq Server #. pip install lumper

Usage

The lumper provides one executable file lumper. You might run this with option --help (or -h)::

$ lumper --help
usage: lumper [-h] {server,worker,mailer} ...

positional arguments:
  {server,worker,mailer}
    server              Run http backend
    worker              Run in worker mode
    mailer              Run as mailer delivery worker

optional arguments:
  -h, --help            show this help message and exit

Notice: exec "lumper <command> --help" for command options

Command line configuration ++++++++++++++++++++++++++

And you might see help about any modes. For web server::

$ lumper server --help
usage: lumper server [-h] [--config CONFIG] [--gen-config] [-a ADDRESS]
                     [-p PORT] [--secret COOKIE_SECRET] [--gzip] [--debug]
                     [--logging LOGGING] [--github-secret GITHUB_SECRET]
                     [-A RMQ_ADDRESS] [-P RMQ_PORT] [-H RMQ_VHOST]
                     [--user RMQ_USER] [--password RMQ_PASSWORD]

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG       Load configuration from file
  --gen-config          Create example of the config_file.json

Server options:
  -a ADDRESS, --address ADDRESS
                        Listen this address
  -p PORT, --port PORT  Listen this port
  --secret COOKIE_SECRET
                        Cookie secret
  --gzip                Gzip HTTP responses
  --debug               Debugging mode
  --logging LOGGING     Logging level
  --github-secret GITHUB_SECRET
                        Github webhook's secret
  -A RMQ_ADDRESS, --rmq-address RMQ_ADDRESS
                        RMQ host address
  -P RMQ_PORT, --rmq-port RMQ_PORT
                        RMQ host port
  -H RMQ_VHOST, --vhost RMQ_VHOST
                        RMQ virtual host
  --user RMQ_USER       RMQ virtual host
  --password RMQ_PASSWORD
                        RMQ virtual host

For worker::

$ lumper worker --help
usage: lumper worker [-h] [--config CONFIG] [--gen-config] [--logging LOGGING]
                     [-a AMQP_ADDRESS] [-p AMQP_PORT] [-H AMQP_VHOST]
                     [-U AMQP_USER] [-P AMQP_PASSWORD]
                     [--docker-url DOCKER_URL] [--docker-tls]
                     [--docker-ca DOCKER_CA_CERT]
                     [--docker-cert DOCKER_CLIENT_CERT]
                     [--docker-key DOCKER_CLIENT_KEY] [--docker-tls-strict]
                     [--docker-registry DOCKER_REGISTRY]
                     [--docker-ssl-registry] [--docker-publish]

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG       Load configuration from file
  --gen-config          Create example of the config_file.json

Main options:
  --logging LOGGING     Logging level

RabbitMQ options:
  -a AMQP_ADDRESS, --address AMQP_ADDRESS
                        RMQ host address
  -p AMQP_PORT, --port AMQP_PORT
                        RMQ host port
  -H AMQP_VHOST, --vhost AMQP_VHOST
                        RMQ virtual host
  -U AMQP_USER, --user AMQP_USER
                        RMQ username
  -P AMQP_PASSWORD, --password AMQP_PASSWORD
                        RMQ password

Docker options:
  --docker-url DOCKER_URL
                        Docker daemon url ["unix:///var/run/docker.sock"]
  --docker-tls          Set when a docker daemon use TLS
  --docker-ca DOCKER_CA_CERT
                        TLS certificate authority
  --docker-cert DOCKER_CLIENT_CERT
                        TLS client certificate
  --docker-key DOCKER_CLIENT_KEY
                        TLS client private key
  --docker-tls-strict   Strict verification server certificate
  --docker-registry DOCKER_REGISTRY
                        Set if you have a private registry
  --docker-ssl-registry
                        The private registry use ssl
  --docker-publish      Set if you want push images to registry

And for mailer::

$ lumper mailer --help
usage: lumper mailer [-h] [--config CONFIG] [--gen-config] [--logging LOGGING]
                     [-a AMQP_ADDRESS] [-p AMQP_PORT] [-H AMQP_VHOST]
                     [-U AMQP_USER] [-P AMQP_PASSWORD] [--smtp-host SMTP_HOST]
                     [--smtp-port SMTP_PORT] [--smtp-user SMTP_USER]
                     [--smtp-password SMTP_PASSWORD] [--smtp-tls]
                     [--smtp-sender SMTP_SENDER] [--mail-map MAIL_MAP]
                     [--admin-mail ADMIN_MAIL]

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG       Load configuration from file
  --gen-config          Create example of the config_file.json

Main options:
  --logging LOGGING     Logging level

RabbitMQ options:
  -a AMQP_ADDRESS, --address AMQP_ADDRESS
                        RMQ host address
  -p AMQP_PORT, --port AMQP_PORT
                        RMQ host port
  -H AMQP_VHOST, --vhost AMQP_VHOST
                        RMQ virtual host
  -U AMQP_USER, --user AMQP_USER
                        RMQ username
  -P AMQP_PASSWORD, --password AMQP_PASSWORD
                        RMQ password

SMTP options:
  --smtp-host SMTP_HOST
                        Server host
  --smtp-port SMTP_PORT
                        Server port
  --smtp-user SMTP_USER
                        Authentication username. Do auth if set.
  --smtp-password SMTP_PASSWORD
                        Password.
  --smtp-tls            Use TLS.
  --smtp-sender SMTP_SENDER
                        Sender of messages [default: lumper@localhost]

Delivery options:
  --mail-map MAIL_MAP   github user to E-mail map json file with hash.
  --admin-mail ADMIN_MAIL
                        admin email for unknown users [default: root@localhost]

Config files ++++++++++++

You might generate and save configuration from the command line::

$ lumper mailer --gen-conf
{
 "admin_mail": "root@localhost",
 "amqp_address": "localhost",
 "amqp_password": null,
 "amqp_port": 5672,
 "amqp_user": null,
 "amqp_vhost": "/",
 "logging": null,
 "mail_map": null,
 "smtp_host": "localhost",
 "smtp_password": null,
 "smtp_port": 25,
 "smtp_sender": "lumper@localhost",
 "smtp_tls": false,
 "smtp_user": null
}

And load it with --config option. E.g lumper mailer --config /etc/lumper/mailer.json

And convert your command line to config-file::

$ lumper mailer --smtp-host mail.google.com --gen-conf
{
 ...
 "smtp_host": "mail.google.com",
 ...
}

Notice: Option --gen-conf must be defined in the end.