Dibctl

Dibctl

Dibctl is a software for image lifecycle management. It helps to build, test, upload and delete images in multipie openstack regions in a consitent way.

Internally it uses diskimage-builder to build images and pytest with testinfra to test them.

Dibctl uses configuration files to describe how to build image, which name it should have after upload, what properties (if any) should be set for a given image. Image configuration file also provide list of tests for each image, plus name of environment where tests should happen.

Testing happens under directives from test.yaml. It contains information how to run a test instance: region authorization URL, credentials, flavor, network list, availability zone, security groups and other nova parameters.

Third configuration file (upload.yaml) provides information for uploading that image into Openstack installations.

Image Lifecycle

Dibctl assumes following image lifecycle:

• Image developed and debugged by operator using command line interface.
• Resulting configuration is used in CI server to build image
• Newly build image is then tested
• If tests passed image is uploaded to one or more regions
• During upload process older copies of image are marked as obsolete
• Obosolete images which are no longer in use deleted

Steps "build, test, upload, obsolete, delete" repeated on periodic basis.

Testing frameworks

Dibctl provides few frameworks for image testing. Each of those frameworks provides tests with full information about image,it properties, created instance (it flavor, network settings, credentials to access instance SSH, etc). Test frameworks:

• 'shell': each test is a simple shell script, which is executed outside test VM. It is provided with all information on how to connect to the test VM.
• 'pytest' - tests are implemented by means of py.test, with optional support for testinfra (python library for server verification, similar to ServerSpec). Dibctl provides vast set of fixture containing all available information about the image and the instance, plus few handy operations (wait_for_port), and a direct access to 'nova' object of openstack client for testing instance reactions on nova operations (hard reboot, rebuild, etc).
• 'ssh': each test is a simple shell script, which is executed inside a guest machine (not yet implemented)
• 'image': passive check of the image content (not implemented yet)

Dibctl comes with some generic tests (which should be applicable to any image of any provider).

Few examples:

• Does instance resize rootfs up to a flavor size at a first boot?
• Does it receive settings and configure IP addresses on all attached interfaces?
• Does DNS resolver set up properly?
• Does hostname match the name of the instance?
• Does instance work after reboot?
• Can user install nginx and get access to http port 80?

Workflow in details

After operators describes configurations, following workflow excepted:

• build: create new image py means of diskimage-builder
• test: new instance is spawned from image under test, and corresponding tests are called. If they all report success, images passes the test.
• upload: If test was successful, the image is uploaded to to one or more installations of Openstack.
• Older copies of images marked as obsolete and removed (after they become unsed - see description below).

That process may be repeated on regular basis via cron or CI server (Jenkins?). Comprehensive testing assures that image that passed the test may be uploaded safely in automated manner.

At the same time dibctl may be used by operators from command line and it provides some means of interfactive debug of faulty images:

• --shell option allow to open shell to instance which failed some tests
• --keep-failed-instance allow to keep instance alive (normally it should be deleted at the end of the test)
• shell command allow open shell without running any tests.

Motivational introduction

Diskimage-builder solved most of the issues around process of building images. It reduces all complexity of images to set of elements and well-defined rules of the build process. Nevertheless, there are many problems outside of the diskimage-builder scope:

• Integration testing
• Linking together diskimage-builder-related information (environment variables, command line options) and glance-related information (image name, properties credentials for upload, etc)
• Unified upload to one or more regions
• Recycling of older image copies

Let's look to all those sages. Firstly, one need to set up:

• environment variables for diskimage-builder
• few very inconsistent and cryptic variables for used elements
• some command line arguments for diksimage-builder
• list of elements used to build an image

Resulting command line is a 'golden artifact' - you need to keep it somewhere.

After the image was build, one can upload it to Glance. One need to provide few more pieces of information:

• Image name
• Credentials for Glance
• Additional meta needed to set up on image (hw_property, etc)

This adds few more lines to the 'golden artifact'.

Normally one wants to test images before uploading. Each image should, at least, be able to boot and accept ssh key.

One may create a simple script to boot and test it:

This adds few more lines to 'golden artifact':

• credential to spawn instance
• flavor id
• nics net-id
• security group name

If one have more than one image with more than one configuration for testing, this brings up complexity even higher, as many of those values become variables for reusable part of the script. If tests failed one should not leave broken and forgotten instances, ssh keypairs and test images. So there should be garbage collection code or some kind of 'finally' clause in the script.

After upload one want to remove older copies of the same image. That adds even more lines to the script, bringing it to the scale of a normal application.

Dibctl was created as evolution of such script, which at certain point become unmanageable. Newer version was written with better understanding of the process, without cutting corners.

Outside of its main goal, dibctl gives one more nice feature: image-transfer, which can copy image from one glance to another while preserving every propery, ownership and share information (tenant-name based).

Key concepts

Dibctl uses following concepts:

• label (for image and environment) - internal 'name' for a given image or environment to use in command line and in cross-reference fields. All entries in dibctl configuration have label, or 'namelabel'.
• image: set of attirbutes to build and test actual image.
• test instance: Instance which is used for image testing. Test instance is created at test time and removed afterwards. Dibctl creates custom SSH key for each test.
• test image: image which is uploaded for testing. Dibctl uses separate upload stage for testing and actual 'upload to procution'. Test images normally uploaded to specific project and are not public. Production images are normally public (or upload to selected tenant and shared with specific tenants). Test image is removed after test (succesfull or not). This can be changed by --keep-failed-image option.
• test environment: set of attributes and variables describing how upload test image and how to boot test instance. Every image references to test environement by it's label. They are listed in tests.yaml
• upload environments: Those describe how and where upload images for production use. Uploaded images are subjected to optional 'obsoletion' stage (see Obsoletion part below), which happens automatically every time image is uploaded or manually.

Image lifecycle

Any of those stages is optional an may be ommited.

Build stage

It uses information from corresponding entry in images.yaml to execute diskimage-builder. 'filename' option specify target filename. dibctl translates diskimage-builder exit codes (it exits with same exit code as diskimage-builder).

Test stage

At this stage image should be build. Dibctl uses information from entry in images.yaml to find image file for testing ('filename'), and than uses environment_name to find correspoinding test environment in tests.yaml. Then it upload image, creates new ssh key, spawns instance accoring to test environments settings, and executing tests in tests_list section of image configuration. Those tests recieve information about instance (IP addresses, network settings, hostname, flavor, etc), perform instance validation. If all tests passes successfully, dibctl sets exit code to 0, otherwise error returned. Regardless of the test results all test-time objects in Openstack are removed: test image, test instance, ssh key. User may use --keep-failed-image and --keep-failed-instance to keep them for closer investigation.

By using --shell dibctl may be instructed to open ssh shell to test machine when tests failed. After that shell is closed, instance (and all other pieces of test) are removed. If operator wants to keep instance from been removed after shell is closed, he (she) may use 'exit 42' command. Dibctl will honor exit code 42 by not removing instance.

Upload stage

At the upload stage image is uploaded to specified installation with specific settings for publication. All image-specific things (properties, tags, etc) from images.yaml are used during this stage, as well, as settings from upload.yaml (See variable ordering to see override rules).

After upload done, it triggers obosoletion stage if obsoletion is stated in upload configuration.

Obsolete stage

Obsolete image: If image is in the same tenant and have same glance name as freshly uploaded, it is obsolete. Obsoleted images recieve specific rename pattern (usually adds 'Obsolete ' before name), and specific set of properties.

Obsoletion m