docker2singularity

<img src="img/logo.png" alt="https://www.sylabs.io/guides/latest/user-guide" data-canonical-src="https://www.sylabs.io/guides/latest/user-guide" width="200" height="200">

Are you developing Docker images and you would like to run them on an HPC cluster supporting Singularity? Are you working on Mac or Windows with no easy access to a Linux machine? If the pull, build, and general commands to work with docker images provided by Singularity natively do not fit your needs, docker2singularity is an alternative way to generate Singularity images. The containers are available to you on quay.io, and older versions also available for you on Docker Hub.

Usage

$ docker run quay.io/singularity/docker2singularity
USAGE: docker2singularity [-m "/mount_point1 /mount_point2"] [options] docker_image_name
OPTIONS:

          Image Format
              --folder   -f   build development sandbox (folder)
              --option   -o   add a custom option to build (-o --fakeroot or -option 'section post' )
              --writable -w   non-production writable image (ext3)         
                              Default is squashfs (recommended) (deprecated)
              --name     -n   provide basename for the container (default based on URI)
              --mount    -m   provide list of custom mount points (in quotes!)
              --help     -h   show this help and exit

Options

Image Format

• squashfs (no arguments specified) gives you a squashfs (*.simg) image. This is a compressed, reliable, and read only format that is recommended for production images. Squashfs support was added to Singularity proper in January of 2017 and thus available as early as the 2.2.1 release.
• sandbox (-f) builds your image into a sandbox folder. This is ideal for development, as it will produce a working image in a folder on your system.
• ext3 (-w) builds an older format (ext3) image (*.img). This format is not recommended for production images as we have observed degradation of the images over time, and they tend to be upwards of 1.5x to 2x the size of squashfs.

Note that you are able to convert easily from a folder or ext3 image using Singularity 2.4. If your choice is to develop, making changes, and then finalize, this approach is not recommended - your changes are not recorded and thus the image not reproducible.

Mount Points

• -m specify one or more mount points to create in the image.

Options

If you look at singularity build --help there are a variety of options available. You can specify some custom option to the command using the --option flag. Make sure that each option that you specify is captured as a single string. E.g.,:

--option --fakeroot 
--option '--section post'

Image Name

The last argument (without a letter) is the name of the docker image, as you would specify to run with Docker (e.g., docker run ubuntu:latest)

Legacy

If you want a legacy version, see the repository branches and tag history on the registry.

Containers were previous built on Docker Hub and now are provided on quay.io. A tag with prefix v corresponds to a release of the Singularity software, while the others are in reference to releases of Docker. Previously used scripts, including environment and action files, are provided in this repository for reference.

Requirements

• Docker (native Linux or Docker for Mac or Docker for Windows) - to create the Singularity image.
• Singularity >= 2.1 - to run the Singularity image (versions 2.0 and older are not supported!). Note that if running a 2.4 image using earlier versions, not all (later developed) features may be available.

Examples

Build a Squashfs Image

Squashfs is the recommended image type, it is compressed and less prone to degradation over time. You don't need to specify anything special to create it:

This is a path on my host, the image will be written here

$ mkdir -p /tmp/test

And here is the command to run. Notice that I am mounting the path /tmp/test that I created above to /output in the container, where the container image will be written (and seen on my host).

$ docker run -v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp/test:/output \
--privileged -t --rm \
quay.io/singularity/docker2singularity \
ubuntu:14.04

Image Format: squashfs
Inspected Size: 188 MB

(1/10) Creating a build sandbox...
(2/10) Exporting filesystem...
(3/10) Creating labels...
(4/10) Adding run script...
(5/10) Setting ENV variables...
(6/10) Adding mount points...
(7/10) Fixing permissions...
(8/10) Stopping and removing the container...
(9/10) Building squashfs container...
Building image from sandbox: /tmp/ubuntu_14.04-2017-09-13-3e51deeadc7b.build
Building Singularity image...
Singularity container built: /tmp/ubuntu_14.04-2017-09-13-3e51deeadc7b.simg
Cleaning up...
(10/10) Moving the image to the output folder...
     62,591,007 100%  340.92MB/s    0:00:00 (xfr#1, to-chk=0/1)
Final Size: 60MB

We can now see the finished image!

$ ls /tmp/test
ubuntu_14.04-2018-04-27-c7e04ea7fa32.simg

And use it!

$ singularity shell /tmp/test/ubuntu_14.04-2018-04-27-c7e04ea7fa32.simg
Singularity: Invoking an interactive shell within container...

Singularity ubuntu_14.04-2018-04-27-c7e04ea7fa32.simg:~/Documents/Dropbox/Code/singularity/docker2singularity> 

Take a look again at the generation code above, and notice how the image went from 188MB to 60MB? This is one of the great things about the squashfs filesystem! This reduction is even more impressive when we are dealing with very large images (e.g., ~3600 down to ~1800). A few notes on the inputs shown above that you should edit:

• /tmp/test: the path you want to have the final image reside. If you are on windows this might look like D:\host\path\where\to\output\singularity\image. -ubuntu:14.04: the docker image name you wish to convert (it will be pulled from Docker Hub if it does not exist on your host system).

docker2singularity uses the Docker daemon located on the host system. It will access the Docker image cache from the host system avoiding having to redownload images that are already present locally.

If you ever need to make changes, you can easily export the squashfs image into either a sandbox folder or ext3 (legacy) image, both of which have writable.

sudo singularity build --sandbox sandbox/ production.simg
sudo singularity build --writable ext3.img production.simg

Custom Naming

Added for version 2.5.1, you can specify the name of your container with the -n/--name argument, as follows:

docker run -v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp/test:/output \
--privileged -t --rm \
quay.io/singularity/docker2singularity \
--name meatballs ubuntu:14.04

...

$ ls /tmp/test/
meatballs.simg

Inspect Your Image

New with docker2singularity 2.4, the labels for the container are available with inspect:

 singularity inspect ubuntu_14.04-2017-09-13-3e51deeadc7b.simg 
{
    "org.label-schema.singularity.build": "squashfs",
    "org.label-schema.docker.version": "17.06.2-ce",
    "org.label-schema.schema-version": "1.0",
    "org.label-schema.singularity.build-type": "docker2singularity",
    "org.label-schema.docker.id": "sha256:dea1945146b96542e6e20642830c78df702d524a113605a906397db1db022703",
    "org.label-schema.build-date": "2017-10-28-17:19:18",
    "org.label-schema.singularity.version": "2.4-dist",
    "org.label-schema.docker.created": "2017-09-13"
}

as is the runscript and environment

singularity inspect --json -e -r ubuntu_14.04-2017-09-13-3e51deeadc7b.simg 
{
    "data": {
        "attributes": {
            "environment": "# Custom environment shell code should follow\n\n",
            "runscript": "#!/bin/sh\n/bin/bash $@\n"
        },
        "type": "container"
    }
}

Build a Sandbox Image

A sandbox image is a folder that is ideal for development. You can view it on your desktop, cd inside and browse, and it works like a Singularity image. To create a sandbox, specify the -f flag:

docker run -v /var/run/docker.sock:/var/run/docker.sock \
-v /host/path/change/me:/output \
--privileged -t --rm \
quay.io/singularity/docker2singularity \
-f \
ubuntu:14.04

Importantly, you can use --writable, and if needed, you can convert a sandbox folder into a production image:

sudo singularity build sandbox/ production.simg

Build a Legacy (ext3) Image

You can build a legacy ext3 image (with --writable) with the -w flag. This is an older image format that is more prone to degradation over time, and (building) may not be supported for future versions of the software.

docker run -v /var/run/docker.sock:/var/run/docker.sock \
-v /host/path/change/me:/output \
--privileged -t --rm \
quay.io/singularity/docker2singularity \
-w \
ubuntu:14.04

You can also use --writable and convert an ext3 image into a production image:

sudo singularity build ext3.img production.simg

Contributed Examples

The following are a list of brief e