Cresset: The One Template to Train Them All

---

TL;DR

A new MLOps system for deep learning development using Docker Compose with the aim of providing reproducible and easy-to-use interactive development environments for deep learning practitioners. Hopefully, the methods presented here will become best practice in both academia and industry.

Introductory Video (In English)

Installation on a New Host

If this is your first time using this project, follow these steps:

1. Install the NVIDIA CUDA Driver appropriate for the target host and NVIDIA GPU. If the driver has already been installed, check that the installed version is compatible with the target CUDA version. CUDA driver version mismatch is the single most common issue for new users. See the compatibility matrix for compatible versions of the CUDA driver and CUDA Toolkit.

2. Install Docker (v23.0+ is recommended) or update to a recent version compatible with Docker Compose V2. Docker incompatibility with Docker Compose V2 is another common issue for new users. Note that Windows users may use WSL (Windows Subsystem for Linux). Cresset has been tested on Windows 11 WSL2 with the Windows CUDA driver using Docker Desktop for Windows. There is no need to install a separate WSL CUDA driver or Docker for Linux inside WSL. Note that only Docker Desktop is under a commercial EULA and Docker Engine (for Linux) and Lima Docker (for Mac) are still both open-source. N.B. Windows Security real-time protection causes significant slowdown if enabled. Disable any active antivirus programs on Windows for best performance. N.B. Linux hosts may also install via this repo.

3. Install the NVIDIA Container Toolkit as specified in this link.

4. Run make install-compose to install Docker Compose V2 for Linux hosts. Installation does not require root permissions. Visit the documentation for the latest installation information. Note that Docker Compose V2 is available for WSL users with Docker Desktop by default.

5. Run make env SERVICE=(train|devel|ngc|simple) on the terminal at project root to create a basic .env file. The .env file provides environment variables for docker-compose.yaml, allowing different users and machines to set their own variables as required. The Makefile has also been configured to read values from the .env file if it exists, allowing non-default values to be specified only once. Each host should have a separate .env file for host-specific configurations.

6. Run make over to create a docker-compose.override.yaml file. Add configurations that should not be shared via source control there. For example, volume-mount pairs specific to each host machine.

7. If Cresset is being placed within a pre-existing project's subdirectory, change the volume pairing from .:${PROJECT_ROOT} to ..:${PROJECT_ROOT}. All commands in Cresset assume that they are being run at project root but this can be changed easily.

Explanation of services

Different Docker Compose services are organized to serve different needs.

• train, the default service, should be used when compiled dependencies are necessary or when PyTorch needs to be compiled from source due to Compute Capability issues, etc.
• devel is designed for PyTorch CUDA/C++ developers who need to recompile frequently and have many complex dependencies.
• ngc is derived from the official NVIDIA PyTorch NGC images with the option to install additional packages. It is recommended for users who wish to base their projects on the NGC images provided by NVIDIA. Note that the NGC images change between different releases and that configurations for one release may not work for another one.
• simple is derived from the Official Ubuntu Linux image by default as some corporations restrict the use of Docker images not officially verified by Docker. It installs all packages via conda by default and can optionally install highly reproducible environments via conda-lock. Note that pip packages can also be installed via conda. Also, the base image can be configured to use images other than the Official Linux Docker images by specifying the BASE_IMAGE argument directly in the .env file. PyTorch runtime performance may be superior in official NVIDIA CUDA images under certain circumstances. Use the tests to benchmark runtime speeds. The simple service is recommended for users without compiled dependencies.

The Makefile has been configured to take values specified in the .env file if the .env file exists. Therefore, all make commands will automatically use the ${SERVICE} specified by make env SERVICE=${SERVICE} after the .env file is created.

Notes for Rootless Users

Many institutions forbid the use of Docker because it requires root permissions, compromising security. For users without Docker root access, using rootless Docker link is recommended.

While installing rootless Docker requires root permissions on the host, root permissions are not necessary after the initial installation.

When using rootless Docker, it is most convenient to set ADD_USER=exclude in the .env file as the root user will be the host user in rootless Docker.

Project Configuration

1. To build PyTorch from source, set BUILD_MODE=include and the CUDA Compute Capability (CCC) of the target NVIDIA GPU in the .env file. Visit the NVIDIA website to find compute capabilities of NVIDIA GPUs. Visit the documentation for an explanation of compute capability and its relevance. Note that the Docker cache will save previously built binaries if the given configurations are identical.

2. Read the docker-compose.yaml file to fill in extra variables in .env. Also, feel free to edit docker-compose.yaml as necessary by changing session names, hostnames, etc. for different projects and configurations. The docker-compose.yaml file provides reasonable default values but these can be overridden by values specified in the .env file. An important configuration is ipc: host, which allows the container to access the shared memory of the host. This is required for multiprocessing, e.g., to use num_workers in the PyTorch DataLoader class. Disable this configuration on WSL and specify shm_size: instead as WSL cannot use host IPC as of the time of writing.

3. Edit requirements in reqs/apt-train.requirements.txt and reqs/train-environment.yaml. These contain project package dependencies. The apt requirements are designed to resemble an ordinary Python requirements.txt file.

4. Edit the volumes section of a service to include external directories in the container environment. Run make over to create a docker-compose.override.yaml file to add custom volumes and configurations. The docker-compose.override.yaml file is excluded from version control to allow per-user and per-server settings.

5. (Advanced) If an external file must be included in the Docker image build process, edit the .dockerignore file to allow the Docker context to find the external file. By default, all files except requirements files are excluded from the Docker build context.

Example .env file for user with username USERNAME, group name GROUPNAME, user id 1000, group id 1000 on service train. Use the simple service if no dependencies need to be compiled and requirements can either be downloaded or installed via apt, conda, or pip.

# Generated automatically by `make env`.
# When using the `root` user with `UID=0`/`USR=root`, set `ADD_USER=exclude`.
GID=1000
UID=1000
GRP=GROUPNAME
USR=USERNAME
HOST_ROOT=.
SERVICE=train
# Do not use the same `PROJECT` name for different projects on the same host!
PROJECT=train-username