Ascender

What is Ascender?

Ascender (Accelerator of SCiENtific DEvelopment and Research) is a GitHub repository template designed for research projects using Python. It incorporates several pre-implemented features to expedite development:

• Containerization: Dependency minimization and code portability enhancement using Docker.
• Virtual Environment / Package Management: Development environment reproducibility ensured by Poetry.
• Coding Style: Automatic code linting and formatting with Ruff.
• Static Type Checking: Early bug detection assisted by Mypy.
• Testing: Testing simplification achieved through pytest.
• Task Runner: Simple task automation with Poe the Poet.
• GitHub Integration: Integration features including GitHub Actions workflows, issue templates, and more.

Please also view resources about Ascender (in Japanese).

Project Organization

    ├── .github/               <- GitHub settings.
    │
    ├── data/                  <- Datasets.
    │
    ├── environments/          <- Environment-specific configurations.
    │
    ├── models/                <- Pretrained and serialized models.
    │
    ├── notebooks/             <- Jupyter notebooks.
    │
    ├── outputs/               <- Outputs.
    │
    ├── src/                   <- Python Source code.
    │
    ├── tests/                 <- Test code.
    │
    ├── .dockerignore
    ├── .gitignore
    ├── LICENSE
    ├── poetry.lock            <- Auto-generated lock file (do not edit manually).
    ├── poetry.toml            <- Poetry configuration.
    ├── pyproject.toml         <- Main project configuration file.
    └── README.md              <- Top-level README for developers.

Prerequisites

• Docker
• Docker Compose
• (Optional) NVIDIA Container Toolkit (nvidia-docker2)

[!Note] The example codes in the README.md are written for Docker Compose v2. However, Ascender is also compatible with Docker Compose v1. If you are using Docker Compose v1, simply replace docker compose with docker-compose in the example commands.

Prerequisites Installation

This section provides installation instructions for Ubuntu. If you have already installed the prerequisites, you may skip this section. For installations on other operating systems, please refer to the official documentation.

• Docker and Docker Compose: Install Docker Engine
• NVIDIA Container Toolkit (nvidia-docker2): Installation Guide

Install Docker and Docker Compose

# Set up the repository
$ sudo apt update
$ sudo apt install ca-certificates curl gnupg lsb-release
$ sudo mkdir -p /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
$ echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker and Docker Compose
$ sudo apt update
$ sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin

If sudo docker run hello-world works, the installation was successful.

(Optional) Install NVIDIA Container Toolkit

To use GPUs with Ascender, install the NVIDIA Container Toolkit as well. This toolkit has specific prerequisites, detailed in the official documentation.

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

$ sudo apt update
$ sudo apt install -y nvidia-docker2
$ sudo systemctl restart docker

If sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base nvidia-smi works, the installation was successful.

Quick Start

This section outlines how to begin using Ascender. For more detailed information, please refer to this slide (in Japanese).

Create a GitHub Repository from Ascender

To start, you need to create your own GitHub repository from Ascender:

• Visit the GitHub repository page of Ascender.
• Click the "Use this template" button at the top right of the page.
• Complete the form and click the "Create repository from template" button.

Your new repository should now be set up in your GitHub account.

(Optional) Modify Names Used in the Template

In the Ascender template, names used in the system by default, such as those for Docker containers and bind-mounted volumes, are set to "ascender." Particularly when using Ascender’s template to create and manage multiple containers, to avoid name conflicts between containers, it is necessary to change the container names for each project (refer also to the FAQ section "Building multiple containers with Ascender’s template").

If you want to change the default names used in the system, please modify the value of PROJECT_NAME_ENV in environments/[cpu,gpu,ci]/.env.

(Optional) Set Environment Variables for Development Within the Container

Depending on the services, frameworks, and libraries used during development, it may be necessary to specify API keys, database hostnames, and passwords as environment variables. To use environment variables within the container, please follow the steps below.

• Copy environments/envs.env.sample to create environments/envs.env

• Edit environments/envs.env to set the environment variables you want to use inside the container

• Add the env_file option to environments/[cpu,gpu]/docker-compose.yaml, and specify the path to environments/envs.env that was created above (by uncommenting the section that is commented out by default)."

  # env_file:       # <- uncomment here
  #   - ../envs.env # <- uncomment here
  

[!Note] The envs.env file may contain sensitive information such as API keys and passwords and should not be version-controlled by Git. In Ascender, files named *.env are excluded from Git tracking by default, as they are listed in the .gitignore file.

Start Development

# Clone the repository
$ git clone git@github.com:cvpaperchallenge/<YOUR_REPO_NAME>.git
$ cd <YOUR_REPO_NAME>

# Build the Docker image and run the container
$ cd environments/gpu  # For CPU only, navigate to `environments/cpu`
$ sudo docker compose up -d

# Enter the container shell
$ sudo docker compose exec core bash

# Set up the virtual environment and install dependencies with Poetry
$ poetry install

You are now ready to start developing with Ascender.

Stop Development

# Stop the container
$ cd environments/gpu  # Or `cd environments/cpu`
$ sudo docker compose stop

FAQ

Using Ascender Without Docker

While we recommend using Docker as described, you may encounter issues installing Docker due to permissions or other constraints.

If you cannot use Docker, Ascender can be operated without it. Simply install Poetry on your computer and proceed as described in the "Start Development" section, omitting the Docker steps.

# Install Poetry
$ pip3 install poetry

# Clone the repository
$ git clone git@github.com:<YOUR_USER_NAME>/<YOUR_REPO_NAME>.git
$ cd <YOUR_REPO_NAME>

# Set up the virtual environment and install dependencies with Poetry
$ poetry install

[!Note] The CI jobs in Ascender's GitHub Actions workflows utilize a Dockerfile. Running without Docker may cause these jobs to fail, necessitating modifications to the Dockerfile or the deletion of the CI job (.github/workflows/lint-and-test.yaml).

Permission Errors When Running poetry install

Sometimes, running poetry install may result in a permission error:

$ poetry install
...

virtualenv: error: argument dest: the destination . is not write-able at /home/challenger/ascender

If this occurs, check your local PC's UID (user ID) and GID (group ID) with the following commands:

$ id -u $USER  # Check U