Plantit

<div align="center"> <img src="https://github.com/Computational-Plant-Science/plantit/blob/master/plantit/front_end/src/assets/logo.png?raw=true" style="position:relative; top: 40px; height: 150px" />

</div> <!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

Contents

• About
  • Status
  • Motivation
  • Overview
• Development
  • Requirements
  • Installation
    • Setting up a development environment
      • Running tests
      • Reverse proxying with ngrok
    • Deploying to production
      • SSL Certificates
  • Configuring environment variables
  • Configuring deployment targets
    • Agent requirements
    • Authenticating with Docker
  • Building the documentation
  • Testing DIRT migrations

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

About

PlantIT is a framework for deploying research apps to high-performance/-throughput clusters. Specifically, it's a science gateway for image-based plant phenotyping. Future work may generalize the platform for any domain.

Status

PlantIT debuted (with pre-release v0.1.0) at NAPPN 2022 (Feb. 22-25). See Releases for changelog and Roadmap for planned features and fixes. Capabilities will likely continue to evolve for some time, with "official" releases following an eventually forthcoming publication.

Motivation

High-throughput phenotyping is resource-intensive and often demands virtual computing resources. This presents deployment challenges related to packaging and portability, and raises barriers to entry. Research software should:

• be highly configurable when necessary
• automate deployment details where possible
• let users (and developers) focus on the problem domain

Overview

PlantIT aims to bridge two user groups: researchers and developers. Of course one may wear both hats. The idea is an open-source conveyor belt for scientific software: make it easier to 1) package and share research applications, and 2) deploy them to clusters.

PlantIT is just glue between version control, data storage, container engine, and cluster scheduler. To publish an app, containerize it (e.g., write a Dockerfile) and add a plantit.yaml file to the GitHub repository. Then run it from the browser with a few clicks.

Development

Read on if you're interested in contributing to plantit or hosting your own instance somewhere.

Requirements

The following are required to develop or deploy plantit in a Unix environment:

• Docker
• npm

Installation

First, clone the repository:

git clone https://github.com/Computational-Plant-Science/plantit.git

Setting up a development environment

To set up a new (or restore a clean) development environment, run scripts/bootstrap.sh from the project root (you may need to use chmod +x first). You can use the -n option to disable the Docker build cache. This command will:

• Stop and remove project containers and networks
• If an .env file (to configure environment variables) does not exist, generate one with default values
• Build the Vue front end
• Build Docker images
• Run migrations

Then bring everything up with docker-compose -f docker-compose.dev.yml up (-d for detached mode).

This will start a number of containers:

• plantit: Django web application (http://localhost:3000)
• postgres: PostgreSQL database
• celery: Celery prefork worker
• celerye: Celery eventlet worker
• flower: Flower web UI for Celery (http://localhost:5555)
• redis: Redis instance (caching, Celery message broker)
• sandbox: Ubuntu test environment

The Django admin interface is at http://localhost:3000/admin/. To use it, you'll need to log into the site at least once (this will create a Django account for you), then shell into the plantit container, run ./manage.py shell, and update your profile with staff/superuser privileges. For instance:

from django.contrib.auth.models import User
user = User.objects.get(username=<your CyVerse username>)
user.is_staff = True
user.is_superuser = True
user.save()

You can also run ./scripts/configure-superuser.sh -u <your CyVerse username> to accomplish the same thing.

Note that the bootstrap script will not clear migrations. To restore to a totally clean database state, you will need to remove all *.py files from the plantit/plantit/migrations directory (except for __init__.py).

Running tests

Once the containers are up, tests can be run with docker-compose -f docker-compose.dev.yml exec plantit ./manage.py test.

Reverse proxying with ngrok

To test remote job submissions using a local development version of the plantit web application, you will need a service of some kind to accept and forward job completion signals to your machine's localhost. One convenient tool is ngrok. After downloading and adding the ngrok executable to your path, run ngrok http 3000 to start a tunnel, then set the DJANGO_API_URL variable in .env to the URL ngrok reports it's listening on.

Deploying to production

In production configuration, NGINX serves static assets and reverse-proxies Django via Gunicorn (both in the same container).

To configure PlantIT for deployment, first clone the repo, then, from the root directory, run:

chmod +x /scripts/deploy.sh
./scripts/deploy.sh <configuration ('rc' or 'prod')> <host IP or FQDN> <admin email address>

This script is idempotent and may safely be triggered to run by e.g., a CI/CD server. This will:

• Bring containers down
• Fetch the latest version of the project
• Pull the latest versions of Docker containers
• Build the Vue front end
• Collect static files
• Configure NGINX (replace localhost in config/ngnix/conf.d/local.conf with the host's IP or FQDN, configured via environment variable)
• Update environment variables (disable debugging, enable SSL and secure cookies, etc)
• Bring containers up
• Run migrations

At this point the following containers should be running:

• nginx: NGINX server (reverse proxy)
• plantit: Django web application behind Gunicorn (http://localhost:80)
• postgres: PostgreSQL database
• celery: Celery background worker
• redis: Redis instance

SSL Certificates

PlantIT uses Let's Encrypt and Certbot for SSL certification. The production configuration includes a certbot container which can be used to request new or renew existing certificates from Let's Encrypt. Standard certificates last 90 days.

In production the certbot container is configured by default to automatically renew certs when necessary:

certbot:
  image: certbot/certbot
  volumes:
    - ./config/certbot/conf:/etc/letsencrypt/
    - ./config/certbot/www:/var/www/certbot
  entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 24h & wait $${!}; done;'"

To manually request a new certificate, run:

docker-compose -f docker-compose.prod.yml run certbot

To renew an existing certificate, use the renew command, then restart all containers:

docker-compose -f docker-compose.prod.yml run certbot renew
docker-compose -f docker-compose.prod.yml restart

Use the --dry-run flag with any command to test without writing anything to disk.

Configuring environment variables

Docker will read environment variables in the following format from a file named .env in the project root directory (if the file exists):

key=value
key=value
...

bootstrap.sh will generate an .env file like the following if one does not exist:

VITE_TITLE=plantit
MAPBOX_TOKEN=<your Mapbox token>
MAPBOX_FEATURE_REFRESH_MINUTES=60
CYVERSE_REDIRECT_URL=http://localhost:3000/apis/v1/idp/cyverse_handle_temporary_code/
CYVERSE_CLIENT_ID=<your cyverse client id>
CYVERSE_CLIENT_SECRET=<your cyverse client secret>
CVVERSE_USERNAME=<your cyverse username>
CYVERSE_PASSWORD=<your cyverse password>
CYVERSE_TOKEN_REFRESH_MINUTES=60
NODE_ENV=development
DJANGO_SETTINGS_MODULE=plantit.settings
DJANGO_SECRET_KEY=<your django secret key>
DJANGO_DEBUG=True
DJANGO_API_URL=http://plantit:3000/apis/v1/
DJANGO_SECURE_SSL_REDIRECT=False
DJANGO_SESSION_COOKIE_SECURE=False
DJANGO_CSRF_COOKIE_SECURE=False
DJANGO_ALLOWED_HOSTS=*
DJANGO_ADMIN_USERNAME=<your django admin username>
DJANGO_ADMIN_PASSWORD=<your django admin password>
DJANGO_ADMIN_EMAIL=<your django admin email>
CELERY_EVENTLET_QUEUE=eventlet
USERS_CACHE=/code/users.json
USERS_REFRESH_MINUTES=60
USERS