AmpliconRepository.org

Authors: Forrest Kim, Edwin Huang, Ted Liefeld, Jens Luebeck, Thorin Tabor, Michael Chan, Dhruv Khatri, Kyra Fetter, Gino Prasad, Rohil Ahuja, Rishaan Kenkre, Tushar Agashe, Devika Torvi, Madalina Giurgiu, Vineet Bafna

This is the main repository for the AmpliconRepository website. The documentation below provides intsructions on deploying the site locally, for development purposes.

How to install the development environment for AmpliconRepository
How to set up your development environment using docker compose
Testing datasets
Pushing changes to GitHub and merging PRs
Using the development server
Logging in as admin
Running the automated tests
How to deploy and update the production server for AmpliconRepository

There are two options for running the server locally:

Option A: Manually install modules and configure the environment step-by-step.

Option B: Use Docker to deploy the server and its environment on your system.

Option A - install the development environment for AmpliconRepository: <a name="aa-env-install"></a>

Requirements

Python Virtual Environment (3.8 or higher)

1. Clone the repository from GitHub

Clone repo using https, ssh, or GitHub Desktop to your local machine

2. Set up the virtual environment and install packages:

In a terminal window, move to the cloned GitHub repo
Go to the AmpliconRepository top level directory (should see requirements.txt)

Option A: Using python's environment manager

Create a new Python virtual environment:

python -m venv ampliconenv

Activate the new environment (you need to do this everytime before running the server):

source ampliconenv/bin/activate

Install required packages

pip install -r requirements.txt

Option B: Using conda's environment manager

Create a new Conda environment

conda create -n ampliconenv "python>=3.8,<3.13"

To activate

conda activate ampliconenv

Install pip to that environment

conda install pip -n ampliconenv

Install required packages

~/[anaconda3/miniconda3]/envs/ampliconenv/bin/pip install -r requirements.txt

3. Set up MongoDB locally (for development)

Install MongoDB
- In Ubuntu this can be done with sudo apt install mongodb-server-core
  - For newer versions of Ubuntu (e.g. 22.04+), follow the instructions here: https://www.fosstechnix.com/how-to-install-mongodb-on-ubuntu-22-04-lts/
- In macOS this can be done with
  
  git config --global init.defaultBranch main
  
  brew tap mongodb/brew
  
  brew install mongodb-community@6.0
- If the package is not found you may need to follow the directions here.
If you don't have a database location set up, set up a location:

mkdir -p ~/data/db/

In a terminal window or tab with the ampliconenv environment active, run MongoDB locally:

mongod --dbpath ~/data/db or mongod --dbpath <DB_PATH>

3a. View MongoDB data in MongoDB Compass

Download MongoDB Compass: https://www.mongodb.com/docs/compass/current/install/#download-and-install-compass
Open the MongoDB Compass app after starting MongoDB locally
Connect to your local instance of MongoDB:

URI: mongodb://localhost:27017

Relevant data will be located in /AmpliconRepository/projects/
You can periodically clear your local deployment mongodb files using Compass so that your disk does not fill up.
Run export DB_URI_SECRET='mongodb://localhost:27017' in your terminal to set the environment variable for your local database.
- So that this is active every time, you can add the command above to your ~/.bashrc file
Note that the latest version of Compass (1.34.2) won't work with our older DB version. You can get an old compass for mac at https://downloads.mongodb.com/compass/mongodb-compass-1.28.4-darwin-x64.dmg

3b. Clearing your local DB

Periodically, you will want to purge old or excessively large accumulated data from you DB. You can do this using the provided script

python purge-local-db.py

4. Neo4j Download Instructions

Docker

the easiest way... edit the path at the end to the local drive you want it to use

docker run -d --name neo4j -p 7474:7474 -p 7687:7687 --env NEO4J_AUTH=neo4j/$NEO4J_PASSWORD_SECRET -v /home/ubuntu/AmpliconRepository-dev/neo4j neo4j

macOS

Download and unzip the tar file: curl -O -C - http://dist.neo4j.org/neo4j-community-5.12.0-unix.tar.gz tar -xvzf neo4j-community-5.12.0-unix.tar.gz

Start neo4j with the console command: cd neo4j-community-5.12.0 bin/neo4j console

Go to http://localhost:7474/browser/ and change the auth settings. By default, both user and password are 'neo4j'. Keep user as 'neo4j' and change password to 'password'.

The environment is now set up. Ensure that neo4j is running before querying the graph.

Alternatively, go to https://neo4j.com/deployment-center/, then download the rpm file for the latest Community Edition under the section titled 'Graph Database Self-Managed'. Further instructions are available upon clicking Download. Note that this method has not been tested by our team.

Ubuntu (or Windows via WSL/WSL2)

Please follow this documentation to set up the latest version of Neo4j Community Edition

In brief, you can do

wget -O - https://debian.neo4j.com/neotechnology.gpg.key | sudo apt-key add -
echo 'deb https://debian.neo4j.com stable latest' | sudo tee /etc/apt/sources.list.d/neo4j.list
sudo apt-get update
sudo apt-get install neo4j

Then launch it by running

sudo neo4j start

Visit http://localhost:7474 and login with neo4j as both the user and password. You will be prompted to set a password for future use. You must set the updated password to the value in your config.sh file (value of NEO4J_PASSWORD_SECRET)

For shutdown at the end of your session, you can do sudo neo4j stop

5. Set up secret keys for OAuth2 and other environment variables

Make sure you have the config.sh file from another developer (this contains secret key information)
Run the command to initialize variables: source config.sh

For local deployments, you will need to ensure that the following two variables are set to FALSE, as shown below

export S3_STATIC_FILES=FALSE
export S3_FILE_DOWNLOADS='FALSE'

IMPORTANT: After recieving your config.sh, please ensure you do not upload it to Github or make it available publicly anywhere.

6. Run development server (Django)

Open a terminal window or tab with the ampliconenv environment active
Move to the caper/ folder (should see manage.py)
Run the server locally:

python manage.py runserver

Open the application on a web browser (recommend using a private/incognito window for faster development):

https://localhost:8000

Troublshooting tip: If you face an error that says port 8000 is already in use, you can kill all tasks using that port by doing sudo fuser -k 8000/tcp

Option B - Local deployment with Docker: <a name="dev-docker"></a>

These steps guide users on how to set up their development environment using Docker and docker compose as an alternative to python or conda-based package management and installation. This is the simplest way to locally deploy the server for new users.

Important: You first need to install docker>=20.10 on your machine.

To test the installation of Docker please do:

# check version: e.g. Docker version 20.10.8, build 3967b7d
docker --version

# check if compose module is present
docker compose --help

# check docker engine installation
sudo docker run hello-world

Quickstart

Build and run your development webserver and mongo db using docker:

cd AmpliconRepository
# place config.sh in caper/, and place .env in current dir
# change UID and GID in .env to match the host configuration
# create all folders which you want to expose to the container
mkdir -p logs tmp .aws .git
docker compose -f docker-compose-dev.yml build --no-cache --progress=plain
docker compose -f docker-compose-dev.yml up -d
# then visit http://localhost:8000/ in your web browser
# once finished, to shutdown:
docker compose -f docker-compose-dev.yml down

Complete steps

i. Start your docker daemon and make sure is running:

# for linux
sudo systemctl start docker
docker --help
docker compose --help

# or alternatively start manually from interface (macos or windows)

ii. Clone the repository (skip this if you have already done this):

git clone https://github.com/AmpliconSuite/AmpliconRepository.git

iii. Build a local Docker image:

This command will create a Docker image genepattern/amplicon-repo:dev-test with your environment, all dependencies and application code you need to run the webserver. Additionally, this command will pull a mongo:4 image for the test database.

First, obtain the secret files .env and config.sh from another developer. Do not share these files with others outside the project. Do not upload them anywhere. Keep them private.

Next, Place .env under AmpliconRepository/ and config.sh under AmpliconRepository/caper/.

You should see these required files:

docker-compose-dev.yml
Dockerfile
.env
requirements.txt
caper/config.sh

cd AmpliconRepository
docker compose -f docker-compose-dev.yml build --progress=plain --no-cache

AmpliconRepository

Install / Use

README