I am looking for collaborators (dev help). Please get in contact

NGSeasy: A Dockerized NGS pipeline and tool-box

NGSeasy

With NGSeasy you can now have full suite of NGS tools up and running on any high end workstation in an afternoon

We present NGSeasy (Easy Analysis of Next Generation Sequencing), a flexible and easy-to-use NGS pipeline for automated alignment, quality control, variant calling and annotation. The pipeline allows users with minimal computational/bioinformatic skills to set up and run an NGS analysis on their own samples, in less than an afternoon, on any operating system (Windows, iOS or Linux) or infrastructure (workstation, cluster or cloud).

Authors: Stephen J Newhouse and Amos Folarin
Release Version: 1.0-r001
Release: dirty_tango
Citation: Folarin AA, Dobson RJ and Newhouse SJ. NGSeasy: a next generation sequencing pipeline in Docker containers [version 1; referees: 3 approved with reservations] F1000Research 2015, 4(ISCB Comm J):997 (doi: 10.12688/f1000research.7104.1).

Acknowledgements

This work is funded by the National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London.

• Lets us know if you want other tools added to NGSeasy

NGSeasy is completely open source and we encourage interested folks to jump in and get involved in the dev with us.

Contributing to NGSEASY

• Fork it!
• Create your feature branch: git checkout -b my-new-feature
• Commit your changes: git commit -am 'Add some feature'
• Push to the branch: git push origin my-new-feature
• Submit a pull request!

NGSeasy: Genome Comparison & Analytic Testing (GCAT) Reports

Here we provide a quick look at basic NGSeasy performance (more results coming soon).

|GCAT Report|Test Data |Pipeline| |----|----|----| |NGSEASY-NTRIM-BWA-FREEBAYES-D|illumina-100bp-pe-exome-150x|fastq > bwa > freebayes| |NGSEASY-NTRIM-BWA-PLATYPUS-D|illumina-100bp-pe-exome-150x|fastq > bwa > platypus|

An example of the run commands:

ngseasy -c ngseasy_test.config.freebayes.tsv -d /media/Data/ngs_projects
ngseasy -c ngseasy_test.config.platypus.tsv  -d /media/Data/ngs_projects

Author Contact Details

Please contact us for help/guidance on using the beta release.

|Author|email|Twitter| LinkedIn| |----|----|----|----| |Dr Stephen J Newhouse| stephen.j.newhouse@gmail.com| @s_j_newhouse| View Steve's profile on LinkedIn| |Dr Amos Folarin| amosfolarin@gmail.com| @amosfolarin|View Amos's profile on LinkedIn|

Issues, Questions and Queries

Please Direct all queries to [https://github.com/KHP-Informatics/ngseasy/issues]

When sending bug reports etc please provide:-

• Date of Download
• OS and version
• Basic Machine Specs (CPU, RAM)
• Docker Version
• Network Speed (Testing Internet Connection Speed)
• The Code you ran eg:- ngseasy -c my.config.tsv -d /My/Dir
• your config file
• The exact error as printed to screen

WARNING! NGSeasy is not numpty or bad data proof!

Please read the docs, stay calm, take your time and think about what you are doing...and if [www.google.com] doesnt help, then please direct all queries to [https://github.com/KHP-Informatics/ngseasy/issues].

Install Docker

Full instructions at https://docs.docker.com/.

Some fixes to make life easy...allows you to run docker without sudo.

This may differ for your OS, and mostly applies to flavours of Linux. Check with your sys admin or just Google https://www.google.com.

MAC/Windows users using http://boot2docker.io/ should be fine. Read the docs or just Google https://www.google.com.

Create a docker group

sudo addgroup docker

Add user to docker group

Here user is ec2-user

sudo usermod -aG docker ec2-user

Log out and log back in.

This ensures your user is running with the correct permissions.

Verify your work by running docker without sudo.

docker run hello-world

..this is what you should get...

Unable to find image 'hello-world:latest' locally
Pulling repository hello-world
91c95931e552: Download complete
a8219747be10: Download complete
Status: Downloaded newer image for hello-world:latest
Hello from Docker.
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (Assuming it was not already locally available.)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

For more examples and ideas, visit:
 http://docs.docker.com/userguide/

Docker Security...

This post reviews the various security implications of using Docker to run applications within containers, and how to address them: How Secure are Containers?

Docker containers are, by default, quite secure; especially if you take care of running your processes inside the containers as non-privileged users (i.e. non root).

Get NGSeasy

#############################################
## Get NGSeasy                             ##
#############################################

cd /home/${USER}

git clone https://github.com/KHP-Informatics/ngseasy.git

Install NGSeasy

• Default install directory is /home/${USER}
• in this example user home is /home/ec2-user
• make INSTALLDIR="/home/ec2-user" all
  • sets up top level directory structure
  • gets all docker images
  • gets indexed hg19 and b37 genomes
  • gets GATK recources for hg19 and b37 genomes
  • gets whole genome and exome test data
• Always set your INSTALLDIR : If you run sudo make all the install path will be /home/root. Please dont do this!
• sudo make install installs scripts to /usr/local/bin/

#############################################
## install NGSeasy                         ##
#############################################

cd ngseasy

## 1.
make INSTALLDIR="/home/ec2-user" all

## 2.
sudo make install

Installation can take a while, 1-2 hours, so go get a coffee../just chill...if your network is bad...then who knows how long...still..just chill...or go get fast internet!

NGSeasy Security

All NGSeasy applications are run as the non-root user pipeman within each container

Recommended Network Speed

> 500 Mbit/s : anything less will add a lot of time to set up (days - weeks).

Testing Internet Connection Speed

source : http://askubuntu.com/questions/104755/how-to-check-internet-speed-via-terminal

wget -O speedtest-cli https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
chmod +x speedtest-cli
./speedtest-cli

Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Comcast Cable (x.x.x.x)...
Selecting best server based on ping...
Hosted by FiberCloud, Inc (Seattle, WA) [12.03 km]: 44.028 ms
Testing download speed........................................
Download: 32.29 Mbit/s
Testing upload speed..................................................
Upload: 5.18 Mbit/s

Install time on Amazon EC2

Connection Speed: ~ 800 Mbit/s

real    94m54.237s
user    12m26.960s
sys     28m46.648s

Note: We have only tested NGSeasy installation on Amazon EC2, Openstack and UK University Networks. These are all fairly fast networks with speeds exceeding 800 Mbit/s on average.

Running NGSeasy for the first time on the test data

Important! NGSeasy is controlled from a single config file. See ngseasy_test.config.tsv for a basic template. It is important that the user sets this up properly before running NGSeasy.

######################