Survival Integration of Multi-omics using Deep-Learning (DeepProg)

This package allows to combine multi-omics data together with survival. Using autoencoders, the pipeline creates new features and identify those linked with survival, using CoxPH regression. The omic data used in the original study are RNA-Seq, MiR and Methylation. However, this approach can be extended to any combination of omic data.

The current package contains the omic data used in the study and a copy of the model computed. However, it is very easy to recreate a new model from scratch using any combination of omic data. The omic data and the survival files should be in tsv (Tabular Separated Values) format and examples are provided. The deep-learning framework uses Keras, which is a embedding of Theano / tensorflow/ CNTK.

A more complete documentation with API description is also available at https://deepprog-garmires-lab.readthedocs.io/en/latest/index.html/

Documentation section

• Installation
• Tutorial: Simple DeepProg model
• Tutorial: Ensemble of DeepProg model
• Tutorial: Advanced usage of DeepProg model
• Tutorial: use DeepProg from the docker image
• Case study: Analyzing TCGA HCC dataset
• Tutorial: Tuning DeepProg

Requirements

• Python 2 or 3 (Python3 is recommended)
• Either theano, tensorflow or CNTK (tensorflow is recommended)
• theano (the used version for the manuscript was 0.8.2)
• tensorflow as a more robust alternative to theano
• cntk CNTK is anoter DL library that can present some advantages compared to tensorflow or theano. See https://docs.microsoft.com/en-us/cognitive-toolkit/
• scikit-learn (>=0.18)
• numpy, scipy
• lifelines
• (if using python3) scikit-survival
• (For distributed computing) ray (ray >= 0.8.4) framework
• (For hyperparameter tuning) scikit-optimize

pip3 install tensorflow

# Alternative to tensorflow, original backend used
pip3 install theano

#If you want to use theano or CNTK
nano ~/.keras/keras.json

Tested python package versions

Python 3.8 (tested for Linux and OSX. For Windows Visual C++ is required and LongPathsEnabled shoud be set to 1 in windows registry)

• tensorflow == 2.4.1 (2.4.1 currently doesn't seem to work with python3.9)
• keras == 2.4.3
• ray == 0.8.4
• scikit-learn == 0.23.2
• scikit-survival == 0.14.0 (currently doesn't seem to work with python3.9)
• lifelines == 0.25.5
• scikit-optimize == 0.8.1 (currently doesn't seem to work with python3.9)
• mpld3 == 0.5.1 Since ray and tensorflow are rapidly evolving libraries, newest versions might unfortunatly break DeepProg's API. To avoid any dependencies issues, we recommand working inside a Python 3 virtual environement (virtualenv) and install the tested packages.

installation (local)

# The downloading can take few minutes due to the size of th git project
git clone https://github.com/lanagarmire/DeepProg.git
cd DeepProg

# install with conda
conda env create -n deepprog -f ./environment.yml python=3.8
conda activate deepprog
pip install -e . -r requirements_tested.txt

# (RECOMMENDED) to install the tested python library versions
pip install -e . -r requirements_tested.txt

##Alternative installations

# Basic installation
pip3 install -e . -r requirements.txt
# To intall the distributed frameworks
pip3 install -r requirements_distributed.txt
# Installing scikit-survival (python3 only)
pip3 install -r requirements_pip3.txt

# DeepProg is working also with python2/pip2 however there is no support for scikit-survival in python2
pip2 install -e . -r requirements.txt
pip2 install -e . -r requirements_distributed.txt

# Install ALL required dependencies with the most up to date packages
pip install -e . -r requirements_all.txt

Installation with docker

We have created a docker image (opoirion/deepprog_docker:v1) with all the dependencies already installed. For the docker (and singularity) instruction, please refer to the docker tutorial (see above).

Support for CNTK / tensorflow

• We originally used Keras with theano as backend plateform. However, Tensorflow (currently used as default) or CNTK are more recent DL framework that can be faster or more stable than theano. Because keras supports these 3 backends, it is possible to use them as alternative to theano. To change backend, please configure the $HOME/.keras/keras.json file. (See official instruction here).

The default configuration file looks like this:

{
    "image_data_format": "channels_last",
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "tensorflow"
}

Distributed computation

• It is possible to use the python ray framework https://github.com/ray-project/ray to control the parallel computation of the multiple models. To use this framework, it is required to install it: pip install ray
• Alternatively, it is also possible to create the model one by one without the need of the ray framework

Visualisation module (Experimental)

• To visualise test sets projected into the multi-omic survival space, it is required to install mpld3 module: pip install mpld3
• Note that the pip version of mpld3 installed on my computer presented a bug: TypeError: array([1.]) is not JSON serializable . However, the newest version of the mpld3 available from the github solved this issue. It is therefore recommended to install the newest version to avoid this issue.

Usage

• test if simdeep is functional (all the software are correctly installed):

  python3 test/test_simdeep.py -v

  # Individual examples
  python3 python examples/example_with_dummy_data.py
  python3 python examples/example_with_dummy_data_distributed.py
  python3 python examples/example_with_precomputed_labels.py
  python3 python examples/example_hyperparameters_tuning.py
  python3 python examples/example_hyperparameters_tuning_with_test_dataset.py

• All the default parameters are defined in the config file: ./simdeep/config.py but can be passed dynamically. Three types of parameters must be defined:
  • The training dataset (omics + survival input files)
    • In addition, the parameters of the test set, i.e. the omic dataset and the survival file
  • The parameters of the autoencoder (the default parameters works but it might be fine-tuned.
  • The parameters of the classification procedures (default are still good)

Example datasets and scripts

An omic .tsv file must have this format:

head mir_dummy.tsv

Samples        dummy_mir_0     dummy_mir_1     dummy_mir_2     dummy_mir_3 ...
sample_test_0  0.469656032287  0.347987447237  0.706633335508  0.440068758445 ...
sample_test_1  0.0453108219657 0.0234642968791 0.593393816691  0.981872970341 ...
sample_test_2  0.908784043793  0.854397550009  0.575879144667  0.553333958713 ...
...

a survival file must have this format:

head survival_dummy.tsv

Samples        days event
sample_test_0  134  1
sample_test_1  291  0
sample_test_2  125  1
sample_test_3  43   0
...

As examples, we included two datasets:

• A dummy example dataset in the example/data/ folder:

examples
├── data
│   ├── meth_dummy.tsv
│   ├── mir_dummy.tsv
│   ├── rna_dummy.tsv
│   ├── rna_test_dummy.tsv
│   ├── survival_dummy.tsv
│   └── survival_test_dummy.tsv

• And a real dataset in the data folder. This dataset derives from the TCGA HCC cancer dataset. This dataset needs to be decompressed before processing:

data
├── meth.tsv.gz
├── mir.tsv.gz
├── rna.tsv.gz
└── survival.tsv

Creating a simple DeepProg model with one autoencoder for each omic

First, we will build a model using the example dataset from ./examples/data/ (These example files are set as default in the config.py file). We will use them to show how to construct a single DeepProg model inferring a autoencoder for each omic

# SimDeep class can be used to build one model with one autoencoder for each omic
from simdeep.simdeep_analysis import SimDeep
from simdeep.extract_data import LoadData

help(SimDeep) # to see all the functions
help(LoadData) # to see all the functions related to loading datasets

# Defining training datasets
from simdeep.config import TRAINING_TSV
from simdeep.config import SURVIVAL_TSV

dataset = LoadData(training_tsv=TRAINING_TSV, survival_tsv=SURVIVAL_TSV)

simDeep = SimDeep(dataset=dataset) # instantiate the model with the dummy example training dataset defined in the config file
simDeep.load_training_dataset() # load the training dataset
simDeep.fit() # fit the model

# Defining test datasets
from simdeep.config import TEST_TSV
from simdeep.config import SURVIVAL_TSV_TEST

simDeep.load_new_test_dataset(TEST_TSV, fname_key='dummy', path_survival_file=SURVIVAL_TSV_TEST)

# The test set is a dummy rna expression (generated randomly)
print(simDeep.dataset.test_tsv) # Defined in the config file
# The data type of th