MetaTOR

<!-- [](https://metator.readthedocs.io) -->

Metagenomic Tridimensional Organisation-based Reassembly - A set of scripts that streamlines the processing and binning of metagenomic metaHiC datasets.

<p align="center"> <img src="docs/example/images/metator_logo.png" width="200"> </p>

1. Installation
   1. With conda
   2. With pip
   3. Development version
2. Usage
3. Output files
4. Contributions
5. References
6. Contact
   1. Authors
   2. Research lab

Installation

With conda

conda is the recommended way to install the latest metator release:

conda create -n metator bioconda::metator

With pip

Note: although metator itself is available from Pypi, several additional libraries are not available from Pypi and must be installed separately. Please consider the following before installing metator:

• Python 3.9 to 3.11 is required.

• The following dependencies should also be locally installed and available in the $PATH:

  • bowtie2 or bwa
  • samtools
  • hmmer
  • prodigal
  • java

• The following non-pythonic librairies are embedded when installing metator with pip: louvain 0.3 and leiden 1.3.0.

Warning: note that if you install metator with pip, this may upgrade/downgrade versions of the above dependencies and may cause conflicts with other softwares.

# Install bowtie2, samtools, hmmer, prodigal and java-jdk:
sudo apt update && sudo apt install bowtie2 samtools hmmer prodigal default-jdk

# Install metator from Pypi
pip3 install metator

Development version

You can use metator development version as follows:

git clone https://github.com/koszullab/metator
cd metator
pip3 install -e .[dev]

Usage

metator {network|partition|validation|pipeline} [parameters]

A metator command takes the form metator action --param1 arg1 --param2 arg2 #etc.

There are three actions/steps in the metator pipeline, which must be run in the following order:

• network : Generate metaHiC contigs network from fastq reads or bam files and normalize it.

• partition : Perform the Louvain or Leiden community detection algorithm many times to bin contigs according to the metaHiC signal between contigs.

• validation : Use CheckM to validate the bins, then do a recursive decontamination step to remove contamination.

There are a number of other, optional, miscellaneous actions:

• pipeline : Run all three of the above actions sequentially or only some of them depending on the arguments given. This can take a while.

• contactmap : Generates a contact map from one bin from the final ouptut of metator.

• version : display current version number.

• help : display help message.

A tutorial is available here to explain how to use metator. More advanced tutorials to analyze the output files are also available:

• Anvio manual curation of the contaminated bins. Available here.
• Visualization and scaffolding of the MAGs with the contactmap modules of MetaTOR. Available here.

Principle of MetaTOR pipeline:

Output files

The output files will be in the ouput directory given as parmater or in working directory if no paramater were given. Depending on the command used, different files will be in the ouptut:

| Files/Commands | description | network | partition | validation | pipeline | | - | :-: | :-: | :-: | :-: | :-: | | alignment_N_for.bam |Bam file of the forward alignment |X|||X| | alignment_N_rev.bam |Bam file of the reverse alignment|X|||X| | alignment_N.pairs |Pairs of the merge alignment|X|||X| | network.txt |Normalized network of the metaHiC library|X|||X| | contig_data_network.txt |Information on contigs after network step|X|||| | clustering_matrix_partition.txt |Matrix of clustering from the partition iterations||X||| | contig_data_partition.txt |Information on contigs after partition step||X||| | overlapping_checkm_results.txt |CheckM results summary file from the partition step|||X|X| | overlapping_checkm_taxonomy.txt |CheckM taxonomy file from the partition step|||X|X| | recursive_checkm_results.txt |CheckM results summary file from the recursive step|||X|X| | recursive_checkm_taxonomy.txt |CheckM taxonomy file from the recursive step|||X|X| | clustering_matrix_validation.txt |Matrix of clustering from the recursive iterations|||X|| | clustering_matrix.txt |Matrix of clustering from the partition and recursive iterations||||X| | contig_data_final.txt |Information on contigs after whole pipeline|||X|X| | bin_summary.txt |Information on the final bins|||X|X| | binning.txt |File with contigs names and their final clustering|||X|X| | overlapping_bin |Directory with the fasta of the partition bins||X|X|X| | recursive_bin |Directory with the fasta of the recursive bins|||X|X| | final_bin |Directory with the fasta of the final bins|||X|X|

Bam alignment files For the bam alignments files, only the aligned reads are kept and the bam are sorted by name. The N value correspond to the id (order of the given fastq started at 0)

Pairs aligment files This format is used to store the relevant information of mapping of the merged alignment. It's a s The N value correspond to the id (order of the given fastq started at 0). It is a tab-separated format holding informations about Hi-C pairs. It has an official specification defined by the 4D Nucleome data coordination and integration center. Here we kept 7 columns readID-chr1-pos1-chr2-pos2-strand1-strand2.

Network file This is a tsv file of the network with edgelist form: Id of the first contig, id of the second contig and the weigth of edge normalized.

Contig data files These are the files with all the informations from the contigs: |ID|Name|Size|GC_content|Hit|Shotgun_coverage|Restriction_site|Core_bin_ID|Core_bin_contigs|Core_bin_size|Overlapping_bin_ID|Overlapping_bin_contigs|Overlapping_bin_size|Recursive_bin_ID|Recursive_bin_contigs|Recursive_bin_size|Final_bin| |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| |1|NODE_1|642311|38.6876450815882|3837|41.1565|2006|1|65|2175226|1|396|6322353|1|52|2158803|MetaTOR_1_1| |2|NODE_2|576356|30.235826468363303|1724|24.509|1256|2|40|1735419|2|401|735419|0|-|-|MetaTOR_2_0| |3|NODE_3|540571|42.305266098255366|2188|14.5855|3405|3|127|6409484|3|431|13615480|1|112|6385126|MetaTOR_3_1|

They have to have the header when they are use as input but the order of the columns nd if they are others columns doesn't matter when they are used as input files. Depending on which step of the pipeline have been launch they just have some of these columns:

• contig_data_network.txt: columns: ID, Name, Size, GC content, Hit, Shotgun_coverage and Restriction Site only.
• contig_data_partition.txt: The same as the previous with the information of core bins and overlapping bins.
• contig_data_final.txt: All the columns.

The shotgun coverage will be filled only if the depth.txt file is given, otherwise it will be filled with -. This column is only necessarry for the abundance and the theoritical_hit normalization. The restriction will also be filled with - if no enzyme are given. This column is only necessary for the RS and the theoritical_hit normalization. Moreover, if the contig is not binned (no HiC reads mapped on it) all the columns with binning information will be filled with -, and if a bin is not recursively decontamined because it's smaller than the size threshold or it doesn't have any contamination the recusive bin information will be filled 0, -, -. Finally, if the bin is not in a final bin, it will be annotated ND in the last column (for not determined).

clustering matrix files The clustering matrix files are at the .npz format which is a compresed foramt for sparsed matrix. This sparsed matrix contains the ratio of time each pairs of contigs are clusterize together by the algorithm of clustering (either Louvain or Leiden). The partition matrix contains the information for the partition step, the recursive one for the recursive step and the general one is the mean of both matrices. Be careful the index of the contigs are zero-based and not one-based as in the contig data file.

It's possible to read them in python using the scipy.sparse.load_npz() function. If the users wants a tsv file instead, he or she could load the matrix in python using loa