Topaz

A pipeline for particle detection in cryo-electron microscopy images using convolutional neural networks trained from positive and unlabeled examples. Topaz also includes methods for micrograph and tomogram denoising using deep denoising models.

Check out our Discussion section for general help, suggestions, and tips on using Topaz. You can also find our documentation site here.

New in v0.3.0

• Major refactoring of logic out of commands for modularity
• Refactored datasets and sampling allow larger training sets with mixed micrograph sizes
• Particle extraction can now be performed in patches (thanks to @jake4844 for help with this)
• Topaz train now additionally reports adjusted_precision, as a proportion of the theoretical max precision (pi)
• Can now read long argument lists using "@list_of_many_files.txt" in place of filenames or wildcards (thanks to #192 from @nfrasser)
• 3D particle-picking is in development on the dev-topaz3d branch
• Various bug fixes

New in v0.2.5

• Added Relion integration scripts
• Topaz extract can now write particle coordinates to one file per input micrograph
• Added Gaussian filter option for after 3D denoising
• Added info on Topaz Workshops
• Topaz GUI update
• Various bug fixes

New in v0.2.4

• Added 3D denoising with topaz denoise3d and two pretrained 3D denoising models
• Added argument for setting number of threads to multithreaded commands
• Topaz GUI update
• Various bug fixes

New in v0.2.3

• Improvements to the pretrained denoising models
• Topaz now includes pretrained particle picking models
• Updated tutorials
• Updated GUI to include denoising commands
• Denoising paper preprint is available here

New in v0.2.2

• The Topaz publication is out here
• Bug fixes and GUI update

New in v0.2.0

• Topaz now supports the newest versions of pytorch (>= 1.0.0). If you have pytorch installed for an older version of topaz, it will need to be upgraded. See installation instructions for details.
• Added topaz denoise, a command for denoising micrographs using neural networks.
• Usability improvements to the GUI.

Prerequisites

• An Nvidia GPU with CUDA support for GPU acceleration.

• Basic Unix/Linux knowledge.

Installation

<details><summary>(Recommended) Click here to install using Anaconda</summary><p>

If you do not have the Anaconda python distribution, please install it following the instructions on their website.

We strongly recommend installing Topaz into a separate conda environment. To create a conda environment for Topaz:

conda create -n topaz python=3.8 (3.8-3.12 are currently supported)
source activate topaz # this changes to the topaz conda environment, 'conda activate topaz' can be used with anaconda >= 4.4 if properly configured
# source deactivate # returns to the base conda environment

More information on conda environments can be found here.

Install Topaz

To install the precompiled Topaz package and its dependencies, including pytorch:

conda install topaz -c tbepler -c pytorch

This installs pytorch from the official channel. To install pytorch for specific cuda versions, you will need to add the 'pytorch-cuda=X.X' package. E.g. to install for CUDA 11.8:

conda install pytorch-cuda=11.8 -c pytorch -c nvidia

or combined into a single command:

conda install topaz pytorch-cuda=11.8 -c tbepler -c pytorch -c nvidia

See here for additional pytorch installation instructions.

That's it! Topaz is now installed in your anaconda environment.

</p></details>

<details><summary>Click here to install using Pip</summary><p>

We strongly recommend installing Topaz into a virtual environment. See installation instructions and user guide for virtualenv.

Install Topaz

To install Topaz for Python 3.X

pip3 install topaz-em

See here for additional pytorch installation instructions, including how to install pytorch for specific CUDA versions.

That's it! Topaz is now installed through pip.

</p></details>

<details><summary>Click here to install using Docker</summary><p>

<details><summary>Do you have Docker installed? If not, click here</summary><p>

Linux/MacOS    (command line)

Download and install Docker 1.21 or greater for Linux or MacOS.

Consider using a Docker 'convenience script' to install (search on your OS's Docker installation webpage).

Launch docker according to your Docker engine's instructions, typically docker start.

Note: You must have sudo or root access to install Docker. If you do not wish to run Docker as sudo/root, you need to configure user groups as described here: https://docs.docker.com/install/linux/linux-postinstall/

Windows    (GUI & command line)

Download and install Docker Toolbox for Windows.

Launch Kitematic.

If on first startup Kitematic displays a red error suggesting that you run using VirtualBox, do so.

Note: Docker Toolbox for MacOS has not yet been tested.

What is Docker?

This tutorial explains why Docker is useful.

</p></details> <br/>

A Dockerfile is provided to build images with CUDA support. Build from the github repo:

docker build -t topaz https://github.com/tbepler/topaz.git

or download the source code and build from the source directory

git clone https://github.com/tbepler/topaz
cd topaz
docker build -t topaz .

</p></details>

<details><summary>Click here to install using Singularity</summary><p>

A prebuilt Singularity image for Topaz is available here and can be installed with:

singularity pull shub://nysbc/topaz

Then, you can run topaz from within the singularity image with (paths must be changed appropriately):

singularity exec --nv -B /mounted_path:/mounted_path /path/to/singularity/container/topaz_latest.sif /usr/local/conda/bin/topaz

</p></details>

<details><summary>Click here to install from source</summary><p>

Recommended: install Topaz into a virtual Python environment
See https://conda.io/docs/user-guide/tasks/manage-environments.html or https://virtualenv.pypa.io/en/stable/ for setting one up.

Install the dependencies

Tested with python 3.8-3.12

• pytorch (>= 1.0.0)
• torchvision
• tqdm (>= 4.65.0)
• h5py (>= 3.7.0)
• pillow (>= 6.2.0)
• numpy (>= 1.11)
• pandas (>= 0.20.3)
• scipy (>= 0.19.1)
• scikit-learn (>= 0.19.0)

Easy installation of dependencies with conda

conda install numpy pandas scikit-learn h5py tqdm
conda install -c pytorch pytorch torchvision

For more info on installing pytorch for your CUDA version see https://pytorch.org/get-started/locally/

Download the source code

git clone https://github.com/tbepler/topaz

Install Topaz

Move to the source code directory

cd topaz

By default, this will be the most recent version of the topaz source code. To install a specific older version, checkout that commit. For example, for v0.1.0 of Topaz:

git checkout v0.1.0

Note that older Topaz versions may have different dependencies. Refer to the README for the specific Topaz version.

Install Topaz into your Python path including the topaz command line interface

pip install .

To install for development use

pip install -e .

</p></details>

Topaz is also available through SBGrid.

Tutorial

The tutorials are presented in Jupyter notebooks. Please install Jupyter following the instructions here.

1. Quick start guide
2. Complete walkthrough
3. Cross validation
4. Micrograph denoising

The tutorial data can be downloaded here.

To run the tutorial steps on your own system, you will need to install Jupyter and matplotlib which is used for visualization.

With Anaconda this can be done with:

conda install jupyter matplotlib

If you installed Topaz using anaconda, make sure these are installed into your Topaz evironment.

User guide

<details><summary>Click here for a description of the Topaz pipeline and its commands</summary><p>

The command line interface is structured as a single entry command (topaz) with different steps defined as subcommands. A general usage guide is provided below with brief instructions for the most important subcommands in the particle picking pipeline.

To