HMS2

Another annotation-free whole-slide training approach to pathological classification. This repository provides scripts to reproduce the results in the paper "Deep neural network trained on gigapixel images improves lymph node metastasis detection in clinical settings", including model training, inference, visualization, and statistics calculation, etc.

>> Demo Video << | Journal Link | Our Website

Publications

Huang, SC., Chen, CC., Lan, J. et al. Deep neural network trained on gigapixel images improves lymph node metastasis detection in clinical settings. Nat Commun 13, 3347 (2022). https://doi.org/10.1038/s41467-022-30746-1

License

Requirements

Hardware Requirements

Make sure the system contains adequate amount of main memory space (minimal: 32 GB) to prevent out-of-memory error.

Software Stacks

Although Poetry can set up most Python packages automatically, you should install the following native libraries manually in advance.

CUDA 11.7+

CUDA is essential for PyTorch to enable GPU-accelerated deep neural network training. See https://docs.nvidia.com/cuda/cuda-installation-guide-linux/ .

OpenMPI 3+

OpenMPI is required for multi-GPU distributed training. If sudo is available, you can simply install this by,

sudo apt install libopenmpi-dev

Python 3.9+

The development kit should be installed.

sudo apt install python3.9-dev

OpenSlide

OpenSlide is a library to read slides. See the installation guide in https://github.com/openslide/openslide .

Python Packages

We use Poetry to manage Python packages. The environment can be automatically set up by,

cd [HMS2 folder]
python3 -m pip install poetry
python3 -m poetry install
python3 -m poetry run poe install

Usage

Before initiating a training task, you should prepare several configuration files with the following step-by-step instructions. Refer to projects/Camelyon16 as an example for training an HMS model on the CAMELYON16(https://camelyon16.grand-challenge.org/) dataset.

0. (Optional) Try a pre-trained CAMELYON16

If you would like to try training HMS models using CAMELYON16 or evaluating pre-trained ones, here we provided contour description files and pre-trained weights trained at 2.5x, 5x, and 10x magnifications, which is available at https://drive.google.com/file/d/12Fv-OhAze_t2_bCX7l1S5iMCgQgOvHGF/view?usp=sharing .

After the ZIP file is downloaded, unzip it to the project folder:

unzip -o hms2_camelyon16.zip -d /path/to/hms2

Besides, you should prepare the slides of CAMELYON16 from https://camelyon16.grand-challenge.org/ into projects/Camelyon16/slides. Then follow the instructions below.

| Pre-trained model | AUC (95% CI) | ----------------- | ---------------------------------- | Camelyon16_2.5x | 0.6015 (0.5022-0.7008) | Camelyon16_5x | 0.6242 (0.5194-0.7291) | Camelyon16_10x | 0.9135 (0.8490-0.9781)

Camelyon16_10x_hms2

1. Create a Project Folder

As a convention, create a project folder in projects with four sub-folders, datalists, slides, contours, train_configs, and test_configs.

2. Define Datasets

3 CSV files defining training, validation and testing datasets, respectively, should be placed in projects/YOUR_PROJECT/datalists. See projects/Camelyon16/datalists for example.

These CSV files should follow the format if your datasets were annotated in slide level:

[slide_1_basename],[slide_1_class]
[slide_2_basename],[slide_2_class]
...

, where [slide_name_*] specify the filename without extension of a slide image and [class_id_*] is an integer indicating a slide-level label (e.g. 0 for normal, 1 for cancerous).

Given contour-level (e.g. LN-level) labels, construct the CSV files in:

[slide_1_contour_1],[slide_1_contour_1_class]
[slide_1_contour_2],[slide_1_contour_2_class]
...

You can name each contour whatever you want.

(Optional) Contour Description Files

For each contour, a contour description file in JSON should be composed with content like:

{"slide\_name": "slide\_1\_basename", "contours": contours}

, where contours is a list of contour. Each contour is a list of coordinates in (x, y). See projects/Camelyon16/contours for example. Save these files in projects/YOUR_PROJECT/contours.

3. Prepare Slide Files

Place the slides files in projects/YOUR_PROJECT/slides. Soft links (ln -s) also work.

4. Set Up Training Configurations

Model hyper-parameters are set up in a YAML file. You can copy projects/Camelyon16/train_configs/2.5x.yaml and modify it for your own preference.

The following table describes each field in a config.

| Field | Description | -------------------------- | --------------------------------------------------------------------------------------------- | RESULT_DIR | Directory to store output stuffs, including model weights, testing results, etc. | MODEL_PATH | Path to store the model weight. (default: ${RESULT_DIR}/model.pt) | OPTIMIZER_STATE_PATH | Path to store the state of optimizer. (default: ${RESULT_DIR}/opt_state.pt) | HISTORY_DIR | Directory to store model history. (default: ${RESULT_DIR}/history/) | STATES_PATH | Path to store the states for resuming. (default: ${RESULT_DIR}/states.pt) | CONFIG_RECORD_PATH | Path to back up this config file. (default: ${RESULT_DIR}/config.yaml) | TRAIN_EVENT_LOG_PATH | Path to store the training event log. (default: ${RESULT_DIR}/train_event_log.json) | | | USE_MIXED_PRECISION | Whether to enable mixed precision training. (default: False) | USE_HMS2 | Whether to enable HMS2. (default: True) | GPU_MEMORY_LIMIT_GB | GPU memory limitation in GB. (default: None) | | | RESIZE_RATIO | Resize ratio for downsampling slide images. Can be a float (e.g., 0.25) or a dictionary containing target_pixel_spacing to specify the targeted pixel spacing in um (e.g., {target_pixel_spacing: 0.92}). | GPU_AUGMENTS | Augmentations to do on GPU with patch-based affine transformation. Available options are "flip", "rigid", "hed_perturb", and "gaussian_blur". (default: ["flip", "rigid", "hed_perturb"]) | AUGMENTS | Augmentations to do on CPU. (default: []) | CLASS_WEIGHTS | A list of dictionaries or None to disable class weighting. Each dictionary contains class_index, positivity, and weight. (default: None) | | | NUM_CLASSES | Number of classes. | GPUS | The number of GPUs specified for assertion. (default: None) | MODEL | Model architecture to use. Options: "resnet50_frozenbn", "resnet50_frozenbn_linear", "resnet50_frozenall", "resnet50_frozenall_linear", "resnet50_frozenall_ap_linear", "resnet50V1c_frozenbn". (default: "resnet50_frozenbn") | PRETRAINED | Pretrained weight to initialize the model. Can be no pretrained ({type: "no"}), torchvision weights (e.g., {type: "torchvision", weights: "IMAGENET1K_V1"}), backbone weights (e.g., {type: "backbone", path: "/path/to/weight.pt"}), or HMS2 weights (e.g., {type: "hms2", path: "/path/to/weight.pt"}). default: {type: "torchvision", weights: "IMAGENET1K_V1"}. | PRE_POOLING | Pre-pooling layer to use. Options: "no", "conv_1x1", "conv_1x1_relu". (default: "no") | POOL_USE | Global pooling method. (default: "re_lse") | BATCH_SIZE | Number of slides processed in each training iteration for each MPI worker. (default: 1) | EPOCHS | Maximal number of training epochs. (default: 200) | LOSS | Loss to use with the format {"name": str, "arguments": dict[str, Any]}. (default: {"name": "ce"}) | METRIC_LIST | A list of metrics. (default: ["accuracy"]) | OPTIMIZER | Optimizer for model updating. (default: "adamw") | INIT_LEARNING_RATE | Initial learning rate for Adam optimizer. (default: 0.00001) | REDUCE_LR_FACTOR | The learning rate will be decreased by this factor upon no validation loss improvement in consequent epochs. Set 0.0 to enable early stopping. (default: 0.1) | REDUCE_LR_PATIENCE | Number of consequent epochs to reduce learning rate. (default: 8) | | | TRAIN_DATASET_CONFIGS | A list of training subdatasets (see below). | | | CLASS_NAMES | Class names for display during evaluation. Set None (default) to ignore the setting. | | | USE_EMBED_MODE | Whether to enable embed mode. (default: False) | EMBED_MODE_CACHE_DIR | Directory to store the embedding cache in embed mode. (default: ${RESULT_DIR}/embedding) | EMBED_MODE_GPU_COMPRESSOR | The compressor running on GPU to compress the embeddings in embed mode. Options: "identity", "avg_pool_7x7", "farthest_point_1/7". (default: "farthest_point_1/7"). Be aware that "avg_pool_7x7" is a deprecated method and may cause the inconsistent performance issue (i.e. gain good performance with embed mode but poor performance with standard mode) in some cases, so we recommend using "farthest_point_1/7" instead. | EMBED_MODE_COMPRESSORS | Compressors to use in embed mode. (default: `["int8", "uniq

Hms2

Install / Use

README