CEP
Computational Endoscopy Platform (advanced deep learning toolset for analyzing endoscopy videos) [MICCAI'25, MICCAI'22, MICCAI'21, ISBI'21, CVPR'20]
Install / Use
/learn @nadeemlab/CEPREADME
Computational Endoscopy Platform (CEP) provides an exhaustive deep learning toolset to handle tasks such as haustral fold annotation (in colonoscopy videos), surface coverage visualization, depth estimation, color-lighting-texture-specular reflection augmentation, and more. All our code, AI-ready training/testing data, and pretrained models will be released here with detailed instructions along with easy-to-run docker containers and Google CoLab projects.
Internal Dataset | Public Dataset :-------------------------:|:-------------------------: <img src="imgs/internal_gifs/gifp1.gif" alt="FoldIt_Preview" width = 400 /> | <img src="imgs/public_gifs/pub_gifp1.gif" alt="FoldIt_Preview" width = 400/> <img src="imgs/internal_gifs/gifp2.gif" alt="FoldIt_Preview" width = 400 /> | <img src="imgs/public_gifs/pub_gifp2.gif" alt="FoldIt_Preview" width = 400/> <img src="imgs/internal_gifs/gifp3.gif" alt="FoldIt_Preview" width = 400 /> | <img src="imgs/public_gifs/pub_gifp3.gif" alt="FoldIt_Preview" width = 400/>
© This code is made available for non-commercial academic purposes.
Updates:
- [x] RT-GAN MICCAI'25 code released for adding lightweight temporal consistency to frame-based domain translation approaches. Please cite the following paper:
Mathew S*, Nadeem S*, Alvin C. Goh, Kaufman A. RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches. International Conference on Medical Imaging Computing and Computer Assisted Intervention (MICCAI), 2025. (* Equal Contribution) [Accepted] [Paper Link] [Supplementary Video]
- [x] CLTS-GAN MICCAI'22 code released for color-lighting-texture-specular reflection augmentation in colonoscopy video frames. Please cite the following paper:
Mathew S*, Nadeem S*, Kaufman A. CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy. International Conference on Medical Imaging Computing and Computer Assisted Intervention (MICCAI), 2022. (* Equal Contribution) [Paper Link]
-
[x] AI-ready training and testing data released. This dataset is created from public HyperKvasir optical colonoscopy videos and TCIA CT colonography repositories. Easy-to-run Docker containers and Google CoLab projects are also released.
-
[x] FoldIt MICCAI'21 code released for haustral fold detection/segmentation in colonoscopy videos. Please cite the following paper:
Mathew S*, Nadeem S*, Kaufman A. FoldIt: Haustral Folds Detection and Segmentation in Colonoscopy Videos. International Conference on Medical Imaging Computing and Computer Assisted Intervention (MICCAI), 12903, 221-230, 2021. (* Equal Contribution) [Paper Link] [Supplementary Video] [Reviews]
- [x] XDCycleGAN CVPR'20 code released for scale-consistent depth estimation for colonoscopy videos. Please cite the following paper:
Mathew S*, Nadeem S*, Kumari S, Kaufman A. Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4696-4705, 2021. (* Equal Contribution) [Paper Link] [Supplementary Video]
- [ ] For surface coverage visualization, we will release our ISBI 2021:
Mathew S*, Nadeem S*, Kaufman A. Visualizing Missing Surfaces In Colonoscopy Videos using Shared Latent Space Representations. IEEE 18th International Symposium on Biomedical Imaging (ISBI), 329-333, 2021. (* Equal Contribution) [Paper Link] [Supplementary Video]
Prerequesites
- Python 3
- CPU or NVIDIA GPU + CUDA CuDNN
Installation
To install the CEP, this repo needs to be cloned
git clone https://github.com/nadeemlab/CEP.git
cd CEP
Once the repo is cloned, the python libraries can be installed
- via pip
pip install -r requirements.txt - via conda
conda env create -f environment.yml
Docker
A dockerfile is provided as an additional way to install.
- First, Docker needs to be installed along with NVIDIA Container Toolkit for GPU support
- Build Docker Image
docker build -t cep . - Create and Run Docker Container
docker run --gpus all --name CEP -it cep
[MICCAI'25] RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches
While developing new unsupervised domain translation methods for colonoscopy (e.g. to translate between real optical and virtual/CT colonoscopy), it is thus typical to start with approaches that initially work for individual frames without temporal consistency. Once an individual-frame model has been finalized, additional contiguous frames are added with a modified deep learning architecture to train a new model from scratch for temporal consistency. This transition to temporally-consistent deep learning models, however, requires significantly more computational and memory resources for training. In this paper, we present a lightweight solution with a tunable temporal parameter, RT-GAN (Recurrent Temporal GAN), for adding temporal consistency to individual frame-based approaches that reduces training requirements by a factor of 5. We demonstrate the effectiveness of our approach on two challenging use cases in colonoscopy: haustral fold segmentation (indicative of missed surface) and realistic colonoscopy simulator video generation. We also release a first-of-its kind temporal dataset for colonoscopy for the above use cases.
To train the RT-GAN model, run the following command. During the training process, results can be viewed via visdom. By default it is on http://localhost:8097.
python3 train.py --dataroot path_to_dataset -model rtgan -name "rtgan_model_name"
To test your trained model, run the following command.
python3 test.py --dataroot path_to_dataset -model rtgan -name "rtgan_model_name"
Public Dataset and Model
Both the dataset and the models can be found here.
[MICCAI'22] CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy
Automated analysis of optical colonoscopy (OC) video frames (to assist endoscopists during OC) is challenging due to variations in color, lighting, texture, and specular reflections. Previous methods either remove some of these variations via preprocessing (making pipelines cumbersome) or add diverse training data with annotations (but expensive and time-consuming). We present CLTS-GAN, a new deep learning model that gives fine control over color, lighting, texture, and specular reflection synthesis for OC video frames. We show that adding these colonoscopy-specific augmentations to the training data can improve state-of-the-art polyp detection/segmentation methods as well as drive next generation of OC simulators for training medical students.
<p align="center"> <img src="imgs/color_lighting_preview.png" alt="Color_Lighting_Preview" width="400"/> <img src="imgs/texture_specular_preview.png" alt="Texture_Specular_Preview" width="320"/> </p> <!--  -->To train the CLTS model, run the following command. During the training process, results can be viewed via visdom. By default it is on http://localhost:8097.
python3 train.py --dataroot path_to_dataset -model cltsgan -name "cltsgan_model_name"
To run your trained model to generate texture and colors, run the following command.
python3 test.py --dataroot path_to_dataset -model cltsTest -name "clts_model_name"
The command above will create 5 images with different texture and color added to VC images. Here are some useful arugments:
--freeze_colorwill fix the color information for each iteration of the images--freeze_texturewill fix the texture information for each iteration of the images--augmentwill allow for augmentation of OC image input (image will passthrough both generator to allow which also works with the two argumnents above)
Public Dataset
We augmented a portion
