VR-Caps: A Virtual Environment for Capsule Endoscopy

Overview

We introduce a virtual active capsule endoscopy environment developed in Unity that provides a simulation platform to generate synthetic data as well as a test bed to develop and test algorithms. Using that environment, we perform various evaluations for common robotics and computer vision tasks of active capsule endoscopy such as classification, pose and depth estimation, area coverage, autonomous navigation, learning control of endoscopic capsule robot with magnetic field inside GI-tract organs, super-resolution, etc. The demonstration of our virtual environment is available on YouTube.

Our main contributions are as follows:

We propose synthetic data generating tool for creating fully labeled data.
Using our simulation environment, we provide a platform for testing numerous highly realistic scenarios.

See Summary of our work for details and our Paper.

Getting Started

1. Installation

The VR-Caps contains several components:

Unity
ML-Agents
SOFA
MagnetoDynamics
SC-SfMLearner

Consequently, to install and use the VR-Caps you will need to:

Download and Install Unity(2018.4 or later)
Install Python(3.6.1 or higher)
Install ML-Agents(Release 1 or higher)
Install SofaAPAPI-Unity3D

Clone the VR-Caps Repository

Now that you have installed Unity and Python, you can now clone this repository.

git clone https://github.com/CapsuleEndoscope/VirtualCapsuleEndoscopy.git

Now, you will need open Unity Hub and simply create a new Unity project by adding VR-Caps-Unity. Then, simply open the Unity by clicking on the project you just added. Please note that we have tested the environment on Unity Version 2019.3.3f1.

The opening scene Clinic Setup is our default scene. You can navigate other scenes by Scenes.

2. Creating Synthetic Data

For data creation tool, please open Record Collect scene from Scenes.

This will open a scene where there is one of our GI system models is already placed and a capsule with a mono camera and a light source is attached on it.

You will need Unity Recorder which can be installed using Unity Package Manager (see image)

After installing Unity Recorder, navigate to Recorder Window and open the Recorder panel.

On the panel, click Add New Recorders and then select Image Sequence and AOV Image Sequence for RGB image recording and depth recording respectively.

Adjust image resolutions from Capture Output Resolution and FPS(Frame per second) from Target Value.

Importing new models to the scene<img align="right" src="img/Hierarchy.png" width="200">

You can import other models from GI-Organs folder by simply dragging the model to the scene. You will notice that the imported model has no texture.

In order to add texture, you need to navigate Organs folder and simply drag material files (.mat) to corresponding 3D organs (For example, Colon Material.mat to Colon that can be selected in the Hierarchy window under the Prefab.)

Generating 3D organs from scratch

One can also generate 3D organs from different patients using the publicly available Cancer Imaging Archive. Please select a CT data in the DICOM format among the dataset for colon or stomach (see below image). Please note that the DICOM images consist of two sets, one taken in the supine position and the other in the prone position. The supine position DICOM images were used since that is the patient’s position during the capsule endoscopy session

After downlading DICOM data, use InVesalius or any similar softwarer to convert the DICOM images to 3D objects. The software provides an automatic selection of the regions desired to be converted, which in our case is the Soft Tissue. Then a surface will be created on the selected regions constructing the corresponding 3D model, which is exported as a Wavefront (.obj) file.

The 3D model is then imported into Blender for further processing which includes removal of bones, fat, skin, and other artifacts that the imported model has so that only the geometries of the colon, small intestines and stomach remain. Please note that not all converted 3D models includes the whole colon and intestines, these models should be discarded.

As some models consist of a large number of mesh which makes it hard to process, we reduced the number of mesh by using another software called MeshLab, using an algorithm called Quadric Edge Collapse Decimation for mesh simplification. It reduces the face number of a mesh while preserving its boundaries and normals.

Please note that, due to some imperfections on the CT data, you may need to fill the gaps and fix the topology of the organs. We used Blender for this operation. Please make sure that there is no missing parts in the 3D organs and the connections and the openings between the stomach and small intestines, and the small intestines and colon are all set.

Generating Disease Classes

We create pipeline to mimic 3 classes of diseases in our environment (Polpys with various shapes and sizes, Ulcerative Collitis and Hemorrage for 3 and 4 different amount and severity levels respectively) that can be used to train/test disease classification algorithms.

Polyps

In the Cancer Imaging Archive, you can also find different models of the organs with the cancerous lumps that can be used to mimic real shaped polyps with realistic locations of occurence. Firts, navigate to relavant class in the archive and download the corresponding DICOM format. Then, by following the same steps explained above, you can create 3D organ with polyps. In order to attain the texture particularly generated for polyps, you should use Blender or a similar software to manually depart meshes for the regions of polyp occurences and save them as different models. Then in Unity you can attain polyps texture Polyps.mat in the where other organ textures are located.

Ulcertive Collitis and Hemorrage

Unlike Polyps, Ulcer and Hemorrage do not have differ in the topology of 3D organs but in texture. Therefore, we generate specific textures for these classes. In order to create organs with these diseases, please select and attain a texture from the textures folder where other .mat files exist.

Various camera designs

As there are commercially available options in capsule camera designs in the wireless capsule endoscopy, in our environment, we extend the standard mono camera capsule to different designs such as stereo, dual and 360° camera). You can select these options from the Capsules folder under this folder

Adjusting camera parameters and post processing effects

Adjusting camera parameters can be used for both mimicing real endoscopy cameras and augmenting the data.

You can use the camera intrinsic parameters that we get by calibrating MiroCam and PillCam capsule endoscope cameras or play with them to generate augmented data.

To adjust Unity Camera, use parameters on Inspector window (e.g,, Field of View, Sensor Size, Focal Length etc.) Set the average of <img src="https://render.githubusercontent.com/render/math?math=f_{x}"> and <img src="https://render.githubusercontent.com/render/math?math=f_{y}"> for the focal length, 2x optical center (<img src="https://render.githubusercontent.com/render/math?math=2c_{x}"> and <img src="https://render.githubusercontent.com/render/math?math=2c_{y}">) for sensor size X and Y.

Please note that image resolution is adjusted on Recorder.

Specular reflection which occurs on the surface of organs due to interaction of light source can also be adjusted by the Coat Mask parameter on Unity's Inspector window.

Post-processing effects that HDRP (High Definition Rendering Pipeline) provides (specular reflection, vignette, lens distortion, chromatic aberration and depth of field) can also be adjusted with relevant parameters.

Movement of the capsule<img align="right" src="img/ElectroMagneticFieldController.png" width="300">

For the actuation of the capsule, we have placed a cylinder magnet inside the capsule and a ball magnet attached to the robot arm. Simulation of the magnetic field is modeled as dipole-dipole interactions by using (MagnetoDynamics).

The default Scene has two infinitesimal dipoles (MagneticDipole prefabs) embedded in Rigidbodies of the DiscMagnet(child object of Capsule)

VirtualCapsuleEndoscopy

Install / Use

README