Usiigaci: Instance-aware cell tracking in stain-free phase contrast microscopy enabled by machine learning

Hsieh-Fu Tsai^1,2, Joanna Gajda³, Tyler F.W. Sloan⁴, Andrei Rares⁵, Jason Ting-Chun Chou¹, and Amy Q. Shen¹

^{¹Micro/Bio/Nanofluidics Unit, Okinawa Institute of Science and Technology Graduate University, Okinawa Japan}

^{²Research Fellow of Japan Society of Promotion for Science}

<sup>³AGH University of Science and Technology, Krakow, Poland</sep>

^{⁴Quorumetrix Solutions, Canada}

^{⁵ImagineA, The Netherlands}

Usiigaci, ウシーガチ, ushi:gachi in Hepburn romanization, in Ryukyuan language means "tracing", "透き写し" in Japanese，i.e. drawing the outline of objects based on a template. The process is essentially what we do: following the morphology and position of cells under microscope, analyze how cell respond upon environmental perturbation in the microenvironment. However, this process is bloody tedious if done by human, and now we developed a pipeline using the famous Mask R-CNN to do this for us. Letting us not only track objects by their position but also track how their morphology changes through time.

Zernike's phase contrast microscopy is a brightfield microscopy technique developed by Frits Zernike and by inventing the phase contrast technique, he won the 1953 Nobel Prize for physics. Phase contrast microscopy is favored by biologists because it translates the phase difference caused by cell components into amplitude thus making these transparent structures more visible. Also, in comparison to differential interference contrast microscopy, phase contrast microscopy works without problems with different substrates especially on plastics that are highly birefringent.

Phase contrast microscopy images are notoriously difficult to segment by conventional computer vision methods. However, accurate whole cell outline segmentation and resolution of cells that contact each other are essential as the first step for cell tracking in automated microscopy needs accurate cell identification. Tracking and visualization of the cellular dynamics based on the segmentations help us understand and quantitative analyze cellular dynamics.

We report Usiigaci, a semi-automated pipeline to segment, track, and visualize cell migration in phase contrast microscopy.

High accuracy instance-aware segmentation is achieved by adapting the mask regional convolutional neural network (Mask R-CNN), winner of Marr prize at ICCV 2017 by He et al.. We built Usiigaci's segmentation module based on the Mask R-CNN implementation by Matterport. Using 50 manually-annotated cell images for training, the trained Mask R-CNN neural network can generate high accuracy whole cell segmentation masks that allow us to analyze both cell migration and cell morphology which are difficult even by fluorescent imaging.

Cell tracking and data verification can be done in ImageJ, other existin tracking software such as Lineage Mapper, or Usiigaci tracker that we developed based on open-source trackpy library. A GUI is also developed to allow manual data verification to check tracking results and delete bad results.

A Jupyter Notebook and the corresponding python script are developed for automated processing and visualization of the tracked results. Step-centric and cell-centric parameters are automatically computed and saved into additional spreadsheets where users can access and reuse in statistical software or R. Automated visualization of cell migration is also generated for cell trajectory graphs, box plots, etc.

• Cell trajectory graph
  • 2D hair ball color coded by track
  • 2D hair ball color coded by time (Imaris like)
  • 2D hair ball color coded by direction (Ibidi like)
  • 2D hair ball color coded by direction length
  • 3D hair ball with z as time
  • scatter plot in gif
• Automated cell migration analysis
  • computation of step centric parameters
    • instantaneous displacement
    • instantaneous speed
    • turn angle
    • direction autocorrelation
  • compuatation of cell centric parameters
    • cumulative distance (total traveled distance)
    • Euclidean distance
    • net velocity
    • end point directionality ratio
    • orientation (cell alignment index)
    • Directedness
  • save individual cell track data
  • save summary of each cell throughout experiment
  • save summary of ensemble at each time point
• automated plotting of descriptive statistics
  • rose histogram of cell orientation
  • box plots, violin plots, and time series plots of cell migration parameters
  • frequency histograms

We worked on Usiigaci for our work on cell electrotaxis study, and hopefully can devote to current international effort to standardize cell migration experiments.

We hope Usiigaci is interesting to you and if it is useful for your research, please cite our paper.

Hsieh-Fu Tsai, Joanna Gajda, Tyler F.W. Sloan, Andrei Rares, and Amy Q. Shen, SoftwareX, 9, 230-237, 2019

Accessible here

Usiigaci is released under MIT License.

Copyright (c) 2018 Okinawa Institute of Science & Technology Graduate University

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and 
associated documentation files (the "Software"), to deal in the Software without restriction, including 
without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 
copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the 
following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions 
of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT 
LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO 
EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE 
USE OR OTHER DEALINGS IN THE SOFTWARE.

TensorFlow is open-sourced under Apache 2.0 opensource license.

Keras is released under MIT license

The copyright of PyQT belong to Riverbank Computing ltd.

Pandas is released under BSD 3-Clause License. Copyright owned by AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team.

Trackpy is released under BSD 3-Clause License. Copyright owned by trackpy contributors.

NumPy and SciPy are released under BSD-new License

Scikit-image is released under modified BSD license

PIMS is released under modified BSD license

Matplotlib is released under Python Software Foundation (PDF) license

Seaborn is released under BSD 3-clause license

ffmpeg is licensed under LGPL 2.1 license

PyQtGraph is released under MIT license

Future work

(2019.Sep.01) We are working on a new architecture to become more light weight and flexible.

Mask R-CNN segmentation

• [ ] Add a function to save only the best model weight during training
• [ ] Add a function to compute F1 score, Jaccard index, Accurady, Precision to a test dataset after training
• [ ] pretrain model weights for DIC microscopy.
• [ ] pretrain model weights for recognition of nucleus
• [ ] add bounding box as an output format (overlay over original image)
• [ ] change output mask to 16 bit to support more instance to be recognized
• [ ] Multiclass segmentation to realize identification of mitotic cells.
• [ ] Multiclass segmentation to realize label-free co-cultured cell segmentation.
• [ ] Add multiple GPU training and inference support - likely need to rework the fundamentals.
• [ ] pretrain model weights for Atto cytowatcher microscope
• [ ] Pretrain model weights for Essen Incucyte S3
• [ ] Pretrain model weights for bacteria

Tracking

• [ ] Add lineage tracking function
• [x] Add support for ultra-long time series tracking
• [x] Add batch tracking scripts

data processing

• [x] Add data analysis script for specific time frame
• [ ] Add output into rois for later use coupling imageJ (for users to quantify the roi)
• [ ] plotly based data visualization for rare event analysis.
• [ ] Add calculation of cell doubling time
• [ ] Add rare event detection
• [ ] Add confluency calculation
• [ ] Faster more integrated data processing pipeline, perhaps on C++
• [ ] Add cell contour vector analysis
• [ ] Add multiclass percent stacked bar plot.

Call for collaboration

We are still improving the software. If you'd like, please adapt the source code on your own dataset. If you are willing to share you dataset, but limited by computing power, we can initiate collaboration. We can try to annotate your data and train a model weight for your data or incorporate your data together with our dataset that you can use. If interested, please contact to the corresponding authors.

Acknowledgement:

This work is supported by JSPS KAKENHI Grant JP1700362 and Okinawa Institute of Science and