Phase contrast time-lapse microscopy datasets with automated and manual cell tracking annotations

Deep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives

Deep learning has demonstrated remarkable performance across various tasks in medical imaging. However, these approaches primarily focus on supervised learning, assuming that the training and testing data are drawn from the same distribution. Unfortunately, this assumption may not always hold true in practice. To address these issues, unsupervised domain adaptation (UDA) techniques have been developed to transfer knowledge from a labeled domain to a related but unlabeled domain. In recent years, significant advancements have been made in UDA, resulting in a wide range of methodologies, including feature alignment, image translation, self-supervision, and disentangled representation methods, among others. In this paper, we provide a comprehensive literature review of recent deep UDA approaches in medical imaging from a technical perspective. Specifically, we categorize current UDA research in medical imaging into six groups and further divide them into finer subcategories based on the different tasks they perform. We also discuss the respective datasets used in the studies to assess the divergence between the different domains. Finally, we discuss emerging areas and provide insights and discussions on future research directions to conclude this survey.

ALFI: Cell cycle phenotype annotations of label-free time-lapse imaging data from cultured human cells

Detecting and tracking multiple moving objects in a video is a challenging task. For living cells, the task becomes even more arduous as cells change their morphology over time, can partially overlap, and mitosis leads to new cells. Differently from fluorescence microscopy, label-free techniques can be easily applied to almost all cell lines, reducing sample preparation complexity and phototoxicity. In this study, we present ALFI, a dataset of images and annotations for label-free microscopy, made publicly available to the scientific community, that notably extends the current panorama of expertly labeled data for detection and tracking of cultured living nontransformed and cancer human cells. It consists of 29 time-lapse image sequences from HeLa, U2OS, and hTERT RPE-1 cells under different experimental conditions, acquired by differential interference contrast microscopy, for a total of 237.9 hours. It contains various annotations (pixel-wise segmentation masks, object-wise bounding boxes, tracking information). The dataset is useful for testing and comparing methods for identifying interphase and mitotic events and reconstructing their lineage, and for discriminating different cellular phenotypes.

Differentiable optimization layers enhance GNN-based mitosis detection

Automatic mitosis detection from video is an essential step in analyzing proliferative behaviour of cells. In existing studies, a conventional object detector such as Unet is combined with a link prediction algorithm to find correspondences between parent and daughter cells. However, they do not take into account the biological constraint that a cell in a frame can correspond to up to two cells in the next frame. Our model called GNN-DOL enables mitosis detection by complementing a graph neural network (GNN) with a differentiable optimization layer (DOL) that implements the constraint. In time-lapse microscopy sequences cultured under four different conditions, we observed that the layer substantially improved detection performance in comparison with GNN-based link prediction. Our results illustrate the importance of incorporating biological knowledge explicitly into deep learning models.

DeepSea is an efficient deep-learning model for single-cell segmentation and tracking in time-lapse microscopy

Time-lapse microscopy is the only method that can directly capture the dynamics and heterogeneity of fundamental cellular processes at the single-cell level with high temporal resolution. Successful application of single-cell time-lapse microscopy requires automated segmentation and tracking of hundreds of individual cells over several time points. However, segmentation and tracking of single cells remain challenging for the analysis of time-lapse microscopy images, in particular for widely available and non-toxic imaging modalities such as phase-contrast imaging. This work presents a versatile and trainable deep-learning model, termed DeepSea, that allows for both segmentation and tracking of single cells in sequences of phase-contrast live microscopy images with higher precision than existing models. We showcase the application of DeepSea by analyzing cell size regulation in embryonic stem cells.

Phase Contrast Image Restoration by Formulating Its Imaging Principle and Reversing the Formulation With Deep Neural Networks

Phase contrast microscopy, as a noninvasive imaging technique, has been widely used to monitor the behavior of transparent cells without staining or altering them. Due to the optical principle of the specifically-designed microscope, phase contrast microscopy images contain artifacts such as halo and shade-off which hinder the cell segmentation and detection tasks. Some previous works developed simplified computational imaging models for phase contrast microscopes by linear approximations and convolutions. The approximated models do not exactly reflect the imaging principle of the phase contrast microscope and accordingly the image restoration by solving the corresponding deconvolution process is not perfect. In this paper, we revisit the optical principle of the phase contrast microscope to precisely formulate its imaging model without any approximation. Based on this model, we propose an image restoration procedure by reversing this imaging model with a deep neural network, instead of mathematically deriving the inverse operator of the model which is technically impossible. Extensive experiments are conducted to demonstrate the superiority of the newly derived phase contrast microscopy imaging model and the power of the deep neural network on modeling the inverse imaging procedure. Moreover, the restored images enable that high quality cell segmentation task can be easily achieved by simply thresholding methods. Implementations of this work are publicly available at https://github.com/LiangHann/Phase-Contrast-Microscopy-Image-Restoration.

LapTrack: linear assignment particle tracking with tunable metrics

MOTIVATION

Particle tracking is an important step of analysis in a variety of scientific fields and is particularly indispensable for the construction of cellular lineages from live images. Although various supervised machine learning methods have been developed for cell tracking, the diversity of the data still necessitates heuristic methods that require parameter estimations from small amounts of data. For this, solving tracking as a linear assignment problem (LAP) has been widely applied and demonstrated to be efficient. However, there has been no implementation that allows custom connection costs, parallel parameter tuning with ground truth annotations, and the functionality to preserve ground truth connections, limiting the application to datasets with partial annotations.

RESULTS

We developed LapTrack, a LAP-based tracker which allows including arbitrary cost functions and inputs, parallel parameter tuning and ground-truth track preservation. Analysis of real and artificial datasets demonstrates the advantage of custom metric functions for tracking score improvement from distance-only cases. The tracker can be easily combined with other Python-based tools for particle detection, segmentation and visualization.

AVAILABILITY AND IMPLEMENTATION

LapTrack is available as a Python package on PyPi, and the notebook examples are shared at https://github.com/yfukai/laptrack. The data and code for this publication are hosted at https://github.com/NoneqPhysLivingMatterLab/laptrack-optimisation.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Weakly supervised cell instance segmentation under various conditions

Cell instance segmentation is important in biomedical research. For living cell analysis, microscopy images are captured under various conditions (e.g., the type of microscopy and type of cell). Deep-learning-based methods can be used to perform instance segmentation if sufficient annotations of individual cell boundaries are prepared as training data. Generally, annotations are required for each condition, which is very time-consuming and labor-intensive. To reduce the annotation cost, we propose a weakly supervised cell instance segmentation method that can segment individual cell regions under various conditions by only using rough cell centroid positions as training data. This method dramatically reduces the annotation cost compared with the standard annotation method of supervised segmentation. We demonstrated the efficacy of our method on various cell images; it outperformed several of the conventional weakly-supervised methods on average. In addition, we demonstrated that our method can perform instance cell segmentation without any manual annotation by using pairs of phase contrast and fluorescence images in which cell nuclei are stained as training data.

Spatio-Temporal Mitosis Detection in Time-Lapse Phase-Contrast Microscopy Image Sequences: A Benchmark

In this paper, we report the results of the first international contest on mitosis detection in phase-contrast microscopy image sequences (https://www.iti-tju.org/mitosisdetection), which was held at the workshop of computer vision for microscopy image analysis (CVMI) in CVPR 2019. This contest aims to promote research on spatiotemporal mitosis detection under microscopy images. In this contest, we released a large-scale time-lapse phase-contrast microscopy image dataset (C2C12-16) for the mitosis detection task. Compared with the previous popular datasets (e.g., C2C12, C3H10), C2C12-16 contains more annotated mitotic events and more diverse cell culture environments. A total of ten different mitosis detection methods were submitted in the contest and evaluated on the test sets of four different cell culture environments in C2C12-16. In this benchmark, we describe all methods and conduct a thorough analysis based on their performances and discuss a feasible direction for mitosis detection. To the best of our knowledge, this is the first benchmark for the mitosis detection problem using a time-lapse phase-contrast microscopy spatiotemporal image sequence model.

An open-source, end-to-end workflow for multidimensional photoemission spectroscopy

Characterization of the electronic band structure of solid state materials is routinely performed using photoemission spectroscopy. Recent advancements in short-wavelength light sources and electron detectors give rise to multidimensional photoemission spectroscopy, allowing parallel measurements of the electron spectral function simultaneously in energy, two momentum components and additional physical parameters with single-event detection capability. Efficient processing of the photoelectron event streams at a rate of up to tens of megabytes per second will enable rapid band mapping for materials characterization. We describe an open-source workflow that allows user interaction with billion-count single-electron events in photoemission band mapping experiments, compatible with beamlines at 3rd and 4rd generation light sources and table-top laser-based setups. The workflow offers an end-to-end recipe from distributed operations on single-event data to structured formats for downstream scientific tasks and storage to materials science database integration. Both the workflow and processed data can be archived for reuse, providing the infrastructure for documenting the provenance and lineage of photoemission data for future high-throughput experiments.

Virtual organelle self-coding for fluorescence imaging via adversarial learning

SIGNIFICANCE

Our study introduces an application of deep learning to virtually generate fluorescence images to reduce the burdens of cost and time from considerable effort in sample preparation related to chemical fixation and staining.

AIM

The objective of our work was to determine how successfully deep learning methods perform on fluorescence prediction that depends on structural and/or a functional relationship between input labels and output labels.

APPROACH

We present a virtual-fluorescence-staining method based on deep neural networks (VirFluoNet) to transform co-registered images of cells into subcellular compartment-specific molecular fluorescence labels in the same field-of-view. An algorithm based on conditional generative adversarial networks was developed and trained on microscopy datasets from breast-cancer and bone-osteosarcoma cell lines: MDA-MB-231 and U2OS, respectively. Several established performance metrics-the mean absolute error (MAE), peak-signal-to-noise ratio (PSNR), and structural-similarity-index (SSIM)-as well as a novel performance metric, the tolerance level, were measured and compared for the same algorithm and input data.

RESULTS

For the MDA-MB-231 cells, F-actin signal performed the fluorescent antibody staining of vinculin prediction better than phase-contrast as an input. For the U2OS cells, satisfactory metrics of performance were archieved in comparison with ground truth. MAE is <0.005, 0.017, 0.012; PSNR is >40  /  34  /  33  dB; and SSIM is >0.925  /  0.926  /  0.925 for 4',6-diamidino-2-phenylindole/hoechst, endoplasmic reticulum, and mitochondria prediction, respectively, from channels of nucleoli and cytoplasmic RNA, Golgi plasma membrane, and F-actin.

CONCLUSIONS

These findings contribute to the understanding of the utility and limitations of deep learning image-regression to predict fluorescence microscopy datasets of biological cells. We infer that predicted image labels must have either a structural and/or a functional relationship to input labels. Furthermore, the approach introduced here holds promise for modeling the internal spatial relationships between organelles and biomolecules within living cells, leading to detection and quantification of alterations from a standard training dataset.

Spatial-Temporal Mitosis Detection in Phase-Contrast Microscopy via Likelihood Map Estimation by 3DCNN

Automated mitotic detection in time-lapse phase-contrast microscopy provides us much information for cell behavior analysis, and thus several mitosis detection methods have been proposed. However, these methods still have two problems; 1) they cannot detect multiple mitosis events when there are closely placed. 2) they do not consider the annotation gaps, which may occur since the appearances of mitosis cells are very similar before and after the annotated frame. In this paper, we propose a novel mitosis detection method that can detect multiple mitosis events in a candidate sequence and mitigate the human annotation gap via estimating spatial-temporal likelihood map by 3DCNN. In this training, the loss gradually decreases with the gap size between ground-truth and estimation. This mitigates the annotation gaps. Our method outperformed the compared methods in terms of F1-score using challenging dataset that contains the data under four different conditions. Code is publicly available in https://github.com/naivete5656/MDMLM.

Kinetic Analysis of the Motility of Giant Virus-Infected Amoebae Using Phase-Contrast Microscopic Images

Tracking cell motility is a useful tool for the study of cell physiology and microbiology. Although phase-contrast microscopy is commonly used, the existence of optical artifacts called “halo” and “shade-off” have inhibited image analysis of moving cells. Here we show kinetic image analysis of Acanthamoeba motility using a newly developed computer program named “Phase-contrast-based Kinetic Analysis Algorithm for Amoebae (PKA3),” which revealed giant-virus-infected amoebae-specific motilities and aggregation profiles using time-lapse phase-contrast microscopic images. This program quantitatively detected the time-dependent, sequential changes in cellular number, size, shape, and direction and distance of cell motility. This method expands the potential of kinetic analysis of cultured cells using versatile phase-contrast images. Furthermore, this program could be a useful tool for investigating detailed kinetic mechanisms of cell motility, not only in virus-infected amoebae but also in other cells, including cancer cells, immune response cells, and neurons.