Network Flow Integer Programming to Track Elliptical Cells in Time-Lapse Sequences

H-EMD: A Hierarchical Earth Mover's Distance Method for Instance Segmentation

Deep learning (DL) based semantic segmentation methods have achieved excellent performance in biomedical image segmentation, producing high quality probability maps to allow extraction of rich instance information to facilitate good instance segmentation. While numerous efforts were put into developing new DL semantic segmentation models, less attention was paid to a key issue of how to effectively explore their probability maps to attain the best possible instance segmentation. We observe that probability maps by DL semantic segmentation models can be used to generate many possible instance candidates, and accurate instance segmentation can be achieved by selecting from them a set of "optimized" candidates as output instances. Further, the generated instance candidates form a well-behaved hierarchical structure (a forest), which allows selecting instances in an optimized manner. Hence, we propose a novel framework, called hierarchical earth mover's distance (H-EMD), for instance segmentation in biomedical 2D+time videos and 3D images, which judiciously incorporates consistent instance selection with semantic-segmentation-generated probability maps. H-EMD contains two main stages: (1) instance candidate generation: capturing instance-structured information in probability maps by generating many instance candidates in a forest structure; (2) instance candidate selection: selecting instances from the candidate set for final instance segmentation. We formulate a key instance selection problem on the instance candidate forest as an optimization problem based on the earth mover's distance (EMD), and solve it by integer linear programming. Extensive experiments on eight biomedical video or 3D datasets demonstrate that H-EMD consistently boosts DL semantic segmentation models and is highly competitive with state-of-the-art methods.

CancerCellTracker: a brightfield time-lapse microscopy framework for cancer drug sensitivity estimation

MOTIVATION

Time-lapse microscopy is a powerful technique that relies on images of live cells cultured ex vivo that are captured at regular intervals of time to describe and quantify their behavior under certain experimental conditions. This imaging method has great potential in advancing the field of precision oncology by quantifying the response of cancer cells to various therapies and identifying the most efficacious treatment for a given patient. Digital image processing algorithms developed so far require high-resolution images involving very few cells originating from homogeneous cell line populations. We propose a novel framework that tracks cancer cells to capture their behavior and quantify cell viability to inform clinical decisions in a high-throughput manner.

RESULTS

The brightfield microscopy images a large number of patient-derived cells in an ex vivo reconstruction of the tumor microenvironment treated with 31 drugs for up to 6 days. We developed a robust and user-friendly pipeline CancerCellTracker that detects cells in co-culture, tracks these cells across time and identifies cell death events using changes in cell attributes. We validated our computational pipeline by comparing the timing of cell death estimates by CancerCellTracker from brightfield images and a fluorescent channel featuring ethidium homodimer. We benchmarked our results using a state-of-the-art algorithm implemented in ImageJ and previously published in the literature. We highlighted CancerCellTracker's efficiency in estimating the percentage of live cells in the presence of bone marrow stromal cells.

AVAILABILITY AND IMPLEMENTATION

https://github.com/compbiolabucf/CancerCellTracker.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Cell Trafficking at the Intersection of the Tumor-Immune Compartments

Migration is an essential cellular process that regulates human organ development and homeostasis as well as disease initiation and progression. In cancer, immune and tumor cell migration is strongly associated with immune cell infiltration, immune escape, and tumor cell metastasis, which ultimately account for more than 90% of cancer deaths. The biophysics and molecular regulation of the migration of cancer and immune cells have been extensively studied separately. However, accumulating evidence indicates that, in the tumor microenvironment, the motilities of immune and cancer cells are highly interdependent via secreted factors such as cytokines and chemokines. Tumor and immune cells constantly express these soluble factors, which produce a tightly intertwined regulatory network for these cells' respective migration. A mechanistic understanding of the reciprocal regulation of soluble factor-mediated cell migration can provide critical information for the development of new biomarkers of tumor progression and of tumor response to immuno-oncological treatments. We review the biophysical andbiomolecular basis for the migration of immune and tumor cells and their associated reciprocal regulatory network. We also describe ongoing attempts to translate this knowledge into the clinic.

DaXi-high-resolution, large imaging volume and multi-view single-objective light-sheet microscopy

The promise of single-objective light-sheet microscopy is to combine the convenience of standard single-objective microscopes with the speed, coverage, resolution and gentleness of light-sheet microscopes. We present DaXi, a single-objective light-sheet microscope design based on oblique plane illumination that achieves: (1) a wider field of view and high-resolution imaging via a custom remote focusing objective; (2) fast volumetric imaging over larger volumes without compromising image quality or necessitating tiled acquisition; (3) fuller image coverage for large samples via multi-view imaging and (4) higher throughput multi-well imaging via remote coverslip placement. Our instrument achieves a resolution of 450 nm laterally and 2 μm axially over an imaging volume of 3,000 × 800 × 300 μm. We demonstrate the speed, field of view, resolution and versatility of our instrument by imaging various systems, including Drosophila egg chamber development, zebrafish whole-brain activity and zebrafish embryonic development – up to nine embryos at a time.

The DaXi single-objective light-sheet microscope achieves fast, high-quality imaging of large volumes. DaXi’s design allows increased scanning range without sacrificing imaging speed or quality, multiview imaging and versatile sample mounting.

Bayesian Multi-Targets Strategy to Track Apis mellifera Movements at Colony Level

Simple Summary

The number of honey bee, Apis mellifera L., colonies has reduced around the globe, and one potential cause is their unintended exposure to sublethal stressors such as agricultural pesticides. The quantification of such effects at colony level is a very complex task due to the innumerable collective activities done by the individual within colonies. Here, we present a Bayesian and computational approach capable of tracking the movements of bees within colonies, which allows the comparison of the collective activities of colonies that received bees previously exposed to uncontaminated diets or to diets containing sublethal concentrations of an agricultural pesticide (a commercial formulation containing the synthetic fungicides thiophanate-methyl and chlorothalonil). Our Bayesian tracking technique proved successful and superior to comparable algorithms, allowing the estimation of dynamical parameters such as entropy and kinetic energy. Our efforts demonstrated that fungicide-contaminated colonies behaved differently from uncontaminated colonies, as the former exhibited anticipated collective activities in peripheral hive areas and had reduced swarm entropy and kinetic energies. Such findings may facilitate the electronic monitoring of potential unintended effects in social pollinators, at colony level, mediated by environmental stressors (e.g., pesticides, electromagnetic fields, noise, and light intensities) alone or in combination.

Abstract

Interactive movements of bees facilitate the division and organization of collective tasks, notably when they need to face internal or external environmental challenges. Here, we present a Bayesian and computational approach to track the movement of several honey bee, Apis mellifera, workers at colony level. We applied algorithms that combined tracking and Kernel Density Estimation (KDE), allowing measurements of entropy and Probability Distribution Function (PDF) of the motion of tracked organisms. We placed approximately 200 recently emerged and labeled bees inside an experimental colony, which consists of a mated queen, approximately 1000 bees, and a naturally occurring beehive background. Before release, labeled bees were fed for one hour with uncontaminated diets or diets containing a commercial mixture of synthetic fungicides (thiophanate-methyl and chlorothalonil). The colonies were filmed (12 min) at the 1st hour, 5th and 10th days after the bees’ release. Our results revealed that the algorithm tracked the labeled bees with great accuracy. Pesticide-contaminated colonies showed anticipated collective activities in peripheral hive areas, far from the brood area, and exhibited reduced swarm entropy and energy values when compared to uncontaminated colonies. Collectively, our approach opens novel possibilities to quantify and predict potential alterations mediated by pollutants (e.g., pesticides) at the bee colony-level.

Online Multiple Object Tracking Using a Novel Discriminative Module for Autonomous Driving

Multi object tracking (MOT) is a key research technology in the environment sensing system of automatic driving, which is very important to driving safety. Online multi object tracking needs to accurately extend the trajectory of multiple objects without using future frame information, so it will face greater challenges. Most of the existing online MOT methods are anchor-based detectors, which have many misdetections and missed detection problems, and have a poor effect on the trajectory extension of adjacent object objects when they are occluded and overlapped. In this paper, we propose a discrimination learning online tracker that can effectively solve the occlusion problem based on an anchor-free detector. This method uses the different weight characteristics of the object when the occlusion occurs and realizes the extension of the competition trajectory through the discrimination module to prevent the ID-switch problem. In the experimental part, we compared the algorithm with other trackers on two public benchmark datasets, MOT16 and MOT17, and proved that our algorithm has achieved state-of-the-art performance, and conducted a qualitative analysis on the convincing autonomous driving dataset KITTI.

DMNet: Dual-Stream Marker Guided Deep Network for Dense Cell Segmentation and Lineage Tracking

Accurate segmentation and tracking of cells in microscopy image sequences is extremely beneficial in clinical diagnostic applications and biomedical research. A continuing challenge is the segmentation of dense touching cells and deforming cells with indistinct boundaries, in low signal-to-noise-ratio images. In this paper, we present a dual-stream marker-guided network (DMNet) for segmentation of touching cells in microscopy videos of many cell types. DMNet uses an explicit cell marker-detection stream, with a separate mask-prediction stream using a distance map penalty function, which enables supervised training to focus attention on touching and nearby cells. For multi-object cell tracking we use M2Track tracking-by-detection approach with multi-step data association. Our M2Track with mask overlap includes short term track-to-cell association followed by track-to-track association to re-link tracklets with missing segmentation masks over a short sequence of frames. Our combined detection, segmentation and tracking algorithm has proven its potential on the IEEE ISBI 2021 6th Cell Tracking Challenge (CTC-6) where we achieved multiple top three rankings for diverse cell types. Our team name is MU-Ba-US, and the implementation of DMNet is available at, http://celltrackingchallenge.net/participants/MU-Ba-US/.

A graph-based cell tracking algorithm with few manually tunable parameters and automated segmentation error correction

Automatic cell segmentation and tracking enables to gain quantitative insights into the processes driving cell migration. To investigate new data with minimal manual effort, cell tracking algorithms should be easy to apply and reduce manual curation time by providing automatic correction of segmentation errors. Current cell tracking algorithms, however, are either easy to apply to new data sets but lack automatic segmentation error correction, or have a vast set of parameters that needs either manual tuning or annotated data for parameter tuning. In this work, we propose a tracking algorithm with only few manually tunable parameters and automatic segmentation error correction. Moreover, no training data is needed. We compare the performance of our approach to three well-performing tracking algorithms from the Cell Tracking Challenge on data sets with simulated, degraded segmentation—including false negatives, over- and under-segmentation errors. Our tracking algorithm can correct false negatives, over- and under-segmentation errors as well as a mixture of the aforementioned segmentation errors. On data sets with under-segmentation errors or a mixture of segmentation errors our approach performs best. Moreover, without requiring additional manual tuning, our approach ranks several times in the top 3 on the 6^th edition of the Cell Tracking Challenge.

EllipTrack: A Global-Local Cell-Tracking Pipeline for 2D Fluorescence Time-Lapse Microscopy

Time-lapse microscopy provides an unprecedented opportunity to monitor single-cell dynamics. However, tracking cells for long periods remains a technical challenge, especially for multi-day, large-scale movies with rapid cell migration, high cell density, and drug treatments that alter cell morphology/behavior. Here, we present EllipTrack, a global-local cell-tracking pipeline optimized for tracking such movies. EllipTrack first implements a global track-linking algorithm to construct tracks that maximize the probability of cell lineages. Tracking mistakes are then corrected with a local track-correction module in which tracks generated by the global algorithm are systematically examined and amended if a more probable alternative can be found. Through benchmarking, we show that EllipTrack outperforms state-of-the-art cell trackers and generates nearly error-free cell lineages for multiple large-scale movies. In addition, EllipTrack can adapt to time- and cell-density-dependent changes in cell migration speeds and requires minimal training datasets. EllipTrack is available at https://github.com/tianchengzhe/EllipTrack.

A novel machine learning based approach for iPS progenitor cell identification

Identification of induced pluripotent stem (iPS) progenitor cells, the iPS forming cells in early stage of reprogramming, could provide valuable information for studying the origin and underlying mechanism of iPS cells. However, it is very difficult to identify experimentally since there are no biomarkers known for early progenitor cells, and only about 6 days after reprogramming initiation, iPS cells can be experimentally determined via fluorescent probes. What is more, the ratio of progenitor cells during early reprograming period is below 5%, which is too low to capture experimentally in the early stage. In this paper, we propose a novel computational approach for the identification of iPS progenitor cells based on machine learning and microscopic image analysis. Firstly, we record the reprogramming process using a live cell imaging system after 48 hours of infection with retroviruses expressing Oct4, Sox2 and Klf4, later iPS progenitor cells and normal murine embryonic fibroblasts (MEFs) within 3 to 5 days after infection are labeled by retrospectively tracing the time-lapse microscopic image. We then calculate 11 types of cell morphological and motion features such as area, speed, etc., and select best time windows for modeling and perform feature selection. Finally, a prediction model using XGBoost is built based on the selected six types of features and best time windows. Our model allows several missing values/frames in the sample datasets, thus it is applicable to a wide range of scenarios. Cross-validation, holdout validation and independent test experiments show that the minimum precision is above 52%, that is, the ratio of predicted progenitor cells within 3 to 5 days after viral infection is above 52%. The results also confirm that the morphology and motion pattern of iPS progenitor cells is different from that of normal MEFs, which helps with the machine learning methods for iPS progenitor cell identification.

Identification of induced pluripotent stem (iPS) progenitor cells could provide valuable information for studying the origin and underlying mechanism of iPS cells. However, it is very difficult to identify experimentally since there are no biomarkers known for early progenitor cells, and only after about 6 days of induction, iPS cells can be experimentally determined via fluorescent probes. What is more, the percentage of the progenitor cells during the early induction period is below 5%, too low to capture experimentally in early stage. In this work, we proposed an approach for the identification of iPS progenitor cells, the iPS forming cells, based on machine learning and microscopic image analysis. The aim is to help biologists to enrich iPS progenitor cells during the early stage of induction, which allows experimentalists to select iPS progenitor cells with much higher probability, and furthermore to study the biomarkers which trigger the reprogramming process.

Imaging, Visualization, and Computation in Developmental Biology

Embryonic development is highly complex and dynamic, requiring the coordination of numerous molecular and cellular events at precise times and places. Advances in imaging technology have made it possible to follow developmental processes at cellular, tissue, and organ levels over time as they take place in the intact embryo. Parallel innovations of in vivo probes permit imaging to report on molecular, physiological, and anatomical events of embryogenesis, but the resulting multidimensional data sets pose significant challenges for extracting knowledge. In this review, we discuss recent and emerging advances in imaging technologies, in vivo labeling, and data processing that offer the greatest potential for jointly deciphering the intricate cellular dynamics and the underlying molecular mechanisms. Our discussion of the emerging area of “image-omics” highlights both the challenges of data analysis and the promise of more fully embracing computation and data science for rapidly advancing our understanding of biology.

Globally optimal segmentation of cell nuclei in fluorescence microscopy images using shape and intensity information

Accurate and efficient segmentation of cell nuclei in fluorescence microscopy images plays a key role in many biological studies. Besides coping with image noise and other imaging artifacts, the separation of touching and partially overlapping cell nuclei is a major challenge. To address this, we introduce a globally optimal model-based approach for cell nuclei segmentation which jointly exploits shape and intensity information. Our approach is based on implicitly parameterized shape models, and we propose single-object and multi-object schemes. In the single-object case, the used shape parameterization leads to convex energies which can be directly minimized without requiring approximation. The multi-object scheme is based on multiple collaborating shapes and has the advantage that prior detection of individual cell nuclei is not needed. This scheme performs joint segmentation and cluster splitting. We describe an energy minimization scheme which converges close to global optima and exploits convex optimization such that our approach does not depend on the initialization nor suffers from local energy minima. The proposed approach is robust and computationally efficient. In contrast, previous shape-based approaches for cell segmentation either are computationally expensive, not globally optimal, or do not jointly exploit shape and intensity information. We successfully applied our approach to fluorescence microscopy images of five different cell types and performed a quantitative comparison with previous methods.

Segmenting and tracking cell instances with cosine embeddings and recurrent hourglass networks

Differently to semantic segmentation, instance segmentation assigns unique labels to each individual instance of the same object class. In this work, we propose a novel recurrent fully convolutional network architecture for tracking such instance segmentations over time, which is highly relevant, e.g., in biomedical applications involving cell growth and migration. Our network architecture incorporates convolutional gated recurrent units (ConvGRU) into a stacked hourglass network to utilize temporal information, e.g., from microscopy videos. Moreover, we train our network with a novel embedding loss based on cosine similarities, such that the network predicts unique embeddings for every instance throughout videos, even in the presence of dynamic structural changes due to mitosis of cells. To create the final tracked instance segmentations, the pixel-wise embeddings are clustered among subsequent video frames by using the mean shift algorithm. After showing the performance of the instance segmentation on a static in-house dataset of muscle fibers from H&E-stained microscopy images, we also evaluate our proposed recurrent stacked hourglass network regarding instance segmentation and tracking performance on six datasets from the ISBI celltracking challenge, where it delivers state-of-the-art results.

A novel generic dictionary-based denoising method for improving noisy and densely packed nuclei segmentation in 3D time-lapse fluorescence microscopy images

Time-lapse fluorescence microscopy is an essential technique for quantifying various characteristics of cellular processes, i.e. cell survival, migration, and differentiation. To perform high-throughput quantification of cellular processes, nuclei segmentation and tracking should be performed in an automated manner. Nevertheless, nuclei segmentation and tracking are challenging tasks due to embedded noise, intensity inhomogeneity, shape variation as well as a weak boundary of nuclei. Although several nuclei segmentation approaches have been reported in the literature, dealing with embedded noise remains the most challenging part of any segmentation algorithm. We propose a novel denoising algorithm, based on sparse coding, that can both enhance very faint and noisy nuclei signal but simultaneously detect nuclei position accurately. Furthermore our method is based on a limited number of parameters, with only one being critical, which is the approximate size of the objects of interest. We also show that our denoising method coupled with classical segmentation method works properly in the context of the most challenging cases. To evaluate the performance of the proposed method, we tested our method on two datasets from the cell tracking challenge. Across all datasets, the proposed method achieved satisfactory results with 96:96% recall for the C. elegans dataset. Besides, in the Drosophila dataset, our method achieved very high recall (99:3%).

Cell Segmentation Using a Similarity Interface With a Multi-Task Convolutional Neural Network

Even though convolutional neural networks (CNN) have been used for cell segmentation, they require pixel-level ground truth annotations. This paper proposes a multitask learning algorithm for cell detection and segmentation using CNNs. We use dot annotations placed inside each cell indicating approximate cell centroids to create training datasets for the detection and segmentation tasks. The segmentation task is used to map the input image to foreground versus background regions, whereas the detection task is used to predict the centroids of the cells. Our multitask model shares convolutional layers between the two tasks, while having task-specific output layers. Learning two tasks simultaneously reduces the risks of overfitting and also helps in separating overlapping cells better. We also introduce a similarity interface (SI) that can be integrated with our multitask network to allow easy adaptation between domains, and to compensate for the variability in contrast and texture of cells seen in microscopy images. The SI comprises an unsupervised first layer in combination with a neighborhood similarity layer (NSL). A layer of logistic sigmoid functions is used as an unsupervised first layer to separate clustered image patches from each other. The NSL transforms its input feature map at a given pixel by computing its similarity to the surrounding neighborhood. Our proposed method achieves higher/comparable detection and segmentation scores as compared to recent state-of-the-art methods with significantly reduced effort for generating training data.

Interacting Tracklets for Multi-object Tracking

In this paper, we propose to exploit the interactions between non-associable tracklets to facilitate multi-object tracking. We introduce two types of tracklet interactions, close interaction and distant interaction. The close interaction imposes physical constraints between two temporally overlapping tracklets and more importantly, allows us to learn local classifiers to distinguish targets that are close to each other in the spatiotemporal domain. The distant interaction, on the other hand, accounts for the higher-order motion and appearance consistency between two temporally isolated tracklets. Our approach is modeled as a binary labeling problem and solved using the efficient Quadratic Pseudo-Boolean Optimization (QPBO). It yields promising tracking performance on the challenging PETS09 and MOT16 dataset. Our code will be made publicly available upon the acceptance of the manuscript.

A probabilistic approach to joint cell tracking and segmentation in high-throughput microscopy videos

We present a novel computational framework for the analysis of high-throughput microscopy videos of living cells. The proposed framework is generally useful and can be applied to different datasets acquired in a variety of laboratory settings. This is accomplished by tying together two fundamental aspects of cell lineage construction, namely cell segmentation and tracking, via a Bayesian inference of dynamic models. In contrast to most existing approaches, which aim to be general, no assumption of cell shape is made. Spatial, temporal, and cross-sectional variation of the analysed data are accommodated by two key contributions. First, time series analysis is exploited to estimate the temporal cell shape uncertainty in addition to cell trajectory. Second, a fast marching (FM) algorithm is used to integrate the inferred cell properties with the observed image measurements in order to obtain image likelihood for cell segmentation, and association. The proposed approach has been tested on eight different time-lapse microscopy data sets, some of which are high-throughput, demonstrating promising results for the detection, segmentation and association of planar cells. Our results surpass the state of the art for the Fluo-C2DL-MSC data set of the Cell Tracking Challenge (Maška et al., 2014).

An objective comparison of cell-tracking algorithms

We present a combined report on the results of three editions of the Cell Tracking Challenge, an ongoing initiative aimed at promoting the development and objective evaluation of cell segmentation and tracking algorithms. With 21 participating algorithms and a data repository consisting of 13 data sets from various microscopy modalities, the challenge displays today's state-of-the-art methodology in the field. We analyzed the challenge results using performance measures for segmentation and tracking that rank all participating methods. We also analyzed the performance of all of the algorithms in terms of biological measures and practical usability. Although some methods scored high in all technical aspects, none obtained fully correct solutions. We found that methods that either take prior information into account using learning strategies or analyze cells in a global spatiotemporal video context performed better than other methods under the segmentation and tracking scenarios included in the challenge.

Greedy Batch-Based Minimum-Cost Flows for Tracking Multiple Objects

Minimum-cost flow algorithms have recently achieved state-of-the-art results in multi-object tracking. However, they rely on the whole image sequence as input. When deployed in real-time applications or in distributed settings, these algorithms first operate on short batches of frames and then stitch the results into full trajectories. This decoupled strategy is prone to errors because the batch-based tracking errors may propagate to the final trajectories and cannot be corrected by other batches. In this paper, we propose a greedy batch-based minimum-cost flow approach for tracking multiple objects. Unlike existing approaches that conduct batch-based tracking and stitching sequentially, we optimize consecutive batches jointly so that the tracking results on one batch may benefit the results on the other. Specifically, we apply a generalized minimum-cost flows (MCF) algorithm on each batch and generate a set of conflicting trajectories. These trajectories comprise the ones with high probabilities, but also those with low probabilities potentially missed by detectors and trackers. We then apply the generalized MCF again to obtain the optimal matching between trajectories from consecutive batches. Our proposed approach is simple, effective, and does not require training. We demonstrate the power of our approach on data sets of different scenarios.