Coupled minimum-cost flow cell tracking for high-throughput quantitative analysis

Graph-based automatic detection and classification of lesion changes in pairs of CT studies for oncology follow-up

PURPOSE

Radiological follow-up of oncology patients requires the quantitative analysis of lesion changes in longitudinal imaging studies, which is time-consuming, requires expertise, and is subject to variability. This paper presents a comprehensive graph-based method for the automatic detection and classification of lesion changes in current and prior CT scans.

METHODS

The inputs are the current and prior CT scans and their organ and lesion segmentations. Classification of lesion changes is formalized as bipartite graph matching where lesion pairings are computed by adaptive overlap-based lesion matching. Six types of lesion changes are computed by connected components analysis. The method was evaluated on 208 pairs of lung and liver CT scans from 57 patients with 4600 lesions, 1713 lesion matchings and 2887 lesion changes. Ground-truth lesion segmentations, lesion matchings and lesion changes were created by an expert radiologist.

RESULTS

Our method yields a lesion matching rate accuracy of 99.7% (394/395) and 95.0% (1252/1318) for the lung and liver datasets. Precision and recall are > 0.99 and 0.94 and 0.95 (respectively) for the detection of lesion changes. The analysis of lesion changes helped the radiologist detect 48 missed lesions and 8 spurious lesions in the input ground-truth lesion datasets.

CONCLUSION

The classification of lesion classification provides the clinician with a readily accessible and intuitive identification and classification of the lesion changes and their patterns in support of clinical decision making. Comprehensive automatic computer-aided lesion matching and analysis of lesion changes may improve quantitative follow-up and evaluation of disease status, assessment of treatment efficacy and response to therapy.

Unsupervised Contour Tracking of Live Cells by Mechanical and Cycle Consistency Losses

Analyzing the dynamic changes of cellular morphology is important for understanding the various functions and characteristics of live cells, including stem cells and metastatic cancer cells. To this end, we need to track all points on the highly deformable cellular contour in every frame of live cell video. Local shapes and textures on the contour are not evident, and their motions are complex, often with expansion and contraction of local contour features. The prior arts for optical flow or deep point set tracking are unsuited due to the fluidity of cells, and previous deep contour tracking does not consider point correspondence. We propose the first deep learning-based tracking of cellular (or more generally viscoelastic materials) contours with point correspondence by fusing dense representation between two contours with cross attention. Since it is impractical to manually label dense tracking points on the contour, unsupervised learning comprised of the mechanical and cyclical consistency losses is proposed to train our contour tracker. The mechanical loss forcing the points to move perpendicular to the contour effectively helps out. For quantitative evaluation, we labeled sparse tracking points along the contour of live cells from two live cell datasets taken with phase contrast and confocal fluorescence microscopes. Our contour tracker quantitatively outperforms compared methods and produces qualitatively more favorable results. Our code and data are publicly available at https://github.com/JunbongJang/contour-tracking/.

The Cell Tracking Challenge: 10 years of objective benchmarking

The Cell Tracking Challenge is an ongoing benchmarking initiative that has become a reference in cell segmentation and tracking algorithm development. Here, we present a significant number of improvements introduced in the challenge since our 2017 report. These include the creation of a new segmentation-only benchmark, the enrichment of the dataset repository with new datasets that increase its diversity and complexity, and the creation of a silver standard reference corpus based on the most competitive results, which will be of particular interest for data-hungry deep learning-based strategies. Furthermore, we present the up-to-date cell segmentation and tracking leaderboards, an in-depth analysis of the relationship between the performance of the state-of-the-art methods and the properties of the datasets and annotations, and two novel, insightful studies about the generalizability and the reusability of top-performing methods. These studies provide critical practical conclusions for both developers and users of traditional and machine learning-based cell segmentation and tracking algorithms.

Adaptive tracking algorithm for trajectory analysis of cells and layer-by-layer assessment of motility dynamics

Tracking biological objects such as cells or subcellular components imaged with time-lapse microscopy enables us to understand the molecular principles about the dynamics of cell behaviors. However, automatic object detection, segmentation and extracting trajectories remain as a rate-limiting step due to intrinsic challenges of video processing. This paper presents an adaptive tracking algorithm (Adtari) that automatically finds the optimum search radius and cell linkages to determine trajectories in consecutive frames. A critical assumption in most tracking studies is that displacement remains unchanged throughout the movie and cells in a few frames are usually analyzed to determine its magnitude. Tracking errors and inaccurate association of cells may occur if the user does not correctly evaluate the value or prior knowledge is not present on cell movement. The key novelty of our method is that minimum intercellular distance and maximum displacement of cells between frames are dynamically computed and used to determine the threshold distance. Since the space between cells is highly variable in a given frame, our software recursively alters the magnitude to determine all plausible matches in the trajectory analysis. Our method therefore eliminates a major preprocessing step where a constant distance was used to determine the neighbor cells in tracking methods. Cells having multiple overlaps and splitting events were further evaluated by using the shape attributes including perimeter, area, ellipticity and distance. The features were applied to determine the closest matches by minimizing the difference in their magnitudes. Finally, reporting section of our software were used to generate instant maps by overlaying cell features and trajectories. Adtari was validated by using videos with variable signal-to-noise, contrast ratio and cell density. We compared the adaptive tracking with constant distance and other methods to evaluate performance and its efficiency. Our algorithm yields reduced mismatch ratio, increased ratio of whole cell track, higher frame tracking efficiency and allows layer-by-layer assessment of motility to characterize single-cells. Adaptive tracking provides a reliable, accurate, time efficient and user-friendly open source software that is well suited for analysis of 2D fluorescence microscopy video datasets.

No Cell Left behind: Automated, Stochastic, Physics-Based Tracking of Every Cell in a Dense, Growing Colony

Motivation: Precise tracking of individual cells—especially tracking the family lineage, for example in a developing embryo—has widespread applications in biology and medicine. Due to significant noise in microscope images, existing methods have difficulty precisely tracking cell activities. These difficulties often require human intervention to resolve. Humans are helpful because our brain naturally and automatically builds a simulation “model” of any scene that we observe. Because we understand simple truths about the world—for example cells can move and divide, but they cannot instantaneously move vast distances—this model “in our heads” helps us to severely constrain the possible interpretations of what we see, allowing us to easily distinguish signal from noise, and track the motion of cells even in the presence of extreme levels of noise that would completely confound existing automated methods. Results: Here, we mimic the ability of the human brain by building an explicit computer simulation model of the scene. Our simulated cells are programmed to allow movement and cell division consistent with reality. At each video frame, we stochastically generate millions of nearby “Universes” and evolve them stochastically to the next frame. We then find and fit the best universes to reality by minimizing the residual between the real image frame and a synthetic image of the simulation. The rule-based simulation puts extremely stringent constraints on possible interpretations of the data, allowing our system to perform far better than existing methods even in the presense of extreme levels of image noise. We demonstrate the viability of this method by accurately tracking every cell in a colony that grows from 4 to over 300 individuals, doing about as well as a human can in the difficult task of tracking cell lineages.

CellSeg: a robust, pre-trained nucleus segmentation and pixel quantification software for highly multiplexed fluorescence images

BACKGROUND

Algorithmic cellular segmentation is an essential step for the quantitative analysis of highly multiplexed tissue images. Current segmentation pipelines often require manual dataset annotation and additional training, significant parameter tuning, or a sophisticated understanding of programming to adapt the software to the researcher's need. Here, we present CellSeg, an open-source, pre-trained nucleus segmentation and signal quantification software based on the Mask region-convolutional neural network (R-CNN) architecture. CellSeg is accessible to users with a wide range of programming skills.

RESULTS

CellSeg performs at the level of top segmentation algorithms in the 2018 Kaggle Data Challenge both qualitatively and quantitatively and generalizes well to a diverse set of multiplexed imaged cancer tissues compared to established state-of-the-art segmentation algorithms. Automated segmentation post-processing steps in the CellSeg pipeline improve the resolution of immune cell populations for downstream single-cell analysis. Finally, an application of CellSeg to a highly multiplexed colorectal cancer dataset acquired on the CO-Detection by indEXing (CODEX) platform demonstrates that CellSeg can be integrated into a multiplexed tissue imaging pipeline and lead to accurate identification of validated cell populations.

CONCLUSION

CellSeg is a robust cell segmentation software for analyzing highly multiplexed tissue images, accessible to biology researchers of any programming skill level.

Visualization and quantitative analyses for mouse embryonic stem cell tracking by manipulating hierarchical data structures using time-lapse confocal microscopy images

We present a cell tracking method for time-lapse confocal microscopy (3D) images that uses dynamic hierarchical data structures to assist cell and colony segmentation and tracking. During the segmentation, the cell and colony numbers and their geometric data are recorded for each 3D image set. In tracking, the colony correspondences between neighboring frames of time-lapse 3D images are first computed using the recorded colony centers. Then, cell correspondences in the correspondent colonies are computed using the recorded cell centers. The examples show the proposed cell tracking method can achieve high tracking accuracy for time-lapse 3D images of undifferentiated but self-renewing mouse embryonic stem (mES) cells where the number and mobility of ES cells in a cell colony may change suddenly by a colony merging or splitting, and cell proliferation or death. The geometric data in the hierarchical data structures also help the visualization and quantitation of the cell shapes and mobility.

A review for Cell and Particle Tracking on Microscopy Images using Algorithms and Deep Learning Technologies

Time-lapse microscopy images generated by biological experiments have been widely used for observing target activities, such as the motion trajectories and survival states. Based on these observations, biologists can conclude experimental results or present new hypotheses for several biological applications, i.e. virus research or drug design. Many methods or tools have been proposed in the past to observe cell and particle activities, which are defined as single cell tracking and single particle tracking problems, by using algorithms and deep learning technologies. In this article, a review for these works is presented in order to summarize the past methods and research topics at first, then points out the problems raised by these works, and finally proposes future research directions. The contributions of this article will help researchers to understand past development trends and further propose innovative technologies.

A graph-based cell tracking algorithm with few manually tunable parameters and automated segmentation error correction

Automatic cell segmentation and tracking enables to gain quantitative insights into the processes driving cell migration. To investigate new data with minimal manual effort, cell tracking algorithms should be easy to apply and reduce manual curation time by providing automatic correction of segmentation errors. Current cell tracking algorithms, however, are either easy to apply to new data sets but lack automatic segmentation error correction, or have a vast set of parameters that needs either manual tuning or annotated data for parameter tuning. In this work, we propose a tracking algorithm with only few manually tunable parameters and automatic segmentation error correction. Moreover, no training data is needed. We compare the performance of our approach to three well-performing tracking algorithms from the Cell Tracking Challenge on data sets with simulated, degraded segmentation—including false negatives, over- and under-segmentation errors. Our tracking algorithm can correct false negatives, over- and under-segmentation errors as well as a mixture of the aforementioned segmentation errors. On data sets with under-segmentation errors or a mixture of segmentation errors our approach performs best. Moreover, without requiring additional manual tuning, our approach ranks several times in the top 3 on the 6^th edition of the Cell Tracking Challenge.

A Joint Tracking Approach via Ant Colony Evolution for Quantitative Cell Cycle Analysis

In this paper, we use an ant colony heuristic method to tackle the integration of data association and state estimation in the presence of cell mitosis, morphological change and uncertainty of measurement. Our approach first models the scouting behavior of an unlabeled ant colony as a chaotic process to generate a set of cell candidates in the current frame, then a labeled ant colony foraging process is modeled to construct an interframe matching between previously estimated cell states and current cell candidates through minimizing the optimal sub-pattern assignment metric for track (OSPA-T). The states of cells in the current frame are finally estimated using labeled ant colonies via a multi-Bernoulli parameter set approximated by individual food pheromone fields and heuristic information within the same region of support, the resulting trail pheromone fields over frames constitutes the cell lineage trees of the tracks. A four-stage track recovery strategy is proposed to monitor the history of all established tracks to reconstruct broken tracks in a computationally economic way. The labeling method used in this work is an improvement on previous techniques. The method has been evaluated on publicly available, challenging cell image sequences, and a satisfied performance improvement is achieved in contrast to the state-of-the-art methods.

Cell segmentation and tracking using CNN-based distance predictions and a graph-based matching strategy

The accurate segmentation and tracking of cells in microscopy image sequences is an important task in biomedical research, e.g., for studying the development of tissues, organs or entire organisms. However, the segmentation of touching cells in images with a low signal-to-noise-ratio is still a challenging problem. In this paper, we present a method for the segmentation of touching cells in microscopy images. By using a novel representation of cell borders, inspired by distance maps, our method is capable to utilize not only touching cells but also close cells in the training process. Furthermore, this representation is notably robust to annotation errors and shows promising results for the segmentation of microscopy images containing in the training data underrepresented or not included cell types. For the prediction of the proposed neighbor distances, an adapted U-Net convolutional neural network (CNN) with two decoder paths is used. In addition, we adapt a graph-based cell tracking algorithm to evaluate our proposed method on the task of cell tracking. The adapted tracking algorithm includes a movement estimation in the cost function to re-link tracks with missing segmentation masks over a short sequence of frames. Our combined tracking by detection method has proven its potential in the IEEE ISBI 2020 Cell Tracking Challenge (http://celltrackingchallenge.net/) where we achieved as team KIT-Sch-GE multiple top three rankings including two top performances using a single segmentation model for the diverse data sets.

Fully-automatic deep learning-based analysis for determination of the invasiveness of breast cancer cells in an acoustic trap

A single-beam acoustic trapping technique has been shown to be very useful for determining the invasiveness of suspended breast cancer cells in an acoustic trap with a manual calcium analysis method. However, for the rapid translation of the technology into the clinic, the development of an efficient/accurate analytical method is needed. We, therefore, develop a fully-automatic deep learning-based calcium image analysis algorithm for determining the invasiveness of suspended breast cancer cells using a single-beam acoustic trapping system. The algorithm allows to segment cells, find trapped cells, and quantify their calcium changes over time. For better segmentation of calcium fluorescent cells even with vague boundaries, a novel deep learning architecture with multi-scale/multi-channel convolution operations (MM-Net) is devised and constructed by a target inversion training method. The MM-Net outperforms other deep learning models in the cell segmentation. Also, a detection/quantification algorithm is developed and implemented to automatically determine the invasiveness of a trapped cell. For the evaluation of the algorithm, it is applied to quantify the invasiveness of breast cancer cells. The results show that the algorithm offers similar performance to the manual calcium analysis method for determining the invasiveness of cancer cells, suggesting that it may serve as a novel tool to automatically determine the invasiveness of cancer cells with high-efficiency.

Automated Cell Tracking Using Motion Prediction-Based Matching and Event Handling

Automated cell segmentation and tracking enables the quantification of static and dynamic cell characteristics and is significant for disease diagnosis, treatment, drug development, and other biomedical applications. This paper introduces a method for fully automated cell tracking, lineage construction, and quantification. Cell detection is performed in the joint spatio-temporal domain by a motion diffusion-based Partial Differential Equation (PDE) combined with energy minimizing active contours. In the tracking stage, we adopt a variational joint local-global optical flow technique to determine the motion vector field. We utilize the predicted cell motion jointly with spatial cell features to define a maximum likelihood criterion to find inter-frame cell correspondences assuming Markov dependency. We formulate cell tracking and cell event detection as a graph partitioning problem. We propose a solution obtained by minimization of a global cost function defined over the set of all cell tracks. We construct a cell lineage tree that represents the cell tracks and cell events. Finally, we compute morphological, motility, and diffusivity measures and validate cell tracking against manually generated reference standards. The automated tracking method applied to reference segmentation maps produces an average tracking accuracy score ( TRA) of 99 percent, and the fully automated segmentation and tracking system produces an average TRA of 89 percent.

Multiscale, multimodal analysis of tumor heterogeneity in IDH1 mutant vs wild-type diffuse gliomas

Glioma is recognized to be a highly heterogeneous CNS malignancy, whose diverse cellular composition and cellular interactions have not been well characterized. To gain new clinical- and biological-insights into the genetically-bifurcated IDH1 mutant (mt) vs wildtype (wt) forms of glioma, we integrated data from protein, genomic and MR imaging from 20 treatment-naïve glioma cases and 16 recurrent GBM cases. Multiplexed immunofluorescence (MxIF) was used to generate single cell data for 43 protein markers representing all cancer hallmarks, Genomic sequencing (exome and RNA (normal and tumor) and magnetic resonance imaging (MRI) quantitative features (protocols were T1-post, FLAIR and ADC) from whole tumor, peritumoral edema and enhancing core vs equivalent normal region were also collected from patients. Based on MxIF analysis, 85,767 cells (glioma cases) and 56,304 cells (GBM cases) were used to generate cell-level data for 24 biomarkers. K-means clustering was used to generate 7 distinct groups of cells with divergent biomarker profiles and deconvolution was used to assign RNA data into three classes. Spatial and molecular heterogeneity metrics were generated for the cell data. All features were compared between IDH mt and IDHwt patients and were finally combined to provide a holistic/integrated comparison. Protein expression by hallmark was generally lower in the IDHmt vs wt patients. Molecular and spatial heterogeneity scores for angiogenesis and cell invasion also differed between IDHmt and wt gliomas irrespective of prior treatment and tumor grade; these differences also persisted in the MR imaging features of peritumoral edema and contrast enhancement volumes. A coherent picture of enhanced angiogenesis in IDHwt tumors was derived from multiple platforms (genomic, proteomic and imaging) and scales from individual proteins to cell clusters and heterogeneity, as well as bulk tumor RNA and imaging features. Longer overall survival for IDH1mt glioma patients may reflect mutation-driven alterations in cellular, molecular, and spatial heterogeneity which manifest in discernable radiological manifestations.

Characterization of Biological Motion Using Motion Sensing Superpixels

Precise spatiotemporal regulation is the foundation for the healthy development and maintenance of living organisms. All cells must correctly execute their function in the right place at the right time. Cellular motion is thus an important dynamic readout of signaling in key disease-relevant molecular pathways. However despite the rapid advancement of imaging technology, a comprehensive quantitative description of motion imaged under different imaging modalities at all spatiotemporal scales; molecular, cellular and tissue-level is still lacking. Generally, cells move either 'individually' or 'collectively' as a group with nearby cells. Current computational tools specifically focus on one or the other regime, limiting their general applicability. To address this, we recently developed and reported a new computational framework, Motion Sensing Superpixels (MOSES). Incorporating the individual advantages of single cell trackers for individual cell and particle image velocimetry (PIV) for collective cell motion analyses, MOSES enables 'mesoscale' analysis of both single-cell and collective motion over arbitrarily long times. At the same time, MOSES readily complements existing single-cell tracking workflows with additional characterization of global motion patterns and interaction analysis between cells and also operates directly on PIV extracted motion fields to yield rich motion trajectories analogous for single-cell tracks suitable for high-throughput motion phenotyping. This protocol provides a step-by-step practical guide for those interested in applying MOSES to their own datasets. The protocol highlights the salient features of a MOSES analysis and demonstrates the ease-of-use and wide applicability of MOSES to biological imaging through demo experimental analyses with ready-to-use code snippets of four datasets from different microscope modalities; phase-contrast, fluorescent, lightsheet and intra-vital microscopy. In addition we discuss critical points of consideration in the analysis.

Motion sensing superpixels (MOSES) is a systematic computational framework to quantify and discover cellular motion phenotypes

Correct cell/cell interactions and motion dynamics are fundamental in tissue homeostasis, and defects in these cellular processes cause diseases. Therefore, there is strong interest in identifying factors, including drug candidates that affect cell/cell interactions and motion dynamics. However, existing quantitative tools for systematically interrogating complex motion phenotypes in timelapse datasets are limited. We present Motion Sensing Superpixels (MOSES), a computational framework that measures and characterises biological motion with a unique superpixel 'mesh' formulation. Using published datasets, MOSES demonstrates single-cell tracking capability and more advanced population quantification than Particle Image Velocimetry approaches. From > 190 co-culture videos, MOSES motion-mapped the interactions between human esophageal squamous epithelial and columnar cells mimicking the esophageal squamous-columnar junction, a site where Barrett's esophagus and esophageal adenocarcinoma often arise clinically. MOSES is a powerful tool that will facilitate unbiased, systematic analysis of cellular dynamics from high-content time-lapse imaging screens with little prior knowledge and few assumptions.

A simple computer vision pipeline reveals the effects of isolation on social interaction dynamics in Drosophila

Isolation profoundly influences social behavior in all animals. In humans, isolation has serious effects on health. Drosophila melanogaster is a powerful model to study small-scale, temporally-transient social behavior. However, longer-term analysis of large groups of flies is hampered by the lack of effective and reliable tools. We built a new imaging arena and improved the existing tracking algorithm to reliably follow a large number of flies simultaneously. Next, based on the automatic classification of touch and graph-based social network analysis, we designed an algorithm to quantify changes in the social network in response to prior social isolation. We observed that isolation significantly and swiftly enhanced individual and local social network parameters depicting near-neighbor relationships. We explored the genome-wide molecular correlates of these behavioral changes and found that whereas behavior changed throughout the six days of isolation, gene expression alterations occurred largely on day one. These changes occurred mostly in metabolic genes, and we verified the metabolic changes by showing an increase of lipid content in isolated flies. In summary, we describe a highly reliable tracking and analysis pipeline for large groups of flies that we use to unravel the behavioral, molecular and physiological impact of isolation on social network dynamics in Drosophila.

Social isolation severely affects the behavior and physiology of social animals, including humans. The fruit fly is a powerful model for studying the mechanisms of development, health and disease and is also used to study social behaviors such as mating and aggression. However, these studies are limited to examining few individuals for shorts amounts of time, due to the lack of effective computational tools for the analysis of large groups over prolonged time. To overcome this hurdle, we built a new behavioral arena and developed new software that accurately tracks many flies simultaneously over long time periods. The arena is cheap and easy to build and the software works with low resolution videos. Using these improved tools, we studied social isolation in groups of male flies. We found that isolation caused flies to form stronger interactions with neighboring flies in their social network. These behavioral changes were preceded by transient changes in the expression of metabolism genes and eventually resulted in isolated flies accumulating fat, as has been previously observed in studies in mice and humans. Our study opens the door for the use of fruit flies in future studies of social isolation.

A probabilistic approach to joint cell tracking and segmentation in high-throughput microscopy videos

We present a novel computational framework for the analysis of high-throughput microscopy videos of living cells. The proposed framework is generally useful and can be applied to different datasets acquired in a variety of laboratory settings. This is accomplished by tying together two fundamental aspects of cell lineage construction, namely cell segmentation and tracking, via a Bayesian inference of dynamic models. In contrast to most existing approaches, which aim to be general, no assumption of cell shape is made. Spatial, temporal, and cross-sectional variation of the analysed data are accommodated by two key contributions. First, time series analysis is exploited to estimate the temporal cell shape uncertainty in addition to cell trajectory. Second, a fast marching (FM) algorithm is used to integrate the inferred cell properties with the observed image measurements in order to obtain image likelihood for cell segmentation, and association. The proposed approach has been tested on eight different time-lapse microscopy data sets, some of which are high-throughput, demonstrating promising results for the detection, segmentation and association of planar cells. Our results surpass the state of the art for the Fluo-C2DL-MSC data set of the Cell Tracking Challenge (Maška et al., 2014).

Cell Membrane Tracking in Living Brain Tissue Using Differential Interference Contrast Microscopy

Differential interference contrast (DIC) microscopy is widely used for observing unstained biological samples that are otherwise optically transparent. Combining this optical technique with machine vision could enable the automation of many life science experiments; however, identifying relevant features under DIC is challenging. In particular, precise tracking of cell boundaries in a thick ( ) slice of tissue has not previously been accomplished. We present a novel deconvolution algorithm that achieves the state-of-the-art performance at identifying and tracking these membrane locations. Our proposed algorithm is formulated as a regularized least squares optimization that incorporates a filtering mechanism to handle organic tissue interference and a robust edge-sparsity regularizer that integrates dynamic edge tracking capabilities. As a secondary contribution, this paper also describes new community infrastructure in the form of a MATLAB toolbox for accurately simulating DIC microscopy images of in vitro brain slices. Building on existing DIC optics modeling, our simulation framework additionally contributes an accurate representation of interference from organic tissue, neuronal cell-shapes, and tissue motion due to the action of the pipette. This simulator allows us to better understand the image statistics (to improve algorithms), as well as quantitatively test cell segmentation and tracking algorithms in scenarios, where ground truth data is fully known.

Diatrack particle tracking software: Review of applications and performance evaluation

Object tracking is an instrumental tool supporting studies of cellular trafficking. There are three challenges in object tracking: the identification of targets; the precise determination of their position and boundaries; and the assembly of correct trajectories. This last challenge is particularly relevant when dealing with densely populated images with low signal-to-noise ratios-conditions that are often encountered in applications such as organelle tracking, virus particle tracking or single-molecule imaging. We have developed a set of methods that can handle a wide variety of signal complexities. They are compiled into a free software package called Diatrack. Here we review its main features and utility in a range of applications, providing a survey of the dynamic imaging field together with recommendations for effective use. The performance of our framework is shown to compare favorably to a wide selection of custom-developed algorithms, whether in terms of localization precision, processing speed or correctness of tracks.

Accurate Correspondence of Cone Photoreceptor Neurons in the Human Eye Using Graph Matching Applied to Longitudinal Adaptive Optics Images

Loss of cone photoreceptor neurons is a leading cause of many blinding retinal diseases. Direct visualization of these cells in the living human eye is now feasible using adaptive optics scanning light ophthalmoscopy (AOSLO). However, it remains challenging to monitor the state of specific cells across multiple visits, due to inherent eye-motion-based distortions that arise during data acquisition, artifacts when overlapping images are montaged, as well as substantial variability in the data itself. This paper presents an accurate graph matching framework that integrates (1) robust local intensity order patterns (LIOP) to describe neuron regions with illumination variation from different visits; (2) a sparse-coding based voting process to measure visual similarities of neuron pairs using LIOP descriptors; and (3) a graph matching model that combines both visual similarity and geometrical cone packing information to determine the correspondence of repeated imaging of cone photoreceptor neurons across longitudinal AOSLO datasets. The matching framework was evaluated on imaging data from ten subjects using a validation dataset created by removing 15% of the neurons from 713 neuron correspondences across image pairs. An overall matching accuracy of 98% was achieved. The framework was robust to differences in the amount of overlap between image pairs. Evaluation on a test dataset showed that the matching accuracy remained at 98% on approximately 3400 neuron correspondences, despite image quality degradation, illumination variation, large image deformation, and edge artifacts. These experimental results show that our graph matching approach can accurately identify cone photoreceptor neuron correspondences on longitudinal AOSLO images.

Trajectory energy minimization for cell growth tracking and genealogy analysis

Cell growth experiments with a microfluidic device produce large-scale time-lapse image data, which contain important information on cell growth and patterns in their genealogy. To extract such information, we propose a scheme to segment and track bacterial cells automatically. In contrast with most published approaches, which often split segmentation and tracking into two independent procedures, we focus on designing an algorithm that describes cell properties evolving between consecutive frames by feeding segmentation and tracking results from one frame to the next one. The cell boundaries are extracted by minimizing the distance regularized level set evolution (DRLSE) model. Each individual cell was identified and tracked by identifying cell septum and membrane as well as developing a trajectory energy minimization function along time-lapse series. Experiments show that by applying this scheme, cell growth and division can be measured automatically. The results show the efficiency of the approach when testing on different datasets while comparing with other existing algorithms. The proposed approach demonstrates great potential for large-scale bacterial cell growth analysis.

Network Flow Integer Programming to Track Elliptical Cells in Time-Lapse Sequences

We propose a novel approach to automatically tracking elliptical cell populations in time-lapse image sequences. Given an initial segmentation, we account for partial occlusions and overlaps by generating an over-complete set of competing detection hypotheses. To this end, we fit ellipses to portions of the initial regions and build a hierarchy of ellipses, which are then treated as cell candidates. We then select temporally consistent ones by solving to optimality an integer program with only one type of flow variables. This eliminates the need for heuristics to handle missed detections due to partial occlusions and complex morphology. We demonstrate the effectiveness of our approach on a range of challenging sequences consisting of clumped cells and show that it outperforms state-of-the-art techniques.

Robust cell tracking in epithelial tissues through identification of maximum common subgraphs

Tracking of cells in live-imaging microscopy videos of epithelial sheets is a powerful tool for investigating fundamental processes in embryonic development. Characterizing cell growth, proliferation, intercalation and apoptosis in epithelia helps us to understand how morphogenetic processes such as tissue invagination and extension are locally regulated and controlled. Accurate cell tracking requires correctly resolving cells entering or leaving the field of view between frames, cell neighbour exchanges, cell removals and cell divisions. However, current tracking methods for epithelial sheets are not robust to large morphogenetic deformations and require significant manual interventions. Here, we present a novel algorithm for epithelial cell tracking, exploiting the graph-theoretic concept of a 'maximum common subgraph' to track cells between frames of a video. Our algorithm does not require the adjustment of tissue-specific parameters, and scales in sub-quadratic time with tissue size. It does not rely on precise positional information, permitting large cell movements between frames and enabling tracking in datasets acquired at low temporal resolution due to experimental constraints such as phototoxicity. To demonstrate the method, we perform tracking on the Drosophila embryonic epidermis and compare cell-cell rearrangements to previous studies in other tissues. Our implementation is open source and generally applicable to epithelial tissues.

Convolutional Deep Belief Networks for Single-Cell/Object Tracking in Computational Biology and Computer Vision

In this paper, we propose deep architecture to dynamically learn the most discriminative features from data for both single-cell and object tracking in computational biology and computer vision. Firstly, the discriminative features are automatically learned via a convolutional deep belief network (CDBN). Secondly, we design a simple yet effective method to transfer features learned from CDBNs on the source tasks for generic purpose to the object tracking tasks using only limited amount of training data. Finally, to alleviate the tracker drifting problem caused by model updating, we jointly consider three different types of positive samples. Extensive experiments validate the robustness and effectiveness of the proposed method.

A Hybrid Approach for Segmentation and Tracking of Myxococcus Xanthus Swarms

Cell segmentation and motion tracking in time-lapse images are fundamental problems in computer vision, and are also crucial for various biomedical studies. Myxococcus xanthus is a type of rod-like cells with highly coordinated motion. The segmentation and tracking of M. xanthus are challenging, because cells may touch tightly and form dense swarms that are difficult to identify individually in an accurate manner. The known cell tracking approaches mainly fall into two frameworks, detection association and model evolution, each having its own advantages and disadvantages. In this paper, we propose a new hybrid framework combining these two frameworks into one and leveraging their complementary advantages. Also, we propose an active contour model based on the Ribbon Snake, which is seamlessly integrated with our hybrid framework. Evaluated by 10 different datasets, our approach achieves considerable improvement over the state-of-the-art cell tracking algorithms on identifying complete cell trajectories, and higher segmentation accuracy than performing segmentation in individual 2D images.

Deformable Graph Model for Tracking Epithelial Cell Sheets in Fluorescence Microscopy

We propose a novel method for tracking cells that are connected through a visible network of membrane junctions. Tissues of this form are common in epithelial cell sheets and resemble planar graphs where each face corresponds to a cell. We leverage this structure and develop a method to track the entire tissue as a deformable graph. This coupled model in which vertices inform the optimal placement of edges and vice versa captures global relationships between tissue components and leads to accurate and robust cell tracking. We compare the performance of our method with that of four reference tracking algorithms on four data sets that present unique tracking challenges. Our method exhibits consistently superior performance in tracking all cells accurately over all image frames, and is robust over a wide range of image intensity and cell shape profiles. This may be an important tool for characterizing tissues of this type especially in the field of developmental biology where automated cell analysis can help elucidate the mechanisms behind controlled cell-shape changes.

Curvelet initialized level set cell segmentation for touching cells in low contrast images

Cell segmentation is an important element of automatic cell analysis. This paper proposes a method to extract the cell nuclei and the cell boundaries of touching cells in low contrast images. First, the contrast of the low contrast cell images is improved by a combination of multiscale top hat filter and h-maxima. Then, a curvelet initialized level set method has been proposed to detect the cell nuclei and the boundaries. The image enhancement results have been verified using PSNR (Peak Signal to noise ratio) and the segmentation results have been verified using accuracy, sensitivity and precision metrics. The results show improved values of the performance metrics with the proposed method.

Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review

Digital pathology and microscopy image analysis is widely used for comprehensive studies of cell morphology or tissue structure. Manual assessment is labor intensive and prone to interobserver variations. Computer-aided methods, which can significantly improve the objectivity and reproducibility, have attracted a great deal of interest in recent literature. Among the pipeline of building a computer-aided diagnosis system, nucleus or cell detection and segmentation play a very important role to describe the molecular morphological information. In the past few decades, many efforts have been devoted to automated nucleus/cell detection and segmentation. In this review, we provide a comprehensive summary of the recent state-of-the-art nucleus/cell segmentation approaches on different types of microscopy images including bright-field, phase-contrast, differential interference contrast, fluorescence, and electron microscopies. In addition, we discuss the challenges for the current methods and the potential future work of nucleus/cell detection and segmentation.

Cell Tracking Accuracy Measurement Based on Comparison of Acyclic Oriented Graphs

Tracking motile cells in time-lapse series is challenging and is required in many biomedical applications. Cell tracks can be mathematically represented as acyclic oriented graphs. Their vertices describe the spatio-temporal locations of individual cells, whereas the edges represent temporal relationships between them. Such a representation maintains the knowledge of all important cellular events within a captured field of view, such as migration, division, death, and transit through the field of view. The increasing number of cell tracking algorithms calls for comparison of their performance. However, the lack of a standardized cell tracking accuracy measure makes the comparison impracticable. This paper defines and evaluates an accuracy measure for objective and systematic benchmarking of cell tracking algorithms. The measure assumes the existence of a ground-truth reference, and assesses how difficult it is to transform a computed graph into the reference one. The difficulty is measured as a weighted sum of the lowest number of graph operations, such as split, delete, and add a vertex and delete, add, and alter the semantics of an edge, needed to make the graphs identical. The measure behavior is extensively analyzed based on the tracking results provided by the participants of the first Cell Tracking Challenge hosted by the 2013 IEEE International Symposium on Biomedical Imaging. We demonstrate the robustness and stability of the measure against small changes in the choice of weights for diverse cell tracking algorithms and fluorescence microscopy datasets. As the measure penalizes all possible errors in the tracking results and is easy to compute, it may especially help developers and analysts to tune their algorithms according to their needs.

Analysis of in vivo single cell behavior by high throughput, human-in-the-loop segmentation of three-dimensional images

BACKGROUND

Analysis of single cells in their native environment is a powerful method to address key questions in developmental systems biology. Confocal microscopy imaging of intact tissues, followed by automatic image segmentation, provides a means to conduct cytometric studies while at the same time preserving crucial information about the spatial organization of the tissue and morphological features of the cells. This technique is rapidly evolving but is still not in widespread use among research groups that do not specialize in technique development, perhaps in part for lack of tools that automate repetitive tasks while allowing experts to make the best use of their time in injecting their domain-specific knowledge.

RESULTS

Here we focus on a well-established stem cell model system, the C. elegans gonad, as well as on two other model systems widely used to study cell fate specification and morphogenesis: the pre-implantation mouse embryo and the developing mouse olfactory epithelium. We report a pipeline that integrates machine-learning-based cell detection, fast human-in-the-loop curation of these detections, and running of active contours seeded from detections to segment cells. The procedure can be bootstrapped by a small number of manual detections, and outperforms alternative pieces of software we benchmarked on C. elegans gonad datasets. Using cell segmentations to quantify fluorescence contents, we report previously-uncharacterized cell behaviors in the model systems we used. We further show how cell morphological features can be used to identify cell cycle phase; this provides a basis for future tools that will streamline cell cycle experiments by minimizing the need for exogenous cell cycle phase labels.

CONCLUSIONS

High-throughput 3D segmentation makes it possible to extract rich information from images that are routinely acquired by biologists, and provides insights - in particular with respect to the cell cycle - that would be difficult to derive otherwise.

Automated profiling of individual cell-cell interactions from high-throughput time-lapse imaging microscopy in nanowell grids (TIMING)

MOTIVATION

There is a need for effective automated methods for profiling dynamic cell-cell interactions with single-cell resolution from high-throughput time-lapse imaging data, especially, the interactions between immune effector cells and tumor cells in adoptive immunotherapy.

RESULTS

Fluorescently labeled human T cells, natural killer cells (NK), and various target cells (NALM6, K562, EL4) were co-incubated on polydimethylsiloxane arrays of sub-nanoliter wells (nanowells), and imaged using multi-channel time-lapse microscopy. The proposed cell segmentation and tracking algorithms account for cell variability and exploit the nanowell confinement property to increase the yield of correctly analyzed nanowells from 45% (existing algorithms) to 98% for wells containing one effector and a single target, enabling automated quantification of cell locations, morphologies, movements, interactions, and deaths without the need for manual proofreading. Automated analysis of recordings from 12 different experiments demonstrated automated nanowell delineation accuracy >99%, automated cell segmentation accuracy >95%, and automated cell tracking accuracy of 90%, with default parameters, despite variations in illumination, staining, imaging noise, cell morphology, and cell clustering. An example analysis revealed that NK cells efficiently discriminate between live and dead targets by altering the duration of conjugation. The data also demonstrated that cytotoxic cells display higher motility than non-killers, both before and during contact.

CONTACT

broysam@central.uh.edu or nvaradar@central.uh.edu

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Multi-Target Tracking With Time-Varying Clutter Rate and Detection Profile: Application to Time-Lapse Cell Microscopy Sequences

Quantitative analysis of the dynamics of tiny cellular and sub-cellular structures, known as particles, in time-lapse cell microscopy sequences requires the development of a reliable multi-target tracking method capable of tracking numerous similar targets in the presence of high levels of noise, high target density, complex motion patterns and intricate interactions. In this paper, we propose a framework for tracking these structures based on the random finite set Bayesian filtering framework. We focus on challenging biological applications where image characteristics such as noise and background intensity change during the acquisition process. Under these conditions, detection methods usually fail to detect all particles and are often followed by missed detections and many spurious measurements with unknown and time-varying rates. To deal with this, we propose a bootstrap filter composed of an estimator and a tracker. The estimator adaptively estimates the required meta parameters for the tracker such as clutter rate and the detection probability of the targets, while the tracker estimates the state of the targets. Our results show that the proposed approach can outperform state-of-the-art particle trackers on both synthetic and real data in this regime.

Segmentation and tracking of adherens junctions in 3D for the analysis of epithelial tissue morphogenesis

Epithelial morphogenesis generates the shape of tissues, organs and embryos and is fundamental for their proper function. It is a dynamic process that occurs at multiple spatial scales from macromolecular dynamics, to cell deformations, mitosis and apoptosis, to coordinated cell rearrangements that lead to global changes of tissue shape. Using time lapse imaging, it is possible to observe these events at a system level. However, to investigate morphogenetic events it is necessary to develop computational tools to extract quantitative information from the time lapse data. Toward this goal, we developed an image-based computational pipeline to preprocess, segment and track epithelial cells in 4D confocal microscopy data. The computational pipeline we developed, for the first time, detects the adherens junctions of epithelial cells in 3D, without the need to first detect cell nuclei. We accentuate and detect cell outlines in a series of steps, symbolically describe the cells and their connectivity, and employ this information to track the cells. We validated the performance of the pipeline for its ability to detect vertices and cell-cell contacts, track cells, and identify mitosis and apoptosis in surface epithelia of Drosophila imaginal discs. We demonstrate the utility of the pipeline to extract key quantitative features of cell behavior with which to elucidate the dynamics and biomechanical control of epithelial tissue morphogenesis. We have made our methods and data available as an open-source multiplatform software tool called TTT (http://github.com/morganrcu/TTT)

Epithelia are the most common tissue type in multicellular organisms. Understanding processes that make them acquire their final shape has implications to pathologies such as cancer progression and birth defects such as spina bifida. During development, epithelial tissues are remodeled by mechanical forces applied at the Adherens Junctions (AJs). The AJs form a belt-like structure below the apical surface that functions to both mechanically link epithelial cells and enable cells to remodel their shape and contacts with their neighbors. In order to study epithelial morphogenesis in a quantitative and systematic way, it is necessary to measure the changes in the shape of the AJs over time. To this end we have built a complete computational pipeline to process image volumes generated by laser scanning confocal microscopy of epithelial tissues where the AJs have been marked with AJ proteins tagged with GFP. The system transforms input voxel intensity values into a symbolic description of the cells in the tissue, their connectivity and their temporal evolution, including the discovery of mitosis and apoptosis. As a proof of concept, we employed the data generated by our system to study aspects of morphogenesis of the Drosophila notum.

Cell-sensitive phase contrast microscopy imaging by multiple exposures

We propose a novel way of imaging live cells in a Petri dish by the phase contrast microscope. By taking multiple exposures of phase contrast microscopy images on the same cell dish, we estimate a cell-sensitive camera response function which responds to cells' irradiance signals but generates a constant on non-cell background signal. The result of this new microscopy imaging is visually superior quality, which reveals the appearance details of cells and suppresses background noise near zero. Using the cell-sensitive microscopy imaging, cells' original irradiance signals are restored from all exposures and the irradiance signals on non-cell background regions are restored as a uniform constant (i.e., the imaging system is sensitive to cells only but insensitive to non-cell background). The restored irradiance signals greatly facilitate the cell segmentation by simple thresholding. The experimental results validate that high quality cell segmentation can be achieved by our approach.

Segmentation and tracking of stem cells in time lapse microscopy to quantify dynamic behavioral changes during spheroid formation

Dynamic behavior of stem cells during in vitro development is diverse. Previous cell tracking studies have focused more on cell proliferation than on cell aggregation. However, the enhancement of cell proliferation in association with cell aggregation has been reported. In a previous study, we also demonstrated that the aggregation of adult human mesenchymal stem cells to form three-dimensional (3D) cellular spheroids helped maintain the expression of stemness marker genes in the cells. However, the dynamic behavioral changes triggered by spheroid formation remain to be investigated. A scheme of image processing techniques is proposed to meet this need. A hybrid-thresholding technique was first developed for efficient segmentation of cell clusters, after which a cell tracking method based on pair-matching with topological constraints was designed. Two morphological indices were derived to track the timing of 3D spheroid formation during the cellular aggregation process. Five cell motility indices measured from single cells and 3D spheroids were then compared. After confirmation of more than 90% correspondence between the results obtained by manual tracking and the proposed methods, an analysis of cellular behavior reveals a significant increase in motility in association with spheroid formation, consistent with a previous report that used a gene expression approach. This study proposed a systematic image analysis method to quantify the dynamic behavior of stem cells for stemness evaluation during cell culturing in vitro. Results demonstrated the validity of the developed platform in investigation of the dynamic behavior of cell aggregation in stem cell cultures in vitro.