Advanced level-set-based cell tracking in time-lapse fluorescence microscopy

A Weakly Supervised Learning Method for Cell Detection and Tracking Using Incomplete Initial Annotations

The automatic detection of cells in microscopy image sequences is a significant task in biomedical research. However, routine microscopy images with cells, which are taken during the process whereby constant division and differentiation occur, are notoriously difficult to detect due to changes in their appearance and number. Recently, convolutional neural network (CNN)-based methods have made significant progress in cell detection and tracking. However, these approaches require many manually annotated data for fully supervised training, which is time-consuming and often requires professional researchers. To alleviate such tiresome and labor-intensive costs, we propose a novel weakly supervised learning cell detection and tracking framework that trains the deep neural network using incomplete initial labels. Our approach uses incomplete cell markers obtained from fluorescent images for initial training on the Induced Pluripotent Stem (iPS) cell dataset, which is rarely studied for cell detection and tracking. During training, the incomplete initial labels were updated iteratively by combining detection and tracking results to obtain a model with better robustness. Our method was evaluated using two fields of the iPS cell dataset, along with the cell detection accuracy (DET) evaluation metric from the Cell Tracking Challenge (CTC) initiative, and it achieved 0.862 and 0.924 DET, respectively. The transferability of the developed model was tested using the public dataset FluoN2DH-GOWT1, which was taken from CTC; this contains two datasets with reference annotations. We randomly removed parts of the annotations in each labeled data to simulate the initial annotations on the public dataset. After training the model on the two datasets, with labels that comprise 10% cell markers, the DET improved from 0.130 to 0.903 and 0.116 to 0.877. When trained with labels that comprise 60% cell markers, the performance was better than the model trained using the supervised learning method. This outcome indicates that the model's performance improved as the quality of the labels used for training increased.

Biosensor Cell-Fit-HD4D for correlation of single-cell fate and microscale energy deposition in complex ion beams

We present a protocol for the biosensor Cell-Fit-HD^4D. It enables long-term monitoring and correlation of single-cell fate with subcellular-deposited energy of ionizing radiation. Cell fate tracking using widefield time-lapse microscopy is uncoupled in time from confocal ion track imaging. Registration of both image acquisition steps allows precise ion track assignment to cells and correlation with cellular readouts.

For complete details on the use and execution of this protocol, please refer to Niklas et al. (2022).

•

Cell-Fit-HD^4D is an in vitro biosensor for clinical ion beams

•

Cell-Fit-HD^4D combines single-cell dosimetry with individual tracking of tumor cells

•

Cell-Fit-HD^4D visualizes variability in radiation response in tumor cell population

Publisher’s note: Undertaking any experimental protocol requires adherence to local institutional guidelines for laboratory safety and ethics.

Demonstration of Neural Networks to Reconstruct Temperatures from Simulated Fluorescent Data Toward Use in Bio-microfluidics

Biological systems often have a narrow temperature range of operation, which require highly accurate spatially resolved temperature measurements, often near ±0.1 K. However, many temperature sensors cannot meet both accuracy and spatial distribution requirements, often because their accuracy is limited by data fitting and temperature reconstruction models. Machine learning algorithms have the potential to meet this need, but their usage in generating spatial distributions of temperature is severely lacking in the literature. This work presents the first instance of using neural networks to process fluorescent images to map the spatial distribution of temperature. Three standard network architectures were investigated using non-spatially resolved fluorescent thermometry (simply-connected feed-forward network) or during image or pixel identification (U-net and convolutional neural network, CNN). Simulated fluorescent images based on experimental data were generated based on known temperature distributions where Gaussian white noise with a standard deviation of ±0.1 K was added. The poor results from these standard networks motivated the creation of what is termed a moving CNN, with an RMSE error of ±0.23 K, where the elements of the matrix represent the neighboring pixels. Finally, the performance of this MCNN is investigated when trained and applied to three distinctive temperature distributions characteristic within microfluidic devices, where the fluorescent image is simulated at either three or five different wavelengths. The results demonstrate that having a minimum of 10 ^3.5 data points per temperature and the broadest range of temperatures during training provides temperature predictions nearest to the true temperatures of the images, with a minimum RMSE of ±0.15 K. When compared to traditional curve fitting techniques, this work demonstrates that greater accuracy when spatially mapping temperature from fluorescent images can be achieved when using convolutional neural networks.

Adaptive tracking algorithm for trajectory analysis of cells and layer-by-layer assessment of motility dynamics

Tracking biological objects such as cells or subcellular components imaged with time-lapse microscopy enables us to understand the molecular principles about the dynamics of cell behaviors. However, automatic object detection, segmentation and extracting trajectories remain as a rate-limiting step due to intrinsic challenges of video processing. This paper presents an adaptive tracking algorithm (Adtari) that automatically finds the optimum search radius and cell linkages to determine trajectories in consecutive frames. A critical assumption in most tracking studies is that displacement remains unchanged throughout the movie and cells in a few frames are usually analyzed to determine its magnitude. Tracking errors and inaccurate association of cells may occur if the user does not correctly evaluate the value or prior knowledge is not present on cell movement. The key novelty of our method is that minimum intercellular distance and maximum displacement of cells between frames are dynamically computed and used to determine the threshold distance. Since the space between cells is highly variable in a given frame, our software recursively alters the magnitude to determine all plausible matches in the trajectory analysis. Our method therefore eliminates a major preprocessing step where a constant distance was used to determine the neighbor cells in tracking methods. Cells having multiple overlaps and splitting events were further evaluated by using the shape attributes including perimeter, area, ellipticity and distance. The features were applied to determine the closest matches by minimizing the difference in their magnitudes. Finally, reporting section of our software were used to generate instant maps by overlaying cell features and trajectories. Adtari was validated by using videos with variable signal-to-noise, contrast ratio and cell density. We compared the adaptive tracking with constant distance and other methods to evaluate performance and its efficiency. Our algorithm yields reduced mismatch ratio, increased ratio of whole cell track, higher frame tracking efficiency and allows layer-by-layer assessment of motility to characterize single-cells. Adaptive tracking provides a reliable, accurate, time efficient and user-friendly open source software that is well suited for analysis of 2D fluorescence microscopy video datasets.

CancerCellTracker: a brightfield time-lapse microscopy framework for cancer drug sensitivity estimation

MOTIVATION

Time-lapse microscopy is a powerful technique that relies on images of live cells cultured ex vivo that are captured at regular intervals of time to describe and quantify their behavior under certain experimental conditions. This imaging method has great potential in advancing the field of precision oncology by quantifying the response of cancer cells to various therapies and identifying the most efficacious treatment for a given patient. Digital image processing algorithms developed so far require high-resolution images involving very few cells originating from homogeneous cell line populations. We propose a novel framework that tracks cancer cells to capture their behavior and quantify cell viability to inform clinical decisions in a high-throughput manner.

RESULTS

The brightfield microscopy images a large number of patient-derived cells in an ex vivo reconstruction of the tumor microenvironment treated with 31 drugs for up to 6 days. We developed a robust and user-friendly pipeline CancerCellTracker that detects cells in co-culture, tracks these cells across time and identifies cell death events using changes in cell attributes. We validated our computational pipeline by comparing the timing of cell death estimates by CancerCellTracker from brightfield images and a fluorescent channel featuring ethidium homodimer. We benchmarked our results using a state-of-the-art algorithm implemented in ImageJ and previously published in the literature. We highlighted CancerCellTracker's efficiency in estimating the percentage of live cells in the presence of bone marrow stromal cells.

AVAILABILITY AND IMPLEMENTATION

https://github.com/compbiolabucf/CancerCellTracker.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Efficient automatic 3D segmentation of cell nuclei for high-content screening

BACKGROUND

High-content screening (HCS) is a pre-clinical approach for the assessment of drug efficacy. On modern platforms, it involves fluorescent image capture using three-dimensional (3D) scanning microscopy. Segmentation of cell nuclei in 3D images is an essential prerequisite to quantify captured fluorescence in cells for screening. However, this segmentation is challenging due to variabilities in cell confluency, drug-induced alterations in cell morphology, and gradual degradation of fluorescence with the depth of scanning. Despite advances in algorithms for segmenting nuclei for HCS, robust 3D methods that are insensitive to these conditions are still lacking.

RESULTS

We have developed an algorithm which first generates a 3D nuclear mask in the original images. Next, an iterative 3D marker-controlled watershed segmentation is applied to downsized images to segment adjacent nuclei under the mask. In the last step, borders of segmented nuclei are adjusted in the original images based on local nucleus and background intensities. The method was developed using a set of 10 3D images. Extensive tests on a separate set of 27 3D images containing 2,367 nuclei demonstrated that our method, in comparison with 6 reference methods, achieved the highest precision (PR = 0.97), recall (RE = 0.88) and F1-score (F1 = 0.93) of nuclei detection. The Jaccard index (JI = 0.83), which reflects the accuracy of nuclei delineation, was similar to that yielded by all reference approaches. Our method was on average more than twice as fast as the reference method that produced the best results. Additional tests carried out on three stacked 3D images comprising heterogenous nuclei yielded average PR = 0.96, RE = 0.84, F1 = 0.89, and JI = 0.80.

CONCLUSIONS

The high-performance metrics yielded by the proposed approach suggest that it can be used to reliably delineate nuclei in 3D images of monolayered and stacked cells exposed to cytotoxic drugs.

TAIM: Tool for Analyzing Root Images to Calculate the Infection Rate of Arbuscular Mycorrhizal Fungi

Arbuscular mycorrhizal fungi (AMF) infect plant roots and are hypothesized to improve plant growth. Recently, AMF is now available for axenic culture. Therefore, AMF is expected to be used as a microbial fertilizer. To evaluate the usefulness of AMF as a microbial fertilizer, we need to investigate the relationship between the degree of root colonization of AMF and plant growth. The method popularly used for calculation of the degree of root colonization, termed the magnified intersections method, is performed manually and is too labor-intensive to enable an extensive survey to be undertaken. Therefore, we automated the magnified intersections method by developing an application named "Tool for Analyzing root images to calculate the Infection rate of arbuscular Mycorrhizal fungi: TAIM." TAIM is a web-based application that calculates the degree of AMF colonization from images using automated computer vision and pattern recognition techniques. Experimental results showed that TAIM correctly detected sampling areas for calculation of the degree of infection and classified the sampling areas with 87.4% accuracy. TAIM is publicly accessible at http://taim.imlab.jp/.

Biosensor for deconvolution of individual cell fate in response to ion beam irradiation

Clonogenic survival assay constitutes the gold standard method for quantifying radiobiological effects. However, it neglects cellular radiation response variability and heterogeneous energy deposition by ion beams on the microscopic scale. We introduce "Cell-Fit-HD4D" a biosensor that enables a deconvolution of individual cell fate in response to the microscopic energy deposition as visualized by optical microscopy. Cell-Fit-HD4D enables single-cell dosimetry in clinically relevant complex radiation fields by correlating microscopic beam parameters with biological endpoints. Decrypting the ion beam's energy deposition and molecular effects at the single-cell level has the potential to improve our understanding of radiobiological dose concepts as well as radiobiological study approaches in general.

Tracking cell lineages in 3D by incremental deep learning

Deep learning is emerging as a powerful approach for bioimage analysis. Its use in cell tracking is limited by the scarcity of annotated data for the training of deep-learning models. Moreover, annotation, training, prediction, and proofreading currently lack a unified user interface. We present ELEPHANT, an interactive platform for 3D cell tracking that addresses these challenges by taking an incremental approach to deep learning. ELEPHANT provides an interface that seamlessly integrates cell track annotation, deep learning, prediction, and proofreading. This enables users to implement cycles of incremental learning starting from a few annotated nuclei. Successive prediction-validation cycles enrich the training data, leading to rapid improvements in tracking performance. We test the software's performance against state-of-the-art methods and track lineages spanning the entire course of leg regeneration in a crustacean over 1 week (504 timepoints). ELEPHANT yields accurate, fully-validated cell lineages with a modest investment in time and effort.

A review for Cell and Particle Tracking on Microscopy Images using Algorithms and Deep Learning Technologies

Time-lapse microscopy images generated by biological experiments have been widely used for observing target activities, such as the motion trajectories and survival states. Based on these observations, biologists can conclude experimental results or present new hypotheses for several biological applications, i.e. virus research or drug design. Many methods or tools have been proposed in the past to observe cell and particle activities, which are defined as single cell tracking and single particle tracking problems, by using algorithms and deep learning technologies. In this article, a review for these works is presented in order to summarize the past methods and research topics at first, then points out the problems raised by these works, and finally proposes future research directions. The contributions of this article will help researchers to understand past development trends and further propose innovative technologies.

DMNet: Dual-Stream Marker Guided Deep Network for Dense Cell Segmentation and Lineage Tracking

Accurate segmentation and tracking of cells in microscopy image sequences is extremely beneficial in clinical diagnostic applications and biomedical research. A continuing challenge is the segmentation of dense touching cells and deforming cells with indistinct boundaries, in low signal-to-noise-ratio images. In this paper, we present a dual-stream marker-guided network (DMNet) for segmentation of touching cells in microscopy videos of many cell types. DMNet uses an explicit cell marker-detection stream, with a separate mask-prediction stream using a distance map penalty function, which enables supervised training to focus attention on touching and nearby cells. For multi-object cell tracking we use M2Track tracking-by-detection approach with multi-step data association. Our M2Track with mask overlap includes short term track-to-cell association followed by track-to-track association to re-link tracklets with missing segmentation masks over a short sequence of frames. Our combined detection, segmentation and tracking algorithm has proven its potential on the IEEE ISBI 2021 6th Cell Tracking Challenge (CTC-6) where we achieved multiple top three rankings for diverse cell types. Our team name is MU-Ba-US, and the implementation of DMNet is available at, http://celltrackingchallenge.net/participants/MU-Ba-US/.

A Joint Tracking Approach via Ant Colony Evolution for Quantitative Cell Cycle Analysis

In this paper, we use an ant colony heuristic method to tackle the integration of data association and state estimation in the presence of cell mitosis, morphological change and uncertainty of measurement. Our approach first models the scouting behavior of an unlabeled ant colony as a chaotic process to generate a set of cell candidates in the current frame, then a labeled ant colony foraging process is modeled to construct an interframe matching between previously estimated cell states and current cell candidates through minimizing the optimal sub-pattern assignment metric for track (OSPA-T). The states of cells in the current frame are finally estimated using labeled ant colonies via a multi-Bernoulli parameter set approximated by individual food pheromone fields and heuristic information within the same region of support, the resulting trail pheromone fields over frames constitutes the cell lineage trees of the tracks. A four-stage track recovery strategy is proposed to monitor the history of all established tracks to reconstruct broken tracks in a computationally economic way. The labeling method used in this work is an improvement on previous techniques. The method has been evaluated on publicly available, challenging cell image sequences, and a satisfied performance improvement is achieved in contrast to the state-of-the-art methods.

Extracting neuronal activity signals from microscopy recordings of contractile tissue using B-spline Explicit Active Surfaces (BEAS) cell tracking

Ca2+ imaging is a widely used microscopy technique to simultaneously study cellular activity in multiple cells. The desired information consists of cell-specific time series of pixel intensity values, in which the fluorescence intensity represents cellular activity. For static scenes, cellular signal extraction is straightforward, however multiple analysis challenges are present in recordings of contractile tissues, like those of the enteric nervous system (ENS). This layer of critical neurons, embedded within the muscle layers of the gut wall, shows optical overlap between neighboring neurons, intensity changes due to cell activity, and constant movement. These challenges reduce the applicability of classical segmentation techniques and traditional stack alignment and regions-of-interest (ROIs) selection workflows. Therefore, a signal extraction method capable of dealing with moving cells and is insensitive to large intensity changes in consecutive frames is needed. Here we propose a b-spline active contour method to delineate and track neuronal cell bodies based on local and global energy terms. We develop both a single as well as a double-contour approach. The latter takes advantage of the appearance of GCaMP expressing cells, and tracks the nucleus' boundaries together with the cytoplasmic contour, providing a stable delineation of neighboring, overlapping cells despite movement and intensity changes. The tracked contours can also serve as landmarks to relocate additional and manually-selected ROIs. This improves the total yield of efficacious cell tracking and allows signal extraction from other cell compartments like neuronal processes. Compared to manual delineation and other segmentation methods, the proposed method can track cells during large tissue deformations and high-intensity changes such as during neuronal firing events, while preserving the shape of the extracted Ca2+ signal. The analysis package represents a significant improvement to available Ca2+ imaging analysis workflows for ENS recordings and other systems where movement challenges traditional Ca2+ signal extraction workflows.

Automatic wavelet-based 3D nuclei segmentation and analysis for multicellular embryo quantification

Identification of individual cells in tissues, organs, and in various developing systems is a well-studied problem because it is an essential part of objectively analyzing quantitative images in numerous biological contexts. We developed a size-dependent wavelet-based segmentation method that provides robust segmentation without any preprocessing, filtering or fine-tuning steps, and is robust to the signal-to-noise ratio. The wavelet-based method achieves robust segmentation results with respect to True Positive rate, Precision, and segmentation accuracy compared with other commonly used methods. We applied the segmentation program to zebrafish embryonic development IN TOTO for nuclei segmentation, image registration, and nuclei shape analysis. These new approaches to segmentation provide a means to carry out quantitative patterning analysis with single-cell precision throughout three dimensional tissues and embryos and they have a high tolerance for non-uniform and noisy image data sets.

DeepSeed Local Graph Matching for Densely Packed Cells Tracking

The tracking of densely packed plant cells across microscopy image sequences is very challenging, because their appearance change greatly over time. A local graph matching algorithm was proposed to track such cells by exploiting the tight spatial topology of neighboring cells, and then an iterative searching strategy was used to grow the correspondence from a seed cell pair. Thus, the performance of the existing tracking approach heavily relies on the robustness of finding seed cell pair. However, the existing local graph matching algorithm cannot guarantee the correctness of the seed cell pair, especially in unregistered image sequences or image sequences with large time intervals. In this paper, we propose a DeepSeed local graph matching model to find seed cell pair robustly, by combining local graph matching and CNN-based similarity learning, which uses cells' spatial-temporal contextual information and cell pairs' similarity information. The CNN-based similarity learning is designed to learn cells' deep feature and measure cell pairs' similarity. Compared with the existing plant cell matching methods, the experimental results show that the DeepSeed local graph matching method can track most cells in unregistered image sequences. Moreover, the DeepSeed tracking algorithm can accurately track cells across image sequences with large time intervals.

Physically asymmetric division of the C. elegans zygote ensures invariably successful embryogenesis

Asymmetric divisions that yield daughter cells of different sizes are frequent during early embryogenesis, but the importance of such a physical difference for successful development remains poorly understood. Here, we investigated this question using the first division of Caenorhabditis elegans embryos, which yields a large AB cell and a small P1 cell. We equalized AB and P1 sizes using acute genetic inactivation or optogenetic manipulation of the spindle positioning protein LIN-5. We uncovered that only some embryos tolerated equalization, and that there was a size asymmetry threshold for viability. Cell lineage analysis of equalized embryos revealed an array of defects, including faster cell cycle progression in P1 descendants, as well as defects in cell positioning, division orientation, and cell fate. Moreover, equalized embryos were more susceptible to external compression. Overall, we conclude that unequal first cleavage is essential for invariably successful embryonic development of C. elegans.

Iron loading is a prominent feature of activated microglia in Alzheimer's disease patients

Brain iron accumulation has been found to accelerate disease progression in amyloid-β(Aβ) positive Alzheimer patients, though the mechanism is still unknown. Microglia have been identified as key players in the disease pathogenesis, and are highly reactive cells responding to aberrations such as increased iron levels. Therefore, using histological methods, multispectral immunofluorescence and an automated in-house developed microglia segmentation and analysis pipeline, we studied the occurrence of iron-accumulating microglia and the effect on its activation state in human Alzheimer brains. We identified a subset of microglia with increased expression of the iron storage protein ferritin light chain (FTL), together with increased Iba1 expression, decreased TMEM119 and P2RY12 expression. This activated microglia subset represented iron-accumulating microglia and appeared morphologically dystrophic. Multispectral immunofluorescence allowed for spatial analysis of FTL+Iba1+-microglia, which were found to be the predominant Aβ-plaque infiltrating microglia. Finally, an increase of FTL+Iba1+-microglia was seen in patients with high Aβ load and Tau load. These findings suggest iron to be taken up by microglia and to influence the functional phenotype of these cells, especially in conjunction with Aβ.

A Curvature-Enhanced Random Walker Segmentation Method for Detailed Capture of 3D Cell Surface Membranes

High-resolution 3D microscopy is a fast advancing field and requires new techniques in image analysis to handle these new datasets. In this work, we focus on detailed 3D segmentation of Dictyostelium cells undergoing macropinocytosis captured on an iSPIM microscope. We propose a novel random walker-based method with a curvature-based enhancement term, with the aim of capturing fine protrusions, such as filopodia and deep invaginations, such as macropinocytotic cups, on the cell surface. We tested our method on both real and synthetic 3D image volumes, demonstrating that the inclusion of the curvature enhancement term can improve the segmentation of the aforementioned features. We show that our method performs better than other state of the art segmentation methods in 3D images of Dictyostelium cells, and performs competitively against CNN-based methods in two Cell Tracking Challenge datasets, demonstrating the ability to obtain accurate segmentations without the requirement of large training datasets. We also present an automated seeding method for microscopy data, which, combined with the curvature-enhanced random walker method, enables the segmentation of large time series with minimal input from the experimenter.

Tracking Objects Based on Multiple Particle Filters for Multipart Combined Moving Directions Information

Object tracking is an important procedure in the computer vision field as it estimates the position, size, and state of an object along the video's timeline. Although many algorithms were proposed with high accuracy, object tracking in diverse contexts is still a challenging problem. The paper presents some methods to track the movement of two types of objects: arbitrary objects and humans. Both problems estimate the state density function of an object using particle filters. For the videos of a static or relatively static camera, we adjusted the state transition model by integrating the movement direction of the object. Also, we propose that partitioning the object needs tracking. To track the human, we partitioned the human into N parts and, then, tracked each part. During tracking, if a part deviated from the object, it was corrected by centering rotation, and the part was, then, combined with other parts.

Sphere estimation network: three-dimensional nuclei detection of fluorescence microscopy images

Purpose: Fluorescence microscopy visualizes three-dimensional subcellular structures in tissue with two-photon microscopy achieving deeper penetration into tissue. Nuclei detection, which is essential for analyzing tissue for clinical and research purposes, remains a challenging problem due to the spatial variability of nuclei. Recent advancements in deep learning techniques have enabled the analysis of fluorescence microscopy data to localize and segment nuclei. However, these localization or segmentation techniques would require additional steps to extract characteristics of nuclei. We develop a 3D convolutional neural network, called Sphere Estimation Network (SphEsNet), to extract characteristics of nuclei without any postprocessing steps. Approach: To simultaneously estimate the center locations of nuclei and their sizes, SphEsNet is composed of two branches to localize nuclei center coordinates and to estimate their radii. Synthetic microscopy volumes automatically generated using a spatially constrained cycle-consistent adversarial network are used for training the network because manually generating 3D real ground truth volumes would be extremely tedious. Results: Three SphEsNet models based on the size of nuclei were trained and tested on five real fluorescence microscopy data sets from rat kidney and mouse intestine. Our method can successfully detect nuclei in multiple locations with various sizes. In addition, our method was compared with other techniques and outperformed them based on object-level precision, recall, and F 1 score. Our model achieved 89.90% for F 1 score. Conclusions: SphEsNet can simultaneously localize nuclei and estimate their size without additional steps. SphEsNet can be potentially used to extract more information from nuclei in fluorescence microscopy images.

Optimising adjacent membrane segmentation and parameterisation in multicellular aggregates by piecewise active contours

In fluorescence microscopy imaging, the segmentation of adjacent cell membranes within cell aggregates, multicellular samples, tissue, organs, or whole organisms remains a challenging task. The lipid bilayer is a very thin membrane when compared to the wavelength of photons in the visual spectra. Fluorescent molecules or proteins used for labelling membranes provide a limited signal intensity, and light scattering in combination with sample dynamics during in vivo imaging lead to poor or ambivalent signal patterns that hinder precise localisation of the membrane sheets. In the proximity of cells, membranes approach and distance each other. Here, the presence of membrane protrusions such as blebs; filopodia and lamellipodia; microvilli; or membrane vesicle trafficking, lead to a plurality of signal patterns, and the accurate localisation of two adjacent membranes becomes difficult. Several computational methods for membrane segmentation have been introduced. However, few of them specifically consider the accurate detection of adjacent membranes. In this article we present ALPACA (ALgorithm for Piecewise Adjacent Contour Adjustment), a novel method based on 2D piecewise parametric active contours that allows: (i) a definition of proximity for adjacent contours, (ii) a precise detection of adjacent, nonadjacent, and overlapping contour sections, (iii) the definition of a polyline for an optimised shared contour within adjacent sections and (iv) a solution for connecting adjacent and nonadjacent sections under the constraint of preserving the inherent cell morphology. We show that ALPACA leads to a precise quantification of adjacent and nonadjacent membrane zones in regular hexagons and live image sequences of cells of the parapineal organ during zebrafish embryo development. The algorithm detects and corrects adjacent, nonadjacent, and overlapping contour sections within a selected adjacency distance d, calculates shared contour sections for neighbouring cells with minimum alterations of the contour characteristics, and presents piecewise active contour solutions, preserving the contour shape and the overall cell morphology. ALPACA quantifies adjacent contours and can improve the meshing of 3D surfaces, the determination of forces, or tracking of contours in combination with previously published algorithms. We discuss pitfalls, strengths, and limits of our approach, and present a guideline to take the best decision for varying experimental conditions for in vivo microscopy.

Deep Reinforcement Learning for Data Association in Cell Tracking

Accurate target detection and association are vital for the development of reliable target tracking, especially for cell tracking based on microscopy images due to the similarity of cells. We propose a deep reinforcement learning method to associate the detected targets between frames. According to the dynamic model of each target, the cost matrix is produced by conjointly considering various features of targets and then used as the input of a neural network. The proposed neural network is trained using reinforcement learning to predict a distribution over the association solution. Furthermore, we design a residual convolutional neural network that results in more efficient learning. We validate our method on two applications: the multiple target tracking simulation and the ISBI cell tracking. The results demonstrate that our approach based on reinforcement learning techniques could effectively track targets following different motion patterns and show competitive results.

A microcarrier-based spheroid 3D invasion assay to monitor dynamic cell movement in extracellular matrix

Background

Cell invasion through extracellular matrix (ECM) is a critical step in tumor metastasis. To study cell invasion in vitro, the internal microenvironment can be simulated via the application of 3D models.

Results

This study presents a method for 3D invasion examination using microcarrier-based spheroids. Cell invasiveness can be evaluated by quantifying cell dispersion in matrices or tracking cell movement through time-lapse imaging. It allows measuring of cell invasion and monitoring of dynamic cell behavior in three dimensions. Here we show different invasive capacities of several cell types using this method. The content and concentration of matrices can influence cell invasion, which should be optimized before large scale experiments. We also introduce further analysis methods of this 3D invasion assay, including manual measurements and homemade semi-automatic quantification. Finally, our results indicate that the position of spheroids in a matrix has a strong impact on cell moving paths, which may be easily overlooked by researchers and may generate false invasion results.

Conclusions

In all, the microcarrier-based spheroids 3D model allows exploration of adherent cell invasion in a fast and highly reproducible way, and provides informative results on dynamic cell behavior in vitro.

A novel machine learning based approach for iPS progenitor cell identification

Identification of induced pluripotent stem (iPS) progenitor cells, the iPS forming cells in early stage of reprogramming, could provide valuable information for studying the origin and underlying mechanism of iPS cells. However, it is very difficult to identify experimentally since there are no biomarkers known for early progenitor cells, and only about 6 days after reprogramming initiation, iPS cells can be experimentally determined via fluorescent probes. What is more, the ratio of progenitor cells during early reprograming period is below 5%, which is too low to capture experimentally in the early stage. In this paper, we propose a novel computational approach for the identification of iPS progenitor cells based on machine learning and microscopic image analysis. Firstly, we record the reprogramming process using a live cell imaging system after 48 hours of infection with retroviruses expressing Oct4, Sox2 and Klf4, later iPS progenitor cells and normal murine embryonic fibroblasts (MEFs) within 3 to 5 days after infection are labeled by retrospectively tracing the time-lapse microscopic image. We then calculate 11 types of cell morphological and motion features such as area, speed, etc., and select best time windows for modeling and perform feature selection. Finally, a prediction model using XGBoost is built based on the selected six types of features and best time windows. Our model allows several missing values/frames in the sample datasets, thus it is applicable to a wide range of scenarios. Cross-validation, holdout validation and independent test experiments show that the minimum precision is above 52%, that is, the ratio of predicted progenitor cells within 3 to 5 days after viral infection is above 52%. The results also confirm that the morphology and motion pattern of iPS progenitor cells is different from that of normal MEFs, which helps with the machine learning methods for iPS progenitor cell identification.

Identification of induced pluripotent stem (iPS) progenitor cells could provide valuable information for studying the origin and underlying mechanism of iPS cells. However, it is very difficult to identify experimentally since there are no biomarkers known for early progenitor cells, and only after about 6 days of induction, iPS cells can be experimentally determined via fluorescent probes. What is more, the percentage of the progenitor cells during the early induction period is below 5%, too low to capture experimentally in early stage. In this work, we proposed an approach for the identification of iPS progenitor cells, the iPS forming cells, based on machine learning and microscopic image analysis. The aim is to help biologists to enrich iPS progenitor cells during the early stage of induction, which allows experimentalists to select iPS progenitor cells with much higher probability, and furthermore to study the biomarkers which trigger the reprogramming process.

An Ant Colony Inspired Multi-Bernoulli Filter for Cell Tracking in Time-Lapse Microscopy Sequences

The analysis of the dynamic behavior of cells in time-lapse microscopy sequences requires the development of reliable and automatic tracking methods capable of estimating individual cell states and delineating the lineage trees corresponding to the tracks. In this paper, we propose a novel approach, i.e., an ant colony inspired multi-Bernoulli filter, to handle the tracking of a collection of cells within which mitosis, morphological change and erratic dynamics occur. The proposed technique treats each ant colony as an independent one in an ant society, and the existence probability of an ant colony and its density distribution approximation are derived from the individual pheromone field and the corresponding heuristic information for the approximation to the multi-Bernoulli parameters. To effectively guide ant foraging between consecutive frames, a dual prediction mechanism is proposed for the ant colony and its pheromone field. The algorithm performance is tested on challenging datasets with varying population density, frequent cell mitosis and uneven motion over time, demonstrating that the algorithm outperforms recently reported approaches.

Deep-Learning-Based Segmentation of Small Extracellular Vesicles in Transmission Electron Microscopy Images

Small extracellular vesicles (sEVs) are cell-derived vesicles of nanoscale size (~30-200 nm) that function as conveyors of information between cells, reflecting the cell of their origin and its physiological condition in their content. Valuable information on the shape and even on the composition of individual sEVs can be recorded using transmission electron microscopy (TEM). Unfortunately, sample preparation for TEM image acquisition is a complex procedure, which often leads to noisy images and renders automatic quantification of sEVs an extremely difficult task. We present a completely deep-learning-based pipeline for the segmentation of sEVs in TEM images. Our method applies a residual convolutional neural network to obtain fine masks and use the Radon transform for splitting clustered sEVs. Using three manually annotated datasets that cover a natural variability typical for sEV studies, we show that the proposed method outperforms two different state-of-the-art approaches in terms of detection and segmentation performance. Furthermore, the diameter and roundness of the segmented vesicles are estimated with an error of less than 10%, which supports the high potential of our method in biological applications.

A segmentation method combining probability map and boundary based on multiple fully convolutional networks and repetitive training

Cell nuclei image segmentation technology can help researchers observe each cell's stress response to drug treatment. However, it is still a challenge to accurately segment the adherent cell nuclei. At present, image segmentation based on a fully convolutional network (FCN) is attracting researchers' attention. We propose a multiple FCN architecture and repetitive training (M-FCN-RT) method to learn features of cell nucleus images. In M-FCN-RT, the multiple FCN (M-FCN) architecture is composed of several single FCNs (S-FCNs) with the same structure, and each FCN is used to learn the specific features of image datasets. In this paper, the M-FCN contains three FCNs; FCN_1-2, FCN₃ and FCN_B. FCN_1-2 and FCN₃ are respectively used to learn the spatial features of cell nuclei for generating probability maps to indicate nucleus regions of an image; FCN_B (boundary FCN) is used to learn the edge features of cell nuclei for generating the nucleus boundary. For the training of each FCN, we propose a repetitive training (RT) method to improve the classification accuracy of the model. To segment cell nuclei, we finally propose an algorithm combining the probability map and boundary (PMB) to segment the adherent nuclei. This paper uses a public opening nucleus image dataset to train, verify and evaluate the proposed M-FCN-RT and PMB methods. Our M-FCN-RT method achieves a high Dice similarity coefficient (DSC) of 92.11%, 95.64% and 87.99% on the three types of sub-datasets respectively for probability maps. In addition, segmentation experimental results show the PMB method is more effective and efficient compared with other methods.

Segmentation of yeast cell's bright-field image with an edge-tracing algorithm

Phenotype analysis of yeast cell requires high-throughput imaging and automatic analysis of abundant image data. At first, each cell needs to be segmented and labeled in the bright-field images. However, the ambiguous boundary of bright-field yeast cell images leads to the failure of traditional segmentation algorithms. We propose a segmentation algorithm based on the morphological characteristics of yeast cells. Seed points are first identified along the cell contour and then connected by an edge tracing approach. In this way, "ill-detected" noise points are removed so that edges of yeast cells can be successfully extracted in bright-field images with sparsely distributed cells. In densely packed images, yeast cells with normal morphology can also be correctly segmented and labeled.

Cell Population Tracking in a Honeycomb Structure Using an IMM Filter Based 3D Local Graph Matching Model

Developing algorithms for plant cell population tracking is very critical for the modeling of plant cell growth pattern and gene expression dynamics. The tracking of plant cells in microscopic image stacks is very challenging for several reasons: (1) plant cells are densely packed in a specific honeycomb structure; (2) they are frequently dividing; and (3) they are imaged in different layers within 3D image stacks. Based on an existing 2D local graph matching algorithm, this paper focuses on building a 3D plant cell matching model, by exploiting the cells' 3D spatiotemporal context. Furthermore, the Interacting Multi-Model filter (IMM) is combined with the 3D local graph matching model to track the plant cell population simultaneously. Because our tracking algorithm does not require the identification of "tracking seeds", the tracking stability and efficiency are greatly enhanced. Last, the plant cell lineages are achieved by associating the cell tracklets, using a maximum-a-posteriori (MAP) method. Compared with the 2D matching method, the experimental results on multiple datasets show that our proposed approach does not only greatly improve the tracking accuracy by 18 percent, but also successfully tracks the plant cells located at the high curvature primordial region, which is not addressed in previous work.

Unsupervised Learning for Cell-Level Visual Representation in Histopathology Images With Generative Adversarial Networks

The visual attributes of cells, such as the nuclear morphology and chromatin openness, are critical for histopathology image analysis. By learning cell-level visual representation, we can obtain a rich mix of features that are highly reusable for various tasks, such as cell-level classification, nuclei segmentation, and cell counting. In this paper, we propose a unified generative adversarial networks architecture with a new formulation of loss to perform robust cell-level visual representation learning in an unsupervised setting. Our model is not only label-free and easily trained but also capable of cell-level unsupervised classification with interpretable visualization, which achieves promising results in the unsupervised classification of bone marrow cellular components. Based on the proposed cell-level visual representation learning, we further develop a pipeline that exploits the varieties of cellular elements to perform histopathology image classification, the advantages of which are demonstrated on bone marrow datasets.

A probabilistic approach to joint cell tracking and segmentation in high-throughput microscopy videos

We present a novel computational framework for the analysis of high-throughput microscopy videos of living cells. The proposed framework is generally useful and can be applied to different datasets acquired in a variety of laboratory settings. This is accomplished by tying together two fundamental aspects of cell lineage construction, namely cell segmentation and tracking, via a Bayesian inference of dynamic models. In contrast to most existing approaches, which aim to be general, no assumption of cell shape is made. Spatial, temporal, and cross-sectional variation of the analysed data are accommodated by two key contributions. First, time series analysis is exploited to estimate the temporal cell shape uncertainty in addition to cell trajectory. Second, a fast marching (FM) algorithm is used to integrate the inferred cell properties with the observed image measurements in order to obtain image likelihood for cell segmentation, and association. The proposed approach has been tested on eight different time-lapse microscopy data sets, some of which are high-throughput, demonstrating promising results for the detection, segmentation and association of planar cells. Our results surpass the state of the art for the Fluo-C2DL-MSC data set of the Cell Tracking Challenge (Maška et al., 2014).

Robust Cell Detection for Large-Scale 3D Microscopy Using GPU-Accelerated Iterative Voting

High-throughput imaging techniques, such as Knife-Edge Scanning Microscopy (KESM),are capable of acquiring three-dimensional whole-organ images at sub-micrometer resolution. These images are challenging to segment since they can exceed several terabytes (TB) in size, requiring extremely fast and fully automated algorithms. Staining techniques are limited to contrast agents that can be applied to large samples and imaged in a single pass. This requires maximizing the number of structures labeled in a single channel, resulting in images that are densely packed with spatial features. In this paper, we propose a three-dimensional approach for locating cells based on iterative voting. Due to the computational complexity of this algorithm, a highly efficient GPU implementation is required to make it practical on large data sets. The proposed algorithm has a limited number of input parameters and is highly parallel.

Automated Cell Segmentation for Quantitative Phase Microscopy

Automated cell segmentation and tracking is essential for dynamic studies of cellular morphology, movement, and interactions as well as other cellular behaviors. However, accurate, automated, and easy-to-use cell segmentation remains a challenge, especially in cases of high cell densities, where discrete boundaries are not easily discernable. Here, we present a fully automated segmentation algorithm that iteratively segments cells based on the observed distribution of optical cell volumes measured by quantitative phase microscopy. By fitting these distributions to known probability density functions, we are able to converge on volumetric thresholds that enable valid segmentation cuts. Since each threshold is determined from the observed data itself, virtually no input is needed from the user. We demonstrate the effectiveness of this approach over time using six cell types that display a range of morphologies, and evaluate these cultures over a range of confluencies. Facile dynamic measures of cell mobility and function revealed unique cellular behaviors that relate to tissue origins, state of differentiation, and real-time signaling. These will improve our understanding of multicellular communication and organization.

An objective comparison of cell-tracking algorithms

We present a combined report on the results of three editions of the Cell Tracking Challenge, an ongoing initiative aimed at promoting the development and objective evaluation of cell segmentation and tracking algorithms. With 21 participating algorithms and a data repository consisting of 13 data sets from various microscopy modalities, the challenge displays today's state-of-the-art methodology in the field. We analyzed the challenge results using performance measures for segmentation and tracking that rank all participating methods. We also analyzed the performance of all of the algorithms in terms of biological measures and practical usability. Although some methods scored high in all technical aspects, none obtained fully correct solutions. We found that methods that either take prior information into account using learning strategies or analyze cells in a global spatiotemporal video context performed better than other methods under the segmentation and tracking scenarios included in the challenge.

Accurate Correspondence of Cone Photoreceptor Neurons in the Human Eye Using Graph Matching Applied to Longitudinal Adaptive Optics Images

Loss of cone photoreceptor neurons is a leading cause of many blinding retinal diseases. Direct visualization of these cells in the living human eye is now feasible using adaptive optics scanning light ophthalmoscopy (AOSLO). However, it remains challenging to monitor the state of specific cells across multiple visits, due to inherent eye-motion-based distortions that arise during data acquisition, artifacts when overlapping images are montaged, as well as substantial variability in the data itself. This paper presents an accurate graph matching framework that integrates (1) robust local intensity order patterns (LIOP) to describe neuron regions with illumination variation from different visits; (2) a sparse-coding based voting process to measure visual similarities of neuron pairs using LIOP descriptors; and (3) a graph matching model that combines both visual similarity and geometrical cone packing information to determine the correspondence of repeated imaging of cone photoreceptor neurons across longitudinal AOSLO datasets. The matching framework was evaluated on imaging data from ten subjects using a validation dataset created by removing 15% of the neurons from 713 neuron correspondences across image pairs. An overall matching accuracy of 98% was achieved. The framework was robust to differences in the amount of overlap between image pairs. Evaluation on a test dataset showed that the matching accuracy remained at 98% on approximately 3400 neuron correspondences, despite image quality degradation, illumination variation, large image deformation, and edge artifacts. These experimental results show that our graph matching approach can accurately identify cone photoreceptor neuron correspondences on longitudinal AOSLO images.

Geometrical characterization of fluorescently labelled surfaces from noisy 3D microscopy data

Modern fluorescence microscopy enables fast 3D imaging of biological and inert systems alike. In many studies, it is important to detect the surface of objects and quantitatively characterize its local geometry, including its mean curvature. We present a fully automated algorithm to determine the location and curvatures of an object from 3D fluorescence images, such as those obtained using confocal or light-sheet microscopy. The algorithm aims at reconstructing surface labelled objects with spherical topology and mild deformations from the spherical geometry with high accuracy, rather than reconstructing arbitrarily deformed objects with lower fidelity. Using both synthetic data with known geometrical characteristics and experimental data of spherical objects, we characterize the algorithm's accuracy over the range of conditions and parameters typically encountered in 3D fluorescence imaging. We show that the algorithm can detect the location of the surface and obtain a map of local mean curvatures with relative errors typically below 2% and 20%, respectively, even in the presence of substantial levels of noise. Finally, we apply this algorithm to analyse the shape and curvature map of fluorescently labelled oil droplets embedded within multicellular aggregates and deformed by cellular forces.

Automated Functional Analysis of Astrocytes from Chronic Time-Lapse Calcium Imaging Data

Recent discoveries that astrocytes exert proactive regulatory effects on neural information processing and that they are deeply involved in normal brain development and disease pathology have stimulated broad interest in understanding astrocyte functional roles in brain circuit. Measuring astrocyte functional status is now technically feasible, due to recent advances in modern microscopy and ultrasensitive cell-type specific genetically encoded Ca²⁺ indicators for chronic imaging. However, there is a big gap between the capability of generating large dataset via calcium imaging and the availability of sophisticated analytical tools for decoding the astrocyte function. Current practice is essentially manual, which not only limits analysis throughput but also risks introducing bias and missing important information latent in complex, dynamic big data. Here, we report a suite of computational tools, called Functional AStrocyte Phenotyping (FASP), for automatically quantifying the functional status of astrocytes. Considering the complex nature of Ca²⁺ signaling in astrocytes and low signal to noise ratio, FASP is designed with data-driven and probabilistic principles, to flexibly account for various patterns and to perform robustly with noisy data. In particular, FASP explicitly models signal propagation, which rules out the applicability of tools designed for other types of data. We demonstrate the effectiveness of FASP using extensive synthetic and real data sets. The findings by FASP were verified by manual inspection. FASP also detected signals that were missed by purely manual analysis but could be confirmed by more careful manual examination under the guidance of automatic analysis. All algorithms and the analysis pipeline are packaged into a plugin for Fiji (ImageJ), with the source code freely available online at https://github.com/VTcbil/FASP.

Trajectory energy minimization for cell growth tracking and genealogy analysis

Cell growth experiments with a microfluidic device produce large-scale time-lapse image data, which contain important information on cell growth and patterns in their genealogy. To extract such information, we propose a scheme to segment and track bacterial cells automatically. In contrast with most published approaches, which often split segmentation and tracking into two independent procedures, we focus on designing an algorithm that describes cell properties evolving between consecutive frames by feeding segmentation and tracking results from one frame to the next one. The cell boundaries are extracted by minimizing the distance regularized level set evolution (DRLSE) model. Each individual cell was identified and tracked by identifying cell septum and membrane as well as developing a trajectory energy minimization function along time-lapse series. Experiments show that by applying this scheme, cell growth and division can be measured automatically. The results show the efficiency of the approach when testing on different datasets while comparing with other existing algorithms. The proposed approach demonstrates great potential for large-scale bacterial cell growth analysis.

Network Flow Integer Programming to Track Elliptical Cells in Time-Lapse Sequences

We propose a novel approach to automatically tracking elliptical cell populations in time-lapse image sequences. Given an initial segmentation, we account for partial occlusions and overlaps by generating an over-complete set of competing detection hypotheses. To this end, we fit ellipses to portions of the initial regions and build a hierarchy of ellipses, which are then treated as cell candidates. We then select temporally consistent ones by solving to optimality an integer program with only one type of flow variables. This eliminates the need for heuristics to handle missed detections due to partial occlusions and complex morphology. We demonstrate the effectiveness of our approach on a range of challenging sequences consisting of clumped cells and show that it outperforms state-of-the-art techniques.

A novel multiphoton microscopy images segmentation method based on superpixel and watershed

Multiphoton microscopy (MPM) imaging technique based on two-photon excited fluorescence (TPEF) and second harmonic generation (SHG) shows fantastic performance for biological imaging. The automatic segmentation of cellular architectural properties for biomedical diagnosis based on MPM images is still a challenging issue. A novel multiphoton microscopy images segmentation method based on superpixels and watershed (MSW) is presented here to provide good segmentation results for MPM images. The proposed method uses SLIC superpixels instead of pixels to analyze MPM images for the first time. The superpixels segmentation based on a new distance metric combined with spatial, CIE Lab color space and phase congruency features, divides the images into patches which keep the details of the cell boundaries. Then the superpixels are used to reconstruct new images by defining an average value of superpixels as image pixels intensity level. Finally, the marker-controlled watershed is utilized to segment the cell boundaries from the reconstructed images. Experimental results show that cellular boundaries can be extracted from MPM images by MSW with higher accuracy and robustness.

MitoGen: A Framework for Generating 3D Synthetic Time-Lapse Sequences of Cell Populations in Fluorescence Microscopy

The proper analysis of biological microscopy images is an important and complex task. Therefore, it requires verification of all steps involved in the process, including image segmentation and tracking algorithms. It is generally better to verify algorithms with computer-generated ground truth datasets, which, compared to manually annotated data, nowadays have reached high quality and can be produced in large quantities even for 3D time-lapse image sequences. Here, we propose a novel framework, called MitoGen, which is capable of generating ground truth datasets with fully 3D time-lapse sequences of synthetic fluorescence-stained cell populations. MitoGen shows biologically justified cell motility, shape and texture changes as well as cell divisions. Standard fluorescence microscopy phenomena such as photobleaching, blur with real point spread function (PSF), and several types of noise, are simulated to obtain realistic images. The MitoGen framework is scalable in both space and time. MitoGen generates visually plausible data that shows good agreement with real data in terms of image descriptors and mean square displacement (MSD) trajectory analysis. Additionally, it is also shown in this paper that four publicly available segmentation and tracking algorithms exhibit similar performance on both real and MitoGen-generated data. The implementation of MitoGen is freely available.

Single cell dynamic phenotyping

Live cell imaging has improved our ability to measure phenotypic heterogeneity. However, bottlenecks in imaging and image processing often make it difficult to differentiate interesting biological behavior from technical artifact. Thus there is a need for new methods that improve data quality without sacrificing throughput. Here we present a 3-step workflow to improve dynamic phenotype measurements of heterogeneous cell populations. We provide guidelines for image acquisition, phenotype tracking, and data filtering to remove erroneous cell tracks using the novel Tracking Aberration Measure (TrAM). Our workflow is broadly applicable across imaging platforms and analysis software. By applying this workflow to cancer cell assays, we reduced aberrant cell track prevalence from 17% to 2%. The cost of this improvement was removing 15% of the well-tracked cells. This enabled detection of significant motility differences between cell lines. Similarly, we avoided detecting a false change in translocation kinetics by eliminating the true cause: varied proportions of unresponsive cells. Finally, by systematically seeking heterogeneous behaviors, we detected subpopulations that otherwise could have been missed, including early apoptotic events and pre-mitotic cells. We provide optimized protocols for specific applications and step-by-step guidelines for adapting them to a variety of biological systems.

Network Consistent Data Association

Existing data association techniques mostly focus on matching pairs of data-point sets and then repeating this process along space-time to achieve long term correspondences. However, in many problems such as person re-identification, a set of data-points may be observed at multiple spatio-temporal locations and/or by multiple agents in a network and simply combining the local pairwise association results between sets of data-points often leads to inconsistencies over the global space-time horizons. In this paper, we propose a Novel Network Consistent Data Association (NCDA) framework formulated as an optimization problem that not only maintains consistency in association results across the network, but also improves the pairwise data association accuracies. The proposed NCDA can be solved as a binary integer program leading to a globally optimal solution and is capable of handling the challenging data-association scenario where the number of data-points varies across different sets of instances in the network. We also present an online implementation of NCDA method that can dynamically associate new observations to already observed data-points in an iterative fashion, while maintaining network consistency. We have tested both the batch and the online NCDA in two application areas-person re-identification and spatio-temporal cell tracking and observed consistent and highly accurate data association results in all the cases.