Large-scale time-lapse microscopy of Oct4 expression in human embryonic stem cell colonies

Memory-efficient semantic segmentation of large microscopy images using graph-based neural networks

We present a graph neural network (GNN)-based framework applied to large-scale microscopy image segmentation tasks. While deep learning models, like convolutional neural networks (CNNs), have become common for automating image segmentation tasks, they are limited by the image size that can fit in the memory of computational hardware. In a GNN framework, large-scale images are converted into graphs using superpixels (regions of pixels with similar color/intensity values), allowing us to input information from the entire image into the model. By converting images with hundreds of millions of pixels to graphs with thousands of nodes, we can segment large images using memory-limited computational resources. We compare the performance of GNN- and CNN-based segmentation in terms of accuracy, training time and required graphics processing unit memory. Based on our experiments with microscopy images of biological cells and cell colonies, GNN-based segmentation used one to three orders-of-magnitude fewer computational resources with only a change in accuracy of ‒2 % to +0.3 %. Furthermore, errors due to superpixel generation can be reduced by either using better superpixel generation algorithms or increasing the number of superpixels, thereby allowing for improvement in the GNN framework's accuracy. This trade-off between accuracy and computational cost over CNN models makes the GNN framework attractive for many large-scale microscopy image segmentation tasks in biology.

High-volume, label-free imaging for quantifying single-cell dynamics in induced pluripotent stem cell colonies

To facilitate the characterization of unlabeled induced pluripotent stem cells (iPSCs) during culture and expansion, we developed an AI pipeline for nuclear segmentation and mitosis detection from phase contrast images of individual cells within iPSC colonies. The analysis uses a 2D convolutional neural network (U-Net) plus a 3D U-Net applied on time lapse images to detect and segment nuclei, mitotic events, and daughter nuclei to enable tracking of large numbers of individual cells over long times in culture. The analysis uses fluorescence data to train models for segmenting nuclei in phase contrast images. The use of classical image processing routines to segment fluorescent nuclei precludes the need for manual annotation. We optimize and evaluate the accuracy of automated annotation to assure the reliability of the training. The model is generalizable in that it performs well on different datasets with an average F1 score of 0.94, on cells at different densities, and on cells from different pluripotent cell lines. The method allows us to assess, in a non-invasive manner, rates of mitosis and cell division which serve as indicators of cell state and cell health. We assess these parameters in up to hundreds of thousands of cells in culture for more than 36 hours, at different locations in the colonies, and as a function of excitation light exposure.

Exact Tile-Based Segmentation Inference for Images Larger than GPU Memory

We address the problem of performing exact (tiling-error free) out-of-core semantic segmentation inference of arbitrarily large images using fully convolutional neural networks (FCN). FCN models have the property that once a model is trained, it can be applied on arbitrarily sized images, although it is still constrained by the available GPU memory. This work is motivated by overcoming the GPU memory size constraint without numerically impacting the final result. Our approach is to select a tile size that will fit into GPU memory with a halo border of half the network receptive field. Next, stride across the image by that tile size without the halo. The input tile halos will overlap, while the output tiles join exactly at the seams. Such an approach enables inference to be performed on whole slide microscopy images, such as those generated by a slide scanner. The novelty of this work is in documenting the formulas for determining tile size and stride and then validating them on U-Net and FC-DenseNet architectures. In addition, we quantify the errors due to tiling configurations which do not satisfy the constraints, and we explore the use of architecture effective receptive fields to estimate the tiling parameters.

Early time-point cell morphology classifiers successfully predict human bone marrow stromal cell differentiation modulated by fiber density in nanofiber scaffolds

Nanofiber scaffolds can induce osteogenic differentiation and cell morphology alterations of human bone marrow stromal cells (hBMSCs) without introduction of chemical cues. In this study, we investigate the predictive power of day 1 cell morphology, quantified by a machine learning based method, as an indicator of osteogenic differentiation modulated by nanofiber density. Nanofiber scaffolds are fabricated via electrospinning. Microscopy, quantitative image processing and clustering analysis are used to systematically quantify scaffold properties as a function of fiber density. hBMSC osteogenic differentiation potential is evaluated after 14 days using osteogenic marker gene expression and after 50 days using calcium mineralization, showing enhanced osteogenic differentiation with an increase in nanofiber density. Cell morphology measurements at day 1 successfully predict differentiation potential when analyzed with the support vector machine (SVM)/supercell tools previously developed and trained on cells from a different donor. A correlation is observed between differentiation potential and cell morphology, demonstrating sensitivity of the morphology measurement to varying degrees of differentiation potential. To further understand how nanofiber density determines hBMSC morphology, both full 3-D morphology measurements as well as other measurements of the 2-D projected morphology are investigated in this study. To achieve predictive power on hBMSC osteogenic differentiation, at least two morphology metrics need to be considered together for each cell, with the majority of metric pairs including one 3-D morphology metric. Analysis of the local nanofiber structure surrounding each cell reveals a correlation with single-cell morphology and indicates that the osteogenic differentiation phenotype may be predictive at the single-cell level.

Probing pluripotency gene regulatory networks with quantitative live cell imaging

Live cell imaging uniquely enables the measurement of dynamic events in single cells, but it has not been used often in the study of gene regulatory networks. Network components can be examined in relation to one another by quantitative live cell imaging of fluorescent protein reporter cell lines that simultaneously report on more than one network component. A series of dual-reporter cell lines would allow different combinations of network components to be examined in individual cells. Dynamical information about interacting network components in individual cells is critical to predictive modeling of gene regulatory networks, and such information is not accessible through omics and other end point techniques. Achieving this requires that gene-edited cell lines are appropriately designed and adequately characterized to assure the validity of the biological conclusions derived from the expression of the reporters. In this brief review we discuss what is known about the importance of dynamics to network modeling and review some recent advances in optical microscopy methods and image analysis approaches that are making the use of quantitative live cell imaging for network analysis possible. We also discuss how strategies for genetic engineering of reporter cell lines can influence the biological relevance of the data.

The role of fluctuations in determining cellular network thermodynamics

The steady state distributions of phenotypic responses within an isogenic population of cells result from both deterministic and stochastic characteristics of biochemical networks. A biochemical network can be characterized by a multidimensional potential landscape based on the distribution of responses and a diffusion matrix of the correlated dynamic fluctuations between N-numbers of intracellular network variables. In this work, we develop a thermodynamic description of biological networks at the level of microscopic interactions between network variables. The Boltzmann H-function defines the rate of free energy dissipation of a network system and provides a framework for determining the heat associated with the nonequilibrium steady state and its network components. The magnitudes of the landscape gradients and the dynamic correlated fluctuations of network variables are experimentally accessible. We describe the use of Fokker-Planck dynamics to calculate housekeeping heat from the experimental data by a method that we refer to as Thermo-FP. The method provides insight into the composition of the network and the relative thermodynamic contributions from network components. We surmise that these thermodynamic quantities allow determination of the relative importance of network components to overall network control. We conjecture that there is an upper limit to the rate of dissipative heat produced by a biological system that is associated with system size or modularity, and we show that the dissipative heat has a lower bound.

Opportunities and Challenges in Implementation of Multiparameter Single Cell Analysis Platforms for Clinical Translation

The high-content interrogation of single cells with platforms optimized for the multiparameter characterization of cells in liquid and solid biopsy samples can enable characterization of heterogeneous populations of cells ex vivo. Doing so will advance the diagnosis, prognosis, and treatment of cancer and other diseases. However, it is important to understand the unique issues in resolving heterogeneity and variability at the single cell level before navigating the validation and regulatory requirements in order for these technologies to impact patient care. Since 2013, leading experts representing industry, academia, and government have been brought together as part of the Foundation for the National Institutes of Health (FNIH) Biomarkers Consortium to foster the potential of high-content data integration for clinical translation.

Asymmetricity Between Sister Cells of Pluripotent Stem Cells at the Onset of Differentiation

Various somatic stem cells divide asymmetrically; however, it is not known whether embryonic stem cells (ESCs) divide symmetrically or asymmetrically, not only while maintaining an undifferentiated state but also at the onset of differentiation. In this study, we observed single ESCs using time-lapse imaging and compared sister cell pairs derived from the same mother cell in either the maintenance or differentiation medium. Mouse ESCs were cultured on E-cadherin-coated glass-based dishes, which allowed us to trace single cells. The undifferentiated cell state was detected by green fluorescent protein (GFP) expression driven by the Nanog promoter, which is active only in undifferentiated cells. Cell population analysis using flow cytometry showed that the peak width indicating distribution of GFP expression broadened when cells were transferred to the differentiation medium compared to when they were in the maintenance medium. This finding suggested that the population of ESCs became more heterogeneous at the onset of differentiation. Using single-cell analysis by time-lapse imaging, we found that although the total survival ratio decreased by changing to differentiation medium, the one-live-one-dead ratio of sister cell pairs was smaller compared with randomly chosen non-sister cell pairs, defined as an unsynchronized cell pair control, in both media. This result suggested that sister cell pairs were more positively synchronized with each other compared to non-sister cell pairs. The differences in interdivision time (the time interval between mother cell division and the subsequent cell division) between sister cells was smaller than that between non-sister cell pairs in both media, suggesting that sister cells divided synchronously. Although the difference in Nanog-GFP intensity between sister cells was smaller than that between non-sister cells in the maintenance medium, it was the same in differentiation medium, suggesting asymmetrical Nanog-GFP intensity. These data suggested that ESCs may divide asymmetrically at the onset of differentiation resulting in heterogeneity.

Illuminating stem cell transcription factor dynamics: long-term single-cell imaging of fluorescent protein fusions

Most single-cell approaches to date are based on destructive snapshot measurements which do not permit to correlate a current molecular state with future fate. However, to understand how cell fate choices are established by transcription factor networks (TFNs) regulating cell fates, TFN dynamics must be continuously monitored in single cells. Here we review how quantitative time-lapse imaging can contribute to understanding TFN dependent cell fate regulation at the single-cell level. We outline potentials of the technology and highlight challenges for interpreting the dynamics of fluorescent protein reporters that may interfere with endogenous TF function. We provide an outlook on how continuous observation of TF dynamics and single-cell fates may be complemented by perturbation studies and be linked to multidimensional molecular profiling in the future.

MIST: Accurate and Scalable Microscopy Image Stitching Tool with Stage Modeling and Error Minimization

Automated microscopy can image specimens larger than the microscope’s field of view (FOV) by stitching overlapping image tiles. It also enables time-lapse studies of entire cell cultures in multiple imaging modalities. We created MIST (Microscopy Image Stitching Tool) for rapid and accurate stitching of large 2D time-lapse mosaics. MIST estimates the mechanical stage model parameters (actuator backlash, and stage repeatability ‘r’) from computed pairwise translations and then minimizes stitching errors by optimizing the translations within a (4r)² square area. MIST has a performance-oriented implementation utilizing multicore hybrid CPU/GPU computing resources, which can process terabytes of time-lapse multi-channel mosaics 15 to 100 times faster than existing tools. We created 15 reference datasets to quantify MIST’s stitching accuracy. The datasets consist of three preparations of stem cell colonies seeded at low density and imaged with varying overlap (10 to 50%). The location and size of 1150 colonies are measured to quantify stitching accuracy. MIST generated stitched images with an average centroid distance error that is less than 2% of a FOV. The sources of these errors include mechanical uncertainties, specimen photobleaching, segmentation, and stitching inaccuracies. MIST produced higher stitching accuracy than three open-source tools. MIST is available in ImageJ at isg.nist.gov.

Lineage mapper: A versatile cell and particle tracker

The ability to accurately track cells and particles from images is critical to many biomedical problems. To address this, we developed Lineage Mapper, an open-source tracker for time-lapse images of biological cells, colonies, and particles. Lineage Mapper tracks objects independently of the segmentation method, detects mitosis in confluence, separates cell clumps mistakenly segmented as a single cell, provides accuracy and scalability even on terabyte-sized datasets, and creates division and/or fusion lineages. Lineage Mapper has been tested and validated on multiple biological and simulated problems. The software is available in ImageJ and Matlab at isg.nist.gov.