A generative model of microtubule distributions, and indirect estimation of its parameters from fluorescence microscopy images

Learning Morphological, Spatial, and Dynamic Models of Cellular Components

In this chapter, we describe protocols for using the CellOrganizer software on the Jupyter Notebook platform to analyze and model cell and organelle shape and spatial arrangement. CellOrganizer is an open-source system for using microscope images to learn statistical models of the structure of cell components and how those components are organized relative to each other. Such models capture the statistical variation in the organization of cellular components by jointly modeling the distributions of their number, shape, and spatial distributions. These models can be created for different cell types or conditions and compared to reflect differences in their spatial organizations. The models are also generative, in that they can be used to synthesize new cell instances reflecting what a model learned and to provide well-structured cell geometries that can be used for biochemical simulations.

DBlink: dynamic localization microscopy in super spatiotemporal resolution via deep learning

Single-molecule localization microscopy (SMLM) has revolutionized biological imaging, improving the spatial resolution of traditional microscopes by an order of magnitude. However, SMLM techniques require long acquisition times, typically a few minutes, to yield a single super-resolved image, because they depend on accumulation of many localizations over thousands of recorded frames. Hence, the capability of SMLM to observe dynamics at high temporal resolution has always been limited. In this work, we present DBlink, a deep-learning-based method for super spatiotemporal resolution reconstruction from SMLM data. The input to DBlink is a recorded video of SMLM data and the output is a super spatiotemporal resolution video reconstruction. We use a convolutional neural network combined with a bidirectional long short-term memory network architecture, designed for capturing long-term dependencies between different input frames. We demonstrate DBlink performance on simulated filaments and mitochondria-like structures, on experimental SMLM data under controlled motion conditions and on live-cell dynamic SMLM. DBlink's spatiotemporal interpolation constitutes an important advance in super-resolution imaging of dynamic processes in live cells.

Deep-ROCS: from speckle patterns to superior-resolved images by deep learning in rotating coherent scattering microscopy

Rotating coherent scattering (ROCS) microscopy is a label-free imaging technique that overcomes the optical diffraction limit by adding up the scattered laser light from a sample obliquely illuminated from different angles. Although ROCS imaging achieves 150 nm spatial and 10 ms temporal resolution, simply summing different speckle patterns may cause loss of sample information. In this paper we present Deep-ROCS, a neural network-based technique that generates a superior-resolved image by efficient numerical combination of a set of differently illuminated images. We show that Deep-ROCS can reconstruct super-resolved images more accurately than conventional ROCS microscopy, retrieving high-frequency information from a small number (6) of speckle images. We demonstrate the performance of Deep-ROCS experimentally on 200 nm beads and by computer simulations, where we show its potential for even more complex structures such as a filament network.

A two-stage method for automated detection of ring-like endosomes in fluorescent microscopy images

Endosomes are subcellular organelles which serve as important transport compartments in eukaryotic cells. Fluorescence microscopy is a widely applied technology to study endosomes at the subcellular level. In general, a microscopy image can contain a large number of organelles and endosomes in particular. Detecting and annotating endosomes in fluorescence microscopy images is a critical part in the study of subcellular trafficking processes. Such annotation is usually performed by human inspection, which is time-consuming and prone to inaccuracy if carried out by inexperienced analysts. This paper proposes a two-stage method for automated detection of ring-like endosomes. The method consists of a localization stage cascaded by an identification stage. Given a test microscopy image, the localization stage generates a voting-map by locally comparing the query endosome patches and the test image based on a bag-of-words model. Using the voting-map, a number of candidate patches of endosomes are determined. Subsequently, in the identification stage, a support vector machine (SVM) is trained using the endosome patches and the background pattern patches. Each of the candidate patches is classified by the SVM to rule out those patches of endosome-like background patterns. The performance of the proposed method is evaluated with real microscopy images of human myeloid endothelial cells. It is shown that the proposed method significantly outperforms several state-of-the-art competing methods using multiple performance metrics.

Extracting microtubule networks from superresolution single-molecule localization microscopy data

Microtubule filaments form ubiquitous networks. However, quantitative analysis of this structure is difficult due to its complex architecture. A tool is given for the automated retrieval of microtubule filaments from superresolution microscopy images and used for a quantitative analysis of microtubule network architecture phenotypes in fibroblasts.

Microtubule filaments form ubiquitous networks that specify spatial organization in cells. However, quantitative analysis of microtubule networks is hampered by their complex architecture, limiting insights into the interplay between their organization and cellular functions. Although superresolution microscopy has greatly facilitated high-resolution imaging of microtubule filaments, extraction of complete filament networks from such data sets is challenging. Here we describe a computational tool for automated retrieval of microtubule filaments from single-molecule-localization–based superresolution microscopy images. We present a user-friendly, graphically interfaced implementation and a quantitative analysis of microtubule network architecture phenotypes in fibroblasts.

Subcellular protein expression models for microsatellite instability in colorectal adenocarcinoma tissue images

Background

New bioimaging techniques capable of visualising the co-location of numerous proteins within individual cells have been proposed to study tumour heterogeneity of neighbouring cells within the same tissue specimen. These techniques have highlighted the need to better understand the interplay between proteins in terms of their colocalisation.

Results

We recently proposed a cellular-level model of the healthy and cancerous colonic crypt microenvironments. Here, we extend the model to include detailed models of protein expression to generate synthetic multiplex fluorescence data. As a first step, we present models for various cell organelles learned from real immunofluorescence data from the Human Protein Atlas. Comparison between the distribution of various features obtained from the real and synthetic organelles has shown very good agreement. This has included both features that have been used as part of the model input and ones that have not been explicitly considered. We then develop models for six proteins which are important colorectal cancer biomarkers and are associated with microsatellite instability, namely MLH1, PMS2, MSH2, MSH6, P53 and PTEN. The protein models include their complex expression patterns and which cell phenotypes express them. The models have been validated by comparing distributions of real and synthesised parameters and by application of frameworks for analysing multiplex immunofluorescence image data.

Conclusions

The six proteins have been chosen as a case study to illustrate how the model can be used to generate synthetic multiplex immunofluorescence data. Further proteins could be included within the model in a similar manner to enable the study of a larger set of proteins of interest and their interactions. To the best of our knowledge, this is the first model for expression of multiple proteins in anatomically intact tissue, rather than within cells in culture.

Point process models for localization and interdependence of punctate cellular structures

Accurate representations of cellular organization for multiple eukaryotic cell types are required for creating predictive models of dynamic cellular function. To this end, we have previously developed the CellOrganizer platform, an open source system for generative modeling of cellular components from microscopy images. CellOrganizer models capture the inherent heterogeneity in the spatial distribution, size, and quantity of different components among a cell population. Furthermore, CellOrganizer can generate quantitatively realistic synthetic images that reflect the underlying cell population. A current focus of the project is to model the complex, interdependent nature of organelle localization. We built upon previous work on developing multiple non-parametric models of organelles or structures that show punctate patterns. The previous models described the relationships between the subcellular localization of puncta and the positions of cell and nuclear membranes and microtubules. We extend these models to consider the relationship to the endoplasmic reticulum (ER), and to consider the relationship between the positions of different puncta of the same type. Our results do not suggest that the punctate patterns we examined are dependent on ER position or inter- and intra-class proximity. With these results, we built classifiers to update previous assignments of proteins to one of 11 patterns in three distinct cell lines. Our generative models demonstrate the ability to construct statistically accurate representations of puncta localization from simple cellular markers in distinct cell types, capturing the complex phenomena of cellular structure interaction with little human input. This protocol represents a novel approach to vesicular protein annotation, a field that is often neglected in high-throughput microscopy. These results suggest that spatial point process models provide useful insight with respect to the spatial dependence between cellular structures. © 2016 International Society for Advancement of Cytometry.

Building cell models and simulations from microscope images

The use of fluorescence microscopy has undergone a major revolution over the past twenty years, both with the development of dramatic new technologies and with the widespread adoption of image analysis and machine learning methods. Many open source software tools provide the ability to use these methods in a wide range of studies, and many molecular and cellular phenotypes can now be automatically distinguished. This article presents the next major challenge in microscopy automation, the creation of accurate models of cell organization directly from images, and reviews the progress that has been made towards this challenge.

Unbiased Rare Event Sampling in Spatial Stochastic Systems Biology Models Using a Weighted Ensemble of Trajectories

The long-term goal of connecting scales in biological simulation can be facilitated by scale-agnostic methods. We demonstrate that the weighted ensemble (WE) strategy, initially developed for molecular simulations, applies effectively to spatially resolved cell-scale simulations. The WE approach runs an ensemble of parallel trajectories with assigned weights and uses a statistical resampling strategy of replicating and pruning trajectories to focus computational effort on difficult-to-sample regions. The method can also generate unbiased estimates of non-equilibrium and equilibrium observables, sometimes with significantly less aggregate computing time than would be possible using standard parallelization. Here, we use WE to orchestrate particle-based kinetic Monte Carlo simulations, which include spatial geometry (e.g., of organelles, plasma membrane) and biochemical interactions among mobile molecular species. We study a series of models exhibiting spatial, temporal and biochemical complexity and show that although WE has important limitations, it can achieve performance significantly exceeding standard parallel simulation—by orders of magnitude for some observables.

Stochastic simulations (simulations where randomness plays a role) of even simple biological systems are often so computationally intensive that it is impossible, in practice, to simulate them exhaustively and gather good statistics about the likelihood of different outcomes. The difficulty is compounded for the observation of rare events in these simulations; unfortunately, rare events, such as state transitions and barrier crossings, are often those of particular interest. Using the weighted ensemble (WE) method, we are able to enhance the characterization of rare events in cell biology simulations, but in such a way that the statistics for these events remain unbiased. The histogram of outcomes that WE produces has the same shape as a naive one, but the resolution of events in the tails of the histogram is greatly improved. This improved resolution in rare event statistics can be used to infer unbiased estimates of long timescale dynamics from short simulations, and we show that using a weighted ensemble can result in a reduction in total simulation time needed to sample certain events of interest in spatial, stochastic models of biological systems.

Automated Learning of Subcellular Variation among Punctate Protein Patterns and a Generative Model of Their Relation to Microtubules

Characterizing the spatial distribution of proteins directly from microscopy images is a difficult problem with numerous applications in cell biology (e.g. identifying motor-related proteins) and clinical research (e.g. identification of cancer biomarkers). Here we describe the design of a system that provides automated analysis of punctate protein patterns in microscope images, including quantification of their relationships to microtubules. We constructed the system using confocal immunofluorescence microscopy images from the Human Protein Atlas project for 11 punctate proteins in three cultured cell lines. These proteins have previously been characterized as being primarily located in punctate structures, but their images had all been annotated by visual examination as being simply "vesicular". We were able to show that these patterns could be distinguished from each other with high accuracy, and we were able to assign to one of these subclasses hundreds of proteins whose subcellular localization had not previously been well defined. In addition to providing these novel annotations, we built a generative approach to modeling of punctate distributions that captures the essential characteristics of the distinct patterns. Such models are expected to be valuable for representing and summarizing each pattern and for constructing systems biology simulations of cell behaviors.

New methods for quantifying and visualizing information from images of cells: An overview

New microscopy imaging techniques have enabled the acquisition of cellular and sub-cellular information with unprecedented accuracy and specificity. Fluorescence techniques have enabled labeling of numerous, previously inaccessible, molecules and organelles, while Raman spectrographic techniques, for example, have enabled label free acquisition. Together with the development of high throughput techniques, these technologies now allow for the acquisition of a significant amount of information about cellular processes and have enabled high throughput and high content screening. Beyond image formation and acquisition, computational techniques comprise an important part of the process of obtaining biological understanding from such experiments. Here we review the pros and cons of the main approaches that have been used to extract information from digital images of cells. In addition, we also offer an overview of modern computational techniques that beyond allowing for discrimination between two hypothesis, also allow for modeling, visualization, and understanding of biological phenomena.

Extraction of Individual Filaments from 2D Confocal Microscopy Images of Flat Cells

A crucial step in understanding the architecture of cells and tissues from microscopy images, and consequently explain important biological events such as wound healing and cancer metastases, is the complete extraction and enumeration of individual filaments from the cellular cytoskeletal network. Current efforts at quantitative estimation of filament length distribution, architecture and orientation from microscopy images are predominantly limited to visual estimation and indirect experimental inference. Here we demonstrate the application of a new algorithm to reliably estimate centerlines of biological filament bundles and extract individual filaments from the centerlines by systematically disambiguating filament intersections. We utilize a filament enhancement step followed by reverse diffusion based filament localization and an integer programming based set combination to systematically extract accurate filaments automatically from microscopy images. Experiments on simulated and real confocal microscope images of flat cells (2D images) show efficacy of the new method.

Localizing and extracting filament distributions from microscopy images

Detailed quantitative measurements of biological filament networks represent a crucial step in understanding architecture and structure of cells and tissues, which in turn explain important biological events such as wound healing and cancer metastases. Microscopic images of biological specimens marked for different structural proteins constitute an important source for observing and measuring meaningful parameters of biological networks. Unfortunately, current efforts at quantitative estimation of architecture and orientation of biological filament networks from microscopy images are predominantly limited to visual estimation and indirect experimental inference. Here, we describe a new method for localizing and extracting filament distributions from 2D microscopy images of different modalities. The method combines a filter-based detection of pixels likely to contain a filament with a constrained reverse diffusion-based approach for localizing the filaments centrelines. We show with qualitative and quantitative experiments, using both simulated and real data, that the new method can provide more accurate centreline estimates of filament in comparison to other approaches currently available. In addition, we show the algorithm is more robust with respect to variations in the initial filter-based filament detection step often used. We demonstrate the application of the method in extracting quantitative parameters from confocal microscopy images of actin filaments and atomic force microscopy images of DNA fragments.

Localizing and extracting filament distributions from microscopy images

Detailed quantitative measurements of biological filament networks represent a crucial step in understanding architecture and structure of cells and tissues, which in turn explain important biological events such as wound healing and cancer metastases. Confocal microscope images of biological specimens marked for different structural proteins constitute an important source for observing and measuring meaningful parameters of biological networks. Unfortunately, current efforts at quantitative estimation of architecture and orientation of biological filament networks from microscopy images are predominantly limited to visual estimation and indirect experimental inference. Here we describe a new method for localizing and extracting filament distributions from 2D confocal microscopy images. The method combines a filter-based detection of pixels likely to contain a filament with a constrained reverse diffusion-based approach for localizing the filaments centrelines. We show with qualitative and quantitative experiments, using both simulated and real data, that the new method can provide more accurate centreline estimates of filament in comparison to other approaches currently available. In addition, we show the algorithm is more robust with respect to variations in the initial filter-based filament detection step often used. We demonstrate the application of the method in extracting quantitative parameters from an experiment that seeks to quantify the effects of carbon nanotubes on actin cytoskeleton in live HeLa cells. We show that their presence can disrupt the overall actin cytoskeletal organization in such cells.

Estimating microtubule distributions from 2D immunofluorescence microscopy images reveals differences among human cultured cell lines

Microtubules are filamentous structures that are involved in several important cellular processes, including cell division, cellular structure and mechanics, and intracellular transportation. Little is known about potential differences in microtubule distributions within and across cell lines. Here we describe a method to estimate information pertaining to 3D microtubule distributions from 2D fluorescence images. Our method allows for quantitative comparisons of microtubule distribution parameters (number of microtubules, mean length) between different cell lines. Among eleven cell lines compared, some showed differences that could be accounted for by differences in the total amount of tubulin per cell while others showed statistically significant differences in the balance between number and length of microtubules. We also observed that some cell lines that visually appear different in their microtubule distributions are quite similar when the model parameters are considered. The method is expected to be generally useful for comparing microtubule distributions between cell lines and for a given cell line after various perturbations. The results are also expected to enable analysis of the differences in gene expression underlying the observed differences in microtubule distributions among cell types.

Toward the virtual cell: automated approaches to building models of subcellular organization "learned" from microscopy images

We review state-of-the-art computational methods for constructing, from image data, generative statistical models of cellular and nuclear shapes and the arrangement of subcellular structures and proteins within them. These automated approaches allow consistent analysis of images of cells for the purposes of learning the range of possible phenotypes, discriminating between them, and informing further investigation. Such models can also provide realistic geometry and initial protein locations to simulations in order to better understand cellular and subcellular processes. To determine the structures of cellular components and how proteins and other molecules are distributed among them, the generative modeling approach described here can be coupled with high throughput imaging technology to infer and represent subcellular organization from data with few a priori assumptions. We also discuss potential improvements to these methods and future directions for research.

CellOrganizer: Image-derived models of subcellular organization and protein distribution

This chapter describes approaches for learning models of subcellular organization from images. The primary utility of these models is expected to be from incorporation into complex simulations of cell behaviors. Most current cell simulations do not consider spatial organization of proteins at all, or treat each organelle type as a single, idealized compartment. The ability to build generative models for all proteins in a proteome and use them for spatially accurate simulations is expected to improve the accuracy of models of cell behaviors. A second use, of potentially equal importance, is expected to be in testing and comparing software for analyzing cell images. The complexity and sophistication of algorithms used in cell-image-based screens and assays (variously referred to as high-content screening, high-content analysis, or high-throughput microscopy) is continuously increasing, and generative models can be used to produce images for testing these algorithms in which the expected answer is known.

AUTOMATED ESTIMATION OF MICROTUBULE MODEL PARAMETERS FROM 3-D LIVE CELL MICROSCOPY IMAGES

While basic principles of microtubule organization are well understood, much remains to be learned about the extent and significance of variation in that organization among cell types and conditions. Large numbers of images of microtubule distributions for many cell types can be readily obtained by high throughput fluorescence microscopy but direct estimation of the parameters underlying the organization is problematic because it is difficult to resolve individual microtubules present at the microtubule-organizing center or at regions of high crossover. Previously, we developed an indirect, generative model-based approach that can estimate such spatial distribution parameters as the number and mean length of microtubules. In order to validate this approach, we have applied it to 3D images of NIH 3T3 cells expressing fluorescently-tagged tubulin in the presence and absence of the microtubule depolymerizing drug nocodazole. We describe here the first application of our inverse modeling approach to live cell images and demonstrate that it yields estimates consistent with expectations.

Image-derived, three-dimensional generative models of cellular organization

Given the importance of subcellular location to protein function, computational simulations of cell behaviors will ultimately require the ability to model the distributions of proteins within organelles and other structures. Toward this end, statistical learning methods have previously been used to build models of sets of two-dimensional microscope images, where each set contains multiple images for a single subcellular location pattern. The model learned from each set of images not only represents the pattern but also captures the variation in that pattern from cell to cell. The models consist of sub-models for nuclear shape, cell shape, organelle size and shape, and organelle distribution relative to nuclear and cell boundaries, and allow synthesis of images with the expectation that they are drawn from the same underlying statistical distribution as the images used to train them. Here we extend this generative models approach to three dimensions using a similar framework, permitting protein subcellular locations to be described more accurately. Models of different patterns can be combined to yield a synthetic multi-channel image containing as many proteins as desired, something that is difficult to obtain by direct microscope imaging for more than a few proteins. In addition, the model parameters represent a more compact and interpretable way of communicating subcellular patterns than descriptive image features and may be particularly effective for automated identification of changes in subcellular organization caused by perturbagens.

AUTOMATED ESTIMATION OF MICROTUBULE MODEL PARAMETERS FROM 3-D LIVE CELL MICROSCOPY IMAGES

While basic principles of microtubule organization are well understood, much remains to be learned about the extent and significance of variation in that organization among cell types and conditions. Large numbers of images of microtubule distributions for many cell types can be readily obtained by high throughput fluorescence microscopy but direct estimation of the parameters underlying the organization is problematic because it is difficult to resolve individual microtubules present at the microtubule-organizing center or at regions of high crossover. Previously, we developed an indirect, generative model-based approach that can estimate such spatial distribution parameters as the number and mean length of microtubules. In order to validate this approach, we have applied it to 3D images of NIH 3T3 cells expressing fluorescently-tagged tubulin in the presence and absence of the microtubule depolymerizing drug nocodazole. We describe here the first application of our inverse modeling approach to live cell images and demonstrate that it yields estimates consistent with expectations.

Communicating subcellular distributions

To build more accurate models of cells and tissues, the ability to incorporate information on the distributions of proteins (and other macromolecules) will become increasingly important. This review describes current progress towards determining and representing protein subcellular patterns so that the information can be used as part of systems biology efforts. Approaches to decomposing an image of the subcellular pattern of a protein give critical information about the fraction of that protein in each of a number of fundamental patterns (e.g., organelles). Methods for learning generative models from images provide a means of capturing the essential properties and variation in those properties of cell shape and organelle patterns. The combination of models of fundamental patterns and vectors specifying the fraction of a protein in each of them provide a much better means of communicating subcellular patterns than the descriptive terms that are currently used. Communicating information about subcellular patterns is important not only for systems biology simulations but also for representing results from microscopy experiments, including high content screening and imaging flow cytometry, in a transportable and generalizable manner.