Detection of molecular particles in live cells via machine learning

Two- and Three-Dimensional Benchmarks for Particle Detection from an Industrial Rotary Kiln Combustion Chamber Based on Light-Field-Camera Recording

This paper describes a benchmark dataset for the detection of fuel particles in 2D and 3D image data in a rotary kiln combustion chamber. The specific challenges of detecting the small particles under demanding environmental conditions allows for the performance of existing and new particle detection techniques to be evaluated. The data set includes a classification of burning and non-burning particles, which can be in the air but also on the rotary kiln wall. The light-field camera used for data generation offers the potential to develop and objectively evaluate new advanced particle detection methods due to the additional 3D information. Besides explanations of the data set and the contained ground truth, an evaluation procedure of the particle detection based on the ground truth and results for an own particle detection procedure for the data set are presented.

Could artificial intelligence revolutionize the development of nanovectors for gene therapy and mRNA vaccines?

Gene therapy enables the introduction of nucleic acids like DNA and RNA into host cells, and is expected to revolutionize the treatment of a wide range of diseases. This growth has been further accelerated by the discovery of CRISPR/Cas technology, which allows accurate genomic editing in a broad range of cells and organisms in vitro and in vivo. Despite many advances in gene delivery and the development of various viral and non-viral gene delivery vectors, the lack of highly efficient non-viral systems with low cellular toxicity remains a challenge. The application of cutting-edge technologies such as artificial intelligence (AI) has great potential to find new paradigms to solve this issue. Herein, we review AI and its major subfields including machine learning (ML), neural networks (NNs), expert systems, deep learning (DL), computer vision and robotics. We discuss the potential of AI-based models and algorithms in the design of targeted gene delivery vehicles capable of crossing extracellular and intracellular barriers by viral mimicry strategies. We finally discuss the role of AI in improving the function of CRISPR/Cas systems, developing novel nanobots, and mRNA vaccine carriers.

Single-Nanoparticle Orientation Sensing by Deep Learning

This paper describes a computational imaging platform to determine the orientation of anisotropic optical probes under differential interference contrast (DIC) microscopy. We established a deep-learning model based on data sets of DIC images collected from metal nanoparticle optical probes at different orientations. This model predicted the in-plane angle of gold nanorods with an error below 20°, the inherent limit of the DIC method. Using low-symmetry gold nanostars as optical probes, we demonstrated the detection of in-plane particle orientation in the full 0-360° range. We also showed that orientation predictions of the same particle were consistent even with variations in the imaging background. Finally, the deep-learning model was extended to enable simultaneous prediction of in-plane and out-of-plane rotation angles for a multibranched nanostar by concurrent analysis of DIC images measured at multiple wavelengths.

A two-stage method for automated detection of ring-like endosomes in fluorescent microscopy images

Endosomes are subcellular organelles which serve as important transport compartments in eukaryotic cells. Fluorescence microscopy is a widely applied technology to study endosomes at the subcellular level. In general, a microscopy image can contain a large number of organelles and endosomes in particular. Detecting and annotating endosomes in fluorescence microscopy images is a critical part in the study of subcellular trafficking processes. Such annotation is usually performed by human inspection, which is time-consuming and prone to inaccuracy if carried out by inexperienced analysts. This paper proposes a two-stage method for automated detection of ring-like endosomes. The method consists of a localization stage cascaded by an identification stage. Given a test microscopy image, the localization stage generates a voting-map by locally comparing the query endosome patches and the test image based on a bag-of-words model. Using the voting-map, a number of candidate patches of endosomes are determined. Subsequently, in the identification stage, a support vector machine (SVM) is trained using the endosome patches and the background pattern patches. Each of the candidate patches is classified by the SVM to rule out those patches of endosome-like background patterns. The performance of the proposed method is evaluated with real microscopy images of human myeloid endothelial cells. It is shown that the proposed method significantly outperforms several state-of-the-art competing methods using multiple performance metrics.

Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D

Particle tracking is a powerful biophysical tool that requires conversion of large video files into position time series, i.e., traces of the species of interest for data analysis. Current tracking methods, based on a limited set of input parameters to identify bright objects, are ill-equipped to handle the spectrum of spatiotemporal heterogeneity and poor signal-to-noise ratios typically presented by submicron species in complex biological environments. Extensive user involvement is frequently necessary to optimize and execute tracking methods, which is not only inefficient but introduces user bias. To develop a fully automated tracking method, we developed a convolutional neural network for particle localization from image data, comprising over 6,000 parameters, and used machine learning techniques to train the network on a diverse portfolio of video conditions. The neural network tracker provides unprecedented automation and accuracy, with exceptionally low false positive and false negative rates on both 2D and 3D simulated videos and 2D experimental videos of difficult-to-track species.

Single-molecule techniques in biophysics: a review of the progress in methods and applications

Single-molecule biophysics has transformed our understanding of biology, but also of the physics of life. More exotic than simple soft matter, biomatter lives far from thermal equilibrium, covering multiple lengths from the nanoscale of single molecules to up to several orders of magnitude higher in cells, tissues and organisms. Biomolecules are often characterized by underlying instability: multiple metastable free energy states exist, separated by levels of just a few multiples of the thermal energy scale k _B T, where k _B is the Boltzmann constant and T absolute temperature, implying complex inter-conversion kinetics in the relatively hot, wet environment of active biological matter. A key benefit of single-molecule biophysics techniques is their ability to probe heterogeneity of free energy states across a molecular population, too challenging in general for conventional ensemble average approaches. Parallel developments in experimental and computational techniques have catalysed the birth of multiplexed, correlative techniques to tackle previously intractable biological questions. Experimentally, progress has been driven by improvements in sensitivity and speed of detectors, and the stability and efficiency of light sources, probes and microfluidics. We discuss the motivation and requirements for these recent experiments, including the underpinning mathematics. These methods are broadly divided into tools which detect molecules and those which manipulate them. For the former we discuss the progress of super-resolution microscopy, transformative for addressing many longstanding questions in the life sciences, and for the latter we include progress in 'force spectroscopy' techniques that mechanically perturb molecules. We also consider in silico progress of single-molecule computational physics, and how simulation and experimentation may be drawn together to give a more complete understanding. Increasingly, combinatorial techniques are now used, including correlative atomic force microscopy and fluorescence imaging, to probe questions closer to native physiological behaviour. We identify the trade-offs, limitations and applications of these techniques, and discuss exciting new directions.

Analysis of in vivo single cell behavior by high throughput, human-in-the-loop segmentation of three-dimensional images

BACKGROUND

Analysis of single cells in their native environment is a powerful method to address key questions in developmental systems biology. Confocal microscopy imaging of intact tissues, followed by automatic image segmentation, provides a means to conduct cytometric studies while at the same time preserving crucial information about the spatial organization of the tissue and morphological features of the cells. This technique is rapidly evolving but is still not in widespread use among research groups that do not specialize in technique development, perhaps in part for lack of tools that automate repetitive tasks while allowing experts to make the best use of their time in injecting their domain-specific knowledge.

RESULTS

Here we focus on a well-established stem cell model system, the C. elegans gonad, as well as on two other model systems widely used to study cell fate specification and morphogenesis: the pre-implantation mouse embryo and the developing mouse olfactory epithelium. We report a pipeline that integrates machine-learning-based cell detection, fast human-in-the-loop curation of these detections, and running of active contours seeded from detections to segment cells. The procedure can be bootstrapped by a small number of manual detections, and outperforms alternative pieces of software we benchmarked on C. elegans gonad datasets. Using cell segmentations to quantify fluorescence contents, we report previously-uncharacterized cell behaviors in the model systems we used. We further show how cell morphological features can be used to identify cell cycle phase; this provides a basis for future tools that will streamline cell cycle experiments by minimizing the need for exogenous cell cycle phase labels.

CONCLUSIONS

High-throughput 3D segmentation makes it possible to extract rich information from images that are routinely acquired by biologists, and provides insights - in particular with respect to the cell cycle - that would be difficult to derive otherwise.

Adaptive Spot Detection With Optimal Scale Selection in Fluorescence Microscopy Images

Accurately detecting subcellular particles in fluorescence microscopy is of primary interest for further quantitative analysis such as counting, tracking, or classification. Our primary goal is to segment vesicles likely to share nearly the same size in fluorescence microscopy images. Our method termed adaptive thresholding of Laplacian of Gaussian (LoG) images with autoselected scale (ATLAS) automatically selects the optimal scale corresponding to the most frequent spot size in the image. Four criteria are proposed and compared to determine the optimal scale in a scale-space framework. Then, the segmentation stage amounts to thresholding the LoG of the intensity image. In contrast to other methods, the threshold is locally adapted given a probability of false alarm (PFA) specified by the user for the whole set of images to be processed. The local threshold is automatically derived from the PFA value and local image statistics estimated in a window whose size is not a critical parameter. We also propose a new data set for benchmarking, consisting of six collections of one hundred images each, which exploits backgrounds extracted from real microscopy images. We have carried out an extensive comparative evaluation on several data sets with ground-truth, which demonstrates that ATLAS outperforms existing methods. ATLAS does not need any fine parameter tuning and requires very low computation time. Convincing results are also reported on real total internal reflection fluorescence microscopy images.

Performance and sensitivity evaluation of 3D spot detection methods in confocal microscopy

Reliable 3D detection of diffraction-limited spots in fluorescence microscopy images is an important task in subcellular observation. Generally, fluorescence microscopy images are heavily degraded by noise and non-specifically stained background, making reliable detection a challenging task. In this work, we have studied the performance and parameter sensitivity of eight recent methods for 3D spot detection. The study is based on both 3D synthetic image data and 3D real confocal microscopy images. The synthetic images were generated using a simulator modeling the complete imaging setup, including the optical path as well as the image acquisition process. We studied the detection performance and parameter sensitivity under different noise levels and under the influence of uneven background signal. To evaluate the parameter sensitivity, we propose a novel measure based on the gradient magnitude of the F1 score. We measured the success rate of the individual methods for different types of the image data and found that the type of image degradation is an important factor. Using the F1 score and the newly proposed sensitivity measure, we found that the parameter sensitivity is not necessarily proportional to the success rate of a method. This also provided an explanation why the best performing method for synthetic data was outperformed by other methods when applied to the real microscopy images. On the basis of the results obtained, we conclude with the recommendation of the HDome method for data with relatively low variations in quality, or the Sorokin method for image sets in which the quality varies more. We also provide alternative recommendations for high-quality images, and for situations in which detailed parameter tuning might be deemed expensive.

CAST: An automated segmentation and tracking tool for the analysis of transcriptional kinetics from single-cell time-lapse recordings

Fluorescence and bioluminescence time-lapse imaging allows to investigate a vast range of cellular processes at single-cell or even subcellular resolution. In particular, time-lapse imaging can provide uniquely detailed information on the fine kinetics of transcription, as well as on biological oscillations such as the circadian and cell cycles. However, we face a paucity of automated methods to quantify time-lapse imaging data with single-cell precision, notably throughout multiple cell cycles. We developed CAST (Cell Automated Segmentation and Tracking platform) to automatically and robustly detect the position and size of cells or nuclei, quantify the corresponding light signals, while taking into account both cell divisions (lineage tracking) and migration events. We present here how CAST analyzes bioluminescence data from a short-lived transcriptional luciferase reporter. However, our flexible and modular implementation makes it easily adaptable to a wide variety of time-lapse recordings. We exemplify how CAST efficiently quantifies single-cell gene expression over multiple cell cycles using mouse NIH3T3 culture cells with a luminescence expression driven by the Bmal1 promoter, a central gene of the circadian oscillator. We further illustrate how such data can be used to quantify transcriptional bursting in conditions of lengthened circadian period, revealing thereby remarkably similar bursting signature compared to the endogenous circadian condition despite marked period lengthening. In summary, we establish CAST as novel tool for the efficient segmentation, signal quantification, and tracking of time-lapse images from mammalian cell culture.

Background fluorescence estimation and vesicle segmentation in live cell imaging with conditional random fields

Image analysis applied to fluorescence live cell microscopy has become a key tool in molecular biology since it enables to characterize biological processes in space and time at the subcellular level. In fluorescence microscopy imaging, the moving tagged structures of interest, such as vesicles, appear as bright spots over a static or nonstatic background. In this paper, we consider the problem of vesicle segmentation and time-varying background estimation at the cellular scale. The main idea is to formulate the joint segmentation-estimation problem in the general conditional random field framework. Furthermore, segmentation of vesicles and background estimation are alternatively performed by energy minimization using a min cut-max flow algorithm. The proposed approach relies on a detection measure computed from intensity contrasts between neighboring blocks in fluorescence microscopy images. This approach permits analysis of either 2D + time or 3D + time data. We demonstrate the performance of the so-called C-CRAFT through an experimental comparison with the state-of-the-art methods in fluorescence video-microscopy. We also use this method to characterize the spatial and temporal distribution of Rab6 transport carriers at the cell periphery for two different specific adhesion geometries.

Chapter 17: bioimage informatics for systems pharmacology

Recent advances in automated high-resolution fluorescence microscopy and robotic handling have made the systematic and cost effective study of diverse morphological changes within a large population of cells possible under a variety of perturbations, e.g., drugs, compounds, metal catalysts, RNA interference (RNAi). Cell population-based studies deviate from conventional microscopy studies on a few cells, and could provide stronger statistical power for drawing experimental observations and conclusions. However, it is challenging to manually extract and quantify phenotypic changes from the large amounts of complex image data generated. Thus, bioimage informatics approaches are needed to rapidly and objectively quantify and analyze the image data. This paper provides an overview of the bioimage informatics challenges and approaches in image-based studies for drug and target discovery. The concepts and capabilities of image-based screening are first illustrated by a few practical examples investigating different kinds of phenotypic changes caEditorsused by drugs, compounds, or RNAi. The bioimage analysis approaches, including object detection, segmentation, and tracking, are then described. Subsequently, the quantitative features, phenotype identification, and multidimensional profile analysis for profiling the effects of drugs and targets are summarized. Moreover, a number of publicly available software packages for bioimage informatics are listed for further reference. It is expected that this review will help readers, including those without bioimage informatics expertise, understand the capabilities, approaches, and tools of bioimage informatics and apply them to advance their own studies.

Automated REcognition of tissue-associated erythrocytes (ARETE)-a new tool in tissue cytometry

Automated microscopic image analysis of immunofluorescence-stained targets on tissue sections is challenged by autofluorescent elements such as erythrocytes, which might interfere with target segmentation and quantification. Therefore, we developed an automated system (Automated REcognition of Tissue-associated Erythrocytes; ARETE) for in silico exclusion of erythrocytes. To detect erythrocytes in transmission images, a cascade of boosted decision trees of Haar-like features was trained on 8,640/4,000 areas (15 × 15 pixels) with/without erythrocytes from images of placental sections (4 µm). Ground truth data were generated on 28 transmission images. At least two human experts labelled the area covered by erythrocytes. For validation, output masks of human experts and ARETE were compared pixel-wise against a mask obtained from majority voting of human experts. F1 score, specificity, and Cohen's κ coefficients were calculated. To study the influence of erythrocyte-derived autofluorescence, we investigated the expression levels of a protein (receptor for advanced glycated end products; RAGE) in placenta and number of Ki-67-positive/cytokeratin 8-positive epithelial cells in colon sections. ARETE exhibited high sensitivity (99.87%) and specificity (99.81%) on a training-subset and processed transmission images (1,392 × 1,024 pixels) within 4 sec. ARETE and human expert's F1-scores were 0.55 versus 0.76, specificities 0.85 versus 0.92 and Cohen's κ coefficients 0.41 versus 0.68. A ranking of Cohen's κ coefficient by the scale of Fleiss certified "good agreement" between ARETE and the human experts. Applying ARETE, we demonstrated 4-14% false-positive RAGE-expression in placenta, and 18% falsely detected proliferative epithelial cells in colon, caused by erythrocyte-autofluorescence. ARETE is a fast system for in silico reduction of erythrocytes, which improves automated image analysis in research and diagnostic pathology.

Automated podosome identification and characterization in fluorescence microscopy images

Podosomes are cellular adhesion structures involved in matrix degradation and invasion that comprise an actin core and a ring of cytoskeletal adaptor proteins. They are most often identified by staining with phalloidin, which binds F-actin and therefore visualizes the core. However, not only podosomes, but also many other cytoskeletal structures contain actin, which makes podosome segmentation by automated image processing difficult. Here, we have developed a quantitative image analysis algorithm that is optimized to identify podosome cores within a typical sample stained with phalloidin. By sequential local and global thresholding, our analysis identifies up to 76% of podosome cores excluding other F-actin-based structures. Based on the overlap in podosome identifications and quantification of podosome numbers, our algorithm performs equally well compared to three experts. Using our algorithm we show effects of actin polymerization and myosin II inhibition on the actin intensity in both podosome core and associated actin network. Furthermore, by expanding the core segmentations, we reveal a previously unappreciated differential distribution of cytoskeletal adaptor proteins within the podosome ring. These applications illustrate that our algorithm is a valuable tool for rapid and accurate large-scale analysis of podosomes to increase our understanding of these characteristic adhesion structures.

A new framework for particle detection in low-SNR fluorescence live-cell images and its application for improved particle tracking

Image denoising and signal enhancement are two common steps to improve particle contrast for detection in low-signal-to-noise ratio (SNR) fluorescence live-cell images. However, denoising may oversmooth features of interest, particularly weak features, leading to false negative detection. Here, we propose a robust framework for particle detection in which image denoising in the grayscale image is not needed, so avoiding image oversmoothing. A key to our approach is the new development of a particle enhancement filter based on the recently proposed particle probability image to obtain significantly enhanced particle features and greatly suppressed background in low-SNR and low-contrast environments. The new detection method is formed by combining foreground and background markers with watershed transform operating in both particle probability and grayscale spaces; dynamical switchings between the two spaces can optimally make use the information in images for accurate determination of particle position, size, and intensity. We further develop the interacting multiple mode filter for particle motion modeling and data association by incorporating the extra information obtained from our particle detector to enhance the efficiency of multiple particle tracking. We find that our methods lead to significant improvements in particle detection and tracking efficiency in fluorescence live-cell applications.

Recent advances in morphological cell image analysis

This paper summarizes the recent advances in image processing methods for morphological cell analysis. The topic of morphological analysis has received much attention with the increasing demands in both bioinformatics and biomedical applications. Among many factors that affect the diagnosis of a disease, morphological cell analysis and statistics have made great contributions to results and effects for a doctor. Morphological cell analysis finds the cellar shape, cellar regularity, classification, statistics, diagnosis, and so forth. In the last 20 years, about 1000 publications have reported the use of morphological cell analysis in biomedical research. Relevant solutions encompass a rather wide application area, such as cell clumps segmentation, morphological characteristics extraction, 3D reconstruction, abnormal cells identification, and statistical analysis. These reports are summarized in this paper to enable easy referral to suitable methods for practical solutions. Representative contributions and future research trends are also addressed.

Towards a versatile automated cell-detection system for science and diagnostics

Analyzing in-situ tissue structures with complex shapes and textures such as multinuclear cells or cells without nuclei is still a challenge for currently available imageprocessing software. This work aims to provide a versatile system to solve such tasks provided that the structures of interests were detected by immunofluorescence microscopy. Images were automatically acquired using slide-based microscopy. Human domain-experts manually marked up tissue samples to evaluate the performance of the computer generated masks. From precision and recall a balanced F-score was computed to measure the correlation between experts and algorithm output. Exhaustive parameter optimization was conducted to ensure that the optimal input parameters were applied during evaluation of the developed algorithm. This procedure significantly increased the performance compared to manually chosen input parameters. We present an approach that can handle huge tissue areas and does not rely on nuclei detection. Once a markup has been created, the algorithm can be parameter-optimized on ground-truth data for the chosen tissue sample. Thereafter, the resulting settings could be applied automatically to the respective stitched image. Concluding, we provide new insights in physiological and pathopysiological cellular mechanisms by automating the in-situ analysis of proteins in intact tissues.

Extended morphological processing: a practical method for automatic spot detection of biological markers from microscopic images

Background

A reliable extraction technique for resolving multiple spots in light or electron microscopic images is essential in investigations of the spatial distribution and dynamics of specific proteins inside cells and tissues. Currently, automatic spot extraction and characterization in complex microscopic images poses many challenges to conventional image processing methods.

Results

A new method to extract closely located, small target spots from biological images is proposed. This method starts with a simple but practical operation based on the extended morphological top-hat transformation to subtract an uneven background. The core of our novel approach is the following: first, the original image is rotated in an arbitrary direction and each rotated image is opened with a single straight line-segment structuring element. Second, the opened images are unified and then subtracted from the original image. To evaluate these procedures, model images of simulated spots with closely located targets were created and the efficacy of our method was compared to that of conventional morphological filtering methods. The results showed the better performance of our method. The spots of real microscope images can be quantified to confirm that the method is applicable in a given practice.

Conclusions

Our method achieved effective spot extraction under various image conditions, including aggregated target spots, poor signal-to-noise ratio, and large variations in the background intensity. Furthermore, it has no restrictions with respect to the shape of the extracted spots. The features of our method allow its broad application in biological and biomedical image information analysis.

Evaluation of methods for detection of fluorescence labeled subcellular objects in microscope images

Background

Several algorithms have been proposed for detecting fluorescently labeled subcellular objects in microscope images. Many of these algorithms have been designed for specific tasks and validated with limited image data. But despite the potential of using extensive comparisons between algorithms to provide useful information to guide method selection and thus more accurate results, relatively few studies have been performed.

Results

To better understand algorithm performance under different conditions, we have carried out a comparative study including eleven spot detection or segmentation algorithms from various application fields. We used microscope images from well plate experiments with a human osteosarcoma cell line and frames from image stacks of yeast cells in different focal planes. These experimentally derived images permit a comparison of method performance in realistic situations where the number of objects varies within image set. We also used simulated microscope images in order to compare the methods and validate them against a ground truth reference result. Our study finds major differences in the performance of different algorithms, in terms of both object counts and segmentation accuracies.

Conclusions

These results suggest that the selection of detection algorithms for image based screens should be done carefully and take into account different conditions, such as the possibility of acquiring empty images or images with very few spots. Our inclusion of methods that have not been used before in this context broadens the set of available detection methods and compares them against the current state-of-the-art methods for subcellular particle detection.

Quantitative comparison of spot detection methods in fluorescence microscopy

Quantitative analysis of biological image data generally involves the detection of many subresolution spots. Especially in live cell imaging, for which fluorescence microscopy is often used, the signal-to-noise ratio (SNR) can be extremely low, making automated spot detection a very challenging task. In the past, many methods have been proposed to perform this task, but a thorough quantitative evaluation and comparison of these methods is lacking in the literature. In this paper, we evaluate the performance of the most frequently used detection methods for this purpose. These include seven unsupervised and two supervised methods. We perform experiments on synthetic images of three different types, for which the ground truth was available, as well as on real image data sets acquired for two different biological studies, for which we obtained expert manual annotations to compare with. The results from both types of experiments suggest that for very low SNRs ( approximately 2), the supervised (machine learning) methods perform best overall. Of the unsupervised methods, the detectors based on the so-called h -dome transform from mathematical morphology or the multiscale variance-stabilizing transform perform comparably, and have the advantage that they do not require a cumbersome learning stage. At high SNRs ( > 5), the difference in performance of all considered detectors becomes negligible.

Tracking in cell and developmental biology

The past decade has seen an unprecedented data explosion in biology. It has become evident that in order to take full advantage of the potential wealth of information hidden in the data produced by even a single experiment, visual inspection and manual analysis are no longer adequate. To ensure efficiency, consistency, and completeness in data processing and analysis, computational tools are essential. Of particular importance to many modern live-cell imaging experiments is the ability to automatically track and analyze the motion of objects in time-lapse microscopy images. This article surveys the recent literature in this area. Covering all scales of microscopic observation, from cells, down to molecules, and up to entire organisms, it discusses the latest trends and successes in the development and application of computerized tracking methods in cell and developmental biology.