Object detection networks and augmented reality for cellular detection in fluorescence microscopy

From pixels to insights: Machine learning and deep learning for bioimage analysis

Bioimage analysis plays a critical role in extracting information from biological images, enabling deeper insights into cellular structures and processes. The integration of machine learning and deep learning techniques has revolutionized the field, enabling the automated, reproducible, and accurate analysis of biological images. Here, we provide an overview of the history and principles of machine learning and deep learning in the context of bioimage analysis. We discuss the essential steps of the bioimage analysis workflow, emphasizing how machine learning and deep learning have improved preprocessing, segmentation, feature extraction, object tracking, and classification. We provide examples that showcase the application of machine learning and deep learning in bioimage analysis. We examine user-friendly software and tools that enable biologists to leverage these techniques without extensive computational expertise. This review is a resource for researchers seeking to incorporate machine learning and deep learning in their bioimage analysis workflows and enhance their research in this rapidly evolving field.

Multiscale spatial segregation analysis in digital images of biofilms

Quantifying the degree of spatial segregation of two bacterial strains in mixed biofilms is an important topic in microbiology. Spatial segregation is dependent on spatial scale as two strains may appear to be well mixed if observed from a distance, but a closer look can reveal strong separation. Typically, this information is encoded in a digital image that represents the binary system, e.g., a microscopy image of a two species biofilm. To decode spatial segregation information, we have developed quantitative measures for evaluating the degree of the spatial scale-dependent segregation of two bacterial strains in a digital image. The constructed algorithm is based on the new segregation measures and overcomes drawbacks of existing approaches for biofilm segregation analysis. The new approach is implemented in a freely available software and was successfully applied to biofilms of two strains and bacterial suspensions for detection of the different spatial scale-dependent segregation levels.

COMBINe enables automated detection and classification of neurons and astrocytes in tissue-cleared mouse brains

Tissue clearing renders entire organs transparent to accelerate whole-tissue imaging; for example, with light-sheet fluorescence microscopy. Yet, challenges remain in analyzing the large resulting 3D datasets that consist of terabytes of images and information on millions of labeled cells. Previous work has established pipelines for automated analysis of tissue-cleared mouse brains, but the focus there was on single-color channels and/or detection of nuclear localized signals in relatively low-resolution images. Here, we present an automated workflow (COMBINe, Cell detectiOn in Mouse BraIN) to map sparsely labeled neurons and astrocytes in genetically distinct mouse forebrains using mosaic analysis with double markers (MADM). COMBINe blends modules from multiple pipelines with RetinaNet at its core. We quantitatively analyzed the regional and subregional effects of MADM-based deletion of the epidermal growth factor receptor (EGFR) on neuronal and astrocyte populations in the mouse forebrain.

Event-driven acquisition for content-enriched microscopy

A common goal of fluorescence microscopy is to collect data on specific biological events. Yet, the event-specific content that can be collected from a sample is limited, especially for rare or stochastic processes. This is due in part to photobleaching and phototoxicity, which constrain imaging speed and duration. We developed an event-driven acquisition framework, in which neural-network-based recognition of specific biological events triggers real-time control in an instant structured illumination microscope. Our setup adapts acquisitions on-the-fly by switching between a slow imaging rate while detecting the onset of events, and a fast imaging rate during their progression. Thus, we capture mitochondrial and bacterial divisions at imaging rates that match their dynamic timescales, while extending overall imaging durations. Because event-driven acquisition allows the microscope to respond specifically to complex biological events, it acquires data enriched in relevant content.

DeepBacs for multi-task bacterial image analysis using open-source deep learning approaches

This work demonstrates and guides how to use a range of state-of-the-art artificial neural-networks to analyse bacterial microscopy images using the recently developed ZeroCostDL4Mic platform. We generated a database of image datasets used to train networks for various image analysis tasks and present strategies for data acquisition and curation, as well as model training. We showcase different deep learning (DL) approaches for segmenting bright field and fluorescence images of different bacterial species, use object detection to classify different growth stages in time-lapse imaging data, and carry out DL-assisted phenotypic profiling of antibiotic-treated cells. To also demonstrate the ability of DL to enhance low-phototoxicity live-cell microscopy, we showcase how image denoising can allow researchers to attain high-fidelity data in faster and longer imaging. Finally, artificial labelling of cell membranes and predictions of super-resolution images allow for accurate mapping of cell shape and intracellular targets. Our purposefully-built database of training and testing data aids in novice users' training, enabling them to quickly explore how to analyse their data through DL. We hope this lays a fertile ground for the efficient application of DL in microbiology and fosters the creation of tools for bacterial cell biology and antibiotic research.

An Open-Source Modular Framework for Automated Pipetting and Imaging Applications

The number of samples in biological experiments is continuously increasing, but complex protocols and human error in many cases lead to suboptimal data quality and hence difficulties in reproducing scientific findings. Laboratory automation can alleviate many of these problems by precisely reproducing machine-readable protocols. These instruments generally require high up-front investments, and due to the lack of open application programming interfaces (APIs), they are notoriously difficult for scientists to customize and control outside of the vendor-supplied software. Here, automated, high-throughput experiments are demonstrated for interdisciplinary research in life science that can be replicated on a modest budget, using open tools to ensure reproducibility by combining the tools OpenFlexure, Opentrons, ImJoy, and UC2. This automated sample preparation and imaging pipeline can easily be replicated and established in many laboratories as well as in educational contexts through easy-to-understand algorithms and easy-to-build microscopes. Additionally, the creation of feedback loops, with later pipetting or imaging steps depending on the analysis of previously acquired images, enables the realization of fully autonomous "smart" microscopy experiments. All documents and source files are publicly available to prove the concept of smart lab automation using inexpensive, open tools. It is believed this democratizes access to the power and repeatability of automated experiments.

A deep learning model (FociRad) for automated detection of γ-H2AX foci and radiation dose estimation

DNA double-strand breaks (DSBs) are the most lethal form of damage to cells from irradiation. γ-H2AX (phosphorylated form of H2AX histone variant) has become one of the most reliable and sensitive biomarkers of DNA DSBs. However, the γ-H2AX foci assay still has limitations in the time consumed for manual scoring and possible variability between scorers. This study proposed a novel automated foci scoring method using a deep convolutional neural network based on a You-Only-Look-Once (YOLO) algorithm to quantify γ-H2AX foci in peripheral blood samples. FociRad, a two-stage deep learning approach, consisted of mononuclear cell (MNC) and γ-H2AX foci detections. Whole blood samples were irradiated with X-rays from a 6 MV linear accelerator at 1, 2, 4 or 6 Gy. Images were captured using confocal microscopy. Then, dose-response calibration curves were established and implemented with unseen dataset. The results of the FociRad model were comparable with manual scoring. MNC detection yielded 96.6% accuracy, 96.7% sensitivity and 96.5% specificity. γ-H2AX foci detection showed very good F1 scores (> 0.9). Implementation of calibration curve in the range of 0-4 Gy gave mean absolute difference of estimated doses less than 1 Gy compared to actual doses. In addition, the evaluation times of FociRad were very short (< 0.5 min per 100 images), while the time for manual scoring increased with the number of foci. In conclusion, FociRad was the first automated foci scoring method to use a YOLO algorithm with high detection performance and fast evaluation time, which opens the door for large-scale applications in radiation triage.

Detection and Recognition of Pollen Grains in Multilabel Microscopic Images

Analysis of pollen material obtained from the Hirst-type apparatus, which is a tedious and labor-intensive process, is usually performed by hand under a microscope by specialists in palynology. This research evaluated the automatic analysis of pollen material performed based on digital microscopic photos. A deep neural network called YOLO was used to analyze microscopic images containing the reference grains of three taxa typical of Central and Eastern Europe. YOLO networks perform recognition and detection; hence, there is no need to segment the image before classification. The obtained results were compared to other deep learning object detection methods, i.e., Faster R-CNN and RetinaNet. YOLO outperformed the other methods, as it gave the mean average precision (mAP@.5:.95) between 86.8% and 92.4% for the test sets included in the study. Among the difficulties related to the correct classification of the research material, the following should be noted: significant similarities of the grains of the analyzed taxa, the possibility of their simultaneous occurrence in one image, and mutual overlapping of objects.

Neural network fast-classifies biological images through features selecting to power automated microscopy

Artificial intelligence is nowadays used for cell detection and classification in optical microscopy during post-acquisition analysis. The microscopes are now fully automated and next expected to be smart by making acquisition decisions based on the images. It calls for analysing them on the fly. Biology further imposes training on a reduced data set due to cost and time to prepare the samples and have the data sets annotated by experts. We propose a real-time image processing compliant with these specifications by balancing accurate detection and execution performance. We characterised the images using a generic, high-dimensional feature extractor. We then classified the images using machine learning to understand the contribution of each feature in decision and execution time. We found that the non-linear-classifier random forests outperformed Fisher's linear discriminant. More importantly, the most discriminant and time-consuming features could be excluded without significant accuracy loss, offering a substantial gain in execution time. It suggests a feature-group redundancy likely related to the biology of the observed cells. We offer a method to select fast and discriminant features. In our assay, a 79.6 ± 2.4% accurate classification of a cell took 68.7 ± 3.5 ms (mean ± SD, 5-fold cross-validation nested in 10 bootstrap repeats), corresponding to 14 cells per second, dispatched into eight phases of the cell cycle, using 12 feature groups and operating a consumer market ARM-based embedded system. A simple neural network offered similar performances paving the way to faster training and classification, using parallel execution on a general-purpose graphic processing unit. Finally, this strategy is also usable for deep neural networks paving the way to optimizing these algorithms for smart microscopy.

Python-Microscope - a new open-source Python library for the control of microscopes

Custom-built microscopes often require control of multiple hardware devices and precise hardware coordination. It is also desirable to have a solution that is scalable to complex systems and that is translatable between components from different manufacturers. Here we report Python-Microscope, a free and open-source Python library for high-performance control of arbitrarily complex and scalable custom microscope systems. Python-Microscope offers simple to use Python-based tools, abstracting differences between physical devices by providing a defined interface for different device types. Concrete implementations are provided for a range of specific hardware, and a framework exists for further expansion. Python-Microscope supports the distribution of devices over multiple computers while maintaining synchronisation via highly precise hardware triggers. We discuss the architectural features of Python-Microscope that overcome the performance problems often raised against Python and demonstrate the different use cases that drove its design: integration with user-facing projects, namely the Microscope-Cockpit project; control of complex microscopes at high speed while using the Python programming language; and use as a microscope simulation tool for software development.

Detection and classification of neurons and glial cells in the MADM mouse brain using RetinaNet

The ability to automatically detect and classify populations of cells in tissue sections is paramount in a wide variety of applications ranging from developmental biology to pathology. Although deep learning algorithms are widely applied to microscopy data, they typically focus on segmentation which requires extensive training and labor-intensive annotation. Here, we utilized object detection networks (neural networks) to detect and classify targets in complex microscopy images, while simplifying data annotation. To this end, we used a RetinaNet model to classify genetically labeled neurons and glia in the brains of Mosaic Analysis with Double Markers (MADM) mice. Our initial RetinaNet-based model achieved an average precision of 0.90 across six classes of cells differentiated by MADM reporter expression and their phenotype (neuron or glia). However, we found that a single RetinaNet model often failed when encountering dense and saturated glial clusters, which show high variability in their shape and fluorophore densities compared to neurons. To overcome this, we introduced a second RetinaNet model dedicated to the detection of glia clusters. Merging the predictions of the two computational models significantly improved the automated cell counting of glial clusters. The proposed cell detection workflow will be instrumental in quantitative analysis of the spatial organization of cellular populations, which is applicable not only to preparations in neuroscience studies, but also to any tissue preparation containing labeled populations of cells.

Deep learning for bioimage analysis in developmental biology

Deep learning has transformed the way large and complex image datasets can be processed, reshaping what is possible in bioimage analysis. As the complexity and size of bioimage data continues to grow, this new analysis paradigm is becoming increasingly ubiquitous. In this Review, we begin by introducing the concepts needed for beginners to understand deep learning. We then review how deep learning has impacted bioimage analysis and explore the open-source resources available to integrate it into a research project. Finally, we discuss the future of deep learning applied to cell and developmental biology. We analyze how state-of-the-art methodologies have the potential to transform our understanding of biological systems through new image-based analysis and modelling that integrate multimodal inputs in space and time.

Developing open-source software for bioimage analysis: opportunities and challenges

Fast-paced innovations in imaging have resulted in single systems producing exponential amounts of data to be analyzed. Computational methods developed in computer science labs have proven to be crucial for analyzing these data in an unbiased and efficient manner, reaching a prominent role in most microscopy studies. Still, their use usually requires expertise in bioimage analysis, and their accessibility for life scientists has therefore become a bottleneck.

Open-source software for bioimage analysis has developed to disseminate these computational methods to a wider audience, and to life scientists in particular. In recent years, the influence of many open-source tools has grown tremendously, helping tens of thousands of life scientists in the process. As creators of successful open-source bioimage analysis software, we here discuss the motivations that can initiate development of a new tool, the common challenges faced, and the characteristics required for achieving success.

Data science in cell imaging

Cell imaging has entered the 'Big Data' era. New technologies in light microscopy and molecular biology have led to an explosion in high-content, dynamic and multidimensional imaging data. Similar to the 'omics' fields two decades ago, our current ability to process, visualize, integrate and mine this new generation of cell imaging data is becoming a critical bottleneck in advancing cell biology. Computation, traditionally used to quantitatively test specific hypotheses, must now also enable iterative hypothesis generation and testing by deciphering hidden biologically meaningful patterns in complex, dynamic or high-dimensional cell image data. Data science is uniquely positioned to aid in this process. In this Perspective, we survey the rapidly expanding new field of data science in cell imaging. Specifically, we highlight how data science tools are used within current image analysis pipelines, propose a computation-first approach to derive new hypotheses from cell image data, identify challenges and describe the next frontiers where we believe data science will make an impact. We also outline steps to ensure broad access to these powerful tools - democratizing infrastructure availability, developing sensitive, robust and usable tools, and promoting interdisciplinary training to both familiarize biologists with data science and expose data scientists to cell imaging.