Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction

Optical tissue clearing and machine learning can precisely characterize extravasation and blood vessel architecture in brain tumors

Precise methods for quantifying drug accumulation in brain tissue are currently very limited, challenging the development of new therapeutics for brain disorders. Transcardial perfusion is instrumental for removing the intravascular fraction of an injected compound, thereby allowing for ex vivo assessment of extravasation into the brain. However, pathological remodeling of tissue microenvironment can affect the efficiency of transcardial perfusion, which has been largely overlooked. We show that, in contrast to healthy vasculature, transcardial perfusion cannot remove an injected compound from the tumor vasculature to a sufficient extent leading to considerable overestimation of compound extravasation. We demonstrate that 3D deep imaging of optically cleared tumor samples overcomes this limitation. We developed two machine learning-based semi-automated image analysis workflows, which provide detailed quantitative characterization of compound extravasation patterns as well as tumor angioarchitecture in large three-dimensional datasets from optically cleared samples. This methodology provides a precise and comprehensive analysis of extravasation in brain tumors and allows for correlation of extravasation patterns with specific features of the heterogeneous brain tumor vasculature.

Learning Optimal Wavefront Shaping for Multi-Channel Imaging

Fast acquisition of depth information is crucial for accurate 3D tracking of moving objects. Snapshot depth sensing can be achieved by wavefront coding, in which the point-spread function (PSF) is engineered to vary distinctively with scene depth by altering the detection optics. In low-light applications, such as 3D localization microscopy, the prevailing approach is to condense signal photons into a single imaging channel with phase-only wavefront modulation to achieve a high pixel-wise signal to noise ratio. Here we show that this paradigm is generally suboptimal and can be significantly improved upon by employing multi-channel wavefront coding, even in low-light applications. We demonstrate our multi-channel optimization scheme on 3D localization microscopy in densely labelled live cells where detectability is limited by overlap of modulated PSFs. At extreme densities, we show that a split-signal system, with end-to-end learned phase masks, doubles the detection rate and reaches improved precision compared to the current state-of-the-art, single-channel design. We implement our method using a bifurcated optical system, experimentally validating our approach by snapshot volumetric imaging and 3D tracking of fluorescently labelled subcellular elements in dense environments.

Deep denoising for multi-dimensional synchrotron X-ray tomography without high-quality reference data

Synchrotron X-ray tomography enables the examination of the internal structure of materials at submicron spatial resolution and subsecond temporal resolution. Unavoidable experimental constraints can impose dose and time limits on the measurements, introducing noise in the reconstructed images. Convolutional neural networks (CNNs) have emerged as a powerful tool to remove noise from reconstructed images. However, their training typically requires collecting a dataset of paired noisy and high-quality measurements, which is a major obstacle to their use in practice. To circumvent this problem, methods for CNN-based denoising have recently been proposed that require no separate training data beyond the already available noisy reconstructions. Among these, the Noise2Inverse method is specifically designed for tomography and related inverse problems. To date, applications of Noise2Inverse have only taken into account 2D spatial information. In this paper, we expand the application of Noise2Inverse in space, time, and spectrum-like domains. This development enhances applications to static and dynamic micro-tomography as well as X-ray diffraction tomography. Results on real-world datasets establish that Noise2Inverse is capable of accurate denoising and enables a substantial reduction in acquisition time while maintaining image quality.

Adaptive adversarial neural networks for the analysis of lossy and domain-shifted datasets of medical images

In machine learning for image-based medical diagnostics, supervised convolutional neural networks are typically trained with large and expertly annotated datasets obtained using high-resolution imaging systems. Moreover, the network's performance can degrade substantially when applied to a dataset with a different distribution. Here, we show that adversarial learning can be used to develop high-performing networks trained on unannotated medical images of varying image quality. Specifically, we used low-quality images acquired using inexpensive portable optical systems to train networks for the evaluation of human embryos, the quantification of human sperm morphology and the diagnosis of malarial infections in the blood, and show that the networks performed well across different data distributions. We also show that adversarial learning can be used with unlabelled data from unseen domain-shifted datasets to adapt pretrained supervised networks to new distributions, even when data from the original distribution are not available. Adaptive adversarial networks may expand the use of validated neural-network models for the evaluation of data collected from multiple imaging systems of varying quality without compromising the knowledge stored in the network.

Image-based cell phenotyping with deep learning

A cell's phenotype is the culmination of several cellular processes through a complex network of molecular interactions that ultimately result in a unique morphological signature. Visual cell phenotyping is the characterization and quantification of these observable cellular traits in images. Recently, cellular phenotyping has undergone a massive overhaul in terms of scale, resolution, and throughput, which is attributable to advances across electronic, optical, and chemical technologies for imaging cells. Coupled with the rapid acceleration of deep learning-based computational tools, these advances have opened up new avenues for innovation across a wide variety of high-throughput cell biology applications. Here, we review applications wherein deep learning is powering the recognition, profiling, and prediction of visual phenotypes to answer important biological questions. As the complexity and scale of imaging assays increase, deep learning offers computational solutions to elucidate the details of previously unexplored cellular phenotypes.

Deep learning-enhanced light-field imaging with continuous validation

Visualizing dynamic processes over large, three-dimensional fields of view at high speed is essential for many applications in the life sciences. Light-field microscopy (LFM) has emerged as a tool for fast volumetric image acquisition, but its effective throughput and widespread use in biology has been hampered by a computationally demanding and artifact-prone image reconstruction process. Here, we present a framework for artificial intelligence-enhanced microscopy, integrating a hybrid light-field light-sheet microscope and deep learning-based volume reconstruction. In our approach, concomitantly acquired, high-resolution two-dimensional light-sheet images continuously serve as training data and validation for the convolutional neural network reconstructing the raw LFM data during extended volumetric time-lapse imaging experiments. Our network delivers high-quality three-dimensional reconstructions at video-rate throughput, which can be further refined based on the high-resolution light-sheet images. We demonstrate the capabilities of our approach by imaging medaka heart dynamics and zebrafish neural activity with volumetric imaging rates up to 100 Hz.

Image enhancement for fluorescence microscopy based on deep learning with prior knowledge of aberration

In this Letter, we propose a deep learning method with prior knowledge of potential aberration to enhance the fluorescence microscopy without additional hardware. The proposed method could effectively reduce noise and improve the peak signal-to-noise ratio of the acquired images at high speed. The enhancement performance and generalization of this method is demonstrated on three commercial fluorescence microscopes. This work provides a computational alternative to overcome the degradation induced by the biological specimen, and it has the potential to be further applied in biological applications.

Special Series Guest Editorial: Artificial Intelligence and Machine Learning in Biomedical Optics

Guest editors Behrouz Shabestri, Mark Anastasio, Baowei Fei, and Frédéric Leblond provide an overview of the JBO Special Series on Artificial Intelligence Machine Learning in Biomedical Optics.

Restoration of Two-Photon Ca2+ Imaging Data Through Model Blind Spatiotemporal Filtering

Two-photon Ca²⁺ imaging is a leading technique for recording neuronal activities in vivo with cellular or subcellular resolution. However, during experiments, the images often suffer from corruption due to complex noises. Therefore, the analysis of Ca²⁺ imaging data requires preprocessing steps, such as denoising, to extract biologically relevant information. We present an approach that facilitates imaging data restoration through image denoising performed by a neural network combining spatiotemporal filtering and model blind learning. Tests with synthetic and real two-photon Ca²⁺ imaging datasets demonstrate that the proposed approach enables efficient restoration of imaging data. In addition, we demonstrate that the proposed approach outperforms the current state-of-the-art methods by evaluating the qualities of the denoising performance of the models quantitatively. Therefore, our method provides an invaluable tool for denoising two-photon Ca²⁺ imaging data by model blind spatiotemporal processing.

Democratising deep learning for microscopy with ZeroCostDL4Mic

Deep Learning (DL) methods are powerful analytical tools for microscopy and can outperform conventional image processing pipelines. Despite the enthusiasm and innovations fuelled by DL technology, the need to access powerful and compatible resources to train DL networks leads to an accessibility barrier that novice users often find difficult to overcome. Here, we present ZeroCostDL4Mic, an entry-level platform simplifying DL access by leveraging the free, cloud-based computational resources of Google Colab. ZeroCostDL4Mic allows researchers with no coding expertise to train and apply key DL networks to perform tasks including segmentation (using U-Net and StarDist), object detection (using YOLOv2), denoising (using CARE and Noise2Void), super-resolution microscopy (using Deep-STORM), and image-to-image translation (using Label-free prediction - fnet, pix2pix and CycleGAN). Importantly, we provide suitable quantitative tools for each network to evaluate model performance, allowing model optimisation. We demonstrate the application of the platform to study multiple biological processes.

Quantification of Dendritic Spines Remodeling under Physiological Stimuli and in Pathological Conditions

Numerous brain diseases are associated with abnormalities in morphology and density of dendritic spines, small membranous protrusions whose structural geometry correlates with the strength of synaptic connections. Thus, the quantitative analysis of dendritic spines remodeling in microscopic images is one of the key elements towards understanding mechanisms of structural neuronal plasticity and bases of brain pathology. In the following article, we review experimental approaches designed to assess quantitative features of dendritic spines under physiological stimuli and in pathological conditions. We compare various methodological pipelines of biological models, sample preparation, data analysis, image acquisition, sample size, and statistical analysis. The methodology and results of relevant experiments are systematically summarized in a tabular form. In particular, we focus on quantitative data regarding the number of animals, cells, dendritic spines, types of studied parameters, size of observed changes, and their statistical significance.

Data science in cell imaging

Cell imaging has entered the 'Big Data' era. New technologies in light microscopy and molecular biology have led to an explosion in high-content, dynamic and multidimensional imaging data. Similar to the 'omics' fields two decades ago, our current ability to process, visualize, integrate and mine this new generation of cell imaging data is becoming a critical bottleneck in advancing cell biology. Computation, traditionally used to quantitatively test specific hypotheses, must now also enable iterative hypothesis generation and testing by deciphering hidden biologically meaningful patterns in complex, dynamic or high-dimensional cell image data. Data science is uniquely positioned to aid in this process. In this Perspective, we survey the rapidly expanding new field of data science in cell imaging. Specifically, we highlight how data science tools are used within current image analysis pipelines, propose a computation-first approach to derive new hypotheses from cell image data, identify challenges and describe the next frontiers where we believe data science will make an impact. We also outline steps to ensure broad access to these powerful tools - democratizing infrastructure availability, developing sensitive, robust and usable tools, and promoting interdisciplinary training to both familiarize biologists with data science and expose data scientists to cell imaging.

Deep learning-based point-scanning super-resolution imaging

Point-scanning imaging systems are among the most widely used tools for high-resolution cellular and tissue imaging, benefiting from arbitrarily defined pixel sizes. The resolution, speed, sample preservation and signal-to-noise ratio (SNR) of point-scanning systems are difficult to optimize simultaneously. We show these limitations can be mitigated via the use of deep learning-based supersampling of undersampled images acquired on a point-scanning system, which we term point-scanning super-resolution (PSSR) imaging. We designed a 'crappifier' that computationally degrades high SNR, high-pixel resolution ground truth images to simulate low SNR, low-resolution counterparts for training PSSR models that can restore real-world undersampled images. For high spatiotemporal resolution fluorescence time-lapse data, we developed a 'multi-frame' PSSR approach that uses information in adjacent frames to improve model predictions. PSSR facilitates point-scanning image acquisition with otherwise unattainable resolution, speed and sensitivity. All the training data, models and code for PSSR are publicly available at 3DEM.org.

Fluorescence imaging in pediatric surgery: State-of-the-art and future perspectives

BACKGROUND

The employment of fluorescence imaging has gained popularity in many fields of adult surgery where it has demonstrated great potentials to improve both surgical and oncological outcomes while minimizing anesthetic time and lowering health-care costs. However, the clinical application of fluorescence-guided surgery (FGS) in pediatrics is just at the initial phase.

MATERIAL AND METHODS

A systematic review of current clinical uses of FGS in pediatric surgery was performed along with a discussion on its advantages, limitations and future developments.

RESULTS

21 studies were included: 9 retrospective and 1 prospective study, 8 case reports, 2 case series and a review article reporting authors' institutional experience. Great emphasis was given to surgical resection of hepatoblastoma and its metastasis (n = 6), real-time imaging of the biliary tree (n = 3) and urogenital system (n = 2). Other current uses concern the assessment of blood perfusion (intestine, n = 3; myocutaneous flap, n = 1; transplanted liver, n = 1) and lymphatic flow imaging (n = 4).

CONCLUSION

Despite a paucity of clinical studies evaluating its role in pediatric surgery, FGS has shown promising results in helping guide tumor resection and improving the accuracy of anatomical delineation.

TYPE OF STUDY

Review article.

LEVEL OF CONFIDENCE

Level IV.

Investigating heterogeneities of live mesenchymal stromal cells using AI-based label-free imaging

Mesenchymal stromal cells (MSCs) are multipotent cells that have great potential for regenerative medicine, tissue repair, and immunotherapy. Unfortunately, the outcomes of MSC-based research and therapies can be highly inconsistent and difficult to reproduce, largely due to the inherently significant heterogeneity in MSCs, which has not been well investigated. To quantify cell heterogeneity, a standard approach is to measure marker expression on the protein level via immunochemistry assays. Performing such measurements non-invasively and at scale has remained challenging as conventional methods such as flow cytometry and immunofluorescence microscopy typically require cell fixation and laborious sample preparation. Here, we developed an artificial intelligence (AI)-based method that converts transmitted light microscopy images of MSCs into quantitative measurements of protein expression levels. By training a U-Net+ conditional generative adversarial network (cGAN) model that accurately (mean [Formula: see text] = 0.77) predicts expression of 8 MSC-specific markers, we showed that expression of surface markers provides a heterogeneity characterization that is complementary to conventional cell-level morphological analyses. Using this label-free imaging method, we also observed a multi-marker temporal-spatial fluctuation of protein distributions in live MSCs. These demonstrations suggest that our AI-based microscopy can be utilized to perform quantitative, non-invasive, single-cell, and multi-marker characterizations of heterogeneous live MSC culture. Our method provides a foundational step toward the instant integrative assessment of MSC properties, which is critical for high-throughput screening and quality control in cellular therapies.

Super-Resolution Enhancement Method Based on Generative Adversarial Network for Integral Imaging Microscopy

The integral imaging microscopy system provides a three-dimensional visualization of a microscopic object. However, it has a low-resolution problem due to the fundamental limitation of the F-number (the aperture stops) by using micro lens array (MLA) and a poor illumination environment. In this paper, a generative adversarial network (GAN)-based super-resolution algorithm is proposed to enhance the resolution where the directional view image is directly fed as input. In a GAN network, the generator regresses the high-resolution output from the low-resolution input image, whereas the discriminator distinguishes between the original and generated image. In the generator part, we use consecutive residual blocks with the content loss to retrieve the photo-realistic original image. It can restore the edges and enhance the resolution by ×2, ×4, and even ×8 times without seriously hampering the image quality. The model is tested with a variety of low-resolution microscopic sample images and successfully generates high-resolution directional view images with better illumination. The quantitative analysis shows that the proposed model performs better for microscopic images than the existing algorithms.

Selective Imaging of Microplastic and Organic Particles in Flow by Multimodal Coherent Anti-Stokes Raman Scattering and Two-Photon Excited Autofluorescence Analysis

Microplastic pollution is an urgent global issue. While spectroscopic techniques have been widely used for the identification of plastics collected from aquatic environments, these techniques are often labor-intensive and time-consuming due to sample collection, preparation, and long measurement times. In this study, a method for the two-dimensional detection and classification of flowing microplastic and organic biotic particles with high spatial and temporal resolutions has been proposed based on the simultaneous detection of coherent anti-Stokes Raman scattering (CARS) and two-photon excited autofluorescence (TPEAF) signals. Poly(methyl methacrylate) (PMMA), polystyrene (PS), and low-density polyethylene (LDPE) particles with sizes ranging from several tens to hundreds of micrometers were selectively detected in flow with an average velocity of 4.17 mm/s by CARS line scanning. With the same flow velocity, flowing PMMA and alga particles were measured using a multimodal system of CARS and TPEAF signals. The average intensities of both PMMA and alga particles in the CARS signals at a frequency of 2940 cm^-1 were higher than the background level, while only algae emitted TPEAF signals. This allowed the classification of PMMA and alga particles to be successfully performed in flow by the simultaneous detection of CARS and TPEAF signals. With the proposed method, the monitoring of microplastics in a continuous water flow without collection or extraction is possible, which is game-changing for the current sampling-based microplastic analysis.

A versatile deep learning architecture for classification and label-free prediction of hyperspectral images

Hyperspectral imaging is a technique that provides rich chemical or compositional information not regularly available to traditional imaging modalities such as intensity imaging or color imaging based on the reflection, transmission, or emission of light. Analysis of hyperspectral imaging often relies on machine learning methods to extract information. Here, we present a new flexible architecture, the U-within-U-Net, that can perform classification, segmentation, and prediction of orthogonal imaging modalities on a variety of hyperspectral imaging techniques. Specifically, we demonstrate feature segmentation and classification on the Indian Pines hyperspectral dataset and simultaneous location prediction of multiple drugs in mass spectrometry imaging of rat liver tissue. We further demonstrate label-free fluorescence image prediction from hyperspectral stimulated Raman scattering microscopy images. The applicability of the U-within-U-Net architecture on diverse datasets with widely varying input and output dimensions and data sources suggest that it has great potential in advancing the use of hyperspectral imaging across many different application areas ranging from remote sensing, to medical imaging, to microscopy.

Advancing Drug Discovery for Neurological Disorders Using iPSC-Derived Neural Organoids

In the last decade, different research groups in the academic setting have developed induced pluripotent stem cell-based protocols to generate three-dimensional, multicellular, neural organoids. Their use to model brain biology, early neural development, and human diseases has provided new insights into the pathophysiology of neuropsychiatric and neurological disorders, including microcephaly, autism, Parkinson’s disease, and Alzheimer’s disease. However, the adoption of organoid technology for large-scale drug screening in the industry has been hampered by challenges with reproducibility, scalability, and translatability to human disease. Potential technical solutions to expand their use in drug discovery pipelines include Clustered Regularly Interspaced Short Palindromic Repeats (CRISPR) to create isogenic models, single-cell RNA sequencing to characterize the model at a cellular level, and machine learning to analyze complex data sets. In addition, high-content imaging, automated liquid handling, and standardized assays represent other valuable tools toward this goal. Though several open issues still hamper the full implementation of the organoid technology outside academia, rapid progress in this field will help to prompt its translation toward large-scale drug screening for neurological disorders.

Unsupervised content-preserving transformation for optical microscopy

The development of deep learning and open access to a substantial collection of imaging data together provide a potential solution for computational image transformation, which is gradually changing the landscape of optical imaging and biomedical research. However, current implementations of deep learning usually operate in a supervised manner, and their reliance on laborious and error-prone data annotation procedures remains a barrier to more general applicability. Here, we propose an unsupervised image transformation to facilitate the utilization of deep learning for optical microscopy, even in some cases in which supervised models cannot be applied. Through the introduction of a saliency constraint, the unsupervised model, named Unsupervised content-preserving Transformation for Optical Microscopy (UTOM), can learn the mapping between two image domains without requiring paired training data while avoiding distortions of the image content. UTOM shows promising performance in a wide range of biomedical image transformation tasks, including in silico histological staining, fluorescence image restoration, and virtual fluorescence labeling. Quantitative evaluations reveal that UTOM achieves stable and high-fidelity image transformations across different imaging conditions and modalities. We anticipate that our framework will encourage a paradigm shift in training neural networks and enable more applications of artificial intelligence in biomedical imaging.

Harnessing non-destructive 3D pathology

High-throughput methods for slide-free three-dimensional (3D) pathological analyses of whole biopsies and surgical specimens offer the promise of modernizing traditional histology workflows and delivering improvements in diagnostic performance. Advanced optical methods now enable the interrogation of orders of magnitude more tissue than previously possible, where volumetric imaging allows for enhanced quantitative analyses of cell distributions and tissue structures that are prognostic and predictive. Non-destructive imaging processes can simplify laboratory workflows, potentially reducing costs, and can ensure that samples are available for subsequent molecular assays. However, the large size of the feature-rich datasets that they generate poses challenges for data management and computer-aided analysis. In this Perspective, we provide an overview of the imaging technologies that enable 3D pathology, and the computational tools-machine learning, in particular-for image processing and interpretation. We also discuss the integration of various other diagnostic modalities with 3D pathology, along with the challenges and opportunities for clinical adoption and regulatory approval.

Analysing errors in single-molecule localisation microscopy

In single molecule localisation microscopy (SMLM) a super-resolution image of the distribution of fluorophores in the sample is built up from the localised positions of many individual molecules. It has become widely used due to its experimental simplicity and the high resolution that can be achieved. However, the factors which limit resolution in a reconstructed image, and the artefacts which can be present, are completely different to those present in standard fluorescent microscopy techniques. Artefacts may be difficult for users to identify, particularly as they can cause images to appear falsely sharp, an effect called artificial sharpening. Here we discuss the different sources of error and bias in SMLM, and the methods available for avoiding or detecting them.

Pushing the super-resolution limit: recent improvements in microscopy below the diffraction limit

Super-resolution microscopy has revolutionised the way we observe biological systems. These methods are now a staple of fluorescence microscopy. Researchers have used super-resolution methods in myriad systems to extract nanoscale spatial information on multiple interacting parts. These methods are continually being extended and reimagined to further push their resolving power and achieve truly single protein resolution. Here, we explore the most recent advances at the frontier of the 'super-resolution' limit and what opportunities remain for further improvements in the near future.

Two-Photon Excitation-Based Imaging Postprocessing Algorithm Model for Background-Free Bioimaging

Bioimaging is a powerful strategy for studying biological activities, which is still limited by the difficulty of distinguishing obscured signals from high background. Despite the development of various new imaging materials and methods, target signals are still likely to be submerged in spontaneous fluorescence or scattering signals. Herein, a novel two-photon excitation-process-based imaging postprocessing algorithm model (2PIA) is introduced to minimize background noise, and triplet-triplet annihilation upconversion metal-organic frameworks (UCMOFs) are chosen as demonstration. Through the collection of several image stacks, the related polynomial of the luminescence intensity and excitation power was established, following splitting the desired signals from noise and obtaining the background-free images definitely. Both in vitro and in vivo experiments show that improved signal visibility is achieved through 2PIA and UCMOFs by removing the interference of scattering, bioluminescence, and other fluorescence materials. The imaging spatial resolution and tissue penetration depth were greatly enhanced. Benefiting from 2PIA, as low as 100 UCMOFs labeled cells can be identified from obscuring background easily after intravenous injection. This image postprocessing method combined with special two-photon excited luminescent materials can conduct biological imaging from complex background interference without using expensive instruments or delicate materials, which holds great promise for accurate biological imaging.

Application of Super-Resolution and Advanced Quantitative Microscopy to the Spatio-Temporal Analysis of Influenza Virus Replication

With an estimated three to five million human cases annually and the potential to infect domestic and wild animal populations, influenza viruses are one of the greatest health and economic burdens to our society, and pose an ongoing threat of large-scale pandemics. Despite our knowledge of many important aspects of influenza virus biology, there is still much to learn about how influenza viruses replicate in infected cells, for instance, how they use entry receptors or exploit host cell trafficking pathways. These gaps in our knowledge are due, in part, to the difficulty of directly observing viruses in living cells. In recent years, advances in light microscopy, including super-resolution microscopy and single-molecule imaging, have enabled many viral replication steps to be visualised dynamically in living cells. In particular, the ability to track single virions and their components, in real time, now allows specific pathways to be interrogated, providing new insights to various aspects of the virus-host cell interaction. In this review, we discuss how state-of-the-art imaging technologies, notably quantitative live-cell and super-resolution microscopy, are providing new nanoscale and molecular insights into influenza virus replication and revealing new opportunities for developing antiviral strategies.

A Single-Shot Autofocus Approach for Surface Plasmon Resonance Microscopy

Surface plasmon resonance microscopy (SPRM) has been widely used as a sensitive imaging platform for chemical and biological analysis. The SPRM system inevitably suffers from focus inhomogeneity and drifts, especially in long-term recordings, leading to distorted images and inaccurate quantification. Traditional focus correction approaches require additional optical parts to detect and adjust focal conditions. Herein, we propose a deep-learning-based image processing method to gain autofocused SPRM images, without increasing the complexity of the optical systems. We trained a generative adversarial network (GAN) model with thousands of SPRM images of nanoparticles acquired at different focal distances. The trained model was able to directly generate focused SPRM images from single-shot defocused images, with no prior knowledge of the focus conditions during recording. Experiments using Au nanoparticles show that this method is effective in both static and time-lapse monitoring. The proposed autofocus technique thus provides an approach for improving the consistency among SPRM studies and for long-term monitoring.

Evaluation and development of deep neural networks for image super-resolution in optical microscopy

Deep neural networks have enabled astonishing transformations from low-resolution (LR) to super-resolved images. However, whether, and under what imaging conditions, such deep-learning models outperform super-resolution (SR) microscopy is poorly explored. Here, using multimodality structured illumination microscopy (SIM), we first provide an extensive dataset of LR-SR image pairs and evaluate the deep-learning SR models in terms of structural complexity, signal-to-noise ratio and upscaling factor. Second, we devise the deep Fourier channel attention network (DFCAN), which leverages the frequency content difference across distinct features to learn precise hierarchical representations of high-frequency information about diverse biological structures. Third, we show that DFCAN's Fourier domain focalization enables robust reconstruction of SIM images under low signal-to-noise ratio conditions. We demonstrate that DFCAN achieves comparable image quality to SIM over a tenfold longer duration in multicolor live-cell imaging experiments, which reveal the detailed structures of mitochondrial cristae and nucleoids and the interaction dynamics of organelles and cytoskeleton.

What machine learning can do for developmental biology

Developmental biology has grown into a data intensive science with the development of high-throughput imaging and multi-omics approaches. Machine learning is a versatile set of techniques that can help make sense of these large datasets with minimal human intervention, through tasks such as image segmentation, super-resolution microscopy and cell clustering. In this Spotlight, I introduce the key concepts, advantages and limitations of machine learning, and discuss how these methods are being applied to problems in developmental biology. Specifically, I focus on how machine learning is improving microscopy and single-cell 'omics' techniques and data analysis. Finally, I provide an outlook for the futures of these fields and suggest ways to foster new interdisciplinary developments.

A Picture Worth a Thousand Molecules-Integrative Technologies for Mapping Subcellular Molecular Organization and Plasticity in Developing Circuits

A key challenge in developmental neuroscience is identifying the local regulatory mechanisms that control neurite and synaptic refinement over large brain volumes. Innovative molecular techniques and high-resolution imaging tools are beginning to reshape our view of how local protein translation in subcellular compartments drives axonal, dendritic, and synaptic development and plasticity. Here we review recent progress in three areas of neurite and synaptic study in situ-compartment-specific transcriptomics/translatomics, targeted proteomics, and super-resolution imaging analysis of synaptic organization and development. We discuss synergies between sequencing and imaging techniques for the discovery and validation of local molecular signaling mechanisms regulating synaptic development, plasticity, and maintenance in circuits.

Single-molecule localization microscopy

Single-molecule localization microscopy (SMLM) describes a family of powerful imaging techniques that dramatically improve spatial resolution over standard, diffraction-limited microscopy techniques and can image biological structures at the molecular scale. In SMLM, individual fluorescent molecules are computationally localized from diffraction-limited image sequences and the localizations are used to generate a super-resolution image or a time course of super-resolution images, or to define molecular trajectories. In this Primer, we introduce the basic principles of SMLM techniques before describing the main experimental considerations when performing SMLM, including fluorescent labelling, sample preparation, hardware requirements and image acquisition in fixed and live cells. We then explain how low-resolution image sequences are computationally processed to reconstruct super-resolution images and/or extract quantitative information, and highlight a selection of biological discoveries enabled by SMLM and closely related methods. We discuss some of the main limitations and potential artefacts of SMLM, as well as ways to alleviate them. Finally, we present an outlook on advanced techniques and promising new developments in the fast-evolving field of SMLM. We hope that this Primer will be a useful reference for both newcomers and practitioners of SMLM.

Batch equalization with a generative adversarial network

Motivation

Advances in automation and imaging have made it possible to capture a large image dataset that spans multiple experimental batches of data. However, accurate biological comparison across the batches is challenged by batch-to-batch variation (i.e. batch effect) due to uncontrollable experimental noise (e.g. varying stain intensity or cell density). Previous approaches to minimize the batch effect have commonly focused on normalizing the low-dimensional image measurements such as an embedding generated by a neural network. However, normalization of the embedding could suffer from over-correction and alter true biological features (e.g. cell size) due to our limited ability to interpret the effect of the normalization on the embedding space. Although techniques like flat-field correction can be applied to normalize the image values directly, they are limited transformations that handle only simple artifacts due to batch effect.

Results

We present a neural network-based batch equalization method that can transfer images from one batch to another while preserving the biological phenotype. The equalization method is trained as a generative adversarial network (GAN), using the StarGAN architecture that has shown considerable ability in style transfer. After incorporating new objectives that disentangle batch effect from biological features, we show that the equalized images have less batch information and preserve the biological information. We also demonstrate that the same model training parameters can generalize to two dramatically different types of cells, indicating this approach could be broadly applicable.

Availability and implementation

https://github.com/tensorflow/gan/tree/master/tensorflow_gan/examples/stargan

Supplementary information

Supplementary data are available at Bioinformatics online.

Establishment of a morphological atlas of the Caenorhabditis elegans embryo using deep-learning-based 4D segmentation

The invariant development and transparent body of the nematode Caenorhabditis elegans enables complete delineation of cell lineages throughout development. Despite extensive studies of cell division, cell migration and cell fate differentiation, cell morphology during development has not yet been systematically characterized in any metazoan, including C. elegans. This knowledge gap substantially hampers many studies in both developmental and cell biology. Here we report an automatic pipeline, CShaper, which combines automated segmentation of fluorescently labeled membranes with automated cell lineage tracing. We apply this pipeline to quantify morphological parameters of densely packed cells in 17 developing C. elegans embryos. Consequently, we generate a time-lapse 3D atlas of cell morphology for the C. elegans embryo from the 4- to 350-cell stages, including cell shape, volume, surface area, migration, nucleus position and cell-cell contact with resolved cell identities. We anticipate that CShaper and the morphological atlas will stimulate and enhance further studies in the fields of developmental biology, cell biology and biomechanics.

The systematic characterization of C. elegans morphology during development has yet to be performed. Here, the authors produce a 3D atlas of C. elegans morphology from 17 embryos and 54 developmental stages, using an automated pipeline, CShaper (combining segmentation of fluorescently labeled membranes with automated cell lineage tracing).

Resolution enhancement and realistic speckle recovery with generative adversarial modeling of micro-optical coherence tomography

A resolution enhancement technique for optical coherence tomography (OCT), based on Generative Adversarial Networks (GANs), was developed and investigated. GANs have been previously used for resolution enhancement of photography and optical microscopy images. We have adapted and improved this technique for OCT image generation. Conditional GANs (cGANs) were trained on a novel set of ultrahigh resolution spectral domain OCT volumes, termed micro-OCT, as the high-resolution ground truth (∼1 μm isotropic resolution). The ground truth was paired with a low-resolution image obtained by synthetically degrading resolution 4x in one of (1-D) or both axial and lateral axes (2-D). Cross-sectional image (B-scan) volumes obtained from in vivo imaging of human labial (lip) tissue and mouse skin were used in separate feasibility experiments. Accuracy of resolution enhancement compared to ground truth was quantified with human perceptual accuracy tests performed by an OCT expert. The GAN loss in the optimization objective, noise injection in both the generator and discriminator models, and multi-scale discrimination were found to be important for achieving realistic speckle appearance in the generated OCT images. The utility of high-resolution speckle recovery was illustrated by an example of micro-OCT imaging of blood vessels in lip tissue. Qualitative examples applying the models to image data from outside of the training data distribution, namely human retina and mouse bladder, were also demonstrated, suggesting potential for cross-domain transferability. This preliminary study suggests that deep learning generative models trained on OCT images from high-performance prototype systems may have potential in enhancing lower resolution data from mainstream/commercial systems, thereby bringing cutting-edge technology to the masses at low cost.

Practical fluorescence reconstruction microscopy for large samples and low-magnification imaging

Fluorescence reconstruction microscopy (FRM) describes a class of techniques where transmitted light images are passed into a convolutional neural network that then outputs predicted epifluorescence images. This approach enables many benefits including reduced phototoxicity, freeing up of fluorescence channels, simplified sample preparation, and the ability to re-process legacy data for new insights. However, FRM can be complex to implement, and current FRM benchmarks are abstractions that are difficult to relate to how valuable or trustworthy a reconstruction is. Here, we relate the conventional benchmarks and demonstrations to practical and familiar cell biology analyses to demonstrate that FRM should be judged in context. We further demonstrate that it performs remarkably well even with lower-magnification microscopy data, as are often collected in screening and high content imaging. Specifically, we present promising results for nuclei, cell-cell junctions, and fine feature reconstruction; provide data-driven experimental design guidelines; and provide researcher-friendly code, complete sample data, and a researcher manual to enable more widespread adoption of FRM.

Autofocusing technologies for whole slide imaging and automated microscopy

Whole slide imaging (WSI) has moved digital pathology closer to diagnostic practice in recent years. Due to the inherent tissue topography variability, accurate autofocusing remains a critical challenge for WSI and automated microscopy systems. The traditional focus map surveying method is limited in its ability to acquire a high degree of focus points while still maintaining high throughput. Real-time approaches decouple image acquisition from focusing, thus allowing for rapid scanning while maintaining continuous accurate focus. This work reviews the traditional focus map approach and discusses the choice of focus measure for focal plane determination. It also discusses various real-time autofocusing approaches including reflective-based triangulation, confocal pinhole detection, low-coherence interferometry, tilted sensor approach, independent dual sensor scanning, beam splitter array, phase detection, dual-LED illumination and deep-learning approaches. The technical concepts, merits and limitations of these methods are explained and compared to those of a traditional WSI system. This review may provide new insights for the development of high-throughput automated microscopy imaging systems that can be made broadly available and utilizable without loss of capacity.

Alternation of inverse problem approach and deep learning for lens-free microscopy image reconstruction

A lens-free microscope is a simple imaging device performing in-line holographic measurements. In the absence of focusing optics, a reconstruction algorithm is used to retrieve the sample image by solving the inverse problem. This is usually performed by optimization algorithms relying on gradient computation. However the presence of local minima leads to unsatisfactory convergence when phase wrapping errors occur. This is particularly the case in large optical thickness samples, for example cells in suspension and cells undergoing mitosis. To date, the occurrence of phase wrapping errors in the holographic reconstruction limits the application of lens-free microscopy in live cell imaging. To overcome this issue, we propose a novel approach in which the reconstruction alternates between two approaches, an inverse problem optimization and deep learning. The computation starts with a first reconstruction guess of the cell sample image. The result is then fed into a neural network, which is trained to correct phase wrapping errors. The neural network prediction is next used as the initialization of a second and last reconstruction step, which corrects to a certain extent the neural network prediction errors. We demonstrate the applicability of this approach in solving the phase wrapping problem occurring with cells in suspension at large densities. This is a challenging sample that typically cannot be reconstructed without phase wrapping errors, when using inverse problem optimization alone.

Recent advances in point spread function engineering and related computational microscopy approaches: from one viewpoint

This personal hybrid review piece, written in light of my recipience of the UIPAB 2020 young investigator award, contains a mixture of my scientific biography and work so far. This paper is not intended to be a comprehensive review, but only to highlight my contributions to computation-related aspects of super-resolution microscopy, as well as their origins and future directions.

Rapid mono and biexponential 3D-T1ρ mapping of knee cartilage using variational networks

In this study we use undersampled MRI acquisition methods to obtain accelerated 3D mono and biexponential spin-lattice relaxation time in the rotating frame (T1ρ) mapping of knee cartilage, reducing the usual long scan time. We compare the accelerated T1ρ maps obtained by deep learning-based variational network (VN) and compressed sensing (CS). Both methods were compared with spatial (S) and spatio-temporal (ST) filters. Complex-valued fitting was used for T1ρ parameters estimation. We tested with seven in vivo and six synthetic datasets, with acceleration factors (AF) from 2 to 10. Median normalized absolute deviation (MNAD), analysis of variance (ANOVA), and coefficient of variation (CV) were used for analysis. The methods CS-ST, VN-S, and VN-ST performed well for accelerating monoexponential T1ρ mapping, with MNAD around 5% for AF = 2, which increases almost linearly with the AF to an MNAD of 13% for AF = 8, with all methods. For biexponential mapping, the VN-ST was the best method starting with MNAD of 7.4% for AF = 2 and reaching MNAD of 13.1% for AF = 8. The VN was able to produce 3D-T1ρ mapping of knee cartilage with lower error than CS. The best results were obtained by VN-ST, improving CS-ST method by nearly 7.5%.

Topaz-Denoise: general deep denoising models for cryoEM and cryoET

Cryo-electron microscopy (cryoEM) is becoming the preferred method for resolving protein structures. Low signal-to-noise ratio (SNR) in cryoEM images reduces the confidence and throughput of structure determination during several steps of data processing, resulting in impediments such as missing particle orientations. Denoising cryoEM images can not only improve downstream analysis but also accelerate the time-consuming data collection process by allowing lower electron dose micrographs to be used for analysis. Here, we present Topaz-Denoise, a deep learning method for reliably and rapidly increasing the SNR of cryoEM images and cryoET tomograms. By training on a dataset composed of thousands of micrographs collected across a wide range of imaging conditions, we are able to learn models capturing the complexity of the cryoEM image formation process. The general model we present is able to denoise new datasets without additional training. Denoising with this model improves micrograph interpretability and allows us to solve 3D single particle structures of clustered protocadherin, an elongated particle with previously elusive views. We then show that low dose collection, enabled by Topaz-Denoise, improves downstream analysis in addition to reducing data collection time. We also present a general 3D denoising model for cryoET. Topaz-Denoise and pre-trained general models are now included in Topaz. We expect that Topaz-Denoise will be of broad utility to the cryoEM community for improving micrograph and tomogram interpretability and accelerating analysis.

DeepACP: A Novel Computational Approach for Accurate Identification of Anticancer Peptides by Deep Learning Algorithm

Cancer is one of the most dangerous diseases to human health. The accurate prediction of anticancer peptides (ACPs) would be valuable for the development and design of novel anticancer agents. Current deep neural network models have obtained state-of-the-art prediction accuracy for the ACP classification task. However, based on existing studies, it remains unclear which deep learning architecture achieves the best performance. Thus, in this study, we first present a systematic exploration of three important deep learning architectures: convolutional, recurrent, and convolutional-recurrent networks for distinguishing ACPs from non-ACPs. We find that the recurrent neural network with bidirectional long short-term memory cells is superior to other architectures. By utilizing the proposed model, we implement a sequence-based deep learning tool (DeepACP) to accurately predict the likelihood of a peptide exhibiting anticancer activity. The results indicate that DeepACP outperforms several existing methods and can be used as an effective tool for the prediction of anticancer peptides. Furthermore, we visualize and understand the deep learning model. We hope that our strategy can be extended to identify other types of peptides and may provide more assistance to the development of proteomics and new drugs.

Deep learning-enabled efficient image restoration for 3D microscopy of turbid biological specimens

Though three-dimensional (3D) fluorescence microscopy has been an essential tool for modern life science research, the light scattering by biological specimens fundamentally prevents its more widespread applications in live imaging. We hereby report a deep-learning approach, termed ScatNet, that enables reversion of 3D fluorescence microscopy from high-resolution targets to low-quality, light-scattered measurements, thereby allowing restoration for a blurred and light-scattered 3D image of deep tissue. Our approach can computationally extend the imaging depth for current 3D fluorescence microscopes, without the addition of complicated optics. Combining ScatNet approach with cutting-edge light-sheet fluorescence microscopy (LSFM), we demonstrate the image restoration of cell nuclei in the deep layer of live Drosophilamelanogaster embryos at single-cell resolution. Applying our approach to two-photon excitation microscopy, we could improve the signal-to-noise ratio (SNR) and resolution of neurons in mouse brain beyond the photon ballistic region.

Virtual organelle self-coding for fluorescence imaging via adversarial learning

SIGNIFICANCE

Our study introduces an application of deep learning to virtually generate fluorescence images to reduce the burdens of cost and time from considerable effort in sample preparation related to chemical fixation and staining.

AIM

The objective of our work was to determine how successfully deep learning methods perform on fluorescence prediction that depends on structural and/or a functional relationship between input labels and output labels.

APPROACH

We present a virtual-fluorescence-staining method based on deep neural networks (VirFluoNet) to transform co-registered images of cells into subcellular compartment-specific molecular fluorescence labels in the same field-of-view. An algorithm based on conditional generative adversarial networks was developed and trained on microscopy datasets from breast-cancer and bone-osteosarcoma cell lines: MDA-MB-231 and U2OS, respectively. Several established performance metrics-the mean absolute error (MAE), peak-signal-to-noise ratio (PSNR), and structural-similarity-index (SSIM)-as well as a novel performance metric, the tolerance level, were measured and compared for the same algorithm and input data.

RESULTS

For the MDA-MB-231 cells, F-actin signal performed the fluorescent antibody staining of vinculin prediction better than phase-contrast as an input. For the U2OS cells, satisfactory metrics of performance were archieved in comparison with ground truth. MAE is <0.005, 0.017, 0.012; PSNR is >40  /  34  /  33  dB; and SSIM is >0.925  /  0.926  /  0.925 for 4',6-diamidino-2-phenylindole/hoechst, endoplasmic reticulum, and mitochondria prediction, respectively, from channels of nucleoli and cytoplasmic RNA, Golgi plasma membrane, and F-actin.

CONCLUSIONS

These findings contribute to the understanding of the utility and limitations of deep learning image-regression to predict fluorescence microscopy datasets of biological cells. We infer that predicted image labels must have either a structural and/or a functional relationship to input labels. Furthermore, the approach introduced here holds promise for modeling the internal spatial relationships between organelles and biomolecules within living cells, leading to detection and quantification of alterations from a standard training dataset.

Correction of refractive index mismatch-induced aberrations under radially polarized illumination by deep learning

Radially polarized field under strong focusing has emerged as a powerful manner for fluorescence microscopy. However, the refractive index (RI) mismatch-induced aberrations seriously degrade imaging performance, especially under high numerical aperture (NA). Traditional adaptive optics (AO) method is limited by its tedious procedure. Here, we present a computational strategy that uses artificial neural networks to correct the aberrations induced by RI mismatch. There are no requirements for expensive hardware and complicated wavefront sensing in our framework when the deep network training is completed. The structural similarity index (SSIM) criteria and spatial frequency spectrum analysis demonstrate that our deep-learning-based method has a better performance compared to the widely used Richardson-Lucy (RL) deconvolution method at different imaging depth on simulation data. Additionally, the generalization of our trained network model is tested on new types of samples that are not present in the training procedure to further evaluate the utility of the network, and the performance is also superior to RL deconvolution.

Revealing architectural order with quantitative label-free imaging and deep learning

We report quantitative label-free imaging with phase and polarization (QLIPP) for simultaneous measurement of density, anisotropy, and orientation of structures in unlabeled live cells and tissue slices. We combine QLIPP with deep neural networks to predict fluorescence images of diverse cell and tissue structures. QLIPP images reveal anatomical regions and axon tract orientation in prenatal human brain tissue sections that are not visible using brightfield imaging. We report a variant of U-Net architecture, multi-channel 2.5D U-Net, for computationally efficient prediction of fluorescence images in three dimensions and over large fields of view. Further, we develop data normalization methods for accurate prediction of myelin distribution over large brain regions. We show that experimental defects in labeling the human tissue can be rescued with quantitative label-free imaging and neural network model. We anticipate that the proposed method will enable new studies of architectural order at spatial scales ranging from organelles to tissue.

The future of cerebral organoids in drug discovery

Until the discovery of human embryonic stem cells and human induced pluripotent stem cells, biotechnology companies were severely limited in the number of human tissues that they could model in large-scale in vitro studies. Until this point, companies have been limited to immortalized cancer lines or a small number of primary cell types that could be extracted and expanded. Nowadays, protocols continue to be developed in the stem cell field, enabling researchers to model an ever-growing library of cell types in controlled, large-scale screens. One differentiation method in particular- cerebral organoids- shows substantial potential in the field of neuroscience and developmental neurobiology. Cerebral organoid technology is still in an early phase of development, and there are several challenges that are currently being addressed by academic and industrial researchers alike. Here we briefly describe some of the early adopters of cerebral organoids, several of the challenges that they are likely facing, and various technologies that are currently being implemented to overcome them.

Single-Cell Classification Using Mass Spectrometry through Interpretable Machine Learning

The brain consists of organized ensembles of cells that exhibit distinct morphologies, cellular connectivity, and dynamic biochemistries that control the executive functions of an organism. However, the relationships between chemical heterogeneity, cell function, and phenotype are not always understood. Recent advancements in matrix-assisted laser desorption/ionization mass spectrometry have enabled the high-throughput, multiplexed chemical analysis of single cells, capable of resolving hundreds of molecules in each mass spectrum. We developed a machine learning workflow to classify single cells according to their mass spectra based on cell groups of interest (GOI), e.g., neurons vs astrocytes. Three data sets from various cell groups were acquired on three different mass spectrometer platforms representing thousands of individual cell spectra that were collected and used to validate the single cell classification workflow. The trained models achieved >80% classification accuracy and were subjected to the recently developed instance-based model interpretation framework, SHapley Additive exPlanations (SHAP), which locally assigns feature importance for each single-cell spectrum. SHAP values were used for both local and global interpretations of our data sets, preserving the chemical heterogeneity uncovered by the single-cell analysis while offering the ability to perform supervised analysis. The top contributing mass features to each of the GOI were ranked and selected using mean absolute SHAP values, highlighting the features that are specific to the defined GOI. Our approach provides insight into discriminating the chemical profiles of the single cells through interpretable machine learning, facilitating downstream analysis and validation.

Deep-learning-based whole-brain imaging at single-neuron resolution

Obtaining fine structures of neurons is necessary for understanding brain function. Simple and effective methods for large-scale 3D imaging at optical resolution are still lacking. Here, we proposed a deep-learning-based fluorescence micro-optical sectioning tomography (DL-fMOST) method for high-throughput, high-resolution whole-brain imaging. We utilized a wide-field microscope for imaging, a U-net convolutional neural network for real-time optical sectioning, and histological sectioning for exceeding the imaging depth limit. A 3D dataset of a mouse brain with a voxel size of 0.32 × 0.32 × 2 µm was acquired in 1.5 days. We demonstrated the robustness of DL-fMOST for mouse brains with labeling of different types of neurons.