DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning

Single Molecules Are Your Quanta: A Bottom-Up Approach toward Multidimensional Super-resolution Microscopy

The rise of single-molecule localization microscopy (SMLM) and related super-resolution methods over the past 15 years has revolutionized how we study biological and materials systems. In this Perspective, we reflect on the underlying philosophy of how diffraction-unlimited pictures containing rich spatial and functional information may gradually emerge through the local accumulation of single-molecule measurements. Starting with the basic concepts, we analyze the uniqueness of and opportunities in building up the final picture one molecule at a time. After brief introductions to the more established multicolor and three-dimensional measurements, we highlight emerging efforts to extend SMLM to new dimensions and functionalities as fluorescence polarization, emission spectra, and molecular motions, and discuss rising opportunities and future directions. With single molecules as our quanta, the bottom-up accumulation approach provides a powerful conduit for multidimensional microscopy at the nanoscale.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Deep-ROCS: from speckle patterns to superior-resolved images by deep learning in rotating coherent scattering microscopy

Rotating coherent scattering (ROCS) microscopy is a label-free imaging technique that overcomes the optical diffraction limit by adding up the scattered laser light from a sample obliquely illuminated from different angles. Although ROCS imaging achieves 150 nm spatial and 10 ms temporal resolution, simply summing different speckle patterns may cause loss of sample information. In this paper we present Deep-ROCS, a neural network-based technique that generates a superior-resolved image by efficient numerical combination of a set of differently illuminated images. We show that Deep-ROCS can reconstruct super-resolved images more accurately than conventional ROCS microscopy, retrieving high-frequency information from a small number (6) of speckle images. We demonstrate the performance of Deep-ROCS experimentally on 200 nm beads and by computer simulations, where we show its potential for even more complex structures such as a filament network.

Multiplexed PSF Engineering for Three-Dimensional Multicolor Particle Tracking

Three-dimensional spatiotemporal tracking of microscopic particles in multiple colors is a challenging optical imaging task. Existing approaches require a trade-off between photon efficiency, field of view, mechanical complexity, spectral specificity, and speed. Here, we introduce multiplexed point-spread-function engineering that achieves photon-efficient, 3D multicolor particle tracking over a large field of view. This is accomplished by first chromatically splitting the emission path of a microscope to different channels, engineering the point-spread function of each, and then recombining them onto the same region of the camera. We demonstrate our technique for simultaneously tracking five types of emitters in vitro as well as colocalization of DNA loci in live yeast cells.

Learning Optimal Wavefront Shaping for Multi-Channel Imaging

Fast acquisition of depth information is crucial for accurate 3D tracking of moving objects. Snapshot depth sensing can be achieved by wavefront coding, in which the point-spread function (PSF) is engineered to vary distinctively with scene depth by altering the detection optics. In low-light applications, such as 3D localization microscopy, the prevailing approach is to condense signal photons into a single imaging channel with phase-only wavefront modulation to achieve a high pixel-wise signal to noise ratio. Here we show that this paradigm is generally suboptimal and can be significantly improved upon by employing multi-channel wavefront coding, even in low-light applications. We demonstrate our multi-channel optimization scheme on 3D localization microscopy in densely labelled live cells where detectability is limited by overlap of modulated PSFs. At extreme densities, we show that a split-signal system, with end-to-end learned phase masks, doubles the detection rate and reaches improved precision compared to the current state-of-the-art, single-channel design. We implement our method using a bifurcated optical system, experimentally validating our approach by snapshot volumetric imaging and 3D tracking of fluorescently labelled subcellular elements in dense environments.

3D printable diffractive optical elements by liquid immersion

Diffractive optical elements (DOEs) are used to shape the wavefront of incident light. This can be used to generate practically any pattern of interest, albeit with varying efficiency. A fundamental challenge associated with DOEs comes from the nanoscale-precision requirements for their fabrication. Here we demonstrate a method to controllably scale up the relevant feature dimensions of a device from tens-of-nanometers to tens-of-microns by immersing the DOEs in a near-index-matched solution. This makes it possible to utilize modern 3D-printing technologies for fabrication, thereby significantly simplifying the production of DOEs and decreasing costs by orders of magnitude, without hindering performance. We demonstrate the tunability of our design for varying experimental conditions, and the suitability of this approach to ultrasensitive applications by localizing the 3D positions of single molecules in cells using our microscale fabricated optical element to modify the point-spread-function (PSF) of a microscope.

Spatial and temporal super-resolution for fluorescence microscopy by a recurrent neural network

A novel spatial and temporal super-resolution (SR) framework based on a recurrent neural network (RNN) is demonstrated. In this work, we learn the complex yet useful features from the temporal data by taking advantage of structural characteristics of RNN and a skip connection. The usage of supervision mechanism is not only making full use of the intermediate output of each recurrent layer to recover the final output, but also alleviating vanishing/exploding gradients during the back-propagation. The proposed scheme achieves excellent reconstruction results, improving both the spatial and temporal resolution of fluorescence images including the simulated and real tubulin datasets. Besides, robustness against various critical metrics, such as the full-width at half-maximum (FWHM) and molecular density, can also be incorporated. In the validation, the performance can be increased by more than 20% for intensity profile, and 8% for FWHM, and the running time can be saved at least 40% compared with the classic Deep-STORM method, a high-performance net which is popularly used for comparison.

DeepSinse: deep learning based detection of single molecules

MOTIVATION

Imaging single molecules has emerged as a powerful characterization tool in the biological sciences. The detection of these under various noise conditions requires the use of algorithms that are dependent on the end-user inputting several parameters, the choice of which can be challenging and subjective.

RESULTS

In this work, we propose DeepSinse, an easily-trainable and useable deep neural network that can detect single molecules with little human input and across a wide range of signal-to-noise ratios. We validate the neural network on the detection of single bursts in simulated and experimental data and compare its performance with the best-in-class, domain-specific algorithms.

AVAILABILITY

Ground truth ROI simulating code, neural network training, validation code, classification code, ROI picker, GUI for simulating, training and validating DeepSinse as well as pre-trained networks are all released under the MIT License on www.github.com/jdanial/DeepSinse.The dSTORM dataset processing code is released under the MIT License on www.github.com/jdanial/StormProcessor.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Data science in cell imaging

Cell imaging has entered the 'Big Data' era. New technologies in light microscopy and molecular biology have led to an explosion in high-content, dynamic and multidimensional imaging data. Similar to the 'omics' fields two decades ago, our current ability to process, visualize, integrate and mine this new generation of cell imaging data is becoming a critical bottleneck in advancing cell biology. Computation, traditionally used to quantitatively test specific hypotheses, must now also enable iterative hypothesis generation and testing by deciphering hidden biologically meaningful patterns in complex, dynamic or high-dimensional cell image data. Data science is uniquely positioned to aid in this process. In this Perspective, we survey the rapidly expanding new field of data science in cell imaging. Specifically, we highlight how data science tools are used within current image analysis pipelines, propose a computation-first approach to derive new hypotheses from cell image data, identify challenges and describe the next frontiers where we believe data science will make an impact. We also outline steps to ensure broad access to these powerful tools - democratizing infrastructure availability, developing sensitive, robust and usable tools, and promoting interdisciplinary training to both familiarize biologists with data science and expose data scientists to cell imaging.

Review of deep learning: concepts, CNN architectures, challenges, applications, future directions

In the last few years, the deep learning (DL) computing paradigm has been deemed the Gold Standard in the machine learning (ML) community. Moreover, it has gradually become the most widely used computational approach in the field of ML, thus achieving outstanding results on several complex cognitive tasks, matching or even beating those provided by human performance. One of the benefits of DL is the ability to learn massive amounts of data. The DL field has grown fast in the last few years and it has been extensively used to successfully address a wide range of traditional applications. More importantly, DL has outperformed well-known ML techniques in many domains, e.g., cybersecurity, natural language processing, bioinformatics, robotics and control, and medical information processing, among many others. Despite it has been contributed several works reviewing the State-of-the-Art on DL, all of them only tackled one aspect of the DL, which leads to an overall lack of knowledge about it. Therefore, in this contribution, we propose using a more holistic approach in order to provide a more suitable starting point from which to develop a full understanding of DL. Specifically, this review attempts to provide a more comprehensive survey of the most important aspects of DL and including those enhancements recently added to the field. In particular, this paper outlines the importance of DL, presents the types of DL techniques and networks. It then presents convolutional neural networks (CNNs) which the most utilized DL network type and describes the development of CNNs architectures together with their main features, e.g., starting with the AlexNet network and closing with the High-Resolution network (HR.Net). Finally, we further present the challenges and suggested solutions to help researchers understand the existing research gaps. It is followed by a list of the major DL applications. Computational tools including FPGA, GPU, and CPU are summarized along with a description of their influence on DL. The paper ends with the evolution matrix, benchmark datasets, and summary and conclusion.

Challenges facing quantitative large-scale optical super-resolution, and some simple solutions

Optical super-resolution microscopy (SRM) has enabled biologists to visualize cellular structures with near-molecular resolution, giving unprecedented access to details about the amounts, sizes, and spatial distributions of macromolecules in the cell. Precisely quantifying these molecular details requires large datasets of high-quality, reproducible SRM images. In this review, we discuss the unique set of challenges facing quantitative SRM, giving particular attention to the shortcomings of conventional specimen preparation techniques and the necessity for optimal labeling of molecular targets. We further discuss the obstacles to scaling SRM methods, such as lengthy image acquisition and complex SRM data analysis. For each of these challenges, we review the recent advances in the field that circumvent these pitfalls and provide practical advice to biologists for optimizing SRM experiments.

Analysing errors in single-molecule localisation microscopy

In single molecule localisation microscopy (SMLM) a super-resolution image of the distribution of fluorophores in the sample is built up from the localised positions of many individual molecules. It has become widely used due to its experimental simplicity and the high resolution that can be achieved. However, the factors which limit resolution in a reconstructed image, and the artefacts which can be present, are completely different to those present in standard fluorescent microscopy techniques. Artefacts may be difficult for users to identify, particularly as they can cause images to appear falsely sharp, an effect called artificial sharpening. Here we discuss the different sources of error and bias in SMLM, and the methods available for avoiding or detecting them.

Single-molecule orientation localization microscopy II: a performance comparison

Various techniques have been developed to measure the 2D and 3D positions and 2D and 3D orientations of fluorescent molecules with improved precision over standard epifluorescence microscopes. Due to the challenging signal-to-background ratio in typical single-molecule experiments, it is essential to choose an imaging system optimized for the specific target sample. In this work, we compare the performance of multiple state-of-the-art and commonly used methods for orientation localization microscopy against the fundamental limits of measurement precision. Our analysis reveals optimal imaging methods for various experiment conditions and sample geometries. Interestingly, simple modifications to the standard fluorescence microscope exhibit superior performance in many imaging scenarios.

Accelerating 3D single-molecule localization microscopy using blind sparse inpainting

SIGNIFICANCE

Single-molecule localization-based super-resolution microscopy has enabled the imaging of microscopic objects beyond the diffraction limit. However, this technique is limited by the requirements of imaging an extremely large number of frames of biological samples to generate a super-resolution image, thus requiring a longer acquisition time. Additionally, the processing of such a large image sequence leads to longer data processing time. Therefore, accelerating image acquisition and processing in single-molecule localization microscopy (SMLM) has been of perennial interest.

AIM

To accelerate three-dimensional (3D) SMLM imaging by leveraging a computational approach without compromising the resolution.

APPROACH

We used blind sparse inpainting to reconstruct high-density 3D images from low-density ones. The low-density images are generated using much fewer frames than usually needed, thus requiring a shorter acquisition and processing time. Therefore, our technique will accelerate 3D SMLM without changing the existing standard SMLM hardware system and labeling protocol.

RESULTS

The performance of the blind sparse inpainting was evaluated on both simulation and experimental datasets. Superior reconstruction results of 3D SMLM images using up to 10-fold fewer frames in simulation and up to 50-fold fewer frames in experimental data were achieved.

CONCLUSIONS

We demonstrate the feasibility of fast 3D SMLM imaging leveraging a computational approach to reduce the number of acquired frames. We anticipate our technique will enable future real-time live-cell 3D imaging to investigate complex nanoscopic biological structures and their functions.

Single-molecule localization microscopy

Single-molecule localization microscopy (SMLM) describes a family of powerful imaging techniques that dramatically improve spatial resolution over standard, diffraction-limited microscopy techniques and can image biological structures at the molecular scale. In SMLM, individual fluorescent molecules are computationally localized from diffraction-limited image sequences and the localizations are used to generate a super-resolution image or a time course of super-resolution images, or to define molecular trajectories. In this Primer, we introduce the basic principles of SMLM techniques before describing the main experimental considerations when performing SMLM, including fluorescent labelling, sample preparation, hardware requirements and image acquisition in fixed and live cells. We then explain how low-resolution image sequences are computationally processed to reconstruct super-resolution images and/or extract quantitative information, and highlight a selection of biological discoveries enabled by SMLM and closely related methods. We discuss some of the main limitations and potential artefacts of SMLM, as well as ways to alleviate them. Finally, we present an outlook on advanced techniques and promising new developments in the fast-evolving field of SMLM. We hope that this Primer will be a useful reference for both newcomers and practitioners of SMLM.

5D superresolution imaging for a live cell nucleus

With a spatial resolution breaking the diffraction limit of light, superresolution imaging allows the visualization of detailed structures of organelles such as mitochondria, cytoskeleton, nucleus, and so on. With multi-dimensional imaging (x, y, z, t, λ), namely, multi-color 3D live imaging enables us fully understand the function of the cell. It is necessary to analyze structural changes or molecular interactions across a large volume in 3D with different labelled targets. To achieve this goal, scientists recently have expanded the original 2D superresolution microscopic tools into 3D imaging techniques. In this review, we will discuss recent development in superresolution microscopy for live imaging with minimal phototoxicity. We will focus our discussion on the cell nucleus where the genetic materials are stored and processed. Machine learning algorism will be introduced to improve the axial resolution of superresolution imaging.

Axially resolved volumetric two-photon microscopy with an extended field of view using depth localization under mirrored Airy beams

It is a great challenge in two-photon microscopy (2PM) to have a high volumetric imaging speed without sacrificing the spatial and temporal resolution in three dimensions (3D). The structure in 2PM images could be reconstructed with better spatial and temporal resolution by the proper choice of the data processing algorithm. Here, we propose a method to reconstruct 3D volume from 2D projections imaged by mirrored Airy beams. We verified that our approach can achieve high accuracy in 3D localization over a large axial range and is applicable to continuous and dense sample. The effective field of view after reconstruction is expanded. It is a promising technique for rapid volumetric 2PM with axial localization at high resolution.

Inference in artificial intelligence with deep optics and photonics

Artificial intelligence tasks across numerous applications require accelerators for fast and low-power execution. Optical computing systems may be able to meet these domain-specific needs but, despite half a century of research, general-purpose optical computing systems have yet to mature into a practical technology. Artificial intelligence inference, however, especially for visual computing applications, may offer opportunities for inference based on optical and photonic systems. In this Perspective, we review recent work on optical computing for artificial intelligence applications and discuss its promise and challenges.

Recent advances in point spread function engineering and related computational microscopy approaches: from one viewpoint

This personal hybrid review piece, written in light of my recipience of the UIPAB 2020 young investigator award, contains a mixture of my scientific biography and work so far. This paper is not intended to be a comprehensive review, but only to highlight my contributions to computation-related aspects of super-resolution microscopy, as well as their origins and future directions.

Super-resolution Microscopy with Single Molecules in Biology and Beyond-Essentials, Current Trends, and Future Challenges

Single-molecule super-resolution microscopy has developed from a specialized technique into one of the most versatile and powerful imaging methods of the nanoscale over the past two decades. In this perspective, we provide a brief overview of the historical development of the field, the fundamental concepts, the methodology required to obtain maximum quantitative information, and the current state of the art. Then, we will discuss emerging perspectives and areas where innovation and further improvement are needed. Despite the tremendous progress, the full potential of single-molecule super-resolution microscopy is yet to be realized, which will be enabled by the research ahead of us.

100,000 frames-per-second compressive imaging with a conventional rolling-shutter camera by random point-spread-function engineering

We demonstrate an approach that allows taking videos at very high frame-rates of over 100,000 frames per second by exploiting the fast sampling rate of the standard rolling-shutter readout mechanism, common to most conventional sensors, and a compressive-sampling acquisition scheme. Our approach is directly applied to a conventional imaging system by the simple addition of a diffuser to the pupil plane that randomly encodes the entire field-of-view to each camera row, while maintaining diffraction-limited resolution. A short video is reconstructed from a single camera frame via a compressed-sensing reconstruction algorithm, exploiting the inherent sparsity of the imaged scene.

Microscopic scan-free surface profiling over extended axial ranges by point-spread-function engineering

The shape of a surface, i.e., its topography, influences many functional properties of a material; hence, characterization is critical in a wide variety of applications. Two notable challenges are profiling temporally changing structures, which requires high-speed acquisition, and capturing geometries with large axial steps. Here, we leverage point-spread-function engineering for scan-free, dynamic, microsurface profiling. The presented method is robust to axial steps and acquires full fields of view at camera-limited framerates. We present two approaches for implementation: fluorescence-based and label-free surface profiling, demonstrating the applicability to a variety of sample geometries and surface types.

Fast-tracking of single emitters in large volumes with nanometer precision

Multifocal plane microscopy allows for capturing images at different focal planes simultaneously. Using a proprietary prism which splits the emitted light into paths of different lengths, images at 8 different focal depths were obtained, covering a volume of 50x50x4 µm³. The position of single emitters was retrieved using a phasor-based approach across the different imaging planes, with better than 10 nm precision in the axial direction. We validated the accuracy of this approach by tracking fluorescent beads in 3D to calculate water viscosity. The fast acquisition rate (>100 fps) also enabled us to follow the capturing of 0.2 µm fluorescent beads into an optical trap.

Integrating engineered point spread functions into the phasor-based single-molecule localization microscopy framework

In single-molecule localization microscopy (SMLM), the use of engineered point spread functions (PSFs) provides access to three-dimensional localization information. The conventional approach of fitting PSFs with a single 2-dimensional Gaussian profile, however, often falls short in analyzing complex PSFs created by placing phase masks, deformable mirrors or spatial light modulators in the optical detection pathway. Here, we describe the integration of PSF modalities known as double-helix, saddle-point or tetra-pod into the phasor-based SMLM (pSMLM) framework enabling fast CPU based localization of single-molecule emitters with sub-pixel accuracy in three dimensions. For the double-helix PSF, pSMLM identifies the two individual lobes and uses their relative rotation for obtaining z-resolved localizations. For the analysis of saddle-point or tetra-pod PSFs, we present a novel phasor-based deconvolution approach entitled circular-tangent pSMLM. Saddle-point PSFs were experimentally realized by placing a deformable mirror in the Fourier plane and modulating the incoming wavefront with specific Zernike modes. Our pSMLM software package delivers similar precision and recall rates to the best-in-class software package (SMAP) at signal-to-noise ratios typical for organic fluorophores and achieves localization rates of up to 15 kHz (double-helix) and 250 kHz (saddle-point/tetra-pod) on a standard CPU. We further integrated pSMLM into an existing software package (SMALL-LABS) suitable for single-particle imaging and tracking in environments with obscuring backgrounds. Taken together, we provide a powerful hardware and software environment for advanced single-molecule studies.