Multicolor localization microscopy and point-spread-function engineering by deep learning

Machine Learning for Single-Molecule Localization Microscopy: From Data Analysis to Quantification

Single-molecule localization microscopy (SMLM) is a versatile tool for realizing nanoscale imaging with visible light and providing unprecedented opportunities to observe bioprocesses. The integration of machine learning with SMLM enhances data analysis by improving efficiency and accuracy. This tutorial aims to provide a comprehensive overview of the data analysis process and theoretical aspects of SMLM, while also highlighting the typical applications of machine learning in this field. By leveraging advanced analytical techniques, SMLM is becoming a powerful quantitative analysis tool for biological research.

Depth-enhanced high-throughput microscopy by compact PSF engineering

High-throughput microscopy is vital for screening applications, where three-dimensional (3D) cellular models play a key role. However, due to defocus susceptibility, current 3D high-throughput microscopes require axial scanning, which lowers throughput and increases photobleaching and photodamage. Point spread function (PSF) engineering is an optical method that enables various 3D imaging capabilities, yet it has not been implemented in high-throughput microscopy due to the cumbersome optical extension it typically requires. Here we demonstrate compact PSF engineering in the objective lens, which allows us to enhance the imaging depth of field and, combined with deep learning, recover 3D information using single snapshots. Beyond the applications shown here, this work showcases the usefulness of high-throughput microscopy in obtaining training data for deep learning-based algorithms, applicable to a variety of microscopy modalities.

Deep learning-based spectroscopic single-molecule localization microscopy

Significance

Spectroscopic single-molecule localization microscopy (sSMLM) takes advantage of nanoscopy and spectroscopy, enabling sub-10 nm resolution as well as simultaneous multicolor imaging of multi-labeled samples. Reconstruction of raw sSMLM data using deep learning is a promising approach for visualizing the subcellular structures at the nanoscale.

Aim

Develop a novel computational approach leveraging deep learning to reconstruct both label-free and fluorescence-labeled sSMLM imaging data.

Approach

We developed a two-network-model based deep learning algorithm, termed DsSMLM, to reconstruct sSMLM data. The effectiveness of DsSMLM was assessed by conducting imaging experiments on diverse samples, including label-free single-stranded DNA (ssDNA) fiber, fluorescence-labeled histone markers on COS-7 and U2OS cells, and simultaneous multicolor imaging of synthetic DNA origami nanoruler.

Results

For label-free imaging, a spatial resolution of 6.22 nm was achieved on ssDNA fiber; for fluorescence-labeled imaging, DsSMLM revealed the distribution of chromatin-rich and chromatin-poor regions defined by histone markers on the cell nucleus and also offered simultaneous multicolor imaging of nanoruler samples, distinguishing two dyes labeled in three emitting points with a separation distance of 40 nm. With DsSMLM, we observed enhanced spectral profiles with 8.8% higher localization detection for single-color imaging and up to 5.05% higher localization detection for simultaneous two-color imaging.

Conclusions

We demonstrate the feasibility of deep learning-based reconstruction for sSMLM imaging applicable to label-free and fluorescence-labeled sSMLM imaging data. We anticipate our technique will be a valuable tool for high-quality super-resolution imaging for a deeper understanding of DNA molecules' photophysics and will facilitate the investigation of multiple nanoscopic cellular structures and their interactions.

Advanced optical imaging for the rational design of nanomedicines

Despite the enormous potential of nanomedicines to shape the future of medicine, their clinical translation remains suboptimal. Translational challenges are present in every step of the development pipeline, from a lack of understanding of patient heterogeneity to insufficient insights on nanoparticle properties and their impact on material-cell interactions. Here, we discuss how the adoption of advanced optical microscopy techniques, such as super-resolution optical microscopies, correlative techniques, and high-content modalities, could aid the rational design of nanocarriers, by characterizing the cell, the nanomaterial, and their interaction with unprecedented spatial and/or temporal detail. In this nanomedicine arena, we will discuss how the implementation of these techniques, with their versatility and specificity, can yield high volumes of multi-parametric data; and how machine learning can aid the rapid advances in microscopy: from image acquisition to data interpretation.

Enhancing Light-Sheet Fluorescence Microscopy Illumination Beams through Deep Design Optimization

Light sheet fluorescence microscopy (LSFM) provides the benefit of optical sectioning coupled with rapid acquisition times for imaging of tissue-cleared specimen. This allows for high-resolution 3D imaging of large tissue volumes. Inherently to LSFM, the quality of the imaging heavily relies on the characteristics of the illumination beam, with the notion that the illumination beam only illuminates a thin section that is being imaged. Therefore, substantial efforts are dedicated to identifying slender, non-diffracting beam profiles that can yield uniform and high-contrast images. An ongoing debate concerns the employment of the most optimal illumination beam; Gaussian, Bessel, Airy patterns and/or others. Comparisons among different beam profiles is challenging as their optimization objective is often different. Given that our large imaging datasets (~0.5TB images per sample) is already analyzed using deep learning models, we envisioned a different approach to this problem by hypothesizing that we can tailor the illumination beam to boost the deep learning models performance. We achieve this by integrating the physical LSFM illumination model after passing through a variable phase mask into the training of a cell detection network. Here we report that the joint optimization continuously updates the phase mask, improving the image quality for better cell detection. Our method's efficacy is demonstrated through both simulations and experiments, revealing substantial enhancements in imaging quality compared to traditional Gaussian light sheet. We offer valuable insights for designing microscopy systems through a computational approach that exhibits significant potential for advancing optics design that relies on deep learning models for analysis of imaging datasets.

Evaluating the resolution of conventional optical microscopes through point spread function measurement

In the imaging process of conventional optical microscopy, the primary factor hindering microscope resolution is the energy diffusion of incident light, most directly described by the point spread function (PSF). Therefore, accurate calculation and measurement of PSF are essential for evaluating and enhancing imaging resolution. Currently, there are various methods to obtain PSFs, each with different advantages and disadvantages suitable for different scenarios. To provide a comprehensive analysis of PSF-obtaining methods, this study classifies them into four categories based on different acquisition principles and analyzes their advantages and disadvantages, starting from the propagation property of light in optical physics. Finally, two PSF-obtaining methods are proposed based on mathematical modeling and deep learning, demonstrating their effectiveness through experimental results. This study compares and analyzes these results, highlighting the practical applications of image deblurring.

Simultaneous spectral differentiation of multiple fluorophores in super-resolution imaging using a glass phase plate

By engineering the point-spread function (PSF) of single molecules, different fluorophore species can be imaged simultaneously and distinguished by their unique PSF patterns. Here, we insert a silicon-dioxide phase plate at the Fourier plane of the detection path of a wide-field fluorescence microscope to produce distinguishable PSFs (X-PSFs) at different wavelengths. We demonstrate that the resulting PSFs can be localized spatially and spectrally using a maximum-likelihood estimation algorithm and can be utilized for hyper-spectral super-resolution microscopy of biological samples. We produced superresolution images of fixed U2OS cells using X-PSFs for dSTORM imaging with simultaneous illumination of up to three fluorophore species. The species were distinguished only by the PSF pattern. We achieved ∼21-nm lateral localization precision (FWHM) and ∼17-nm axial precision (FWHM) with an average of 1,800 - 3,500 photons per PSF and a background as high as 130 - 400 photons per pixel. The modified PSF distinguished fluorescent probes with ∼80 nm separation between spectral peaks.

Advancing Spectrally-Resolved Single Molecule Localization Microscopy with Deep Learning

Spectrally-resolved single molecule localization microscopy (srSMLM) is a recent technique enriching single molecule localization microscopy with the simultaneous recording of spectra of the single emitters. srSMLM resolution is limited by the number of photons collected per emitters. Sharing a photon budget to record the localization and the spectroscopic information results in a loss of spatial and spectral resolution-or forces the sacrifice of one at the expense of the other. Here, srUnet-a deep-learning Unet-based image processing routine trained to increase the spectral and spatial signals to compensate for the resolution loss inherent in additionally recording the spectral component is reported. Both localization and spectral precision are improved by srUnet-particularly for the low-emitting species. srUnet increases the fraction of localization whose signal can be both spatially and spectrally characterized. It preserves spectral shifts and the linearity of the dispersion of light. It strongly facilitates wavelength assignment in multicolor experiments. srUnet is a simple post-processing add-on boosting srSMLM performance close to conventional SMLM with the potential to turn srSMLM into the new standard for multicolor single molecule imaging.

Quantitative Single-Molecule Localization Microscopy

Super-resolution fluorescence microscopy allows the investigation of cellular structures at nanoscale resolution using light. Current developments in super-resolution microscopy have focused on reliable quantification of the underlying biological data. In this review, we first describe the basic principles of super-resolution microscopy techniques such as stimulated emission depletion (STED) microscopy and single-molecule localization microscopy (SMLM), and then give a broad overview of methodological developments to quantify super-resolution data, particularly those geared toward SMLM data. We cover commonly used techniques such as spatial point pattern analysis, colocalization, and protein copy number quantification but also describe more advanced techniques such as structural modeling, single-particle tracking, and biosensing. Finally, we provide an outlook on exciting new research directions to which quantitative super-resolution microscopy might be applied.

Deep learning using a residual deconvolutional network enables real-time high-density single-molecule localization microscopy

High-density localization based on deep learning is a very effective method to accelerate single molecule localization microscopy (SMLM). Compared with traditional high-density localization methods, deep learning-based methods enable a faster data processing speed and a higher localization accuracy. However, the reported high-density localization methods based on deep learning are still not fast enough to enable real time data processing for large batches of raw images, which is probably due to the heavy computational burden and computation complexity in the U-shape architecture used in these models. Here we propose a high-density localization method called FID-STORM, which is based on an improved residual deconvolutional network for the real-time processing of raw images. In FID-STORM, we use a residual network to extract the features directly from low-resolution raw images rather than the U-shape network from interpolated images. We also use a model fusion from TensorRT to further accelerate the inference of the model. In addition, we process the sum of the localization images directly on GPU to obtain an additional speed gain. Using simulated and experimental data, we verified that the FID-STORM method achieves a processing speed of 7.31 ms/frame at 256 × 256 pixels @ Nvidia RTX 2080 Ti graphic card, which is shorter than the typical exposure time of 10∼30 ms, thus enabling real-time data processing in high-density SMLM. Moreover, compared with a popular interpolated image-based method called Deep-STORM, FID-STORM enables a speed gain of ∼26 times, without loss of reconstruction accuracy. We also provided an ImageJ plugin for our new method.

Machine-learning-powered extraction of molecular diffusivity from single-molecule images for super-resolution mapping

While critical to biological processes, molecular diffusion is difficult to quantify, and spatial mapping of local diffusivity is even more challenging. Here we report a machine-learning-enabled approach, pixels-to-diffusivity (Pix2D), to directly extract the diffusion coefficient D from single-molecule images, and consequently enable super-resolved D spatial mapping. Working with single-molecule images recorded at a fixed framerate under typical single-molecule localization microscopy (SMLM) conditions, Pix2D exploits the often undesired yet evident motion blur, i.e., the convolution of single-molecule motion trajectory during the frame recording time with the diffraction-limited point spread function (PSF) of the microscope. Whereas the stochastic nature of diffusion imprints diverse diffusion trajectories to different molecules diffusing at the same given D, we construct a convolutional neural network (CNN) model that takes a stack of single-molecule images as the input and evaluates a D-value as the output. We thus validate robust D evaluation and spatial mapping with simulated data, and with experimental data successfully characterize D differences for supported lipid bilayers of different compositions and resolve gel and fluidic phases at the nanoscale.

Single-molecule imaging in the primary cilium

The primary cilium is an important signaling organelle critical for normal development and tissue homeostasis. Its small dimensions and complexity necessitate advanced imaging approaches to uncover the molecular mechanisms behind its function. Here, we outline how single-molecule fluorescence microscopy can be used for tracking molecular dynamics and interactions and for super-resolution imaging of nanoscale structures in the primary cilium. Specifically, we describe in detail how to capture and quantify the 2D dynamics of individual transmembrane proteins PTCH1 and SMO and how to map the 3D nanoscale distributions of the inversin compartment proteins INVS, ANKS6, and NPHP3. This protocol can, with minor modifications, be adapted for studies of other proteins and cell lines to further elucidate the structure and function of the primary cilium at the molecular level.

Deep learning in single-molecule imaging and analysis: recent advances and prospects

Single-molecule microscopy is advantageous in characterizing heterogeneous dynamics at the molecular level. However, there are several challenges that currently hinder the wide application of single molecule imaging in bio-chemical studies, including how to perform single-molecule measurements efficiently with minimal run-to-run variations, how to analyze weak single-molecule signals efficiently and accurately without the influence of human bias, and how to extract complete information about dynamics of interest from single-molecule data. As a new class of computer algorithms that simulate the human brain to extract data features, deep learning networks excel in task parallelism and model generalization, and are well-suited for handling nonlinear functions and extracting weak features, which provide a promising approach for single-molecule experiment automation and data processing. In this perspective, we will highlight recent advances in the application of deep learning to single-molecule studies, discuss how deep learning has been used to address the challenges in the field as well as the pitfalls of existing applications, and outline the directions for future development.

Monocular kilometer-scale passive ranging by point-spread function engineering

Standard imaging systems are designed for 2D representation of objects, while information about the third dimension remains implicit, as imaging-based distance estimation is a difficult challenge. Existing long-range distance estimation technologies mostly rely on active emission of signal, which as a subsystem, constitutes a significant portion of the complexity, size and cost of the active-ranging apparatus. Despite the appeal of alleviating the requirement for signal-emission, passive distance estimation methods are essentially nonexistent for ranges greater than a few hundreds of meters. Here, we present monocular long-range, telescope-based passive ranging, realized by integration of point-spread-function engineering into a telescope, extending the scale of point-spread-function engineering-based ranging to distances where it has never been tested before. We provide experimental demonstrations of the optical system in a variety of challenging imaging scenarios, including adversarial weather conditions, dynamic targets and scenes of diversified textures, at distances extending beyond 1.7 km. We conclude with brief quantification of the effect of atmospheric turbulence on estimation precision, which becomes a significant error source in long-range optical imaging.

Development of Deep-Learning-Based Single-Molecule Localization Image Analysis

Recent developments in super-resolution fluorescence microscopic techniques (SRM) have allowed for nanoscale imaging that greatly facilitates our understanding of nanostructures. However, the performance of single-molecule localization microscopy (SMLM) is significantly restricted by the image analysis method, as the final super-resolution image is reconstructed from identified localizations through computational analysis. With recent advancements in deep learning, many researchers have employed deep learning-based algorithms to analyze SMLM image data. This review discusses recent developments in deep-learning-based SMLM image analysis, including the limitations of existing fitting algorithms and how the quality of SMLM images can be improved through deep learning. Finally, we address possible future applications of deep learning methods for SMLM imaging.

Optimal Control of Coherent Light Scattering for Binary Decision Problems

Because of quantum noise fluctuations, the rate of error achievable in decision problems involving several possible configurations of a scattering system is subject to a fundamental limit known as the Helstrom bound. Here, we present a general framework to calculate and minimize this bound using coherent probe fields with tailored spatial distributions. As an example, we experimentally study a target located in between two disordered scattering media. We first show that the optimal field distribution can be directly identified using a general approach based on scattering matrix measurements. We then demonstrate that this optimal light field successfully probes the presence of the target with a number of photons that is reduced by more than 2 orders of magnitude as compared to unoptimized fields.

High-axial-resolution single-molecule localization under dense excitation with a multi-channel deep U-Net

Single-molecule localization microscopy (SMLM) can bypass the diffraction limit of optical microscopes and greatly improve the resolution in fluorescence microscopy. By introducing the point spread function (PSF) engineering technique, we can customize depth varying PSF to achieve higher axial resolution. However, most existing 3D single-molecule localization algorithms require excited fluorescent molecules to be sparse and captured at high signal-to-noise ratios, which results in a long acquisition time and precludes SMLM's further applications in many potential fields. To address this problem, we propose a novel 3D single-molecular localization method based on a multi-channel neural network based on U-Net. By leveraging the deep network's great advantages in feature extraction, the proposed network can reliably discriminate dense fluorescent molecules with overlapped PSFs and corrupted by sensor noise. Both simulated and real experiments demonstrate its superior performance in PSF engineered microscopes with short exposure and dense excitations, which holds great potential in fast 3D super-resolution microscopy.

Two-color super-resolution localization microscopy via joint encoding of emitter location and color

Multi-color super-resolution localization microscopy (SRLM) provides great opportunities for studying the structural and functional details of biological samples. However, current multi-color SRLM methods either suffer from medium to high crosstalk, or require a dedicated optical system and a complicated image analysis procedure. To address these problems, here we propose a completely different method to realize multi-color SRLM. This method is built upon a customized RGBW camera with a repeated pattern of filtered (Red, Green, Blue and Near-infrared) and unfiltered (White) pixels. With a new insight that RGBW camera is advantageous for color recognition instead of color reproduction, we developed a joint encoding scheme of emitter location and color. By combing this RGBW camera with the joint encoding scheme and a simple optical set-up, we demonstrated two-color SRLM with ∼20 nm resolution and < 2% crosstalk (which is comparable to the best-reported values). This study significantly reduces the complexity of two-color SRLM (and potentially multi-color SRLM), and thus offers good opportunities for general biomedical research laboratories to use multi-color SRLM, which is currently mastered only by well-trained researchers.

Advancing biological super-resolution microscopy through deep learning: a brief review

Biological super-resolution microscopy is a new generation of imaging techniques that overcome the ~200 nm diffraction limit of conventional light microscopy in spatial resolution. By providing novel spatial or spatiotemporal information on biological processes at nanometer resolution with molecular specificity, it plays an increasingly important role in biomedical sciences. However, its technical constraints also require trade-offs to balance its spatial resolution, temporal resolution, and light exposure of samples. Recently, deep learning has achieved breakthrough performance in many image processing and computer vision tasks. It has also shown great promise in pushing the performance envelope of biological super-resolution microscopy. In this brief review, we survey recent advances in using deep learning to enhance the performance of biological super-resolution microscopy, focusing primarily on computational reconstruction of super-resolution images. Related key technical challenges are discussed. Despite the challenges, deep learning is expected to play an important role in the development of biological super-resolution microscopy. We conclude with an outlook into the future of this new research area.

Single Molecules Are Your Quanta: A Bottom-Up Approach toward Multidimensional Super-resolution Microscopy

The rise of single-molecule localization microscopy (SMLM) and related super-resolution methods over the past 15 years has revolutionized how we study biological and materials systems. In this Perspective, we reflect on the underlying philosophy of how diffraction-unlimited pictures containing rich spatial and functional information may gradually emerge through the local accumulation of single-molecule measurements. Starting with the basic concepts, we analyze the uniqueness of and opportunities in building up the final picture one molecule at a time. After brief introductions to the more established multicolor and three-dimensional measurements, we highlight emerging efforts to extend SMLM to new dimensions and functionalities as fluorescence polarization, emission spectra, and molecular motions, and discuss rising opportunities and future directions. With single molecules as our quanta, the bottom-up accumulation approach provides a powerful conduit for multidimensional microscopy at the nanoscale.

Towards Visual Proteomics at High Resolution

Traditionally, structural biologists approach the complexity of cellular proteomes in a reductionist manner. Proteomes are fractionated, their molecular components purified and studied one-by-one using the experimental methods for structure determination at their disposal. Visual proteomics aims at obtaining a holistic picture of cellular proteomes by studying them in situ, ideally in unperturbed cellular environments. The method that enables doing this at highest resolution is cryo-electron tomography. It allows to visualize cellular landscapes with molecular resolution generating maps or atlases revealing the interaction networks which underlie cellular functions in health and in disease states. Current implementations of cryo ET do not yet realize the full potential of the method in terms of resolution and interpretability. To this end, further improvements in technology and methodology are needed. This review describes the state of the art as well as measures which we expect will help overcoming current limitations.

Multimodal single-molecule microscopy with continuously controlled spectral resolution

Color is a fundamental contrast mechanism in fluorescence microscopy, providing the basis for numerous imaging and spectroscopy techniques. Building on spectral imaging schemes that encode color into a fixed spatial intensity distribution, here, we introduce continuously controlled spectral-resolution (CoCoS) microscopy, which allows the spectral resolution of the system to be adjusted in real-time. By optimizing the spectral resolution for each experiment, we achieve maximal sensitivity and throughput, allowing for single-frame acquisition of multiple color channels with single-molecule sensitivity and 140-fold larger fields of view compared with previous super-resolution spectral imaging techniques. Here, we demonstrate the utility of CoCoS in three experimental formats, single-molecule spectroscopy, single-molecule Förster resonance energy transfer, and multicolor single-particle tracking in live neurons, using a range of samples and 12 distinct fluorescent markers. A simple add-on allows CoCoS to be integrated into existing fluorescence microscopes, rendering spectral imaging accessible to the wider scientific community.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Neural network-based image reconstruction in swept-source optical coherence tomography using undersampled spectral data

Optical coherence tomography (OCT) is a widely used non-invasive biomedical imaging modality that can rapidly provide volumetric images of samples. Here, we present a deep learning-based image reconstruction framework that can generate swept-source OCT (SS-OCT) images using undersampled spectral data, without any spatial aliasing artifacts. This neural network-based image reconstruction does not require any hardware changes to the optical setup and can be easily integrated with existing swept-source or spectral-domain OCT systems to reduce the amount of raw spectral data to be acquired. To show the efficacy of this framework, we trained and blindly tested a deep neural network using mouse embryo samples imaged by an SS-OCT system. Using 2-fold undersampled spectral data (i.e., 640 spectral points per A-line), the trained neural network can blindly reconstruct 512 A-lines in 0.59 ms using multiple graphics-processing units (GPUs), removing spatial aliasing artifacts due to spectral undersampling, also presenting a very good match to the images of the same samples, reconstructed using the full spectral OCT data (i.e., 1280 spectral points per A-line). We also successfully demonstrate that this framework can be further extended to process 3× undersampled spectral data per A-line, with some performance degradation in the reconstructed image quality compared to 2× spectral undersampling. Furthermore, an A-line-optimized undersampling method is presented by jointly optimizing the spectral sampling locations and the corresponding image reconstruction network, which improved the overall imaging performance using less spectral data points per A-line compared to 2× or 3× spectral undersampling results. This deep learning-enabled image reconstruction approach can be broadly used in various forms of spectral-domain OCT systems, helping to increase their imaging speed without sacrificing image resolution and signal-to-noise ratio.

Deep-ROCS: from speckle patterns to superior-resolved images by deep learning in rotating coherent scattering microscopy

Rotating coherent scattering (ROCS) microscopy is a label-free imaging technique that overcomes the optical diffraction limit by adding up the scattered laser light from a sample obliquely illuminated from different angles. Although ROCS imaging achieves 150 nm spatial and 10 ms temporal resolution, simply summing different speckle patterns may cause loss of sample information. In this paper we present Deep-ROCS, a neural network-based technique that generates a superior-resolved image by efficient numerical combination of a set of differently illuminated images. We show that Deep-ROCS can reconstruct super-resolved images more accurately than conventional ROCS microscopy, retrieving high-frequency information from a small number (6) of speckle images. We demonstrate the performance of Deep-ROCS experimentally on 200 nm beads and by computer simulations, where we show its potential for even more complex structures such as a filament network.

Learning Optimal Wavefront Shaping for Multi-Channel Imaging

Fast acquisition of depth information is crucial for accurate 3D tracking of moving objects. Snapshot depth sensing can be achieved by wavefront coding, in which the point-spread function (PSF) is engineered to vary distinctively with scene depth by altering the detection optics. In low-light applications, such as 3D localization microscopy, the prevailing approach is to condense signal photons into a single imaging channel with phase-only wavefront modulation to achieve a high pixel-wise signal to noise ratio. Here we show that this paradigm is generally suboptimal and can be significantly improved upon by employing multi-channel wavefront coding, even in low-light applications. We demonstrate our multi-channel optimization scheme on 3D localization microscopy in densely labelled live cells where detectability is limited by overlap of modulated PSFs. At extreme densities, we show that a split-signal system, with end-to-end learned phase masks, doubles the detection rate and reaches improved precision compared to the current state-of-the-art, single-channel design. We implement our method using a bifurcated optical system, experimentally validating our approach by snapshot volumetric imaging and 3D tracking of fluorescently labelled subcellular elements in dense environments.

Power Phase Apodization Study on Compensation Defocusing and Chromatic Aberration in the Imaging System

We performed a detailed comparative study of the parametric high degree (cubic, fourth, and fifth) power phase apodization on compensation defocusing and chromatic aberration in the imaging system. The research results showed that increasing the power degree of the apodization function provided better independence (invariance) of the point spread function (PSF) from defocusing while reducing the depth of field (DOF). This reduction could be compensated by increasing the parameter α; however, this led to an increase in the size of the light spot. A nonlinear relationship between the increase in the DOF and spot size was shown (due to a small increase in the size of the light spot, the DOF can be significantly increased). Thus, the search for the best solution was based on a compromise of restrictions on the circle of confusion (CoC) and DOF. The modeling of color image formation under defocusing conditions for the considered apodization functions was performed. The subsequent deconvolution of the resulting color image was demonstrated.

Improving spatial precision and field-of-view in wavelength-tagged single-particle tracking using spectroscopic single-molecule localization microscopy

Spectroscopic single-molecule localization microscopy (sSMLM) generates super-resolution images of single molecules while simultaneously capturing the spectra of their fluorescence emissions. However, sSMLM splits photons from single-molecule emissions into a spatial channel and a spectral channel, reducing both channels' precisions. It is also challenging in transmission grating-based sSMLM to achieve a large field-of-view (FOV) and avoid overlap between the spatial and spectral channels. The challenge in FOV has further significance in single-molecule tracking applications. In this work, we analyzed the correlation between the spatial and spectral channels in sSMLM to improve its spatial precision, and we developed a split-mirror assembly to enlarge its FOV. We demonstrate the benefits of these improvements by tracking quantum dots. We also show that we can reduce particle-identification ambiguity by tagging each particle with its unique spectral characteristics.

Challenges facing quantitative large-scale optical super-resolution, and some simple solutions

Optical super-resolution microscopy (SRM) has enabled biologists to visualize cellular structures with near-molecular resolution, giving unprecedented access to details about the amounts, sizes, and spatial distributions of macromolecules in the cell. Precisely quantifying these molecular details requires large datasets of high-quality, reproducible SRM images. In this review, we discuss the unique set of challenges facing quantitative SRM, giving particular attention to the shortcomings of conventional specimen preparation techniques and the necessity for optimal labeling of molecular targets. We further discuss the obstacles to scaling SRM methods, such as lengthy image acquisition and complex SRM data analysis. For each of these challenges, we review the recent advances in the field that circumvent these pitfalls and provide practical advice to biologists for optimizing SRM experiments.

Single-molecule orientation localization microscopy II: a performance comparison

Various techniques have been developed to measure the 2D and 3D positions and 2D and 3D orientations of fluorescent molecules with improved precision over standard epifluorescence microscopes. Due to the challenging signal-to-background ratio in typical single-molecule experiments, it is essential to choose an imaging system optimized for the specific target sample. In this work, we compare the performance of multiple state-of-the-art and commonly used methods for orientation localization microscopy against the fundamental limits of measurement precision. Our analysis reveals optimal imaging methods for various experiment conditions and sample geometries. Interestingly, simple modifications to the standard fluorescence microscope exhibit superior performance in many imaging scenarios.

Accelerating 3D single-molecule localization microscopy using blind sparse inpainting

SIGNIFICANCE

Single-molecule localization-based super-resolution microscopy has enabled the imaging of microscopic objects beyond the diffraction limit. However, this technique is limited by the requirements of imaging an extremely large number of frames of biological samples to generate a super-resolution image, thus requiring a longer acquisition time. Additionally, the processing of such a large image sequence leads to longer data processing time. Therefore, accelerating image acquisition and processing in single-molecule localization microscopy (SMLM) has been of perennial interest.

AIM

To accelerate three-dimensional (3D) SMLM imaging by leveraging a computational approach without compromising the resolution.

APPROACH

We used blind sparse inpainting to reconstruct high-density 3D images from low-density ones. The low-density images are generated using much fewer frames than usually needed, thus requiring a shorter acquisition and processing time. Therefore, our technique will accelerate 3D SMLM without changing the existing standard SMLM hardware system and labeling protocol.

RESULTS

The performance of the blind sparse inpainting was evaluated on both simulation and experimental datasets. Superior reconstruction results of 3D SMLM images using up to 10-fold fewer frames in simulation and up to 50-fold fewer frames in experimental data were achieved.

CONCLUSIONS

We demonstrate the feasibility of fast 3D SMLM imaging leveraging a computational approach to reduce the number of acquired frames. We anticipate our technique will enable future real-time live-cell 3D imaging to investigate complex nanoscopic biological structures and their functions.

Quantifying accuracy and heterogeneity in single-molecule super-resolution microscopy

The resolution and accuracy of single-molecule localization microscopes (SMLMs) are routinely benchmarked using simulated data, calibration rulers, or comparisons to secondary imaging modalities. However, these methods cannot quantify the nanoscale accuracy of an arbitrary SMLM dataset. Here, we show that by computing localization stability under a well-chosen perturbation with accurate knowledge of the imaging system, we can robustly measure the confidence of individual localizations without ground-truth knowledge of the sample. We demonstrate that our method, termed Wasserstein-induced flux (WIF), measures the accuracy of various reconstruction algorithms directly on experimental 2D and 3D data of microtubules and amyloid fibrils. We further show that WIF confidences can be used to evaluate the mismatch between computational models and imaging data, enhance the accuracy and resolution of reconstructed structures, and discover hidden molecular heterogeneities. As a computational methodology, WIF is broadly applicable to any SMLM dataset, imaging system, and localization algorithm.

Standard benchmarking of single-molecule localization microscopy cannot quantify nanoscale accuracy of arbitrary datasets. Here, the authors present Wasserstein-induced flux, a method using a chosen perturbation and knowledge of the imaging system to measure confidence of individual localizations.

Recent advances in point spread function engineering and related computational microscopy approaches: from one viewpoint

This personal hybrid review piece, written in light of my recipience of the UIPAB 2020 young investigator award, contains a mixture of my scientific biography and work so far. This paper is not intended to be a comprehensive review, but only to highlight my contributions to computation-related aspects of super-resolution microscopy, as well as their origins and future directions.

Confocal Fluorescence-Lifetime Single-Molecule Localization Microscopy

Fluorescence lifetime imaging microscopy is an important technique that adds another dimension to intensity and color acquired by conventional microscopy. In particular, it allows for multiplexing fluorescent labels that have otherwise similar spectral properties. Currently, the only super-resolution technique that is capable of recording super-resolved images with lifetime information is stimulated emission depletion microscopy. In contrast, all single-molecule localization microscopy (SMLM) techniques that employ wide-field cameras completely lack the lifetime dimension. Here, we combine fluorescence-lifetime confocal laser-scanning microscopy with SMLM for realizing single-molecule localization-based fluorescence-lifetime super-resolution imaging. Besides yielding images with a spatial resolution much beyond the diffraction limit, it determines the fluorescence lifetime of all localized molecules. We validate our technique by applying it to direct stochastic optical reconstruction microscopy and points accumulation for imaging in nanoscale topography imaging of fixed cells, and we demonstrate its multiplexing capability on samples with two different labels that differ only by fluorescence lifetime but not by their spectral properties.

Super-resolution Microscopy with Single Molecules in Biology and Beyond-Essentials, Current Trends, and Future Challenges

Single-molecule super-resolution microscopy has developed from a specialized technique into one of the most versatile and powerful imaging methods of the nanoscale over the past two decades. In this perspective, we provide a brief overview of the historical development of the field, the fundamental concepts, the methodology required to obtain maximum quantitative information, and the current state of the art. Then, we will discuss emerging perspectives and areas where innovation and further improvement are needed. Despite the tremendous progress, the full potential of single-molecule super-resolution microscopy is yet to be realized, which will be enabled by the research ahead of us.

Multi-element microscope optimization by a learned sensing network with composite physical layers

Standard microscopes offer a variety of settings to help improve the visibility of different specimens to the end microscope user. Increasingly, however, digital microscopes are used to capture images for automated interpretation by computer algorithms (e.g., for feature classification, detection, or segmentation), often without any human involvement. In this work, we investigate an approach to jointly optimize multiple microscope settings, together with a classification network, for improved performance with such automated tasks. We explore the interplay between optimization of programmable illumination and pupil transmission, using experimentally imaged blood smears for automated malaria parasite detection, to show that multi-element "learned sensing" outperforms its single-element counterpart. While not necessarily ideal for human interpretation, the network's resulting low-resolution microscope images (20X-comparable) offer a machine learning network sufficient contrast to match the classification performance of corresponding high-resolution imagery (100X-comparable), pointing a path toward accurate automation over large fields-of-view.

Cryogenic single-molecule fluorescence annotations for electron tomography reveal in situ organization of key proteins in Caulobacter

Superresolution fluorescence microscopy and cryogenic electron tomography (CET) are powerful imaging methods for exploring the subcellular organization of biomolecules. Superresolution fluorescence microscopy based on covalent labeling highlights specific proteins and has sufficient sensitivity to observe single fluorescent molecules, but the reconstructions lack detailed cellular context. CET has molecular-scale resolution but lacks specific and nonperturbative intracellular labeling techniques. Here, we describe an imaging scheme that correlates cryogenic single-molecule fluorescence localizations with CET reconstructions. Our approach achieves single-molecule localizations with an average lateral precision of 9 nm, and a relative registration error between the set of localizations and CET reconstruction of ∼30 nm. We illustrate the workflow by annotating the positions of three proteins in the bacterium Caulobacter crescentus: McpA, PopZ, and SpmX. McpA, which forms a part of the chemoreceptor array, acts as a validation structure by being visible under both imaging modalities. In contrast, PopZ and SpmX cannot be directly identified in CET. While not directly discernable, PopZ fills a region at the cell poles that is devoid of electron-dense ribosomes. We annotate the position of PopZ with single-molecule localizations and confirm its position within the ribosome excluded region. We further use the locations of PopZ to provide context for localizations of SpmX, a low-copy integral membrane protein sequestered by PopZ as part of a signaling pathway that leads to an asymmetric cell division. Our correlative approach reveals that SpmX localizes along one side of the cell pole and its extent closely matches that of the PopZ region.

DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning

An outstanding challenge in single-molecule localization microscopy is the accurate and precise localization of individual point emitters in three dimensions in densely labeled samples. One established approach for three-dimensional single-molecule localization is point-spread-function (PSF) engineering, in which the PSF is engineered to vary distinctively with emitter depth using additional optical elements. However, images of dense emitters, which are desirable for improving temporal resolution, pose a challenge for algorithmic localization of engineered PSFs, due to lateral overlap of the emitter PSFs. Here we train a neural network to localize multiple emitters with densely overlapping Tetrapod PSFs over a large axial range. We then use the network to design the optimal PSF for the multi-emitter case. We demonstrate our approach experimentally with super-resolution reconstructions of mitochondria and volumetric imaging of fluorescently labeled telomeres in cells. Our approach, DeepSTORM3D, enables the study of biological processes in whole cells at timescales that are rarely explored in localization microscopy.

VIPR: vectorial implementation of phase retrieval for fast and accurate microscopic pixel-wise pupil estimation

In microscopy, proper modeling of the image formation has a substantial effect on the precision and accuracy in localization experiments and facilitates the correction of aberrations in adaptive optics experiments. The observed images are subject to polarization effects, refractive index variations, and system specific constraints. Previously reported techniques have addressed these challenges by using complicated calibration samples, computationally heavy numerical algorithms, and various mathematical simplifications. In this work, we present a phase retrieval approach based on an analytical derivation of the vectorial diffraction model. Our method produces an accurate estimate of the system's phase information, without any prior knowledge about the aberrations, in under a minute.

Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]

Deep learning-based data analysis methods have gained considerable attention in all fields of science over the last decade. In recent years, this trend has reached the single-molecule community. In this review, we will survey significant contributions of the application of deep learning in single-molecule imaging experiments. Additionally, we will describe the historical events that led to the development of modern deep learning methods, summarize the fundamental concepts of deep learning, and highlight the importance of proper data composition for accurate, unbiased results.

Machine-learning based spectral classification for spectroscopic single-molecule localization microscopy

Spectroscopic single-molecule localization microscopy (sSMLM) simultaneously captures the spatial locations and emission spectra of single molecular emissions and enables simultaneous multicolor super-resolution imaging. Existing sSMLM relies on extracting spectral signatures, such as weighted spectral centroids, to distinguish different molecular labels. However, the rich information carried by the complete spectral profiles is not fully utilized; thus, the misclassification rate between molecular labels can be high at low spectral analysis photon budget. We developed a machine learning (ML)-based method to analyze the full spectral profiles of each molecular emission and reduce the misclassification rate. We experimentally validated our method by imaging immunofluorescently labeled COS-7 cells using two far-red dyes typically used in sSMLM (AF647 and CF660) to resolve mitochondria and microtubules, respectively. We showed that the ML method achieved 10-fold reduction in misclassification and two-fold improvement in spectral data utilization comparing with the existing spectral centroid method.

Learned sensing: jointly optimized microscope hardware for accurate image classification

Since its invention, the microscope has been optimized for interpretation by a human observer. With the recent development of deep learning algorithms for automated image analysis, there is now a clear need to re-design the microscope's hardware for specific interpretation tasks. To increase the speed and accuracy of automated image classification, this work presents a method to co-optimize how a sample is illuminated in a microscope, along with a pipeline to automatically classify the resulting image, using a deep neural network. By adding a "physical layer" to a deep classification network, we are able to jointly optimize for specific illumination patterns that highlight the most important sample features for the particular learning task at hand, which may not be obvious under standard illumination. We demonstrate how our learned sensing approach for illumination design can automatically identify malaria-infected cells with up to 5-10% greater accuracy than standard and alternative microscope lighting designs. We show that this joint hardware-software design procedure generalizes to offer accurate diagnoses for two different blood smear types, and experimentally show how our new procedure can translate across different experimental setups while maintaining high accuracy.

Joint 3D localization and classification of space debris using a multispectral rotating point spread function

We consider the problem of joint three-dimensional (3D) localization and material classification of unresolved space debris using a multispectral rotating point spread function (RPSF). The use of RPSF allows one to estimate the 3D locations of point sources from their rotated images acquired by a single 2D sensor array, since the amount of rotation of each source image about its x, y location depends on its axial distance z. Using multispectral images, with one RPSF per spectral band, we are able not only to localize the 3D positions of the space debris but also classify their material composition. We propose a three-stage method for achieving joint localization and classification. In stage 1, we adopt an optimization scheme for localization in which the spectral signature of each material is assumed to be uniform, which significantly improves efficiency and yields better localization results than possible with a single spectral band. In stage 2, we estimate the spectral signature and refine the localization result via an alternating approach. We process classification in the final stage. Both Poisson noise and Gaussian noise models are considered, and the implementation of each is discussed. Numerical tests using multispectral data from NASA show the efficiency of our three-stage approach and illustrate the improvement of point source localization and spectral classification from using multiple bands over a single band.

Computational inverse design for cascaded systems of metasurface optics

Metasurfaces are an emerging technology that may supplant many of the conventional optics found in imaging devices, displays, and precision scientific instruments. Here, we develop a method for designing optical systems composed of multiple unique metasurfaces aligned in sequence and separated by distances much larger than the design wavelengths. Our approach is based on computational inverse design, also known as the adjoint-gradient method. This technique enables thousands or millions of independent design variables (e.g., the shapes of individual meta-atoms) to be optimized in parallel, with little or no intervention required by the user. The assumptions underlying our method are as follows: we use the local periodic approximation to determine the phase-response of a given meta-atom, we use the scalar wave approximation to propagate light fields between metasurface layers, and we do not consider multiple reflections between metasurface layers (analogous to a sequential-optics ray-tracer). To demonstrate the broad applicability of our method, we use it to design an achromatic doublet metasurface lens, a spectrally-multiplexed holographic element, and an ultra-compact optical neural network for classifying handwritten digits.

Single-Particle Diffusion Characterization by Deep Learning

Diffusion plays a crucial role in many biological processes including signaling, cellular organization, transport mechanisms, and more. Direct observation of molecular movement by single-particle-tracking experiments has contributed to a growing body of evidence that many cellular systems do not exhibit classical Brownian motion but rather anomalous diffusion. Despite this evidence, characterization of the physical process underlying anomalous diffusion remains a challenging problem for several reasons. First, different physical processes can exist simultaneously in a system. Second, commonly used tools for distinguishing between these processes are based on asymptotic behavior, which is experimentally inaccessible in most cases. Finally, an accurate analysis of the diffusion model requires the calculation of many observables because different transport modes can result in the same diffusion power-law α, which is typically obtained from the mean-square displacements (MSDs). The outstanding challenge in the field is to develop a method to extract an accurate assessment of the diffusion process using many short trajectories with a simple scheme that is applicable at the nonexpert level. Here, we use deep learning to infer the underlying process resulting in anomalous diffusion. We implement a neural network to classify single-particle trajectories by diffusion type: Brownian motion, fractional Brownian motion and continuous time random walk. Further, we demonstrate the applicability of our network architecture for estimating the Hurst exponent for fractional Brownian motion and the diffusion coefficient for Brownian motion on both simulated and experimental data. These networks achieve greater accuracy than time-averaged MSD analysis on simulated trajectories while only requiring as few as 25 steps. When tested on experimental data, both net and ensemble MSD analysis converge to similar values; however, the net needs only half the number of trajectories required for ensemble MSD to achieve the same confidence interval. Finally, we extract diffusion parameters from multiple extremely short trajectories (10 steps) using our approach.

Information-rich localization microscopy through machine learning

Recent years have witnessed the development of single-molecule localization microscopy as a generic tool for sampling diverse biologically relevant information at the super-resolution level. While current approaches often rely on the target-specific alteration of the point spread function to encode the multidimensional contents of single fluorophores, the details of the point spread function in an unmodified microscope already contain rich information. Here we introduce a data-driven approach in which artificial neural networks are trained to make a direct link between an experimental point spread function image and its underlying, multidimensional parameters, and compare results with alternative approaches based on maximum likelihood estimation. To demonstrate this concept in real systems, we decipher in fixed cells both the colors and the axial positions of single molecules in regular localization microscopy data.