Information-rich localization microscopy through machine learning

Machine Learning for Single-Molecule Localization Microscopy: From Data Analysis to Quantification

Single-molecule localization microscopy (SMLM) is a versatile tool for realizing nanoscale imaging with visible light and providing unprecedented opportunities to observe bioprocesses. The integration of machine learning with SMLM enhances data analysis by improving efficiency and accuracy. This tutorial aims to provide a comprehensive overview of the data analysis process and theoretical aspects of SMLM, while also highlighting the typical applications of machine learning in this field. By leveraging advanced analytical techniques, SMLM is becoming a powerful quantitative analysis tool for biological research.

Deep learning-based spectroscopic single-molecule localization microscopy

Significance

Spectroscopic single-molecule localization microscopy (sSMLM) takes advantage of nanoscopy and spectroscopy, enabling sub-10 nm resolution as well as simultaneous multicolor imaging of multi-labeled samples. Reconstruction of raw sSMLM data using deep learning is a promising approach for visualizing the subcellular structures at the nanoscale.

Aim

Develop a novel computational approach leveraging deep learning to reconstruct both label-free and fluorescence-labeled sSMLM imaging data.

Approach

We developed a two-network-model based deep learning algorithm, termed DsSMLM, to reconstruct sSMLM data. The effectiveness of DsSMLM was assessed by conducting imaging experiments on diverse samples, including label-free single-stranded DNA (ssDNA) fiber, fluorescence-labeled histone markers on COS-7 and U2OS cells, and simultaneous multicolor imaging of synthetic DNA origami nanoruler.

Results

For label-free imaging, a spatial resolution of 6.22 nm was achieved on ssDNA fiber; for fluorescence-labeled imaging, DsSMLM revealed the distribution of chromatin-rich and chromatin-poor regions defined by histone markers on the cell nucleus and also offered simultaneous multicolor imaging of nanoruler samples, distinguishing two dyes labeled in three emitting points with a separation distance of 40 nm. With DsSMLM, we observed enhanced spectral profiles with 8.8% higher localization detection for single-color imaging and up to 5.05% higher localization detection for simultaneous two-color imaging.

Conclusions

We demonstrate the feasibility of deep learning-based reconstruction for sSMLM imaging applicable to label-free and fluorescence-labeled sSMLM imaging data. We anticipate our technique will be a valuable tool for high-quality super-resolution imaging for a deeper understanding of DNA molecules' photophysics and will facilitate the investigation of multiple nanoscopic cellular structures and their interactions.

Large-FOV 3D localization microscopy by spatially variant point spread function generation

Accurate characterization of the microscopic point spread function (PSF) is crucial for achieving high-performance localization microscopy (LM). Traditionally, LM assumes a spatially invariant PSF to simplify the modeling of the imaging system. However, for large fields of view (FOV) imaging, it becomes important to account for the spatially variant nature of the PSF. Here, we propose an accurate and fast principal components analysis-based field-dependent 3D PSF generator (PPG3D) and localizer for LM. Through simulations and experimental three-dimensional (3D) single-molecule localization microscopy (SMLM), we demonstrate the effectiveness of PPG3D, enabling super-resolution imaging of mitochondria and microtubules with high fidelity over a large FOV. A comparison of PPG3D with a shift-variant PSF generator for 3D LM reveals a threefold improvement in accuracy. Moreover, PPG3D is approximately 100 times faster than existing PSF generators, when used in image plane-based interpolation mode. Given its user-friendliness, we believe that PPG3D holds great potential for widespread application in SMLM and other imaging modalities.

Computational drug development for membrane protein targets

The application of computational biology in drug development for membrane protein targets has experienced a boost from recent developments in deep learning-driven structure prediction, increased speed and resolution of structure elucidation, machine learning structure-based design and the evaluation of big data. Recent protein structure predictions based on machine learning tools have delivered surprisingly reliable results for water-soluble and membrane proteins but have limitations for development of drugs that target membrane proteins. Structural transitions of membrane proteins have a central role during transmembrane signaling and are often influenced by therapeutic compounds. Resolving the structural and functional basis of dynamic transmembrane signaling networks, especially within the native membrane or cellular environment, remains a central challenge for drug development. Tackling this challenge will require an interplay between experimental and computational tools, such as super-resolution optical microscopy for quantification of the molecular interactions of cellular signaling networks and their modulation by potential drugs, cryo-electron microscopy for determination of the structural transitions of proteins in native cell membranes and entire cells, and computational tools for data analysis and prediction of the structure and function of cellular signaling networks, as well as generation of promising drug candidates.

Switchable and Functional Fluorophores for Multidimensional Single-Molecule Localization Microscopy

Multidimensional single-molecule localization microscopy (mSMLM) represents a paradigm shift in the realm of super-resolution microscopy techniques. It affords the simultaneous detection of single-molecule spatial locations at the nanoscale and functional information by interrogating the emission properties of switchable fluorophores. The latter is finely tuned to report its local environment through carefully manipulated laser illumination and single-molecule detection strategies. This Perspective highlights recent strides in mSMLM with a focus on fluorophore designs and their integration into mSMLM imaging systems. Particular interests are the accomplishments in simultaneous multiplexed super-resolution imaging, nanoscale polarity and hydrophobicity mapping, and single-molecule orientational imaging. Challenges and prospects in mSMLM are also discussed, which include the development of more vibrant and functional fluorescent probes, the optimization of optical implementation to judiciously utilize the photon budget, and the advancement of imaging analysis and machine learning techniques.

Machine-learning-powered extraction of molecular diffusivity from single-molecule images for super-resolution mapping

While critical to biological processes, molecular diffusion is difficult to quantify, and spatial mapping of local diffusivity is even more challenging. Here we report a machine-learning-enabled approach, pixels-to-diffusivity (Pix2D), to directly extract the diffusion coefficient D from single-molecule images, and consequently enable super-resolved D spatial mapping. Working with single-molecule images recorded at a fixed framerate under typical single-molecule localization microscopy (SMLM) conditions, Pix2D exploits the often undesired yet evident motion blur, i.e., the convolution of single-molecule motion trajectory during the frame recording time with the diffraction-limited point spread function (PSF) of the microscope. Whereas the stochastic nature of diffusion imprints diverse diffusion trajectories to different molecules diffusing at the same given D, we construct a convolutional neural network (CNN) model that takes a stack of single-molecule images as the input and evaluates a D-value as the output. We thus validate robust D evaluation and spatial mapping with simulated data, and with experimental data successfully characterize D differences for supported lipid bilayers of different compositions and resolve gel and fluidic phases at the nanoscale.

Deep Learning Enhanced Electrochemiluminescence Microscopy

Limited by the efficiency of electrochemiluminescence, tens of seconds of exposure time are typically required to get a high-quality image. Image enhancement of short exposure time images to obtain a well-defined electrochemiluminescence image can meet the needs of high-throughput or dynamic imaging. Here, we propose deep enhanced ECL microscopy (DEECL), a general strategy that utilizes artificial neural networks to reconstruct electrochemiluminescence images with millisecond exposure times to have similar quality as high-quality electrochemiluminescence images with second-long exposure time. Electrochemiluminescence imaging of fixed cells demonstrates that DEECL allows improvement of the imaging efficiency by 1 to 2 orders than usual. This approach is further used for a data-intensive analysis application, cell classification, achieving an accuracy of 85% with ECL data at an exposure time of 50 ms. We anticipate that the computationally enhanced electrochemiluminescence microscopy will enable fast and information-rich imaging and prove useful for understanding dynamic chemical and biological processes.

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

Single-molecule localization microscopy in a typical wide-field setup has been widely used for investigating subcellular structures with super resolution; however, field-dependent aberrations restrict the field of view (FOV) to only tens of micrometers. Here, we present a deep-learning method for precise localization of spatially variant point emitters (FD-DeepLoc) over a large FOV covering the full chip of a modern sCMOS camera. Using a graphic processing unit-based vectorial point spread function (PSF) fitter, we can fast and accurately model the spatially variant PSF of a high numerical aperture objective in the entire FOV. Combined with deformable mirror-based optimal PSF engineering, we demonstrate high-accuracy three-dimensional single-molecule localization microscopy over a volume of ~180 × 180 × 5 μm³, allowing us to image mitochondria and nuclear pore complexes in entire cells in a single imaging cycle without hardware scanning; a 100-fold increase in throughput compared to the state of the art.

Similar color analysis based on deep learning (SCAD) for multiplex digital PCR via a single fluorescent channel

Digital PCR (dPCR) has recently attracted great interest due to its high sensitivity and accuracy. However, the existing dPCR depends on multicolor fluorescent dyes and multiple fluorescent channels to achieve multiplex detection, resulting in increased detection cost and limited detection throughput. Here, we developed a deep learning-based similar color analysis method, namely SCAD, to achieve multiplex dPCR in a single fluorescent channel. As a demonstration, we designed a microwell chip-based diplex dPCR system for detecting two genes (bla_NDM and bla_VIM) with two kinds of green fluorescent probes, whose emission colors are difficult to discriminate by traditional fluorescence intensity-based methods. To verify the possibility of deep learning algorithms to distinguish the similar colors, we first applied t-distributed stochastic neighbor embedding (tSNE) to make a clustering map for the microwells with similar fluorescence. Then, we trained a Vision Transformer (ViT) model on 10 000 microwells with two similar colors and tested it with 262 202 microwells. Lastly, the trained model was proven to have highly accurate classification ability (>98% for both the training set and the test set) and precise quantification ability on both bla_NDM and bla_VIM (ratio difference <0.10). We envision that the developed SCAD method would significantly expand the detection throughput of dPCR without the need for other auxiliary equipment.

Minimizing Molecular Misidentification in Imaging Low-Abundance Protein Interactions Using Spectroscopic Single-Molecule Localization Microscopy

Super-resolution microscopy can capture spatiotemporal organizations of protein interactions with resolution down to 10 nm; however, the analyses of more than two proteins involving low-abundance protein are challenging because spectral crosstalk and heterogeneities of individual fluorescent labels result in molecular misidentification. Here we developed a deep learning-based imaging analysis method for spectroscopic single-molecule localization microscopy to minimize molecular misidentification in three-color super-resolution imaging. We characterized the 3-fold reduction of molecular misidentification in the new imaging method using pure samples of different photoswitchable fluorophores and visualized three distinct subcellular proteins in U2-OS cell lines. We further validated the protein counts and interactions of TOMM20, DRP1, and SUMO1 in a well-studied biological process, Staurosporine-induced apoptosis, by comparing the imaging results with Western-blot analyses of different subcellular portions.

Development of Deep-Learning-Based Single-Molecule Localization Image Analysis

Recent developments in super-resolution fluorescence microscopic techniques (SRM) have allowed for nanoscale imaging that greatly facilitates our understanding of nanostructures. However, the performance of single-molecule localization microscopy (SMLM) is significantly restricted by the image analysis method, as the final super-resolution image is reconstructed from identified localizations through computational analysis. With recent advancements in deep learning, many researchers have employed deep learning-based algorithms to analyze SMLM image data. This review discusses recent developments in deep-learning-based SMLM image analysis, including the limitations of existing fitting algorithms and how the quality of SMLM images can be improved through deep learning. Finally, we address possible future applications of deep learning methods for SMLM imaging.

Robust emitter localization with enhanced harmonic analysis

We present a non-iterative and model-free algorithm for three-dimensional (3D) single emitter localization. Our algorithm decodes the axial position and the emitter width via the ratio of the first and second Fourier harmonic. The retrieved width information is further used for dynamic extraction of the proper region of interest to robustly eliminate the outer noisy background, thus improving the localization precision over existing non-iterative algorithms. Using simulated and experimental datasets, we demonstrate that our algorithm achieves localization precision approaching the state-of-the-art iterative fitting-based methods in all three dimensions at two orders of magnitude faster speed, applicable in various 3D single-molecule localization techniques.

Tutorial: methods for three-dimensional visualization of archival tissue material

Analysis of three-dimensional patient specimens is gaining increasing relevance for understanding the principles of tissue structure as well as the biology and mechanisms underlying disease. New technologies are improving our ability to visualize large volume of tissues with subcellular resolution. One resource often overlooked is archival tissue maintained for decades in hospitals and research archives around the world. Accessing the wealth of information stored within these samples requires the use of appropriate methods. This tutorial introduces the range of sample preparation and microscopy approaches available for three-dimensional visualization of archival tissue. We summarize key aspects of the relevant techniques and common issues encountered when using archival tissue, including registration and antibody penetration. We also discuss analysis pipelines required to process, visualize and analyze the data and criteria to guide decision-making. The methods outlined in this tutorial provide an important and sustainable avenue for validating three-dimensional tissue organization and mechanisms of disease.

Single Molecules Are Your Quanta: A Bottom-Up Approach toward Multidimensional Super-resolution Microscopy

The rise of single-molecule localization microscopy (SMLM) and related super-resolution methods over the past 15 years has revolutionized how we study biological and materials systems. In this Perspective, we reflect on the underlying philosophy of how diffraction-unlimited pictures containing rich spatial and functional information may gradually emerge through the local accumulation of single-molecule measurements. Starting with the basic concepts, we analyze the uniqueness of and opportunities in building up the final picture one molecule at a time. After brief introductions to the more established multicolor and three-dimensional measurements, we highlight emerging efforts to extend SMLM to new dimensions and functionalities as fluorescence polarization, emission spectra, and molecular motions, and discuss rising opportunities and future directions. With single molecules as our quanta, the bottom-up accumulation approach provides a powerful conduit for multidimensional microscopy at the nanoscale.

Accelerating 3D single-molecule localization microscopy using blind sparse inpainting

SIGNIFICANCE

Single-molecule localization-based super-resolution microscopy has enabled the imaging of microscopic objects beyond the diffraction limit. However, this technique is limited by the requirements of imaging an extremely large number of frames of biological samples to generate a super-resolution image, thus requiring a longer acquisition time. Additionally, the processing of such a large image sequence leads to longer data processing time. Therefore, accelerating image acquisition and processing in single-molecule localization microscopy (SMLM) has been of perennial interest.

AIM

To accelerate three-dimensional (3D) SMLM imaging by leveraging a computational approach without compromising the resolution.

APPROACH

We used blind sparse inpainting to reconstruct high-density 3D images from low-density ones. The low-density images are generated using much fewer frames than usually needed, thus requiring a shorter acquisition and processing time. Therefore, our technique will accelerate 3D SMLM without changing the existing standard SMLM hardware system and labeling protocol.

RESULTS

The performance of the blind sparse inpainting was evaluated on both simulation and experimental datasets. Superior reconstruction results of 3D SMLM images using up to 10-fold fewer frames in simulation and up to 50-fold fewer frames in experimental data were achieved.

CONCLUSIONS

We demonstrate the feasibility of fast 3D SMLM imaging leveraging a computational approach to reduce the number of acquired frames. We anticipate our technique will enable future real-time live-cell 3D imaging to investigate complex nanoscopic biological structures and their functions.

Quantifying accuracy and heterogeneity in single-molecule super-resolution microscopy

The resolution and accuracy of single-molecule localization microscopes (SMLMs) are routinely benchmarked using simulated data, calibration rulers, or comparisons to secondary imaging modalities. However, these methods cannot quantify the nanoscale accuracy of an arbitrary SMLM dataset. Here, we show that by computing localization stability under a well-chosen perturbation with accurate knowledge of the imaging system, we can robustly measure the confidence of individual localizations without ground-truth knowledge of the sample. We demonstrate that our method, termed Wasserstein-induced flux (WIF), measures the accuracy of various reconstruction algorithms directly on experimental 2D and 3D data of microtubules and amyloid fibrils. We further show that WIF confidences can be used to evaluate the mismatch between computational models and imaging data, enhance the accuracy and resolution of reconstructed structures, and discover hidden molecular heterogeneities. As a computational methodology, WIF is broadly applicable to any SMLM dataset, imaging system, and localization algorithm.

Standard benchmarking of single-molecule localization microscopy cannot quantify nanoscale accuracy of arbitrary datasets. Here, the authors present Wasserstein-induced flux, a method using a chosen perturbation and knowledge of the imaging system to measure confidence of individual localizations.

Super-resolution Microscopy with Single Molecules in Biology and Beyond-Essentials, Current Trends, and Future Challenges

Single-molecule super-resolution microscopy has developed from a specialized technique into one of the most versatile and powerful imaging methods of the nanoscale over the past two decades. In this perspective, we provide a brief overview of the historical development of the field, the fundamental concepts, the methodology required to obtain maximum quantitative information, and the current state of the art. Then, we will discuss emerging perspectives and areas where innovation and further improvement are needed. Despite the tremendous progress, the full potential of single-molecule super-resolution microscopy is yet to be realized, which will be enabled by the research ahead of us.

Accelerating multicolor spectroscopic single-molecule localization microscopy using deep learning

Spectroscopic single-molecule localization microscopy (sSMLM) simultaneously provides spatial localization and spectral information of individual single-molecules emission, offering multicolor super-resolution imaging of multiple molecules in a single sample with the nanoscopic resolution. However, this technique is limited by the requirements of acquiring a large number of frames to reconstruct a super-resolution image. In addition, multicolor sSMLM imaging suffers from spectral cross-talk while using multiple dyes with relatively broad spectral bands that produce cross-color contamination. Here, we present a computational strategy to accelerate multicolor sSMLM imaging. Our method uses deep convolution neural networks to reconstruct high-density multicolor super-resolution images from low-density, contaminated multicolor images rendered using sSMLM datasets with much fewer frames, without compromising spatial resolution. High-quality, super-resolution images are reconstructed using up to 8-fold fewer frames than usually needed. Thus, our technique generates multicolor super-resolution images within a much shorter time, without any changes in the existing sSMLM hardware system. Two-color and three-color sSMLM experimental results demonstrate superior reconstructions of tubulin/mitochondria, peroxisome/mitochondria, and tubulin/mitochondria/peroxisome in fixed COS-7 and U2-OS cells with a significant reduction in acquisition time.

Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]

Deep learning-based data analysis methods have gained considerable attention in all fields of science over the last decade. In recent years, this trend has reached the single-molecule community. In this review, we will survey significant contributions of the application of deep learning in single-molecule imaging experiments. Additionally, we will describe the historical events that led to the development of modern deep learning methods, summarize the fundamental concepts of deep learning, and highlight the importance of proper data composition for accurate, unbiased results.

Design of task-specific optical systems using broadband diffractive neural networks

Deep learning has been transformative in many fields, motivating the emergence of various optical computing architectures. Diffractive optical network is a recently introduced optical computing framework that merges wave optics with deep-learning methods to design optical neural networks. Diffraction-based all-optical object recognition systems, designed through this framework and fabricated by 3D printing, have been reported to recognize hand-written digits and fashion products, demonstrating all-optical inference and generalization to sub-classes of data. These previous diffractive approaches employed monochromatic coherent light as the illumination source. Here, we report a broadband diffractive optical neural network design that simultaneously processes a continuum of wavelengths generated by a temporally incoherent broadband source to all-optically perform a specific task learned using deep learning. We experimentally validated the success of this broadband diffractive neural network architecture by designing, fabricating and testing seven different multi-layer, diffractive optical systems that transform the optical wavefront generated by a broadband THz pulse to realize (1) a series of tuneable, single-passband and dual-passband spectral filters and (2) spatially controlled wavelength de-multiplexing. Merging the native or engineered dispersion of various material systems with a deep-learning-based design strategy, broadband diffractive neural networks help us engineer the light-matter interaction in 3D, diverging from intuitive and analytical design methods to create task-specific optical components that can all-optically perform deterministic tasks or statistical inference for optical machine learning.

Machine-learning based spectral classification for spectroscopic single-molecule localization microscopy

Spectroscopic single-molecule localization microscopy (sSMLM) simultaneously captures the spatial locations and emission spectra of single molecular emissions and enables simultaneous multicolor super-resolution imaging. Existing sSMLM relies on extracting spectral signatures, such as weighted spectral centroids, to distinguish different molecular labels. However, the rich information carried by the complete spectral profiles is not fully utilized; thus, the misclassification rate between molecular labels can be high at low spectral analysis photon budget. We developed a machine learning (ML)-based method to analyze the full spectral profiles of each molecular emission and reduce the misclassification rate. We experimentally validated our method by imaging immunofluorescently labeled COS-7 cells using two far-red dyes typically used in sSMLM (AF647 and CF660) to resolve mitochondria and microtubules, respectively. We showed that the ML method achieved 10-fold reduction in misclassification and two-fold improvement in spectral data utilization comparing with the existing spectral centroid method.