Divide and conquer: real-time maximum likelihood fitting of multiple emitters for super-resolution localization microscopy

Multiplexed and Millimeter-Scale Fluorescence Nanoscopy of Cells and Tissue Sections via Prism-Illumination and Microfluidics-Enhanced DNA-PAINT

Fluorescence nanoscopy has become increasingly powerful for biomedical research, but it has historically afforded a small field-of-view (FOV) of around 50 μm × 50 μm at once and more recently up to ∼200 μm × 200 μm. Efforts to further increase the FOV in fluorescence nanoscopy have thus far relied on the use of fabricated waveguide substrates, adding cost and sample constraints to the applications. Here we report PRism-Illumination and Microfluidics-Enhanced DNA-PAINT (PRIME-PAINT) for multiplexed fluorescence nanoscopy across millimeter-scale FOVs. Built upon the well-established prism-type total internal reflection microscopy, PRIME-PAINT achieves robust single-molecule localization with up to ∼520 μm × 520 μm single FOVs and 25-40 nm lateral resolutions. Through stitching, nanoscopic imaging over mm2 sample areas can be completed in as little as 40 min per target. An on-stage microfluidics chamber facilitates probe exchange for multiplexing and enhances image quality, particularly for formalin-fixed paraffin-embedded (FFPE) tissue sections. We demonstrate the utility of PRIME-PAINT by analyzing ∼106 caveolae structures in ∼1,000 cells and imaging entire pancreatic cancer lesions from patient tissue biopsies. By imaging from nanometers to millimeters with multiplexity and broad sample compatibility, PRIME-PAINT will be useful for building multiscale, Google-Earth-like views of biological systems.

Quantitatively Lighting up the Spatial Organization of CD47/SIRPα Immune Checkpoints on the Cellular Membrane with Single-Molecule Localization Microscopy

Immunotherapy including immune checkpoint inhibition has reinvigorated the current cancer treatment field. The development of efficient cancer immunotherapies depends on a thorough understanding of the status of immune checkpoints and how they interact. However, the distribution and spatial organization changes of immune checkpoints during their interactions at the single-molecule level remain difficult to directly visualize due to the lack of in situ imaging techniques with appropriate spatial and stoichiometric resolution. Herein, we report the direct visualization and quantification of the spatial distribution and organization of CD47 on the bladder tumor cell membrane and SIRPα on the macrophage membrane by using a single-molecule localization imaging technique called quantitative direct stochastic optical reconstruction microscopy (QdSTORM). Results showed that a portion of CD47 and SIRPα was present on cell membranes as heterogeneous clusters of varying sizes and densities prior to activation. Quantitative analyses of the reconstructed super-resolution images and theoretical simulation revealed that CD47 and SIRPα were reorganized into larger clusters upon binding to each other. Furthermore, we found that blocking the immune checkpoint interaction with small-molecule inhibitors or antibodies significantly impacted the spatial clustering behavior of CD47 on bladder tumor cells, demonstrating the promise of our QdSTORM strategy in elucidating the molecular mechanisms underlying immunotherapy. This work offers a promising strategy to advance our understanding of immune checkpoint state and interactions while also contributing to the fields including signal regulation and cancer therapy.

Deep learning using a residual deconvolutional network enables real-time high-density single-molecule localization microscopy

High-density localization based on deep learning is a very effective method to accelerate single molecule localization microscopy (SMLM). Compared with traditional high-density localization methods, deep learning-based methods enable a faster data processing speed and a higher localization accuracy. However, the reported high-density localization methods based on deep learning are still not fast enough to enable real time data processing for large batches of raw images, which is probably due to the heavy computational burden and computation complexity in the U-shape architecture used in these models. Here we propose a high-density localization method called FID-STORM, which is based on an improved residual deconvolutional network for the real-time processing of raw images. In FID-STORM, we use a residual network to extract the features directly from low-resolution raw images rather than the U-shape network from interpolated images. We also use a model fusion from TensorRT to further accelerate the inference of the model. In addition, we process the sum of the localization images directly on GPU to obtain an additional speed gain. Using simulated and experimental data, we verified that the FID-STORM method achieves a processing speed of 7.31 ms/frame at 256 × 256 pixels @ Nvidia RTX 2080 Ti graphic card, which is shorter than the typical exposure time of 10∼30 ms, thus enabling real-time data processing in high-density SMLM. Moreover, compared with a popular interpolated image-based method called Deep-STORM, FID-STORM enables a speed gain of ∼26 times, without loss of reconstruction accuracy. We also provided an ImageJ plugin for our new method.

LuckyProfiler: an ImageJ plug-in capable of quantifying FWHM resolution easily and effectively for super-resolution images

Quantifying the resolution of a super-resolution image is vital for biologists trying to apply super-resolution microscopy in various research fields. Among the reported image resolution estimation methods, the one that calculates the full width at half maximum (FWHM) of line profile, called FWHM resolution, continues the traditional resolution criteria and has been popularly used by many researchers. However, quantifying the FWHM resolution of a super-resolution image is a time-consuming, labor-intensive, and error-prone process because this method typically involves a manual and careful selection of one or several of the smallest structures. In this paper, we investigate the influencing factors in FWHM resolution quantification systematically and present an ImageJ plug-in called LuckyProfiler for biologists so that they can have an easy and effective way of quantifying the FWHM resolution of super-resolution images.

Real-time image resolution measurement for single molecule localization microscopy

Recent advancements in single molecule localization microscopy (SMLM) have demonstrated outstanding potential applications in high-throughput and high-content screening imaging. One major limitation to such applications is to find a way to optimize imaging throughput without scarifying image quality, especially the homogeneity in image resolution, during the imaging of hundreds of field-of-views (FOVs) in heterogeneous samples. Here we introduce a real-time image resolution measurement method for SMLM to solve this problem. This method is under the heuristic framework of overall image resolution that counts on localization precision and localization density. Rather than estimating the mean localization density after completing the entire SMLM process, this method uses the spatial Poisson process to model the random activation of molecules and thus determines the localization density in real-time. We demonstrate that the method is valid in real-time resolution measurement and is effective in guaranteeing homogeneous image resolution across multiple representative FOVs with optimized imaging throughput.

PCIe-based FPGA-GPU heterogeneous computation for real-time multi-emitter fitting in super-resolution localization microscopy

Real-time multi-emitter fitting is a key technology for advancing super-resolution localization microscopy (SRLM), especially when it is necessary to achieve dynamic imaging quality control and/or optimization of experimental conditions. However, with the increase of activation densities, the requirements in the computing resources would increase rapidly due to the complexity of the fitting algorithms, making it difficult to realize real-time multi-emitter fitting for emitter density more than 0.6 mol/µm² in large field of view (FOV), even after acceleration with the popular Graphics Processing Unit (GPU) computation. Here we adopt the task parallelism strategy in computer science to construct a Peripheral Component Interconnect Express (PCIe) based all-in-one heterogeneous computing platform (AIO-HCP), where the data between two major parallel computing hardware, Field Programmable Gate Array (FPGA) and GPU, are interacted directly and executed simultaneously. Using simulated and experimental data, we verify that AIO-HCP could achieve a data throughput of up to ∼ 1.561 GB/s between FPGA and GPU. With this new platform, we develop a multi-emitter fitting method, called AIO-STORM, under big data stream parallel scheduling. We show that AIO-STORM is capable of providing real-time image processing on raw images with 100 µm × 100 µm FOV, 10 ms exposure time and 5.5 mol/µm² structure density, without scarifying image quality. This study overcomes the data throughput limitation of heterogeneous devices, demonstrates the power of the PCIe-based heterogeneous computation platform, and offers opportunities for multi-scale stitching of super-resolution images.

Influence of drift correction precision on super-resolution localization microscopy

Super-resolution localization microscopy (SRLM) breaks the diffraction limit successfully and improves the resolution of optical imaging systems by nearly an order of magnitude. However, SRLM typically takes several minutes or longer to collect a sufficient number of image frames that are required for reconstructing a final super-resolution image. During this long image acquisition period, system drift should be tightly controlled to ensure the imaging quality; thus, several drift correction methods have been developed. However, it is still unclear whether the performance of these methods is able to ensure sufficient image quality in SRLM. Without a clear answer to this question, it is hard to choose a suitable drift correction method for a specific SRLM experiment. In this paper, we use both theoretical analysis and simulation to investigate the relationship among drift correction precision, localization precision, and position estimation precision. We propose a concept of relative localization precision for evaluating the effect of drift correction on imaging resolution, which would help to select an appropriate drift correction method for a specific experiment.

Blood Cell Analysis: From Traditional Methods to Super-Resolution Microscopy

Blood cell analysis is essential for the diagnosis and identification of hematological malignancies. The use of digital microscopy systems has been extended in clinical laboratories. Super-resolution microscopy (SRM) has attracted wide attention in the medical field due to its nanoscale spatial resolution and high sensitivity. It is considered to be a potential method of blood cell analysis that may have more advantages than traditional approaches such as conventional optical microscopy and hematology analyzers in certain examination projects. In this review, we firstly summarize several common blood cell analysis technologies in the clinic, and analyze the advantages and disadvantages of these technologies. Then, we focus on the basic principles and characteristics of three representative SRM techniques, as well as the latest advances in these techniques for blood cell analysis. Finally, we discuss the developmental trend and possible research directions of SRM, and provide some discussions on further development of technologies for blood cell analysis.

Can super-resolution microscopy become a standard characterization technique for materials chemistry?

The characterization of newly synthesized materials is a cornerstone of all chemistry and nanotechnology laboratories. For this purpose, a wide array of analytical techniques have been standardized and are used routinely by laboratories across the globe. With these methods we can understand the structure, dynamics and function of novel molecular architectures and their relations with the desired performance, guiding the development of the next generation of materials. Moreover, one of the challenges in materials chemistry is the lack of reproducibility due to improper publishing of the sample preparation protocol. In this context, the recent adoption of the reporting standard MIRIBEL (Minimum Information Reporting in Bio-Nano Experimental Literature) for material characterization and details of experimental protocols aims to provide complete, reproducible and reliable sample preparation for the scientific community. Thus, MIRIBEL should be immediately adopted in publications by scientific journals to overcome this challenge. Besides current standard spectroscopy and microscopy techniques, there is a constant development of novel technologies that aim to help chemists unveil the structure of complex materials. Among them super-resolution microscopy (SRM), an optical technique that bypasses the diffraction limit of light, has facilitated the study of synthetic materials with multicolor ability and minimal invasiveness at nanometric resolution. Although still in its infancy, the potential of SRM to unveil the structure, dynamics and function of complex synthetic architectures has been highlighted in pioneering reports during the last few years. Currently, SRM is a sophisticated technique with many challenges in sample preparation, data analysis, environmental control and automation, and moreover the instrumentation is still expensive. Therefore, SRM is currently limited to expert users and is not implemented in characterization routines. This perspective discusses the potential of SRM to transition from a niche technique to a standard routine method for material characterization. We propose a roadmap for the necessary developments required for this purpose based on a collaborative effort from scientists and engineers across disciplines.

Raw Data to Results: A Hands-On Introduction and Overview of Computational Analysis for Single-Molecule Localization Microscopy

Single-molecule localization microscopy (SMLM) is an advanced microscopy method that uses the blinking of fluorescent molecules to determine the position of these molecules with a resolution below the diffraction limit (∼5-40 nm). While SMLM imaging itself is becoming more popular, the computational analysis surrounding the technique is still a specialized area and often remains a "black box" for experimental researchers. Here, we provide an introduction to the required computational analysis of SMLM imaging, post-processing and typical data analysis. Importantly, user-friendly, ready-to-use and well-documented code in Python and MATLAB with exemplary data is provided as an interactive experience for the reader, as well as a starting point for further analysis. Our code is supplemented by descriptions of the computational problems and their implementation. We discuss the state of the art in computational methods and software suites used in SMLM imaging and data analysis. Finally, we give an outlook into further computational challenges in the field.

A labeling strategy with effective preservation of fluorophores for expansion single-molecule localization microscopy (Ex-SMLM)

Expansion microscopy (ExM) significantly improves the resolution of conventional diffraction-limited optical microscopy by using physically expanding biological samples. Combining ExM with single-molecule localization microscopy (SMLM) could further enhance the resolving power of SMLM, which is typically in the order of 20-30 nm. However, to make this combination successful, we need to solve three key issues related to sample preparation, including mainly hydrogel shrinking in an ionic photoswitching buffer, fluorescence photobleaching due to a free-radical reaction and reduced labelling efficiency from protease digestion. Re-embedding polyacrylamide gel or using an improved photoswitching buffer with a low ionic strength is able to minimize or even solve the hydrogel shrinking problem, while the development of post-expansion labelling approaches avoids fluorescence bleaching. However, the preservation of protein epitopes (which determines the labelling efficiency) remains to be challenging. In this paper, we propose to tackle this challenge by introducing the highly selective and stable biotin-streptavidin interaction into the post-expansion labelling strategy. After upgrading the popular immunolabelling linkage scheme from Epitope-Primary antibody-Secondary antibody-Fluorophores to Epitope-Primary antibody-Secondary antibody-Biotin-Streptavidin-Fluorophores, we were able to label protein epitopes with biotin, which was stable during the expansion process, and thus avoid the troublesome problem in preserving protein epitopes or antibodies. We demonstrate that combining Ex-SMLM with the new post-expansion linkage scheme enables new possibilities in resolving the detailed arrangement of Nup133 proteins in the nuclear pore complex, which helps researchers to observe a clearer structure. This study provides new opportunities for studying the ultrastructural details of subcellular organelles or even biomacromolecules, using the conventional SMLM system.

State-of-the-Art Approaches for Image Deconvolution Problems, including Modern Deep Learning Architectures

In modern digital microscopy, deconvolution methods are widely used to eliminate a number of image defects and increase resolution. In this review, we have divided these methods into classical, deep learning-based, and optimization-based methods. The review describes the major architectures of neural networks, such as convolutional and generative adversarial networks, autoencoders, various forms of recurrent networks, and the attention mechanism used for the deconvolution problem. Special attention is paid to deep learning as the most powerful and flexible modern approach. The review describes the major architectures of neural networks used for the deconvolution problem. We describe the difficulties in their application, such as the discrepancy between the standard loss functions and the visual content and the heterogeneity of the images. Next, we examine how to deal with this by introducing new loss functions, multiscale learning, and prior knowledge of visual content. In conclusion, a review of promising directions and further development of deconvolution methods in microscopy is given.

Accelerating multi-emitter localization in super-resolution localization microscopy with FPGA-GPU cooperative computation

The real-time multi-emitter localization method is essential for advancing high-throughput super-resolution localization microscopy (HT-SRLM). In the past decade, the graphics processing unit (GPU) computation has been dominantly used to accelerate the execution speed of the multi-emitter localization method. However, if HT-SRLM is combined with a scientific complementary metal-oxide-semiconductor (sCMOS) camera working at full frame rate, real-time image processing is still difficult to achieve using this acceleration approach, thus resulting in a massive data storage challenge and even system crash. Here we take advantage of the cooperative acceleration power of field programming gate array (FPGA) computation and GPU computation, and propose a method called HCP-STORM to enable real-time multi-emitter localization. Using simulated images, we verified that HCP-STORM is capable of providing real-time image processing for raw images from a representative Hamamatsu Flash 4 V3 sCMOS camera working at full frame rate (that is, 2048×2048 pixels @ 10 ms exposure time). Using experimental images, we prove that HCP-STORM is 25 times faster than QC-STORM and 295 times faster than ThunderSTORM, with a small but acceptable degradation in image quality. This study shows the potential of FPGA-GPU cooperative computation in accelerating multi-emitter localization, and pushes a significant step toward the maturity of HT-SRLM technology.

High-precision 3D drift correction with differential phase contrast images

Single molecule localization microscopy (SMLM) usually requires long image acquisition time at the order of minutes and thus suffers from sample drift, which deteriorates image quality. A drift estimation method with high precision is typically used in SMLM, which can be further combined with a drift compensation device to enable active microscope stabilization. Among all the reported methods, the drift estimation method based on bright-field image correlation requires no extra sample preparation or complicated modification to the imaging setup. However, the performance of this method is limited by the contrast of bright-field images, especially for the structures without sufficient features. In this paper, we proposed to use differential phase contrast (DPC) microscopy to enhance the image contrast and presented a 3D drift correction method with higher precision and robustness. This DPC-based drift correction method is suitable even for biological samples without clear morphological features. We demonstrated that this method can achieve a correction precision of < 6 nm in both the lateral direction and axial direction. Using SMLM imaging of microtubules, we verified that this method provides a comparable drift estimation performance as redundant cross-correlation.

Estimating the dynamic range of quantitative single-molecule localization microscopy

In recent years, there have been significant advances in quantifying molecule copy number and protein stoichiometry with single-molecule localization microscopy (SMLM). However, as the density of fluorophores per diffraction-limited spot increases, distinguishing between detection events from different fluorophores becomes progressively more difficult, affecting the accuracy of such measurements. Although essential to the design of quantitative experiments, the dynamic range of SMLM counting techniques has not yet been studied in detail. Here, we provide a working definition of the dynamic range for quantitative SMLM in terms of the relative number of missed localizations or blinks and explore the photophysical and experimental parameters that affect it. We begin with a simple two-state model of blinking fluorophores, then extend the model to incorporate photobleaching and temporal binning by the detection camera. From these models, we first show that our estimates of the dynamic range agree with realistic simulations of the photoswitching. We find that the dynamic range scales inversely with the duty cycle when counting both blinks and localizations. Finally, we validate our theoretical approach on direct stochastic optical reconstruction microscopy (dSTORM) data sets of photoswitching Alexa Fluor 647 dyes. Our results should help guide researchers in designing and implementing SMLM-based molecular counting experiments.

Real-time 3D single-molecule localization microscopy analysis using lookup tables

Herein, we present a new algorithm for real-time analysis of 3D single molecule localization microscopy images with a small impact on fitting accuracy using lookup-tables with discrete xyz-positions. The algorithm realizes real-time visualization during acquisition. We demonstrate its performance on simulated and measured data. Additionally, combining real-time fitting with a feedback loop controlling the activation laser pulse keeps the number of emitters per image frame constant.

Probing Chromatin Compaction and Its Epigenetic States in situ With Single-Molecule Localization-Based Super-Resolution Microscopy

Chromatin organization play a vital role in gene regulation and genome maintenance in normal biological processes and in response to environmental insults. Disruption of chromatin organization imposes a significant effect on many cellular processes and is often associated with a range of pathological processes such as aging and cancer. Extensive attention has been attracted to understand the structural and functional studies of chromatin architecture. Biochemical assays coupled with the state-of-the-art genomic technologies have been traditionally used to probe chromatin architecture. Recent advances in single molecule localization microscopy (SMLM) open up new opportunities to directly visualize higher-order chromatin architecture, its compaction status and its functional states at nanometer resolution in the intact cells or tissue. In this review, we will first discuss the recent technical advantages and challenges of using SMLM to image chromatin architecture. Next, we will focus on the recent applications of SMLM for structural and functional studies to probe chromatin architecture in key cellular processes. Finally, we will provide our perspectives on the recent development and potential applications of super-resolution imaging of chromatin architecture in improving our understanding in diseases.

Comparing Super-Resolution Microscopy Techniques to Analyze Chromosomes

The importance of fluorescence light microscopy for understanding cellular and sub-cellular structures and functions is undeniable. However, the resolution is limited by light diffraction (~200–250 nm laterally, ~500–700 nm axially). Meanwhile, super-resolution microscopy, such as structured illumination microscopy (SIM), is being applied more and more to overcome this restriction. Instead, super-resolution by stimulated emission depletion (STED) microscopy achieving a resolution of ~50 nm laterally and ~130 nm axially has not yet frequently been applied in plant cell research due to the required specific sample preparation and stable dye staining. Single-molecule localization microscopy (SMLM) including photoactivated localization microscopy (PALM) has not yet been widely used, although this nanoscopic technique allows even the detection of single molecules. In this study, we compared protein imaging within metaphase chromosomes of barley via conventional wide-field and confocal microscopy, and the sub-diffraction methods SIM, STED, and SMLM. The chromosomes were labeled by DAPI (4′,6-diamidino-2-phenylindol), a DNA-specific dye, and with antibodies against topoisomerase IIα (Topo II), a protein important for correct chromatin condensation. Compared to the diffraction-limited methods, the combination of the three different super-resolution imaging techniques delivered tremendous additional insights into the plant chromosome architecture through the achieved increased resolution.

Super-resolution Microscopy with Single Molecules in Biology and Beyond-Essentials, Current Trends, and Future Challenges

Single-molecule super-resolution microscopy has developed from a specialized technique into one of the most versatile and powerful imaging methods of the nanoscale over the past two decades. In this perspective, we provide a brief overview of the historical development of the field, the fundamental concepts, the methodology required to obtain maximum quantitative information, and the current state of the art. Then, we will discuss emerging perspectives and areas where innovation and further improvement are needed. Despite the tremendous progress, the full potential of single-molecule super-resolution microscopy is yet to be realized, which will be enabled by the research ahead of us.

Potential quality improvement of stochastic optical localization nanoscopy images obtained by frame by frame localization algorithms

A data movie of stochastic optical localization nanoscopy contains spatial and temporal correlations, both providing information of emitter locations. The majority of localization algorithms in the literature estimate emitter locations by frame-by-frame localization (FFL), which exploit only the spatial correlation and leave the temporal correlation into the FFL nanoscopy images. The temporal correlation contained in the FFL images, if exploited, can improve the localization accuracy and the image quality. In this paper, we analyze the properties of the FFL images in terms of root mean square minimum distance (RMSMD) and root mean square error (RMSE). It is shown that RMSMD and RMSE can be potentially reduced by a maximum fold equal to the square root of the average number of activations per emitter. Analyzed and revealed are also several statistical properties of RMSMD and RMSE and their relationship with respect to a large number of data frames, bias and variance of localization errors, small localization errors, sample drift, and the worst FFL image. Numerical examples are taken and the results confirm the prediction of analysis. The ideas about how to develop an algorithm to exploit the temporal correlation of FFL images are also briefly discussed. The results suggest development of two kinds of localization algorithms: the algorithms that can exploit the temporal correlation of FFL images and the unbiased localization algorithms.