Accelerating 3D single-molecule localization microscopy using blind sparse inpainting

Deep learning-based spectroscopic single-molecule localization microscopy

Significance

Spectroscopic single-molecule localization microscopy (sSMLM) takes advantage of nanoscopy and spectroscopy, enabling sub-10 nm resolution as well as simultaneous multicolor imaging of multi-labeled samples. Reconstruction of raw sSMLM data using deep learning is a promising approach for visualizing the subcellular structures at the nanoscale.

Aim

Develop a novel computational approach leveraging deep learning to reconstruct both label-free and fluorescence-labeled sSMLM imaging data.

Approach

We developed a two-network-model based deep learning algorithm, termed DsSMLM, to reconstruct sSMLM data. The effectiveness of DsSMLM was assessed by conducting imaging experiments on diverse samples, including label-free single-stranded DNA (ssDNA) fiber, fluorescence-labeled histone markers on COS-7 and U2OS cells, and simultaneous multicolor imaging of synthetic DNA origami nanoruler.

Results

For label-free imaging, a spatial resolution of 6.22 nm was achieved on ssDNA fiber; for fluorescence-labeled imaging, DsSMLM revealed the distribution of chromatin-rich and chromatin-poor regions defined by histone markers on the cell nucleus and also offered simultaneous multicolor imaging of nanoruler samples, distinguishing two dyes labeled in three emitting points with a separation distance of 40 nm. With DsSMLM, we observed enhanced spectral profiles with 8.8% higher localization detection for single-color imaging and up to 5.05% higher localization detection for simultaneous two-color imaging.

Conclusions

We demonstrate the feasibility of deep learning-based reconstruction for sSMLM imaging applicable to label-free and fluorescence-labeled sSMLM imaging data. We anticipate our technique will be a valuable tool for high-quality super-resolution imaging for a deeper understanding of DNA molecules' photophysics and will facilitate the investigation of multiple nanoscopic cellular structures and their interactions.

ResNet-based image inpainting method for enhancing the imaging speed of single molecule localization microscopy

Single molecule localization microscopy (SMLM) is a mainstream method in the field of super-resolution fluorescence microscopy that can achieve a spatial resolution of 20∼30 nm through a simple optical system. SMLM usually requires thousands of raw images to reconstruct a super-resolution image, and thus suffers from a slow imaging speed. Recently, several methods based on image inpainting have been developed to enhance the imaging speed of SMLM. However, these image inpainting methods may also produce erroneous local features (or called image artifacts), for example, incorrectly joined or split filaments. In this study, we use the ResNet generator, a network with strong local feature extraction capability, to replace the popularly-used U-Net generator to minimize the image artifact problem in current image inpainting methods, and develop an image inpainting method called DI-STORM. We validate our method using both simulated and experimental data, and demonstrate that DI-STORM has the best acceleration capability and produces the least artifacts in the repaired images, as compared with VDSR (the simplest CNN-based image inpainting method in SMLM) and ANNA-PALM (the best GAN-based image inpainting method in SMLM). We believe that DI-STORM could facilitate the application of deep learning-based image inpainting methods for SMLM.

Accelerating multi-emitter localization in super-resolution localization microscopy with FPGA-GPU cooperative computation

The real-time multi-emitter localization method is essential for advancing high-throughput super-resolution localization microscopy (HT-SRLM). In the past decade, the graphics processing unit (GPU) computation has been dominantly used to accelerate the execution speed of the multi-emitter localization method. However, if HT-SRLM is combined with a scientific complementary metal-oxide-semiconductor (sCMOS) camera working at full frame rate, real-time image processing is still difficult to achieve using this acceleration approach, thus resulting in a massive data storage challenge and even system crash. Here we take advantage of the cooperative acceleration power of field programming gate array (FPGA) computation and GPU computation, and propose a method called HCP-STORM to enable real-time multi-emitter localization. Using simulated images, we verified that HCP-STORM is capable of providing real-time image processing for raw images from a representative Hamamatsu Flash 4 V3 sCMOS camera working at full frame rate (that is, 2048×2048 pixels @ 10 ms exposure time). Using experimental images, we prove that HCP-STORM is 25 times faster than QC-STORM and 295 times faster than ThunderSTORM, with a small but acceptable degradation in image quality. This study shows the potential of FPGA-GPU cooperative computation in accelerating multi-emitter localization, and pushes a significant step toward the maturity of HT-SRLM technology.

From whole organism to ultrastructure: progress in axonal imaging for decoding circuit development

Since the pioneering work of Ramón y Cajal, scientists have sought to unravel the complexities of axon development underlying neural circuit formation. Micrometer-scale axonal growth cones navigate to targets that are often centimeters away. To reach their targets, growth cones react to dynamic environmental cues that change in the order of seconds to days. Proper axon growth and guidance are essential to circuit formation, and progress in imaging has been integral to studying these processes. In particular, advances in high- and super-resolution microscopy provide the spatial and temporal resolution required for studying developing axons. In this Review, we describe how improved microscopy has revolutionized our understanding of axonal development. We discuss how novel technologies, specifically light-sheet and super-resolution microscopy, led to new discoveries at the cellular scale by imaging axon outgrowth and circuit wiring with extreme precision. We next examine how advanced microscopy broadened our understanding of the subcellular dynamics driving axon growth and guidance. We finally assess the current challenges that the field of axonal biology still faces for imaging axons, and examine how future technology could meet these needs.

Automated and semi-automated enhancement, segmentation and tracing of cytoskeletal networks in microscopic images: A review

Cytoskeletal filaments are structures of utmost importance to biological cells and organisms due to their versatility and the significant functions they perform. These biopolymers are most often organised into network-like scaffolds with a complex morphology. Understanding the geometrical and topological organisation of these networks provides key insights into their functional roles. However, this non-trivial task requires a combination of high-resolution microscopy and sophisticated image processing/analysis software. The correct analysis of the network structure and connectivity needs precise segmentation of microscopic images. While segmentation of filament-like objects is a well-studied concept in biomedical imaging, where tracing of neurons and blood vessels is routine, there are comparatively fewer studies focusing on the segmentation of cytoskeletal filaments and networks from microscopic images. The developments in the fields of microscopy, computer vision and deep learning, however, began to facilitate the task, as reflected by an increase in the recent literature on the topic. Here, we aim to provide a short summary of the research on the (semi-)automated enhancement, segmentation and tracing methods that are particularly designed and developed for microscopic images of cytoskeletal networks. In addition to providing an overview of the conventional methods, we cover the recently introduced, deep-learning-assisted methods alongside the advantages they offer over classical methods.