DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning

Annotation and automated segmentation of single-molecule localisation microscopy data

Single Molecule Localisation Microscopy (SMLM) is becoming a widely used technique in cell biology. After processing the images, the molecular localisations are typically stored in a table as xy (or xyz) coordinates, with additional information, such as number of photons, etc. This set of coordinates can be used to generate an image to visualise the molecular distribution, for example, a 2D or 3D histogram of localisations. Many different methods have been devised to analyse SMLM data, among which cluster analysis of the localisations is popular. However, it can be useful to first segment the data, to extract the localisations in a specific region of a cell or in individual cells, prior to downstream analysis. Here we describe a pipeline for annotating localisations in an SMLM dataset in which we compared membrane segmentation approaches, including Otsu thresholding and machine learning models, and subsequent cell segmentation. We used an SMLM dataset derived from dSTORM images of sectioned cell pellets, stained for the membrane proteins EGFR (epidermal growth factor receptor) and EREG (epiregulin) as a test dataset. We found that a Cellpose model retrained on our data performed the best in the membrane segmentation task, allowing us to perform downstream cluster analysis of membrane versus cell interior localisations. We anticipate this will be generally useful for SMLM analysis.

In silico labeling in cell biology: Potential and limitations

In silico labeling is the computational cross-modality image translation where the output modality is a subcellular marker that is not specifically encoded in the input image, for example, in silico localization of organelles from transmitted light images. In principle, in silico labeling has the potential to facilitate rapid live imaging of multiple organelles with reduced photobleaching and phototoxicity, a technology enabling a major leap toward understanding the cell as an integrated complex system. However, five years have passed since feasibility was attained, without any demonstration of using in silico labeling to uncover new biological insight. In here, we discuss the current state of in silico labeling, the limitations preventing it from becoming a practical tool, and how we can overcome these limitations to reach its full potential.

Machine Learning for Single-Molecule Localization Microscopy: From Data Analysis to Quantification

Single-molecule localization microscopy (SMLM) is a versatile tool for realizing nanoscale imaging with visible light and providing unprecedented opportunities to observe bioprocesses. The integration of machine learning with SMLM enhances data analysis by improving efficiency and accuracy. This tutorial aims to provide a comprehensive overview of the data analysis process and theoretical aspects of SMLM, while also highlighting the typical applications of machine learning in this field. By leveraging advanced analytical techniques, SMLM is becoming a powerful quantitative analysis tool for biological research.

Functional Assessment of Cerebral Capillaries using Single Capillary Reporters in Ultrasound Localization Microscopy

The brain's microvascular cerebral capillary network plays a vital role in maintaining neuronal health, yet capillary dynamics are still not well understood due to limitations in existing imaging techniques. Here, we present Single Capillary Reporters (SCaRe) for transcranial Ultrasound Localization Microscopy (ULM), a novel approach enabling non-invasive, whole-brain mapping of single capillaries and estimates of their transit-time as a neurovascular biomarker. We accomplish this first through computational Monte Carlo and ultrasound simulations of microbubbles flowing through a fully-connected capillary network. We unveil distinct capillary flow behaviors which informs methodological changes to ULM acquisitions to better capture capillaries in vivo. Subsequently, applying SCaRe-ULM in vivo, we achieve unprecedented visualization of single capillary tracks across brain regions, analysis of layer-specific capillary heterogeneous transit times (CHT), and characterization of whole microbubble trajectories from arterioles to venules. Lastly, we evaluate capillary biomarkers using injected lipopolysaccharide to induce systemic neuroinflammation and track the increase in SCaRe-ULM CHT, demonstrating the capability to detect subtle capillary functional changes. SCaRe-ULM represents a significant advance in studying microvascular dynamics, offering novel avenues for investigating capillary patterns in neurological disorders and potential diagnostic applications.

Intelligent Beam Optimization for Light-Sheet Fluorescence Microscopy through Deep Learning

Light-sheet fluorescence microscopy (LSFM) provides the benefit of optical sectioning coupled with rapid acquisition times, enabling high-resolution 3-dimensional imaging of large tissue-cleared samples. Inherent to LSFM, the quality of the imaging heavily relies on the characteristics of the illumination beam, which only illuminates a thin section of the sample. Therefore, substantial efforts are dedicated to identifying slender, nondiffracting beam profiles that yield uniform and high-contrast images. An ongoing debate concerns the identification of optimal illumination beams for different samples: Gaussian, Bessel, Airy patterns, and/or others. However, comparisons among different beam profiles are challenging as their optimization objectives are often different. Given that our large imaging datasets (approximately 0.5 TB of images per sample) are already analyzed using deep learning models, we envisioned a different approach to the problem by designing an illumination beam tailored to boost the performance of the deep learning model. We hypothesized that integrating the physical LSFM illumination model (after passing it through a variable phase mask) into the training of a cell detection network would achieve this goal. Here, we report that joint optimization continuously updates the phase mask and results in improved image quality for better cell detection. The efficacy of our method is demonstrated through both simulations and experiments that reveal substantial enhancements in imaging quality compared to the traditional Gaussian light sheet. We discuss how designing microscopy systems through a computational approach provides novel insights for advancing optical design that relies on deep learning models for the analysis of imaging datasets.

EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging

Ultrafast 3D imaging is indispensable for visualizing complex and dynamic biological processes. Conventional scanning-based techniques necessitate an inherent trade-off between acquisition speed and space-bandwidth product (SBP). Emerging single-shot 3D wide-field techniques offer a promising alternative but are bottlenecked by the synchronous readout constraints of conventional CMOS systems, thus restricting data throughput to maintain high SBP at limited frame rates. To address this, we introduce EventLFM, a straightforward and cost-effective system that overcomes these challenges by integrating an event camera with Fourier light field microscopy (LFM), a state-of-the-art single-shot 3D wide-field imaging technique. The event camera operates on a novel asynchronous readout architecture, thereby bypassing the frame rate limitations inherent to conventional CMOS systems. We further develop a simple and robust event-driven LFM reconstruction algorithm that can reliably reconstruct 3D dynamics from the unique spatiotemporal measurements captured by EventLFM. Experimental results demonstrate that EventLFM can robustly reconstruct fast-moving and rapidly blinking 3D fluorescent samples at kHz frame rates. Furthermore, we highlight EventLFM's capability for imaging of blinking neuronal signals in scattering mouse brain tissues and 3D tracking of GFP-labeled neurons in freely moving C. elegans. We believe that the combined ultrafast speed and large 3D SBP offered by EventLFM may open up new possibilities across many biomedical applications.

Depth-enhanced high-throughput microscopy by compact PSF engineering

High-throughput microscopy is vital for screening applications, where three-dimensional (3D) cellular models play a key role. However, due to defocus susceptibility, current 3D high-throughput microscopes require axial scanning, which lowers throughput and increases photobleaching and photodamage. Point spread function (PSF) engineering is an optical method that enables various 3D imaging capabilities, yet it has not been implemented in high-throughput microscopy due to the cumbersome optical extension it typically requires. Here we demonstrate compact PSF engineering in the objective lens, which allows us to enhance the imaging depth of field and, combined with deep learning, recover 3D information using single snapshots. Beyond the applications shown here, this work showcases the usefulness of high-throughput microscopy in obtaining training data for deep learning-based algorithms, applicable to a variety of microscopy modalities.

Temporal analysis of relative distances (TARDIS) is a robust, parameter-free alternative to single-particle tracking

In single-particle tracking, individual particles are localized and tracked over time to probe their diffusion and molecular interactions. Temporal crossing of trajectories, blinking particles, and false-positive localizations present computational challenges that have remained difficult to overcome. Here we introduce a robust, parameter-free alternative to single-particle tracking: temporal analysis of relative distances (TARDIS). In TARDIS, an all-to-all distance analysis between localizations is performed with increasing temporal shifts. These pairwise distances represent either intraparticle distances originating from the same particle, or interparticle distances originating from unrelated particles, and are fitted analytically to obtain quantitative measures on particle dynamics. We showcase that TARDIS outperforms tracking algorithms, benchmarked on simulated and experimental data of varying complexity. We further show that TARDIS performs accurately in complex conditions characterized by high particle density, strong emitter blinking or false-positive localizations, and is in fact limited by the capabilities of localization algorithms. TARDIS' robustness enables fivefold shorter measurements without loss of information.

Deep learning-based spectroscopic single-molecule localization microscopy

Significance

Spectroscopic single-molecule localization microscopy (sSMLM) takes advantage of nanoscopy and spectroscopy, enabling sub-10 nm resolution as well as simultaneous multicolor imaging of multi-labeled samples. Reconstruction of raw sSMLM data using deep learning is a promising approach for visualizing the subcellular structures at the nanoscale.

Aim

Develop a novel computational approach leveraging deep learning to reconstruct both label-free and fluorescence-labeled sSMLM imaging data.

Approach

We developed a two-network-model based deep learning algorithm, termed DsSMLM, to reconstruct sSMLM data. The effectiveness of DsSMLM was assessed by conducting imaging experiments on diverse samples, including label-free single-stranded DNA (ssDNA) fiber, fluorescence-labeled histone markers on COS-7 and U2OS cells, and simultaneous multicolor imaging of synthetic DNA origami nanoruler.

Results

For label-free imaging, a spatial resolution of 6.22 nm was achieved on ssDNA fiber; for fluorescence-labeled imaging, DsSMLM revealed the distribution of chromatin-rich and chromatin-poor regions defined by histone markers on the cell nucleus and also offered simultaneous multicolor imaging of nanoruler samples, distinguishing two dyes labeled in three emitting points with a separation distance of 40 nm. With DsSMLM, we observed enhanced spectral profiles with 8.8% higher localization detection for single-color imaging and up to 5.05% higher localization detection for simultaneous two-color imaging.

Conclusions

We demonstrate the feasibility of deep learning-based reconstruction for sSMLM imaging applicable to label-free and fluorescence-labeled sSMLM imaging data. We anticipate our technique will be a valuable tool for high-quality super-resolution imaging for a deeper understanding of DNA molecules' photophysics and will facilitate the investigation of multiple nanoscopic cellular structures and their interactions.

DeepDOF-SE: affordable deep-learning microscopy platform for slide-free histology

Histopathology plays a critical role in the diagnosis and surgical management of cancer. However, access to histopathology services, especially frozen section pathology during surgery, is limited in resource-constrained settings because preparing slides from resected tissue is time-consuming, labor-intensive, and requires expensive infrastructure. Here, we report a deep-learning-enabled microscope, named DeepDOF-SE, to rapidly scan intact tissue at cellular resolution without the need for physical sectioning. Three key features jointly make DeepDOF-SE practical. First, tissue specimens are stained directly with inexpensive vital fluorescent dyes and optically sectioned with ultra-violet excitation that localizes fluorescent emission to a thin surface layer. Second, a deep-learning algorithm extends the depth-of-field, allowing rapid acquisition of in-focus images from large areas of tissue even when the tissue surface is highly irregular. Finally, a semi-supervised generative adversarial network virtually stains DeepDOF-SE fluorescence images with hematoxylin-and-eosin appearance, facilitating image interpretation by pathologists without significant additional training. We developed the DeepDOF-SE platform using a data-driven approach and validated its performance by imaging surgical resections of suspected oral tumors. Our results show that DeepDOF-SE provides histological information of diagnostic importance, offering a rapid and affordable slide-free histology platform for intraoperative tumor margin assessment and in low-resource settings.

Large-FOV 3D localization microscopy by spatially variant point spread function generation

Accurate characterization of the microscopic point spread function (PSF) is crucial for achieving high-performance localization microscopy (LM). Traditionally, LM assumes a spatially invariant PSF to simplify the modeling of the imaging system. However, for large fields of view (FOV) imaging, it becomes important to account for the spatially variant nature of the PSF. Here, we propose an accurate and fast principal components analysis-based field-dependent 3D PSF generator (PPG3D) and localizer for LM. Through simulations and experimental three-dimensional (3D) single-molecule localization microscopy (SMLM), we demonstrate the effectiveness of PPG3D, enabling super-resolution imaging of mitochondria and microtubules with high fidelity over a large FOV. A comparison of PPG3D with a shift-variant PSF generator for 3D LM reveals a threefold improvement in accuracy. Moreover, PPG3D is approximately 100 times faster than existing PSF generators, when used in image plane-based interpolation mode. Given its user-friendliness, we believe that PPG3D holds great potential for widespread application in SMLM and other imaging modalities.

High-density volumetric super-resolution microscopy

Volumetric super-resolution microscopy typically encodes the 3D position of single-molecule fluorescence into a 2D image by changing the shape of the point spread function (PSF) as a function of depth. However, the resulting large and complex PSF spatial footprints reduce biological throughput and applicability by requiring lower labeling densities to avoid overlapping fluorescent signals. We quantitatively compare the density dependence of single-molecule light field microscopy (SMLFM) to other 3D PSFs (astigmatism, double helix and tetrapod) showing that SMLFM enables an order-of-magnitude speed improvement compared to the double helix PSF by resolving overlapping emitters through parallax. We demonstrate this optical robustness experimentally with high accuracy ( > 99.2 ± 0.1%, 0.1 locs μm-2) and sensitivity ( > 86.6 ± 0.9%, 0.1 locs μm-2) through whole-cell (scan-free) imaging and tracking of single membrane proteins in live primary B cells. We also exemplify high-density volumetric imaging (0.15 locs μm-2) in dense cytosolic tubulin datasets.

TIE-GANs: single-shot quantitative phase imaging using transport of intensity equation with integration of GANs

Significance

Artificial intelligence (AI) has become a prominent technology in computational imaging over the past decade. The expeditious and label-free characteristics of quantitative phase imaging (QPI) render it a promising contender for AI investigation. Though interferometric methodologies exhibit potential efficacy, their implementation involves complex experimental platforms and computationally intensive reconstruction procedures. Hence, non-interferometric methods, such as transport of intensity equation (TIE), are preferred over interferometric methods.

Aim

TIE method, despite its effectiveness, is tedious as it requires the acquisition of many images at varying defocus planes. The proposed methodology holds the ability to generate a phase image utilizing a single intensity image using generative adversarial networks (GANs). We present a method called TIE-GANs to overcome the multi-shot scheme of conventional TIE.

Approach

The present investigation employs the TIE as a QPI methodology, which necessitates reduced experimental and computational efforts. TIE is being used for the dataset preparation as well. The proposed method captures images from different defocus planes for training. Our approach uses an image-to-image translation technique to produce phase maps and is based on GANs. The main contribution of this work is the introduction of GANs with TIE (TIE:GANs) that can give better phase reconstruction results with shorter computation times. This is the first time the GANs is proposed for TIE phase retrieval.

Results

The characterization of the system was carried out with microbeads of 4    μ m size and structural similarity index (SSIM) for microbeads was found to be 0.98. We demonstrated the application of the proposed method with oral cells, which yielded a maximum SSIM value of 0.95. The key characteristics include mean squared error and peak-signal-to-noise ratio values of 140 and 26.42 dB for oral cells and 100 and 28.10 dB for microbeads.

Conclusions

The proposed methodology holds the ability to generate a phase image utilizing a single intensity image. Our method is feasible for digital cytology because of its reported high value of SSIM. Our approach can handle defocused images in such a way that it can take intensity image from any defocus plane within the provided range and able to generate phase map.

Advanced optical imaging for the rational design of nanomedicines

Despite the enormous potential of nanomedicines to shape the future of medicine, their clinical translation remains suboptimal. Translational challenges are present in every step of the development pipeline, from a lack of understanding of patient heterogeneity to insufficient insights on nanoparticle properties and their impact on material-cell interactions. Here, we discuss how the adoption of advanced optical microscopy techniques, such as super-resolution optical microscopies, correlative techniques, and high-content modalities, could aid the rational design of nanocarriers, by characterizing the cell, the nanomaterial, and their interaction with unprecedented spatial and/or temporal detail. In this nanomedicine arena, we will discuss how the implementation of these techniques, with their versatility and specificity, can yield high volumes of multi-parametric data; and how machine learning can aid the rapid advances in microscopy: from image acquisition to data interpretation.

Enhancing Light-Sheet Fluorescence Microscopy Illumination Beams through Deep Design Optimization

Light sheet fluorescence microscopy (LSFM) provides the benefit of optical sectioning coupled with rapid acquisition times for imaging of tissue-cleared specimen. This allows for high-resolution 3D imaging of large tissue volumes. Inherently to LSFM, the quality of the imaging heavily relies on the characteristics of the illumination beam, with the notion that the illumination beam only illuminates a thin section that is being imaged. Therefore, substantial efforts are dedicated to identifying slender, non-diffracting beam profiles that can yield uniform and high-contrast images. An ongoing debate concerns the employment of the most optimal illumination beam; Gaussian, Bessel, Airy patterns and/or others. Comparisons among different beam profiles is challenging as their optimization objective is often different. Given that our large imaging datasets (~0.5TB images per sample) is already analyzed using deep learning models, we envisioned a different approach to this problem by hypothesizing that we can tailor the illumination beam to boost the deep learning models performance. We achieve this by integrating the physical LSFM illumination model after passing through a variable phase mask into the training of a cell detection network. Here we report that the joint optimization continuously updates the phase mask, improving the image quality for better cell detection. Our method's efficacy is demonstrated through both simulations and experiments, revealing substantial enhancements in imaging quality compared to traditional Gaussian light sheet. We offer valuable insights for designing microscopy systems through a computational approach that exhibits significant potential for advancing optics design that relies on deep learning models for analysis of imaging datasets.

Roadmap on Label-Free Super-Resolution Imaging

Label-free super-resolution (LFSR) imaging relies on light-scattering processes in nanoscale objects without a need for fluorescent (FL) staining required in super-resolved FL microscopy. The objectives of this Roadmap are to present a comprehensive vision of the developments, the state-of-the-art in this field, and to discuss the resolution boundaries and hurdles which need to be overcome to break the classical diffraction limit of the LFSR imaging. The scope of this Roadmap spans from the advanced interference detection techniques, where the diffraction-limited lateral resolution is combined with unsurpassed axial and temporal resolution, to techniques with true lateral super-resolution capability which are based on understanding resolution as an information science problem, on using novel structured illumination, near-field scanning, and nonlinear optics approaches, and on designing superlenses based on nanoplasmonics, metamaterials, transformation optics, and microsphere-assisted approaches. To this end, this Roadmap brings under the same umbrella researchers from the physics and biomedical optics communities in which such studies have often been developing separately. The ultimate intent of this paper is to create a vision for the current and future developments of LFSR imaging based on its physical mechanisms and to create a great opening for the series of articles in this field.

High-fidelity 3D live-cell nanoscopy through data-driven enhanced super-resolution radial fluctuation

Live-cell super-resolution microscopy enables the imaging of biological structure dynamics below the diffraction limit. Here we present enhanced super-resolution radial fluctuations (eSRRF), substantially improving image fidelity and resolution compared to the original SRRF method. eSRRF incorporates automated parameter optimization based on the data itself, giving insight into the trade-off between resolution and fidelity. We demonstrate eSRRF across a range of imaging modalities and biological systems. Notably, we extend eSRRF to three dimensions by combining it with multifocus microscopy. This realizes live-cell volumetric super-resolution imaging with an acquisition speed of ~1 volume per second. eSRRF provides an accessible super-resolution approach, maximizing information extraction across varied experimental conditions while minimizing artifacts. Its optimal parameter prediction strategy is generalizable, moving toward unbiased and optimized analyses in super-resolution microscopy.

DBlink: dynamic localization microscopy in super spatiotemporal resolution via deep learning

Single-molecule localization microscopy (SMLM) has revolutionized biological imaging, improving the spatial resolution of traditional microscopes by an order of magnitude. However, SMLM techniques require long acquisition times, typically a few minutes, to yield a single super-resolved image, because they depend on accumulation of many localizations over thousands of recorded frames. Hence, the capability of SMLM to observe dynamics at high temporal resolution has always been limited. In this work, we present DBlink, a deep-learning-based method for super spatiotemporal resolution reconstruction from SMLM data. The input to DBlink is a recorded video of SMLM data and the output is a super spatiotemporal resolution video reconstruction. We use a convolutional neural network combined with a bidirectional long short-term memory network architecture, designed for capturing long-term dependencies between different input frames. We demonstrate DBlink performance on simulated filaments and mitochondria-like structures, on experimental SMLM data under controlled motion conditions and on live-cell dynamic SMLM. DBlink's spatiotemporal interpolation constitutes an important advance in super-resolution imaging of dynamic processes in live cells.

Optical imaging technologies for in vivo cancer detection in low-resource settings

Cancer continues to affect underserved populations disproportionately. Novel optical imaging technologies, which can provide rapid, non-invasive, and accurate cancer detection at the point of care, have great potential to improve global cancer care. This article reviews the recent technical innovations and clinical translation of low-cost optical imaging technologies, highlighting the advances in both hardware and software, especially the integration of artificial intelligence, to improve in vivo cancer detection in low-resource settings. Additionally, this article provides an overview of existing challenges and future perspectives of adapting optical imaging technologies into clinical practice, which can potentially contribute to novel insights and programs that effectively improve cancer detection in low-resource settings.

LocNet: deep learning-based localization on a rotating point spread function with applications to telescope imaging

Three-dimensional (3D) point source recovery from two-dimensional (2D) data is a challenging problem with wide-ranging applications in single-molecule localization microscopy and space-debris localization telescops. Point spread function (PSF) engineering is a promising technique to solve this 3D localization problem. Specifically, we consider the problem of 3D localization of space debris from a 2D image using a rotating PSF where the depth information is encoded in the angle of rotation of a single-lobe PSF for each point source. Instead of applying a model-based optimization, we introduce a convolution neural network (CNN)-based approach to localize space debris in full 3D space automatically. A hard sample training strategy is proposed to improve the performance of CNN further. Contrary to the traditional model-based methods, our technique is efficient and outperforms the current state-of-the-art method by more than 11% in the precision rate with a comparable improvement in the recall rate.

E2E-BPF microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution

Several image-based biomedical diagnoses require high-resolution imaging capabilities at large spatial scales. However, conventional microscopes exhibit an inherent trade-off between depth-of-field (DoF) and spatial resolution, and thus require objects to be refocused at each lateral location, which is time consuming. Here, we present a computational imaging platform, termed E2E-BPF microscope, which enables large-area, high-resolution imaging of large-scale objects without serial refocusing. This method involves a physics-incorporated, deep-learned design of binary phase filter (BPF) and jointly optimized deconvolution neural network, which altogether produces high-resolution, high-contrast images over extended depth ranges. We demonstrate the method through numerical simulations and experiments with fluorescently labeled beads, cells and tissue section, and present high-resolution imaging capability over a 15.5-fold larger DoF than the conventional microscope. Our method provides highly effective and scalable strategy for DoF-extended optical imaging system, and is expected to find numerous applications in rapid image-based diagnosis, optical vision, and metrology.

Fluorescence Microscopy: a statistics-optics perspective

Fundamental properties of light unavoidably impose features on images collected using fluorescence microscopes. Modeling these features is ever more important in quantitatively interpreting microscopy images collected at scales on par or smaller than light's wavelength. Here we review the optics responsible for generating fluorescent images, fluorophore properties, microscopy modalities leveraging properties of both light and fluorophores, in addition to the necessarily probabilistic modeling tools imposed by the stochastic nature of light and measurement.

Pupil engineering for extended depth-of-field imaging in a fluorescence miniscope

Significance

Fluorescence head-mounted microscopes, i.e., miniscopes, have emerged as powerful tools to analyze in-vivo neural populations but exhibit a limited depth-of-field (DoF) due to the use of high numerical aperture (NA) gradient refractive index (GRIN) objective lenses.

Aim

We present extended depth-of-field (EDoF) miniscope, which integrates an optimized thin and lightweight binary diffractive optical element (DOE) onto the GRIN lens of a miniscope to extend the DoF by 2.8× between twin foci in fixed scattering samples.

Approach

We use a genetic algorithm that considers the GRIN lens' aberration and intensity loss from scattering in a Fourier optics-forward model to optimize a DOE and manufacture the DOE through single-step photolithography. We integrate the DOE into EDoF-Miniscope with a lateral accuracy of 70  μm to produce high-contrast signals without compromising the speed, spatial resolution, size, or weight.

Results

We characterize the performance of EDoF-Miniscope across 5- and 10-μm fluorescent beads embedded in scattering phantoms and demonstrate that EDoF-Miniscope facilitates deeper interrogations of neuronal populations in a 100-μm-thick mouse brain sample and vessels in a whole mouse brain sample.

Conclusions

Built from off-the-shelf components and augmented by a customizable DOE, we expect that this low-cost EDoF-Miniscope may find utility in a wide range of neural recording applications.

Machine learning for cross-scale microscopy of viruses

Despite advances in virological sciences and antiviral research, viruses continue to emerge, circulate, and threaten public health. We still lack a comprehensive understanding of how cells and individuals remain susceptible to infectious agents. This deficiency is in part due to the complexity of viruses, including the cell states controlling virus-host interactions. Microscopy samples distinct cellular infection stages in a multi-parametric, time-resolved manner at molecular resolution and is increasingly enhanced by machine learning and deep learning. Here we discuss how state-of-the-art artificial intelligence (AI) augments light and electron microscopy and advances virological research of cells. We describe current procedures for image denoising, object segmentation, tracking, classification, and super-resolution and showcase examples of how AI has improved the acquisition and analyses of microscopy data. The power of AI-enhanced microscopy will continue to help unravel virus infection mechanisms, develop antiviral agents, and improve viral vectors.

Simultaneous spectral differentiation of multiple fluorophores in super-resolution imaging using a glass phase plate

By engineering the point-spread function (PSF) of single molecules, different fluorophore species can be imaged simultaneously and distinguished by their unique PSF patterns. Here, we insert a silicon-dioxide phase plate at the Fourier plane of the detection path of a wide-field fluorescence microscope to produce distinguishable PSFs (X-PSFs) at different wavelengths. We demonstrate that the resulting PSFs can be localized spatially and spectrally using a maximum-likelihood estimation algorithm and can be utilized for hyper-spectral super-resolution microscopy of biological samples. We produced superresolution images of fixed U2OS cells using X-PSFs for dSTORM imaging with simultaneous illumination of up to three fluorophore species. The species were distinguished only by the PSF pattern. We achieved ∼21-nm lateral localization precision (FWHM) and ∼17-nm axial precision (FWHM) with an average of 1,800 - 3,500 photons per PSF and a background as high as 130 - 400 photons per pixel. The modified PSF distinguished fluorescent probes with ∼80 nm separation between spectral peaks.

Neural network-assisted single-molecule localization microscopy with a weak-affinity protein tag

Single-molecule localization microscopy achieves nanometer spatial resolution by localizing single fluorophores separated in space and time. A major challenge of single-molecule localization microscopy is the long acquisition time, leading to low throughput, as well as to a poor temporal resolution that limits its use to visualize the dynamics of cellular structures in live cells. Another challenge is photobleaching, which reduces information density over time and limits throughput and the available observation time in live-cell applications. To address both challenges, we combine two concepts: first, we integrate the neural network DeepSTORM to predict super-resolution images from high-density imaging data, which increases acquisition speed. Second, we employ a direct protein label, HaloTag7, in combination with exchangeable ligands (xHTLs), for fluorescence labeling. This labeling method bypasses photobleaching by providing a constant signal over time and is compatible with live-cell imaging. The combination of both a neural network and a weak-affinity protein label reduced the acquisition time up to ∼25-fold. Furthermore, we demonstrate live-cell imaging with increased temporal resolution, and capture the dynamics of the endoplasmic reticulum over extended time without signal loss.

Near index matching enables solid diffractive optical element fabrication via additive manufacturing

Diffractive optical elements (DOEs) have a wide range of applications in optics and photonics, thanks to their capability to perform complex wavefront shaping in a compact form. However, widespread applicability of DOEs is still limited, because existing fabrication methods are cumbersome and expensive. Here, we present a simple and cost-effective fabrication approach for solid, high-performance DOEs. The method is based on conjugating two nearly refractive index-matched solidifiable transparent materials. The index matching allows for extreme scaling up of the elements in the axial dimension, which enables simple fabrication of a template using commercially available 3D printing at tens-of-micrometer resolution. We demonstrated the approach by fabricating and using DOEs serving as microlens arrays, vortex plates, including for highly sensitive applications such as vector beam generation and super-resolution microscopy using MINSTED, and phase-masks for three-dimensional single-molecule localization microscopy. Beyond the advantage of making DOEs widely accessible by drastically simplifying their production, the method also overcomes difficulties faced by existing methods in fabricating highly complex elements, such as high-order vortex plates, and spectrum-encoding phase masks for microscopy.

Incoherent diffractive optical elements for extendable field-of-view imaging

We present a diffractive optics design for incoherent imaging with an extendable field-of-view. In our design method, multiple layers of diffractive optical elements (DOEs) are synthesized so that images on the input plane illuminated with spatially incoherent light are reproduced upright on the output plane. In addition, our method removes the need for an approximation of shift invariance, which has been assumed in conventional optical designs for incoherent imaging systems. Once the DOE cascade is calculated, the field-of-view can be extended by using an array of such DOEs without further calculation. We derive the optical condition to calculate the DOEs and numerically demonstrate the proposed method with the condition.

Combining deep learning approaches and point spread function engineering for simultaneous 3D position and 3D orientation measurements of fluorescent single molecules

Point Spread Function (PSF) engineering is an effective method to increase the sensitivity of single-molecule fluorescence images to specific parameters. Classical phase mask optimization approaches have enabled the creation of new PSFs that can achieve, for example, localization precision of a few nanometers axially over a capture range of several microns with bright emitters. However, for complex high-dimensional optimization problems, classical approaches are difficult to implement and can be very time-consuming for computation. The advent of deep learning methods and their application to single-molecule imaging has provided a way to solve these problems. Here, we propose to combine PSF engineering and deep learning approaches to obtain both an optimized phase mask and a neural network structure to obtain the 3D position and 3D orientation of fixed fluorescent molecules. Our approach allows us to obtain an axial localization precision around 30 nanometers, as well as an orientation precision around 5 degrees for orientations and positions over a one micron depth range for a signal-to-noise ratio consistent with what is typical in single-molecule cellular imaging experiments.

Optimizing self-interference digital holography for single-molecule localization

Self-interference digital holography (SIDH) can image incoherently emitting objects over large axial ranges from three two-dimensional images. By combining SIDH with single-molecule localization microscopy (SMLM), incoherently emitting objects can be localized with nanometer precision over a wide axial range without mechanical refocusing. However, background light substantially degrades the performance of SIDH due to the relatively large size of the hologram. To optimize the performance of SIDH, we performed simulations to study the optimal hologram radius (Rh) for different levels of background photons. The results show that by reducing the size of the hologram, we can achieve a localization precision of better than 60 nm laterally and 80 nm axially over a 10 µm axial range under the conditions of low signal level (6000 photons) with 10 photons/pixel of background noise. We then performed experiments to demonstrate our optimized SIDH system. The results show that point sources emitting as few as 2120 photons can be successfully detected. We further demonstrated that we can successfully reconstruct point-like sources emitting 4200 photons over a 10 µm axial range by light-sheet SIDH.

Deep learning enables fast, gentle STED microscopy

STED microscopy is widely used to image subcellular structures with super-resolution. Here, we report that restoring STED images with deep learning can mitigate photobleaching and photodamage by reducing the pixel dwell time by one or two orders of magnitude. Our method allows for efficient and robust restoration of noisy 2D and 3D STED images with multiple targets and facilitates long-term imaging of mitochondrial dynamics.

When Super-Resolution Microscopy Meets Microfluidics: Enhanced Biological Imaging and Analysis with Unprecedented Resolution

Super-resolution microscopy is rapidly developed in recent years, allowing biologists to extract more quantitative information on subcellular processes in live cells that is usually not accessible with conventional techniques. However, super-resolution imaging is not fully exploited because of the lack of an appropriate and multifunctional experimental platform. As an important tool in life sciences, microfluidics is capable of cell manipulation and the regulation of the cellular environment because of its superior flexibility and biocompatibility. The combination of microfluidics and super-resolution microscopy revolutionizes the study of complex cellular properties and dynamics, providing valuable insights into cellular structure and biological functions at the single-molecule level. In this perspective, an overview of the main advantages of microfluidic technology that are essential to the performance of super-resolution microscopy are offered. The main benefits of performing super-resolution imaging with microfluidic devices are highlighted and perspectives on the diverse applications that are facilitated by combining these two powerful techniques are provided.

Quantitative Single-Molecule Localization Microscopy

Super-resolution fluorescence microscopy allows the investigation of cellular structures at nanoscale resolution using light. Current developments in super-resolution microscopy have focused on reliable quantification of the underlying biological data. In this review, we first describe the basic principles of super-resolution microscopy techniques such as stimulated emission depletion (STED) microscopy and single-molecule localization microscopy (SMLM), and then give a broad overview of methodological developments to quantify super-resolution data, particularly those geared toward SMLM data. We cover commonly used techniques such as spatial point pattern analysis, colocalization, and protein copy number quantification but also describe more advanced techniques such as structural modeling, single-particle tracking, and biosensing. Finally, we provide an outlook on exciting new research directions to which quantitative super-resolution microscopy might be applied.

Unidirectional imaging using deep learning-designed materials

A unidirectional imager would only permit image formation along one direction, from an input field-of-view (FOV) A to an output FOV B, and in the reverse path, B → A, the image formation would be blocked. We report the first demonstration of unidirectional imagers, presenting polarization-insensitive and broadband unidirectional imaging based on successive diffractive layers that are linear and isotropic. After their deep learning-based training, the resulting diffractive layers are fabricated to form a unidirectional imager. Although trained using monochromatic illumination, the diffractive unidirectional imager maintains its functionality over a large spectral band and works under broadband illumination. We experimentally validated this unidirectional imager using terahertz radiation, well matching our numerical results. We also created a wavelength-selective unidirectional imager, where two unidirectional imaging operations, in reverse directions, are multiplexed through different illumination wavelengths. Diffractive unidirectional imaging using structured materials will have numerous applications in, e.g., security, defense, telecommunications, and privacy protection.

Frequency-domain compression imaging for extending the field of view of infrared thermometers

We propose a computational imaging technique for expanding the field of view of infrared thermometers. The contradiction between the field of view and the focal length has always been a chief problem for researchers, especially in infrared optical systems. Large-area infrared detectors are expensive and technically arduous to be manufactured, which enormously limits the performance of the infrared optical system. On the other hand, the extensive use of infrared thermometers in COVID-19 has created a considerable demand for infrared optical systems. Therefore, improving the performance of infrared optical systems and increasing the utilization of infrared detectors is vital. This work proposes a multi-channel frequency-domain compression imaging method based on point spread function (PSF) engineering. Compared with conventional compressed sensing, the submitted method images once without an intermediate image plane. Furthermore, phase encoding is used without loss of illumination of the image surface. These facts can significantly reduce the volume of the optical system and improve the energy efficiency of the compressed imaging system. Therefore, its application in COVID-19 is of great value. We design a dual-channel frequency-domain compression imaging system to verify the proposed method's feasibility. Then, the wavefront coded PSF and optical transfer function (OTF) are used, and the two-step iterative shrinkage/thresholding (TWIST) algorithm is used to restore the image to get the final result. This compression imaging method provides a new idea for the large field of view monitoring systems, especially in infrared optical systems.

Deep learning using a residual deconvolutional network enables real-time high-density single-molecule localization microscopy

High-density localization based on deep learning is a very effective method to accelerate single molecule localization microscopy (SMLM). Compared with traditional high-density localization methods, deep learning-based methods enable a faster data processing speed and a higher localization accuracy. However, the reported high-density localization methods based on deep learning are still not fast enough to enable real time data processing for large batches of raw images, which is probably due to the heavy computational burden and computation complexity in the U-shape architecture used in these models. Here we propose a high-density localization method called FID-STORM, which is based on an improved residual deconvolutional network for the real-time processing of raw images. In FID-STORM, we use a residual network to extract the features directly from low-resolution raw images rather than the U-shape network from interpolated images. We also use a model fusion from TensorRT to further accelerate the inference of the model. In addition, we process the sum of the localization images directly on GPU to obtain an additional speed gain. Using simulated and experimental data, we verified that the FID-STORM method achieves a processing speed of 7.31 ms/frame at 256 × 256 pixels @ Nvidia RTX 2080 Ti graphic card, which is shorter than the typical exposure time of 10∼30 ms, thus enabling real-time data processing in high-density SMLM. Moreover, compared with a popular interpolated image-based method called Deep-STORM, FID-STORM enables a speed gain of ∼26 times, without loss of reconstruction accuracy. We also provided an ImageJ plugin for our new method.

Machine-learning-powered extraction of molecular diffusivity from single-molecule images for super-resolution mapping

While critical to biological processes, molecular diffusion is difficult to quantify, and spatial mapping of local diffusivity is even more challenging. Here we report a machine-learning-enabled approach, pixels-to-diffusivity (Pix2D), to directly extract the diffusion coefficient D from single-molecule images, and consequently enable super-resolved D spatial mapping. Working with single-molecule images recorded at a fixed framerate under typical single-molecule localization microscopy (SMLM) conditions, Pix2D exploits the often undesired yet evident motion blur, i.e., the convolution of single-molecule motion trajectory during the frame recording time with the diffraction-limited point spread function (PSF) of the microscope. Whereas the stochastic nature of diffusion imprints diverse diffusion trajectories to different molecules diffusing at the same given D, we construct a convolutional neural network (CNN) model that takes a stack of single-molecule images as the input and evaluates a D-value as the output. We thus validate robust D evaluation and spatial mapping with simulated data, and with experimental data successfully characterize D differences for supported lipid bilayers of different compositions and resolve gel and fluidic phases at the nanoscale.

Double helix point spread function with variable spacing for precise 3D particle localization

To extend the axial depth of nanoscale 3D-localization microscopy, we propose here a splicing-type vortex singularities (SVS) phase mask, which has been meticulously optimized with a Fresnel approximation imaging inverse operation. The optimized SVS DH-PSF has proven to have high transfer function efficiency with adjustable performance in its axial range. The axial position of the particle was computed by using both the main lobes' spacing and the rotation angle, an improvement of the localization precision of the particle. Concretely, the proposed optimized SVS DH-PSF, with a smaller spatial extent, can effectively reduce the overlap of nanoparticle images and realize the 3D localization of multiple nanoparticles with small spacing, with respect to PSFs for large axial 3D localization. Finally, we successfully conducted extensive experiments on 3D localization for tracking dense nanoparticles at 8µm depth with a numerical aperture of 1.4, demonstrating its great potential.

Optofluidic imaging meets deep learning: from merging to emerging

Propelled by the striking advances in optical microscopy and deep learning (DL), the role of imaging in lab-on-a-chip has dramatically been transformed from a silo inspection tool to a quantitative "smart" engine. A suite of advanced optical microscopes now enables imaging over a range of spatial scales (from molecules to organisms) and temporal window (from microseconds to hours). On the other hand, the staggering diversity of DL algorithms has revolutionized image processing and analysis at the scale and complexity that were once inconceivable. Recognizing these exciting but overwhelming developments, we provide a timely review of their latest trends in the context of lab-on-a-chip imaging, or coined optofluidic imaging. More importantly, here we discuss the strengths and caveats of how to adopt, reinvent, and integrate these imaging techniques and DL algorithms in order to tailor different lab-on-a-chip applications. In particular, we highlight three areas where the latest advances in lab-on-a-chip imaging and DL can form unique synergisms: image formation, image analytics and intelligent image-guided autonomous lab-on-a-chip. Despite the on-going challenges, we anticipate that they will represent the next frontiers in lab-on-a-chip imaging that will spearhead new capabilities in advancing analytical chemistry research, accelerating biological discovery, and empowering new intelligent clinical applications.

Single Molecule Localization Microscopy for Studying Small Extracellular Vesicles

Small extracellular vesicles (sEVs) are 30-200 nm nanovesicles enriched with unique cargoes of nucleic acids, lipids, and proteins. sEVs are released by all cell types and have emerged as a critical mediator of cell-to-cell communication. Although many studies have dealt with the role of sEVs in health and disease, the exact mechanism of sEVs biogenesis and uptake remain unexplored due to the lack of suitable imaging technologies. For sEVs functional studies, imaging has long relied on conventional fluorescence microscopy that has only 200-300 nm resolution, thereby generating blurred images. To break this resolution limit, recent developments in super-resolution microscopy techniques, specifically single-molecule localization microscopy (SMLM), expanded the understanding of subcellular details at the few nanometer level. SMLM success relies on the use of appropriate fluorophores with excellent blinking properties. In this review, the basic principle of SMLM is highlighted and the state of the art of SMLM use in sEV biology is summarized. Next, how SMLM techniques implemented for cell imaging can be translated to sEV imaging is discussed by applying different labeling strategies to study sEV biogenesis and their biomolecular interaction with the distant recipient cells.

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

Single-molecule localization microscopy in a typical wide-field setup has been widely used for investigating subcellular structures with super resolution; however, field-dependent aberrations restrict the field of view (FOV) to only tens of micrometers. Here, we present a deep-learning method for precise localization of spatially variant point emitters (FD-DeepLoc) over a large FOV covering the full chip of a modern sCMOS camera. Using a graphic processing unit-based vectorial point spread function (PSF) fitter, we can fast and accurately model the spatially variant PSF of a high numerical aperture objective in the entire FOV. Combined with deformable mirror-based optimal PSF engineering, we demonstrate high-accuracy three-dimensional single-molecule localization microscopy over a volume of ~180 × 180 × 5 μm³, allowing us to image mitochondria and nuclear pore complexes in entire cells in a single imaging cycle without hardware scanning; a 100-fold increase in throughput compared to the state of the art.

Single-molecule imaging in the primary cilium

The primary cilium is an important signaling organelle critical for normal development and tissue homeostasis. Its small dimensions and complexity necessitate advanced imaging approaches to uncover the molecular mechanisms behind its function. Here, we outline how single-molecule fluorescence microscopy can be used for tracking molecular dynamics and interactions and for super-resolution imaging of nanoscale structures in the primary cilium. Specifically, we describe in detail how to capture and quantify the 2D dynamics of individual transmembrane proteins PTCH1 and SMO and how to map the 3D nanoscale distributions of the inversin compartment proteins INVS, ANKS6, and NPHP3. This protocol can, with minor modifications, be adapted for studies of other proteins and cell lines to further elucidate the structure and function of the primary cilium at the molecular level.

Six-Dimensional Single-Molecule Imaging with Isotropic Resolution using a Multi-View Reflector Microscope

Imaging both the positions and orientations of single fluorophores, termed single-molecule orientation-localisation microscopy, is a powerful tool to study biochemical processes. However, the limited photon budget associated with single-molecule fluorescence makes high-dimensional imaging with isotropic, nanoscale spatial resolution a formidable challenge. Here, we realise a radially and azimuthally polarized multi-view reflector (raMVR) microscope for the imaging of the 3D positions and 3D orientations of single molecules, with precision of 10.9 nm and 2.0° over a 1.5 μm depth range. The raMVR microscope achieves 6D super-resolution imaging of Nile red (NR) molecules transiently bound to lipid-coated spheres, accurately resolving their spherical morphology despite refractive-index mismatch. By observing the rotational dynamics of NR, raMVR images also resolve the infiltration of lipid membranes by amyloid-beta oligomers without covalent labelling. Finally, we demonstrate 6D imaging of cell membranes, where the orientations of specific fluorophores reveal heterogeneity in membrane fluidity. With its nearly isotropic 3D spatial resolution and orientation measurement precision, we expect the raMVR microscope to enable 6D imaging of molecular dynamics within biological and chemical systems with exceptional detail.

Deep learning enables fast, gentle STED microscopy

STED microscopy is widely used to image subcellular structures with super-resolution. Here, we report that denoising STED images with deep learning can mitigate photobleaching and photodamage by reducing the pixel dwell time by one or two orders of magnitude. Our method allows for efficient and robust restoration of noisy 2D and 3D STED images with multiple targets and facilitates long-term imaging of mitochondrial dynamics.

Quantifying postsynaptic receptor dynamics: insights into synaptic function

The molecular composition of presynaptic and postsynaptic neuronal terminals is dynamic, and yet long-term stabilizations in postsynaptic responses are necessary for synaptic development and long-term plasticity. The need to reconcile these concepts is further complicated by learning- and memory-related plastic changes in the molecular make-up of synapses. Advances in single-particle tracking mean that we can now quantify the number and diffusive properties of specific synaptic molecules, while statistical thermodynamics provides a framework to analyse these molecular fluctuations. In this Review, we discuss the use of these approaches to gain quantitative descriptions of the processes underlying the turnover, long-term stability and plasticity of postsynaptic receptors and show how these can help us to understand the balance between local molecular turnover and synaptic structural identity and integrity.

Super-Resolution Microscopy: A Technique to Revolutionize Research and Diagnosis of Glomerulopathies

Background

For decades, knowledge about glomerular (patho)physiology has been tightly linked with advances in microscopic imaging technology. For example, the invention of electron microscopy was required to hypothesize about the mode of glomerular filtration barrier function.

Summary

Super-resolution techniques, defined as fluorescence microscopy approaches that surpass the optical resolution limit of around 200 nm, have been made available to the scientific community. Several of these different techniques are currently in use in glomerular research. Using three-dimensional structured illumination microscopy, the exact morphology of the podocyte filtration slit can be morphometrically analyzed and quantitatively compared across samples originating from animal models or human biopsies.

Key Messages

Several quantitative image analysis approaches and their potential influence on glomerular research and diagnostics are discussed. By improving not only optical resolution but also information content and turnaround time, super-resolution microscopy has the potential to expand the diagnosis of glomerular disease. Soon, these approaches could be introduced into glomerular disease diagnosis.

Unravelling cell migration: defining movement from the cell surface

Cell motility is essential for life and development. Unfortunately, cell migration is also linked to several pathological processes, such as cancer metastasis. Cells' ability to migrate relies on many actors. Cells change their migratory strategy based on their phenotype and the properties of the surrounding microenvironment. Cell migration is, therefore, an extremely complex phenomenon. Researchers have investigated cell motility for more than a century. Recent discoveries have uncovered some of the mysteries associated with the mechanisms involved in cell migration, such as intracellular signaling and cell mechanics. These findings involve different players, including transmembrane receptors, adhesive complexes, cytoskeletal components , the nucleus, and the extracellular matrix. This review aims to give a global overview of our current understanding of cell migration.

Three-dimensional tracking using a single-spot rotating point spread function created by a multiring spiral phase plate

SIGNIFICANCE

Three-dimensional (3D) imaging and object tracking is critical for medical and biological research and can be achieved by multifocal imaging with diffractive optical elements (DOEs) converting depth ( z ) information into a modification of the two-dimensional image. Physical insight into DOE designs will spur this expanding field.

AIM

To precisely track microscopic fluorescent objects in biological systems in 3D with a simple low-cost DOE system.

APPROACH

We designed a multiring spiral phase plate (SPP) generating a single-spot rotating point spread function (SS-RPSF) in a microscope. Our simple, analytically transparent design process uses Bessel beams to avoid rotational ambiguities and achieve a significant depth range. The SPP was inserted into the Nomarski prism slider of a standard microscope. Performance was evaluated using fluorescent beads and in live cells expressing a fluorescent chromatin marker.

RESULTS

Bead localization precision was < 25    nm in the transverse dimensions and ≤ 70    nm along the axial dimension over an axial range of 6    μ m . Higher axial precision ( ≤ 50    nm ) was achieved over a shallower focal depth of 2.9    μ m . 3D diffusion constants of chromatin matched expected values.

CONCLUSIONS

Precise 3D localization and tracking can be achieved with a SS-RPSF SPP in a standard microscope with minor modifications.

Enabling Spectrally Resolved Single-Molecule Localization Microscopy at High Emitter Densities

Single-molecule localization microscopy (SMLM) is a powerful super-resolution technique for elucidating structure and dynamics in the life- and material sciences. Simultaneously acquiring spectral information (spectrally resolved SMLM, sSMLM) has been hampered by several challenges: an increased complexity of the optical detection pathway, lower accessible emitter densities, and compromised spatio-spectral resolution. Here we present a single-component, low-cost implementation of sSMLM that addresses these challenges. Using a low-dispersion transmission grating positioned close to the image plane, the +1^stdiffraction order is minimally elongated and is analyzed using existing single-molecule localization algorithms. The distance between the 0^th and 1^st order provides accurate information on the spectral properties of individual emitters. This method enables a 5-fold higher emitter density while discriminating between fluorophores whose peak emissions are less than 15 nm apart. Our approach can find widespread use in single-molecule applications that rely on distinguishing spectrally different fluorophores under low photon conditions.

Deep learning in single-molecule imaging and analysis: recent advances and prospects

Single-molecule microscopy is advantageous in characterizing heterogeneous dynamics at the molecular level. However, there are several challenges that currently hinder the wide application of single molecule imaging in bio-chemical studies, including how to perform single-molecule measurements efficiently with minimal run-to-run variations, how to analyze weak single-molecule signals efficiently and accurately without the influence of human bias, and how to extract complete information about dynamics of interest from single-molecule data. As a new class of computer algorithms that simulate the human brain to extract data features, deep learning networks excel in task parallelism and model generalization, and are well-suited for handling nonlinear functions and extracting weak features, which provide a promising approach for single-molecule experiment automation and data processing. In this perspective, we will highlight recent advances in the application of deep learning to single-molecule studies, discuss how deep learning has been used to address the challenges in the field as well as the pitfalls of existing applications, and outline the directions for future development.