Learning Optimal Wavefront Shaping for Multi-Channel Imaging

Machine Learning for Single-Molecule Localization Microscopy: From Data Analysis to Quantification

Single-molecule localization microscopy (SMLM) is a versatile tool for realizing nanoscale imaging with visible light and providing unprecedented opportunities to observe bioprocesses. The integration of machine learning with SMLM enhances data analysis by improving efficiency and accuracy. This tutorial aims to provide a comprehensive overview of the data analysis process and theoretical aspects of SMLM, while also highlighting the typical applications of machine learning in this field. By leveraging advanced analytical techniques, SMLM is becoming a powerful quantitative analysis tool for biological research.

Intelligent Beam Optimization for Light-Sheet Fluorescence Microscopy through Deep Learning

Light-sheet fluorescence microscopy (LSFM) provides the benefit of optical sectioning coupled with rapid acquisition times, enabling high-resolution 3-dimensional imaging of large tissue-cleared samples. Inherent to LSFM, the quality of the imaging heavily relies on the characteristics of the illumination beam, which only illuminates a thin section of the sample. Therefore, substantial efforts are dedicated to identifying slender, nondiffracting beam profiles that yield uniform and high-contrast images. An ongoing debate concerns the identification of optimal illumination beams for different samples: Gaussian, Bessel, Airy patterns, and/or others. However, comparisons among different beam profiles are challenging as their optimization objectives are often different. Given that our large imaging datasets (approximately 0.5 TB of images per sample) are already analyzed using deep learning models, we envisioned a different approach to the problem by designing an illumination beam tailored to boost the performance of the deep learning model. We hypothesized that integrating the physical LSFM illumination model (after passing it through a variable phase mask) into the training of a cell detection network would achieve this goal. Here, we report that joint optimization continuously updates the phase mask and results in improved image quality for better cell detection. The efficacy of our method is demonstrated through both simulations and experiments that reveal substantial enhancements in imaging quality compared to the traditional Gaussian light sheet. We discuss how designing microscopy systems through a computational approach provides novel insights for advancing optical design that relies on deep learning models for the analysis of imaging datasets.

Depth-enhanced high-throughput microscopy by compact PSF engineering

High-throughput microscopy is vital for screening applications, where three-dimensional (3D) cellular models play a key role. However, due to defocus susceptibility, current 3D high-throughput microscopes require axial scanning, which lowers throughput and increases photobleaching and photodamage. Point spread function (PSF) engineering is an optical method that enables various 3D imaging capabilities, yet it has not been implemented in high-throughput microscopy due to the cumbersome optical extension it typically requires. Here we demonstrate compact PSF engineering in the objective lens, which allows us to enhance the imaging depth of field and, combined with deep learning, recover 3D information using single snapshots. Beyond the applications shown here, this work showcases the usefulness of high-throughput microscopy in obtaining training data for deep learning-based algorithms, applicable to a variety of microscopy modalities.

Universal inverse modeling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single-molecule localization, aberration correction and deconvolution. Here we present universal inverse modeling of point spread functions (uiPSF), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single-molecule localization microscopy (SMLM). Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system- or sample-specific characteristics, for example, the bead size, field- and depth- dependent aberrations, and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single-molecule blinking data.

Dynamic microenvironments shape nuclear organization and gene expression

Live imaging has revealed that the regulation of gene expression is largely driven by transient interactions. For example, many regulatory proteins bind chromatin for just seconds, and loop-like genomic contacts are rare and last only minutes. These discoveries have been difficult to reconcile with our canonical models that are predicated on stable and hierarchical interactions. Proteomic microenvironments that concentrate nuclear factors may explain how brief interactions can still mediate gene regulation by creating conditions where reactions occur more frequently. Here, we summarize new imaging technologies and recent discoveries implicating microenvironments as a potential driver of nuclear function. Finally, we propose that key properties of proteomic microenvironments, such as their size, enrichment, and lifetimes, are directly linked to regulatory function.

E2E-BPF microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution

Several image-based biomedical diagnoses require high-resolution imaging capabilities at large spatial scales. However, conventional microscopes exhibit an inherent trade-off between depth-of-field (DoF) and spatial resolution, and thus require objects to be refocused at each lateral location, which is time consuming. Here, we present a computational imaging platform, termed E2E-BPF microscope, which enables large-area, high-resolution imaging of large-scale objects without serial refocusing. This method involves a physics-incorporated, deep-learned design of binary phase filter (BPF) and jointly optimized deconvolution neural network, which altogether produces high-resolution, high-contrast images over extended depth ranges. We demonstrate the method through numerical simulations and experiments with fluorescently labeled beads, cells and tissue section, and present high-resolution imaging capability over a 15.5-fold larger DoF than the conventional microscope. Our method provides highly effective and scalable strategy for DoF-extended optical imaging system, and is expected to find numerous applications in rapid image-based diagnosis, optical vision, and metrology.

Universal inverse modelling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single molecule localization, aberration correction and deconvolution. Here we present uiPSF (universal inverse modelling of Point Spread Functions), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single molecule localization microscopy (SMLM). The resulting PSF model enables accurate 3D super-resolution imaging using SMLM. Additionally, uiPSF can be used to characterize and optimize a microscope system by quantifying the aberrations, including field-dependent aberrations, and resolutions. Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system or sample specific characteristics, e.g., the bead size, depth dependent aberrations and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single molecule blinking data.

Fluorescence-based super-resolution-microscopy strategies for chromatin studies

Super-resolution microscopy (SRM) is a prime tool to study chromatin organisation at near biomolecular resolution in the native cellular environment. With fluorescent labels DNA, chromatin-associated proteins and specific epigenetic states can be identified with high molecular specificity. The aim of this review is to introduce the field of diffraction-unlimited SRM to enable an informed selection of the most suitable SRM method for a specific chromatin-related research question. We will explain both diffraction-unlimited approaches (coordinate-targeted and stochastic-localisation-based) and list their characteristic spatio-temporal resolutions, live-cell compatibility, image-processing, and ability for multi-colour imaging. As the increase in resolution, compared to, e.g. confocal microscopy, leads to a central role of the sample quality, important considerations for sample preparation and concrete examples of labelling strategies applicable to chromatin research are discussed. To illustrate how SRM-based methods can significantly improve our understanding of chromatin functioning, and to serve as an inspiring starting point for future work, we conclude with examples of recent applications of SRM in chromatin research.

Double helix point spread function with variable spacing for precise 3D particle localization

To extend the axial depth of nanoscale 3D-localization microscopy, we propose here a splicing-type vortex singularities (SVS) phase mask, which has been meticulously optimized with a Fresnel approximation imaging inverse operation. The optimized SVS DH-PSF has proven to have high transfer function efficiency with adjustable performance in its axial range. The axial position of the particle was computed by using both the main lobes' spacing and the rotation angle, an improvement of the localization precision of the particle. Concretely, the proposed optimized SVS DH-PSF, with a smaller spatial extent, can effectively reduce the overlap of nanoparticle images and realize the 3D localization of multiple nanoparticles with small spacing, with respect to PSFs for large axial 3D localization. Finally, we successfully conducted extensive experiments on 3D localization for tracking dense nanoparticles at 8µm depth with a numerical aperture of 1.4, demonstrating its great potential.

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

Single-molecule localization microscopy in a typical wide-field setup has been widely used for investigating subcellular structures with super resolution; however, field-dependent aberrations restrict the field of view (FOV) to only tens of micrometers. Here, we present a deep-learning method for precise localization of spatially variant point emitters (FD-DeepLoc) over a large FOV covering the full chip of a modern sCMOS camera. Using a graphic processing unit-based vectorial point spread function (PSF) fitter, we can fast and accurately model the spatially variant PSF of a high numerical aperture objective in the entire FOV. Combined with deformable mirror-based optimal PSF engineering, we demonstrate high-accuracy three-dimensional single-molecule localization microscopy over a volume of ~180 × 180 × 5 μm³, allowing us to image mitochondria and nuclear pore complexes in entire cells in a single imaging cycle without hardware scanning; a 100-fold increase in throughput compared to the state of the art.

Co-designed metaoptoelectronic deep learning

A metaoptical system is co-designed with electronic hardware to implement deep learning image recognition. The optical convolution block includes a reflective metasurface to perform one layer of a deep neural network. The optical and digital components are jointly optimized to perform an image classification task attaining 65% accuracy, which is close to the 66% accuracy of a fully-digital network where the optical block is replaced by a digital convolution layer.

Six-Dimensional Single-Molecule Imaging with Isotropic Resolution using a Multi-View Reflector Microscope

Imaging both the positions and orientations of single fluorophores, termed single-molecule orientation-localisation microscopy, is a powerful tool to study biochemical processes. However, the limited photon budget associated with single-molecule fluorescence makes high-dimensional imaging with isotropic, nanoscale spatial resolution a formidable challenge. Here, we realise a radially and azimuthally polarized multi-view reflector (raMVR) microscope for the imaging of the 3D positions and 3D orientations of single molecules, with precision of 10.9 nm and 2.0° over a 1.5 μm depth range. The raMVR microscope achieves 6D super-resolution imaging of Nile red (NR) molecules transiently bound to lipid-coated spheres, accurately resolving their spherical morphology despite refractive-index mismatch. By observing the rotational dynamics of NR, raMVR images also resolve the infiltration of lipid membranes by amyloid-beta oligomers without covalent labelling. Finally, we demonstrate 6D imaging of cell membranes, where the orientations of specific fluorophores reveal heterogeneity in membrane fluidity. With its nearly isotropic 3D spatial resolution and orientation measurement precision, we expect the raMVR microscope to enable 6D imaging of molecular dynamics within biological and chemical systems with exceptional detail.

Monocular kilometer-scale passive ranging by point-spread function engineering

Standard imaging systems are designed for 2D representation of objects, while information about the third dimension remains implicit, as imaging-based distance estimation is a difficult challenge. Existing long-range distance estimation technologies mostly rely on active emission of signal, which as a subsystem, constitutes a significant portion of the complexity, size and cost of the active-ranging apparatus. Despite the appeal of alleviating the requirement for signal-emission, passive distance estimation methods are essentially nonexistent for ranges greater than a few hundreds of meters. Here, we present monocular long-range, telescope-based passive ranging, realized by integration of point-spread-function engineering into a telescope, extending the scale of point-spread-function engineering-based ranging to distances where it has never been tested before. We provide experimental demonstrations of the optical system in a variety of challenging imaging scenarios, including adversarial weather conditions, dynamic targets and scenes of diversified textures, at distances extending beyond 1.7 km. We conclude with brief quantification of the effect of atmospheric turbulence on estimation precision, which becomes a significant error source in long-range optical imaging.

Towards optimal point spread function design for resolving closely spaced emitters in three dimensions

The past decade has brought many innovations in optical design for 3D super-resolution imaging of point-like emitters, but these methods often focus on single-emitter localization precision as a performance metric. Here, we propose a simple heuristic for designing a point spread function (PSF) that allows for precise measurement of the distance between two emitters. We discover that there are two types of PSFs that achieve high performance for resolving emitters in 3D, as quantified by the Cramér-Rao bounds for estimating the separation between two closely spaced emitters. One PSF is very similar to the existing Tetrapod PSFs; the other is a rotating single-spot PSF, which we call the crescent PSF. The latter exhibits excellent performance for localizing single emitters throughout a 1-µm focal volume (localization precisions of 7.3 nm in x, 7.7 nm in y, and 18.3 nm in z using 1000 detected photons), and it distinguishes between one and two closely spaced emitters with superior accuracy (25-53% lower error rates than the best-performing Tetrapod PSF, averaged throughout a 1-µm focal volume). Our study provides additional insights into optimal strategies for encoding 3D spatial information into optical PSFs.

Diffractive optical system design by cascaded propagation

Modern design of complex optical systems relies heavily on computational tools. These frequently use geometrical optics as well as Fourier optics. Fourier optics is typically used for designing thin diffractive elements, placed in the system's aperture, generating a shift-invariant Point Spread Function (PSF). A major bottleneck in applying Fourier Optics in many cases of interest, e.g. when dealing with multiple, or out-of-aperture elements, comes from numerical complexity. In this work, we propose and implement an efficient and differentiable propagation model based on the Collins integral, which enables the optimization of diffractive optical systems with unprecedented design freedom using backpropagation. We demonstrate the applicability of our method, numerically and experimentally, by engineering shift-variant PSFs via thin plate elements placed in arbitrary planes inside complex imaging systems, performing cascaded optimization of multiple planes, and designing optimal machine-vision systems by deep learning.

Global fitting for high-accuracy multi-channel single-molecule localization

Multi-channel detection in single-molecule localization microscopy greatly increases information content for various biological applications. Here, we present globLoc, a graphics processing unit based global fitting algorithm with flexible PSF modeling and parameter sharing, to extract maximum information from multi-channel single molecule data. As signals in multi-channel data are highly correlated, globLoc links parameters such as 3D coordinates or photon counts across channels, improving localization precision and robustness. We show, both in simulations and experiments, that global fitting can substantially improve the 3D localization precision for biplane and 4Pi single-molecule localization microscopy and color assignment for ratiometric multicolor imaging.

Comparison of methods for end-to-end co-optimization of optical systems and image processing with commercial lens design software

We compare three different methods to co-optimize hybrid optical/digital imaging systems with a commercial lens design software: conventional optimization based on spot diagram minimization, optimization of a surrogate criterion based on a priori equalization of modulation transfer functions (MTFs), and minimization of the mean square error (MSE) between the ideal sharp image and the image restored by a unique deconvolution filter. To implement the latter method, we integrate - for the first time to our knowledge - MSE optimization to the software Synopsys CodeV. Taking as an application example the design of a Cooke triplet having good image quality everywhere in the field of view (FoV), we show that it is possible, by leveraging deconvolution during the optimization process, to adapt the spatial distribution of imaging performance to a prescribed goal. We also demonstrate the superiority of MSE co-optimization over the other methods, both in terms of quantitative and visual image quality.

Deep learning multi-shot 3D localization microscopy using hybrid optical-electronic computing

Current 3D localization microscopy approaches are fundamentally limited in their ability to image thick, densely labeled specimens. Here, we introduce a hybrid optical-electronic computing approach that jointly optimizes an optical encoder (a set of multiple, simultaneously imaged 3D point spread functions) and an electronic decoder (a neural-network-based localization algorithm) to optimize 3D localization performance under these conditions. With extensive simulations and biological experiments, we demonstrate that our deep-learning-based microscope achieves significantly higher 3D localization accuracy than existing approaches, especially in challenging scenarios with high molecular density over large depth ranges.

Single-Molecule Localization Microscopy of 3D Orientation and Anisotropic Wobble Using a Polarized Vortex Point Spread Function

Within condensed matter, single fluorophores are sensitive probes of their chemical environments, but it is difficult to use their limited photon budget to image precisely their positions, 3D orientations, and rotational diffusion simultaneously. We demonstrate the polarized vortex point spread function (PSF) for measuring these parameters, including characterizing the anisotropy of a molecule's wobble, simultaneously from a single image. Even when imaging dim emitters (∼500 photons detected), the polarized vortex PSF can obtain 12 nm localization precision, 4°-8° orientation precision, and 26° wobble precision. We use the vortex PSF to measure the emission anisotropy of fluorescent beads, the wobble dynamics of Nile red (NR) within supported lipid bilayers, and the distinct orientation signatures of NR in contact with amyloid-beta fibrils, oligomers, and tangles. The unparalleled sensitivity of the vortex PSF transforms single-molecule microscopes into nanoscale orientation imaging spectrometers, where the orientations and wobbles of individual probes reveal structures and organization of soft matter that are nearly impossible to perceive by using molecular positions alone.