Optimal point spread function design for 3D imaging

Depth-enhanced high-throughput microscopy by compact PSF engineering

High-throughput microscopy is vital for screening applications, where three-dimensional (3D) cellular models play a key role. However, due to defocus susceptibility, current 3D high-throughput microscopes require axial scanning, which lowers throughput and increases photobleaching and photodamage. Point spread function (PSF) engineering is an optical method that enables various 3D imaging capabilities, yet it has not been implemented in high-throughput microscopy due to the cumbersome optical extension it typically requires. Here we demonstrate compact PSF engineering in the objective lens, which allows us to enhance the imaging depth of field and, combined with deep learning, recover 3D information using single snapshots. Beyond the applications shown here, this work showcases the usefulness of high-throughput microscopy in obtaining training data for deep learning-based algorithms, applicable to a variety of microscopy modalities.

Universal inverse modeling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single-molecule localization, aberration correction and deconvolution. Here we present universal inverse modeling of point spread functions (uiPSF), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single-molecule localization microscopy (SMLM). Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system- or sample-specific characteristics, for example, the bead size, field- and depth- dependent aberrations, and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single-molecule blinking data.

Large-FOV 3D localization microscopy by spatially variant point spread function generation

Accurate characterization of the microscopic point spread function (PSF) is crucial for achieving high-performance localization microscopy (LM). Traditionally, LM assumes a spatially invariant PSF to simplify the modeling of the imaging system. However, for large fields of view (FOV) imaging, it becomes important to account for the spatially variant nature of the PSF. Here, we propose an accurate and fast principal components analysis-based field-dependent 3D PSF generator (PPG3D) and localizer for LM. Through simulations and experimental three-dimensional (3D) single-molecule localization microscopy (SMLM), we demonstrate the effectiveness of PPG3D, enabling super-resolution imaging of mitochondria and microtubules with high fidelity over a large FOV. A comparison of PPG3D with a shift-variant PSF generator for 3D LM reveals a threefold improvement in accuracy. Moreover, PPG3D is approximately 100 times faster than existing PSF generators, when used in image plane-based interpolation mode. Given its user-friendliness, we believe that PPG3D holds great potential for widespread application in SMLM and other imaging modalities.

High-density volumetric super-resolution microscopy

Volumetric super-resolution microscopy typically encodes the 3D position of single-molecule fluorescence into a 2D image by changing the shape of the point spread function (PSF) as a function of depth. However, the resulting large and complex PSF spatial footprints reduce biological throughput and applicability by requiring lower labeling densities to avoid overlapping fluorescent signals. We quantitatively compare the density dependence of single-molecule light field microscopy (SMLFM) to other 3D PSFs (astigmatism, double helix and tetrapod) showing that SMLFM enables an order-of-magnitude speed improvement compared to the double helix PSF by resolving overlapping emitters through parallax. We demonstrate this optical robustness experimentally with high accuracy ( > 99.2 ± 0.1%, 0.1 locs μm-2) and sensitivity ( > 86.6 ± 0.9%, 0.1 locs μm-2) through whole-cell (scan-free) imaging and tracking of single membrane proteins in live primary B cells. We also exemplify high-density volumetric imaging (0.15 locs μm-2) in dense cytosolic tubulin datasets.

Optimal sampling rate for 3D single molecule localization

Resolution of single molecule localization microscopy (SMLM) depends on the localization accuracy, which can be improved by utilizing engineered point spread functions (PSF) with delicate shapes. However, the intrinsic pixelation effect of the detector sensor will deteriorate PSFs under different sampling rates. The influence of the pixelation effect to the achieved 3D localization accuracy for different PSF shapes under different signal to background ratio (SBR) and pixel dependent readout noise has not been investigated in detail so far. In this work, we proposed a framework to characterize the 3D localization accuracy of pixelated PSF at different sampling rates. Four different PSFs (astigmatic PSF, double helix (DH) PSF, Tetrapod PSF and 4Pi PSF) were evaluated and the pixel size with optimal 3D localization performance were derived. This work provides a theoretical guide for the optimal design of sampling rate for 3D super resolution imaging.

E2E-BPF microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution

Several image-based biomedical diagnoses require high-resolution imaging capabilities at large spatial scales. However, conventional microscopes exhibit an inherent trade-off between depth-of-field (DoF) and spatial resolution, and thus require objects to be refocused at each lateral location, which is time consuming. Here, we present a computational imaging platform, termed E2E-BPF microscope, which enables large-area, high-resolution imaging of large-scale objects without serial refocusing. This method involves a physics-incorporated, deep-learned design of binary phase filter (BPF) and jointly optimized deconvolution neural network, which altogether produces high-resolution, high-contrast images over extended depth ranges. We demonstrate the method through numerical simulations and experiments with fluorescently labeled beads, cells and tissue section, and present high-resolution imaging capability over a 15.5-fold larger DoF than the conventional microscope. Our method provides highly effective and scalable strategy for DoF-extended optical imaging system, and is expected to find numerous applications in rapid image-based diagnosis, optical vision, and metrology.

Universal inverse modelling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single molecule localization, aberration correction and deconvolution. Here we present uiPSF (universal inverse modelling of Point Spread Functions), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single molecule localization microscopy (SMLM). The resulting PSF model enables accurate 3D super-resolution imaging using SMLM. Additionally, uiPSF can be used to characterize and optimize a microscope system by quantifying the aberrations, including field-dependent aberrations, and resolutions. Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system or sample specific characteristics, e.g., the bead size, depth dependent aberrations and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single molecule blinking data.

Fluorescence Microscopy: a statistics-optics perspective

Fundamental properties of light unavoidably impose features on images collected using fluorescence microscopes. Modeling these features is ever more important in quantitatively interpreting microscopy images collected at scales on par or smaller than light's wavelength. Here we review the optics responsible for generating fluorescent images, fluorophore properties, microscopy modalities leveraging properties of both light and fluorophores, in addition to the necessarily probabilistic modeling tools imposed by the stochastic nature of light and measurement.

Generalized aberration regularization in the single-lens computational imaging system design

The restoration algorithm can reduce the aberration blur in the single-lens computational imaging system. The single lens and the restoration algorithm should be co-designed to improve the imaging performance of the whole computational imaging system. By investigating the influence of aberrations on computational imaging quality, we find there are aberrations that cannot be removed by restoration algorithm, which cause the loss of high-resolution information. However, such aberrations are not constrained in existing co-design methods. To solve this problem, we propose the generalized aberration regularization (GAR), which mainly constrains the aberrations that reduce the modulation transfer function (MTF) at the high frequency. Experimental results demonstrate that our approach can significantly improve the quality of high-resolution information in the recovery image compared with the existing co-design methods.

Whole-cell multi-target single-molecule super-resolution imaging in 3D with microfluidics and a single-objective tilted light sheet

Multi-target single-molecule super-resolution fluorescence microscopy offers a powerful means of understanding the distributions and interplay between multiple subcellular structures at the nanoscale. However, single-molecule super-resolution imaging of whole mammalian cells is often hampered by high fluorescence background and slow acquisition speeds, especially when imaging multiple targets in 3D. In this work, we have mitigated these issues by developing a steerable, dithered, single-objective tilted light sheet for optical sectioning to reduce fluorescence background and a pipeline for 3D nanoprinting microfluidic systems for reflection of the light sheet into the sample and for efficient and automated solution exchange. By combining these innovations with PSF engineering for nanoscale localization of individual molecules in 3D, deep learning for analysis of overlapping emitters, active 3D stabilization for drift correction and long-term imaging, and Exchange-PAINT for sequential multi-target imaging without chromatic offsets, we demonstrate whole-cell multi-target 3D single-molecule super-resolution imaging with improved precision and imaging speed.

Simultaneous spectral differentiation of multiple fluorophores in super-resolution imaging using a glass phase plate

By engineering the point-spread function (PSF) of single molecules, different fluorophore species can be imaged simultaneously and distinguished by their unique PSF patterns. Here, we insert a silicon-dioxide phase plate at the Fourier plane of the detection path of a wide-field fluorescence microscope to produce distinguishable PSFs (X-PSFs) at different wavelengths. We demonstrate that the resulting PSFs can be localized spatially and spectrally using a maximum-likelihood estimation algorithm and can be utilized for hyper-spectral super-resolution microscopy of biological samples. We produced superresolution images of fixed U2OS cells using X-PSFs for dSTORM imaging with simultaneous illumination of up to three fluorophore species. The species were distinguished only by the PSF pattern. We achieved ∼21-nm lateral localization precision (FWHM) and ∼17-nm axial precision (FWHM) with an average of 1,800 - 3,500 photons per PSF and a background as high as 130 - 400 photons per pixel. The modified PSF distinguished fluorescent probes with ∼80 nm separation between spectral peaks.

Near index matching enables solid diffractive optical element fabrication via additive manufacturing

Diffractive optical elements (DOEs) have a wide range of applications in optics and photonics, thanks to their capability to perform complex wavefront shaping in a compact form. However, widespread applicability of DOEs is still limited, because existing fabrication methods are cumbersome and expensive. Here, we present a simple and cost-effective fabrication approach for solid, high-performance DOEs. The method is based on conjugating two nearly refractive index-matched solidifiable transparent materials. The index matching allows for extreme scaling up of the elements in the axial dimension, which enables simple fabrication of a template using commercially available 3D printing at tens-of-micrometer resolution. We demonstrated the approach by fabricating and using DOEs serving as microlens arrays, vortex plates, including for highly sensitive applications such as vector beam generation and super-resolution microscopy using MINSTED, and phase-masks for three-dimensional single-molecule localization microscopy. Beyond the advantage of making DOEs widely accessible by drastically simplifying their production, the method also overcomes difficulties faced by existing methods in fabricating highly complex elements, such as high-order vortex plates, and spectrum-encoding phase masks for microscopy.

Combining deep learning approaches and point spread function engineering for simultaneous 3D position and 3D orientation measurements of fluorescent single molecules

Point Spread Function (PSF) engineering is an effective method to increase the sensitivity of single-molecule fluorescence images to specific parameters. Classical phase mask optimization approaches have enabled the creation of new PSFs that can achieve, for example, localization precision of a few nanometers axially over a capture range of several microns with bright emitters. However, for complex high-dimensional optimization problems, classical approaches are difficult to implement and can be very time-consuming for computation. The advent of deep learning methods and their application to single-molecule imaging has provided a way to solve these problems. Here, we propose to combine PSF engineering and deep learning approaches to obtain both an optimized phase mask and a neural network structure to obtain the 3D position and 3D orientation of fixed fluorescent molecules. Our approach allows us to obtain an axial localization precision around 30 nanometers, as well as an orientation precision around 5 degrees for orientations and positions over a one micron depth range for a signal-to-noise ratio consistent with what is typical in single-molecule cellular imaging experiments.

On the equivalence of binary phase masks optimized for localization or detection in extended depth-of-field localization microscopy

Binary annular masks have recently been proposed to extend the depth of field (DoF) of single-molecule localization microscopy. A strategy for designing optimal masks has been introduced based on maximizing the emitter localization accuracy, expressed in terms of Fisher information, over a targeted DoF range. However, the complete post-processing pipeline to localize a single emitter consists of two successive steps: detection, where the regions containing emitters are determined, and localization, where the sub-pixel position of each detected emitter is estimated. Phase masks usually optimize only this second step. The presence of a phase mask also affecting detection, the purpose of this paper is to quantify and mitigate this effect. Using a rigorous framework built from a detection-oriented information theoretical criterion (Bhattacharyya distance), we demonstrate that in most cases of practical significance, annular binary phase masks maximizing Fisher information also maximize the detection probability. This result supports the common design practice consisting of optimizing a phase mask by maximizing Fisher information only.

Information optimization of laser scanning microscopes for real-time feedback-driven single particle tracking

Real-time feedback-driven single particle tracking (RT-FD-SPT) is a class of microscopy techniques that uses measurements of finite excitation/detection volume in a feedback control loop to actuate that volume and track with high spatio-temporal resolution a single particle moving in three dimensions. A variety of methods have been developed, each defined by a set of user-defined choices. Selection of those values is typically done through ad hoc, off-line tuning for the best perceived performance. Here we present a mathematical framework, based on optimization of the Fisher information, to select those parameters such that the best information is acquired for estimating parameters of interest, such as the location of the particle, specifics of the excitation beam such as its dimensions or peak intensity, or the background noise. For concreteness, we focus on tracking of a fluorescently-labeled particle and apply this framework to determine the optimal parameters for three existing fluorescence-based RT-FD-SPT techniques with respect to particle localization.

Frequency-domain compression imaging for extending the field of view of infrared thermometers

We propose a computational imaging technique for expanding the field of view of infrared thermometers. The contradiction between the field of view and the focal length has always been a chief problem for researchers, especially in infrared optical systems. Large-area infrared detectors are expensive and technically arduous to be manufactured, which enormously limits the performance of the infrared optical system. On the other hand, the extensive use of infrared thermometers in COVID-19 has created a considerable demand for infrared optical systems. Therefore, improving the performance of infrared optical systems and increasing the utilization of infrared detectors is vital. This work proposes a multi-channel frequency-domain compression imaging method based on point spread function (PSF) engineering. Compared with conventional compressed sensing, the submitted method images once without an intermediate image plane. Furthermore, phase encoding is used without loss of illumination of the image surface. These facts can significantly reduce the volume of the optical system and improve the energy efficiency of the compressed imaging system. Therefore, its application in COVID-19 is of great value. We design a dual-channel frequency-domain compression imaging system to verify the proposed method's feasibility. Then, the wavefront coded PSF and optical transfer function (OTF) are used, and the two-step iterative shrinkage/thresholding (TWIST) algorithm is used to restore the image to get the final result. This compression imaging method provides a new idea for the large field of view monitoring systems, especially in infrared optical systems.

Double helix point spread function with variable spacing for precise 3D particle localization

To extend the axial depth of nanoscale 3D-localization microscopy, we propose here a splicing-type vortex singularities (SVS) phase mask, which has been meticulously optimized with a Fresnel approximation imaging inverse operation. The optimized SVS DH-PSF has proven to have high transfer function efficiency with adjustable performance in its axial range. The axial position of the particle was computed by using both the main lobes' spacing and the rotation angle, an improvement of the localization precision of the particle. Concretely, the proposed optimized SVS DH-PSF, with a smaller spatial extent, can effectively reduce the overlap of nanoparticle images and realize the 3D localization of multiple nanoparticles with small spacing, with respect to PSFs for large axial 3D localization. Finally, we successfully conducted extensive experiments on 3D localization for tracking dense nanoparticles at 8µm depth with a numerical aperture of 1.4, demonstrating its great potential.

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

Single-molecule localization microscopy in a typical wide-field setup has been widely used for investigating subcellular structures with super resolution; however, field-dependent aberrations restrict the field of view (FOV) to only tens of micrometers. Here, we present a deep-learning method for precise localization of spatially variant point emitters (FD-DeepLoc) over a large FOV covering the full chip of a modern sCMOS camera. Using a graphic processing unit-based vectorial point spread function (PSF) fitter, we can fast and accurately model the spatially variant PSF of a high numerical aperture objective in the entire FOV. Combined with deformable mirror-based optimal PSF engineering, we demonstrate high-accuracy three-dimensional single-molecule localization microscopy over a volume of ~180 × 180 × 5 μm³, allowing us to image mitochondria and nuclear pore complexes in entire cells in a single imaging cycle without hardware scanning; a 100-fold increase in throughput compared to the state of the art.

Versatile all-digital transport-of-intensity based wavefront sensor and adaptive optics using a DMD

Measuring and correcting wavefront aberrations is an important process in a wide variety of disciplines, from ophthalmology, laser cutting, and astronomy to free-space communication and microscopy, and always relies on measuring intensities to infer phase. One approach is to use the transport-of-intensity as a means for phase retrieval, exploiting the connection between observed energy flow in optical fields and their wavefronts. Here we present a simple scheme, using a digital micro-mirror device (DMD), to perform angular spectrum propagation and extract the wavefront of optical fields at various wavelengths, dynamically, with high resolution and tuneable sensitivity. We verify the capability of our approach by extracting common Zernike aberrations, turbulent phase screens, and lens phases under static and dynamic conditions at multiple wavelengths and polarizations. We use this setup for adaptive optics, correcting distortion using a second DMD to apply conjugate phase modulation. We observed effective wavefront recovery under a variety of conditions which allowed for convenient real-time adaptive correction in a compact arrangement. Our approach provides an all-digital system that is versatile, cheap, fast, accurate, broadband and polarization invariant.

Direct generation of spatially entangled qudits using quantum nonlinear optical holography

Nonlinear holography shapes the amplitude and phase of generated new harmonics using nonlinear processes. Classical nonlinear holography influenced many fields in optics, from information storage, demultiplexing of spatial information, and all-optical control of accelerating beams. Here, we extend the concept of nonlinear holography to the quantum regime. We directly shape the spatial quantum correlations of entangled photon pairs in two-dimensional patterned nonlinear photonic crystals using spontaneous parametric down conversion, without any pump shaping. The generated signal-idler pair obeys a parity conservation law that is governed by the nonlinear crystal. Furthermore, the quantum states exhibit quantum correlations and violate the Clauser-Horne-Shimony-Holt inequality, thus enabling entanglement-based quantum key distribution. Our demonstration paves the way for controllable on-chip quantum optics schemes using the high-dimensional spatial degree of freedom.

Single-molecule imaging in the primary cilium

The primary cilium is an important signaling organelle critical for normal development and tissue homeostasis. Its small dimensions and complexity necessitate advanced imaging approaches to uncover the molecular mechanisms behind its function. Here, we outline how single-molecule fluorescence microscopy can be used for tracking molecular dynamics and interactions and for super-resolution imaging of nanoscale structures in the primary cilium. Specifically, we describe in detail how to capture and quantify the 2D dynamics of individual transmembrane proteins PTCH1 and SMO and how to map the 3D nanoscale distributions of the inversin compartment proteins INVS, ANKS6, and NPHP3. This protocol can, with minor modifications, be adapted for studies of other proteins and cell lines to further elucidate the structure and function of the primary cilium at the molecular level.

Monocular metasurface camera for passive single-shot 4D imaging

It is a grand challenge for an imaging system to simultaneously obtain multi-dimensional light field information, such as depth and polarization, of a scene for the accurate perception of the physical world. However, such a task would conventionally require bulky optical components, time-domain multiplexing, and active laser illumination. Here, we experimentally demonstrate a compact monocular camera equipped with a single-layer metalens that can capture a 4D image, including 2D all-in-focus intensity, depth, and polarization of a target scene in a single shot under ambient illumination conditions. The metalens is optimized to have a conjugate pair of polarization-decoupled rotating single-helix point-spread functions that are strongly dependent on the depth of the target object. Combined with a straightforward, physically interpretable image retrieval algorithm, the camera can simultaneously perform high-accuracy depth sensing and high-fidelity polarization imaging over an extended depth of field for both static and dynamic scenes in both indoor and outdoor environments. Such a compact multi-dimensional imaging system could enable new applications in diverse areas ranging from machine vision to microscopy.

Fisher information and the Cramér-Rao lower bound in single-pixel localization microscopy with spatiotemporally modulated illumination

Single-pixel imaging, the concept that an image can be captured via a single-pixel detector, is a cost-effective yet powerful technique to reduce data acquisition duration without sacrificing image resolution when properly structured illumination patterns are introduced. Normally, the image reconstruction process is subject to the diffraction limit. Here, we study the possibility of exploiting the information contained in the illumination patterns to enable a form of single-pixel localization microscopy (SPLM) for super-resolution. This concept is inspired by coherent holographic image reconstruction by phase transfer (CHIRPT) microscopy. CHIRPT microscopy is a single-pixel imaging technique that uses structured illumination that is spatiotemporally modulated (STM) so that a unique temporal modulation pattern is imparted to each point within a large illumination volume. The fluorescent light emitted by molecules contains the same temporal modulations as the illumination patterns at the locations of the molecules. By recording a portion of the total emitted fluorescent power, the signal may be numerically processed to form an image. Unique temporal modulation patterns that excite fluorescent probes at each point can also be used to localize individual molecules by matching their particular temporal light emission patterns to the measured temporal signal. This paper evaluates the feasibility of SPLM with STM illuminations used in and inspired by CHIRPT microscopy via the information content its data carry about the emitter location(s). More specifically, we provide the mathematical formalism of Fisher information (FI) and the Cramér-Rao lower bound (CRLB) associated with the location parameters of the emitter(s). The FI and CRLB are then numerically evaluated under different experimental assumptions to assess the effects of experimental parameters on localization precision. Last, we compare the single-pixel CRLB to that from camera-based single-molecule localization microscopy in the localization of a single emitter. We show that SPLM has several distinguishing characteristics that provide certain advantages, such as relatively constant CRLB over a very large illumination volume and improved CRLB for 3D localization due to the information coupling introduced by simultaneous modulations of the transverse axes.

Three-dimensional wide-field fluorescence microscopy for transcranial mapping of cortical microcirculation

Wide-field fluorescence imaging is an indispensable tool for studying large-scale biodynamics. Limited space-bandwidth product and strong light diffusion make conventional implementations incapable of high-resolution mapping of fluorescence biodistribution in three dimensions. We introduce a volumetric wide-field fluorescence microscopy based on optical astigmatism combined with fluorescence source localization, covering 5.6×5.6×0.6 mm³ imaging volume. Two alternative configurations are proposed exploiting multifocal illumination or sparse localization of point emitters, which are herein seamlessly integrated in one system. We demonstrate real-time volumetric mapping of the murine cortical microcirculation at capillary resolution without employing cranial windows, thus simultaneously delivering quantitative perfusion information across both brain hemispheres. Morphological and functional changes of cerebral vascular networks are further investigated after an acute ischemic stroke, enabling cortex-wide observation of concurrent collateral recruitment events occurring on a sub-second scale. The reported technique thus offers a wealth of unmatched possibilities for non- or minimally invasive imaging of biodynamics across scales.

Three-dimensional tracking using a single-spot rotating point spread function created by a multiring spiral phase plate

SIGNIFICANCE

Three-dimensional (3D) imaging and object tracking is critical for medical and biological research and can be achieved by multifocal imaging with diffractive optical elements (DOEs) converting depth ( z ) information into a modification of the two-dimensional image. Physical insight into DOE designs will spur this expanding field.

AIM

To precisely track microscopic fluorescent objects in biological systems in 3D with a simple low-cost DOE system.

APPROACH

We designed a multiring spiral phase plate (SPP) generating a single-spot rotating point spread function (SS-RPSF) in a microscope. Our simple, analytically transparent design process uses Bessel beams to avoid rotational ambiguities and achieve a significant depth range. The SPP was inserted into the Nomarski prism slider of a standard microscope. Performance was evaluated using fluorescent beads and in live cells expressing a fluorescent chromatin marker.

RESULTS

Bead localization precision was < 25    nm in the transverse dimensions and ≤ 70    nm along the axial dimension over an axial range of 6    μ m . Higher axial precision ( ≤ 50    nm ) was achieved over a shallower focal depth of 2.9    μ m . 3D diffusion constants of chromatin matched expected values.

CONCLUSIONS

Precise 3D localization and tracking can be achieved with a SS-RPSF SPP in a standard microscope with minor modifications.

Monocular kilometer-scale passive ranging by point-spread function engineering

Standard imaging systems are designed for 2D representation of objects, while information about the third dimension remains implicit, as imaging-based distance estimation is a difficult challenge. Existing long-range distance estimation technologies mostly rely on active emission of signal, which as a subsystem, constitutes a significant portion of the complexity, size and cost of the active-ranging apparatus. Despite the appeal of alleviating the requirement for signal-emission, passive distance estimation methods are essentially nonexistent for ranges greater than a few hundreds of meters. Here, we present monocular long-range, telescope-based passive ranging, realized by integration of point-spread-function engineering into a telescope, extending the scale of point-spread-function engineering-based ranging to distances where it has never been tested before. We provide experimental demonstrations of the optical system in a variety of challenging imaging scenarios, including adversarial weather conditions, dynamic targets and scenes of diversified textures, at distances extending beyond 1.7 km. We conclude with brief quantification of the effect of atmospheric turbulence on estimation precision, which becomes a significant error source in long-range optical imaging.

Technologies Enabling Single-Molecule Super-Resolution Imaging of mRNA

The transient nature of RNA has rendered it one of the more difficult biological targets for imaging. This difficulty stems both from the physical properties of RNA as well as the temporal constraints associated therewith. These concerns are further complicated by the difficulty in imaging endogenous RNA within a cell that has been transfected with a target sequence. These concerns, combined with traditional concerns associated with super-resolution light microscopy has made the imaging of this critical target difficult. Recent advances have provided researchers the tools to image endogenous RNA in live cells at both the cellular and single-molecule level. Here, we review techniques used for labeling and imaging RNA with special emphases on various labeling methods and a virtual 3D super-resolution imaging technique.

Investigating deep optics model representation in affecting resolved all-in-focus image quality and depth estimation fidelity

The end-to-end (E2E) optimization of optics and image processing, dubbed deep optics, has renewed the state-of-the-art in various computer vision tasks. However, specifying the proper model representation or parameterization of the optical elements remains elusive. This article comprehensibly investigates three modeling hypotheses of the phase coded-aperture imaging under a representative context of deep optics, joint all-in-focus (AiF) imaging and monocular depth estimation (MDE). Specifically, we analyze the respective trade-off of these models and provide insights into relevant domain-specific requirements, explore the connection between the spatial feature of the point spread function (PSF) and the performance trade-off between the AiF and MDE tasks, and discuss the model sensitivity to possible fabrication errors. This study provides new prospects for future deep optics designs, particularly those aiming for AiF and/or MDE.

Single-shot optical surface profiling using extended depth of field 3D microscopy

The measurement of three-dimensional samples at high speed is essential for many applications, either due to the requirement for measuring samples that change fast over time, or due to the requirement of reducing the scanning time, and therefore inspection cost, in industrial environments. Conventionally, the measurement of surface topographies at high resolution typically requires an axial scanning of the sample. We report the implementation of a technique able to reconstruct surface topographies at high resolution, only from the acquisition of a single camera shot, dropping the need to perform an axial scan. A system prototype is reported and assessed as an ultra-fast optical surface profiler. We propose robust calibration and operation methods and algorithms to reconstruct surface topographies of optically-rough samples, and compare the experimental results with a reference confocal optical profiler.

Axial accuracy in localization microscopy with 3D point spread function engineering

Single-molecule localization microscopy has developed into a widely used technique to overcome the diffraction limit and enables 3D localization of single-emitters with nanometer precision. A widely used method to enable 3D encoding is to use a cylindrical lens or a phase mask to engineer the point spread function (PSF). The performance of these PSFs is often assessed by comparing the precision they achieve, ignoring accuracy. Nonetheless, accurate localization is required in many applications, such as multi-plane imaging, measuring and modelling of physical processes based on volumetric data, and 3D particle averaging. However, there are PSF model mismatches in the localization schemes due to how reference PSFs are obtained, look-up-tables are created, or spots are fitted. Currently there is little insight in how these model mismatches give rise to systematic axial localization errors, how large these errors are, and how to mitigate them. In this theoretical and simulation work we use a vector PSF model, which incorporates super-critical angle fluorescence (SAF) and the appropriate aplanatic correction factor, to analyze the errors in z-localization. We introduce theory for defining the focal plane in SAF conditions and analyze the predicted axial errors for an astigmatic PSF, double-helix PSF, and saddle-point PSF. These simulations indicate that the absolute axial biases can be as large as 140 nm, 250 nm, and 120 nm for the astigmatic, saddle-point, and double-helix PSF respectively, with relative errors of more than 50%. Finally, we discuss potential experimental methods to verify these findings and propose a workflow to mitigate these effects.

Diffractive optical system design by cascaded propagation

Modern design of complex optical systems relies heavily on computational tools. These frequently use geometrical optics as well as Fourier optics. Fourier optics is typically used for designing thin diffractive elements, placed in the system's aperture, generating a shift-invariant Point Spread Function (PSF). A major bottleneck in applying Fourier Optics in many cases of interest, e.g. when dealing with multiple, or out-of-aperture elements, comes from numerical complexity. In this work, we propose and implement an efficient and differentiable propagation model based on the Collins integral, which enables the optimization of diffractive optical systems with unprecedented design freedom using backpropagation. We demonstrate the applicability of our method, numerically and experimentally, by engineering shift-variant PSFs via thin plate elements placed in arbitrary planes inside complex imaging systems, performing cascaded optimization of multiple planes, and designing optimal machine-vision systems by deep learning.

Real-Time Feedback-Driven Single-Particle Tracking: A Survey and Perspective

Real-time feedback-driven single-particle tracking (RT-FD-SPT) is a class of techniques in the field of single-particle tracking that uses feedback control to keep a particle of interest in a detection volume. These methods provide high spatiotemporal resolution on particle dynamics and allow for concurrent spectroscopic measurements. This review article begins with a survey of existing techniques and of applications where RT-FD-SPT has played an important role. Each of the core components of RT-FD-SPT are systematically discussed in order to develop an understanding of the trade-offs that must be made in algorithm design and to create a clear picture of the important differences, advantages, and drawbacks of existing approaches. These components are feedback tracking and control, ranging from simple proportional-integral-derivative control to advanced nonlinear techniques, estimation to determine particle location from the measured data, including both online and offline algorithms, and techniques for calibrating and characterizing different RT-FD-SPT methods. Then a collection of metrics for RT-FD-SPT is introduced to help guide experimentalists in selecting a method for their particular application and to help reveal where there are gaps in the techniques that represent opportunities for further development. Finally, this review is concluded with a discussion on future perspectives in the field.

Accurate single image depth detection using multiple rotating point spread functions

In this article we present the simulation and experimental implementation of a camera-based sensor with low object-space numerical aperture that is capable of measuring the distance of multiple object points with an accuracy of 8.51 µm over a range of 20 mm. The overall measurement volume is 70 mm × 50 mm × 20 mm. The lens of the camera is upgraded with a diffractive optical element (DOE) which fulfills two tasks: replication of the single object point to a predefined pattern of K spots in the image plane and adding a vortex point spread function (PSF), whose shape and rotation is sensitive to defocus. We analyze the parameters of the spiral phase mask and discuss the depth reconstruction approach. By applying the depth reconstruction to each of the K replications and averaging the results, we experimentally show that the accuracy of the reconstructed depth signal can be improved by a factor of up to 3 by the replication approach. This replication method (also called multipoint method) not only improves accuracy of depth reconstruction but also of lateral position measurement. Therefore, the presented concept can be used as a single camera 3D position sensor for multiple points with high lateral as well as depth resolution.

Deformable mirror based optimal PSF engineering for 3D super-resolution imaging

Point spread function (PSF) engineering is an important technique to encode the properties (e.g., 3D positions, color, and orientation) of a single molecule in the shape of the PSF, often with the help of a programmable phase modulator. A deformable mirror (DM) is currently the most widely used phase modulator for fluorescence detection as it shows negligible photon loss. However, it relies on careful calibration for precise wavefront control. Therefore, design of an optimal PSF not only relies on the theoretical calculation of the maximum information content, but also the physical behavior of the phase modulator, which is often ignored during the optimization process. Here, we develop a framework for PSF engineering which could generate a device specific optimal PSF for 3D super-resolution imaging using a DM. We use our method to generate two types of PSFs with depths of field comparable to the widely used astigmatism and tetrapod PSFs, respectively. We demonstrate the superior performance of the DM specific optimal PSF over the conventional astigmatism and tetrapod PSF both theoretically and experimentally.

Dipole-spread-function engineering for simultaneously measuring the 3D orientations and 3D positions of fluorescent molecules

Interactions between biomolecules are characterized by both where they occur and how they are organized, e.g., the alignment of lipid molecules to form a membrane. However, spatial and angular information are mixed within the image of a fluorescent molecule-the microscope's dipole-spread function (DSF). We demonstrate the pixOL algorithm for simultaneously optimizing all pixels within a phase mask to produce an engineered Green's tensor-the dipole extension of point-spread function engineering. The pixOL DSF achieves optimal precision for measuring simultaneously the 3D orientation and 3D location of a single molecule, i.e., 4.1° orientation, 0.44 sr wobble angle, 23.2 nm lateral localization, and 19.5 nm axial localization precisions in simulations over a 700-nm depth range using 2500 detected photons. The pixOL microscope accurately and precisely resolves the 3D positions and 3D orientations of Nile red within a spherical supported lipid bilayer, resolving both membrane defects and differences in cholesterol concentration in 6 dimensions.

3D Single Molecule Super-Resolution Microscopy of Whole Nuclear Lamina

Single molecule (SM) super-resolution microscopies bypass the diffraction limit of conventional optical techniques and provide excellent spatial resolutions in the tens of nanometers without overly complex microscope hardware. SM imaging using optical astigmatism is an efficient strategy for visualizing subcellular features in 3D with a z-range of up to ∼1 µm per acquisition. This approach however, places high demands on fluorophore brightness and photoswitching resilience meaning that imaging entire cell volumes in 3D using SM super-resolution remains challenging. Here we employ SM astigmatism together with multiplane acquisition to visualize the whole nuclear lamina of COS-7 and T cells in 3D. Nuclear lamina provides structural support to the nuclear envelope and participates in vital nuclear functions including internuclear transport, chromatin organization and gene regulation. Its position at the periphery of the nucleus provides a visible reference of the nuclear boundary and can be used to quantify the spatial distribution of intranuclear components such as histone modifications and transcription factors. We found Alexa Fluor 647, a popular photoswitchable fluorophore, remained viable for over an hour of continuous high laser power exposure, and provided sufficient brightness detectable up to 8 µm deep into a cell, allowing us to capture the entire nuclear lamina in 3D. Our approach provides sufficient super-resolution detail of nuclear lamina morphology to enable quantification of overall nuclear dimensions and local membrane features.

Learning to autofocus in whole slide imaging via physics-guided deep cascade networks

Whole slide imaging (WSI), is an essential technology for digital pathology, the performance of which is primarily affected by the autofocusing process. Conventional autofocusing methods either are time-consuming or require additional hardware and thus are not compatible with the current WSI systems. In this paper, we propose an effective learning-based method for autofocusing in WSI, which can realize accurate autofocusing at high speed as well as without any optical hardware modifications. Our method is inspired by an observation that sample images captured by WSI have distinctive characteristics with respect to positive / negative defocus offsets, due to the asymmetry effect of optical aberrations. Based on this physical knowledge, we develop novel deep cascade networks to enhance autofocusing quality. Specifically, to handle the effect of optical aberrations, a binary classification network is tailored to distinguish sample images with positive / negative defocus. As such, samples within the same category share similar characteristics. It facilitates the followed refocusing network, which is designed to learn the mapping between the defocus image and defocus distance. Experimental results demonstrate that our method achieves superior autofocusing performance to other related methods.

Modulated pattern scanning microscopy

In confocal microscopy, the effective optical transfer function (OTF_eff) with Gaussian plane wave illumination covers very few high-frequency components, which prohibits further improvement of the resolution. We propose modulated pattern scanning microscopy (MPSM) to achieve super-resolution imaging. In MPSM, the phase of the illumination beam is modulated to reassign the OTF_eff in the Fourier domain. The phase mask is designed using an optimization algorithm to obtain the fluorescence emission pattern with rich high-frequency components. Then, the postprocessing algorithms are adapted to retrieve the super-resolved images from the modulated recordings. Simulation and experiment demonstrate that MPSM increases the resolution approximately 1.3 times better than confocal microscopy. Compared with conventional deconvolution, MPSM exhibits a higher signal-to-noise ratio.

Self-rotating beam in the free space propagation

We introduce a class of self-rotating beams whose intensity profile tends to self-rotate and self-bend in the free space propagation. The feature of the self-rotating beams is acceleration in the three-dimensional (3D) space. The acceleration dynamics of the self-rotating beams is controllable. Furthermore, multiple self-rotating beams can be generated by a combined diffractive optical element (DOE) simultaneously. Such a beam can be viewed as evolution of a vortex beam by changing the exponential constant of phase. We have generated this beam successfully in the experiment and observed the expected phenomenon, which is basically consistent with the result of the numerical simulation. Our results may provide new insight into the self-rotating beam and extend potential applications in optical imaging.

Raw Data to Results: A Hands-On Introduction and Overview of Computational Analysis for Single-Molecule Localization Microscopy

Single-molecule localization microscopy (SMLM) is an advanced microscopy method that uses the blinking of fluorescent molecules to determine the position of these molecules with a resolution below the diffraction limit (∼5-40 nm). While SMLM imaging itself is becoming more popular, the computational analysis surrounding the technique is still a specialized area and often remains a "black box" for experimental researchers. Here, we provide an introduction to the required computational analysis of SMLM imaging, post-processing and typical data analysis. Importantly, user-friendly, ready-to-use and well-documented code in Python and MATLAB with exemplary data is provided as an interactive experience for the reader, as well as a starting point for further analysis. Our code is supplemented by descriptions of the computational problems and their implementation. We discuss the state of the art in computational methods and software suites used in SMLM imaging and data analysis. Finally, we give an outlook into further computational challenges in the field.

Exploring Cell Surface-Nanopillar Interactions with 3D Super-Resolution Microscopy

Plasma membrane topography has been shown to strongly influence the behavior of many cellular processes such as clathrin-mediated endocytosis, actin rearrangements, and others. Recent studies have used three-dimensional (3D) nanostructures such as nanopillars to imprint well-defined membrane curvatures (the "nano-bio interface"). In these studies, proteins and their interactions were probed by two-dimensional fluorescence microscopy. However, the low resolution and limited axial detail of such methods are not optimal to determine the relative spatial position and distribution of proteins along a 100 nm-diameter object, which is below the optical diffraction limit. Here, we introduce a general method to explore the nanoscale distribution of proteins at the nano-bio interface with 10-20 nm precision using 3D single-molecule super-resolution (SR) localization microscopy. This is achieved by combining a silicone-oil immersion objective and 3D double-helix point spread function microscopy. We carefully adjust the objective to minimize spherical aberrations between quartz nanopillars and the cell. To validate the 3D SR method, we imaged the 3D shape of surface-labeled nanopillars and compared the results with electron microscopy measurements. Turning to transmembrane-anchored labels in cells, the high quality 3D SR reconstructions reveal the membrane tightly wrapping around the nanopillars. Interestingly, the cytoplasmic protein AP-2 involved in clathrin-mediated endocytosis accumulates along the nanopillar above a specific threshold of 1/R (the reciprocal of the radius) membrane curvature. Finally, we observe that AP-2 and actin preferentially accumulate at positive Gaussian curvature near the pillar caps. Our results establish a general method to investigate the nanoscale distribution of proteins at the nano-bio interface using 3D SR microscopy.

On the validity domain of maximum likelihood estimators for depth-of-field extension in single-molecule localization microscopy

Localization microscopy approaches with enhanced depth-of-field (EDoF) are commonly optimized using the Cramér-Rao bound (CRB) as a criterion. It is widely believed that the CRB can be attained in practice by using the maximum-likelihood estimator (MLE). This is, however, an approximation, of which we define in this paper the precise domain of validity. Exploring a wide range of settings and noise levels, we show that the MLE is efficient when the signal-to-noise ratio (SNR) is such that the localization standard deviation of a single molecule is less than 20 nm. Thus, our results provide an explicit and quantitative validity boundary for the use of the MLE in EDoF localization microscopy setups optimized with the CRB.

Holographic single-image depth reconstruction

In this article a camera-based single-image sensor is presented, that is able to measure the distance of multiple object points. The experimental results show an accuracy of 8,51 μm within a depth range of 20 mm. The sensor consists of a camera, whose lens is upgraded with a diffractive optical element (DOE). It fulfils two tasks: adding a vortex point spread function (PSF) and replication of the vortex PSFs to a predefined pattern of K spots. Both, shape and rotation of the vortex PSF is sensitive to defocus. By applying the depth reconstruction to each of the K replications and averaging the results, we experimentally show, that the reconstruction of the depth signal can be improved by a factor of up to 3.

Phase optimization algorithm for 3D particle localization with large axial depth

We propose an optimization algorithm based on Fresnel approximation (FA) imaging to optimize an extended-axial-depth point spread function (PSF) for 3D particle localization. The transfer function efficiency of the PSF is improved by repeatedly imposing constraints in the object plane, the spatial domain, and the Fourier domain. During the iterative calculation, the effective photon number or Cramer-Rao lower bound is used as the termination condition of the iteration. The algorithm allows flexible adjustment of the peak intensity ratio of the two main lobes. Moreover, the transfer function efficiency can be balanced by increasing the weight of the modulation function of the expected PSF at each axial position. The twin-Airy (TA) PSF optimized by the FA optimization algorithm does not require complex post-processing, whereas post-processing is an essential step for the unoptimized TA-PSF. The optimization algorithm is significant for extended-axial-depth PSFs used for 3D particle localization, as it improves localization precision and temporal resolution.

Sub-millisecond Translational and Orientational Dynamics of a Freely Moving Single Nanoprobe

This paper presents a new experiment with which we are able to measure the 3D translational motion of a single particle at 10 μs time resolution and with ∼10 nm spatial resolution while at the same time determining the 3D orientation of the same single particle with 250 μs time resolution. These high time resolutions are ∼40 times greater than previous simultaneous measurements of 3D position and 3D orientation. Detailed numerical simulations and experiments are used to demonstrate that the technique can measure 3D orientation at the shot-noise limit. The microscope is also able to simultaneously measure the length or width (with the other assumed) of the plasmonic nanorods used here in situ and nondestructively, which should yield a greater understanding of the underlying dynamics. This technique should be applicable to a broad range of problems where environments which change in space and time may perturb physical and chemical dynamics.

Deep learning multi-shot 3D localization microscopy using hybrid optical-electronic computing

Current 3D localization microscopy approaches are fundamentally limited in their ability to image thick, densely labeled specimens. Here, we introduce a hybrid optical-electronic computing approach that jointly optimizes an optical encoder (a set of multiple, simultaneously imaged 3D point spread functions) and an electronic decoder (a neural-network-based localization algorithm) to optimize 3D localization performance under these conditions. With extensive simulations and biological experiments, we demonstrate that our deep-learning-based microscope achieves significantly higher 3D localization accuracy than existing approaches, especially in challenging scenarios with high molecular density over large depth ranges.

Three-dimensional diffusion coefficient measurement by a large depth-of-field rotating point spread function

A prominent challenge in single-molecule localization microscopy is the real-time, fast, and accurate localization of nano-objects moving in three-dimensional (3D) samples. A well-established method for 3D single-molecule localization is the double-helix pointspread-function (DH-PSF) engineering, which uses additional optical elements to make the PSF exhibit different rotation angles with different nanoparticle depths. However, the compact main lobe size, effective detection depth, and precise conversion between rotation angle and depth are necessary, posing challenges to the DH-PSF generation method. Here we generate a more compact DH-PSF using Fresnel-zone-based spiral phases, and the pure phase mask achieves high transmission efficiency. The final generated DH-PSFs have a linear rotation rate at each axial position, showing a more accurate rotation angle and depth conversion. The Cramer-Rao lower limit calculation results show that the axial depth of DH-PSF extends to ∼11µm with an axial localization precision of ∼45nm at 3000 photons and average background noise of 15. We measured the diffusion coefficient of nanospheres in different concentrations of glycerol using the generated DH-PSF. The measured results are within 6% error from the theoretical values, indicating the superior performance of the DH-PSF for nanoparticle diffusion coefficient measurements.

Imaging through a scattering medium: the Fisher information and the generalized Abbe limit

Enhanced-resolution imaging in complex scattering media is revisited from a parameter estimation perspective. A suitably defined Fisher information is shown to offer useful insights into the limiting precision of parameter estimation in a scattering environment and, hence, into the limiting spatial resolution that can be achieved in imaging-through-scattering settings. The Fisher information that defines this resolution limit via the Cramér-Rao lower bound is shown to scale with the number of adaptively controlled space-time modes of the probe field, suggesting a physically intuitive generalization of the Abbe limit to the spatial resolution attainable for complex scattering systems. In a conventional, direct-imaging microscopy setting, this bound is shown to converge to the canonical Abbe limit.

Light Sheet Illumination for 3D Single-Molecule Super-Resolution Imaging of Neuronal Synapses

The function of the neuronal synapse depends on the dynamics and interactions of individual molecules at the nanoscale. With the development of single-molecule super-resolution microscopy over the last decades, researchers now have a powerful and versatile imaging tool for mapping the molecular mechanisms behind the biological function. However, imaging of thicker samples, such as mammalian cells and tissue, in all three dimensions is still challenging due to increased fluorescence background and imaging volumes. The combination of single-molecule imaging with light sheet illumination is an emerging approach that allows for imaging of biological samples with reduced fluorescence background, photobleaching, and photodamage. In this review, we first present a brief overview of light sheet illumination and previous super-resolution techniques used for imaging of neurons and synapses. We then provide an in-depth technical review of the fundamental concepts and the current state of the art in the fields of three-dimensional single-molecule tracking and super-resolution imaging with light sheet illumination. We review how light sheet illumination can improve single-molecule tracking and super-resolution imaging in individual neurons and synapses, and we discuss emerging perspectives and new innovations that have the potential to enable and improve single-molecule imaging in brain tissue.

Fisher information for optimal planning of X-ray diffraction experiments

Fisher information is a powerful mathematical tool suitable for quantification of data `informativity' and optimization of the experimental setup and measurement conditions. Here, it is applied to X-ray diffraction and an informational approach to choosing the optimal measurement configuration is proposed. The core idea is maximization of the information which can be extracted from the measured data set by the selected analysis technique, over the sets of accessible reflections and measurement geometries. The developed approach is applied to high-resolution X-ray diffraction measurements and microstructure analysis of multilayer samples, and its efficiency and consistency are demonstrated with the results of more straightforward Monte Carlo simulations.