Optimal point spread function design for 3D imaging

Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet

Background fluorescence, especially when it exhibits undesired spatial features, is a primary factor for reduced image quality in optical microscopy. Structured background is particularly detrimental when analyzing single-molecule images for 3-dimensional localization microscopy or single-molecule tracking. Here, we introduce BGnet, a deep neural network with a U-net-type architecture, as a general method to rapidly estimate the background underlying the image of a point source with excellent accuracy, even when point-spread function (PSF) engineering is in use to create complex PSF shapes. We trained BGnet to extract the background from images of various PSFs and show that the identification is accurate for a wide range of different interfering background structures constructed from many spatial frequencies. Furthermore, we demonstrate that the obtained background-corrected PSF images, for both simulated and experimental data, lead to a substantial improvement in localization precision. Finally, we verify that structured background estimation with BGnet results in higher quality of superresolution reconstructions of biological structures.

Accurate phase retrieval of complex 3D point spread functions with deep residual neural networks

Phase retrieval, i.e., the reconstruction of phase information from intensity information, is a central problem in many optical systems. Imaging the emission from a point source such as a single molecule is one example. Here, we demonstrate that a deep residual neural net is able to quickly and accurately extract the hidden phase for general point spread functions (PSFs) formed by Zernike-type phase modulations. Five slices of the 3D PSF at different focal positions within a two micrometer range around the focus are sufficient to retrieve the first six orders of Zernike coefficients.

Fluorescence imaging with tailored light

Fluorescence microscopy has long been a valuable tool for biological and medical imaging. Control of optical parameters such as the amplitude, phase, polarization and propagation angle of light gives fluorescence imaging great capabilities ranging from super-resolution imaging to long-term real-time observation of living organisms. In this review, we discuss current fluorescence imaging techniques in terms of the use of tailored or structured light for the sample illumination and fluorescence detection, providing a clear overview of their working principles and capabilities.

Multi-line fluorescence scanning microscope for multi-focal imaging with unlimited field of view

Confocal scanning microscopy is the de facto standard modality for fluorescence imaging. Point scanning, however, leads to a limited throughput and makes the technique unsuitable for fast multi-focal scanning over large areas. We propose an architecture for multi-focal fluorescence imaging that is scalable to large area imaging. The design is based on the concept of line scanning with continuous 'push broom' scanning. Instead of a line sensor, we use an area sensor that is tilted with respect to the optical axis to acquire image data from multiple depths inside the sample simultaneously. A multi-line illumination where the lines span a plane conjugate to the tilted sensor is created by means of a diffractive optics design, implemented on a spatial light modulator. In particular, we describe a design that uses higher order astigmatism to generate focal lines of substantially constant peak intensity along the lines. The proposed method is suitable for fast 3D image acquisition with unlimited field of view, it requires no moving components except for the sample scanning stage, and provides intrinsic alignment of the simultaneously scanned focal slices. As proof of concept, we have scanned 9 focal slices simultaneously over an area of 36 mm² at 0.29 µm pixel size in object space. The projected ultimate throughput that can be realized with the proposed architecture is in excess of 100 Mpixel/s.

Joint 3D localization and classification of space debris using a multispectral rotating point spread function

We consider the problem of joint three-dimensional (3D) localization and material classification of unresolved space debris using a multispectral rotating point spread function (RPSF). The use of RPSF allows one to estimate the 3D locations of point sources from their rotated images acquired by a single 2D sensor array, since the amount of rotation of each source image about its x, y location depends on its axial distance z. Using multispectral images, with one RPSF per spectral band, we are able not only to localize the 3D positions of the space debris but also classify their material composition. We propose a three-stage method for achieving joint localization and classification. In stage 1, we adopt an optimization scheme for localization in which the spectral signature of each material is assumed to be uniform, which significantly improves efficiency and yields better localization results than possible with a single spectral band. In stage 2, we estimate the spectral signature and refine the localization result via an alternating approach. We process classification in the final stage. Both Poisson noise and Gaussian noise models are considered, and the implementation of each is discussed. Numerical tests using multispectral data from NASA show the efficiency of our three-stage approach and illustrate the improvement of point source localization and spectral classification from using multiple bands over a single band.

Nanoscale Resolution 3D Snapshot Particle Tracking by Multifocal Microscopy

Accurate, precise, and rapid particle tracking in three dimensions remains a challenge; yet, its achievement will significantly enhance our understanding of living systems. We developed a multifocal microscopy (MFM) that allows snapshot acquisition of the imaging data, and an associated image processing approach, that together allow simultaneous 3D tracking of many fluorescent particles with nanoscale resolution. The 3D tracking was validated by measuring a known trajectory of a fluorescent bead with an axial accuracy of 19 nm through an image depth (axial range) of 3 μm and 4 nm precision of axial localization through an image depth of 4 μm. A second test obtained a uniform axial probability distribution and Brownian dynamics of beads diffusing in solution. We also validated the MFM approach by imaging fluorescent beads immobilized in gels and comparing the 3D localizations to their "ground truth" positions obtained from a confocal microscopy z-stack of finely spaced images. Finally, we applied our MFM and image processing approach to obtain 3D trajectories of insulin granules in pseudoislets of MIN6 cells to demonstrate its compatibility with complex biological systems. Our study demonstrates that multifocal microscopy allows rapid (video rate) and simultaneous 3D tracking of many "particles" with nanoscale accuracy and precision in a wide range of systems, including over spatial scales relevant to whole live cells.

3D Tracking-Free Approach for Obtaining 3D Super-Resolution Information in Rotationally Symmetric Biostructures

Currently, it is highly desirable but still challenging to obtain high-resolution (<50 nm) three-dimensional (3D) super-resolution information on structures in fixed specimens as well as for dynamic processes in live cells. Here we introduce a simple approach, without using 3D super-resolution microscopy or real-time 3D particle tracking, to estimate 3D sub-diffraction-limited structural or dynamic information in rotationally symmetric biostructures. This is a postlocalization analysis that transforms 2D super-resolution images or 2D single-molecule localization distributions into their corresponding 3D spatial probability distributions on the basis of prior known structural knowledge. This analysis is ideal in cases where the ultrastructure of a cellular structure is known but the substructural localization of a particular (usually mobile) protein is not. The method has been successfully applied to achieve 3D structural and functional sub-diffraction-limited information for 25-300 nm subcellular organelles that meet the rotational symmetry requirement, such as nuclear pore complex, primary cilium, and microtubule. In this Article, we will provide comprehensive analyses of this method by using experimental data and computational simulations. Finally, open source code of the 2D to 3D transformation algorithm (MATLAB) and simulations (Python) have also been developed.

Combining 3D single molecule localization strategies for reproducible bioimaging

Here, we present a 3D localization-based super-resolution technique providing a slowly varying localization precision over a 1 μm range with precisions down to 15 nm. The axial localization is performed through a combination of point spread function (PSF) shaping and supercritical angle fluorescence (SAF), which yields absolute axial information. Using a dual-view scheme, the axial detection is decoupled from the lateral detection and optimized independently to provide a weakly anisotropic 3D resolution over the imaging range. This method can be readily implemented on most homemade PSF shaping setups and provides drift-free, tilt-insensitive and achromatic results. Its insensitivity to these unavoidable experimental biases is especially adapted for multicolor 3D super-resolution microscopy, as we demonstrate by imaging cell cytoskeleton, living bacteria membranes and axon periodic submembrane scaffolds. We further illustrate the interest of the technique for biological multicolor imaging over a several-μm range by direct merging of multiple acquisitions at different depths.

3D single molecule localization microscopy suffers from several experimental biases that degrade the resolution or localization precision. Here the authors present a dual-view detection scheme combining supercritical angle fluorescence and astigmatic imaging to obtain precise and unbiased 3D super resolution images.

Computational correction of spatially variant optical aberrations in 3D single-molecule localization microscopy

3D single-molecule localization microscopy relies on fitting the shape of point-spread-functions (PSFs) recorded on a wide-field detector. However, optical aberrations distort those shapes, which compromises the accuracy and precision of single-molecule localization microscopy. Here, we employ a computational phase retrieval based on a vectorial PSF model to quantify the spatial variance of optical aberrations in a two-channel ultrawide-field single-molecule localization microscope. The use of a spatially variant PSF model enables accurate and precise emitter localization in x-, y- and z-directions throughout the entire field of view.

A Light-Field Metasurface for High-Resolution Single-Particle Tracking

Three-dimensional (3D) single-particle tracking (SPT) is a key tool for studying dynamic processes in the life sciences. However, conventional optical elements utilizing light fields impose an inherent trade-off between lateral and axial resolution, preventing SPT with high spatiotemporal resolution across an extended volume. We overcome the typical loss in spatial resolution that accompanies light-field-based approaches to obtain 3D information by placing a standard microscope coverslip patterned with a multifunctional, light-field metasurface on a specimen. This approach enables an otherwise unmodified microscope to gather 3D information at an enhanced spatial resolution. We demonstrate simultaneous tracking of multiple fluorescent particles within a large 0.5 × 0.5 × 0.3 mm³ volume using a standard epi-fluorescent microscope with submicron lateral and micron-level axial resolution.

Resolving Cytosolic Diffusive States in Bacteria by Single-Molecule Tracking

The trajectory of a single protein in the cytosol of a living cell contains information about its molecular interactions in its native environment. However, it has remained challenging to accurately resolve and characterize the diffusive states that can manifest in the cytosol using analytical approaches based on simplifying assumptions. Here, we show that multiple intracellular diffusive states can be successfully resolved if sufficient single-molecule trajectory information is available to generate well-sampled distributions of experimental measurements and if experimental biases are taken into account during data analysis. To address the inherent experimental biases in camera-based and MINFLUX-based single-molecule tracking, we use an empirical data analysis framework based on Monte Carlo simulations of confined Brownian motion. This framework is general and adaptable to arbitrary cell geometries and data acquisition parameters employed in two-dimensional or three-dimensional single-molecule tracking. We show that, in addition to determining the diffusion coefficients and populations of prevalent diffusive states, the timescales of diffusive state switching can be determined by stepwise increasing the time window of averaging over subsequent single-molecule displacements. Time-averaged diffusion analysis of single-molecule tracking data may thus provide quantitative insights into binding and unbinding reactions among rapidly diffusing molecules that are integral for cellular functions.

Multicolor localization microscopy and point-spread-function engineering by deep learning

Deep learning has become an extremely effective tool for image classification and image restoration problems. Here, we apply deep learning to microscopy and demonstrate how neural networks can exploit the chromatic dependence of the point-spread function to classify the colors of single emitters imaged on a grayscale camera. While existing localization microscopy methods for spectral classification require additional optical elements in the emission path, e.g., spectral filters, prisms, or phase masks, our neural net correctly identifies static and mobile emitters with high efficiency using a standard, unmodified single-channel configuration. Furthermore, we show how deep learning can be used to design new phase-modulating elements that, when implemented into the imaging path, result in further improved color differentiation between species, including simultaneously differentiating four species in a single image.

Ring resonator based polarization diversity WDM receiver

A ring resonator based 4 channel wavelength division multiplexing (WDM) receiver with polarization diversity is demonstrated at 10 Gb/s per channel. By forming a waveguide loop between the two output ports of a polarization splitter-rotator (PSR), the input signals in the quasi-transverse-electric (quasi-TE) and the quasi-transverse-magnetic (quasi-TM) polarizations can be demultiplexed by the same set of ring resonator filters, thus reducing the number of required channel control circuits by half compared to methods which process the two polarizations individually. Large signal measurement results indicate that the design can tolerate a signal delay of up to 30% of the unit interval (UI) between the two polarizations, which implies that compensating for manufacturing variability with optical delay lines on chip is not necessary for a robust operation. The inter-channel crosstalk is found negligible down to 0.4nm (50 GHz) spacing, at which point the adjacent channel isolation is 17 dB, proving the design's compatibility for dense WDM application.

Generalized method to design phase masks for 3D super-resolution microscopy

Point spread function (PSF) engineering by phase modulation is a novel approach to three-dimensional (3D) super-resolution microscopy, with different point spread functions being proposed for specific applications. It is often not easy to achieve the desired shape of engineered point spread functions because it is challenging to determine the correct phase mask. Additionally, a phase mask can either encode 3D space information or additional time information, but not both simultaneously. A robust algorithm for recovering a phase mask to generate arbitrary point spread functions is needed. In this work, a generalized phase mask design method is introduced by performing an optimization. A stochastic gradient descent algorithm and a Gauss-Newton algorithm are developed and compared for their ability to recover the phase masks for previously reported point spread functions. The new Gauss-Newton algorithm converges to a minimum at much higher speeds. This algorithm is used to design a novel stretching-lobe phase mask to encode temporal and 3D spatial information simultaneously. The stretching-lobe phase mask and other masks are fabricated in-house for proof-of-concept using multi-level light lithography and an optimized commercially sourced stretching-lobe phase mask (PM) is validated experimentally to encode 3D spatial and temporal information. The algorithms' generalizability is further demonstrated by generating a phase mask that comprises four different letters at different depths.

The finite state projection based Fisher information matrix approach to estimate information and optimize single-cell experiments

Modern optical imaging experiments not only measure single-cell and single-molecule dynamics with high precision, but they can also perturb the cellular environment in myriad controlled and novel settings. Techniques, such as single-molecule fluorescence in-situ hybridization, microfluidics, and optogenetics, have opened the door to a large number of potential experiments, which begs the question of how to choose the best possible experiment. The Fisher information matrix (FIM) estimates how well potential experiments will constrain model parameters and can be used to design optimal experiments. Here, we introduce the finite state projection (FSP) based FIM, which uses the formalism of the chemical master equation to derive and compute the FIM. The FSP-FIM makes no assumptions about the distribution shapes of single-cell data, and it does not require precise measurements of higher order moments of such distributions. We validate the FSP-FIM against well-known Fisher information results for the simple case of constitutive gene expression. We then use numerical simulations to demonstrate the use of the FSP-FIM to optimize the timing of single-cell experiments with more complex, non-Gaussian fluctuations. We validate optimal simulated experiments determined using the FSP-FIM with Monte-Carlo approaches and contrast these to experiment designs chosen by traditional analyses that assume Gaussian fluctuations or use the central limit theorem. By systematically designing experiments to use all of the measurable fluctuations, our method enables a key step to improve co-design of experiments and quantitative models.

Revealing Nanoscale Morphology of the Primary Cilium Using Super-Resolution Fluorescence Microscopy

Super-resolution (SR) microscopy has been used to observe structural details beyond the diffraction limit of ∼250 nm in a variety of biological and materials systems. By combining this imaging technique with both computer-vision algorithms and topological methods, we reveal and quantify the nanoscale morphology of the primary cilium, a tiny tubular cellular structure (∼2-6 μm long and 200-300 nm in diameter). The cilium in mammalian cells protrudes out of the plasma membrane and is important in many signaling processes related to cellular differentiation and disease. After tagging individual ciliary transmembrane proteins, specifically Smoothened, with single fluorescent labels in fixed cells, we use three-dimensional (3D) single-molecule SR microscopy to determine their positions with a precision of 10-25 nm. We gain a dense, pointillistic reconstruction of the surfaces of many cilia, revealing large heterogeneity in membrane shape. A Poisson surface reconstruction algorithm generates a fine surface mesh, allowing us to characterize the presence of deformations by quantifying the surface curvature. Upon impairment of intracellular cargo transport machinery by genetic knockout or small-molecule treatment of cells, our quantitative curvature analysis shows significant morphological differences not visible by conventional fluorescence microscopy techniques. Furthermore, using a complementary SR technique, two-color, two-dimensional stimulated emission depletion microscopy, we find that the cytoskeleton in the cilium, the axoneme, also exhibits abnormal morphology in the mutant cells, similar to our 3D results on the Smoothened-measured ciliary surface. Our work combines 3D SR microscopy and computational tools to quantitatively characterize morphological changes of the primary cilium under different treatments and uses stimulated emission depletion to discover correlated changes in the underlying structure. This approach can be useful for studying other biological or nanoscale structures of interest.

Minimizing Structural Bias in Single-Molecule Super-Resolution Microscopy

Single-molecule localization microscopy (SMLM) depends on sequential detection and localization of individual molecular blinking events. Due to the stochasticity of single-molecule blinking and the desire to improve SMLM’s temporal resolution, algorithms capable of analyzing frames with a high density (HD) of active molecules, or molecules whose images overlap, are a prerequisite for accurate location measurements. Thus far, HD algorithms are evaluated using scalar metrics, such as root-mean-square error, that fail to quantify the structure of errors caused by the structure of the sample. Here, we show that the spatial distribution of localization errors within super-resolved images of biological structures are vectorial in nature, leading to systematic structural biases that severely degrade image resolution. We further demonstrate that the shape of the microscope’s point-spread function (PSF) fundamentally affects the characteristics of imaging artifacts. We built a Robust Statistical Estimation algorithm (RoSE) to minimize these biases for arbitrary structures and PSFs. RoSE accomplishes this minimization by estimating the likelihood of blinking events to localize molecules more accurately and eliminate false localizations. Using RoSE, we measure the distance between crossing microtubules, quantify the morphology of and separation between vesicles, and obtain robust recovery using diverse 3D PSFs with unmatched accuracy compared to state-of-the-art algorithms.

Fundamental Precision Bounds for Three-Dimensional Optical Localization Microscopy with Poisson Statistics

Point source localization is a problem of persistent interest in optical imaging. In particular, a number of widely used biological microscopy techniques rely on precise three-dimensional localization of single fluorophores. As emitter depth localization is more challenging than lateral localization, considerable effort has been spent on engineering the response of the microscope in a way that reveals increased depth information. Here, we prove the (sub)optimality of these approaches by deriving and comparing to the measurement-independent quantum Cramér-Rao bound (QCRB). We show that existing methods for depth localization with single-objective collection exceed the QCRB, and we gain insight into the bound by proposing an interferometer arrangement that approaches it. We also show that for light collection with two opposed objectives, an established interferometric technique globally reaches the QCRB in all three dimensions simultaneously, and so this represents an interesting case study from the point of view of quantum multiparameter estimation.

ZOLA-3D allows flexible 3D localization microscopy over an adjustable axial range

Single molecule localization microscopy can generate 3D super-resolution images without scanning by leveraging the axial variations of normal or engineered point spread functions (PSF). Successful implementation of these approaches for extended axial ranges remains, however, challenging. We present Zernike Optimized Localization Approach in 3D (ZOLA-3D), an easy-to-use computational and optical solution that achieves optimal resolution over a tunable axial range. We use ZOLA-3D to demonstrate 3D super-resolution imaging of mitochondria, nuclear pores and microtubules in entire nuclei or cells up to ~5 μm deep.

3D single-molecule localization is limited in depth and often requires using a wide range of point spread functions (PSFs). Here the authors present an optical solution featuring a deformable mirror to generate different PSFs and easy-to-use software for super-resolution imaging up to 5 µm deep.

Observing DNA in live cells

The structural organization and dynamics of DNA are known to be of paramount importance in countless cellular processes, but capturing these events poses a unique challenge. Fluorescence microscopy is well suited for these live-cell investigations, but requires attaching fluorescent labels to the species under investigation. Over the past several decades, a suite of techniques have been developed for labeling and imaging DNA, each with various advantages and drawbacks. Here, we provide an overview of the labeling and imaging tools currently available for visualizing DNA in live cells, and discuss their suitability for various applications.

Light sheet approaches for improved precision in 3D localization-based super-resolution imaging in mammalian cells [Invited]

The development of imaging techniques beyond the diffraction limit has paved the way for detailed studies of nanostructures and molecular mechanisms in biological systems. Imaging thicker samples, such as mammalian cells and tissue, in all three dimensions, is challenging due to increased background and volumes to image. Light sheet illumination is a method that allows for selective irradiation of the image plane, and its inherent optical sectioning capability allows for imaging of biological samples with reduced background, photobleaching, and photodamage. In this review, we discuss the advantage of combining single-molecule imaging with light sheet illumination. We begin by describing the principles of single-molecule localization microscopy and of light sheet illumination. Finally, we present examples of designs that successfully have married single-molecule super-resolution imaging with light sheet illumination for improved precision in mammalian cells.

Nonlinear Focal Modulation Microscopy

We demonstrate nonlinear focal modulation microscopy (NFOMM) to achieve superresolution imaging. Traditional approaches to superresolution that utilize point scanning often rely on spatially reducing the size of the emission pattern by directly narrowing (e.g., through minimizing the detection pinhole in Airyscan, Zeiss) or indirectly peeling its outer profiles [e.g., through depleting the outer emission region in stimulated emission depletion (STED) microscopy]. We show that an alternative conceptualization that focuses on maximizing the optical system's frequency shifting ability offers advantages in further improving resolution while reducing system complexity. In NFOMM, a spatial light modulator and a suitably intense laser illumination are used to implement nonlinear focal-field modulation to achieve a transverse spatial resolution of ∼60  nm (∼λ/10). We show that NFOMM is comparable with STED microscopy and suitable for fundamental biology studies, as evidenced in imaging nuclear pore complexes, tubulin and vimentin in Vero cells. Since NFOMM is readily implemented as an add-on module to a laser-scanning microscope, we anticipate wide utility of this new imaging technique.

Real-time 3D single-molecule localization using experimental point spread functions

We present a real-time fitter for 3D single-molecule localization microscopy using experimental point spread functions (PSFs) that achieves minimal uncertainty in 3D on any microscope and is compatible with any PSF engineering approach. We used this method to image cellular structures and attained unprecedented image quality for astigmatic PSFs. The fitter compensates for most optical aberrations and makes accurate 3D super-resolution microscopy broadly accessible, even on standard microscopes without dedicated 3D optics.

High precision wavefront control in point spread function engineering for single emitter localization

Point spread function (PSF) engineering is used in single emitter localization to measure the emitter position in 3D and possibly other parameters such as the emission color or dipole orientation as well. Advanced PSF models such as spline fits to experimental PSFs or the vectorial PSF model can be used in the corresponding localization algorithms in order to model the intricate spot shape and deformations correctly. The complexity of the optical architecture and fit model makes PSF engineering approaches particularly sensitive to optical aberrations. Here, we present a calibration and alignment protocol for fluorescence microscopes equipped with a spatial light modulator (SLM) with the goal of establishing a wavefront error well below the diffraction limit for optimum application of complex engineered PSFs. We achieve high-precision wavefront control, to a level below 20 mλ wavefront aberration over a 30 minute time window after the calibration procedure, using a separate light path for calibrating the pixel-to-pixel variations of the SLM, and alignment of the SLM with respect to the optical axis and Fourier plane within 3 μm (x/y) and 100 μm (z) error. Aberrations are retrieved from a fit of the vectorial PSF model to a bead z-stack and compensated with a residual wavefront error comparable to the error of the SLM calibration step. This well-calibrated and corrected setup makes it possible to create complex '3D+λ' PSFs that fit very well to the vectorial PSF model. Proof-of-principle bead experiments show precisions below 10 nm in x, y, and λ, and below 20 nm in z over an axial range of 1 μm with 2000 signal photons and 12 background photons.

Computational localization microscopy with extended axial range

A new single-aperture 3D particle-localization and tracking technique is presented that demonstrates an increase in depth range by more than an order of magnitude without compromising optical resolution and throughput. We exploit the extended depth range and depth-dependent translation of an Airy-beam PSF for 3D localization over an extended volume in a single snapshot. The technique is applicable to all bright-field and fluorescence modalities for particle localization and tracking, ranging from super-resolution microscopy through to the tracking of fluorescent beads and endogenous particles within cells. We demonstrate and validate its application to real-time 3D velocity imaging of fluid flow in capillaries using fluorescent tracer beads. An axial localization precision of 50 nm was obtained over a depth range of 120μm using a 0.4NA, 20× microscope objective. We believe this to be the highest ratio of axial range-to-precision reported to date.

Tilted Light Sheet Microscopy with 3D Point Spread Functions for Single-Molecule Super-Resolution Imaging in Mammalian Cells

To obtain a complete picture of subcellular nanostructures, cells must be imaged with high resolution in all three dimensions (3D). Here, we present tilted light sheet microscopy with 3D point spread functions (TILT3D), an imaging platform that combines a novel, tilted light sheet illumination strategy with engineered long axial range point spread functions (PSFs) for low-background, 3D super localization of single molecules as well as 3D super-resolution imaging in thick cells. TILT3D is built upon a standard inverted microscope and has minimal custom parts. The axial positions of the single molecules are encoded in the shape of the PSF rather than in the position or thickness of the light sheet, and the light sheet can therefore be formed using simple optics. The result is flexible and user-friendly 3D super-resolution imaging with tens of nm localization precision throughout thick mammalian cells. We validated TILT3D for 3D super-resolution imaging in mammalian cells by imaging mitochondria and the full nuclear lamina using the double-helix PSF for single-molecule detection and the recently developed Tetrapod PSF for fiducial bead tracking and live axial drift correction. We envision TILT3D to become an important tool not only for 3D super-resolution imaging, but also for live whole-cell single-particle and single-molecule tracking.

Single-molecule techniques in biophysics: a review of the progress in methods and applications

Single-molecule biophysics has transformed our understanding of biology, but also of the physics of life. More exotic than simple soft matter, biomatter lives far from thermal equilibrium, covering multiple lengths from the nanoscale of single molecules to up to several orders of magnitude higher in cells, tissues and organisms. Biomolecules are often characterized by underlying instability: multiple metastable free energy states exist, separated by levels of just a few multiples of the thermal energy scale k _B T, where k _B is the Boltzmann constant and T absolute temperature, implying complex inter-conversion kinetics in the relatively hot, wet environment of active biological matter. A key benefit of single-molecule biophysics techniques is their ability to probe heterogeneity of free energy states across a molecular population, too challenging in general for conventional ensemble average approaches. Parallel developments in experimental and computational techniques have catalysed the birth of multiplexed, correlative techniques to tackle previously intractable biological questions. Experimentally, progress has been driven by improvements in sensitivity and speed of detectors, and the stability and efficiency of light sources, probes and microfluidics. We discuss the motivation and requirements for these recent experiments, including the underpinning mathematics. These methods are broadly divided into tools which detect molecules and those which manipulate them. For the former we discuss the progress of super-resolution microscopy, transformative for addressing many longstanding questions in the life sciences, and for the latter we include progress in 'force spectroscopy' techniques that mechanically perturb molecules. We also consider in silico progress of single-molecule computational physics, and how simulation and experimentation may be drawn together to give a more complete understanding. Increasingly, combinatorial techniques are now used, including correlative atomic force microscopy and fluorescence imaging, to probe questions closer to native physiological behaviour. We identify the trade-offs, limitations and applications of these techniques, and discuss exciting new directions.

3D single-molecule super-resolution microscopy with a tilted light sheet

Tilted light sheet microscopy with 3D point spread functions (TILT3D) combines a novel, tilted light sheet illumination strategy with long axial range point spread functions (PSFs) for low-background, 3D super-localization of single molecules as well as 3D super-resolution imaging in thick cells. Because the axial positions of the single emitters are encoded in the shape of each single-molecule image rather than in the position or thickness of the light sheet, the light sheet need not be extremely thin. TILT3D is built upon a standard inverted microscope and has minimal custom parts. The result is simple and flexible 3D super-resolution imaging with tens of nm localization precision throughout thick mammalian cells. We validate TILT3D for 3D super-resolution imaging in mammalian cells by imaging mitochondria and the full nuclear lamina using the double-helix PSF for single-molecule detection and the recently developed tetrapod PSFs for fiducial bead tracking and live axial drift correction.

Dual-Color and 3D Super-Resolution Microscopy of Multi-protein Assemblies

Breaking the resolution limit of conventional microscopy by super-resolution microscopy (SRM) led to many new biological insights into protein assemblies at the nanoscale. Here we provide detailed protocols for single-molecule localization microscopy (SMLM) to image the structure of a protein complex. As examples, we show how to acquire single- and dual-color super-resolution images of the nuclear pore complex (NPC) and dual-color 3D data on actin and paxillin in focal adhesions.

Spatial and spectral imaging of point-spread functions using a spatial light modulator

We develop a point-spread function (PSF) engineering approach to imaging the spatial and spectral information of molecular emissions using a spatial light modulator (SLM). We show that a dispersive grating pattern imposed upon the emission reveals spectral information. We also propose a deconvolution model that allows the decoupling of the spectral and 3D spatial information in engineered PSFs. The work is readily applicable to single-molecule measurements and fluorescent microscopy.

Observation of live chromatin dynamics in cells via 3D localization microscopy using Tetrapod point spread functions

We report the observation of chromatin dynamics in living budding yeast (Saccharomyces cerevisiae) cells, in three-dimensions (3D). Using dual color localization microscopy and employing a Tetrapod point spread function, we analyze the spatio-temporal dynamics of two fluorescently labeled DNA loci surrounding the GAL locus. From the measured trajectories, we obtain different dynamical characteristics in terms of inter-loci distance and temporal variance; when the GAL locus is activated, the 3D inter-loci distance and temporal variance increase compared to the inactive state. These changes are visible in spite of the large thermally- and biologically-driven heterogeneity in the relative motion of the two loci. Our observations are consistent with current euchromatin vs. heterochromatin models.

Point-spread function engineering enhances digital Fourier microscopy

While numerous optical methods exist to probe the dynamics of biological or complex fluid samples, in recent years digital Fourier microscopy techniques such as differential dynamic microscopy have emerged as ways to efficiently combine elements of imaging and scattering methods. Here, we demonstrate, through experiments and simulations, how point-spread function engineering can be used to extend the reach of differential dynamic microscopy.

Incoherent digital holograms acquired by interferenceless coded aperture correlation holography system without refractive lenses

We present a lensless, interferenceless incoherent digital holography technique based on the principle of coded aperture correlation holography. The acquired digital hologram by this technique contains a three-dimensional image of some observed scene. Light diffracted by a point object (pinhole) is modulated using a random-like coded phase mask (CPM) and the intensity pattern is recorded and composed as a point spread hologram (PSH). A library of PSHs is created using the same CPM by moving the pinhole to all possible axial locations. Intensity diffracted through the same CPM from an object placed within the axial limits of the PSH library is recorded by a digital camera. The recorded intensity this time is composed as the object hologram. The image of the object at any axial plane is reconstructed by cross-correlating the object hologram with the corresponding component of the PSH library. The reconstruction noise attached to the image is suppressed by various methods. The reconstruction results of multiplane and thick objects by this technique are compared with regular lens-based imaging.

Fluorescence nanoscopy in cell biology

Fluorescence nanoscopy uniquely combines minimally invasive optical access to the internal nanoscale structure and dynamics of cells and tissues with molecular detection specificity. While the basic physical principles of 'super-resolution' imaging were discovered in the 1990s, with initial experimental demonstrations following in 2000, the broad application of super-resolution imaging to address cell-biological questions has only more recently emerged. Nanoscopy approaches have begun to facilitate discoveries in cell biology and to add new knowledge. One current direction for method improvement is the ambition to quantitatively account for each molecule under investigation and assess true molecular colocalization patterns via multi-colour analyses. In pursuing this goal, the labelling of individual molecules to enable their visualization has emerged as a central challenge. Extending nanoscale imaging into (sliced) tissue and whole-animal contexts is a further goal. In this Review we describe the successes to date and discuss current obstacles and possibilities for further development.

Vortex topographic microscopy for full-field reference-free imaging and testing

Light vortices carry orbital angular momentum and have a variety of applications in optical manipulation, high-capacity communications or microscopy. Here we propose a new concept of full-field vortex topographic microscopy enabling a reference-free displacement and shape measurement of reflective samples. The sample surface is mapped by an array of light spots enabling quantitative reconstruction of the local depths from defocused wavefronts. Light from the spots is converted to a lattice of mutually uncorrelated double-helix point spread functions (PSFs) whose angular rotation enables depth estimation. The PSFs are created by self-interference of optical vortices that originate from the same wavefront and are shaped by a spiral phase mask (SPM). The method benefits from the isoplanatic PSFs whose shape and size remain unchanged under defocusing, ensuring high precision in a wide range of measured depths. The technique was tested using a microscope Nikon Eclipse E600 working with a micro-hole plate providing structured illumination and the SPM placed in the imaging path. The depth measurement was demonstrated in the range of 11 µm exceeding the depth of field of the microscope objective up to 19 times. Throughout this range, the surface depth was mapped with the precision better than 30 nm at the lateral positions given with the precision better than 10 nm. Application potential of the method was demonstrated by profiling the top surface of a bearing ball and reconstructing the three-dimensional relief of a reflection phase grating.

Interference based localization of single emitters

The ability to localize precisely a single optical emitter is important for particle tracking applications and super resolution microscopy. It is known that for a traditional microscope the ability to localize such an emitter is limited by the photon count. Here we analyze the ability to improve such localization by imposing interference fringes. We show here that a simple grating interferometer can introduce such improvement in certain circumstances and analyze what is required to increase the localization precision further.

Interferometric rotating point spread function

Rotating point spread functions (PSF), such as the double helix (DH) PSF, are widely used in localization-based super-resolution imaging because of their large working depth range. In this article, we propose an interferometric DH PSF (iDH PSF) using two opposed objective lenses as in the 4Pi microscope. In the proposed iDH PSF, the super-resolution in the axial PSF is transferred to the azimuthal rotation. Moreover, we design an iDH PSF whose imaging range reaches 3 μm, which is roughly 3 times as much as that which can be obtained by using other interferometric localization-based super-resolution imaging methods.

Multiplexed phase-space imaging for 3D fluorescence microscopy

Optical phase-space functions describe spatial and angular information simultaneously; examples of optical phase-space functions include light fields in ray optics and Wigner functions in wave optics. Measurement of phase-space enables digital refocusing, aberration removal and 3D reconstruction. High-resolution capture of 4D phase-space datasets is, however, challenging. Previous scanning approaches are slow, light inefficient and do not achieve diffraction-limited resolution. Here, we propose a multiplexed method that solves these problems. We use a spatial light modulator (SLM) in the pupil plane of a microscope in order to sequentially pattern multiplexed coded apertures while capturing images in real space. Then, we reconstruct the 3D fluorescence distribution of our sample by solving an inverse problem via regularized least squares with a proximal accelerated gradient descent solver. We experimentally reconstruct a 101 Megavoxel 3D volume (1010×510×500µm with NA 0.4), demonstrating improved acquisition time, light throughput and resolution compared to scanning aperture methods. Our flexible patterning scheme further allows sparsity in the sample to be exploited for reduced data capture.

Three-Dimensional Localization of Single Molecules for Super-Resolution Imaging and Single-Particle Tracking

Single-molecule super-resolution fluorescence microscopy and single-particle tracking are two imaging modalities that illuminate the properties of cells and materials on spatial scales down to tens of nanometers or with dynamical information about nanoscale particle motion in the millisecond range, respectively. These methods generally use wide-field microscopes and two-dimensional camera detectors to localize molecules to much higher precision than the diffraction limit. Given the limited total photons available from each single-molecule label, both modalities require careful mathematical analysis and image processing. Much more information can be obtained about the system under study by extending to three-dimensional (3D) single-molecule localization: without this capability, visualization of structures or motions extending in the axial direction can easily be missed or confused, compromising scientific understanding. A variety of methods for obtaining both 3D super-resolution images and 3D tracking information have been devised, each with their own strengths and weaknesses. These include imaging of multiple focal planes, point-spread-function engineering, and interferometric detection. These methods may be compared based on their ability to provide accurate and precise position information on single-molecule emitters with limited photons. To successfully apply and further develop these methods, it is essential to consider many practical concerns, including the effects of optical aberrations, field dependence in the imaging system, fluorophore labeling density, and registration between different color channels. Selected examples of 3D super-resolution imaging and tracking are described for illustration from a variety of biological contexts and with a variety of methods, demonstrating the power of 3D localization for understanding complex systems.

Interferenceless coded aperture correlation holography-a new technique for recording incoherent digital holograms without two-wave interference

Recording digital holograms without wave interference simplifies the optical systems, increases their power efficiency and avoids complicated aligning procedures. We propose and demonstrate a new technique of digital hologram acquisition without two-wave interference. Incoherent light emitted from an object propagates through a random-like coded phase mask and recorded directly without interference by a digital camera. In the training stage of the system, a point spread hologram (PSH) is first recorded by modulating the light diffracted from a point object by the coded phase masks. At least two different masks should be used to record two different intensity distributions at all possible axial locations. The various recorded patterns at every axial location are superposed in the computer to obtain a complex valued PSH library cataloged to its axial location. Following the training stage, an object is placed within the axial boundaries of the PSH library and the light diffracted from the object is once again modulated by the same phase masks. The intensity patterns are recorded and superposed exactly as the PSH to yield a complex hologram of the object. The object information at any particular plane is reconstructed by a cross-correlation between the complex valued hologram and the appropriate element of the PSH library. The characteristics and the performance of the proposed system were compared with an equivalent regular imaging system.

Engineering and Optimization of Quasi-Nondiffracting Helicon-Like Beams With an Evolutionary Algorithm

Nondiffracting beams maintain their intensity profiles over a large propagation distance without substantial diffraction and exhibit unique propagation trajectories, leading to scientific impacts in various fields. However, the nonlocalized intensity distribution of non-diffracting beams is restrictive for many practical applications. Thus, strategies to optimize the beam profiles remain much in demand. In this report, we demonstrate an evolutionary algorithmic framework for optical beam engineering and optimization and experimentally validate it by realizing quasi-nondiffracting radially self-accelerating (or self-rotating) beams in a high-resolution imaging system. The work reports a tightly confined side-lobe-suppressed helicon-like beam that largely maintains its properties of radial self-acceleration and non-diffraction in the 3-D space. The optimization method represents a new methodological avenue that can be extended to a broad range of beam engineering problems.

Single Particle Tracking: From Theory to Biophysical Applications

After three decades of developments, single particle tracking (SPT) has become a powerful tool to interrogate dynamics in a range of materials including live cells and novel catalytic supports because of its ability to reveal dynamics in the structure-function relationships underlying the heterogeneous nature of such systems. In this review, we summarize the algorithms behind, and practical applications of, SPT. We first cover the theoretical background including particle identification, localization, and trajectory reconstruction. General instrumentation and recent developments to achieve two- and three-dimensional subdiffraction localization and SPT are discussed. We then highlight some applications of SPT to study various biological and synthetic materials systems. Finally, we provide our perspective regarding several directions for future advancements in the theory and application of SPT.

Three-Dimensional Super-Resolution in Eukaryotic Cells Using the Double-Helix Point Spread Function

Single-molecule localization microscopy, typically based on total internal reflection illumination, has taken our understanding of protein organization and dynamics in cells beyond the diffraction limit. However, biological systems exist in a complicated three-dimensional environment, which has required the development of new techniques, including the double-helix point spread function (DHPSF), to accurately visualize biological processes. The application of the DHPSF approach has so far been limited to the study of relatively small prokaryotic cells. By matching the refractive index of the objective lens immersion liquid to that of the sample media, we demonstrate DHPSF imaging of up to 15-μm-thick whole eukaryotic cell volumes in three to five imaging planes. We illustrate the capabilities of the DHPSF by exploring large-scale membrane reorganization in human T cells after receptor triggering, and by using single-particle tracking to image several mammalian proteins, including membrane, cytoplasmic, and nuclear proteins in T cells and embryonic stem cells.

Analyzing Single Molecule Localization Microscopy Data Using Cubic Splines

The resolution of super-resolution microscopy based on single molecule localization is in part determined by the accuracy of the localization algorithm. In most published approaches to date this localization is done by fitting an analytical function that approximates the point spread function (PSF) of the microscope. However, particularly for localization in 3D, analytical functions such as a Gaussian, which are computationally inexpensive, may not accurately capture the PSF shape leading to reduced fitting accuracy. On the other hand, analytical functions that can accurately capture the PSF shape, such as those based on pupil functions, can be computationally expensive. Here we investigate the use of cubic splines as an alternative fitting approach. We demonstrate that cubic splines can capture the shape of any PSF with high accuracy and that they can be used for fitting the PSF with only a 2–3x increase in computation time as compared to Gaussian fitting. We provide an open-source software package that measures the PSF of any microscope and uses the measured PSF to perform 3D single molecule localization microscopy analysis with reasonable accuracy and speed.

Measurement-based estimation of global pupil functions in 3D localization microscopy

We report the use of a phase retrieval procedure based on maximum likelihood estimation (MLE) to produce an improved, experimentally calibrated model of a point spread function (PSF) for use in three-dimensional (3D) localization microscopy experiments. The method estimates a global pupil phase function (which includes both the PSF and system aberrations) over the full axial range from a simple calibration scan. The pupil function is used to refine the PSF model and hence enable superior localizations from experimental data. To demonstrate the utility of the procedure, we apply it to experimental data acquired with a microscope employing a tetrapod PSF with a 6 µm axial range. The phase-retrieved model demonstrates significant improvements in both accuracy and precision of 3D localizations relative to the model based on scalar diffraction theory. The localization precision of the phase-retrieved model is shown to be near the limits imposed by estimation theory, and the reproducibility of the procedure is characterized and discussed. Code which performs the phase retrieval algorithm is provided.

Spiral phase mask shadow-imaging for 3D-measurement of flow fields

Particle tracking velocimetry (PTV) is a valuable tool for microfluidic analysis. Especially mixing processes and the environmental interaction of fluids on a microscopic scale are of particular importance for pharmaceutical and biomedical applications. However, currently applied techniques suffer from the lag of instantaneous depth information. Here we present a scan-free, shadow-imaging PTV-technique for 3D trajectory and velocity measurement of flow fields in micro-channels with 2 µm spatial resolution. By using an incoherent light source, one camera and a spatial light modulator (LCoS-SLM) that generates double-images of the seeding particle shadows, it is a simply applicable and highly scalable technique.

A simple approach to spectrally resolved fluorescence and bright field microscopy over select regions of interest

A standard wide field inverted microscope was converted to a spatially selective spectrally resolved microscope through the addition of a polarizing beam splitter, a pair of polarizers, an amplitude-mode liquid crystal-spatial light modulator, and a USB spectrometer. The instrument is capable of simultaneously imaging and acquiring spectra over user defined regions of interest. The microscope can also be operated in a bright-field mode to acquire absorption spectra of micron scale objects. The utility of the instrument is demonstrated on three different samples. First, the instrument is used to resolve three differently labeled fluorescent beads in vitro. Second, the instrument is used to recover time dependent bleaching dynamics that have distinct spectral changes in the cyanobacteria, Synechococcus leopoliensis UTEX 625. Lastly, the technique is used to acquire the absorption spectra of CH3NH3PbBr3 perovskites and measure differences between nanocrystal films and micron scale crystals.

Dual-color multiple-particle tracking at 50-nm localization and over 100-µm range in 3D with temporal focusing two-photon microscopy

Nanoscale particle tracking in three dimensions is crucial to directly observe dynamics of molecules and nanoparticles in living cells. Here we present a three-dimensional particle tracking method based on temporally focused two-photon excitation. Multiple particles are imaged at 30 frames/s in volume up to 180 × 180 × 100 µm³. The spatial localization precision can reach 50 nm. We demonstrate its capability of tracking fast swimming microbes at speed of ~200 µm/s. Two-photon dual-color tracking is achieved by simultaneously exciting two kinds of fluorescent beads at 800 nm to demonstrate its potential in molecular interaction studies. Our method provides a simple wide-field fluorescence imaging approach for deep multiple-particle tracking.