Optimal 3D single-molecule localization for superresolution microscopy with aberrations and engineered point spread functions

Universal inverse modeling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single-molecule localization, aberration correction and deconvolution. Here we present universal inverse modeling of point spread functions (uiPSF), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single-molecule localization microscopy (SMLM). Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system- or sample-specific characteristics, for example, the bead size, field- and depth- dependent aberrations, and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single-molecule blinking data.

Large-FOV 3D localization microscopy by spatially variant point spread function generation

Accurate characterization of the microscopic point spread function (PSF) is crucial for achieving high-performance localization microscopy (LM). Traditionally, LM assumes a spatially invariant PSF to simplify the modeling of the imaging system. However, for large fields of view (FOV) imaging, it becomes important to account for the spatially variant nature of the PSF. Here, we propose an accurate and fast principal components analysis-based field-dependent 3D PSF generator (PPG3D) and localizer for LM. Through simulations and experimental three-dimensional (3D) single-molecule localization microscopy (SMLM), we demonstrate the effectiveness of PPG3D, enabling super-resolution imaging of mitochondria and microtubules with high fidelity over a large FOV. A comparison of PPG3D with a shift-variant PSF generator for 3D LM reveals a threefold improvement in accuracy. Moreover, PPG3D is approximately 100 times faster than existing PSF generators, when used in image plane-based interpolation mode. Given its user-friendliness, we believe that PPG3D holds great potential for widespread application in SMLM and other imaging modalities.

From Monomers to Hexamers: A Theoretical Probability of the Neighbor Density Approach to Dissect Protein Oligomerization in Cells

Deciphering the oligomeric state of proteins within cells is pivotal to understanding their role in intricate cellular processes. With the recent advances in single-molecule localization microscopy, previous efforts have harnessed protein location density approaches, coupled with simulations, to extract membrane protein oligomeric states in cells, highlighting the value of such techniques. However, a comprehensive theoretical approach that can be universally applied across different proteins (e.g., membrane and cytosolic proteins) remains elusive. Here, we introduce the theoretical probability of neighbor density (PND) as a robust tool to discern protein oligomeric states in cellular environments. Utilizing our approach, the theoretical PND was validated against simulated data for both membrane and cytosolic proteins, consistently aligning with experimental baselines for membrane proteins. This congruence was maintained even when adjusting for protein concentrations or exploring proteins of various oligomeric states. The strength of our method lies not only in its precision but also in its adaptability, accommodating diverse cellular protein scenarios without compromising the accuracy. The development and validation of the theoretical PND facilitate accurate protein oligomeric state determination and bolster our understanding of protein-mediated cellular functions.

Universal inverse modelling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single molecule localization, aberration correction and deconvolution. Here we present uiPSF (universal inverse modelling of Point Spread Functions), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single molecule localization microscopy (SMLM). The resulting PSF model enables accurate 3D super-resolution imaging using SMLM. Additionally, uiPSF can be used to characterize and optimize a microscope system by quantifying the aberrations, including field-dependent aberrations, and resolutions. Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system or sample specific characteristics, e.g., the bead size, depth dependent aberrations and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single molecule blinking data.

MAxSIM: multi-angle-crossing structured illumination microscopy with height-controlled mirror for 3D topological mapping of live cells

Mapping 3D plasma membrane topology in live cells can bring unprecedented insights into cell biology. Widefield-based super-resolution methods such as 3D-structured illumination microscopy (3D-SIM) can achieve twice the axial ( ~ 300 nm) and lateral ( ~ 100 nm) resolution of widefield microscopy in real time in live cells. However, twice-resolution enhancement cannot sufficiently visualize nanoscale fine structures of the plasma membrane. Axial interferometry methods including fluorescence light interference contrast microscopy and its derivatives (e.g., scanning angle interference microscopy) can determine nanoscale axial locations of proteins on and near the plasma membrane. Thus, by combining super-resolution lateral imaging of 2D-SIM with axial interferometry, we developed multi-angle-crossing structured illumination microscopy (MAxSIM) to generate multiple incident angles by fast, optoelectronic creation of diffraction patterns. Axial localization accuracy can be enhanced by placing cells on a bottom glass substrate, locating a custom height-controlled mirror (HCM) at a fixed axial position above the glass substrate, and optimizing the height reconstruction algorithm for noisy experimental data. The HCM also enables imaging of both the apical and basal surfaces of a cell. MAxSIM with HCM offers high-fidelity nanoscale 3D topological mapping of cell plasma membranes with near-real-time ( ~ 0.5 Hz) imaging of live cells and 3D single-molecule tracking.

Lysosome-Targeted Biosensor for the Super-Resolution Imaging of Lysosome-Mitochondrion Interaction

Background: The interaction between lysosomes and mitochondria includes not only mitophagy but also mitochondrion–lysosome contact (MLC) that enables the two organelles to exchange materials and information. In our study, we synthesised a biosensor with fluorescence characteristics that can image lysosomes for structured illumination microscopy and, in turn, examined morphological changes in mitochondria and the phenomenon of MLC under pathological conditions.

Methods: After designing and synthesising the biosensor, dubbed CNN, we performed an assay with a Cell Counting Kit-8 to detect CNN’s toxicity in relation to H9C2 cardiomyocytes. We next analysed the co-localisation of CNN and the commercial lysosomal probe LTG in cells, qualitatively analysed the imaging characteristics of CNN in different cells (i.e. H9C2, HeLa and HepG2 cells) via structured illumination microscopy and observed how CNN entered cells at different temperatures and levels of endocytosis. Last, we treated the H9C2 cells with mannitol or glucose to observe the morphological changes of mitochondria and their positions relative to lysosomes.

Results: After we endocytosed CNN, a lysosome-targeted biosensor with a wide, stable pH response range, into cells in an energy-dependent manner. SIM also revealed that conditions in high glucose induced stress in lysosomes and changed the morphology of mitochondria from elongated strips to round spheres.

Conclusion: CNN is a new tool for tracking lysosomes in living cells, both physiologically and pathologically, and showcases new options for the design of similar biosensors.

Wavefront engineered light needle microscopy for axially resolved rapid volumetric imaging

Increasing the acquisition speed of three-dimensional volumetric images is important-particularly in biological imaging-to unveil the structural dynamics and functionalities of specimens in detail. In conventional laser scanning fluorescence microscopy, volumetric images are constructed from optical sectioning images sequentially acquired by changing the observation plane, limiting the acquisition speed. Here, we present a novel method to realize volumetric imaging from two-dimensional raster scanning of a light needle spot without sectioning, even in the traditional framework of laser scanning microscopy. Information from multiple axial planes is simultaneously captured using wavefront engineering for fluorescence signals, allowing us to readily survey the entire depth range while maintaining spatial resolution. This technique is applied to real-time and video-rate three-dimensional tracking of micrometer-sized particles, as well as the prompt visualization of thick fixed biological specimens, offering substantially faster volumetric imaging.

Phase optimization algorithm for 3D particle localization with large axial depth

We propose an optimization algorithm based on Fresnel approximation (FA) imaging to optimize an extended-axial-depth point spread function (PSF) for 3D particle localization. The transfer function efficiency of the PSF is improved by repeatedly imposing constraints in the object plane, the spatial domain, and the Fourier domain. During the iterative calculation, the effective photon number or Cramer-Rao lower bound is used as the termination condition of the iteration. The algorithm allows flexible adjustment of the peak intensity ratio of the two main lobes. Moreover, the transfer function efficiency can be balanced by increasing the weight of the modulation function of the expected PSF at each axial position. The twin-Airy (TA) PSF optimized by the FA optimization algorithm does not require complex post-processing, whereas post-processing is an essential step for the unoptimized TA-PSF. The optimization algorithm is significant for extended-axial-depth PSFs used for 3D particle localization, as it improves localization precision and temporal resolution.

High-axial-resolution single-molecule localization under dense excitation with a multi-channel deep U-Net

Single-molecule localization microscopy (SMLM) can bypass the diffraction limit of optical microscopes and greatly improve the resolution in fluorescence microscopy. By introducing the point spread function (PSF) engineering technique, we can customize depth varying PSF to achieve higher axial resolution. However, most existing 3D single-molecule localization algorithms require excited fluorescent molecules to be sparse and captured at high signal-to-noise ratios, which results in a long acquisition time and precludes SMLM's further applications in many potential fields. To address this problem, we propose a novel 3D single-molecular localization method based on a multi-channel neural network based on U-Net. By leveraging the deep network's great advantages in feature extraction, the proposed network can reliably discriminate dense fluorescent molecules with overlapped PSFs and corrupted by sensor noise. Both simulated and real experiments demonstrate its superior performance in PSF engineered microscopes with short exposure and dense excitations, which holds great potential in fast 3D super-resolution microscopy.

Splicing exponential point spread function design for localization of nanoparticles

We propose a point spread function for three-dimensional localization of nanoparticles. The axial detection range of the point spread function can be simply changed by adjusting the design parameters. In addition, the spatial extent of the point spread function can also be changed by adjusting the design parameters, which is an advantage other point spread functions do not have. We used our point spread functions and the existing point spread functions for dense multi-particle imaging, which proved the advantage that the point spread function with a smaller spatial extent we designed can effectively reduce the overlap between the point spread functions. The three-dimensional process of the fluorescent microsphere penetrating HT-22 cell membrane was successfully recorded, which verified the effectiveness of this method.

Super-resolution imaging reveals cytoskeleton-dependent organelle rearrangement within platelets at intermediate stages of maturation

A steady supply of platelets maintains their levels in the blood, and this is achieved by the generation of progeny from platelet intermediates. Using systematic super-resolution microscopy, we examine the ultrastructural organization of various organelles in different platelet intermediates to understand the mechanism of organelle redistribution and sorting in platelet intermediate maturation as the early step of platelet progeny production. We observe the dynamic interconversion between the intermediates and find that microtubules are responsible for controlling the overall shape of platelet intermediates. Super-resolution images show that most of the organelles are located near the cell periphery in oval preplatelets and confined to the bulbous tips in proplatelets. We also find that the distribution of the dense tubular system and α granules is regulated by actin, whereas that of mitochondria and dense granules is governed by microtubules. Altogether, our results call for a reassessment of organelle redistribution in platelet intermediates.

Extended mechanical force measurements using structured illumination microscopy

Quantifying cell generated mechanical forces is key to furthering our understanding of mechanobiology. Traction force microscopy (TFM) is one of the most broadly applied force probing technologies, but its sensitivity is strictly dependent on the spatio-temporal resolution of the underlying imaging system. In previous works, it was demonstrated that increased sampling densities of cell derived forces permitted by super-resolution fluorescence imaging enhanced the sensitivity of the TFM method. However, these recent advances to TFM based on super-resolution techniques were limited to slow acquisition speeds and high illumination powers. Here, we present three novel TFM approaches that, in combination with total internal reflection, structured illumination microscopy and astigmatism, improve the spatial and temporal performance in either two-dimensional or three-dimensional mechanical force quantification, while maintaining low illumination powers. These three techniques can be straightforwardly implemented on a single optical set-up offering a powerful platform to provide new insights into the physiological force generation in a wide range of biological studies. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

Accurate subpixel shifting of point spread functions

The localization of emitters requires accurate subpixel shifting of point spread function (PSF) models. However, the PSF recorded by the camera is not the true PSF of the system due to integration across finite pixels. These errors can be propagated during the shifting process, causing systematic biases in the registration or localization process. This letter proposes a set of filter kernels that, when convolved with the image, accurately shifts it by an arbitrary subpixel shift. Each pixel in the filter is represented by a two-dimensional polynomial function of the possible x and y shift values. These filters are effective; when tested on three different PSFs, they reduced errors by a factor of 20 or more over PSF models evaluated at the pixel center.

Depth-of-field engineering in coded aperture imaging

Extending the depth-of-field (DOF) of an optical imaging system without effecting the other imaging properties has been an important topic of research for a long time. In this work, we propose a new general technique of engineering the DOF of an imaging system beyond just a simple extension of the DOF. Engineering the DOF means in this study that the inherent DOF can be extended to one, or to several, separated different intervals of DOF, with controlled start and end points. Practically, because of the DOF engineering, entire objects in certain separated different input subvolumes are imaged with the same sharpness as if these objects are all in focus. Furthermore, the images from different subvolumes can be laterally shifted, each subvolume in a different shift, relative to their positions in the object space. By doing so, mutual hiding of images can be avoided. The proposed technique is introduced into a system of coded aperture imaging. In other words, the light from the object space is modulated by a coded aperture and recorded into the computer in which the desired image is reconstructed from the recorded pattern. The DOF engineering is done by designing the coded aperture composed of three diffractive elements. One element is a quadratic phase function dictating the start point of the in-focus axial interval and the second element is a quartic phase function which dictates the end point of this interval. Quasi-random coded phase mask is the third element, which enables the digital reconstruction. Multiplexing several sets of diffractive elements, each with different set of phase coefficients, can yield various axial reconstruction curves. The entire diffractive elements are displayed on a spatial light modulator such that real-time DOF engineering is enabled according to the user needs in the course of the observation. Experimental verifications of the proposed system with several examples of DOF engineering are presented, where the entire imaging of the observed scene is done by single camera shot.

Optical topography of rough surfaces using vortex localization of fluorescent markers

Measuring rough surfaces is challenging because the proven topographic methods are impaired by the adverse effects of diffuse light. In our method, the measured surface is marked by fluorescent nanobeads allowing a complete suppression of diffuse light by bandpass filtering. Light emitted by each fluorescent bead is shaped to a double-helix point spread function used for three-dimensional bead localization on the surface. This non-interferometric measurement of rough surface topography is implemented in a vibration resistant setup. The comparison of our method with vertical scanning interferometry shows that a commercial profiler is surpassed when ground glass surfaces with steep slopes are measured.

Addressing systematic errors in axial distance measurements in single-emitter localization microscopy

Nanoscale localization of point emitters is critical to several methods in optical fluorescence microscopy, including single-molecule super-resolution imaging and tracking. While the precision of the localization procedure has been the topic of extensive study, localization accuracy has been less emphasized, in part due to the challenge of producing an experimental sample containing unperturbed point emitters at known three-dimensional positions in a relevant geometry. We report a new experimental system which reproduces a widely-adopted geometry in high-numerical aperture localization microscopy, in which molecules are situated in an aqueous medium above a glass coverslip imaged with an oil-immersion objective. We demonstrate a calibration procedure that enables measurement of the depth-dependent point spread function (PSF) for open aperture imaging as well as imaging with engineered PSFs with index mismatch. We reveal the complicated, depth-varying behavior of the focal plane position in this system and discuss the axial localization biases incurred by common approximations of this behavior. We compare our results to theoretical calculations.

Three-Dimensional Single-Molecule Localization Microscopy in Whole-Cell and Tissue Specimens

Super-resolution microscopy techniques are versatile and powerful tools for visualizing organelle structures, interactions, and protein functions in biomedical research. However, whole-cell and tissue specimens challenge the achievable resolution and depth of nanoscopy methods. We focus on three-dimensional single-molecule localization microscopy and review some of the major roadblocks and developing solutions to resolving thick volumes of cells and tissues at the nanoscale in three dimensions. These challenges include background fluorescence, system- and sample-induced aberrations, and information carried by photons, as well as drift correction, volume reconstruction, and photobleaching mitigation. We also highlight examples of innovations that have demonstrated significant breakthroughs in addressing the abovementioned challenges together with their core concepts as well as their trade-offs.

Hybrid multifocal structured illumination microscopy with enhanced lateral resolution and axial localization capability

Super-resolution (SR) fluorescence microscopy that breaks through the diffraction barrier has drawn great interest in biomedical research. However, obtaining a high precision three-dimensional distribution of the specimen in a short time still remains a challenging task for existing techniques. In this paper, we propose a super-resolution fluorescence microscopy with axial localization capability by combining multifocal structured illumination microscopy with a hybrid detection PSF composed of a Gaussian PSF and a double-helix PSF. A modified reconstruction scheme is presented to accommodate the new hybrid PSF. This method can not only recover the lateral super-resolution image of the specimen but also retain the specimen's depth map within a range of 600 nm with an axial localization precision of 20.8 nm. The performance of this approach is verified by testing fluorescent beads and tubulin in 293-cells. The developed microscope is well suited for observing the precise 3D distribution of thin specimens.

Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet

Background fluorescence, especially when it exhibits undesired spatial features, is a primary factor for reduced image quality in optical microscopy. Structured background is particularly detrimental when analyzing single-molecule images for 3-dimensional localization microscopy or single-molecule tracking. Here, we introduce BGnet, a deep neural network with a U-net-type architecture, as a general method to rapidly estimate the background underlying the image of a point source with excellent accuracy, even when point-spread function (PSF) engineering is in use to create complex PSF shapes. We trained BGnet to extract the background from images of various PSFs and show that the identification is accurate for a wide range of different interfering background structures constructed from many spatial frequencies. Furthermore, we demonstrate that the obtained background-corrected PSF images, for both simulated and experimental data, lead to a substantial improvement in localization precision. Finally, we verify that structured background estimation with BGnet results in higher quality of superresolution reconstructions of biological structures.

Accurate phase retrieval of complex 3D point spread functions with deep residual neural networks

Phase retrieval, i.e., the reconstruction of phase information from intensity information, is a central problem in many optical systems. Imaging the emission from a point source such as a single molecule is one example. Here, we demonstrate that a deep residual neural net is able to quickly and accurately extract the hidden phase for general point spread functions (PSFs) formed by Zernike-type phase modulations. Five slices of the 3D PSF at different focal positions within a two micrometer range around the focus are sufficient to retrieve the first six orders of Zernike coefficients.

A mechanistic examination of salting out in protein-polymer membrane interactions

Developing a mechanistic understanding of protein dynamics and conformational changes at polymer interfaces is critical for a range of processes including industrial protein separations. Salting out is one example of a procedure that is ubiquitous in protein separations yet is optimized empirically because there is no mechanistic description of the underlying interactions that would allow predictive modeling. Here, we investigate peak narrowing in a model transferrin-nylon system under salting out conditions using a combination of single-molecule tracking and ensemble separations. Distinct surface transport modes and protein conformational changes at the negatively charged nylon interface are quantified as a function of salt concentration. Single-molecule kinetics relate macroscale improvements in chromatographic peak broadening with microscale distributions of surface interaction mechanisms such as continuous-time random walks and simple adsorption-desorption. Monte Carlo simulations underpinned by the stochastic theory of chromatography are performed using kinetic data extracted from single-molecule observations. Simulations agree with experiment, revealing a decrease in peak broadening as the salt concentration increases. The results suggest that chemical modifications to membranes that decrease the probability of surface random walks could reduce peak broadening in full-scale protein separations. More broadly, this work represents a proof of concept for combining single-molecule experiments and a mechanistic theory to improve costly and time-consuming empirical methods of optimization.

Three-dimensional biplane spectroscopic single-molecule localization microscopy

Spectroscopic single-molecule localization microscopy (sSMLM) captures the full emission spectra of individual molecules while simultaneously localizing their spatial locations at a precision greatly exceeding the optical diffraction limit. To achieve this, sSMLM uses a dispersive optical component to separate the emitted photons into dedicated spatial and spectral imaging channels for simultaneous acquisition. While adding a cylindrical lens in the spatial imaging channel enabled three-dimensional (3D) imaging in sSMLM, the inherent astigmatism leads to technical hurdles in spectral calibration and nonuniform lateral resolution at different depths. We found that implementing the biplane method based on the already established spatial and spectral imaging channels offers a much more attractive solution for 3D sSMLM. It allows for more efficient use of the limited photon budget and provides homogeneous lateral resolution compared with the astigmatism-based method using a cylindrical lens. Here we report 3D biplane sSMLM and demonstrate its multi-color 3D imaging capability by imaging microtubules and mitochondria in fixed COS-7 cells immunostained with Alexa Fluor 647 and CF 660C dyes, respectively. We showed a lateral localization precision of 20 nm at an average photon count of 550, a spectral precision of 4 nm at an average photon count of 1250, and an axial localization resolution of 50 nm.

Wavelength-scale errors in optical localization due to spin-orbit coupling of light

Far-field optical imaging techniques allow the determination of the position of point-like emitters and scatterers [1-3]. Although the optical wavelength sets a fundamental limit to the image resolution of unknown objects, the position of an individual emitter can in principle be estimated from the image with arbitrary precision. This is used for example in the determination of stars position [4] or in optical super-resolution microscopy [5]. Furthermore, precise position determination is an experimental prerequisite for the manipulation and measurement of individual quantum systems, such as atoms, ions, and solid-state-based quantum emitters [6-8]. Here we demonstrate that spin-orbit coupling of light in the emission of elliptically polarized emitters can lead to systematic, wavelength-scale errors in the estimation of the emitters position. Imaging a single trapped atom as well as a single sub-wavelength-diameter gold nanoparticle, we demonstrate a shift between the emitters measured and actual positions which is comparable to the optical wavelength. For certain settings, the expected shift can become arbitrarily large. Beyond optical imaging techniques, our findings could be relevant for the localization of objects using any type of wave that carries orbital angular momentum relative to the emitters position with a component orthogonal to the direction of observation.

Three-dimensional localization microscopy using deep learning

Single molecule localization microscopy (SMLM) is one of the fastest evolving and most broadly used super-resolving imaging techniques in the biosciences. While image recordings could take up to hours only ten years ago, scientists are now reaching for real-time imaging in order to follow the dynamics of biology. To this end, it is crucial to have data processing strategies available that are capable of handling the vast amounts of data produced by the microscope. In this article, we report on the use of a deep convolutional neural network (CNN) for localizing particles in three dimensions on the basis of single images. In test experiments conducted on fluorescent microbeads, we show that the precision obtained with a CNN can be comparable to that of maximum likelihood estimation (MLE), which is the accepted gold standard. Regarding speed, the CNN performs with about 22k localizations per second more than three orders of magnitude faster than the MLE algorithm of ThunderSTORM. If only five parameters are estimated (3D position, signal and background), our CNN implementation is currently slower than the fastest, recently published GPU-based MLE algorithm. However, in this comparison the CNN catches up with every additional parameter, with only a few percent extra time required per additional dimension. Thus it may become feasible to estimate further variables such as molecule orientation, aberration functions or color. We experimentally demonstrate that jointly estimating Zernike mode magnitudes for aberration modeling can significantly improve the accuracy of the estimates.

Aberration correction method based on double-helix point spread function

Point spread function (PSF) engineering has met with lots of interest in various optical imaging techniques, including super-resolution microscopy, microparticle tracking, and extended depth-of-field microscopy. The intensity distributions of the modified PSFs often suffer from deteriorations caused by system aberrations, which greatly degrade the image contrast, resolution, or localization precision. We present an aberration correction method using a spiral-phase-based double-helix PSF as an aberration indicator, which is sensitive and quantitatively correlated to the spherical aberration, coma, and astigmatism. Superior to the routine iteration-based correction methods, the presented approach is iteration-free and the aberration coefficients can be directly calculated with the measured parameters, relieving the computing burden. The validity of the method is verified by both examining the intensity distribution of the conventional Gaussian PSF in three dimensions and observing muntjac skin fibroblast cells. This iteration-free correction method has a potential application in PSF engineering systems equipped with a spatial light modulator.

Influence of sample surface height for evaluation of peak extraction algorithms in confocal microscopy

The axial resolution of confocal microscopy is not only dependent on optical characteristics but also on the utilized peak extraction algorithms. In previous evaluations of peak extraction algorithms, sample surface height is generally assumed to be zero, and only sampling-noise-induced peak extraction uncertainty was analyzed. Here we propose a sample surface-height-dependent (SHD) evaluation model that takes the combined considerations of sample surface height and noise for comparisons of algorithms' performances. Monte Carlo simulations were first conducted on the centroid algorithm and several nonlinear fitting algorithms such as the parabola fitting algorithm, Gaussian fitting algorithm, and sinc² fitting algorithm. Subsequently, the evaluation indicators, including mean peak extraction error and mean uncertainty were suggested for the algorithms' performance ranking. Finally, experimental verifications of the SHD model were carried out using a fiber-based chromatic confocal system. From our simulations and experiments, we demonstrate that sample surface height is a critical influencing factor in peak extraction computation in terms of both the accuracy and standard deviations. Compared to the conventional standard uncertainty evaluation model, our SHD model can provide a more comprehensive characterization of peak extraction algorithms' performance and offer a more flexible and consistent reference for algorithm selection.

ZOLA-3D allows flexible 3D localization microscopy over an adjustable axial range

Single molecule localization microscopy can generate 3D super-resolution images without scanning by leveraging the axial variations of normal or engineered point spread functions (PSF). Successful implementation of these approaches for extended axial ranges remains, however, challenging. We present Zernike Optimized Localization Approach in 3D (ZOLA-3D), an easy-to-use computational and optical solution that achieves optimal resolution over a tunable axial range. We use ZOLA-3D to demonstrate 3D super-resolution imaging of mitochondria, nuclear pores and microtubules in entire nuclei or cells up to ~5 μm deep.

3D single-molecule localization is limited in depth and often requires using a wide range of point spread functions (PSFs). Here the authors present an optical solution featuring a deformable mirror to generate different PSFs and easy-to-use software for super-resolution imaging up to 5 µm deep.

Optimized protocol for combined PALM-dSTORM imaging

Multi-colour super-resolution localization microscopy is an efficient technique to study a variety of intracellular processes, including protein-protein interactions. This technique requires specific labels that display transition between fluorescent and non-fluorescent states under given conditions. For the most commonly used label types, photoactivatable fluorescent proteins and organic fluorophores, these conditions are different, making experiments that combine both labels difficult. Here, we demonstrate that changing the standard imaging buffer of thiols/oxygen scavenging system, used for organic fluorophores, to the commercial mounting medium Vectashield increased the number of photons emitted by the fluorescent protein mEos2 and enhanced the photoconversion rate between its green and red forms. In addition, the photophysical properties of organic fluorophores remained unaltered with respect to the standard imaging buffer. The use of Vectashield together with our optimized protocol for correction of sample drift and chromatic aberrations enabled us to perform two-colour 3D super-resolution imaging of the nucleolus and resolve its three compartments.

Combined hardware and computational optical wavefront correction

In many optical imaging applications, it is necessary to overcome aberrations to obtain high-resolution images. Aberration correction can be performed by either physically modifying the optical wavefront using hardware components, or by modifying the wavefront during image reconstruction using computational imaging. Here we address a longstanding issue in computational imaging: photons that are not collected cannot be corrected. This severely restricts the applications of computational wavefront correction. Additionally, performance limitations of hardware wavefront correction leave many aberrations uncorrected. We combine hardware and computational correction to address the shortcomings of each method. Coherent optical backscattering data is collected using high-speed optical coherence tomography, with aberrations corrected at the time of acquisition using a wavefront sensor and deformable mirror to maximize photon collection. Remaining aberrations are corrected by digitally modifying the coherently-measured wavefront during imaging reconstruction. This strategy obtains high-resolution images with improved signal-to-noise ratio of in vivo human photoreceptor cells with more complete correction of ocular aberrations, and increased flexibility to image at multiple retinal depths, field locations, and time points. While our approach is not restricted to retinal imaging, this application is one of the most challenging for computational imaging due to the large aberrations of the dilated pupil, time-varying aberrations, and unavoidable eye motion. In contrast with previous computational imaging work, we have imaged single photoreceptors and their waveguide modes in fully dilated eyes with a single acquisition. Combined hardware and computational wavefront correction improves the image sharpness of existing adaptive optics systems, and broadens the potential applications of computational imaging methods.

Real-time 3D single-molecule localization using experimental point spread functions

We present a real-time fitter for 3D single-molecule localization microscopy using experimental point spread functions (PSFs) that achieves minimal uncertainty in 3D on any microscope and is compatible with any PSF engineering approach. We used this method to image cellular structures and attained unprecedented image quality for astigmatic PSFs. The fitter compensates for most optical aberrations and makes accurate 3D super-resolution microscopy broadly accessible, even on standard microscopes without dedicated 3D optics.

High-order-helix point spread functions for monocular three-dimensional imaging with superior aberration robustness

An approach for designing purely refractive optical elements that generate engineered, multi-order-helix point spread functions (PSFs) with large peak separation for passive, optical depth measurement is presented. The influence of aberrations on the PSF's rotation angle, which limits the depth retrieval accuracy, is studied numerically and analytically. It appears that only Zernike modes with an azimuthal index that is an integer multiple of the number of PSF peaks introduce PSF rotation, and hence depth estimation errors. This implies that high-order-helix designs have superior robustness with respect to aberrations. This is experimentally demonstrated by imaging an extended scene in the presence of severe system aberrations using novel, cost-efficient phase elements based on UV-replication on the wafer-scale.

Point-spread function engineering enhances digital Fourier microscopy

While numerous optical methods exist to probe the dynamics of biological or complex fluid samples, in recent years digital Fourier microscopy techniques such as differential dynamic microscopy have emerged as ways to efficiently combine elements of imaging and scattering methods. Here, we demonstrate, through experiments and simulations, how point-spread function engineering can be used to extend the reach of differential dynamic microscopy.

Super-Resolution Microscopy: Shedding Light on the Cellular Plasma Membrane

Lipids and the membranes they form are fundamental building blocks of cellular life, and their geometry and chemical properties distinguish membranes from other cellular environments. Collective processes occurring within membranes strongly impact cellular behavior and biochemistry, and understanding these processes presents unique challenges due to the often complex and myriad interactions between membrane components. Super-resolution microscopy offers a significant gain in resolution over traditional optical microscopy, enabling the localization of individual molecules even in densely labeled samples and in cellular and tissue environments. These microscopy techniques have been used to examine the organization and dynamics of plasma membrane components, providing insight into the fundamental interactions that determine membrane functions. Here, we broadly introduce the structure and organization of the mammalian plasma membrane and review recent applications of super-resolution microscopy to the study of membranes. We then highlight some inherent challenges faced when using super-resolution microscopy to study membranes, and we discuss recent technical advancements that promise further improvements to super-resolution microscopy and its application to the plasma membrane.

Engineering and Optimization of Quasi-Nondiffracting Helicon-Like Beams With an Evolutionary Algorithm

Nondiffracting beams maintain their intensity profiles over a large propagation distance without substantial diffraction and exhibit unique propagation trajectories, leading to scientific impacts in various fields. However, the nonlocalized intensity distribution of non-diffracting beams is restrictive for many practical applications. Thus, strategies to optimize the beam profiles remain much in demand. In this report, we demonstrate an evolutionary algorithmic framework for optical beam engineering and optimization and experimentally validate it by realizing quasi-nondiffracting radially self-accelerating (or self-rotating) beams in a high-resolution imaging system. The work reports a tightly confined side-lobe-suppressed helicon-like beam that largely maintains its properties of radial self-acceleration and non-diffraction in the 3-D space. The optimization method represents a new methodological avenue that can be extended to a broad range of beam engineering problems.

Live endothelial cells imaged by Scanning Near-field Optical Microscopy (SNOM): capabilities and challenges

The scanning near-field optical microscopy (SNOM) shows a potential to study details of biological samples, since it provides the optical images of objects with nanometric spatial resolution (50-200 nm) and the topographic information at the same time. The goal of this work is to demonstrate the capabilities of SNOM in transmission configuration to study human endothelial cells and their morphological changes, sometimes very subtle, upon inflammation. Various sample preparations were tested for SNOM measurements and promising results are collected to show: 1) the influence of α tumor necrosis factor (TNF-α) on EA.hy 926 cells (measurements of the fixed cells); 2) high resolution images of various endothelial cell lines, i.e. EA.hy 926 and HLMVEC (investigations of the fixed cells in buffer environment); 3) imaging of live endothelial cells in physiological buffers. The study demonstrate complementarity of the SNOM measurements performed in air and in liquid environments, on fixed as well as on living cells. Furthermore, it is proved that the SNOM is a very useful method for analysis of cellular morphology and topography. Changes in the cell shape and nucleus size, which are the symptoms of inflammatory reaction, were noticed in TNF-α activated EA.hy 926 cells. The cellular structures of submicron size were observed in high resolution optical images of cells from EA.hy 926 and HLMVEC lines.

Single Particle Tracking: From Theory to Biophysical Applications

After three decades of developments, single particle tracking (SPT) has become a powerful tool to interrogate dynamics in a range of materials including live cells and novel catalytic supports because of its ability to reveal dynamics in the structure-function relationships underlying the heterogeneous nature of such systems. In this review, we summarize the algorithms behind, and practical applications of, SPT. We first cover the theoretical background including particle identification, localization, and trajectory reconstruction. General instrumentation and recent developments to achieve two- and three-dimensional subdiffraction localization and SPT are discussed. We then highlight some applications of SPT to study various biological and synthetic materials systems. Finally, we provide our perspective regarding several directions for future advancements in the theory and application of SPT.

Measurement-based estimation of global pupil functions in 3D localization microscopy

We report the use of a phase retrieval procedure based on maximum likelihood estimation (MLE) to produce an improved, experimentally calibrated model of a point spread function (PSF) for use in three-dimensional (3D) localization microscopy experiments. The method estimates a global pupil phase function (which includes both the PSF and system aberrations) over the full axial range from a simple calibration scan. The pupil function is used to refine the PSF model and hence enable superior localizations from experimental data. To demonstrate the utility of the procedure, we apply it to experimental data acquired with a microscope employing a tetrapod PSF with a 6 µm axial range. The phase-retrieved model demonstrates significant improvements in both accuracy and precision of 3D localizations relative to the model based on scalar diffraction theory. The localization precision of the phase-retrieved model is shown to be near the limits imposed by estimation theory, and the reproducibility of the procedure is characterized and discussed. Code which performs the phase retrieval algorithm is provided.

Fast Three-Dimensional Single-Particle Tracking in Natural Brain Tissue

Observation of molecular dynamics is often biased by the optical very heterogeneous environment of cells and complex tissue. Here, we have designed an algorithm that facilitates molecular dynamic analyses within brain slices. We adjust fast astigmatism-based three-dimensional single-particle tracking techniques to depth-dependent optical aberrations induced by the refractive index mismatch so that they are applicable to complex samples. In contrast to existing techniques, our online calibration method determines the aberration directly from the acquired two-dimensional image stream by exploiting the inherent particle movement and the redundancy introduced by the astigmatism. The method improves the positioning by reducing the systematic errors introduced by the aberrations, and allows correct derivation of the cellular morphology and molecular diffusion parameters in three dimensions independently of the imaging depth. No additional experimental effort for the user is required. Our method will be useful for many imaging configurations, which allow imaging in deep cellular structures.

Color-Coded Super-Resolution Small-Molecule Imaging

Although the development of super-resolution microscopy dates back to 1994, its applications have been primarily focused on visualizing cellular structures and targets, including proteins, DNA and sugars. We now report on a system that allows both monitoring of the localization of exogenous small molecules in live cells at low resolution and subsequent super-resolution imaging by using stochastic optical reconstruction microscopy (STORM) on fixed cells. This represents a powerful new tool to understand the dynamics of subcellular trafficking associated with the mode and mechanism of action of exogenous small molecules.

Single shot three-dimensional imaging using an engineered point spread function

A system approach to acquire a three-dimensional object distribution is presented using a compact and cost efficient camera system with an engineered point spread function. The corresponding monocular setup incorporates a phase-only computer-generated hologram in combination with a conventional imaging objective in order to optically encode the axial information within a single two-dimensional image. The object's depth map is calculated using a novel approach based on the power cepstrum of the image. The in-plane RGB image information is restored with an extended depth of focus by applying an adapted Wiener filter. The presented approach is tested experimentally by estimating the three-dimensional distribution of an extended passively illuminated scene.

Real-time adaptive drift correction for super-resolution localization microscopy

Super-resolution localization microscopy involves acquiring thousands of image frames of sparse collections of single molecules in the sample. The long acquisition time makes the imaging setup prone to drift, affecting accuracy and precision. Localization accuracy is generally improved by a posteriori drift correction. However, localization precision lost due to sample drifting out of focus cannot be recovered as the signal is originally detected at a lower peak signal. Here, we demonstrate a method of stabilizing a super-resolution localization microscope in three dimensions for extended periods of time with nanometer precision. Hence, no localization correction after the experiment is required to obtain super-resolved reconstructions. The method incorporates a closed-loop with a feedback signal generated from camera images and actuation on a 3D nanopositioning stage holding the sample.

New results on the single molecule localization problem in two and three dimensions

Fluorescence microscopy is an optical microscopy technique which has been extensively used to study specifically- labeled subcellular objects, such as proteins, and their functions. The best possible accuracy with which an object of interest can be localized when imaged using a fluorescence microscope is typically calculated using the Cramer- Rao lower bound (CRLB). The calculation of the CRLB, however, so far relied on an analytical expression for the image of the object. This can pose challenges in practice since it is often difficult to find appropriate analytical models for the images of general objects. Even if an appropriate analytical model is available, the lack of knowledge about the precise values of imaging parameters might also impose difficulties in the calculation oxf the CRLB. To address these challenges, we have developed an approach that directly uses an experimentally collected image set to calculate the best possible localization accuracy for a general subcellular object in two and three dimensions. In this approach, we fit smoothly connected piecewise polynomials, known as splines, to the experimentally collected image set to provide a continuous model of the object. This continuous model can then be used for the calculation of the best possible localization accuracy.

Fast weighted centroid algorithm for single particle localization near the information limit

A simple weighting scheme that enhances the localization precision of center of mass calculations for radially symmetric intensity distributions is presented. The algorithm effectively removes the biasing that is common in such center of mass calculations. Localization precision compares favorably with other localization algorithms used in super-resolution microscopy and particle tracking, while significantly reducing the processing time and memory usage. We expect that the algorithm presented will be of significant utility when fast computationally lightweight particle localization or tracking is desired.

Sequential superresolution imaging of multiple targets using a single fluorophore

Fluorescence superresolution (SR) microscopy, or fluorescence nanoscopy, provides nanometer scale detail of cellular structures and allows for imaging of biological processes at the molecular level. Specific SR imaging methods, such as localization-based imaging, rely on stochastic transitions between on (fluorescent) and off (dark) states of fluorophores. Imaging multiple cellular structures using multi-color imaging is complicated and limited by the differing properties of various organic dyes including their fluorescent state duty cycle, photons per switching event, number of fluorescent cycles before irreversible photobleaching, and overall sensitivity to buffer conditions. In addition, multiple color imaging requires consideration of multiple optical paths or chromatic aberration that can lead to differential aberrations that are important at the nanometer scale. Here, we report a method for sequential labeling and imaging that allows for SR imaging of multiple targets using a single fluorophore with negligible cross-talk between images. Using brightfield image correlation to register and overlay multiple image acquisitions with ~10 nm overlay precision in the x-y imaging plane, we have exploited the optimal properties of AlexaFluor647 for dSTORM to image four distinct cellular proteins. We also visualize the changes in co-localization of the epidermal growth factor (EGF) receptor and clathrin upon EGF addition that are consistent with clathrin-mediated endocytosis. These results are the first to demonstrate sequential SR (s-SR) imaging using direct stochastic reconstruction microscopy (dSTORM), and this method for sequential imaging can be applied to any superresolution technique.

Determination of localization accuracy based on experimentally acquired image sets: applications to single molecule microscopy

Fluorescence microscopy is a photon-limited imaging modality that allows the study of subcellular objects and processes with high specificity. The best possible accuracy (standard deviation) with which an object of interest can be localized when imaged using a fluorescence microscope is typically calculated using the Cramér-Rao lower bound, that is, the inverse of the Fisher information. However, the current approach for the calculation of the best possible localization accuracy relies on an analytical expression for the image of the object. This can pose practical challenges since it is often difficult to find appropriate analytical models for the images of general objects. In this study, we instead develop an approach that directly uses an experimentally collected image set to calculate the best possible localization accuracy for a general subcellular object. In this approach, we fit splines, i.e. smoothly connected piecewise polynomials, to the experimentally collected image set to provide a continuous model of the object, which can then be used for the calculation of the best possible localization accuracy. Due to its practical importance, we investigate in detail the application of the proposed approach in single molecule fluorescence microscopy. In this case, the object of interest is a point source and, therefore, the acquired image set pertains to an experimental point spread function.

Determining the rotational mobility of a single molecule from a single image: a numerical study

Measurements of the orientational freedom with which a single molecule may rotate or 'wobble' about a fixed axis have provided researchers invaluable clues about the underlying behavior of a variety of biological systems. In this paper, we propose a measurement and data analysis procedure based on a widefield fluorescence microscope image for quantitatively distinguishing individual molecules that exhibit varying degrees of rotational mobility. Our proposed technique is especially applicable to cases in which the molecule undergoes rotational motions on a timescale much faster than the framerate of the camera used to record fluorescence images. Unlike currently available methods, sophisticated hardware for modulating the polarization of light illuminating the sample is not required. Additional polarization optics may be inserted in the microscope's imaging pathway to achieve superior measurement precision, but are not essential. We present a theoretical analysis, and benchmark our technique with numerical simulations using typical experimental parameters for single-molecule imaging.

Effect of rotational diffusion in an orientational potential well on the point spread function of electric dipole emitters

A study is presented of the point spread function (PSF) of electric dipole emitters that go through a series of absorption-emission cycles while the dipole orientation is changing due to rotational diffusion within the constraint of an orientational potential well. An analytical expression for the PSF is derived valid for arbitrary orientational potential wells in the limit of image acquisition times much larger than the rotational relaxation time. This framework is used to study the effects of the direction of incidence, polarization, and degree of coherence of the illumination. In the limit of fast rotational diffusion on the scale of the fluorescence lifetime the illumination influences only the PSF height, not its shape. In the limit of slow rotational diffusion on the scale of the fluorescence lifetime there is a significant effect on the PSF shape as well, provided the illumination is (partially) coherent. For oblique incidence, illumination asymmetries can arise in the PSF that give rise to position offsets in localization based on Gaussian spot fitting. These asymmetries persist in the limit of free diffusion in a zero orientational potential well.

Spatial information in large-scale neural recordings

To record from a given neuron, a recording technology must be able to separate the activity of that neuron from the activity of its neighbors. Here, we develop a Fisher information based framework to determine the conditions under which this is feasible for a given technology. This framework combines measurable point spread functions with measurable noise distributions to produce theoretical bounds on the precision with which a recording technology can localize neural activities. If there is sufficient information to uniquely localize neural activities, then a technology will, from an information theoretic perspective, be able to record from these neurons. We (1) describe this framework, and (2) demonstrate its application in model experiments. This method generalizes to many recording devices that resolve objects in space and should be useful in the design of next-generation scalable neural recording systems.

Super-resolution imaging in live cells

Over the last twenty years super-resolution fluorescence microscopy has gone from proof-of-concept experiments to commercial systems being available in many labs, improving the resolution achievable by up to a factor of 10 or more. There are three major approaches to super-resolution, stimulated emission depletion microscopy, structured illumination microscopy, and localisation microscopy, which have all produced stunning images of cellular structures. A major current challenge is optimising performance of each technique so that the same sort of data can be routinely taken in live cells. There are several major challenges, particularly phototoxicity and the speed with which images of whole cells, or groups of cells, can be acquired. In this review we discuss the various approaches which can be successfully used in live cells, the tradeoffs in resolution, speed, and ease of implementation which one must make for each approach, and the quality of results that one might expect from each technique.

•

Super-resolution imaging of cell structures can achieve a resolution of tens of nm.

•

There are three major techniques: STED, SIM, and localisation microscopy.

•

Live cell super-resolution requires trading off resolution, speed, and light dose.

The nanoscale organization of the B lymphocyte membrane

The fluid mosaic model of Singer and Nicolson correctly predicted that the plasma membrane (PM) forms a lipid bi-layer containing many integral trans-membrane proteins. This model also suggested that most of these proteins were randomly dispersed and freely diffusing moieties. Initially, this view of a dynamic and rather unorganized membrane was supported by early observations of the cell surfaces using the light microscope. However, recent studies on the PM below the diffraction limit of visible light (~250nm) revealed that, at nanoscale dimensions, membranes are highly organized and compartmentalized structures. Lymphocytes are particularly useful to study this nanoscale membrane organization because they grow as single cells and are not permanently engaged in cell:cell contacts within a tissue that can influence membrane organization. In this review, we describe the methods that can be used to better study the protein:protein interaction and nanoscale organization of lymphocyte membrane proteins, with a focus on the B cell antigen receptor (BCR). Furthermore, we discuss the factors that may generate and maintain these membrane structures.