Designing the focal plane spacing for multifocal plane microscopy

Joint estimation of point spread function and molecule positions in SMLM informed from multiple planes

The advent of single molecule localization microscopy (SMLM) has transformed our capacity to investigate biological structures at the nanoscale. While the research focus has long been on improving localization precision, systematic errors caused by optical aberrations are often overlooked. In the case of 3D SMLM, such errors have the potential to significantly impair the quality of the resulting images. In this paper, we present an imaging and data processing approach that jointly estimates both, molecule positions and optical aberrations in SMLM. Therefore, the method minimizes systematic errors in SMLM reconstructions without the necessity of additional experimental calibration steps, such as the recording of fluorescent bead z-stacks. We investigate the reliability of this approach, especially in situations where the joint retrieval can be expected to be ill-posed, i.e., whenever the sample is "flat" and provides little diversity among the captured single molecule images. To enhance the reliability of the inverse problem solution, we suggest utilizing small SMLM data sets acquired at one or more slightly defocused "auxiliary" planes. We investigate the effectiveness of our approach through numerical simulations and imaging experiments of a calibration probe and nuclear pore complexes. Our method is simple and integrates seamlessly into existing SMLM setups without necessitating modifications or added complexity to the system.

Simultaneous Multifocal Plane Fourier Ptychographic Microscopy Utilizing a Standard RGB Camera

Fourier ptychographic microscopy (FPM) is a computational imaging technology that can acquire high-resolution large-area images for applications ranging from biology to microelectronics. In this study, we utilize multifocal plane imaging to enhance the existing FPM technology. Using an RGB light emitting diode (LED) array to illuminate the sample, raw images are captured using a color camera. Then, exploiting the basic optical principle of wavelength-dependent focal length variation, three focal plane images are extracted from the raw image through simple R, G, and B channel separation. Herein, a single aspherical lens with a numerical aperture (NA) of 0.15 was used as the objective lens, and the illumination NA used for FPM image reconstruction was 0.08. Therefore, simultaneous multifocal plane FPM with a synthetic NA of 0.23 was achieved. The multifocal imaging performance of the enhanced FPM system was then evaluated by inspecting a transparent organic light-emitting diode (OLED) sample. The FPM system was able to simultaneously inspect the individual OLED pixels as well as the surface of the encapsulating glass substrate by separating R, G, and B channel images from the raw image, which was taken in one shot.

Recent Advances in Single-Molecule Tracking and Imaging Techniques

Since the early 1990s, single-molecule detection in solution at room temperature has enabled direct observation of single biomolecules at work in real time and under physiological conditions, providing insights into complex biological systems that the traditional ensemble methods cannot offer. In particular, recent advances in single-molecule tracking techniques allow researchers to follow individual biomolecules in their native environments for a timescale of seconds to minutes, revealing not only the distinct pathways these biomolecules take for downstream signaling but also their roles in supporting life. In this review, we discuss various single-molecule tracking and imaging techniques developed to date, with an emphasis on advanced three-dimensional (3D) tracking systems that not only achieve ultrahigh spatiotemporal resolution but also provide sufficient working depths suitable for tracking single molecules in 3D tissue models. We then summarize the observables that can be extracted from the trajectory data. Methods to perform single-molecule clustering analysis and future directions are also discussed.

Fisher information and the Cramér-Rao lower bound in single-pixel localization microscopy with spatiotemporally modulated illumination

Single-pixel imaging, the concept that an image can be captured via a single-pixel detector, is a cost-effective yet powerful technique to reduce data acquisition duration without sacrificing image resolution when properly structured illumination patterns are introduced. Normally, the image reconstruction process is subject to the diffraction limit. Here, we study the possibility of exploiting the information contained in the illumination patterns to enable a form of single-pixel localization microscopy (SPLM) for super-resolution. This concept is inspired by coherent holographic image reconstruction by phase transfer (CHIRPT) microscopy. CHIRPT microscopy is a single-pixel imaging technique that uses structured illumination that is spatiotemporally modulated (STM) so that a unique temporal modulation pattern is imparted to each point within a large illumination volume. The fluorescent light emitted by molecules contains the same temporal modulations as the illumination patterns at the locations of the molecules. By recording a portion of the total emitted fluorescent power, the signal may be numerically processed to form an image. Unique temporal modulation patterns that excite fluorescent probes at each point can also be used to localize individual molecules by matching their particular temporal light emission patterns to the measured temporal signal. This paper evaluates the feasibility of SPLM with STM illuminations used in and inspired by CHIRPT microscopy via the information content its data carry about the emitter location(s). More specifically, we provide the mathematical formalism of Fisher information (FI) and the Cramér-Rao lower bound (CRLB) associated with the location parameters of the emitter(s). The FI and CRLB are then numerically evaluated under different experimental assumptions to assess the effects of experimental parameters on localization precision. Last, we compare the single-pixel CRLB to that from camera-based single-molecule localization microscopy in the localization of a single emitter. We show that SPLM has several distinguishing characteristics that provide certain advantages, such as relatively constant CRLB over a very large illumination volume and improved CRLB for 3D localization due to the information coupling introduced by simultaneous modulations of the transverse axes.

Instantaneous 4D micro-particle image velocimetry (µPIV) via multifocal microscopy (MUM)

Multifocal microscopy (MUM), a technique to capture multiple fields of view (FOVs) from distinct axial planes simultaneously and on one camera, was used to perform micro-particle image velocimetry (µPIV) to reconstruct velocity and shear stress fields imposed by a liquid flowing around a cell. A diffraction based multifocal relay was used to capture images from three different planes with 630 nm axial spacing from which the axial positions of the flow-tracing particles were calculated using the image sharpness metric. It was shown that MUM can achieve an accuracy on the calculated velocity of around (0.52 ± 0.19) µm/s. Using fixed cells, MUM imaged the flow perturbations at sub-cellular level, which showed characteristics similar to those observed in the literature. Using live cells as an exemplar, MUM observed the effect of changing cell morphology on the local flow during perfusion. Compared to standard confocal laser scanning microscope, MUM offers a clear advantage in acquisition speed for µPIV (over 300 times faster). This is an important characteristic for rapidly evolving biological systems where there is the necessity to monitor in real time entire volumes to correlate the sample responses to the external forces.

Fundamental precision bounds for three-dimensional optical localization microscopy using self-interference digital holography

Localization based microscopy using self-interference digital holography (SIDH) provides three-dimensional (3D) positional information about point sources with nanometer scale precision. To understand the performance limits of SIDH, here we calculate the theoretical limit to localization precision for SIDH when designed with two different configurations. One configuration creates the hologram using a plane wave and a spherical wave while the second configuration creates the hologram using two spherical waves. We further compare the calculated precision bounds to the 3D single molecule localization precision from different Point Spread Functions. SIDH results in almost constant localization precision in all three dimensions for a 20 µm thick depth of field. For high signal-to-background ratio (SBR), SIDH on average achieves better localization precision. For lower SBR values, the large size of the hologram on the detector becomes a problem, and PSF models perform better.

Effect of Pixelation on the Parameter Estimation of Single Molecule Trajectories

The advent of single molecule microscopy has revolutionized biological investigations by providing a powerful tool for the study of intercellular and intracellular trafficking processes of protein molecules which was not available before through conventional microscopy. In practice, pixelated detectors are used to acquire the images of fluorescently labeled objects moving in cellular environments. Then, the acquired fluorescence microscopy images contain the numbers of the photons detected in each pixel, during an exposure time interval. Moreover, instead of having the exact locations of detection of the photons, we only know the pixel areas in which the photons impact the detector. These challenges make the analysis of single molecule trajectories, from pixelated images, a complex problem. Here, we investigate the effect of pixelation on the parameter estimation of single molecule trajectories. In particular, we develop a stochastic framework to calculate the maximum likelihood estimates of the parameters of a stochastic differential equation that describes the motion of the molecule in living cells. We also calculate the Fisher information matrix for this parameter estimation problem. The analytical results are complicated through the fact that the observation process in a microscope prohibits the use of standard Kalman filter type approaches. The analytical framework presented here is illustrated with examples of low photon count scenarios for which we rely on Monte Carlo methods to compute the associated probability distributions.

Design and simulation of a snapshot multi-focal interferometric microscope

Realizing both high temporal and spatial resolution across a large volume is a key challenge for 3D fluorescent imaging. Towards achieving this objective, we introduce an interferometric multifocus microscopy (iMFM) system, a combination of multifocus microscopy (MFM) with two opposing objective lenses. We show that the proposed iMFM is capable of simultaneously producing multiple focal plane interferometry that provides axial super-resolution and hence isotropic 3D resolution with a single exposure. We design and simulate the iMFM microscope by employing two special diffractive optical elements. The point spread function of this new iMFM microscope is simulated and the image formation model is given. For reconstruction, we use the Richardson-Lucy deconvolution algorithm with total variation regularization for 3D extended object recovery, and a maximum likelihood estimator (MLE) for single molecule tracking. A method for determining an initial axial position of the molecule is also proposed to improve the convergence of the MLE. We demonstrate both theoretically and numerically that isotropic 3D nanoscopic localization accuracy is achievable with an axial imaging range of 2um when tracking a fluorescent molecule in three dimensions and that the diffraction limited axial resolution can be improved by 3-4 times in the single shot wide-field 3D extended object recovery. We believe that iMFM will be a useful tool in 3D dynamic event imaging that requires both high temporal and spatial resolution.

Fidelity test for through-focus or volumetric type of optical imaging methods

Rapid increase in interest and applications of through-focus (TF) or volumetric type of optical imaging in biology and other areas has resulted in the development of several TF image collection methods. Achieving quantitative results from images requires standardization and optimization of image acquisition protocols. Several standardization protocols are available for conventional optical microscopy where a best-focus image is used, but to date, rigorous testing protocols do not exist for TF optical imaging. In this paper, we present a method to determine the fidelity of the TF optical data using the TF scanning optical microscopy images.

Through-focus or volumetric type of optical imaging methods: a review

In recent years, the use of through-focus (TF) or volumetric type of optical imaging has gained momentum in several areas such as biological imaging, microscopy, adaptive optics, material processing, optical data storage, and optical inspection. We provide a review of basic TF optical methods highlighting their design, major unique characteristics, and application space.

Three-Dimensional Localization of Single Molecules for Super-Resolution Imaging and Single-Particle Tracking

Single-molecule super-resolution fluorescence microscopy and single-particle tracking are two imaging modalities that illuminate the properties of cells and materials on spatial scales down to tens of nanometers or with dynamical information about nanoscale particle motion in the millisecond range, respectively. These methods generally use wide-field microscopes and two-dimensional camera detectors to localize molecules to much higher precision than the diffraction limit. Given the limited total photons available from each single-molecule label, both modalities require careful mathematical analysis and image processing. Much more information can be obtained about the system under study by extending to three-dimensional (3D) single-molecule localization: without this capability, visualization of structures or motions extending in the axial direction can easily be missed or confused, compromising scientific understanding. A variety of methods for obtaining both 3D super-resolution images and 3D tracking information have been devised, each with their own strengths and weaknesses. These include imaging of multiple focal planes, point-spread-function engineering, and interferometric detection. These methods may be compared based on their ability to provide accurate and precise position information on single-molecule emitters with limited photons. To successfully apply and further develop these methods, it is essential to consider many practical concerns, including the effects of optical aberrations, field dependence in the imaging system, fluorophore labeling density, and registration between different color channels. Selected examples of 3D super-resolution imaging and tracking are described for illustration from a variety of biological contexts and with a variety of methods, demonstrating the power of 3D localization for understanding complex systems.

Fisher information theory for parameter estimation in single molecule microscopy: tutorial

Estimation of a parameter of interest from image data represents a task that is commonly carried out in single molecule microscopy data analysis. The determination of the positional coordinates of a molecule from its image, for example, forms the basis of standard applications such as single molecule tracking and localization-based super-resolution image reconstruction. Assuming that the estimator used recovers, on average, the true value of the parameter, its accuracy, or standard deviation, is then at best equal to the square root of the Cramér-Rao lower bound. The Cramér-Rao lower bound can therefore be used as a benchmark in the evaluation of the accuracy of an estimator. Additionally, as its value can be computed and assessed for different experimental settings, it is useful as an experimental design tool. This tutorial demonstrates a mathematical framework that has been specifically developed to calculate the Cramér-Rao lower bound for estimation problems in single molecule microscopy and, more broadly, fluorescence microscopy. The material includes a presentation of the photon detection process that underlies all image data, various image data models that describe images acquired with different detector types, and Fisher information expressions that are necessary for the calculation of the lower bound. Throughout the tutorial, examples involving concrete estimation problems are used to illustrate the effects of various factors on the accuracy of parameter estimation and, more generally, to demonstrate the flexibility of the mathematical framework.

Deciphering the Structure and Function of Nuclear Pores Using Single-Molecule Fluorescence Approaches

Due to its central role in macromolecular trafficking and nucleocytoplasmic information transfer, the nuclear pore complex (NPC) has been studied in great detail using a wide spectrum of methods. Consequently, many aspects of its architecture, general function, and role in the life cycle of a cell are well understood. Over the last decade, fluorescence microscopy methods have enabled the real-time visualization of single molecules interacting with and transiting through the NPC, allowing novel questions to be examined with nanometer precision. While initial single-molecule studies focused primarily on import pathways using permeabilized cells, it has recently proven feasible to investigate the export of mRNAs in living cells. Single-molecule assays can address questions that are difficult or impossible to answer by other means, yet the complexity of nucleocytoplasmic transport requires that interpretation be based on a firm genetic, biochemical, and structural foundation. Moreover, conceptually simple single-molecule experiments remain technically challenging, particularly with regard to signal intensity, signal-to-noise ratio, and the analysis of noise, stochasticity, and precision. We discuss nuclear transport issues recently addressed by single-molecule microscopy, evaluate the limits of existing assays and data, and identify open questions for future studies. We expect that single-molecule fluorescence approaches will continue to be applied to outstanding nucleocytoplasmic transport questions, and that the approaches developed for NPC studies are extendable to additional complex systems and pathways within cells.

New results on the single molecule localization problem in two and three dimensions

Fluorescence microscopy is an optical microscopy technique which has been extensively used to study specifically- labeled subcellular objects, such as proteins, and their functions. The best possible accuracy with which an object of interest can be localized when imaged using a fluorescence microscope is typically calculated using the Cramer- Rao lower bound (CRLB). The calculation of the CRLB, however, so far relied on an analytical expression for the image of the object. This can pose challenges in practice since it is often difficult to find appropriate analytical models for the images of general objects. Even if an appropriate analytical model is available, the lack of knowledge about the precise values of imaging parameters might also impose difficulties in the calculation oxf the CRLB. To address these challenges, we have developed an approach that directly uses an experimentally collected image set to calculate the best possible localization accuracy for a general subcellular object in two and three dimensions. In this approach, we fit smoothly connected piecewise polynomials, known as splines, to the experimentally collected image set to provide a continuous model of the object. This continuous model can then be used for the calculation of the best possible localization accuracy.

Determination of localization accuracy based on experimentally acquired image sets: applications to single molecule microscopy

Fluorescence microscopy is a photon-limited imaging modality that allows the study of subcellular objects and processes with high specificity. The best possible accuracy (standard deviation) with which an object of interest can be localized when imaged using a fluorescence microscope is typically calculated using the Cramér-Rao lower bound, that is, the inverse of the Fisher information. However, the current approach for the calculation of the best possible localization accuracy relies on an analytical expression for the image of the object. This can pose practical challenges since it is often difficult to find appropriate analytical models for the images of general objects. In this study, we instead develop an approach that directly uses an experimentally collected image set to calculate the best possible localization accuracy for a general subcellular object. In this approach, we fit splines, i.e. smoothly connected piecewise polynomials, to the experimentally collected image set to provide a continuous model of the object, which can then be used for the calculation of the best possible localization accuracy. Due to its practical importance, we investigate in detail the application of the proposed approach in single molecule fluorescence microscopy. In this case, the object of interest is a point source and, therefore, the acquired image set pertains to an experimental point spread function.

3D multifocus astigmatism and compressed sensing (3D MACS) based superresolution reconstruction

Single molecule based superresolution techniques (STORM/PALM) achieve nanometer spatial resolution by integrating the temporal information of the switching dynamics of fluorophores (emitters). When emitter density is low for each frame, they are located to the nanometer resolution. However, when the emitter density rises, causing significant overlapping, it becomes increasingly difficult to accurately locate individual emitters. This is particularly apparent in three dimensional (3D) localization because of the large effective volume of the 3D point spread function (PSF). The inability to precisely locate the emitters at a high density causes poor temporal resolution of localization-based superresolution technique and significantly limits its application in 3D live cell imaging. To address this problem, we developed a 3D high-density superresolution imaging platform that allows us to precisely locate the positions of emitters, even when they are significantly overlapped in three dimensional space. Our platform involves a multi-focus system in combination with astigmatic optics and an ℓ 1-Homotopy optimization procedure. To reduce the intrinsic bias introduced by the discrete formulation of compressed sensing, we introduced a debiasing step followed by a 3D weighted centroid procedure, which not only increases the localization accuracy, but also increases the computation speed of image reconstruction. We implemented our algorithms on a graphic processing unit (GPU), which speeds up processing 10 times compared with central processing unit (CPU) implementation. We tested our method with both simulated data and experimental data of fluorescently labeled microtubules and were able to reconstruct a 3D microtubule image with 1000 frames (512×512) acquired within 20 seconds.

An information-theoretic approach to designing the plane spacing for multifocal plane microscopy

Multifocal plane microscopy (MUM) is a 3D imaging modality which enables the localization and tracking of single molecules at high spatial and temporal resolution by simultaneously imaging distinct focal planes within the sample. MUM overcomes the depth discrimination problem of conventional microscopy and allows high accuracy localization of a single molecule in 3D along the z-axis. An important question in the design of MUM experiments concerns the appropriate number of focal planes and their spacings to achieve the best possible 3D localization accuracy along the z-axis. Ideally, it is desired to obtain a 3D localization accuracy that is uniform over a large depth and has small numerical values, which guarantee that the single molecule is continuously detectable. Here, we address this concern by developing a plane spacing design strategy based on the Fisher information. In particular, we analyze the Fisher information matrix for the 3D localization problem along the z-axis and propose spacing scenarios termed the strong coupling and the weak coupling spacings, which provide appropriate 3D localization accuracies. Using these spacing scenarios, we investigate the detectability of the single molecule along the z-axis and study the effect of changing the number of focal planes on the 3D localization accuracy. We further review a software module we recently introduced, the MUMDesignTool, that helps to design the plane spacings for a MUM setup.