Generalized recovery algorithm for 3D super-resolution microscopy using rotating point spread functions

LocNet: deep learning-based localization on a rotating point spread function with applications to telescope imaging

Three-dimensional (3D) point source recovery from two-dimensional (2D) data is a challenging problem with wide-ranging applications in single-molecule localization microscopy and space-debris localization telescops. Point spread function (PSF) engineering is a promising technique to solve this 3D localization problem. Specifically, we consider the problem of 3D localization of space debris from a 2D image using a rotating PSF where the depth information is encoded in the angle of rotation of a single-lobe PSF for each point source. Instead of applying a model-based optimization, we introduce a convolution neural network (CNN)-based approach to localize space debris in full 3D space automatically. A hard sample training strategy is proposed to improve the performance of CNN further. Contrary to the traditional model-based methods, our technique is efficient and outperforms the current state-of-the-art method by more than 11% in the precision rate with a comparable improvement in the recall rate.

Imaging Heterogeneous 3D Dynamics of Individual Solutes in a Polyelectrolyte Brush

Understanding molecular transport in polyelectrolyte brushes (PEBs) is crucial for applications such as separations, drug delivery, anti-fouling, and biosensors, where structural features of the polymer control intermolecular interactions. The complex structure and local heterogeneity of PEBs, while theoretically predicted, are not easily accessed with conventional experimental methods. In this work, we use 3D single-molecule tracking to understand transport behavior within a cationic poly(2-(N,N-dimethylamino)ethyl acrylate) (PDMAEA) brush using an anionic dye, Alexa Fluor 546, as the probe. The analysis is done by a parallelized, unbiased 3D tracking algorithm. Our results explicitly demonstrate that spatial heterogeneity within the brush manifests as heterogeneity of single-molecule displacements. Two distinct populations of probe motion are identified, with anticorrelated axial and lateral transport confinement, which we believe to correspond to intra- vs inter-chain probe motion.

Three-dimensional tracking using a single-spot rotating point spread function created by a multiring spiral phase plate

SIGNIFICANCE

Three-dimensional (3D) imaging and object tracking is critical for medical and biological research and can be achieved by multifocal imaging with diffractive optical elements (DOEs) converting depth ( z ) information into a modification of the two-dimensional image. Physical insight into DOE designs will spur this expanding field.

AIM

To precisely track microscopic fluorescent objects in biological systems in 3D with a simple low-cost DOE system.

APPROACH

We designed a multiring spiral phase plate (SPP) generating a single-spot rotating point spread function (SS-RPSF) in a microscope. Our simple, analytically transparent design process uses Bessel beams to avoid rotational ambiguities and achieve a significant depth range. The SPP was inserted into the Nomarski prism slider of a standard microscope. Performance was evaluated using fluorescent beads and in live cells expressing a fluorescent chromatin marker.

RESULTS

Bead localization precision was < 25    nm in the transverse dimensions and ≤ 70    nm along the axial dimension over an axial range of 6    μ m . Higher axial precision ( ≤ 50    nm ) was achieved over a shallower focal depth of 2.9    μ m . 3D diffusion constants of chromatin matched expected values.

CONCLUSIONS

Precise 3D localization and tracking can be achieved with a SS-RPSF SPP in a standard microscope with minor modifications.

High speed two-photon laser scanning stereomicroscopy for three-dimension tracking multiple particles simultaneously in three-dimension

In this paper, we will describe a video rate two-photon laser scanning stereomicroscopy for imaging-based three-dimensional particle tracking. Using a resonant galvanometer, we have now achieved 30 volumes per second (frame size 512 × 512) in volumetric imaging. Owing to the pulse multiplexing and demultiplexing techniques, the system does not suffer the speed loss for taking two parallax views of a volume. The switching time between left and right views is reduced to several nanoseconds. The extremely fast view switching and high volumetric imaging speed allow us to track fast transport processes of nanoparticles in deep light-scattering media. For instance, in 1%-intralipid solution and fibrillar scaffolds, the tracking penetration depth can be around 400 μm.

On-the-fly compressive single-pixel foveation using the STOne transform

Compressive imaging allows one to sample an image below the Nyquist rate yet still accurately recover it from the measurements by solving an L1 optimization problem. The L1 solvers, however, are iterative and can require significant time to reconstruct the original signal. Intuitively, the reconstruction time can be reduced by reconstructing fewer total pixels. The human eye reduces the total amount of data it processes by having a spatially varying resolution, a method called foveation. In this work, we use foveation to achieve a 4x improvement in L1 compressive sensing reconstruction speed for hyperspectral images and video. Unlike previous works, the presented technique allows the high-resolution region to be placed anywhere in the scene after the subsampled measurements have been acquired, has no moving parts, and is entirely non-adaptive.

Untying the Gordian KNOT: Unbiased Single Particle Tracking Using Point Clouds and Adaptive Motion Analysis

Achieving mechanistic understanding of transport in complex environments such as inside cells or at polymer interfaces is challenging. We need better ways to image transport in 3-D and better single particle tracking algorithms to determine transport that are not systemically biased toward any classical motion model. Here we present an unbiased single particle tracking algorithm: Knowing Nothing Outside Tracking (KNOT). KNOT uses point clouds provided by iterative deconvolution to educate individual particle localizations and link particle positions between frames to achieve 2-D and 3-D tracking. Information from prior point clouds fuels an independent adaptive motion model for each particle to avoid global models that could introduce biases. KNOT competes with or surpasses other 2-D methods from the 2012 particle tracking challenge while accurately tracking adsorption dynamics of proteins on polymer surfaces and early endosome transport in live cells in 3-D. We apply KNOT to study 3-D endosome transport to reveal new physical insight into locally directed and diffusive transport in live cells. Our analysis demonstrates better accuracy in classifying local motion and its direction compared to previous methods, revealing intricate intracellular transport heterogeneities.

Super-resolution Microscopy for Biological Imaging

Studying the ultra-fine structures and functions of the subcellular organelles and exploring the dynamic biological events in depth are the key issues in contemporary biological research. Fluorescence bio-imaging has been used to study cell biology for decades. However, the structures and functions of the subcellular organelles which fall under the diffraction limit are still not explored fully at a nanoscale level. Several super-resolution microscopy (SRM) techniques have been devised over the years which can be utilized to overcome diffraction limit. These techniques have opened a new window in biological research. However, SRM methods are highly vulnerable to the lack of appropriate fluorophores and other sophisticated technical considerations. Therefore, this chapter briefly summarizes the basic principles of various SRM methods which have been frequently utilized in biological imaging. The chapter not only gives an overview of the technical advantages and drawbacks about using different SRM techniques for bio-imaging applications but also briefly articulates the nitty-gritties of selecting a proper fluorescent probe for a specific SRM experiment with biological samples.

A mechanistic examination of salting out in protein-polymer membrane interactions

Developing a mechanistic understanding of protein dynamics and conformational changes at polymer interfaces is critical for a range of processes including industrial protein separations. Salting out is one example of a procedure that is ubiquitous in protein separations yet is optimized empirically because there is no mechanistic description of the underlying interactions that would allow predictive modeling. Here, we investigate peak narrowing in a model transferrin-nylon system under salting out conditions using a combination of single-molecule tracking and ensemble separations. Distinct surface transport modes and protein conformational changes at the negatively charged nylon interface are quantified as a function of salt concentration. Single-molecule kinetics relate macroscale improvements in chromatographic peak broadening with microscale distributions of surface interaction mechanisms such as continuous-time random walks and simple adsorption-desorption. Monte Carlo simulations underpinned by the stochastic theory of chromatography are performed using kinetic data extracted from single-molecule observations. Simulations agree with experiment, revealing a decrease in peak broadening as the salt concentration increases. The results suggest that chemical modifications to membranes that decrease the probability of surface random walks could reduce peak broadening in full-scale protein separations. More broadly, this work represents a proof of concept for combining single-molecule experiments and a mechanistic theory to improve costly and time-consuming empirical methods of optimization.

Joint 3D localization and classification of space debris using a multispectral rotating point spread function

We consider the problem of joint three-dimensional (3D) localization and material classification of unresolved space debris using a multispectral rotating point spread function (RPSF). The use of RPSF allows one to estimate the 3D locations of point sources from their rotated images acquired by a single 2D sensor array, since the amount of rotation of each source image about its x, y location depends on its axial distance z. Using multispectral images, with one RPSF per spectral band, we are able not only to localize the 3D positions of the space debris but also classify their material composition. We propose a three-stage method for achieving joint localization and classification. In stage 1, we adopt an optimization scheme for localization in which the spectral signature of each material is assumed to be uniform, which significantly improves efficiency and yields better localization results than possible with a single spectral band. In stage 2, we estimate the spectral signature and refine the localization result via an alternating approach. We process classification in the final stage. Both Poisson noise and Gaussian noise models are considered, and the implementation of each is discussed. Numerical tests using multispectral data from NASA show the efficiency of our three-stage approach and illustrate the improvement of point source localization and spectral classification from using multiple bands over a single band.

Generalized method to design phase masks for 3D super-resolution microscopy

Point spread function (PSF) engineering by phase modulation is a novel approach to three-dimensional (3D) super-resolution microscopy, with different point spread functions being proposed for specific applications. It is often not easy to achieve the desired shape of engineered point spread functions because it is challenging to determine the correct phase mask. Additionally, a phase mask can either encode 3D space information or additional time information, but not both simultaneously. A robust algorithm for recovering a phase mask to generate arbitrary point spread functions is needed. In this work, a generalized phase mask design method is introduced by performing an optimization. A stochastic gradient descent algorithm and a Gauss-Newton algorithm are developed and compared for their ability to recover the phase masks for previously reported point spread functions. The new Gauss-Newton algorithm converges to a minimum at much higher speeds. This algorithm is used to design a novel stretching-lobe phase mask to encode temporal and 3D spatial information simultaneously. The stretching-lobe phase mask and other masks are fabricated in-house for proof-of-concept using multi-level light lithography and an optimized commercially sourced stretching-lobe phase mask (PM) is validated experimentally to encode 3D spatial and temporal information. The algorithms' generalizability is further demonstrated by generating a phase mask that comprises four different letters at different depths.

Three-dimensional super-resolved live cell imaging through polarized multi-angle TIRF

Measuring three-dimensional nanoscale cellular structures is challenging, especially when the structure is dynamic. Owing to the informative total internal reflection fluorescence (TIRF) imaging under varied illumination angles, multi-angle (MA) TIRF has been examined to offer a nanoscale axial and a subsecond temporal resolution. However, conventional MA-TIRF still performs badly in lateral resolution and fails to characterize the depth image in densely distributed regions. Here, we emphasize the lateral super-resolution in the MA-TIRF, exampled by simply introducing polarization modulation into the illumination procedure. Equipped with a sparsity and accelerated proximal algorithm, we examine a more precise 3D sample structure compared with previous methods, enabling live cell imaging with a temporal resolution of 2 s and recovering high-resolution mitochondria fission and fusion processes. We also shared the recovery program, which is the first open-source recovery code for MA-TIRF, to the best of our knowledge.

Enhancing Analytical Separations Using Super-Resolution Microscopy

Super-resolution microscopy is becoming an invaluable tool to investigate structure and dynamics driving protein interactions at interfaces. In this review, we highlight the applications of super-resolution microscopy for quantifying the physics and chemistry that occur between target proteins and stationary-phase supports during chromatographic separations. Our discussion concentrates on the newfound ability of super-resolved single-protein spectroscopy to inform theoretical parameters via quantification of adsorption-desorption dynamics, protein unfolding, and nanoconfined transport.

Maximizing the field of view and accuracy in 3D Single Molecule Localization Microscopy

Super-resolution techniques that localize single molecules in three dimensions through point spread function (PSF) engineering are very sensitive to aberrations and optical alignment. Here we show how double-helix point spread function is affected by such mis-alignment and aberration. Specifically, we demonstrate through simulation and experiment how misplacement of phase masks in infinity corrected systems is a common source of significant loss of accuracy. We also describe an optimal alignment and calibration procedure to correct for these errors. In combination, these optimizations allow for a maximal field of view with high accuracy and precision. Though discussed with reference to double-helix point spread function (DHPSF), the optimization techniques are equally applicable to other engineered PSFs.

Variable Lysozyme Transport Dynamics on Oxidatively Functionalized Polystyrene Films

Tuning protein adsorption dynamics at polymeric interfaces is of great interest to many biomedical and material applications. Functionalization of polymer surfaces is a common method to introduce application-specific surface chemistries to a polymer interface. In this work, single-molecule fluorescence microscopy is utilized to determine the adsorption dynamics of lysozyme, a well-studied antibacterial protein, at the interface of polystyrene oxidized via UV exposure and oxygen plasma and functionalized by ligand grafting to produce varying degrees of surface hydrophilicity, surface roughness, and induced oxygen content. Single-molecule tracking indicates lysozyme loading capacities, and surface mobility at the polymer interface is hindered as a result of all functionalization techniques. Adsorption dynamics of lysozyme depend on the extent and the specificity of the oxygen functionalities introduced to the polystyrene surface. Hindered adsorption and mobility are dominated by hydrophobic effects attributed to water hydration layer formation at the functionalized polystyrene surfaces.

Three-Dimensional Localization of Single Molecules for Super-Resolution Imaging and Single-Particle Tracking

Single-molecule super-resolution fluorescence microscopy and single-particle tracking are two imaging modalities that illuminate the properties of cells and materials on spatial scales down to tens of nanometers or with dynamical information about nanoscale particle motion in the millisecond range, respectively. These methods generally use wide-field microscopes and two-dimensional camera detectors to localize molecules to much higher precision than the diffraction limit. Given the limited total photons available from each single-molecule label, both modalities require careful mathematical analysis and image processing. Much more information can be obtained about the system under study by extending to three-dimensional (3D) single-molecule localization: without this capability, visualization of structures or motions extending in the axial direction can easily be missed or confused, compromising scientific understanding. A variety of methods for obtaining both 3D super-resolution images and 3D tracking information have been devised, each with their own strengths and weaknesses. These include imaging of multiple focal planes, point-spread-function engineering, and interferometric detection. These methods may be compared based on their ability to provide accurate and precise position information on single-molecule emitters with limited photons. To successfully apply and further develop these methods, it is essential to consider many practical concerns, including the effects of optical aberrations, field dependence in the imaging system, fluorophore labeling density, and registration between different color channels. Selected examples of 3D super-resolution imaging and tracking are described for illustration from a variety of biological contexts and with a variety of methods, demonstrating the power of 3D localization for understanding complex systems.

Single Particle Tracking: From Theory to Biophysical Applications

After three decades of developments, single particle tracking (SPT) has become a powerful tool to interrogate dynamics in a range of materials including live cells and novel catalytic supports because of its ability to reveal dynamics in the structure-function relationships underlying the heterogeneous nature of such systems. In this review, we summarize the algorithms behind, and practical applications of, SPT. We first cover the theoretical background including particle identification, localization, and trajectory reconstruction. General instrumentation and recent developments to achieve two- and three-dimensional subdiffraction localization and SPT are discussed. We then highlight some applications of SPT to study various biological and synthetic materials systems. Finally, we provide our perspective regarding several directions for future advancements in the theory and application of SPT.

Super Temporal-Resolved Microscopy (STReM)

Super-resolution microscopy typically achieves high spatial resolution, but the temporal resolution remains low. We report super temporal-resolved microscopy (STReM) to improve the temporal resolution of 2D super-resolution microscopy by a factor of 20 compared to that of the traditional camera-limited frame rate. This is achieved by rotating a phase mask in the Fourier plane during data acquisition and then recovering the temporal information by fitting the point spread function (PSF) orientations. The feasibility of this technique is verified with both simulated and experimental 2D adsorption/desorption and 2D emitter transport. When STReM is applied to measure protein adsorption at a glass surface, previously unseen dynamics are revealed.