A stochastic model for electron multiplication charge-coupled devices--from theory to practice

Photophysical image analysis: Unsupervised probabilistic thresholding for images from electron-multiplying charge-coupled devices

We introduce the concept photophysical image analysis (PIA) and an associated pipeline for unsupervised probabilistic image thresholding for images recorded by electron-multiplying charge-coupled device (EMCCD) cameras. We base our approach on a closed-form analytic expression for the characteristic function (Fourier-transform of the probability mass function) for the image counts recorded in an EMCCD camera, which takes into account both stochasticity in the arrival of photons at the imaging camera and subsequent noise induced by the detection system of the camera. The only assumption in our method is that the background photon arrival to the imaging system is described by a stationary Poisson process (we make no assumption about the photon statistics for the signal). We estimate the background photon statistics parameter, λbg, from an image which contains both background and signal pixels by use of a novel truncated fit procedure with an automatically determined image count threshold. Prior to this, the camera noise model parameters are estimated using a calibration step. Utilizing the estimates for the camera parameters and λbg, we then introduce a probabilistic thresholding method, where, for the first time, the fraction of misclassified pixels can be determined a priori for a general image in an unsupervised way. We use synthetic images to validate our a priori estimates and to benchmark against the Otsu method, which is a popular unsupervised non-probabilistic image thresholding method (no a priori estimates for the error rates are provided). For completeness, we lastly present a simple heuristic general-purpose segmentation method based on the thresholding results, which we apply to segmentation of synthetic images and experimental images of fluorescent beads and lung cell nuclei. Our publicly available software opens up for fully automated, unsupervised, probabilistic photophysical image analysis.

A Structured Illumination Microscopy Framework with Spatial-Domain Noise Propagation

Biological images captured by a microscope are characterized by heterogeneous signal-to-noise ratios (SNRs) across the field of view due to spatially varying photon emission and camera noise. State-of-the-art unsupervised structured illumination microscopy (SIM) reconstruction algorithms, commonly implemented in the Fourier domain, do not accurately model this noise and suffer from high-frequency artifacts, user-dependent choices of smoothness constraints making assumptions on biological features, and unphysical negative values in the recovered fluorescence intensity map. On the other hand, supervised methods rely on large datasets for training, and often require retraining for new sample structures. Consequently, achieving high contrast near the maximum theoretical resolution in an unsupervised, physically principled manner remains a challenging task. Here, we propose Bayesian-SIM (B-SIM), an unsupervised Bayesian framework to quantitatively reconstruct SIM data, rectifying these shortcomings by accurately incorporating all noise sources in the spatial domain. To accelerate the reconstruction process for computational feasibility, we devise a parallelized Monte Carlo sampling strategy for inference. We benchmark our framework on both simulated and experimental images, and demonstrate improved contrast permitting feature recovery at up to 25% shorter length scales over state-of-the-art methods at both high- and low-SNR. B-SIM enables unsupervised, quantitative, physically accurate reconstruction without the need for labeled training data, democratizing high-quality SIM reconstruction and expands the capabilities of live-cell SIM to lower SNR, potentially revealing biological features in previously inaccessible regimes.

Short-wave Infrared Photoluminescence Lifetime Mapping of Rare-Earth Doped Nanoparticles Using All-Optical Streak Imaging

The short-wave infrared (SWIR) photoluminescence lifetimes of rare-earth doped nanoparticles (RENPs) have found diverse applications in fundamental and applied research. Despite dazzling progress in the novel design and synthesis of RENPs with attractive optical properties, existing optical systems for SWIR photoluminescence lifetime imaging are still considerably restricted by inefficient photon detection, limited imaging speed, and low sensitivity. To overcome these challenges, SWIR photoluminescence lifetime imaging microscopy using an all-optical streak camera (PLIMASC) is developed. Synergizing scanning optics and a high-sensitivity InGaAs CMOS camera, SWIR-PLIMASC has a 1D imaging speed of up to 138.9 kHz in the spectral range of 900-1700 nm, which quantifies the photoluminescence lifetime of RENPs in a single shot. A 2D photoluminescence lifetime map can be acquired by 1D scanning of the sample. To showcase the power of SWIR-PLIMASC, a series of core-shell RENPs with distinct SWIR photoluminescence lifetimes is synthesized. In particular, using Er3+ -doped RENPs, SWIR-PLIMASC enables multiplexed anti-counterfeiting. Leveraging Ho3+ -doped RENPs as temperature indicators, this system is applied to SWIR photoluminescence lifetime-based thermometry. Opening up a new avenue for efficient SWIR photoluminescence lifetime mapping, this work is envisaged to contribute to advanced materials characterization, information science, and biomedicine.

Physically informed Monte Carlo simulation of dual-wedge prism-based spectroscopic single-molecule localization microscopy

Significance

The dual-wedge prism (DWP)-based spectroscopic single-molecule localization microscopy (sSMLM) system offers improved localization precision and adjustable spectral or localization performance, but its nonlinear spectral dispersion presents a challenge. A systematic method can help understand the challenges and thereafter optimize the DWP system's performance by customizing the system parameters to maximize the spectral or localization performance for various molecular labels.

Aim

We developed a Monte Carlo (MC)-based model that predicts the imaging output of the DWP-based sSMLM system given different system parameters.

Approach

We assessed our MC model's localization and spectral precisions by comparing our simulation against theoretical equations and fluorescent microspheres. Furthermore, we simulated the DWP-based system using beamsplitters (BSs) with a reflectance (R):transmittance (T) of R50:T50 and R30:T70 and their tradeoffs.

Results

Our MC simulation showed average deviations of 2.5 and 2.1 nm for localization and spectral precisions against theoretical equations and 2.3 and 1.0 nm against fluorescent microspheres. An R30:T70 BS improved the spectral precision by 8% but worsened the localization precision by 35% on average compared with an R50:T50 BS.

Conclusions

The MC model accurately predicted the localization precision, spectral precision, spectral peaks, and spectral widths of fluorescent microspheres, as validated by experimental data. Our work enhances the theoretical understanding of DWP-based sSMLM for multiplexed imaging, enabling performance optimization.

Fluorescence Microscopy: a statistics-optics perspective

Fundamental properties of light unavoidably impose features on images collected using fluorescence microscopes. Modeling these features is ever more important in quantitatively interpreting microscopy images collected at scales on par or smaller than light's wavelength. Here we review the optics responsible for generating fluorescent images, fluorophore properties, microscopy modalities leveraging properties of both light and fluorophores, in addition to the necessarily probabilistic modeling tools imposed by the stochastic nature of light and measurement.

Comprehensive study of Raman optical response of typical substrates for thin-film growth under 633 nm and 785 nm laser excitation

Raman spectroscopy is one of the most efficient and non-destructive techniques for characterizing materials. However, it is challenging to analyze thin films using Raman spectroscopy since the substrates beneath the thin film often obscure its optical response. Here, we evaluate the suitability of fourteen commonly employed single-crystal substrates for Raman spectroscopy of thin films using 633 nm and 785 nm laser excitation systems. We determine the optimal wavenumber ranges for thin-film characterization by identifying the most prominent Raman peaks and their relative intensities for each substrate and across substrates. In addition, we compare the intensity of background signals across substrates, which is essential for establishing their applicability for Raman detection in thin films. The substrates LaAlO3 and Al2O3 have the largest free spectral range for both laser systems, while Al2O3 has the lowest background levels, according to our findings. In contrast, the substrates SrTiO3 and Nb:SrTiO3 have the narrowest free spectral range, while GdScO3, NGO and MgO have the highest background levels, making them unsuitable for optical investigations.

Performance assessment of probe-based Raman spectroscopy systems for biomedical analysis

We present a methodology for evaluating the performance of probe-based Raman spectroscopy systems for biomedical analysis. This procedure uses a biological standard sample and data analysis approach to circumvent many of the issues related to accurately measuring and comparing the signal quality of Raman spectra between systems. Dairy milk is selected as the biological standard due to its similarity to tissue spectral properties and because its homogeneity eliminates the dependence of probe orientation on the measured spectrum. A spectral dataset is first collected from milk for each system configuration, followed by a model-based correction step to remove photobleaching artifacts and accurately calculate SNR. Results demonstrate that the proposed strategy, unlike current methods, produces an experimental SNR that agrees with the theoretical value. Four preconfigured imaging spectrographs that share similar manufacturer specifications were compared, showing that their capabilities to detect biological Raman spectra widely differ in terms of throughput and stray light rejection. While the methodology is used to compare spectrographs in this case, it can be adapted for other purposes, such as optimizing the design of a custom-built Raman spectrometer, evaluating inter-probe variability, or examining how altering system subcomponents affects signal quality.

BNP-Track: A framework for superresolved tracking

Assessing dynamic processes at single molecule scales is key toward capturing life at the level of its molecular actors. Widefield superresolution methods, such as STORM, PALM, and PAINT, provide nanoscale localization accuracy, even when distances between fluorescently labeled single molecules ("emitters") fall below light's diffraction limit. However, as these superresolution methods rely on rare photophysical events to distinguish emitters from both each other and background, they are largely limited to static samples. In contrast, here we leverage spatiotemporal correlations of dynamic widefield imaging data to extend superresolution to simultaneous multiple emitter tracking without relying on photodynamics even as emitter distances from one another fall below the diffraction limit. We simultaneously determine emitter numbers and their tracks (localization and linking) with the same localization accuracy per frame as widefield superresolution does for immobilized emitters under similar imaging conditions (≈50nm). We demonstrate our results for both in cellulo data and, for benchmarking purposes, on synthetic data. To this end, we avoid the existing tracking paradigm relying on completely or partially separating the tasks of emitter number determination, localization of each emitter, and linking emitter positions across frames. Instead, we develop a fully joint posterior distribution over the quantities of interest, including emitter tracks and their total, otherwise unknown, number within the Bayesian nonparametric paradigm. Our posterior quantifies the full uncertainty over emitter numbers and their associated tracks propagated from origins including shot noise and camera artefacts, pixelation, stochastic background, and out-of-focus motion. Finally, it remains accurate in more crowded regimes where alternative tracking tools cannot be applied.

Towards Unbiased Fluorophore Counting in Superresolution Fluorescence Microscopy

With the advent of fluorescence superresolution microscopy, nano-sized structures can be imaged with a previously unprecedented accuracy. Therefore, it is rapidly gaining importance as an analytical tool in the life sciences and beyond. However, the images obtained so far lack an absolute scale in terms of fluorophore numbers. Here, we use, for the first time, a detailed statistical model of the temporal imaging process which relies on a hidden Markov model operating on two timescales. This allows us to extract this information from the raw data without additional calibration measurements. We show this on the basis of added data from experiments on single Alexa 647 molecules as well as GSDIM/dSTORM measurements on DNA origami structures with a known number of labeling positions.

Exascale integrated modeling of low-order wavefront sensing and control for the Roman Coronagraph instrument

Astronomical instruments to detect exoplanets require extreme wavefront stability. For these missions to succeed, comprehensive and precise modeling is required to design and analyze suitable coronagraphs and their wavefront control systems. In this paper, we describe techniques for integrated modeling at scale that is, to the best of our knowledge, 1000 times faster than previously published works. We show how this capability has been used to validate performance and perform uncertainty quantification for the Roman Coronagraph instrument. Finally, we show how this modeling capacity may be necessary to design and build the next generation of space-based coronagraph instruments.

Inferring potential landscapes from noisy trajectories of particles within an optical feedback trap

While particle trajectories encode information on their governing potentials, potentials can be challenging to robustly extract from trajectories. Measurement errors may corrupt a particle’s position, and sparse sampling of the potential limits data in higher energy regions such as barriers. We develop a Bayesian method to infer potentials from trajectories corrupted by Markovian measurement noise without assuming prior functional form on the potentials. As an alternative to Gaussian process priors over potentials, we introduce structured kernel interpolation to the Natural Sciences which allows us to extend our analysis to large datasets. Structured-Kernel-Interpolation Priors for Potential Energy Reconstruction (SKIPPER) is validated on 1D and 2D experimental trajectories for particles in a feedback trap.

•

A feedback trap was used to generate noisy Langevin microbead trajectories

•

The potential energy surface is recovered using a Bayesian formulation

•

The formulation uses a structured-kernel-interpolation Gaussian process (SKI-GP) to tractably approximate Gaussian process regression for larger datasets

•

Thanks to our adaptation of SKI-GP, we have broadened the use of Gaussian processes for natural science applications

Modeling Non-additive Effects in Neighboring Chemically Identical Fluorophores

Quantitative fluorescence analysis is often used to derive chemical properties, including stoichiometries, of biomolecular complexes. One fundamental underlying assumption in the analysis of fluorescence data─whether it be the determination of protein complex stoichiometry by super-resolution, or step-counting by photobleaching, or the determination of RNA counts in diffraction-limited spots in RNA fluorescence in situ hybridization (RNA-FISH) experiments─is that fluorophores behave identically and do not interact. However, recent experiments on fluorophore-labeled DNA origami structures such as fluorocubes have shed light on the nature of the interactions between identical fluorophores as these are brought closer together, thereby raising questions on the validity of the modeling assumption that fluorophores do not interact. Here, we analyze photon arrival data under pulsed illumination from fluorocubes where distances between dyes range from 2 to 10 nm. We discuss the implications of non-additivity of brightness on quantitative fluorescence analysis.

Impact of Noise and Background on Measurement Uncertainties in Luminescence Thermometry

Materials with temperature-dependent luminescence can be used as local thermometers when incorporated in, for example, a biological environment or chemical reactor. Researchers have continuously developed new materials aiming for the highest sensitivity of luminescence to temperature. Although the comparison of luminescent materials based on their temperature sensitivity is convenient, this parameter gives an incomplete description of the potential performance of the materials in applications. Here, we demonstrate how the precision of a temperature measurement with luminescent nanocrystals depends not only on the temperature sensitivity of the nanocrystals but also on their luminescence strength compared to measurement noise and background signal. After first determining the noise characteristics of our instrumentation, we show how the uncertainty of a temperature measurement can be predicted quantitatively. Our predictions match the temperature uncertainties that we extract from repeated measurements, over a wide temperature range (303-473 K), for different CCD readout settings, and for different background levels. The work presented here is the first study that incorporates all of these practical issues to accurately calculate the uncertainty of luminescent nanothermometers. This method will be important for the optimization and development of luminescent nanothermometers.

Diffraction-limited molecular cluster quantification with Bayesian nonparametrics

Life’s fundamental processes involve multiple molecules operating in close proximity within cells. To probe the composition and kinetics of molecular clusters confined within small (diffraction-limited) regions, experiments often report on the total fluorescence intensity simultaneously emitted from labeled molecules confined to such regions. Methods exist to enumerate total fluorophore numbers (e.g., step counting by photobleaching). However, methods aimed at step counting by photobleaching cannot treat photophysical dynamics in counting nor learn their associated kinetic rates. Here we propose a method to simultaneously enumerate fluorophores and determine their individual photophysical state trajectories. As the number of active (fluorescent) molecules at any given time is unknown, we rely on Bayesian nonparametrics and use specialized Monte Carlo algorithms to derive our estimates. Our formulation is benchmarked on synthetic and real data sets. While our focus here is on photophysical dynamics (in which labels transition between active and inactive states), such dynamics can also serve as a proxy for other types of dynamics such as assembly and disassembly kinetics of clusters. Similarly, while we focus on the case where all labels are initially fluorescent, other regimes, more appropriate to photoactivated localization microscopy, where fluorophores are instantiated in a non-fluorescent state, fall within the scope of the framework. As such, we provide a complete and versatile framework for the interpretation of complex time traces arising from the simultaneous activity of up to 100 fluorophores.

Rapid ensemble measurement of protein diffusion and probe blinking dynamics in cells

We present a fluorescence fluctuation image correlation analysis method that can rapidly and simultaneously measure the diffusion coefficient, photoblinking rates, and fraction of diffusing particles of fluorescent molecules in cells. Unlike other image correlation techniques, we demonstrated that our method could be applied irrespective of a nonuniformly distributed, immobile blinking fluorophore population. This allows us to measure blinking and transport dynamics in complex cell morphologies, a benefit for a range of super-resolution fluorescence imaging approaches that rely on probe emission blinking. Furthermore, we showed that our technique could be applied without directly accounting for photobleaching. We successfully employed our technique on several simulations with realistic EMCCD noise and photobleaching models, as well as on Dronpa-C12-labeled β-actin in living NIH/3T3 and HeLa cells. We found that the diffusion coefficients measured using our method were consistent with previous literature values. We further found that photoblinking rates measured in the live HeLa cells varied as expected with changing excitation power.

High-axial-resolution single-molecule localization under dense excitation with a multi-channel deep U-Net

Single-molecule localization microscopy (SMLM) can bypass the diffraction limit of optical microscopes and greatly improve the resolution in fluorescence microscopy. By introducing the point spread function (PSF) engineering technique, we can customize depth varying PSF to achieve higher axial resolution. However, most existing 3D single-molecule localization algorithms require excited fluorescent molecules to be sparse and captured at high signal-to-noise ratios, which results in a long acquisition time and precludes SMLM's further applications in many potential fields. To address this problem, we propose a novel 3D single-molecular localization method based on a multi-channel neural network based on U-Net. By leveraging the deep network's great advantages in feature extraction, the proposed network can reliably discriminate dense fluorescent molecules with overlapped PSFs and corrupted by sensor noise. Both simulated and real experiments demonstrate its superior performance in PSF engineered microscopes with short exposure and dense excitations, which holds great potential in fast 3D super-resolution microscopy.

Estimating the dynamic range of quantitative single-molecule localization microscopy

In recent years, there have been significant advances in quantifying molecule copy number and protein stoichiometry with single-molecule localization microscopy (SMLM). However, as the density of fluorophores per diffraction-limited spot increases, distinguishing between detection events from different fluorophores becomes progressively more difficult, affecting the accuracy of such measurements. Although essential to the design of quantitative experiments, the dynamic range of SMLM counting techniques has not yet been studied in detail. Here, we provide a working definition of the dynamic range for quantitative SMLM in terms of the relative number of missed localizations or blinks and explore the photophysical and experimental parameters that affect it. We begin with a simple two-state model of blinking fluorophores, then extend the model to incorporate photobleaching and temporal binning by the detection camera. From these models, we first show that our estimates of the dynamic range agree with realistic simulations of the photoswitching. We find that the dynamic range scales inversely with the duty cycle when counting both blinks and localizations. Finally, we validate our theoretical approach on direct stochastic optical reconstruction microscopy (dSTORM) data sets of photoswitching Alexa Fluor 647 dyes. Our results should help guide researchers in designing and implementing SMLM-based molecular counting experiments.

A gain series method for accurate EMCCD calibration

Calibration of the gain and digital conversion factor of an EMCCD is necessary for accurate photon counting. We present a new method to quickly calibrate multiple gain settings of an EMCCD camera. Acquiring gain-series calibration data and analyzing the resulting images with the EMCCD noise model more accurately estimates the gain response of the camera. Furthermore, we develop a method to compare the results from different calibration approaches. Gain-series calibration outperforms all other methods in this self-consistency test.

Extraction of rapid kinetics from smFRET measurements using integrative detectors

Hidden Markov models (HMMs) are used to learn single-molecule kinetics across a range of experimental techniques. By their construction, HMMs assume that single-molecule events occur on slower timescales than those of data acquisition. To move beyond that HMM limitation and allow for single-molecule events to occur on any timescale, we must treat single-molecule events in continuous time as they occur in nature. We propose a method to learn kinetic rates from single-molecule Förster resonance energy transfer (smFRET) data collected by integrative detectors, even if those rates exceed data acquisition rates. To achieve that, we exploit our recently proposed "hidden Markov jump process" (HMJP), with which we learn transition kinetics from parallel measurements in donor and acceptor channels. HMJPs generalize the HMM paradigm in two critical ways: (1) they deal with physical smFRET systems as they switch between conformational states in continuous time, and (2) they estimate transition rates between conformational states directly without having recourse to transition probabilities or assuming slow dynamics. Our continuous-time treatment learns the transition kinetics and photon emission rates for dynamic regimes that are inaccessible to HMMs, which treat system kinetics in discrete time. We validate our framework's robustness on simulated data and demonstrate its performance on experimental data from FRET-labeled Holliday junctions.

DeepSinse: deep learning based detection of single molecules

MOTIVATION

Imaging single molecules has emerged as a powerful characterization tool in the biological sciences. The detection of these under various noise conditions requires the use of algorithms that are dependent on the end-user inputting several parameters, the choice of which can be challenging and subjective.

RESULTS

In this work, we propose DeepSinse, an easily-trainable and useable deep neural network that can detect single molecules with little human input and across a wide range of signal-to-noise ratios. We validate the neural network on the detection of single bursts in simulated and experimental data and compare its performance with the best-in-class, domain-specific algorithms.

AVAILABILITY

Ground truth ROI simulating code, neural network training, validation code, classification code, ROI picker, GUI for simulating, training and validating DeepSinse as well as pre-trained networks are all released under the MIT License on www.github.com/jdanial/DeepSinse.The dSTORM dataset processing code is released under the MIT License on www.github.com/jdanial/StormProcessor.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

A brief history of the octopus imaging facility to celebrate its 10th anniversary

Octopus (Optics Clustered to OutPut Unique Solutions) celebrated in June 2020 its 10th birthday. Based at Harwell, near Oxford, Octopus is an open access, peer reviewed, national imaging facility that offers successful U.K. applicants supported access to single molecule imaging, confocal microscopy, several flavours of superresolution imaging, light sheet microscopy, optical trapping and cryoscanning electron microscopy. Managed by a multidisciplinary team, Octopus has so far assisted >100 groups of U.K. and international researchers. Cross-fertilisation across fields proved to be a strong propeller of success underpinned by combining access to top-end instrumentation with a strong programme of imaging hardware and software developments. How Octopus was born, and highlights of the multidisciplinary output produced during its 10-year journey are reviewed below, with the aim of celebrating a myriad of collaborations with the U.K. scientific community, and reflecting on their scientific and societal impact.

Switching behaviour of dSTORM dyes in glycerol-containing buffer

To suppress optical aberrations caused by refractive index mismatch, we employ glycerol-immersion objectives in conjunction with fused silica cover glasses and imaging buffers with a high glycerol content. Here we demonstrate that the addition of glycerol to the buffer does not degrade the switching behaviour of the dyes Alexa Fluor 647 and Alexa Fluor 568 in dSTORM measurements, which shows that this approach is suitable for dSTORM. Additionally, we report evidence that sealed sample geometries as used in our experiments reduce photobleaching due to the lower influx of oxygen into the imaging buffer.

Photon-counting 3D integral imaging with less than a single photon per pixel on average using a statistical model of the EM-CCD camera

We investigate photon-counting 3D integral imaging (PCII) with an electron multiplying charged-coupled device (EM-CCD) camera using dedicated statistical models. Using conventional integral imaging reconstruction methods with this camera in photon-counting conditions may result in degraded reconstructed image quality if multiple photons are detected simultaneously in a given pixel. We propose an estimation method derived from the photon detection statistical model of the EM-CCD to address the problems caused by multiple photons detected at the same pixel and provide improved 3D reconstructions. We also present a simplified version of this statistical method that can be used under the correct conditions. The imaging performance of these methods is evaluated on experimental data by the peak signal-to-noise ratio and the structural similarity index measure. The experiments demonstrate that 3D integral imaging substantially outperforms 2D imaging in degraded conditions. Furthermore, we achieve imaging in photon-counting conditions where, on average, less than a single photon per pixel is detected by the camera. To the best of our knowledge, this is the first report of PCII with the EM-CCD camera employing its statistical model in 3D reconstruction of PCII.

Identifying microbial species by single-molecule DNA optical mapping and resampling statistics

Single-molecule DNA mapping has the potential to serve as a powerful complement to high-throughput sequencing in metagenomic analysis. Offering longer read lengths and forgoing the need for complex library preparation and amplification, mapping stands to provide an unbiased view into the composition of complex viromes and/or microbiomes. To fully enable mapping-based metagenomics, sensitivity and specificity of DNA map analysis and identification need to be improved. Using detailed simulations and experimental data, we first demonstrate how fluorescence imaging of surface stretched, sequence specifically labeled DNA fragments can yield highly sensitive identification of targets. Second, a new analysis technique is introduced to increase specificity of the analysis, allowing even closely related species to be resolved. Third, we show how an increase in resolution improves sensitivity. Finally, we demonstrate that these methods are capable of identifying species with long genomes such as bacteria with high sensitivity.

Single-molecule turnover dynamics of actin and membrane coat proteins in clathrin-mediated endocytosis

Actin dynamics generate forces to deform the membrane and overcome the cell’s high turgor pressure during clathrin-mediated endocytosis (CME) in yeast, but precise molecular details are still unresolved. Our previous models predicted that actin filaments of the endocytic meshwork continually polymerize and disassemble, turning over multiple times during an endocytic event, similar to other actin systems. We applied single-molecule speckle tracking in live fission yeast to directly measure molecular turnover within CME sites for the first time. In contrast with the overall ~20 s lifetimes of actin and actin-associated proteins in endocytic patches, we detected single-molecule residence times around 1 to 2 s, and similarly high turnover rates of membrane-associated proteins in CME. Furthermore, we find heterogeneous behaviors in many proteins’ motions. These results indicate that endocytic proteins turn over up to five times during the formation of an endocytic vesicle, and suggest revising quantitative models of force production.

Single-molecule turnover dynamics of actin and membrane coat proteins in clathrin-mediated endocytosis

Actin dynamics generate forces to deform the membrane and overcome the cell's high turgor pressure during clathrin-mediated endocytosis (CME) in yeast, but precise molecular details are still unresolved. Our previous models predicted that actin filaments of the endocytic meshwork continually polymerize and disassemble, turning over multiple times during an endocytic event, similar to other actin systems. We applied single-molecule speckle tracking in live fission yeast to directly measure molecular turnover within CME sites for the first time. In contrast with the overall ~20 s lifetimes of actin and actin-associated proteins in endocytic patches, we detected single-molecule residence times around 1 to 2 s, and similarly high turnover rates of membrane-associated proteins in CME. Furthermore, we find heterogeneous behaviors in many proteins' motions. These results indicate that endocytic proteins turn over up to five times during the formation of an endocytic vesicle, and suggest revising quantitative models of force production.

A High-Throughput Image Correlation Method for Rapid Analysis of Fluorophore Photoblinking and Photobleaching Rates

Super-resolution fluorescence imaging based on localization microscopy requires tuning the photoblinking properties of fluorescent dyes employed. Missing is a rapid way to analyze the blinking rates of the fluorophore probes. Herein we present an ensemble autocorrelation technique for rapidly and simultaneously measuring photoblinking and bleaching rate constants from a microscopy image time series of fluorescent probes that is significantly faster than individual single-molecule trajectory analysis approaches. Our method is accurate for probe densities typically encountered in single-molecule studies as well as for higher density systems which cannot be analyzed by standard single-molecule techniques. We also show that we can resolve characteristic blinking times that are faster than camera detector exposure times, which cannot be accessed by threshold-based single-molecule approaches due to aliasing. We confirm this through computer simulation and single-molecule imaging data of DNA-Cy5 complexes. Finally, we demonstrate that with sufficient sampling our technique can accurately recover rates from stochastic optical reconstruction microscopy super-resolution data.

An extended quantitative model for super-resolution optical fluctuation imaging (SOFI)

Super-resolution optical fluctuation imaging (SOFI) provides super-resolution (SR) fluorescence imaging by analyzing fluctuations in the fluorophore emission. The technique has been used both to acquire quantitative SR images and to provide SR biosensing by monitoring changes in fluorophore blinking dynamics. Proper analysis of such data relies on a fully quantitative model of the imaging. However, previous SOFI imaging models made several assumptions that can not be realized in practice. In this work we address these limitations by developing and verifying a fully quantitative model that better approximates real-world imaging conditions. Our model shows that (i) SOFI images are free of bias, or can be made so, if the signal is stationary and fluorophores blink independently, (ii) allows a fully quantitative description of the link between SOFI imaging and probe dynamics, and (iii) paves the way for more advanced SOFI image reconstruction by offering a computationally fast way to calculate SOFI images for arbitrary probe, sample and instrumental properties.

Modeling Point Spread Function in Fluorescence Microscopy With a Sparse Gaussian Mixture: Tradeoff Between Accuracy and Efficiency

Deblurring is a fundamental inverse problem in bioimaging. It requires modeling the point spread function (PSF), which captures the optical distortions entailed by the image formation process. The PSF limits the spatial resolution attainable for a given microscope. However, recent applications require a higher resolution and have prompted the development of super-resolution techniques to achieve sub-pixel accuracy. This requirement restricts the class of suitable PSF models to analog ones. In addition, deblurring is computationally intensive, hence further requiring computationally efficient models. A custom candidate fitting both the requirements is the Gaussian model. However, this model cannot capture the rich tail structures found in both the theoretical and empirical PSFs. In this paper, we aim at improving the reconstruction accuracy beyond the Gaussian model, while preserving its computational efficiency. We introduce a new class of analog PSF models based on the Gaussian mixtures. The number of Gaussian kernels controls both the modeling accuracy and the computational efficiency of the model: the lower the number of kernels, the lower the accuracy and the higher the efficiency. To explore the accuracy-efficiency tradeoff, we propose a variational formulation of the PSF calibration problem, where a convex sparsity-inducing penalty on the number of Gaussian kernels allows trading accuracy for efficiency. We derive an efficient algorithm based on a fully split formulation of alternating split Bregman. We assess our framework on synthetic and real data, and demonstrate a better reconstruction accuracy in both geometry and photometry in point source localization-a fundamental inverse problem in fluorescence microscopy.

Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software

With the widespread uptake of 2D and 3D single molecule localization microscopy, a large set of different data analysis packages have been developed to generate super-resolution images. In a large community effort we designed a competition to extensively characterise and rank the performance of 2D and 3D single molecule localization microscopy software packages. We generated realistic simulated datasets for popular imaging modalities - 2D, astigmatic 3D, biplane 3D, and double helix 3D - and evaluated 36 participant packages against these data. This provides the first broad assessment of 3D single molecule localization microscopy software and provides a holistic view of how the latest 2D and 3D single molecule localization software perform in realistic conditions. This resource allows researchers to identify optimal analytical software for their experiments, allows 3D SMLM software developers to benchmark new software against current state of the art, and provides insight into the current limits of the field.

This study reports results from the second community-wide single molecule localization microscopy software challenge, which tested over thirty software packages on realistic simulated data for multiple popular 3D image acquisition modes as well as 2D localization microscopy.

Analyzing the Dynamics of Single TBP-DNA-NC2 Complexes Using Hidden Markov Models

Single-pair Förster resonance energy transfer (spFRET) has become an important tool for investigating conformational dynamics in biological systems. To extract dynamic information from the spFRET traces measured with total internal reflection fluorescence microscopy, we extended the hidden Markov model (HMM) approach. In our extended HMM analysis, we incorporated the photon-shot noise from camera-based systems into the HMM. Thus, the variance in Förster resonance energy transfer (FRET) efficiency of the various states, which is typically a fitted parameter, is explicitly included in the analysis estimated from the number of detected photons. It is also possible to include an additional broadening of the FRET state, which would then only reflect the inherent flexibility of the dynamic biological systems. This approach is useful when comparing the dynamics of individual molecules for which the total intensities vary significantly. We used spFRET with the extended HMM analysis to investigate the dynamics of TATA-box-binding protein (TBP) on promoter DNA in the presence of negative cofactor 2 (NC2). We compared the dynamics of two promoters as well as DNAs of different length and labeling location. For the adenovirus major late promoter, four FRET states were observed; three states correspond to different conformations of the DNA in the TBP-DNA-NC2 complex and a four-state model in which the complex has shifted along the DNA. The HMM analysis revealed that the states are connected via a linear, four-well model. For the H2B promoter, more complex dynamics were observed. By clustering the FRET states detected with the HMM analysis, we could compare the general dynamics observed for the two promoter sequences. We observed that the dynamics from a stretched DNA conformation to a bent conformation for the two promoters were similar, whereas the bent conformation of the TBP-DNA-NC2 complex for the H2B promoter is approximately three times more stable than for the adenovirus major late promoter.

A Bayesian framework for the analog reconstruction of kymographs from fluorescence microscopy data

Kymographs are widely used to represent and analyse spatio-temporal dynamics of fluorescence markers along curvilinear biological compartments. These objects have a singular geometry, thus kymograph reconstruction is inherently an analog image processing task. However, the existing approaches are essentially digital: the kymograph photometry is sampled directly from the time-lapse images. As a result, such kymographs rely on raw image data that suffer from the degradations entailed by the image formation process and the spatio-temporal resolution of the imaging setup. In this work, we address these limitations and introduce a well-grounded Bayesian framework for the analog reconstruction of kymographs. To handle the movement of the object, we introduce an intrinsic description of kymographs using differential geometry: a kymograph is a photometry defined on a parameter space that is embedded in physical space by a time-varying map that follows the object geometry. We model the kymograph photometry as a Lévy innovation process, a flexible class of non-parametric signal priors. We account for the image formation process using the virtual microscope framework. We formulate a computationally tractable representation of the associated maximum a posteriori problem and solve it using a class of efficient and modular algorithms based on the alternating split Bregman. We assess the performance of our Bayesian framework on synthetic data and apply it to reconstruct the fluorescence dynamics along microtubules in vivo in the budding yeast S. cerevisiae. We demonstrate that our framework allows revealing patterns from single time-lapse data that are invisible on standard digital kymographs.

Reliable State Identification and State Transition Detection in Fluorescence Intensity-Based Single-Molecule Förster Resonance Energy-Transfer Data

Single-molecule Förster resonance energy transfer (smFRET) is a powerful technique to probe biomolecular structure and dynamics. A popular implementation of smFRET consists of recording fluorescence intensity time traces of surface-immobilized, chromophore-tagged molecules. This approach generates large and complex data sets, the analysis of which is to date not standardized. Here, we address a key challenge in smFRET data analysis: the generation of thermodynamic and kinetic models that describe with statistical rigor the behavior of FRET trajectories recorded from surface-tethered biomolecules in terms of the number of FRET states, the corresponding mean FRET values, and the kinetic rates at which they interconvert. For this purpose, we first perform Monte Carlo simulations to generate smFRET trajectories, in which a relevant space of experimental parameters is explored. Then, we provide an account on current strategies to achieve such model selection, as well as a quantitative assessment of their performances. Specifically, we evaluate the performance of each algorithm (change-point analysis, STaSI, HaMMy, vbFRET, and ebFRET) with respect to accuracy, reproducibility, and computing time, which yields a range of algorithm-specific referential benchmarks for various data qualities. Data simulation and analysis were performed with our MATLAB-based multifunctional analysis software for handling smFRET data (MASH-FRET).

Simulations of camera-based single-molecule fluorescence experiments

Single-molecule microscopy has become a widely used technique in (bio)physics and (bio)chemistry. A popular implementation is single-molecule Förster Resonance Energy Transfer (smFRET), for which total internal reflection fluorescence microscopy is frequently combined with camera-based detection of surface-immobilized molecules. Camera-based smFRET experiments generate large and complex datasets and several methods for video processing and analysis have been reported. As these algorithms often address similar aspects in video analysis, there is a growing need for standardized comparison. Here, we present a Matlab-based software (MASH-FRET) that allows for the simulation of camera-based smFRET videos, yielding standardized data sets suitable for benchmarking video processing algorithms. The software permits to vary parameters that are relevant in cameras-based smFRET, such as video quality, and the properties of the system under study. Experimental noise is modeled taking into account photon statistics and camera noise. Finally, we survey how video test sets should be designed to evaluate currently available data analysis strategies in camera-based sm fluorescence experiments. We complement our study by pre-optimizing and evaluating spot detection algorithms using our simulated video test sets.

In silico optimization of radioluminescence microscopy

Radioluminescence microscopy (RLM) is a high-resolution method for imaging radionuclide uptake in live cells within a fluorescence microscopy environment. Although RLM currently provides sufficient spatial resolution and sensitivity for cell imaging, it has not been systematically optimized. This study seeks to optimize the parameters of the system by computational simulation using a combination of numerical models for the system's various components: Monte-Carlo simulation for radiation transport, 3D optical point-spread function for the microscope, and stochastic photosensor model for the electron multiplying charge coupled device (EMCCD) camera. The relationship between key parameters and performance metrics relevant to image quality is examined. Results show that Lu2 O3 :Eu yields the best performance among 5 different scintillator materials, and a thickness: 8 μm can best balance spatial resolution and sensitivity. For this configuration, a spatial resolution of ~20 μm and sensitivity of 40% can be achieved for all 3 magnifications investigated, provided that the user adjusts pixel binning and electron multiplying (EM) gain accordingly. Hence the primary consideration for selecting the magnification should be the desired field of view and magnification for concurrent optical microscopy studies. In conclusion, this study estimates the optimal imaging performance achievable with RLM and promotes further development for more robust imaging of cellular processes using radiotracers.

Parallel Low-Loss Measurement of Multiple Atomic Qubits

We demonstrate low-loss measurement of the hyperfine ground state of rubidium atoms by state dependent fluorescence detection in a dipole trap array of five sites. The presence of atoms and their internal states are minimally altered by utilizing circularly polarized probe light and a strictly controlled quantization axis. We achieve mean state detection fidelity of 97% without correcting for imperfect state preparation or background losses, and 98.7% when corrected. After state detection and correction for background losses, the probability of atom loss due to the state measurement is <2% and the initial hyperfine state is preserved with >98% probability.

Performance evaluation of 18 F radioluminescence microscopy using computational simulation

PURPOSE

Radioluminescence microscopy can visualize the distribution of beta-emitting radiotracers in live single cells with high resolution. Here, we perform a computational simulation of ¹⁸ F positron imaging using this modality to better understand how radioluminescence signals are formed and to assist in optimizing the experimental setup and image processing.

METHODS

First, the transport of charged particles through the cell and scintillator and the resulting scintillation is modeled using the GEANT4 Monte-Carlo simulation. Then, the propagation of the scintillation light through the microscope is modeled by a convolution with a depth-dependent point-spread function, which models the microscope response. Finally, the physical measurement of the scintillation light using an electron-multiplying charge-coupled device (EMCCD) camera is modeled using a stochastic numerical photosensor model, which accounts for various sources of noise. The simulated output of the EMCCD camera is further processed using our ORBIT image reconstruction methodology to evaluate the endpoint images.

RESULTS

The EMCCD camera model was validated against experimentally acquired images and the simulated noise, as measured by the standard deviation of a blank image, was found to be accurate within 2% of the actual detection. Furthermore, point source simulations found that a reconstructed spatial resolution of 18.5 μm can be achieved near the scintillator. As the source is moved away from the scintillator, spatial resolution degrades at a rate of 3.5 μm per μm distance. These results agree well with the experimentally measured spatial resolution of 30-40 μm (live cells). The simulation also shows that the system sensitivity is 26.5%, which is also consistent with our previous experiments. Finally, an image of a simulated sparse set of single cells is visually similar to the measured cell image.

CONCLUSIONS

Our simulation methodology agrees with experimental measurements taken with radioluminescence microscopy. This in silico approach can be used to guide further instrumentation developments and to provide a framework for improving image reconstruction.

Measurement-based estimation of global pupil functions in 3D localization microscopy

We report the use of a phase retrieval procedure based on maximum likelihood estimation (MLE) to produce an improved, experimentally calibrated model of a point spread function (PSF) for use in three-dimensional (3D) localization microscopy experiments. The method estimates a global pupil phase function (which includes both the PSF and system aberrations) over the full axial range from a simple calibration scan. The pupil function is used to refine the PSF model and hence enable superior localizations from experimental data. To demonstrate the utility of the procedure, we apply it to experimental data acquired with a microscope employing a tetrapod PSF with a 6 µm axial range. The phase-retrieved model demonstrates significant improvements in both accuracy and precision of 3D localizations relative to the model based on scalar diffraction theory. The localization precision of the phase-retrieved model is shown to be near the limits imposed by estimation theory, and the reproducibility of the procedure is characterized and discussed. Code which performs the phase retrieval algorithm is provided.

Cryogenic photoluminescence imaging system for nanoscale positioning of single quantum emitters

We report a photoluminescence imaging system for locating single quantum emitters with respect to alignment features. Samples are interrogated in a 4 K closed-cycle cryostat by a high numerical aperture (NA = 0.9, 100× magnification) objective that sits within the cryostat, enabling high efficiency collection of emitted photons without image distortions due to the cryostat windows. The locations of single InAs/GaAs quantum dots within a >50 μm × 50 μm field of view are determined with ≈4.5 nm uncertainty (one standard deviation) in a 1 s long acquisition. The uncertainty is determined through a combination of a maximum likelihood estimate for localizing the quantum dot emission, and a cross correlation method for determining the alignment mark center. This location technique can be an important step in the high-throughput creation of nanophotonic devices that rely upon the interaction of highly confined optical modes with single quantum emitters.

Important factors determining the nanoscale tracking precision of dynamic microtubule ends

Tracking dynamic microtubule ends in fluorescence microscopy movies provides insight into the statistical properties of microtubule dynamics and is vital for further analysis that requires knowledge of the trajectories of the microtubule ends. Here we analyse the performance of a previously developed automated microtubule end tracking routine; this has been optimized for comparatively low signal-to-noise image sequences that are characteristic of microscopy movies of dynamic microtubules growing in vitro. Sequences of simulated microtubule images were generated assuming a variety of different experimental conditions. The simulated movies were then tracked and the tracking errors were characterized. We found that the growth characteristics of the microtubules within realistic ranges had a negligible effect on the tracking precision. The fluorophore labelling density, the pixel size of the images, and the exposure times were found to be important parameters limiting the tracking precision which could be explained using concepts of single molecule localization microscopy. The signal-to-noise ratio was found to be a good single predictor of the tracking precision: typical experimental signal-to-noise ratios lead to tracking precisions in the range of tens of nanometres, making the tracking program described here a useful tool for dynamic microtubule end tracking with close to molecular precision.

Determining the rotational mobility of a single molecule from a single image: a numerical study

Measurements of the orientational freedom with which a single molecule may rotate or 'wobble' about a fixed axis have provided researchers invaluable clues about the underlying behavior of a variety of biological systems. In this paper, we propose a measurement and data analysis procedure based on a widefield fluorescence microscope image for quantitatively distinguishing individual molecules that exhibit varying degrees of rotational mobility. Our proposed technique is especially applicable to cases in which the molecule undergoes rotational motions on a timescale much faster than the framerate of the camera used to record fluorescence images. Unlike currently available methods, sophisticated hardware for modulating the polarization of light illuminating the sample is not required. Additional polarization optics may be inserted in the microscope's imaging pathway to achieve superior measurement precision, but are not essential. We present a theoretical analysis, and benchmark our technique with numerical simulations using typical experimental parameters for single-molecule imaging.

Single molecule microscopy in 3D cell cultures and tissues

From the onset of the first microscopic visualization of single fluorescent molecules in living cells at the beginning of this century, to the present, almost routine application of single molecule microscopy, the method has well-proven its ability to contribute unmatched detailed insight into the heterogeneous and dynamic molecular world life is composed of. Except for investigations on bacteria and yeast, almost the entire story of success is based on studies on adherent mammalian 2D cell cultures. However, despite this continuous progress, the technique was not able to keep pace with the move of the cell biology community to adapt 3D cell culture models for basic research, regenerative medicine, or drug development and screening. In this review, we will summarize the progress, which only recently allowed for the application of single molecule microscopy to 3D cell systems and give an overview of the technical advances that led to it. While initially posing a challenge, we finally conclude that relevant 3D cell models will become an integral part of the on-going success of single molecule microscopy.

A divide and conquer strategy for the maximum likelihood localization of low intensity objects

In cell biology and other fields the automatic accurate localization of sub-resolution objects in images is an important tool. The signal is often corrupted by multiple forms of noise, including excess noise resulting from the amplification by an electron multiplying charge-coupled device (EMCCD). Here we present our novel Nested Maximum Likelihood Algorithm (NMLA), which solves the problem of localizing multiple overlapping emitters in a setting affected by excess noise, by repeatedly solving the task of independent localization for single emitters in an excess noise-free system. NMLA dramatically improves scalability and robustness, when compared to a general purpose optimization technique. Our method was successfully applied for in vivo localization of fluorescent proteins.

Bayesian total internal reflection fluorescence correlation spectroscopy reveals hIAPP-induced plasma membrane domain organization in live cells

Amyloid fibril deposition of human islet amyloid polypeptide (hIAPP) in pancreatic islet cells is implicated in the pathogenesis of type II diabetes. A growing number of studies suggest that small peptide aggregates are cytotoxic via their interaction with the plasma membrane, which leads to membrane permeabilization or disruption. A recent study using imaging total internal reflection-fluorescence correlation spectroscopy (ITIR-FCS) showed that monomeric hIAPP induced the formation of cellular plasma membrane microdomains containing dense lipids, in addition to the modulation of membrane fluidity. However, the spatial organization of microdomains and their temporal evolution were only partially characterized due to limitations in the conventional analysis and interpretation of imaging FCS datasets. Here, we apply a previously developed Bayesian analysis procedure to ITIR-FCS data to resolve hIAPP-induced microdomain spatial organization and temporal dynamics. Our analysis enables the visualization of the temporal evolution of multiple diffusing species in the spatially heterogeneous cell membrane, lending support to the carpet model for the association mode of hIAPP aggregates with the plasma membrane. The presented Bayesian analysis procedure provides an automated and general approach to unbiased model-based interpretation of imaging FCS data, with broad applicability to resolving the heterogeneous spatial-temporal organization of biological membrane systems.