AIDA: an adaptive image deconvolution algorithm with application to multi-frame and three-dimensional data

Adaptive optics for optical microscopy [Invited]

Optical microscopy is widely used to visualize fine structures. When applied to bioimaging, its performance is often degraded by sample-induced aberrations. In recent years, adaptive optics (AO), originally developed to correct for atmosphere-associated aberrations, has been applied to a wide range of microscopy modalities, enabling high- or super-resolution imaging of biological structure and function in complex tissues. Here, we review classic and recently developed AO techniques and their applications in optical microscopy.

SPITFIR(e): a supermaneuverable algorithm for fast denoising and deconvolution of 3D fluorescence microscopy images and videos

Modern fluorescent microscopy imaging is still limited by the optical aberrations and the photon budget available in the specimen. A direct consequence is the necessity to develop flexible and "off-road" algorithms in order to recover structural details and improve spatial resolution, which is critical when restraining the illumination to low levels in order to limit photo-damages. Here, we report SPITFIR(e) a flexible method designed to accurately and quickly restore 2D-3D fluorescence microscopy images and videos (4D images). We designed a generic sparse-promoting regularizer to subtract undesirable out-of-focus background and we developed a primal-dual algorithm for fast optimization. SPITFIR(e) is a "swiss-knife" method for practitioners as it adapts to any microscopy techniques, to various sources of signal degradation (noise, blur), to variable image contents, as well as to low signal-to-noise ratios. Our method outperforms existing state-of-the-art algorithms, and is more flexible than supervised deep-learning methods requiring ground truth datasets. The performance, the flexibility, and the ability to push the spatiotemporal resolution limit of sub-diffracted fluorescence microscopy techniques are demonstrated on experimental datasets acquired with various microscopy techniques from 3D spinning-disk confocal up to lattice light sheet microscopy.

Live cell microscopy: From image to insight

Live-cell microscopy is a powerful tool that can reveal cellular behavior as well as the underlying molecular processes. A key advantage of microscopy is that by visualizing biological processes, it can provide direct insights. Nevertheless, live-cell imaging can be technically challenging and prone to artifacts. For a successful experiment, many careful decisions are required at all steps from hardware selection to downstream image analysis. Facing these questions can be particularly intimidating due to the requirement for expertise in multiple disciplines, ranging from optics, biophysics, and programming to cell biology. In this review, we aim to summarize the key points that need to be considered when setting up and analyzing a live-cell imaging experiment. While we put a particular focus on yeast, many of the concepts discussed are applicable also to other organisms. In addition, we discuss reporting and data sharing strategies that we think are critical to improve reproducibility in the field.

Using EQ·PET to reduce reconstruction-dependent variations in [18F]FDG-PET brain imaging

This study aims at assessing whether EANM harmonisation strategy combined with EQ·PET methodology could be successfully applied to harmonize brain 2-deoxy-2[¹⁸F]fluoro-D-glucose ([¹⁸F]FDG) positron emission tomography (PET) images. The NEMA NU 2 body phantom was prepared according to the EANM guidelines with an [¹⁸F]FDG solution. Raw PET phantom data were reconstructed with three different reconstruction protocols frequently used in clinical PET brain imaging: ([Formula: see text]) Ordered subset expectation maximization (OSEM) 3D with time of flight (TOF), 2 iterations and 21 subsets; ([Formula: see text]) OSEM 3D with TOF, 6 iterations and 21 subsets; and ([Formula: see text]) OSEM 3D with TOF, point spread function (PSF), and 8 iterations and 21 subsets. EQ·PET filters were computed as the Gaussian smoothing that best independently aligned the recovery coefficients (RCs) of reconstructions [Formula: see text] and [Formula: see text] with the RCs of the reference reconstruction, [Formula: see text]. The performance of the EQ·PET filter to reduce variations in quantification due to differences in reconstruction was investigated using clinical PET brain images of 35 early-onset Alzheimer's disease (EOAD) patients. Qualitative assessments and multiple quantitative metrics on the cortical surface at different scale levels with or without partial volume effect correction were evaluated on the [¹⁸F]FDG brain data before and after application of the EQ·PET filter. The EQ·PET methodology succeeded in finding the optimal smoothing that minimised root-mean-square error (RMSE) calculated using human brain [¹⁸F]FDG-PET datasets of EOAD patients, providing harmonized comparisons in the neurological context. Performance was superior for TOF than for TOF  +  PSF reconstructions. Results showed the capability of the EQ·PET methodology to minimize reconstruction-induced variabilities between brain [¹⁸F]FDG-PET images. However, moderate variabilities remained after harmonizing PSF reconstructions with standard non-PSF OSEM reconstructions, suggesting that precautions should be taken when using PSF modelling.

On-the-fly estimation of a microscopy point spread function

A proper estimation of realistic point-spread function (PSF) in optical microscopy can significantly improve the deconvolution performance and assist the microscope calibration process. In this work, by exemplifying 3D wide-field fluorescence microscopy, we propose an approach for estimating the spherically aberrated PSF of a microscope, directly from the observed samples. The PSF, expressed as a linear combination of 4 basis functions, is obtained directly from the acquired image by minimizing a novel criterion, which is derived from the noise statistics in the microscope. We demonstrate the effectiveness of the PSF approximation model and of our estimation method using both simulations and real experiments that were carried out on quantum dots. The principle of our PSF estimation approach is sufficiently flexible to be generalized non-spherical aberrations and other microscope modalities.

Resolution enhancement in laser scanning microscopy with deconvolution switching laser modes (D-SLAM)

Laser scanning microscopy is limited in lateral resolution by the diffraction of light. Superresolution methods have been developed since the 90s to overcome this limitation. However superresolution is generally achieved at the expense of a greater complexity (high power lasers, very long acquisition times, specific fluorophores) and limitations on the observable samples. In this paper we propose a method to improve the resolution of confocal microscopy by combining different laser modes and deconvolution. Two images of the same field are acquired with the confocal microscope using different laser modes and used as inputs to a deconvolution algorithm. The two laser modes have different Point Spread Functions and thus provide complementary information leading to an image with enhanced resolution compared to using a single confocal image as input to the same deconvolution algorithm. By changing the laser modes to Bessel-Gauss beams we were able to further improve the efficiency of the deconvolution algorithm and obtain images with a residual Point Spread Function having a width of 0.14 λ (72 nm at a wavelength of 532 nm). This method only requires a laser scanning microscope and is not dependent on certain specific properties of fluorescent proteins. The proposed method requires only a few add-ons to classical confocal or two-photon microscopes and can easily be retrofitted into an existing commercial laser scanning microscope.

Fast and accurate three-dimensional point spread function computation for fluorescence microscopy

The point spread function (PSF) plays a fundamental role in fluorescence microscopy. A realistic and accurately calculated PSF model can significantly improve the performance in 3D deconvolution microscopy and also the localization accuracy in single-molecule microscopy. In this work, we propose a fast and accurate approximation of the Gibson-Lanni model, which has been shown to represent the PSF suitably under a variety of imaging conditions. We express the Kirchhoff's integral in this model as a linear combination of rescaled Bessel functions, thus providing an integral-free way for the calculation. The explicit approximation error in terms of parameters is given numerically. Experiments demonstrate that the proposed approach results in a significantly smaller computational time compared with current state-of-the-art techniques to achieve the same accuracy. This approach can also be extended to other microscopy PSF models.

DeconvolutionLab2: An open-source software for deconvolution microscopy

Images in fluorescence microscopy are inherently blurred due to the limit of diffraction of light. The purpose of deconvolution microscopy is to compensate numerically for this degradation. Deconvolution is widely used to restore fine details of 3D biological samples. Unfortunately, dealing with deconvolution tools is not straightforward. Among others, end users have to select the appropriate algorithm, calibration and parametrization, while potentially facing demanding computational tasks. To make deconvolution more accessible, we have developed a practical platform for deconvolution microscopy called DeconvolutionLab. Freely distributed, DeconvolutionLab hosts standard algorithms for 3D microscopy deconvolution and drives them through a user-oriented interface. In this paper, we take advantage of the release of DeconvolutionLab2 to provide a complete description of the software package and its built-in deconvolution algorithms. We examine several standard algorithms used in deconvolution microscopy, notably: Regularized inverse filter, Tikhonov regularization, Landweber, Tikhonov-Miller, Richardson-Lucy, and fast iterative shrinkage-thresholding. We evaluate these methods over large 3D microscopy images using simulated datasets and real experimental images. We distinguish the algorithms in terms of image quality, performance, usability and computational requirements. Our presentation is completed with a discussion of recent trends in deconvolution, inspired by the results of the Grand Challenge on deconvolution microscopy that was recently organized.

Total variation regularized deconvolution for extended depth of field microscopy

The depth of field of an optical system can be extended through a combination of point spread function (PSF) engineering and image processing. A phase mask inserted in the back aperture of the system creates a PSF that is focus-invariant over an extended depth. A digital deconvolution is then used to restore transverse resolution. The application and analysis of this technique to fluorescence microscopy is limited in the literature. In this paper we formalize a microscopy specific imaging model, and experimentally demonstrate a total variation regularized deconvolution approach. Results are compared to the Wiener filter.

High-resolution restoration of 3D structures from widefield images with extreme low signal-to-noise-ratio

Four-dimensional fluorescence microscopy--which records 3D image information as a function of time--provides an unbiased way of tracking dynamic behavior of subcellular components in living samples and capturing key events in complex macromolecular processes. Unfortunately, the combination of phototoxicity and photobleaching can severely limit the density or duration of sampling, thereby limiting the biological information that can be obtained. Although widefield microscopy provides a very light-efficient way of imaging, obtaining high-quality reconstructions requires deconvolution to remove optical aberrations. Unfortunately, most deconvolution methods perform very poorly at low signal-to-noise ratios, thereby requiring moderate photon doses to obtain acceptable resolution. We present a unique deconvolution method that combines an entropy-based regularization function with kernels that can exploit general spatial characteristics of the fluorescence image to push the required dose to extreme low levels, resulting in an enabling technology for high-resolution in vivo biological imaging.

Phase diversity for three-dimensional imaging

Phase diversity (PD) is a powerful technique for estimating wavefront aberrations from two-dimensional images of extended scenes. PD can work with extended incoherent images and, in an adaptive optics system, does not need extra hardware in addition to the deformable mirror. For these reasons, PD should be well suited to aberration measurement in microscopy applications. But, in biological widefield microscopy, the objects being imaged are frequently three-dimensional, and the images contain out-of-focus light. In this paper, we introduce multiplane PD and show that PD can be extended to widefield imaging of three-dimensional objects. This should be particularly useful in the field of biological fluorescence microscopy where the objects are very light sensitive and the aberrations cannot easily be determined.

Kinetic analysis of pre-ribosome structure in vivo

Pre-ribosomal particles undergo numerous structural changes during maturation, but their high complexity and short lifetimes make these changes very difficult to follow in vivo. In consequence, pre-ribosome structure and composition have largely been inferred from purified particles and analyzed in vitro. Here we describe techniques for kinetic analyses of the changes in pre-ribosome structure in living cells of Saccharomyces cerevisiae. To allow this, in vivo structure probing by DMS modification was combined with affinity purification of newly synthesized 20S pre-rRNA over a time course of metabolic labeling with 4-thiouracil. To demonstrate that this approach is generally applicable, we initially analyzed the accessibility of the region surrounding cleavage site D site at the 3' end of the mature 18S rRNA region of the pre-rRNA. This revealed a remarkably flexible structure throughout 40S subunit biogenesis, with little stable RNA-protein interaction apparent. Analysis of folding in the region of the 18S central pseudoknot was consistent with previous data showing U3 snoRNA-18S rRNA interactions. Dynamic changes in the structure of the hinge between helix 28 (H28) and H44 of pre-18S rRNA were consistent with recently reported interactions with the 3' guide region of U3 snoRNA. Finally, analysis of the H18 region indicates that the RNA structure matures early, but additional protection appears subsequently, presumably reflecting protein binding. The structural analyses described here were performed on total, affinity-purified, newly synthesized RNA, so many classes of RNA and RNA-protein complex are potentially amenable to this approach.

Light sheet microscopy in cell biology

Light sheet-based fluorescence microscopy (LSFM) is emerging as a powerful imaging technique for the life sciences. LSFM provides an exceptionally high imaging speed, high signal-to-noise ratio, low level of photo-bleaching, and good optical penetration depth. This unique combination of capabilities makes light sheet-based microscopes highly suitable for live imaging applications. Here, we provide an overview of light sheet-based microscopy assays for in vitro and in vivo imaging of biological samples, including cell extracts, soft gels, and large multicellular organisms. We furthermore describe computational tools for basic image processing and data inspection.

Increasing axial resolution of 3D data sets using deconvolution algorithms

Deconvolution algorithms are tools for the restoration of data degraded by blur and noise. An incorporation of regularization functions into the iterative form of reconstruction algorithms can improve the restoration performance and characteristics (e.g. noise and artefact handling). In this study, algorithms based on Richardson-Lucy deconvolution algorithm are tested. The ability of these algorithms to improve axial resolution of three-dimensional data sets is evaluated on model synthetic data. Finally, unregularized Richardson-Lucy algorithm is selected for the evaluation and reconstruction of three-dimensional chromosomal data sets of Drosophila melanogaster. Problems concerning the reconstruction process are discussed and further improvements are proposed.

Blind image deconvolution using machine learning for three-dimensional microscopy

In this work, we propose a novel method for the regularization of blind deconvolution algorithms. The proposed method employs example-based machine learning techniques for modeling the space of point spread functions. During an iterative blind deconvolution process, a prior term attracts the point spread function estimates to the learned point spread function space. We demonstrate the usage of this regularizer within a Bayesian blind deconvolution framework and also integrate into the latter a method for noise reduction, thus creating a complete blind deconvolution method. The application of the proposed algorithm is demonstrated on synthetic and real-world three-dimensional images acquired by a wide-field fluorescence microscope, where the need for blind deconvolution algorithms is indispensable, yielding excellent results.

Tensor factorization for model-free space-variant blind deconvolution of the single- and multi-frame multi-spectral image

The higher order orthogonal iteration (HOOI) is used for a single-frame and multi-frame space-variant blind deconvolution (BD) performed by factorization of the tensor of blurred multi-spectral image (MSI). This is achieved by conversion of BD into blind source separation (BSS), whereupon sources represent the original image and its spatial derivatives. The HOOI-based factorization enables an essentially unique solution of the related BSS problem with orthogonality constraints imposed on factors and the core tensor of the Tucker3 model of the image tensor. In contrast, the matrix factorization-based unique solution of the same BSS problem demands sources to be statistically independent or sparse which is not true. The consequence of such an approach to BD is that it virtually does not require a priori information about the possibly space-variant point spread function (PSF): neither its model nor size of its support. For the space-variant BD problem, MSI is divided into blocks whereupon the PSF is assumed to be a space-invariant within the blocks. The success of proposed concept is demonstrated in experimentally degraded images: defocused single-frame gray scale and red-green-blue (RGB) images, single-frame gray scale and RGB images blurred by atmospheric turbulence, and a single-frame RGB image blurred by a grating (photon sieve). A comparable or better performance is demonstrated in relation to the blind Richardson-Lucy algorithm which, however, requires a priori information about parametric model of the blur.

A Parallel Product-Convolution approach for representing the depth varying Point Spread Functions in 3D widefield microscopy based on principal component analysis

We address the problem of computational representation of image formation in 3D widefield fluorescence microscopy with depth varying spherical aberrations. We first represent 3D depth-dependent point spread functions (PSFs) as a weighted sum of basis functions that are obtained by principal component analysis (PCA) of experimental data. This representation is then used to derive an approximating structure that compactly expresses the depth variant response as a sum of few depth invariant convolutions pre-multiplied by a set of 1D depth functions, where the convolving functions are the PCA-derived basis functions. The model offers an efficient and convenient trade-off between complexity and accuracy. For a given number of approximating PSFs, the proposed method results in a much better accuracy than the strata based approximation scheme that is currently used in the literature. In addition to yielding better accuracy, the proposed methods automatically eliminate the noise in the measured PSFs.

High-resolution wide-field microscopy with adaptive optics for spherical aberration correction and motionless focusing

Live imaging in cell biology requires three-dimensional data acquisition with the best resolution and signal-to-noise ratio possible. Depth aberrations are a major source of image degradation in three-dimensional microscopy, causing a significant loss of resolution and intensity deep into the sample. These aberrations occur because of the mismatch between the sample refractive index and the immersion medium index. We have built a wide-field fluorescence microscope that incorporates a large-throw deformable mirror to simultaneously focus and correct for depth aberration in three-dimensional imaging. Imaging fluorescent beads in water and glycerol with an oil immersion lens we demonstrate a corrected point spread function and a 2-fold improvement in signal intensity. We apply this new microscope to imaging biological samples, and show sharper images and improved deconvolution.

Visualizing yeast chromosomes and nuclear architecture

We describe here optimized protocols for tagging genomic DNA sequences with bacterial operator sites to enable visualization of specific loci in living budding yeast cells. Quantitative methods for the analysis of locus position relative to the nuclear center or nuclear pores, the analysis of chromatin dynamics and the relative position of tagged loci to other nuclear landmarks are described. Methods for accurate immunolocalization of nuclear proteins without loss of three-dimensional structure, in combination with fluorescence in situ hybridization, are also presented. These methods allow a robust analysis of subnuclear organization of both proteins and DNA in intact yeast cells.

Super-resolution laser scanning microscopy through spatiotemporal modulation

Super-resolution optical microscopy has attracted great interest among researchers in many fields, especially in biology where the scale of physical structures and molecular processes fall below the diffraction limit of resolution for light. As one of the emerging techniques, structured illumination microscopy can double the resolution by shifting unresolvable spatial frequencies into the pass-band of the microscope through spatial frequency mixing with a wide-field structured illumination pattern. However, such a wide-field scheme typically can only image optically thin samples and is incompatible with multiphoton processes such as two-photon fluorescence, which require point scanning with a focused laser beam. Here, we propose two new super-resolution schemes for laser scanning microscopy by generalizing the concept of a spatially nonuniform imaging system. One scheme, scanning patterned illumination (SPIN) microscopy, employs modulation of the excitation combined with temporally cumulative imaging by a nondescanned array detector. The other scheme, scanning patterned detection (SPADE) microscopy, utilizes detection modulation together with spatially cumulative imaging, in this case by a nondescanned single-element detector. When combined with multiphoton excitation, both schemes can image thick samples with three-dimensional optical sectioning and much improved resolution.

Blind deconvolution for thin-layered confocal imaging

We propose an alternate minimization algorithm for estimating the point-spread function (PSF) of a confocal laser scanning microscope and the specimen fluorescence distribution. A three-dimensional separable Gaussian model is used to restrict the PSF solution space and a constraint on the specimen is used so as to favor the stabilization and convergence of the algorithm. The results obtained from the simulation show that the PSF can be estimated to a high degree of accuracy, and those on real data show better deconvolution as compared to a full theoretical PSF model.

Reducing between scanner differences in multi-center PET studies

This work is part of the multi-center Alzheimer's Disease Neuroimaging Initiative (ADNI), a large multi-site study of dementia, including patients having mild cognitive impairment (MCI), probable Alzheimer's disease (AD), as well as healthy elderly controls. A major portion of ADNI involves the use of [(18)F]-fluorodeoxyglucose (FDG) with positron emission tomography (PET). The objective of this paper is the reduction of inter-scanner differences in the FDG-PET scans obtained from the 50 participating PET centers having fifteen different scanner models. In spite of a standardized imaging protocol, systematic inter-scanner variability in PET images from various sites is observed primarily due to differences in scanner resolution, reconstruction techniques, and different implementations of scatter and attenuation corrections. Two correction steps were developed by comparison of 3-D Hoffman brain phantom scans with the 'gold standard' digital 3-D Hoffman brain phantom: i) high frequency correction; where a smoothing kernel for each scanner model was estimated to smooth all images to a common resolution and ii) low frequency correction; where smooth affine correction factors were obtained to reduce the attenuation and scatter correction errors. For the phantom data, the high frequency correction reduced the variability by 20%-50% and the low frequency correction further reduced the differences by another 20%-25%. Correction factors obtained from phantom studies were applied to 95 scans from normal control subjects obtained from the participating sites. The high frequency correction reduced differences similar to the phantom studies. However, the low frequency correction did not further reduce differences; hence further refinement of the procedure is necessary.

Multi-excitation Raman spectroscopy technique for fluorescence rejection

Multi-excitation Raman spectroscopy filters out Raman signals from a fluorescent background by sequentially using multiple excitation frequencies. The filtering method exploits the shift of the Raman spectra with excitation frequency and the static response of the fluorescent background. This technique builds upon previous work which used two slightly shifted excitations, Shifted Excitation Raman Difference Spectroscopy (SERDS), in order to filter the Raman signal. An Expectation-Maximization algorithm is used to estimate the Raman and fluorescence signals from multiple spectra acquired with slightly shifted excitation frequencies. In both simulation and experiment, the efficacy of the algorithm increases with the number of excitation frequencies even when holding the total excitation energy constant, such that the signal to noise ratio is inversely proportional to the number of excitation frequencies. In situations where the intense fluorescence causes significant shot noise compared to the weak Raman signals, the multi-excitation approach is more effective than non-iterative techniques such as polynomial background subtraction.

Shape, size and multiplicity of main-belt asteroids I. Keck Adaptive Optics survey

This paper presents results from a high spatial resolution survey of 33 main-belt asteroids with diameters >40 km using the Keck II Adaptive Optics (AO) facility. Five of these (45 Eugenia, 87 Sylvia, 107 Camilla, 121 Hermione, 130 Elektra) were confirmed to have satellite. Assuming the same albedo as the primary, these moonlets are relatively small (∼5% of the primary size) suggesting that they are fragments captured after a disruptive collision of a parent body or captured ejecta due to an impact. For each asteroid, we have estimated the minimum size of a moonlet that can positively detected within the Hill sphere of the system by estimating and modeling a 2-σ detection profile: in average on the data set, a moonlet located at 2/100 × R(Hill) (1/4 × R(Hill)) with a diameter larger than 6 km (4 km) would have been unambiguously seen. The apparent size and shape of each asteroid was estimated after deconvolution using a new algorithm called AIDA. The mean diameter for the majority of asteroids is in good agreement with IRAS radiometric measurements, though for asteroids with a D < 200 km, it is underestimated on average by 6-8%. Most asteroids had a size ratio that was very close to those determined by lightcurve measurements. One observation of 104 Klymene suggests it has a bifurcated shape. The bi-lobed shape of 121 Hermione described in Marchis et al. [Marchis, F., Hestroffer, D., Descamps, P., Berthier, J., Laver, C., de Pater, I., 2005c. Icarus 178, 450-464] was confirmed after deconvolution. The ratio of contact binaries in our survey, which is limited to asteroids larger than 40 km, is surprisingly high (∼6%), suggesting that a non-single configuration is common in the main-belt. Several asteroids have been analyzed with lightcurve inversions. We compared lightcurve inversion models for plane-of-sky predictions with the observed images (9 Metis, 52 Europa, 87 Sylvia, 130 Elektra, 192 Nausikaa, and 423 Diotima, 511 Davida). The AO images allowed us to determine a unique photometric mirror pole solution, which is normally ambiguous for asteroids moving close to the plane of the ecliptic (e.g., 192 Nausikaa and 52 Europa). The photometric inversion models agree well with the AO images, thus confirming the validity of both the lightcurve inversion method and the AO image reduction technique.