Bayesian estimation of regularization and point spread function parameters for Wiener-Hunt deconvolution

Surface-guided computing to analyze subcellular morphology and membrane-associated signals in 3D

Signal transduction and cell function are governed by the spatiotemporal organization of membrane-associated molecules. Despite significant advances in visualizing molecular distributions by 3D light microscopy, cell biologists still have limited quantitative understanding of the processes implicated in the regulation of molecular signals at the whole cell scale. In particular, complex and transient cell surface morphologies challenge the complete sampling of cell geometry, membrane-associated molecular concentration and activity and the computing of meaningful parameters such as the cofluctuation between morphology and signals. Here, we introduce u-Unwrap3D, a framework to remap arbitrarily complex 3D cell surfaces and membrane-associated signals into equivalent lower dimensional representations. The mappings are bidirectional, allowing the application of image processing operations in the data representation best suited for the task and to subsequently present the results in any of the other representations, including the original 3D cell surface. Leveraging this surface-guided computing paradigm, we track segmented surface motifs in 2D to quantify the recruitment of Septin polymers by blebbing events; we quantify actin enrichment in peripheral ruffles; and we measure the speed of ruffle movement along topographically complex cell surfaces. Thus, u-Unwrap3D provides access to spatiotemporal analyses of cell biological parameters on unconstrained 3D surface geometries and signals.

Using Computer Vision to Track Facial Color Changes and Predict Heart Rate

The current technological advances have pushed the quantification of exercise intensity to new era of physical exercise sciences. Monitoring physical exercise is essential in the process of planning, applying, and controlling loads for performance optimization and health. A lot of research studies applied various statistical approaches to estimate various physiological indices, to our knowledge, no studies found to investigate the relationship of facial color changes and increased exercise intensity. The aim of this study was to develop a non-contact method based on computer vision to determine the heart rate and, ultimately, the exercise intensity. The method was based on analyzing facial color changes during exercise by using RGB, HSV, YCbCr, Lab, and YUV color models. Nine university students participated in the study (mean age = 26.88 ± 6.01 years, mean weight = 72.56 ± 14.27 kg, mean height = 172.88 ± 12.04 cm, six males and three females, and all white Caucasian). The data analyses were carried out separately for each participant (personalized model) as well as all the participants at a time (universal model). The multiple auto regressions, and a multiple polynomial regression model were designed to predict maximum heart rate percentage (maxHR%) from each color models. The results were analyzed and evaluated using Root Mean Square Error (RMSE), F-values, and R-square. The multiple polynomial regression using all participants exhibits the best accuracy with RMSE of 6.75 (R-square = 0.78). Exercise prescription and monitoring can benefit from the use of these methods, for example, to optimize the process of online monitoring, without having the need to use any other instrumentation.

Deep camera obscura: an image restoration pipeline for pinhole photography

Modern machine learning has enhanced the image quality for consumer and mobile photography through low-light denoising, high dynamic range (HDR) imaging, and improved demosaicing among other applications. While most of these advances have been made for normal lens-based cameras, there has been an emerging body of research for improved photography for lensless cameras using thin optics such as amplitude or phase masks, diffraction gratings, or diffusion layers. These lensless cameras are suited for size and cost-constrained applications such as tiny robotics and microscopy that prohibit the use of a large lens. However, the earliest and simplest camera design, the camera obscura or pinhole camera, has been relatively overlooked for machine learning pipelines with minimal research on enhancing pinhole camera images for everyday photography applications. In this paper, we develop an image restoration pipeline of the pinhole system to enhance the pinhole image quality through joint denoising and deblurring. Our pipeline integrates optics-based filtering and reblur losses for reconstructing high resolution still images (2600 × 1952) as well as temporal consistency for video reconstruction to enable practical exposure times (30 FPS) for high resolution video (1920 × 1080). We demonstrate high 2D image quality on real pinhole images that is on-par or slightly improved compared to other lensless cameras. This work opens up the potential of pinhole cameras to be used for photography in size-limited devices such as smartphones in the future.

Design of an achromatic optical polarization-insensitive zoom metalens

A diffractive lens based on metasurfaces has many advantages such as flatness, small aberrations, and compactness. The focal length can be adjusted by changing the lateral displacement between a pair of conjugate metasurfaces while fixing their axial distance, thereby forming a very compact zoom lens. However, chromatic aberration of diffractive optical elements restricts this system to working at one wavelength. This Letter proposes a metalens design method based on novel three-layer polarization-insensitive nanoposts, which can improve transmission amplitude and satisfy the achromatic zoom function with wavelengths ranging from 1310 nm to 1550 nm. The focal length can be adjusted from 31.2 µm to 19.8 µm, corresponding to extremal wavelength-dependent focal length relative deviation of 8.98%. This achromatic zoom metalens design method could have applications in various fields including augmented reality and integrated optical systems.

At the Pulse of Time: Machine Vision in Retinal Videos

Spontaneous venous pulsations (SVP) are a common finding in healthy people. The absence of SVP is associated with rapid progression in glaucoma and increased intracranial pressure. Traditionally, SVP has been documented qualitatively by clinicians during biomicroscopy. Nowadays numerous imaging devices recording the fundus exist. Hence, video data for objectification of SVP is readily available. Still, these clinical datasets are afflicted with various quality issues and artifacts. In this machine vision based study, we explore methods to overcome challenges in identifying SVP in fundus videos of varying quality and provide a detailed protocol thereof. Hereby, we aim to lower the burden of access of implementing machine vision in clinical video datasets and quantification of SVP.

Self-Bayesian aberration removal via constraints for ultracold atom microscopy

High-resolution imaging of ultracold atoms typically requires custom high numerical aperture (NA) optics, as is the case for quantum gas microscopy. These high NA objectives involve many optical elements, each of which contributes to loss and light scattering, making them unsuitable for quantum backaction limited "weak" measurements. We employ a low-cost high NA aspheric lens as an objective for a practical and economical-although aberrated-high-resolution microscope to image ⁸⁷Rb Bose-Einstein condensates. Here, we present a methodology for digitally eliminating the resulting aberrations that is applicable to a wide range of imaging strategies and requires no additional hardware. We recover nearly the full NA of our objective, thereby demonstrating a simple and powerful digital aberration correction method for achieving optimal microscopy of quantum objects. This reconstruction relies on a high-quality measure of our imaging system's even-order aberrations from density-density correlations measured with differing degrees of defocus. We demonstrate our aberration compensation technique using phase-contrast imaging, a dispersive imaging technique directly applicable to quantum backaction limited measurements. Furthermore, we show that our digital correction technique reduces the contribution of photon shot noise to density-density correlation measurements which would otherwise contaminate the desired quantum projection noise signal in weak measurements.

Multiple-camera defocus imaging of ultracold atomic gases

In cold atom experiments, each image of light refracted and absorbed by an atomic ensemble carries a remarkable amount of information. Numerous imaging techniques including absorption, fluorescence, and phase-contrast are commonly used. Other techniques such as off-resonance defocused imaging (ORDI, [1-4]), where an in-focus image is deconvolved from a defocused image, have been demonstrated but find only niche applications. The ORDI inversion process introduces systematic artifacts because it relies on regularization to account for missing information at some spatial frequencies. In the present work, we extend ORDI to use multiple cameras simultaneously at degrees of defocus, eliminating the need for regularization and its attendant artifacts. We demonstrate this technique by imaging Bose-Einstein condensates, and show that the statistical uncertainties in the measured column density using the multiple-camera off-resonance defocused (McORD) imaging method are competitive with absorption imaging near resonance and phase contrast imaging far from resonance. Experimentally, the McORD method may be incorporated into existing set-ups with minimal additional equipment.

Ultra-Fast Electron Microscopic Imaging of Single Molecules With a Direct Electron Detection Camera and Noise Reduction

Time-resolved imaging of molecules and materials made of light elements is an emerging field of transmission electron microscopy (TEM), and the recent development of direct electron detection cameras, capable of taking as many as 1,600 fps, has potentially broadened the scope of the time-resolved TEM imaging in chemistry and nanotechnology. However, such a high frame rate reduces electron dose per frame, lowers the signal-to-noise ratio (SNR), and renders the molecular images practically invisible. Here, we examined image noise reduction to take the best advantage of fast cameras and concluded that the Chambolle total variation denoising algorithm is the method of choice, as illustrated for imaging of a molecule in the 1D hollow space of a carbon nanotube with ~1 ms time resolution. Through the systematic comparison of the performance of multiple denoising algorithms, we found that the Chambolle algorithm improves the SNR by more than an order of magnitude when applied to TEM images taken at a low electron dose as required for imaging at around 1,000 fps. Open-source code and a standalone application to apply Chambolle denoising to TEM images and video frames are available for download.

Measurement of the horizontal beam emittance of undulator radiation by tandem-double-slit optical system

A tandem-double-slit optical system was constructed to evaluate the practical beam emittance of undulator radiation. The optical system was a combination of an upstream slit (S₁) and downstream slit (S₂) aligned on the optical axis with an appropriate separation. The intensity distribution after the double slits, I(x₁, x₂), was measured by scanning S₁ and S₂ in the horizontal direction. Coordinates having 1/\sqrt e intensity were extracted from I(x₁, x₂), whose contour provided the standard deviation ellipse in the x₁-x₂ space. I(x₁, x₂) was converted to the corresponding distribution in the phase space, I(x₁, x₁'). The horizontal beam emittance was evaluated to be 3.1 nm rad, which was larger than the value of 2.4 nm rad estimated by using ray-tracing. It was found that the increase was mainly due to an increase in beam divergence rather than size.

Simultaneous recovery of both bright and dim structures from noisy fluorescence microscopy images using a modified TV constraint

The quality and information content of biological images can be significantly enhanced by postacquisition processing using deconvolution and denoising. However, when imaging complex biological samples, such as neurons, stained with fluorescence labels, the signal level of different structures can differ by several orders of magnitude. This poses a challenge as current image reconstruction algorithms are focused on recovering low signals and generally have sample-dependent performance, requiring tedious manual tuning. This is one of the main hurdles for their wide adoption by nonspecialists. In this work, we modify the general constrained reconstruction method (in our case utilizing a total variation constraint) so that both bright and dim structures can drive the deconvolution with equal force. In this way, we can simultaneously obtain high-quality reconstruction across a wide range of signals within a single image or image sequence. The algorithm is tested on both simulated and experimental data. When compared with current state-of-art algorithms, our algorithm outperforms others in terms of maintaining the resolution in the high-signal areas and reducing artefacts in the low-signal areas. The algorithm was also tested on image sequences where one set of parameters are used to reconstruct all images, with blind evaluation by a group of biologists demonstrating a marked preference for the images produced by our method. This means that our method is suitable for batch processing of image sequences obtained from either spatial or temporal scanning. LAY DESCRIPTION: Fluorescence microscopy images of complex biological samples contain a wide range of signal levels. This signal variation leads current reconstruction algorithms, which aim to enhance the quality of the raw images, to have sample-dependent performance. In this work, we design a new optimization that allows the reconstruction to "pay equal eqattention to" both bright and dim structures. In this way, we can simultaneously recover both bright and dim structures within a single image or image sequence, as validated when the algorithm was quantitatively tested on both simulated and experimental data. When our method was evaluated alongside current state of art algorithms by a group of biologists, our algorithm was considered qualitatively superior.

Spatial and spectral imaging of point-spread functions using a spatial light modulator

We develop a point-spread function (PSF) engineering approach to imaging the spatial and spectral information of molecular emissions using a spatial light modulator (SLM). We show that a dispersive grating pattern imposed upon the emission reveals spectral information. We also propose a deconvolution model that allows the decoupling of the spectral and 3D spatial information in engineered PSFs. The work is readily applicable to single-molecule measurements and fluorescent microscopy.

Super-resolution imaging in a multiple layer EPID

A new portal imager consisting of four vertically stacked conventional electronic portal imaging device (EPID) layers has been constructed in pursuit of improved detective quantum efficiency (DQE). We hypothesize that super-resolution (SR) imaging can also be achieved in such a system by shifting each layer laterally by half a pixel relative to the layer above. Super-resolution imaging will improve resolution and contrast-to-noise ratio (CNR) in megavoltage (MV) planar and cone beam computed tomography (MV-CBCT) applications. Simulations are carried out to test this hypothesis with digital phantoms. To assess planar resolution, 2 mm long iron rods with 0.3 × 0.3 mm² square cross-section are arranged in a grid pattern at the center of a 1 cm thick solid water. For measuring CNR in MV-CBCT, a 20 cm diameter digital phantom with 8 inserts of different electron densities is used. For measuring resolution in MV-CBCT, a digital phantom featuring a bar pattern similar to the Gammex™ phantom is used. A 6 MV beam is attenuated through each phantom and detected by each of the four detector layers. Fill factor of the detector is explicitly considered. Projections are blurred with an estimated point spread function (PSF) before super-resolution reconstruction. When projections from multiple shifted layers are used in SR reconstruction, even a simple shift-add fusion can significantly improve the resolution in reconstructed images. In the reconstructed planar image, the grid pattern becomes visually clearer. In MV-CBCT, combining projections from multiple layers results in increased CNR and resolution. The inclusion of two, three and four layers increases CNR by 40%, 70% and 99%, respectively. Shifting adjacent layers by half a pixel almost doubles resolution. In comparison, using four perfectly aligned layers does not improve resolution relative to a single layer.

Proton pinhole imaging on the National Ignition Facility

Pinhole imaging of large (mm scale) carbon-deuterium (CD) plasmas by proton self-emission has been used for the first time to study the microphysics of shock formation, which is of astrophysical relevance. The 3 MeV deuterium-deuterium (DD) fusion proton self-emission from these plasmas is imaged using a novel pinhole imaging system, with up to five different 1 mm diameter pinholes positioned 25 cm from target-chamber center. CR39 is used as the detector medium, positioned at 100 cm distance from the pinhole for a magnification of 4 ×. A Wiener deconvolution algorithm is numerically demonstrated and used to interpret the images. When the spatial morphology is known, this algorithm accurately reproduces the size of features larger than about half the pinhole diameter. For these astrophysical plasma experiments on the National Ignition Facility, this provides a strong constraint on simulation modeling of the experiment.

Bayesian-based deconvolution fluorescence microscopy using dynamically updated nonstationary expectation estimates

Fluorescence microscopy is widely used for the study of biological specimens. Deconvolution can significantly improve the resolution and contrast of images produced using fluorescence microscopy; in particular, Bayesian-based methods have become very popular in deconvolution fluorescence microscopy. An ongoing challenge with Bayesian-based methods is in dealing with the presence of noise in low SNR imaging conditions. In this study, we present a Bayesian-based method for performing deconvolution using dynamically updated nonstationary expectation estimates that can improve the fluorescence microscopy image quality in the presence of noise, without explicit use of spatial regularization.

Out-of-focus background subtraction for fast structured illumination super-resolution microscopy of optically thick samples

We propose a structured illumination microscopy method to combine super resolution and optical sectioning in three-dimensional (3D) samples that allows the use of two-dimensional (2D) data processing. Indeed, obtaining super-resolution images of thick samples is a difficult task if low spatial frequencies are present in the in-focus section of the sample, as these frequencies have to be distinguished from the out-of-focus background. A rigorous treatment would require a 3D reconstruction of the whole sample using a 3D point spread function and a 3D stack of structured illumination data. The number of raw images required, 15 per optical section in this case, limits the rate at which high-resolution images can be obtained. We show that by a succession of two different treatments of structured illumination data we can estimate the contrast of the illumination pattern and remove the out-of-focus content from the raw images. After this cleaning step, we can obtain super-resolution images of optical sections in thick samples using a two-beam harmonic illumination pattern and a limited number of raw images. This two-step processing makes it possible to obtain super resolved optical sections in thick samples as fast as if the sample was two-dimensional.

Theoretical performance model for single image depth from defocus

In this paper we present a performance model for depth estimation using single image depth from defocus (SIDFD). Our model is based on an original expression of the Cramér-Rao bound (CRB) in this context. We show that this model is consistent with the expected behavior of SIDFD. We then study the influence on the performance of the optical parameters of a conventional camera such as the focal length, the aperture, and the position of the in-focus plane (IFP). We derive an approximate analytical expression of the CRB away from the IFP, and we propose an interpretation of the SIDFD performance in this domain. Finally, we illustrate the predictive capacity of our performance model on experimental data comparing several settings of a consumer camera.

Semi-blind sparse image reconstruction with application to MRFM

We propose a solution to the image deconvolution problem where the convolution kernel or point spread function (PSF) is assumed to be only partially known. Small perturbations generated from the model are exploited to produce a few principal components explaining the PSF uncertainty in a high-dimensional space. Unlike recent developments on blind deconvolution of natural images, we assume the image is sparse in the pixel basis, a natural sparsity arising in magnetic resonance force microscopy (MRFM). Our approach adopts a Bayesian Metropolis-within-Gibbs sampling framework. The performance of our Bayesian semi-blind algorithm for sparse images is superior to previously proposed semi-blind algorithms such as the alternating minimization algorithm and blind algorithms developed for natural images. We illustrate our myopic algorithm on real MRFM tobacco virus data.

Bayesian estimation for optimized structured illumination microscopy

Structured illumination microscopy is a recent imaging technique that aims at going beyond the classical optical resolution by reconstructing high-resolution (HR) images from low-resolution (LR) images acquired through modulation of the transfer function of the microscope. The classical implementation has a number of drawbacks, such as requiring a large number of images to be acquired and parameters to be manually set in an ad-hoc manner that have, until now, hampered its wide dissemination. Here, we present a new framework based on a Bayesian inverse problem formulation approach that enables the computation of one HR image from a reduced number of LR images and has no specific constraints on the modulation. Moreover, it permits to automatically estimate the optimal reconstruction hyperparameters and to compute an uncertainty bound on the estimated values. We demonstrate through numerical evaluations on simulated data and examples on real microscopy data that our approach represents a decisive advance for a wider use of HR microscopy through structured illumination.