Automatic phase aberration compensation for digital holographic microscopy based on deep learning background detection

Quantitative phase imaging based on holography: trends and new perspectives

In 1948, Dennis Gabor proposed the concept of holography, providing a pioneering solution to a quantitative description of the optical wavefront. After 75 years of development, holographic imaging has become a powerful tool for optical wavefront measurement and quantitative phase imaging. The emergence of this technology has given fresh energy to physics, biology, and materials science. Digital holography (DH) possesses the quantitative advantages of wide-field, non-contact, precise, and dynamic measurement capability for complex-waves. DH has unique capabilities for the propagation of optical fields by measuring light scattering with phase information. It offers quantitative visualization of the refractive index and thickness distribution of weak absorption samples, which plays a vital role in the pathophysiology of various diseases and the characterization of various materials. It provides a possibility to bridge the gap between the imaging and scattering disciplines. The propagation of wavefront is described by the complex amplitude. The complex-value in the complex-domain is reconstructed from the intensity-value measurement by camera in the real-domain. Here, we regard the process of holographic recording and reconstruction as a transformation between complex-domain and real-domain, and discuss the mathematics and physical principles of reconstruction. We review the DH in underlying principles, technical approaches, and the breadth of applications. We conclude with emerging challenges and opportunities based on combining holographic imaging with other methodologies that expand the scope and utility of holographic imaging even further. The multidisciplinary nature brings technology and application experts together in label-free cell biology, analytical chemistry, clinical sciences, wavefront sensing, and semiconductor production.

Deep learning based coherence holography reconstruction of 3D objects

We propose a reconstruction method for coherence holography using deep neural networks. cGAN and U-NET models were developed to reconstruct 3D complex objects from recorded interferograms. Our proposed methods, dubbed deep coherence holography (DCH), predict the non-diffracted fields or the sub-objects included in the 3D object from the captured interferograms, yielding better reconstructed objects than the traditional analytical imaging methods in terms of accuracy, resolution, and time. The DCH needs one image per sub-object as opposed to N images for the traditional sin-fit algorithm, and hence the total reconstruction time is reduced by N×. Furthermore, with noisy interferograms the DCH amplitude mean square reconstruction error (MSE) is 5×104× and 104× and phase MSE is 102× and 3×103× better than Fourier fringe and sin-fit algorithms, respectively. The amplitude peak signal to noise ratio (PSNR) is 3× and 2× and phase PSNR is 5× and 3× better than Fourier fringe and sin-fit algorithms, respectively. The reconstruction resolution is the same as sin-fit but 2× better than the Fourier fringe analysis technique.

Spherical wave illumination scanning digital holographic profilometry

In this work, we proposed what we believe to be a novel scanning solution for the assessment of high-NA samples, referred to as spherical-wave illumination scanning digital holographic profilometry (SWS-DHP). This approach introduces a 2F optimization methodology, based on the measurement of the focal length of the object to determine the spherical component of the scanning. Furthermore, re-optimization of 2F, whether it needs to be operated depends on the measured object's NA to inspect more information. Meanwhile, utilizing phase space analysis shows SWS superiority in information transfer for high-NA samples compared to plane-wave illumination scanning. In addition, this method introduces a shape reconstruction algorithm with volumetric aberration compensation based on the propagation of the aberrated object and illumination waves to obtain high-quality measurements. Finally, the imaging merits of SWS-DHP were proved through simulations and were experimentally verified for the object of NA up to 0.87.

On the use of deep learning for phase recovery

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.

Automatic aberration compensation for digital holographic microscopy based on bicubic downsampling and improved minimization of global phase gradients

In digital holographic microscopy, aberrations caused by imperfect optical system settings can greatly affect the quantitative measurement of the target phase, so the compensation of aberrations in the distorted phase has become a key point of research in digital holographic microscopy. Here, we propose a fully automatic numerical phase aberration compensation method with fast computational speed and high robustness. The method uses bicubic downsampling to smooth the sample phase for reducing its disturbance to the background aberration fit, while reducing the computational effort of aberration compensation. Polynomial coefficients of the aberration fitting are iteratively optimized in the process of minimizing the global phase gradient by improving the phase gradient operator and constructing the loss function to achieve accurate fitting of the phase aberration. Simulation and experimental results show that the proposed method can achieve high aberration compensation accuracy without prior knowledge of the hologram recording settings or manual selection of the background area free of samples, and it is suitable for samples with moderate and relatively flat background area, which can be widely used in the quantitative analysis of biological tissues and micro and nano structures.

Quantitative phase imaging of living red blood cells combining digital holographic microscopy and deep learning

Digital holographic microscopy as a non-contacting, non-invasive, and highly accurate measurement technology, is becoming a valuable method for quantitatively investigating cells and tissues. Reconstruction of phases from a digital hologram is a key step in quantitative phase imaging for biological and biomedical research. This study proposes a two-stage deep convolutional neural network named VY-Net, to realize the effective and robust phase reconstruction of living red blood cells. The VY-Net can obtain the phase information of an object directly from a single-shot off-axis digital hologram. We also propose two new indices to evaluate the reconstructed phases. In experiments, the mean of the structural similarity index of reconstructed phases can reach 0.9309, and the mean of the accuracy of reconstructions of reconstructed phases is as high as 91.54%. An unseen phase map of a living human white blood cell is successfully reconstructed by the trained VY-Net, demonstrating its strong generality.

Deep Learning Approach for the Localization and Analysis of Surface Plasmon Scattering

Surface plasmon resonance microscopy (SPRM) combines the principles of traditional microscopy with the versatility of surface plasmons to develop label-free imaging methods. This paper describes a proof-of-principles approach based on deep learning that utilized the Y-Net convolutional neural network model to improve the detection and analysis methodology of SPRM. A machine-learning based image analysis technique was used to provide a method for the one-shot analysis of SPRM images to estimate scattering parameters such as the scatterer location. The method was assessed by applying the approach to SPRM images and reconstructing an image from the network output for comparison with the original image. The results showed that deep learning can localize scatterers and predict other variables of scattering objects with high accuracy in a noisy environment. The results also confirmed that with a larger field of view, deep learning can be used to improve traditional SPRM such that it localizes and produces scatterer characteristics in one shot, considerably increasing the detection capabilities of SPRM.

Deep sound-field denoiser: optically-measured sound-field denoising using deep neural network

This paper proposes a deep sound-field denoiser, a deep neural network (DNN) based denoising of optically measured sound-field images. Sound-field imaging using optical methods has gained considerable attention due to its ability to achieve high-spatial-resolution imaging of acoustic phenomena that conventional acoustic sensors cannot accomplish. However, the optically measured sound-field images are often heavily contaminated by noise because of the low sensitivity of optical interferometric measurements to airborne sound. Here, we propose a DNN-based sound-field denoising method. Time-varying sound-field image sequences are decomposed into harmonic complex-amplitude images by using a time-directional Fourier transform. The complex images are converted into two-channel images consisting of real and imaginary parts and denoised by a nonlinear-activation-free network. The network is trained on a sound-field dataset obtained from numerical acoustic simulations with randomized parameters. We compared the method with conventional ones, such as image filters, a spatiotemporal filter, and other DNN architectures, on numerical and experimental data. The experimental data were measured by parallel phase-shifting interferometry and holographic speckle interferometry. The proposed deep sound-field denoiser significantly outperformed the conventional methods on both the numerical and experimental data. Code is available on GitHub (https://github.com/nttcslab/deep-sound-field-denoiser).

Comprehensive tool for a phase compensation reconstruction method in digital holographic microscopy operating in non-telecentric regime

Quantitative phase imaging (QPI) via Digital Holographic microscopy (DHM) has been widely applied in material and biological applications. The performance of DHM technologies relies heavily on computational reconstruction methods to provide accurate phase measurements. Among the optical configuration of the imaging system in DHM, imaging systems operating in a non-telecentric regime are the most common ones. Nonetheless, the spherical wavefront introduced by the non-telecentric DHM system must be compensated to provide undistorted phase measurements. The proposed reconstruction approach is based on previous work from Kemper's group. Here, we have reformulated the problem, reducing the number of required parameters needed for reconstructing phase images to the sensor pixel size and source wavelength. The developed computational algorithm can be divided into six main steps. In the first step, the selection of the +1-diffraction order in the hologram spectrum. The interference angle is obtained from the selected +1 order. Secondly, the curvature of the spherical wavefront distorting the sample's phase map is estimated by analyzing the size of the selected +1 order in the hologram's spectrum. The third and fourth steps are the spatial filtering of the +1 order and the compensation of the interference angle. The next step involves the estimation of the center of the spherical wavefront. An optional final optimization step has been included to fine-tune the estimated parameters and provide fully compensated phase images. Because the proper implementation of a framework is critical to achieve successful results, we have explicitly described the steps, including functions and toolboxes, required for reconstructing phase images without distortions. As a result, we have provided open-access codes and a user interface tool with minimum user input to reconstruct holograms recorded in a non-telecentric DHM system.

Phase aberration adaptive compensation in digital holography based on phase imitation and metric optimization

We proposed a numerical and accurate quadratic phase aberration compensation method in digital holography. A phase imitation method based on Gaussian 1σ-criterion is used to obtain the morphological features of the object phase using partial differential, filtering and integration successively. We also propose an adaptive compensation method based on a maximum-minimum-average- α-standard deviation (MMAαSD) evaluation metric to obtain optimal compensated coefficients by minimizing the above metric of the compensation function. The effectiveness and robustness of our method are demonstrated by simulation and experiments.

Automatic elimination of phase aberrations in digital holography based on Gaussian 1σ- criterion and histogram segmentation

We propose a numerical and automatic quadratic phase aberration elimination method in digital holography for phase-contrast imaging. A histogram segmentation method based on Gaussian 1σ-criterion is used to obtain the accurate coefficients of quadratic aberrations using the weighted least-squares algorithm. This method needs no manual intervention for specimen-free zone or prior parameters of optical components. We also propose a maximum-minimum-average-standard deviation (MMASD) metric to quantitatively evaluate the effectiveness of quadratic aberration elimination. Simulation and experimental results are demonstrated to verify the efficacy of our proposed method over the traditional least-squares algorithm.

Phase aberration separation for holographic microscopy by alternating direction sparse optimization

The morphology and dynamics of label-free tissues can be exploited by sample-induced changes in the optical field from quantitative phase imaging. Its sensitivity to subtle changes in the optical field makes the reconstructed phase susceptible to phase aberrations. We import variable sparse splitting framework on quantitative phase aberration extraction based on alternating direction aberration free method. The optimization and regularization in the reconstructed phase are decomposed into object terms and aberration terms. By formulating the aberration extraction as a convex quadratic problem, the background phase aberration can be fast and directly decomposed with the specific complete basis functions such as Zernike or standard polynomials. Faithful phase reconstruction can be obtained by eliminating global background phase aberration. The aberration-free two-dimensional and three-dimensional imaging experiments are demonstrated, showing the relaxation of the strict alignment requirements for the holographic microscopes.

Complex wave and phase retrieval from a single off-axis interferogram

Single-frame off-axis holographic reconstruction is promising for quantitative phase imaging. However, reconstruction accuracy and contrast are degraded by noise, frequency spectrum overlap of the interferogram, severe phase distortion, etc. In this work, we propose an iterative single-frame complex wave retrieval based on an explicit model of object and reference waves. We also develop a phase restoration algorithm that does not resort to phase unwrapping. Both simulation and real experiments demonstrate higher accuracy and robustness compared to state-of-the-art methods, for both complex wave estimation and phase reconstruction. Importantly, the allowed bandwidth for the object wave is significantly improved in realistic experimental conditions (similar amplitudes for object and reference waves), which makes it attractive for large field-of-view and high-resolution imaging applications.

Digital Holographic Microscopy to Assess Cell Behavior

Digital holographic microscopy is an imaging technique particularly well suited to the study of living cells in culture, as no labeling is required and computed phase maps produce high contrast, quantitative pixel information. A full experiment involves instrument calibration, cell culture quality checks, selection and setup of imaging chambers, a sampling plan, image acquisition, phase and amplitude map reconstruction, and parameter map post-processing to extract information about cell morphology and/or motility. Each step is described below, focusing on results from imaging four human cell lines. Several post-processing approaches are detailed, with an aim of tracking individual cells and dynamics of cell populations.

pyDHM: A Python library for applications in digital holographic microscopy

pyDHM is an open-source Python library aimed at Digital Holographic Microscopy (DHM) applications. The pyDHM is a user-friendly library written in the robust programming language of Python that provides a set of numerical processing algorithms for reconstructing amplitude and phase images for a broad range of optical DHM configurations. The pyDHM implements phase-shifting approaches for in-line and slightly off-axis systems and enables phase compensation for telecentric and non-telecentric systems. In addition, pyDHM includes three propagation algorithms for numerical focusing complex amplitude distributions in DHM and digital holography (DH) setups. We have validated the library using numerical and experimental holograms.

Accurate extraction of the +1 term spectrum with spurious spectrum elimination in off-axis digital holography

In off-axis digital holography, spatial filtering is a key problem limiting the quality of reconstructed image, especially in the case of spurious spectrum generated by coherent noise in the hologram spectrum. In this paper, a new spatial filtering method with spurious spectrum elimination is proposed. Side band centering judgment is firstly implemented to locate the center point of the +1 term in the hologram spectrum. Then by roughly recognizing the region of +1 term spectrum, most of the -1 term, 0 term and the spurious spectral components are eliminated. Finally, Butterworth filtering is performed to extract the +1 term spectrum as enough as possible without introducing the spurious spectrum. Simulated hologram of E-shaped specimen with the spurious spectrum is generated to evaluate the performance of the proposed method. Experimental data of USAF 1951 resolution target, ovarian slice and microlens array are adopted to verify the effectiveness of the proposed method. Simulation and experimental results demonstrated that the proposed method is able to accurately extract the +1 term spectrum with spurious spectrum elimination and achieve a relatively good balance between the structural detail characterization and noise suppression.

Deep learning in optical metrology: a review

With the advances in scientific foundations and technological implementations, optical metrology has become versatile problem-solving backbones in manufacturing, fundamental research, and engineering applications, such as quality control, nondestructive testing, experimental mechanics, and biomedicine. In recent years, deep learning, a subfield of machine learning, is emerging as a powerful tool to address problems by learning from data, largely driven by the availability of massive datasets, enhanced computational power, fast data storage, and novel training algorithms for the deep neural network. It is currently promoting increased interests and gaining extensive attention for its utilization in the field of optical metrology. Unlike the traditional "physics-based" approach, deep-learning-enabled optical metrology is a kind of "data-driven" approach, which has already provided numerous alternative solutions to many challenging problems in this field with better performances. In this review, we present an overview of the current status and the latest progress of deep-learning technologies in the field of optical metrology. We first briefly introduce both traditional image-processing algorithms in optical metrology and the basic concepts of deep learning, followed by a comprehensive review of its applications in various optical metrology tasks, such as fringe denoising, phase retrieval, phase unwrapping, subset correlation, and error compensation. The open challenges faced by the current deep-learning approach in optical metrology are then discussed. Finally, the directions for future research are outlined.

Speeding up reconstruction of 3D tomograms in holographic flow cytometry via deep learning

Tomographic flow cytometry by digital holography is an emerging imaging modality capable of collecting multiple views of moving and rotating cells with the aim of recovering their refractive index distribution in 3D. Although this modality allows us to access high-resolution imaging with high-throughput, the huge amount of time-lapse holographic images to be processed (hundreds of digital holograms per cell) constitutes the actual bottleneck. This prevents the system from being suitable for lab-on-a-chip platforms in real-world applications, where fast analysis of measured data is mandatory. Here we demonstrate a significant speeding-up reconstruction of phase-contrast tomograms by introducing in the processing pipeline a multi-scale fully-convolutional context aggregation network. Although it was originally developed in the context of semantic image analysis, we demonstrate for the first time that it can be successfully adapted to a holographic lab-on-chip platform for achieving 3D tomograms through a faster computational process. We trained the network with input-output image pairs to reproduce the end-to-end holographic reconstruction process, i.e. recovering quantitative phase maps (QPMs) of single cells from their digital holograms. Then, the sequence of QPMs of the same rotating cell is used to perform the tomographic reconstruction. The proposed approach significantly reduces the computational time for retrieving tomograms, thus making them available in a few seconds instead of tens of minutes, while essentially preserving the high-content information of tomographic data. Moreover, we have accomplished a compact deep convolutional neural network parameterization that can fit into on-chip SRAM and a small memory footprint, thus demonstrating its possible exploitation to provide onboard computations for lab-on-chip devices with low processing hardware resources.

Video-Rate Quantitative Phase Imaging Using a Digital Holographic Microscope and a Generative Adversarial Network

The conventional reconstruction method of off-axis digital holographic microscopy (DHM) relies on computational processing that involves spatial filtering of the sample spectrum and tilt compensation between the interfering waves to accurately reconstruct the phase of a biological sample. Additional computational procedures such as numerical focusing may be needed to reconstruct free-of-distortion quantitative phase images based on the optical configuration of the DHM system. Regardless of the implementation, any DHM computational processing leads to long processing times, hampering the use of DHM for video-rate renderings of dynamic biological processes. In this study, we report on a conditional generative adversarial network (cGAN) for robust and fast quantitative phase imaging in DHM. The reconstructed phase images provided by the GAN model present stable background levels, enhancing the visualization of the specimens for different experimental conditions in which the conventional approach often fails. The proposed learning-based method was trained and validated using human red blood cells recorded on an off-axis Mach–Zehnder DHM system. After proper training, the proposed GAN yields a computationally efficient method, reconstructing DHM images seven times faster than conventional computational approaches.

Deep learning for digital holography: a review

Recent years have witnessed the unprecedented progress of deep learning applications in digital holography (DH). Nevertheless, there remain huge potentials in how deep learning can further improve performance and enable new functionalities for DH. Here, we survey recent developments in various DH applications powered by deep learning algorithms. This article starts with a brief introduction to digital holographic imaging, then summarizes the most relevant deep learning techniques for DH, with discussions on their benefits and challenges. We then present case studies covering a wide range of problems and applications in order to highlight research achievements to date. We provide an outlook of several promising directions to widen the use of deep learning in various DH applications.

Fast-iterative automatic reconstruction method for quantitative phase image with reduced phase perturbations in off-axis digital holographic microscopy

This works presents a reconstruction algorithm to recover the complex object information for an off-axis digital holographic microscope (DHM) operating in the telecentric regimen. We introduce an automatic and fast method to minimize a cost function that finds the best numerical conjugated reference beam to compensate the filtered object information, eliminating any undesired phase perturbation due to the tilt between the reference and object waves. The novelties of the proposed approach, to the best of our knowledge, are a precise estimation of the interference angle between the object and reference waves, reconstructed phase images without phase perturbations, and reduced processing time. The method has been validated using a manufactured phase target and biological samples.

DBDNet for denoising in ESPI wrapped phase patterns with high density and high speckle noise

In this paper, we propose a dilated-blocks-based deep convolution neural network, named DBDNet, for denoising in electronic speckle pattern interferometry (ESPI) wrapped phase patterns with high density and high speckle noise. In our method, the proposed dilated blocks have a specific sequence of dilation rate and a multilayer cascading fusion structure, which can better improve the effect of speckle noise reduction, especially for phase patterns with high noise and high density. Furthermore, we have built an abundant training dataset with varieties of densities and noise levels to train our network; thus, the trained model has a good generalization and can denoise ESPI wrapped phase in various circumstances. The network can get denoised results directly and does not need any pre-process or post-process. We test our method on one group of computer-simulated ESPI phase patterns and one group of experimentally obtained ESPI phase patterns. The test images have a high degree of speckle noise and different densities. We compare our method with two representative methods in the spatial domain and frequency domain, named oriented-couple partial differential equation and windowed Fourier low pass filter (LPF), and a method based on deep learning, named fast and flexible denoising convolutional neural network (FFDNet). The denoising performance is evaluated quantitatively and qualitatively. The results demonstrate that our method can reduce high speckle noise and restore the dense areas of ESPI phase patterns, and get better results than the compared methods. We also apply our method to a series of phase patterns from a dynamic measurement and get successful results.

Enhancement of image sharpness and height measurement using a low-speckle light source based on a patterned quantum dot film in dual-wavelength digital holography

A dual-wavelength single light source based on a patterned quantum dot (QD) film was developed with a 405nm LED and bandpass filters to increase color conversion efficiency as well as to decouple the two peaks of dual-wavelength emitted from the QD film. A QD film was patterned laterally with two different sizes of QDs and was combined with bandpass filters to produce a high efficiency and low-speckle dual-wavelength light source. The experimental results showed that the developed dual-wavelength light source can decrease speckle noise to improve the reconstructed image sharpness and the accuracy on height measurement in dual-wavelength digital holography.

Machine Learning Assisted Classification of Cell Lines and Cell States on Quantitative Phase Images

In this report, we present implementation and validation of machine-learning classifiers for distinguishing between cell types (HeLa, A549, 3T3 cell lines) and states (live, necrosis, apoptosis) based on the analysis of optical parameters derived from cell phase images. Validation of the developed classifier shows the accuracy for distinguishing between the three cell types of about 93% and between different cell states of the same cell line of about 89%. In the field test of the developed algorithm, we demonstrate successful evaluation of the temporal dynamics of relative amounts of live, apoptotic and necrotic cells after photodynamic treatment at different doses.

Volumetric monitoring of airborne particulate matter concentration using smartphone-based digital holographic microscopy and deep learning

Airborne particulate matter (PM) has become a global environmental issue. This PM has harmful effects on public health and precision industries. Conventional air-quality monitoring methods usually utilize expensive equipment, and they are cumbersome to handle for accurate and high throughput measurements. In addition, commercial particle counters have technical limitations in high-concentration measurement, and data fluctuations are induced during air sampling. In this study, a novel smartphone-based technique for monitoring airborne PM concentrations was developed using smartphone-based digital holographic microscopy (S-DHM) and deep learning network called Holo-SpeckleNet. Holographic speckle images of various PM concentrations were recorded by the S-DHM system. The recorded speckle images and the corresponding ground truth PM concentrations were used to train deep learning algorithms consisting of a deep autoencoder and regression layers. The performance of the proposed smartphone-based PM monitoring technique was validated through hyperparameter optimization. The developed S-DHM integrated with Holo-SpeckleNet can be smartly and effectively utilized for portable PM monitoring and safety alarm provision under perilous environmental conditions.

Quantitative phase imaging in digital holographic microscopy based on image inpainting using a two-stage generative adversarial network

Based on the hologram inpainting via a two-stage Generative Adversarial Network (GAN), we present a precise phase aberration compensation method in digital holographic microscopy (DHM). In the proposed methodology, the interference fringes of the sample area in the hologram are firstly removed by the background segmentation via edge detection and morphological image processing. The vacancy area is then inpainted with the fringes generated by a deep learning algorithm. The image inpainting finally results in a sample-free reference hologram containing the total aberration of the system. The phase aberrations could be deleted by subtracting the unwrapped phase of the sample-free hologram from our inpainting network results, in no need of any complex spectrum centering procedure, prior knowledge of the system, or manual intervention. With a full and proper training of the two-stage GAN, our approach can robustly realize a distinct phase mapping, which overcomes the drawbacks of multiple iterations, noise interference or limited field of view in the recent methods using self-extension, Zernike polynomials fitting (ZPF) or geometrical transformations. The validity of the proposed procedure is confirmed by measuring the surface of preprocessed silicon wafer with a Michelson interferometer digital holographic inspection platform. The results of our experiment indicate the viability and accuracy of the presented method. Additionally, this work can pave the way for the evaluation of new applications of GAN in DHM.

Neural network-based image reconstruction in swept-source optical coherence tomography using undersampled spectral data

Optical coherence tomography (OCT) is a widely used non-invasive biomedical imaging modality that can rapidly provide volumetric images of samples. Here, we present a deep learning-based image reconstruction framework that can generate swept-source OCT (SS-OCT) images using undersampled spectral data, without any spatial aliasing artifacts. This neural network-based image reconstruction does not require any hardware changes to the optical setup and can be easily integrated with existing swept-source or spectral-domain OCT systems to reduce the amount of raw spectral data to be acquired. To show the efficacy of this framework, we trained and blindly tested a deep neural network using mouse embryo samples imaged by an SS-OCT system. Using 2-fold undersampled spectral data (i.e., 640 spectral points per A-line), the trained neural network can blindly reconstruct 512 A-lines in 0.59 ms using multiple graphics-processing units (GPUs), removing spatial aliasing artifacts due to spectral undersampling, also presenting a very good match to the images of the same samples, reconstructed using the full spectral OCT data (i.e., 1280 spectral points per A-line). We also successfully demonstrate that this framework can be further extended to process 3× undersampled spectral data per A-line, with some performance degradation in the reconstructed image quality compared to 2× spectral undersampling. Furthermore, an A-line-optimized undersampling method is presented by jointly optimizing the spectral sampling locations and the corresponding image reconstruction network, which improved the overall imaging performance using less spectral data points per A-line compared to 2× or 3× spectral undersampling results. This deep learning-enabled image reconstruction approach can be broadly used in various forms of spectral-domain OCT systems, helping to increase their imaging speed without sacrificing image resolution and signal-to-noise ratio.

Pupil-aberration calibration with controlled illumination for quantitative phase imaging

Quantitative phase imaging (QPI) exploits sample-induced changes in the optical field to analyze biological specimens in a label-free manner. However, the quantitative nature of QPI makes it susceptible to optical aberrations. We propose a method for calibrating pupil aberrations by imaging a sample of interest. The proposed method recovers pupil information by utilizing the cross-spectral density between optical fields at different incident angles and allows both thin and weakly scattering three-dimensional samples for calibration. We experimentally validate the proposed method by imaging various samples, including a resolution target, breast tissue, and a polystyrene bead, and demonstrate aberration-free two- and three-dimensional QPI.

Improving holographic particle characterization by modeling spherical aberration

Holographic microscopy combined with forward modeling and inference allows colloidal particles to be characterized and tracked in three dimensions with high precision. However, current models ignore the effects of optical aberrations on hologram formation. We investigate the effects of spherical aberration on the structure of single-particle holograms and on the accuracy of particle characterization. We find that in a typical experimental setup, spherical aberration can result in systematic shifts of about 2% in the inferred refractive index and radius. We show that fitting with a model that accounts for spherical aberration decreases this aberration-dependent error by a factor of two or more, even when the level of spherical aberration in the optical train is unknown. With the new generative model, the inferred parameters are consistent across different levels of aberration, making particle characterization more robust.

Sensing morphogenesis of bone cells under microfluidic shear stress by holographic microscopy and automatic aberration compensation with deep learning

We present sensing time-lapse morphogenesis of living bone cells under micro-fluidic shear stress (FSS) by digital holographic (DH) microscopy. To remove the effect of aberrations on quantitative measurements, we propose a numerical and automatic method to compensate for aberrations based on a convolutional neural network (CNN). For the first time, the aberration compensation issue is considered as a regression task where optimal coefficients for constructing the phase aberration map act as responses corresponding to the input aberrated phase image. We adopted tens of thousands of living cells' phase images reconstructed from digital holograms for training the CNN. The experiments demonstrate that, based on the trained network, phase aberrations can be totally removed in real-time without any hypothesis of object and aberration phase, knowledge of the setup's physical parameters, and the operation of selecting background regions; hence, the morphogenesis of the bone cells under FSS is accurately detected and quantitatively analyzed. The results show that the proposed method could provide a highly efficient and versatile way to investigate the effects of micro-FSS on living biological cells in microfluidic lab-on-chip platforms thanks to the combination of phase-contrast label-free microcopy with artificial intelligence.

Aberration-free digital holographic phase imaging using the derivative-based principal component analysis

SIGNIFICANCE

Digital holographic microscopy is widely used to get the quantitative phase information of transparent cells.

AIM

However, the sample phase is superimposed with aberrations. To quantify the phase information, aberrations need to be fully compensated.

APPROACH

We propose a technique to obtain aberration-free phase imaging, using the derivative-based principal component analysis (dPCA).

RESULTS

With dPCA, almost all aberrations can be extracted and compensated without requirements on background segmentation, making it efficient and convenient.

CONCLUSIONS

It solves the problem that the conventional principal component analysis (PCA) algorithm cannot compensate the common but intricate higher order cross-term aberrations, such as astigmatism and coma. Moreover, the dPCA strategy proposed here is not only suitable for aberration compensation but also applicable for other cases where there exist cross-terms that cannot be analyzed with the PCA algorithm.

Deep learning in electron microscopy

Deep learning is transforming most areas of science and technology, including electron microscopy. This review paper offers a practical perspective aimed at developers with limited familiarity. For context, we review popular applications of deep learning in electron microscopy. Following, we discuss hardware and software needed to get started with deep learning and interface with electron microscopes. We then review neural network components, popular architectures, and their optimization. Finally, we discuss future directions of deep learning in electron microscopy.

High-throughput label-free cell detection and counting from diffraction patterns with deep fully convolutional neural networks

SIGNIFICANCE

Digital holographic microscopy (DHM) is a promising technique for the study of semitransparent biological specimen such as red blood cells (RBCs). It is important and meaningful to detect and count biological cells at the single cell level in biomedical images for biomarker discovery and disease diagnostics. However, the biological cell analysis based on phase information of images is inefficient due to the complexity of numerical phase reconstruction algorithm applied to raw hologram images. New cell study methods based on diffraction pattern directly are desirable.

AIM

Deep fully convolutional networks (FCNs) were developed on raw hologram images directly for high-throughput label-free cell detection and counting to assist the biological cell analysis in the future.

APPROACH

The raw diffraction patterns of RBCs were recorded by use of DHM. Ground-truth mask images were labeled based on phase images reconstructed from RBC holograms using numerical reconstruction algorithm. A deep FCN, which is UNet, was trained on the diffraction pattern images to achieve the label-free cell detection and counting.

RESULTS

The implemented deep FCNs provide a promising way to high-throughput and label-free counting of RBCs with a counting accuracy of 99% at a throughput rate of greater than 288 cells per second and 200  μm  ×  200  μm field of view at the single cell level. Compared to convolutional neural networks, the FCNs can get much better results in terms of accuracy and throughput rate.

CONCLUSIONS

High-throughput label-free cell detection and counting were successfully achieved from diffraction patterns with deep FCNs. It is a promising approach for biological specimen analysis based on raw hologram directly.

Dual-wavelength interferogram decoupling method for three-frame generalized dual-wavelength phase-shifting interferometry based on deep learning

In dual-wavelength interferometry, the key issue is how to efficiently retrieve the phases at each wavelength using the minimum number of wavelength-multiplexed interferograms. To address this problem, a new dual-wavelength interferogram decoupling method with the help of deep learning is proposed in this study. This method requires only three randomly phase-shifted dual-wavelength interferograms. With a well-trained deep neural network, one can obtain three interferograms with arbitrary phase shifts at each wavelength. Using these interferograms, the wrapped phases of a single wavelength can be extracted, respectively, via an iterative phase retrieval algorithm, and then the phases at different synthetic beat wavelengths can be calculated. The feasibility and applicability of the proposed method are demonstrated by simulation experiments of the spherical cap and red blood cell, respectively. This method will provide a solution for the problem of phase retrieval in multiwavelength interferometry.

Calibration-free quantitative phase imaging using data-driven aberration modeling

We present a data-driven approach to compensate for optical aberrations in calibration-free quantitative phase imaging (QPI). Unlike existing methods that require additional measurements or a background region to correct aberrations, we exploit deep learning techniques to model the physics of aberration in an imaging system. We demonstrate the generation of a single-shot aberration-corrected field image by using a U-net-based deep neural network that learns a translation between an optical field with aberrations and an aberration-corrected field. The high fidelity and stability of our method is demonstrated on 2D and 3D QPI measurements of various confluent eukaryotic cells and microbeads, benchmarking against the conventional method using background subtractions.

The use of deep learning algorithm and digital media art in all-media intelligent electronic music system

In the development of digital media art, to explore the preliminary application of deep learning method in intelligent electronic music system, and promote the integration of deep learning method and digital media technology, thus providing a direction for the development of all media intelligent system, based on deep deterministic policy gradient (DDPG), to solve the multi-task problem in intelligent system, a multi-task learning-based DDPG algorithm (M-DDPG) is proposed. Furthermore, a DDPG algorithm based on hierarchical learning (H-DDPG) is proposed for the hierarchical analysis of images in intelligent system. Aiming at the problem of image classification in intelligent system, through the setting of simulation environment, the application effect of several algorithms in intelligent electronic music system is evaluated. The results show that: M-DDPG algorithm can more accurately complete the operation of related tasks, the reward received by the intelligent system is more than 0.35, and the test results based on eight tasks are more accurate and effective. Even in the case of task error, the algorithm still shows good training results. H-DDPG algorithm has good effect for complex task processing. The accuracy rate of task test corresponding to intelligent system in different scenarios is above 95%, which is better than other conventional algorithms in task test; the self-reinforcement network algorithm can promote the improvement of image classification effect. Several algorithms proposed show excellent performance in image processing of intelligent system, and have great application potential.

Deep neural network for multi-depth hologram generation and its training strategy

We present a deep neural network for generating a multi-depth hologram and its training strategy. The proposed network takes multiple images of different depths as inputs and calculates the complex hologram as an output, which reconstructs each input image at the corresponding depth. We design a structure of the proposed network and develop the dataset compositing method to train the network effectively. The dataset consists of multiple input intensity profiles and their propagated holograms. Rather than simply training random speckle images and their propagated holograms, we generate the training dataset by adjusting the density of the random dots or combining basic shapes to the dataset such as a circle. The proposed dataset composition method improves the quality of reconstructed images by the holograms generated by the network, called deep learning holograms (DLHs). To verify the proposed method, we numerically and optically reconstruct the DLHs. The results confirmed that the DLHs can reconstruct clear images at multiple depths similar to conventional multi-depth computer-generated holograms. To evaluate the performance of the DLH quantitatively, we compute the peak signal-to-noise ratio of the reconstructed images and analyze the reconstructed intensity patterns with various methods.

Deep learning-based hologram generation using a white light source

Digital holographic microscopy enables the recording of sample holograms which contain 3D volumetric information. However, additional optical elements, such as partially or fully coherent light source and a pinhole, are required to induce diffraction and interference. Here, we present a deep neural network based on generative adversarial network (GAN) to perform image transformation from a defocused bright-field (BF) image acquired from a general white light source to a holographic image. Training image pairs of 11,050 for image conversion were gathered by using a hybrid BF and hologram imaging technique. The performance of the trained network was evaluated by comparing generated and ground truth holograms of microspheres and erythrocytes distributed in 3D. Holograms generated from BF images through the trained GAN showed enhanced image contrast with 3-5 times increased signal-to-noise ratio compared to ground truth holograms and provided 3D positional information and light scattering patterns of the samples. The developed GAN-based method is a promising mean for dynamic analysis of microscale objects with providing detailed 3D positional information and monitoring biological samples precisely even though conventional BF microscopic setting is utilized.

Sampling moiré method: a tool for sensing quadratic phase distortion and its correction for accurate quantitative phase microscopy

The advantages of quantitative phase microscopy (QPM) such as label-free imaging with high spatial sensitivity, live cell compatibility and high-speed imaging makes it viable for various biological applications. The measurement accuracy of QPM strongly relies on the shape of the recorded interferograms, whether straight or curved fringes are recorded during the data acquisition. Moreover, for a single shot phase recovery high fringe density is required. The wavefront curvature for the high-density fringes over the entire field of view is difficult to be discerned with the naked eye. As a consequence, there is a quadratic phase aberration in the recovered phase images due to curvature mismatch. In the present work, we have implemented sampling moiré method for real-time sensing of the wavefront curvature mismatch between the object and the reference wavefronts and further for its correction. By zooming out the interferogram, moiré fringes are generated which helps to easily identify the curvature of the fringes. The wavefront curvature mismatch correction accuracy of the method is tested with the help of low temporal coherent light source such as a white light (temporal coherence ∼ 1.6 µm). The proposed scheme is successfully demonstrated to remove the quadratic phase aberration caused due to wavefront mismatch from an USAF resolution target and the biological tissue samples. The phase recovery accuracy of the current scheme is further compared with and found to better than the standard method called principle component analysis. The proposed method enables recording of the corrected wavefront interferogram without needing any additional optical components or modification and also does not need any post-processing correction algorithms. The proposed method of curvature compensation paves the path for a high-throughput and accurate quantitative phase imaging.

Enhanced calibration for freeform surface misalignments in non-null interferometers by convolutional neural network

Most tested surface calibration methods in interferometers, such as the direct coefficients removing method, the sensitive matrix (SM) method, and deep neural network (DNN) calibration method, rely on Zernike coefficients. However, due to the inherent rotationally non-symmetric aberrations in a non-null freeform surface interferometer, the interferograms are usually non-circular even if the surface apertures are circular. The Zernike coefficients based methods are inaccurate due to the non-orthogonality of Zernike polynomials in the non-circular area. A convolutional neural network (CNN)-based misalignment calibration method is proposed. Instead of Zernike coefficients, the well-trained CNN treats the interferogram directly to estimate the specific misalignments. Simulations and experiments are carried out to validate the high accuracy.

RedCap: residual encoder-decoder capsule network for holographic image reconstruction

A capsule network, as an advanced technique in deep learning, is designed to overcome information loss in the pooling operation and internal data representation of a convolutional neural network (CNN). It has shown promising results in several applications, such as digit recognition and image segmentation. In this work, we investigate for the first time the use of capsule network in digital holographic reconstruction. The proposed residual encoder-decoder capsule network, which we call RedCap, uses a novel windowed spatial dynamic routing algorithm and residual capsule block, which extends the idea of a residual block. Compared with the CNN-based neural network, RedCap exhibits much better experimental results in digital holographic reconstruction, while having a dramatic 75% reduction in the number of parameters. It indicates that RedCap is more efficient in the way it processes data and requires a much less memory storage for the learned model, which therefore makes it possible to be applied to some challenging situations with limited computational resources, such as portable devices.

In vitro monitoring of photoinduced necrosis in HeLa cells using digital holographic microscopy and machine learning

Digital holographic microscopy supplemented with the developed cell segmentation and machine learning and classification algorithms is implemented for quantitative description of the dynamics of cellular necrosis induced by photodynamic treatment in vitro. It is demonstrated that the developed algorithms operating with a set of optical, morphological, and physiological parameters of cells, obtained from their phase images, can be used for automatic distinction between live and necrotic cells. The developed classifier provides high accuracy of about 95.5% and allows for calculation of survival rates in the course of cell death.

Deep Learning-Based Single-Cell Optical Image Studies

Optical imaging technology that has the advantages of high sensitivity and cost-effectiveness greatly promotes the progress of nondestructive single-cell studies. Complex cellular image analysis tasks such as three-dimensional reconstruction call for machine-learning technology in cell optical image research. With the rapid developments of high-throughput imaging flow cytometry, big data cell optical images are always obtained that may require machine learning for data analysis. In recent years, deep learning has been prevalent in the field of machine learning for large-scale image processing and analysis, which brings a new dawn for single-cell optical image studies with an explosive growth of data availability. Popular deep learning techniques offer new ideas for multimodal and multitask single-cell optical image research. This article provides an overview of the basic knowledge of deep learning and its applications in single-cell optical image studies. We explore the feasibility of applying deep learning techniques to single-cell optical image analysis, where popular techniques such as transfer learning, multimodal learning, multitask learning, and end-to-end learning have been reviewed. Image preprocessing and deep learning model training methods are then summarized. Applications based on deep learning techniques in the field of single-cell optical image studies are reviewed, which include image segmentation, super-resolution image reconstruction, cell tracking, cell counting, cross-modal image reconstruction, and design and control of cell imaging systems. In addition, deep learning in popular single-cell optical imaging techniques such as label-free cell optical imaging, high-content screening, and high-throughput optical imaging cytometry are also mentioned. Finally, the perspectives of deep learning technology for single-cell optical image analysis are discussed. © 2020 International Society for Advancement of Cytometry.

High temporal and spatial resolution single-shot digital holography with Fresnel domain filtering using witch's hat illumination

High-resolution, single-shot on-axis digital holography is proposed. Generally, an on-axis configuration samples carrier fringes with higher spatial resolution compared to an off-axis configuration. However, the reconstructed image is obtained with unnecessary images of a conjugate image and a zero-order beam. The proposed method uses a phase-modulated illumination beam and image processing to eliminate these unnecessary images. Since time-division and parallel phase-shifting methods are not required, the proposed method has higher temporal and spatial resolutions. During image processing, the conjugate image is removed by filtering on the Fresnel domain while keeping most of the information of the object image intact. The usefulness of the proposed method is confirmed by a numerical simulation and an optical experiment.

Optimal processing scheme for restoration of phase data corrupted by strong decorrelation noise and dislocations

The presence of speckle noise and dislocations makes phase restoration potentially difficult in quantitative phase imaging and metrology. Unfortunately, there is no appropriate approach to deal with phase data corrupted by high speckle noise and phase dislocations. Usually, processing schemes may deal with low-pass phase filtering, phase unwrapping, or phase inpainting. This paper discusses the efficient processing to deal with noisy phase maps corrupted with phase dislocations. Six processing schemes, combining four operations, are evaluated. The investigation is carried out by realistic numerical simulations in which strong decorrelation phase noise and phase dislocations are generated. As a result, most robust and faster processing is established. The applicability of the optimal scheme is demonstrated through deformation measurement in dental materials.

Accurate detection of small particles in digital holography using fully convolutional networks

Particle detection is a key procedure in particle field characterization with digital holography. Due to various background noises, spurious small particles might be generated and real small particles might be lost during particle detection. Therefore, accurate small particle detection remains a challenge in the research of energy and combustion. A deep learning method based on modified fully convolutional networks is proposed to detect small opaque particles (e.g., coal particles) on extended focus images. The model is tested by several experiments and proved to have good small particle detection accuracy.

Phase curvature compensation in digital holographic microscopy based on phase gradient fitting and optimization

We propose a numerical method for phase curvature compensation in digital holographic microscopy, in which the phase curvature is compensated for by subtracting a numerical phase mask from the distorted phase. The parameters of the phase mask are obtained based on phase gradient fitting and optimization, in which the initial mask parameters are obtained by fitting the phase gradient, and then more accurate mask parameters are determined using a spectrum energy search. The compensation can be executed in a hologram without extra devices or any prior knowledge of the setup and specimen. A computer simulation and experimental results demonstrated the feasibility of the proposed method.

Deep Learning Approach for Enhanced Detection of Surface Plasmon Scattering

A deep learning approach has been taken to improve detection characteristics of surface plasmon microscopy (SPM) of light scattering. Deep learning based on the convolutional neural network algorithm was used to estimate the effect of scattering parameters, mainly the number of scatterers. The improvement was assessed on a quantitative basis by applying the approach to SPM images formed by coherent interference of scatterers. It was found that deep learning significantly improves the accuracy over conventional detection: the enhancement in the accuracy was shown to be significantly higher by almost 6 times and useful for scattering by polydisperse mixtures. This suggests that deep learning can be used to find scattering objects effectively in the noisy environment. Furthermore, deep learning can be extended directly to label-free molecular detection assays and provide considerably improved detection in imaging and microscopy techniques.

Adaptive wavefront correction structured illumination holographic tomography

In this study, a novel adaptive wavefront correction (AWC) technique is implemented on a compactly developed structured illumination holographic tomography (SI-HT) system. We propose a mechanical movement-free compact scanning architecture for SI-HT systems with AWC, implemented by designing and displaying a series of computer-generated holograms (CGH) composed of blazed grating with phase Fresnel lens on a phase-only spatial light modulator (SLM). In the proposed SI-HT, the aberrations of the optical system are sensed by digital holography and are used to design the CGH-based AWC to compensate the phase aberrations of the tomographic imaging system. The proposed method was validated using a standard Siemens star target, its potential application was demonstrated using a live candida rugosa sample, and its label-free three-dimensional refractive index profile was generated at its subcellular level. The experimental results obtained reveal the ability of the proposed method to enhance the imaging performance in both lateral and axial directions.

Adaptive frequency filtering based on convolutional neural networks in off-axis digital holographic microscopy

Digital holographic microscopy (DHM) as a label-free quantitative imaging tool has been widely used to investigate the morphology of living cells dynamically. In the off-axis DHM, the spatial filtering in the frequency spectrum of the hologram is vital to the quality of the reconstructed images. In this paper, we propose an adaptive spatial filtering approach based on convolutional neural networks (CNN) to automatically extracts the optimal shape of frequency components. For achieving robust and precise recognition performance, the net model is trained by using the tens of thousands of frequency spectrums with a variety of specimens and imaging conditions. The experimental results demonstrate that the trained network produce an adaptive spatial filtering window which can accurately select the frequency components of the object term and eliminate the frequency components of the interference terms, especially the coherent noise that overlaps with the object term in the spatial frequency domain. We find that the proposed approach has a fast, robust, and outstanding frequency filtering capability without any manual intervention and initial input parameters compared to previous techniques. Furthermore, the applicability of the proposed method in off-axis DHM for dynamic analysis is demonstrated by real-time monitoring the morphologic changes of living MLO-Y4 cells that are constantly subject to Fluid Shear Stress (FSS).