Automatic phase aberration compensation for digital holographic microscopy based on deep learning background detection

Focus prediction in digital holographic microscopy using deep convolutional neural networks

Deep artificial neural network learning is an emerging tool in image analysis. We demonstrate its potential in the field of digital holographic microscopy by addressing the challenging problem of determining the in-focus reconstruction depth of Madin-Darby canine kidney cell clusters encoded in digital holograms. A deep convolutional neural network learns the in-focus depths from half a million hologram amplitude images. The trained network correctly determines the in-focus depth of new holograms with high probability, without performing numerical propagation. This paper reports on extensions to preliminary work published earlier as one of the first applications of deep learning in the field of digital holographic microscopy.

Multi-step phase aberration compensation method based on optimal principal component analysis and subsampling for digital holographic microscopy

Digital holographic microscopy (DHM) is a well-known powerful technique allowing measurement of the spatial distributions of both the amplitude and phase produced by a transparent sample. Nevertheless, in order to improve the transverse resolution of the DHM system, a microscope objective has to be introduced in the object beam path, which inevitably leads to phase aberration in the object wavefront. In recent decades, a multitude of techniques have been proposed to compensate for this phase aberration, and the principal component analysis (PCA) technique has proven to be one of the most promising approaches due to its high compensation accuracy, low computational complexity, and simplicity to implement. However, when it comes to high-order phase aberration, which is common for a mal-aligned DHM system, the PCA technique usually performs poorly since it is unable to fit the cross-terms of the standard Zernike polynomials. To address this problem, here we propose a multi-step phase-aberration-compensation method based on optimal PCA and sub-sampling where PCA is first applied to remove the non-cross-aberration terms, followed by sub-sampled fitting for the remaining cross-aberration correction. The key advantage of our approach is that it can handle both the conventional objective phase curvature and high-order aberrations such as astigmatism and anamorphism with very little computational overhead. Simulation and experimental results demonstrate that our method outperforms state-of-the-art approaches, and the compensation results are consistent with those obtained from the double-exposure method.

Digital holographic phase imaging with aberrations totally compensated

Digital holography is a well-accepted method for phase imaging. However, the phase of the object is always embedded in aberrations. Here, we present a digital holographic phase imaging with the aberrations fully compensated, including the high order aberrations. Instead of using pre-defined aberration models or 2D fitting, we used the simpler and more flexible 1D fitting. Although it is 1D fitting, data across the whole plane could be used. Theoretically, all types of aberrations can be compensated with this method. Experimental results show that the aberrations have been fully compensated and the pure object phase can be obtained for further studies.

Spectral pre-modulation of training examples enhances the spatial resolution of the phase extraction neural network (PhENN)

The phase extraction neural network (PhENN) [Optica 4, 1117 (2017)] is a computational architecture, based on deep machine learning, for lens-less quantitative phase retrieval from raw intensity data. PhENN is a deep convolutional neural network trained through examples consisting of pairs of true phase objects and their corresponding intensity diffraction patterns; thereafter, given a test raw intensity pattern, PhENN is capable of reconstructing the original phase object robustly, in many cases even for objects outside the database where the training examples were drawn from. Here, we show that the spatial frequency content of the training examples is an important factor limiting PhENN's spatial frequency response. For example, if the training database is relatively sparse in high spatial frequencies, as most natural scenes are, PhENN's ability to resolve fine spatial features in test patterns will be correspondingly limited. To combat this issue, we propose "flattening" the power spectral density of the training examples before presenting them to PhENN. For phase objects following the statistics of natural scenes, we demonstrate experimentally that the spectral pre-modulation method enhances the spatial resolution of PhENN by a factor of 2.

Wavelength-multiplexed digital holography for quantitative phase measurement using quantum dot film

We propose an enhanced quantitative three-dimensional measurement system using wavelength-multiplexed digital holography. To simplify the configuration, a dual-peak quantum dot wavelength converter, combined with a blue LED, is adapted as a single low-coherence light source. Rather than a conventional dual-wavelength method, which records and reconstruct the object wave for each wavelength, the proposed system can capture the holograms of two wavelengths simultaneously with fewer acquisitions, simple setup, and low noise. To verify the system's performance, the measurements of the step height sample are presented.

Multi-parallax views synthesis for three-dimensional light-field display using unsupervised CNN

Multi-view applications have been used in a wide range, especially Three-Dimensional (3D) display. Since capturing dense multiple views for 3D light-field display is still a difficult work, view synthesis becomes an accessible way. Convolutional neural networks (CNN) has been used to synthesize new views of the scene. However, training targets are sometimes difficult to obtain, and it views are very difficult to synthesize at arbitrary positions. Here, an unsupervised network of Multi-Parallax View Net (MPVN) is proposed, which can synthesize multi-parallax views for 3D light-field display. Existing parallax views are re-projected to the target position to build input towers. The network is operated on these towers, and outputs a color tower and a selection tower. These two towers yield the final output image by per-pixel weight summing. MPVN adopts end-to-end unsupervised training to minimize prediction errors at existing positions. It can predict virtual views at any parallax position between existing views in a high quality. Experimental results demonstrate the validation of our proposed network, and SSIM of synthetic views are mostly over 0.95. We believe that this method can effectively provide enough views for 3D light-field display in the future work.

Deep learning approach for Fourier ptychography microscopy

Convolutional neural networks (CNNs) have gained tremendous success in solving complex inverse problems. The aim of this work is to develop a novel CNN framework to reconstruct video sequences of dynamic live cells captured using a computational microscopy technique, Fourier ptychographic microscopy (FPM). The unique feature of the FPM is its capability to reconstruct images with both wide field-of-view (FOV) and high resolution, i.e. a large space-bandwidth-product (SBP), by taking a series of low resolution intensity images. For live cell imaging, a single FPM frame contains thousands of cell samples with different morphological features. Our idea is to fully exploit the statistical information provided by these large spatial ensembles so as to make predictions in a sequential measurement, without using any additional temporal dataset. Specifically, we show that it is possible to reconstruct high-SBP dynamic cell videos by a CNN trained only on the first FPM dataset captured at the beginning of a time-series experiment. Our CNN approach reconstructs a 12800×10800 pixel phase image using only ∼25 seconds, a 50× speedup compared to the model-based FPM algorithm. In addition, the CNN further reduces the required number of images in each time frame by ∼ 6×. Overall, this significantly improves the imaging throughput by reducing both the acquisition and computational times. The proposed CNN is based on the conditional generative adversarial network (cGAN) framework. We further propose a mixed loss function that combines the standard image domain loss and a weighted Fourier domain loss, which leads to improved reconstruction of the high frequency information. Additionally, we also exploit transfer learning so that our pre-trained CNN can be further optimized to image other cell types. Our technique demonstrates a promising deep learning approach to continuously monitor large live-cell populations over an extended time and gather useful spatial and temporal information with sub-cellular resolution.

eHoloNet: a learning-based end-to-end approach for in-line digital holographic reconstruction

It is well known that in-line digital holography (DH) makes use of the full pixel count in forming the holographic imaging. But it usually requires phase-shifting or phase retrieval techniques to remove the zero-order and twin-image terms, resulting in the so-called two-step reconstruction process, i.e., phase recovery and focusing. Here, we propose a one-step end-to-end learning-based method for in-line holography reconstruction, namely, the eHoloNet, which can reconstruct the object wavefront directly from a single-shot in-line digital hologram. In addition, the proposed learning-based DH technique has strong robustness to the change of optical path difference between reference beam and object light and does not require the reference beam to be a plane or spherical wave.

Compression of Phase-Only Holograms with JPEG Standard and Deep Learning

It is a critical issue to reduce the enormous amount of data in the processing, storage and transmission of a hologram in digital format. In photograph compression, the JPEG standard is commonly supported by almost every system and device. It will be favorable if JPEG standard is applicable to hologram compression, with advantages of universal compatibility. However, the reconstructed image from a JPEG compressed hologram suffers from severe quality degradation since some high frequency features in the hologram will be lost during the compression process. In this work, we employ a deep convolutional neural network to reduce the artifacts in a JPEG compressed hologram. Simulation and experimental results reveal that our proposed “JPEG + deep learning” hologram compression scheme can achieve satisfactory reconstruction results for a computer-generated phase-only hologram after compression.

Fast phase retrieval in off-axis digital holographic microscopy through deep learning

Traditional digital holographic imaging algorithms need multiple iterations to obtain focused reconstructed image, which is time-consuming. In terms of phase retrieval, there is also the problem of phase compensation in addition to focusing task. Here, a new method is proposed for fast digital focus, where we use U-type convolutional neural network (U-net) to recover the original phase of microscopic samples. Generated data sets are used to simulate different degrees of defocused image, and verify that the U-net can restore the original phase to a great extent and realize phase compensation at the same time. We apply this method in the construction of real-time off-axis digital holographic microscope and obtain great breakthroughs in imaging speed.

Automatic phase aberration compensation for digital holographic microscopy based on phase variation minimization

We propose a numerical and totally automatic phase aberration compensation method in digital holographic microscopy. The phase aberrations are extracted in a nonlinear optimization procedure in which the phase variation of the reconstructed object wave is minimized. Not only phase curvature but also high-order aberrations could be corrected without extra devices. The correction is directly carried out with the wrapped phase map, which is not affected by phase unwrapping or fitting errors. Numerical simulation proves that the proposed method is more accurate than the conventional surface fitting method without selecting a cell-free background. Experimental results demonstrate the availability of the proposed method in real-time analysis of living cells.

Three-dimensional facial digitization using advanced digital image correlation

Presented in this paper is an effective technique to acquire the three-dimensional (3D) digital images of the human face without the use of active lighting and artificial patterns. The technique is based on binocular stereo imaging and digital image correlation, and it includes two key steps: camera calibration and image matching. The camera calibration involves a pinhole model and a bundle-adjustment approach, and the governing equations of the 3D digitization process are described. For reliable pixel-to-pixel image matching, the skin pores and freckles or lentigines on the human face serve as the required pattern features to facilitate the process. It employs feature-matching-based initial guess, multiple subsets, iterative optimization algorithm, and reliability-guided computation path to achieve fast and accurate image matching. Experiments have been conducted to demonstrate the validity of the proposed technique. The simplicity of the approach and the affordable cost of the implementation show its practicability in scientific and engineering applications.

Region Based CNN for Foreign Object Debris Detection on Airfield Pavement

In this paper, a novel algorithm based on convolutional neural network (CNN) is proposed to detect foreign object debris (FOD) based on optical imaging sensors. It contains two modules, the improved region proposal network (RPN) and spatial transformer network (STN) based CNN classifier. In the improved RPN, some extra select rules are designed and deployed to generate high quality candidates with fewer numbers. Moreover, the efficiency of CNN detector is significantly improved by introducing STN layer. Compared to faster R-CNN and single shot multiBox detector (SSD), the proposed algorithm achieves better result for FOD detection on airfield pavement in the experiment.

Quantitative assessment of cancer cell morphology and motility using telecentric digital holographic microscopy and machine learning

The noninvasive, fast acquisition of quantitative phase maps using digital holographic microscopy (DHM) allows tracking of rapid cellular motility on transparent substrates. On two-dimensional surfaces in vitro, MDA-MB-231 cancer cells assume several morphologies related to the mode of migration and substrate stiffness, relevant to mechanisms of cancer invasiveness in vivo. The quantitative phase information from DHM may accurately classify adhesive cancer cell subpopulations with clinical relevance. To test this, cells from the invasive breast cancer MDA-MB-231 cell line were cultured on glass, tissue-culture treated polystyrene, and collagen hydrogels, and imaged with DHM followed by epifluorescence microscopy after staining F-actin and nuclei. Trends in cell phase parameters were tracked on the different substrates, during cell division, and during matrix adhesion, relating them to F-actin features. Support vector machine learning algorithms were trained and tested using parameters from holographic phase reconstructions and cell geometric features from conventional phase images, and used to distinguish between elongated and rounded cell morphologies. DHM was able to distinguish between elongated and rounded morphologies of MDA-MB-231 cells with 94% accuracy, compared to 83% accuracy using cell geometric features from conventional brightfield microscopy. This finding indicates the potential of DHM to detect and monitor cancer cell morphologies relevant to cell cycle phase status, substrate adhesion, and motility. © 2017 International Society for Advancement of Cytometry.