Focus prediction in digital holographic microscopy using deep convolutional neural networks

Quantitative phase imaging based on holography: trends and new perspectives

In 1948, Dennis Gabor proposed the concept of holography, providing a pioneering solution to a quantitative description of the optical wavefront. After 75 years of development, holographic imaging has become a powerful tool for optical wavefront measurement and quantitative phase imaging. The emergence of this technology has given fresh energy to physics, biology, and materials science. Digital holography (DH) possesses the quantitative advantages of wide-field, non-contact, precise, and dynamic measurement capability for complex-waves. DH has unique capabilities for the propagation of optical fields by measuring light scattering with phase information. It offers quantitative visualization of the refractive index and thickness distribution of weak absorption samples, which plays a vital role in the pathophysiology of various diseases and the characterization of various materials. It provides a possibility to bridge the gap between the imaging and scattering disciplines. The propagation of wavefront is described by the complex amplitude. The complex-value in the complex-domain is reconstructed from the intensity-value measurement by camera in the real-domain. Here, we regard the process of holographic recording and reconstruction as a transformation between complex-domain and real-domain, and discuss the mathematics and physical principles of reconstruction. We review the DH in underlying principles, technical approaches, and the breadth of applications. We conclude with emerging challenges and opportunities based on combining holographic imaging with other methodologies that expand the scope and utility of holographic imaging even further. The multidisciplinary nature brings technology and application experts together in label-free cell biology, analytical chemistry, clinical sciences, wavefront sensing, and semiconductor production.

Digital in-line holographic microscopy for label-free identification and tracking of biological cells

Digital in-line holographic microscopy (DIHM) is a non-invasive, real-time, label-free technique that captures three-dimensional (3D) positional, orientational, and morphological information from digital holographic images of living biological cells. Unlike conventional microscopies, the DIHM technique enables precise measurements of dynamic behaviors exhibited by living cells within a 3D volume. This review outlines the fundamental principles and comprehensive digital image processing procedures employed in DIHM-based cell tracking methods. In addition, recent applications of DIHM technique for label-free identification and digital tracking of various motile biological cells, including human blood cells, spermatozoa, diseased cells, and unicellular microorganisms, are thoroughly examined. Leveraging artificial intelligence has significantly enhanced both the speed and accuracy of digital image processing for cell tracking and identification. The quantitative data on cell morphology and dynamics captured by DIHM can effectively elucidate the underlying mechanisms governing various microbial behaviors and contribute to the accumulation of diagnostic databases and the development of clinical treatments.

Real-time 3D tracking of swimming microbes using digital holographic microscopy and deep learning

The three-dimensional swimming tracks of motile microorganisms can be used to identify their species, which holds promise for the rapid identification of bacterial pathogens. The tracks also provide detailed information on the cells' responses to external stimuli such as chemical gradients and physical objects. Digital holographic microscopy (DHM) is a well-established, but computationally intensive method for obtaining three-dimensional cell tracks from video microscopy data. We demonstrate that a common neural network (NN) accelerates the analysis of holographic data by an order of magnitude, enabling its use on single-board computers and in real time. We establish a heuristic relationship between the distance of a cell from the focal plane and the size of the bounding box assigned to it by the NN, allowing us to rapidly localise cells in three dimensions as they swim. This technique opens the possibility of providing real-time feedback in experiments, for example by monitoring and adapting the supply of nutrients to a microbial bioreactor in response to changes in the swimming phenotype of microbes, or for rapid identification of bacterial pathogens in drinking water or clinical samples.

Refocus criterion from image-plane speckle correlation in digital holographic interferometry

In digital holography and holographic interferometry, refocusing to the correct image plane can be challenging and may be obtained by various metrics. This paper proposes a digital refocus approach utilizing the linear relationship between in-plane speckle motion and defocus as a response to an induced phase gradient. The theory based on cross-correlations between pairs of intensity images reconstructed at different distances from the recording plane is discussed. Two simple metrics, based on the cross-correlation properties of the reconstructed speckle images, are proposed and evaluated utilizing both simulations and experiments. Experiments exhibit similar trends in which the estimate of the correct reconstruction distance differs by a small amount between the two metrics. The difference is found less than 1% in the estimate of the true reconstruction distance. The results show that either metric is able to yield a sufficient reconstruction distance for the reconstruction of the image plane.

On the use of deep learning for phase recovery

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.

High-precision microscopic autofocus with a single natural image

In industrial microscopic detection, learning-based autofocus methods have empowered operators to acquire high-quality images quickly. However, there are two parts of errors in Learning-based methods: the fitting error of the network model and the making error of the prior dataset, which limits the potential for further improvements in focusing accuracy. In this paper, a high-precision autofocus pipeline was introduced, which predicts the defocus distance from a single natural image. A new method for making datasets was proposed, which overcomes the limitations of the sharpness metric itself and improves the overall accuracy of the dataset. Furthermore, a lightweight regression network was built, namely Natural-image Defocus Prediction Model (NDPM), to improve the focusing accuracy. A realistic dataset of sufficient size was made to train all models. The experiment shows NDPM has better focusing performance compared with other models, with a mean focusing error of 0.422µm.

In-focus quantitative phase imaging from defocused off-axis holograms: synergistic reconstruction framework

Digital holographic microscopy (DHM) enables the three-dimensional (3D) reconstruction of quantitative phase distributions from a defocused hologram. Traditional computational algorithms follow a sequential approach in which one first reconstructs the complex amplitude distribution and later applies focusing algorithms to provide an in-focus phase map. In this work, we have developed a synergistic computational framework to compensate for the linear tilt introduced in off-axis DHM systems and autofocus the defocused holograms by minimizing a cost function, providing in-focus reconstructed phase images without phase distortions. The proposed computational tool has been validated in defocused holograms of human red blood cells and three-dimensional images of dynamic sperm cells.

Field recovery from digital inline holographic images of composite propellant combustion base on denoising diffusion model

Digital inline holography has gained extensive application in the optical diagnosis of solid propellant combustion. However, this method confronts several challenges. Firstly, the calculation time required for reconstruction and depth of field extension is excessively long. Secondly, the excessive smoke, airflow, and flame during combustion cause significant interference and poor reconstruction quality, which reduces the accuracy of particle identification. To address these issues, we have developed a holographic image reconstruction technique for aluminum particle combustion based on the Attention Mechanism, U-net, and Diffusion models. This approach enables end-to-end reconstruction of aluminum particle combustion holographic images, while effectively circumventing the interference of airflow combustion and flame.

Autofocus methods based on laser illumination

Autofocusing system plays an important role in microscopic measurement. However, natural-image-based autofocus methods encounter difficulties in improving focusing accuracy and robustness due to the diversity of detection objects. In this paper, a high-precision autofocus method with laser illumination was proposed, termed laser split-image autofocus (LSA), which actively endows the detection scene with image features. The common non-learning-based and learning-based methods for LSA were quantitatively analyzed and evaluated. Furthermore, a lightweight comparative framework model for LSA, termed split-image comparison model (SCM), was proposed to further improve the focusing accuracy and robustness, and a realistic split-image dataset of sufficient size was made to train all models. The experiment showed LSA has better focusing performance than natural-image-based method. In addition, SCM has a great improvement in accuracy and robustness compared with previous learning and non-learning methods, with a mean focusing error of 0.317µm in complex scenes. Therefore, SCM is more suitable for industrial measurement.

A Machine Learning Specklegram Wavemeter (MaSWave) Based on a Short Section of Multimode Fiber as the Dispersive Element

Wavemeters are very important for precise and accurate measurements of both pulses and continuous-wave optical sources. Conventional wavemeters employ gratings, prisms, and other wavelength-sensitive devices in their design. Here, we report a simple and low-cost wavemeter based on a section of multimode fiber (MMF). The concept is to correlate the multimodal interference pattern (i.e., speckle patterns or specklegrams) at the end face of an MMF with the wavelength of the input light source. Through a series of experiments, specklegrams from the end face of an MMF as captured by a CCD camera (acting as a low-cost interrogation unit) were analyzed using a convolutional neural network (CNN) model. The developed machine learning specklegram wavemeter (MaSWave) can accurately map specklegrams of wavelengths up to 1 pm resolution when employing a 0.1 m long MMF. Moreover, the CNN was trained with several categories of image datasets (from 10 nm to 1 pm wavelength shifts). In addition, analysis for different step-index and graded-index MMF types was carried out. The work shows how further robustness to the effects of environmental changes (mainly vibrations and temperature changes) can be achieved at the expense of decreased wavelength shift resolution, by employing a shorter length MMF section (e.g., 0.02 m long MMF). In summary, this work demonstrates how a machine learning model can be used for the analysis of specklegrams in the design of a wavemeter.

Multi-Class Classification and Multi-Output Regression of Three-Dimensional Objects Using Artificial Intelligence Applied to Digital Holographic Information

Digital holographically sensed 3D data processing, which is useful for AI-based vision, is demonstrated. Three prominent methods of learning from datasets such as sensed holograms, computationally retrieved intensity and phase from holograms forming concatenated intensity-phase (whole information) images, and phase-only images (depth information) were utilized for the proposed multi-class classification and multi-output regression tasks of the chosen 3D objects in supervised learning. Each dataset comprised 2268 images obtained from the chosen eighteen 3D objects. The efficacy of our approaches was validated on experimentally generated digital holographic data then further quantified and compared using specific evaluation matrices. The machine learning classifiers had better AUC values for different classes on the holograms and whole information datasets compared to the CNN, whereas the CNN had a better performance on the phase-only image dataset compared to these classifiers. The MLP regressor was found to have a stable prediction in the test and validation sets with a fixed EV regression score of 0.00 compared to the CNN, the other regressors for holograms, and the phase-only image datasets, whereas the RF regressor showed a better performance in the validation set for the whole information dataset with a fixed EV regression score of 0.01 compared to the CNN and other regressors.

Learning-based complex field recovery from digital hologram with various depth objects

In this paper, we investigate a learning-based complex field recovery technique of an object from its digital hologram. Most of the previous learning-based approaches first propagate the captured hologram to the object plane and then suppress the DC and conjugate noise in the reconstruction. To the contrary, the proposed technique utilizes a deep learning network to extract the object complex field in the hologram plane directly, making it robust to the object depth variations and well suited for three-dimensional objects. Unlike the previous approaches which concentrate on transparent biological samples having near-uniform amplitude, the proposed technique is applied to more general objects which have large amplitude variations. The proposed technique is verified by numerical simulations and optical experiments, demonstrating its feasibility.

Advances in Digital Holographic Interferometry

Holographic interferometry is a well-established field of science and optical engineering. It has a half-century history of successful implementation as the solution to numerous technical tasks and problems. However, fast progress in digital and computer holography has promoted it to a new level of possibilities and has opened brand new fields of its application. In this review paper, we consider some such new techniques and applications.

Fast autofocusing using tiny transformer networks for digital holographic microscopy

The numerical wavefront backpropagation principle of digital holography confers unique extended focus capabilities, without mechanical displacements along z-axis. However, the determination of the correct focusing distance is a non-trivial and time consuming issue. A deep learning (DL) solution is proposed to cast the autofocusing as a regression problem and tested over both experimental and simulated holograms. Single wavelength digital holograms were recorded by a Digital Holographic Microscope (DHM) with a 10x microscope objective from a patterned target moving in 3D over an axial range of 92 μm. Tiny DL models are proposed and compared such as a tiny Vision Transformer (TViT), tiny VGG16 (TVGG) and a tiny Swin-Transfomer (TSwinT). The proposed tiny networks are compared with their original versions (ViT/B16, VGG16 and Swin-Transformer Tiny) and the main neural networks used in digital holography such as LeNet and AlexNet. The experiments show that the predicted focusing distance ZRPred is accurately inferred with an accuracy of 1.2 μm in average in comparison with the DHM depth of field of 15 µm. Numerical simulations show that all tiny models give the ZRPred with an error below 0.3 µm. Such a prospect would significantly improve the current capabilities of computer vision position sensing in applications such as 3D microscopy for life sciences or micro-robotics. Moreover, all models reach an inference time on CPU, inferior to 25 ms per inference. In terms of occlusions, TViT based on its Transformer architecture is the most robust.

Deep learning-based image processing in optical microscopy

Optical microscopy has emerged as a key driver of fundamental research since it provides the ability to probe into imperceptible structures in the biomedical world. For the detailed investigation of samples, a high-resolution image with enhanced contrast and minimal damage is preferred. To achieve this, an automated image analysis method is preferable over manual analysis in terms of both speed of acquisition and reduced error accumulation. In this regard, deep learning (DL)-based image processing can be highly beneficial. The review summarises and critiques the use of DL in image processing for the data collected using various optical microscopic techniques. In tandem with optical microscopy, DL has already found applications in various problems related to image classification and segmentation. It has also performed well in enhancing image resolution in smartphone-based microscopy, which in turn enablse crucial medical assistance in remote places.

Graphical abstract

Hologram classification of occluded and deformable objects with speckle noise contamination by deep learning

Advancements in optical, computing, and electronic technologies have enabled holograms of physical three-dimensional (3D) objects to be captured. The hologram can be displayed with a spatial light modulator to reconstruct a visible image. Although holography is an ideal solution for recording 3D images, a hologram comprises high-frequency fringe patterns that are almost impossible to recognize with traditional computer vision methods. Recently, it has been shown that holograms can be classified with deep learning based on convolution neural networks. However, the method can only achieve a high success classification rate if the image represented in the hologram is without speckle noise and occlusion. Minor occlusion of the image generally leads to a substantial drop in the success rate. This paper proposes a method known as ensemble deep-learning invariant occluded hologram classification to overcome this problem. The proposed new method attains over 95% accuracy in the classification of holograms of partially occluded handwritten numbers contaminated with speckle noise. To achieve the performance, a new augmentation scheme and a new enhanced ensemble structure are necessary. The new augmentation process includes occluded objects and simulates the worst-case scenario of speckle noise.

Deep learning in optical metrology: a review

With the advances in scientific foundations and technological implementations, optical metrology has become versatile problem-solving backbones in manufacturing, fundamental research, and engineering applications, such as quality control, nondestructive testing, experimental mechanics, and biomedicine. In recent years, deep learning, a subfield of machine learning, is emerging as a powerful tool to address problems by learning from data, largely driven by the availability of massive datasets, enhanced computational power, fast data storage, and novel training algorithms for the deep neural network. It is currently promoting increased interests and gaining extensive attention for its utilization in the field of optical metrology. Unlike the traditional "physics-based" approach, deep-learning-enabled optical metrology is a kind of "data-driven" approach, which has already provided numerous alternative solutions to many challenging problems in this field with better performances. In this review, we present an overview of the current status and the latest progress of deep-learning technologies in the field of optical metrology. We first briefly introduce both traditional image-processing algorithms in optical metrology and the basic concepts of deep learning, followed by a comprehensive review of its applications in various optical metrology tasks, such as fringe denoising, phase retrieval, phase unwrapping, subset correlation, and error compensation. The open challenges faced by the current deep-learning approach in optical metrology are then discussed. Finally, the directions for future research are outlined.

Comprehensive deep learning model for 3D color holography

Holography is a vital tool used in various applications from microscopy, solar energy, imaging, display to information encryption. Generation of a holographic image and reconstruction of object/hologram information from a holographic image using the current algorithms are time-consuming processes. Versatile, fast in the meantime, accurate methodologies are required to compute holograms performing color imaging at multiple observation planes and reconstruct object/sample information from a holographic image for widely accommodating optical holograms. Here, we focus on design of optical holograms for generation of holographic images at multiple observation planes and colors via a deep learning model, the CHoloNet. The CHoloNet produces optical holograms which show multitasking performance as multiplexing color holographic image planes by tuning holographic structures. Furthermore, our deep learning model retrieves an object/hologram information from an intensity holographic image without requiring phase and amplitude information from the intensity image. We show that reconstructed objects/holograms show excellent agreement with the ground-truth images. The CHoloNet does not need iteratively reconstruction of object/hologram information while conventional object/hologram recovery methods rely on multiple holographic images at various observation planes along with the iterative algorithms. We openly share the fast and efficient framework that we develop in order to contribute to the design and implementation of optical holograms, and we believe that the CHoloNet based object/hologram reconstruction and generation of holographic images will speed up wide-area implementation of optical holography in microscopy, data encryption, and communication technologies.

Speeding up reconstruction of 3D tomograms in holographic flow cytometry via deep learning

Tomographic flow cytometry by digital holography is an emerging imaging modality capable of collecting multiple views of moving and rotating cells with the aim of recovering their refractive index distribution in 3D. Although this modality allows us to access high-resolution imaging with high-throughput, the huge amount of time-lapse holographic images to be processed (hundreds of digital holograms per cell) constitutes the actual bottleneck. This prevents the system from being suitable for lab-on-a-chip platforms in real-world applications, where fast analysis of measured data is mandatory. Here we demonstrate a significant speeding-up reconstruction of phase-contrast tomograms by introducing in the processing pipeline a multi-scale fully-convolutional context aggregation network. Although it was originally developed in the context of semantic image analysis, we demonstrate for the first time that it can be successfully adapted to a holographic lab-on-chip platform for achieving 3D tomograms through a faster computational process. We trained the network with input-output image pairs to reproduce the end-to-end holographic reconstruction process, i.e. recovering quantitative phase maps (QPMs) of single cells from their digital holograms. Then, the sequence of QPMs of the same rotating cell is used to perform the tomographic reconstruction. The proposed approach significantly reduces the computational time for retrieving tomograms, thus making them available in a few seconds instead of tens of minutes, while essentially preserving the high-content information of tomographic data. Moreover, we have accomplished a compact deep convolutional neural network parameterization that can fit into on-chip SRAM and a small memory footprint, thus demonstrating its possible exploitation to provide onboard computations for lab-on-chip devices with low processing hardware resources.

Holographic 3D particle reconstruction using a one-stage network

Volumetric reconstruction of a three-dimensional (3D) particle field with high resolution and low latency is an ambitious and valuable task. As a compact and high-throughput imaging system, digital holography (DH) encodes the 3D information of a particle volume into a two-dimensional (2D) interference pattern. In this work, we propose a one-stage network (OSNet) for 3D particle volumetric reconstruction. Specifically, by a single feed-forward process, OSNet can retrieve the 3D coordinates of the particles directly from the holograms without high-fidelity image reconstruction at each depth slice. Evaluation results from both synthetic and experimental data confirm the feasibility and robustness of our method under different particle concentrations and noise levels in terms of detection rate and position accuracy, with improved processing speed. The additional applications of 3D particle tracking are also investigated, facilitating the analysis of the dynamic displacements and motions for micro-objects or cells. It can be further extended to various types of computational imaging problems sharing similar traits.

Video-Rate Quantitative Phase Imaging Using a Digital Holographic Microscope and a Generative Adversarial Network

The conventional reconstruction method of off-axis digital holographic microscopy (DHM) relies on computational processing that involves spatial filtering of the sample spectrum and tilt compensation between the interfering waves to accurately reconstruct the phase of a biological sample. Additional computational procedures such as numerical focusing may be needed to reconstruct free-of-distortion quantitative phase images based on the optical configuration of the DHM system. Regardless of the implementation, any DHM computational processing leads to long processing times, hampering the use of DHM for video-rate renderings of dynamic biological processes. In this study, we report on a conditional generative adversarial network (cGAN) for robust and fast quantitative phase imaging in DHM. The reconstructed phase images provided by the GAN model present stable background levels, enhancing the visualization of the specimens for different experimental conditions in which the conventional approach often fails. The proposed learning-based method was trained and validated using human red blood cells recorded on an off-axis Mach–Zehnder DHM system. After proper training, the proposed GAN yields a computationally efficient method, reconstructing DHM images seven times faster than conventional computational approaches.

Deep learning for digital holography: a review

Recent years have witnessed the unprecedented progress of deep learning applications in digital holography (DH). Nevertheless, there remain huge potentials in how deep learning can further improve performance and enable new functionalities for DH. Here, we survey recent developments in various DH applications powered by deep learning algorithms. This article starts with a brief introduction to digital holographic imaging, then summarizes the most relevant deep learning techniques for DH, with discussions on their benefits and challenges. We then present case studies covering a wide range of problems and applications in order to highlight research achievements to date. We provide an outlook of several promising directions to widen the use of deep learning in various DH applications.

Machine Learning Assisted Classification of Cell Lines and Cell States on Quantitative Phase Images

In this report, we present implementation and validation of machine-learning classifiers for distinguishing between cell types (HeLa, A549, 3T3 cell lines) and states (live, necrosis, apoptosis) based on the analysis of optical parameters derived from cell phase images. Validation of the developed classifier shows the accuracy for distinguishing between the three cell types of about 93% and between different cell states of the same cell line of about 89%. In the field test of the developed algorithm, we demonstrate successful evaluation of the temporal dynamics of relative amounts of live, apoptotic and necrotic cells after photodynamic treatment at different doses.

Volumetric monitoring of airborne particulate matter concentration using smartphone-based digital holographic microscopy and deep learning

Airborne particulate matter (PM) has become a global environmental issue. This PM has harmful effects on public health and precision industries. Conventional air-quality monitoring methods usually utilize expensive equipment, and they are cumbersome to handle for accurate and high throughput measurements. In addition, commercial particle counters have technical limitations in high-concentration measurement, and data fluctuations are induced during air sampling. In this study, a novel smartphone-based technique for monitoring airborne PM concentrations was developed using smartphone-based digital holographic microscopy (S-DHM) and deep learning network called Holo-SpeckleNet. Holographic speckle images of various PM concentrations were recorded by the S-DHM system. The recorded speckle images and the corresponding ground truth PM concentrations were used to train deep learning algorithms consisting of a deep autoencoder and regression layers. The performance of the proposed smartphone-based PM monitoring technique was validated through hyperparameter optimization. The developed S-DHM integrated with Holo-SpeckleNet can be smartly and effectively utilized for portable PM monitoring and safety alarm provision under perilous environmental conditions.

Label-Free White Blood Cell Classification Using Refractive Index Tomography and Deep Learning

Objective and Impact Statement. We propose a rapid and accurate blood cell identification method exploiting deep learning and label-free refractive index (RI) tomography. Our computational approach that fully utilizes tomographic information of bone marrow (BM) white blood cell (WBC) enables us to not only classify the blood cells with deep learning but also quantitatively study their morphological and biochemical properties for hematology research. Introduction. Conventional methods for examining blood cells, such as blood smear analysis by medical professionals and fluorescence-activated cell sorting, require significant time, costs, and domain knowledge that could affect test results. While label-free imaging techniques that use a specimen's intrinsic contrast (e.g., multiphoton and Raman microscopy) have been used to characterize blood cells, their imaging procedures and instrumentations are relatively time-consuming and complex. Methods. The RI tomograms of the BM WBCs are acquired via Mach-Zehnder interferometer-based tomographic microscope and classified by a 3D convolutional neural network. We test our deep learning classifier for the four types of bone marrow WBC collected from healthy donors (n = 10 ): monocyte, myelocyte, B lymphocyte, and T lymphocyte. The quantitative parameters of WBC are directly obtained from the tomograms. Results. Our results show >99% accuracy for the binary classification of myeloids and lymphoids and >96% accuracy for the four-type classification of B and T lymphocytes, monocyte, and myelocytes. The feature learning capability of our approach is visualized via an unsupervised dimension reduction technique. Conclusion. We envision that the proposed cell classification framework can be easily integrated into existing blood cell investigation workflows, providing cost-effective and rapid diagnosis for hematologic malignancy.

Dual-wavelength interferogram decoupling method for three-frame generalized dual-wavelength phase-shifting interferometry based on deep learning

In dual-wavelength interferometry, the key issue is how to efficiently retrieve the phases at each wavelength using the minimum number of wavelength-multiplexed interferograms. To address this problem, a new dual-wavelength interferogram decoupling method with the help of deep learning is proposed in this study. This method requires only three randomly phase-shifted dual-wavelength interferograms. With a well-trained deep neural network, one can obtain three interferograms with arbitrary phase shifts at each wavelength. Using these interferograms, the wrapped phases of a single wavelength can be extracted, respectively, via an iterative phase retrieval algorithm, and then the phases at different synthetic beat wavelengths can be calculated. The feasibility and applicability of the proposed method are demonstrated by simulation experiments of the spherical cap and red blood cell, respectively. This method will provide a solution for the problem of phase retrieval in multiwavelength interferometry.

High-speed imaging of the sound field by parallel phase-shifting digital holography

Sound field imaging techniques have been found very useful for acoustic designs. Building on this idea, innovative techniques are needed and presented in this paper, where we report on developed imaging of the sound field radiated from speakers by parallel phase-shifting digital holography. We adopted an ultrasonic wave radiated from a speaker for an object. The phase distribution of the light wave was modulated by the sound field radiated from the speaker. The modulated phase distribution was recorded in the form of multiplexed phase-shifted holograms at the frame rate of 100,000 fps. A 40,000 Hz sound field radiated from a speaker is used as an observation target. Our proposed method can implement the imaging of the sound field successfully. Also, in order to demonstrate the digital refocusing capability of digital holography, we set two speakers, whose difference in depth positions was 6.6 cm, as a long-depth object. We demonstrated the digital refocusing on the two speakers along with the capability of measuring the positions of the objects. Furthermore, we succeeded in imaging of 40,000 Hz and 41,000 Hz sound fields radiated from the two speakers. The presented experimental results showed that parallel phase-shifting digital holography is very useful and suitable for sound field imaging.

Large volume holographic imaging for biological sample analysis

SIGNIFICANCE

Particle field holography is a versatile technique to determine the size and distribution of moving or stationary particles in air or in a liquid without significant disturbance of the sample volume. Although this technique is applied in biological sample analysis, it is limited to small sample volumes, thus increasing the number of measurements per sample. In this work, we characterize the maximum achievable volume limit based on the specification of a given sensor to realize the development of a potentially low-cost, single-shot, large-volume holographic microscope.

AIM

We present mathematical formulas that will aid in the design and development and improve the focusing speed for the numerical reconstruction of registered holograms in particle field holographic microscopes. Our proposed methodology has potential application in the detection of Schistosoma haematobium eggs in human urine samples.

APPROACH

Using the Fraunhofer holography theory for opaque objects, we derived an exact formula for the maximum diffraction-limited volume for an in-line holographic setup. The proof-of-concept device built based on the derived formulas was experimentally validated with urine spiked with cultured Schistosoma haematobium eggs.

RESULTS

Results obtained show that for urine spiked with Schistosoma haematobium eggs, the volume thickness is limited to several millimeters due to scattering properties of the sample. The distances of the target particles could be estimated directly from the hologram fringes.

CONCLUSION

The methodology proposed will aid in the development of large-volume holographic microscopes.

Deep neural network for multi-depth hologram generation and its training strategy

We present a deep neural network for generating a multi-depth hologram and its training strategy. The proposed network takes multiple images of different depths as inputs and calculates the complex hologram as an output, which reconstructs each input image at the corresponding depth. We design a structure of the proposed network and develop the dataset compositing method to train the network effectively. The dataset consists of multiple input intensity profiles and their propagated holograms. Rather than simply training random speckle images and their propagated holograms, we generate the training dataset by adjusting the density of the random dots or combining basic shapes to the dataset such as a circle. The proposed dataset composition method improves the quality of reconstructed images by the holograms generated by the network, called deep learning holograms (DLHs). To verify the proposed method, we numerically and optically reconstruct the DLHs. The results confirmed that the DLHs can reconstruct clear images at multiple depths similar to conventional multi-depth computer-generated holograms. To evaluate the performance of the DLH quantitatively, we compute the peak signal-to-noise ratio of the reconstructed images and analyze the reconstructed intensity patterns with various methods.

Classification of cell morphology with quantitative phase microscopy and machine learning

We describe and compare two machine learning approaches for cell classification based on label-free quantitative phase imaging with transport of intensity equation methods. In one approach, we design a multilevel integrated machine learning classifier including various individual models such as artificial neural network, extreme learning machine and generalized logistic regression. In another approach, we apply a pretrained convolutional neural network using transfer learning for the classification. As a validation, we show the performances of both approaches on classification between macrophages cultured in normal gravity and microgravity with quantitative phase imaging. The multilevel integrated classifier achieves average accuracy 93.1%, which is comparable to the average accuracy 93.5% obtained by convolutional neural network. The presented quantitative phase imaging system with two classification approaches could be helpful to biomedical scientists for easy and accurate cell analysis.

Deep learning-based hologram generation using a white light source

Digital holographic microscopy enables the recording of sample holograms which contain 3D volumetric information. However, additional optical elements, such as partially or fully coherent light source and a pinhole, are required to induce diffraction and interference. Here, we present a deep neural network based on generative adversarial network (GAN) to perform image transformation from a defocused bright-field (BF) image acquired from a general white light source to a holographic image. Training image pairs of 11,050 for image conversion were gathered by using a hybrid BF and hologram imaging technique. The performance of the trained network was evaluated by comparing generated and ground truth holograms of microspheres and erythrocytes distributed in 3D. Holograms generated from BF images through the trained GAN showed enhanced image contrast with 3-5 times increased signal-to-noise ratio compared to ground truth holograms and provided 3D positional information and light scattering patterns of the samples. The developed GAN-based method is a promising mean for dynamic analysis of microscale objects with providing detailed 3D positional information and monitoring biological samples precisely even though conventional BF microscopic setting is utilized.

Does deep learning always outperform simple linear regression in optical imaging?

Deep learning has been extensively applied in many optical imaging problems in recent years. Despite the success, the limitations and drawbacks of deep learning in optical imaging have been seldom investigated. In this work, we show that conventional linear-regression-based methods can outperform the previously proposed deep learning approaches for two black-box optical imaging problems in some extent. Deep learning demonstrates its weakness especially when the number of training samples is small. The advantages and disadvantages of linear-regression-based methods and deep learning are analyzed and compared. Since many optical systems are essentially linear, a deep learning network containing many nonlinearity functions sometimes may not be the most suitable option.

Deep Learning-Based Single-Cell Optical Image Studies

Optical imaging technology that has the advantages of high sensitivity and cost-effectiveness greatly promotes the progress of nondestructive single-cell studies. Complex cellular image analysis tasks such as three-dimensional reconstruction call for machine-learning technology in cell optical image research. With the rapid developments of high-throughput imaging flow cytometry, big data cell optical images are always obtained that may require machine learning for data analysis. In recent years, deep learning has been prevalent in the field of machine learning for large-scale image processing and analysis, which brings a new dawn for single-cell optical image studies with an explosive growth of data availability. Popular deep learning techniques offer new ideas for multimodal and multitask single-cell optical image research. This article provides an overview of the basic knowledge of deep learning and its applications in single-cell optical image studies. We explore the feasibility of applying deep learning techniques to single-cell optical image analysis, where popular techniques such as transfer learning, multimodal learning, multitask learning, and end-to-end learning have been reviewed. Image preprocessing and deep learning model training methods are then summarized. Applications based on deep learning techniques in the field of single-cell optical image studies are reviewed, which include image segmentation, super-resolution image reconstruction, cell tracking, cell counting, cross-modal image reconstruction, and design and control of cell imaging systems. In addition, deep learning in popular single-cell optical imaging techniques such as label-free cell optical imaging, high-content screening, and high-throughput optical imaging cytometry are also mentioned. Finally, the perspectives of deep learning technology for single-cell optical image analysis are discussed. © 2020 International Society for Advancement of Cytometry.

Accurate detection of small particles in digital holography using fully convolutional networks

Particle detection is a key procedure in particle field characterization with digital holography. Due to various background noises, spurious small particles might be generated and real small particles might be lost during particle detection. Therefore, accurate small particle detection remains a challenge in the research of energy and combustion. A deep learning method based on modified fully convolutional networks is proposed to detect small opaque particles (e.g., coal particles) on extended focus images. The model is tested by several experiments and proved to have good small particle detection accuracy.

Ensemble convolutional neural network for classifying holograms of deformable objects

Recently, a method known as "ensemble deep learning invariant hologram classification" (EDL-IHC) for classifying of holograms of deformable objects with deep learning network (DLN) has been demonstrated. However DL-IHC requires substantial computational resources to attain near perfect success rate (≥99%). In practice, it is always desirable to have higher success rate with a low complexity DLN. In this paper we propose a low complexity DLN known as "ensemble deep learning invariant hologram classification" (EDL-IHC). In comparison with DL-IHC, our proposed hologram classifier has promoted the success rate by 2.86% in the classification of holograms of handwritten numerals.

Y-Net: a one-to-two deep learning framework for digital holographic reconstruction

In this Letter, for the first time, to the best of our knowledge, we propose a digital holographic reconstruction method with a one-to-two deep learning framework (Y-Net). Perfectly fitting the holographic reconstruction process, the Y-Net can simultaneously reconstruct intensity and phase information from a single digital hologram. As a result, this compact network with reduced parameters brings higher performance than typical network variants. The experimental results of the mouse phagocytes demonstrate the advantages of the proposed Y-Net.

Robust autofocusing method for multi-wavelength lensless imaging

Lensless imaging based on multi-wavelength phase retrieval becomes a promising technology widely used as it has simple acquisition, miniaturized size and low-cost setup. However, measuring the sample-to-sensor distance with high accuracy, which is the key for high-resolution reconstruction, is still a challenge. In this work, we propose a multi-wavelength criterion to realize autofocusing modulation, i.e., achieving much higher accuracy in determining the sample-to-sensor distance, compared to the conventional methods. Three beams in different spectrums are adopted to illuminate the sample, and the resulting holograms are recorded by a CCD camera. The patterns calculated by performing back propagation of the recorded holograms, with exhaustively searched sample-to-sensor distance value, are adopted to access the criterion. Image sharpness can be accessed and the optimal sample-to-sensor distance can be finely determined by targeting the valley of the curve given by the criterion. Through our novel multi-wavelength based autofocusing strategy and executing further phase retrieval process, high-resolution images can be finally retrieved. The applicability and robustness of our method is validated both in simulations and experiments. Our technique provides a useful tool for multi-wavelength lensless imaging under limited experimental conditions.

No-search focus prediction at the single cell level in digital holographic imaging with deep convolutional neural network

Digital propagation of an off-axis hologram can provide the quantitative phase-contrast image if the exact distance between the sensor plane (such as CCD) and the reconstruction plane is correctly provided. In this paper, we present a deep-learning convolutional neural network with a regression layer as the top layer to estimate the best reconstruction distance. The experimental results obtained using microsphere beads and red blood cells show that the proposed method can accurately predict the propagation distance from a filtered hologram. The result is compared with the conventional automatic focus-evaluation function. Additionally, our approach can be utilized at the single-cell level, which is useful for cell-to-cell depth measurement and cell adherent studies.

One-step robust deep learning phase unwrapping

Phase unwrapping is an important but challenging issue in phase measurement. Even with the research efforts of a few decades, unfortunately, the problem remains not well solved, especially when heavy noise and aliasing (undersampling) are present. We propose a database generation method for phase-type objects and a one-step deep learning phase unwrapping method. With a trained deep neural network, the unseen phase fields of living mouse osteoblasts and dynamic candle flame are successfully unwrapped, demonstrating that the complicated nonlinear phase unwrapping task can be directly fulfilled in one step by a single deep neural network. Excellent anti-noise and anti-aliasing performances outperforming classical methods are highlighted in this paper.