Diffraction model-informed neural network for unsupervised layer-based computer-generated holography

Non-convex optimization for inverse problem solving in computer-generated holography

Computer-generated holography is a promising technique that modulates user-defined wavefronts with digital holograms. Computing appropriate holograms with faithful reconstructions is not only a problem closely related to the fundamental basis of holography but also a long-standing challenge for researchers in general fields of optics. Finding the exact solution of a desired hologram to reconstruct an accurate target object constitutes an ill-posed inverse problem. The general practice of single-diffraction computation for synthesizing holograms can only provide an approximate answer, which is subject to limitations in numerical implementation. Various non-convex optimization algorithms are thus designed to seek an optimal solution by introducing different constraints, frameworks, and initializations. Herein, we overview the optimization algorithms applied to computer-generated holography, incorporating principles of hologram synthesis based on alternative projections and gradient descent methods. This is aimed to provide an underlying basis for optimized hologram generation, as well as insights into the cutting-edge developments of this rapidly evolving field for potential applications in virtual reality, augmented reality, head-up display, data encryption, laser fabrication, and metasurface design.

Compensated DOE in a VHG-based waveguide display to improve uniformity

Augmented reality head-mounted displays (AR-HMDs) utilizing diffractive waveguides have emerged as a popular research focus. However, the illuminance uniformity over the fields of view (FOV) is often unsatisfactory in volume holographic grating (VHG) based waveguide displays. This paper proposes a high uniformity AR waveguide display system. Firstly, the angular uniformity of the VHG-based waveguide displays is analyzed. Subsequently, diffractive optical elements (DOEs) are seamlessly integrated onto the outer coupling surface of the waveguide substrate to improve the angular uniformity through phase compensation. To design the DOE phase, the multi-objective stochastic gradient descent (MO-SGD) algorithm is proposed. A single DOE is used to compensating various images form the image source. A hybrid loss, which includes the learned perceptual image patch similarity (LPIPS) metric, is applied to enhance the algorithm performance. Simulation results show that the proposed method effectively suppresses illumination degradation at the edge FOV in exit pupil images of the waveguide display system. In the results, the peak signal-to-noise ratio (PSNR) is improved by 5.54 dB. Optical experiments validate the effectiveness of the proposed method. The measured nonuniformity (NU) against FOVs is improved by 53.05% from 0.3749 to 0.1760.

Vision transformer empowered physics-driven deep learning for omnidirectional three-dimensional holography

The inter-plane crosstalk and limited axial resolution are two key points that hinder the performance of three-dimensional (3D) holograms. The state-of-the-art methods rely on increasing the orthogonality of the cross-sections of a 3D object at different depths to lower the impact of inter-plane crosstalk. Such strategy either produces unidirectional 3D hologram or induces speckle noise. Recently, learning-based methods provide a new way to solve this problem. However, most related works rely on convolution neural networks and the reconstructed 3D holograms have limited axial resolution and display quality. In this work, we propose a vision transformer (ViT) empowered physics-driven deep neural network which can realize the generation of omnidirectional 3D holograms. Owing to the global attention mechanism of ViT, our 3D CGH has small inter-plane crosstalk and high axial resolution. We believe our work not only promotes high-quality 3D holographic display, but also opens a new avenue for complex inverse design in photonics.

Adaptive layer-based computer-generated holograms

We propose a novel, to the best of our knowledge, and fast adaptive layer-based (ALB) method for generating a computer-generated hologram (CGH) with accurate depth information. A complex three-dimensional (3D) object is adaptively divided into layers along the depth direction according to its own non-uniformly distributed depth coordinates, which reduces the depth error caused by the conventional layer-based method. Each adaptive layer generates a single-layer hologram using the angular spectrum method for diffraction, and the final hologram of a complex three-dimensional object is obtained by superimposing all the adaptive layer holograms. A hologram derived with the proposed method is referred to as an adaptive layer-based hologram (ALBH). Our demonstration shows that the desired reconstruction can be achieved with 52 adaptive layers in 8.7 s, whereas the conventional method requires 397 layers in 74.9 s.

Liquid lens based holographic camera for real 3D scene hologram acquisition using end-to-end physical model-driven network

With the development of artificial intelligence, neural network provides unique opportunities for holography, such as high fidelity and dynamic calculation. How to obtain real 3D scene and generate high fidelity hologram in real time is an urgent problem. Here, we propose a liquid lens based holographic camera for real 3D scene hologram acquisition using an end-to-end physical model-driven network (EEPMD-Net). As the core component of the liquid camera, the first 10 mm large aperture electrowetting-based liquid lens is proposed by using specially fabricated solution. The design of the liquid camera ensures that the multi-layers of the real 3D scene can be obtained quickly and with great imaging performance. The EEPMD-Net takes the information of real 3D scene as the input, and uses two new structures of encoder and decoder networks to realize low-noise phase generation. By comparing the intensity information between the reconstructed image after depth fusion and the target scene, the composite loss function is constructed for phase optimization, and the high-fidelity training of hologram with true depth of the 3D scene is realized for the first time. The holographic camera achieves the high-fidelity and fast generation of the hologram of the real 3D scene, and the reconstructed experiment proves that the holographic image has the advantage of low noise. The proposed holographic camera is unique and can be used in 3D display, measurement, encryption and other fields.

Real-time 4K computer-generated hologram based on encoding conventional neural network with learned layered phase

Learning-based computer-generated hologram (CGH) demonstrates great potential for real-time high-quality holographic displays. However, real-time 4K CGH generation for 3D scenes remains a challenge due to the computational burden. Here, a variant conventional neural network (CNN) is presented for CGH encoding with learned layered initial phases for layered CGH generation. Specifically, the CNN predicts the CGH based on the input complex amplitude on the CGH plane, and the learned initial phases act as a universal phase for any target images at the target depth layer. These phases are generated during the training process of the coding CNN to further optimize the quality. The CNN is trained to learn encoding 3D CGH by randomly selecting the depth layer in the training process, and contains only 938 parameters. The generation time for a 2D 4K CGH is 18 ms, and is increased by 12 ms for each layer in a layered 3D scene. The average Peak Signal to Noise Ratio (PSNR) of each layer is above 30dB in the depth range from 160 to 210 mm. Experiments verify that our method can achieve real-time layered 4K CGH generation.

Asymmetrical neural network for real-time and high-quality computer-generated holography

Computer-generated holography based on neural network holds great promise as a real-time hologram generation method. However, existing neural network-based approaches prioritize lightweight networks to achieve real-time display, which limits their capacity for network fitting. Here, we propose an asymmetrical neural network with a non-end-to-end structure that enhances fitting capacity and delivers superior real-time display quality. The non-end-to-end structure decomposes the overall task into two sub-tasks: phase prediction and hologram encoding. The asymmetrical design tailors each sub-network to its specific sub-task using distinct basic net-layers rather than relying on similar net-layers. This method allows for a sub-network with strong feature extraction and inference capabilities to match the phase predictor, while another sub-network with efficient coding capability matches the hologram encoder. By matching network functions to tasks, our method enhances the overall network's fitting capacity while maintaining a lightweight architecture. Both numerical reconstructions and optical experiments validate the reliability and effectiveness of our proposed method.

Diffraction model-driven neural network trained using hybrid domain loss for real-time and high-quality computer-generated holography

Learning-based computer-generated holography (CGH) has demonstrated great potential in enabling real-time, high-quality holographic displays. However, most existing learning-based algorithms still struggle to produce high-quality holograms, due to the difficulty of convolutional neural networks (CNNs) in learning cross-domain tasks. Here, we present a diffraction model-driven neural network (Res-Holo) using hybrid domain loss for phase-only hologram (POH) generation. Res-Holo utilizes the weights of the pretrained ResNet34 as the initialization during the encoder stage of the initial phase prediction network to extract more generic features and also to help prevent overfitting. Also, frequency domain loss is added to further constrain the information that the spatial domain loss is insensitive. The peak signal-to-noise ratio (PSNR) of the reconstructed image is improved by 6.05 dB using hybrid domain loss compared to using spatial domain loss alone. Simulation results show that the proposed Res-Holo can generate high-fidelity 2 K resolution POHs with an average PSNR of 32.88 dB at 0.014 seconds/frame on the DIV2K validation set. Both monochrome and full-color optical experiments show that the proposed method can effectively improve the quality of reproduced images and suppress image artifacts.

End-to-end real-time holographic display based on real-time capture of real scenes

Holographic display is considered as a promising three-dimensional (3D) display technology and has been widely studied. However, to date, the real-time holographic display for real scenes is still far from being incorporated in our life. The speed and quality of information extraction and holographic computing need to be further improved. In this paper, we propose an end-to-end real-time holographic display based on real-time capture of real scenes, where the parallax images are collected from the scene and a convolutional neural network (CNN) builds the mapping from the parallax images to the hologram. Parallax images are acquired in real time by a binocular camera, and contain depth information and amplitude information needed for 3D hologram calculation. The CNN, which can transform parallax images into 3D holograms, is trained by datasets consisting of parallax images and high-quality 3D holograms. The static colorful reconstruction and speckle-free real-time holographic display based on real-time capture of real scenes have been verified by the optical experiments. With simple system composition and affordable hardware requirements, the proposed technique will break the dilemma of the existing real-scene holographic display, and open up a new direction for the application of real-scene holographic 3D display such as holographic live video and solving vergence-accommodation conflict (VAC) problems for head-mounted display devices.

Focus Issue Introduction: 3D Image Acquisition and Display: Technology, Perception and Applications

This Feature Issue of Optics Express is organized in conjunction with the 2022 Optica conference on 3D Image Acquisition and Display: Technology, Perception and Applications which was held in hybrid format from 11 to 15, July 2022 as part of the Imaging and Applied Optics Congress and Optical Sensors and Sensing Congress 2022 in Vancouver, Canada. This Feature Issue presents 31 articles which cover the topics and scope of the 2022 3D Image Acquisition and Display conference. This Introduction provides a summary of these published articles that appear in this Feature Issue.

Reducing crosstalk of a multi-plane holographic display by the time-multiplexing stochastic gradient descent

Multi-plane reconstruction is essential for realizing a holographic three-dimensional (3D) display. One fundamental issue in conventional multi-plane Gerchberg-Saxton (GS) algorithm is the inter-plane crosstalk, mainly caused by the neglect of other planes' interference in the process of amplitude replacement at each object plane. In this paper, we proposed the time-multiplexing stochastic gradient descent (TM-SGD) optimization algorithm to reduce the multi-plane reconstruction crosstalk. First, the global optimization feature of stochastic gradient descent (SGD) was utilized to reduce the inter-plane crosstalk. However, the crosstalk optimization effect would degrade as the number of object planes increases, due to the imbalance between input and output information. Thus, we further introduced the time-multiplexing strategy into both the iteration and reconstruction process of multi-plane SGD to increase input information. In TM-SGD, multiple sub-holograms are obtained through multi-loop iteration and then sequentially refreshed on spatial light modulator (SLM). The optimization condition between the holograms and the object planes converts from one-to-many to many-to-many, improving the optimization of inter-plane crosstalk. During the persistence of vision, multiple sub-hologram jointly reconstruct the crosstalk-free multi-plane images. Through simulation and experiment, we confirmed that TM-SGD could effectively reduce the inter-plane crosstalk and improve image quality.The proposed TM-SGD-based holographic display has wide applications in tomographic 3D visualization for biology, medical science, and engineering design, which need to reconstruct multiple independent tomographic images without inter-plane crosstalk.

Diffraction model-informed neural network for unsupervised layer-based computer-generated holography

Learning-based computer-generated holography (CGH) has shown remarkable promise to enable real-time holographic displays. Supervised CGH requires creating a large-scale dataset with target images and corresponding holograms. We propose a diffraction model-informed neural network framework (self-holo) for 3D phase-only hologram generation. Due to the angular spectrum propagation being incorporated into the neural network, the self-holo can be trained in an unsupervised manner without the need of a labeled dataset. Utilizing the various representations of a 3D object and randomly reconstructing the hologram to one layer of a 3D object keeps the complexity of the self-holo independent of the number of depth layers. The self-holo takes amplitude and depth map images as input and synthesizes a 3D hologram or a 2D hologram. We demonstrate 3D reconstructions with a good 3D effect and the generalizability of self-holo in numerical and optical experiments.