3D Imaging Based on Depth Measurement Technologies

Development of Augmented Reality Vision for Osteosynthesis Using a 3D Camera

BACKGROUND

We developed a 3D camera system to track motion in a surgical field. This system has the potential to introduce augmented reality (AR) systems non-invasively, eliminating the need for the invasive AR markers conventionally required. The present study was performed to verify the real-time tracking accuracy of this system, assess the feasibility of integrating this system into the surgical workflow, and establish its potential to enhance the accuracy and efficiency of orthopedic procedures.

METHODS

To evaluate the accuracy of AR technology using a 3D camera, a forearm bone model was created. The forearm model was depicted using a 3D camera, and its accuracy was verified in terms of the positional relationship with a 3D bone model created from previously imaged CT data. Images of the surgical field (capturing the actual forearm) were taken and saved in nine poses by rotating the forearm from pronation to supination. The alignment of the reference points was computed at the three points of CT versus the three points of the 3D camera, yielding a 3D rotation matrix representing the positional relationship. In the original system, a stereo vision-based 3D camera, with a depth image resolution of 1280×720 pixels, 30 frames per second, and a lens field of view of 64 specifications, with a baseline of 3 cm, capable of optimally acquiring real-time 3D data at a distance of 40-60 cm from the subject was used. In the modified system, the following modifications were made to improve tracking performance: (1) color filter processing was changed from HSV to RGB, (2) positional detection accuracy was modified with supporting marker sizes of 8 mm in diameter, and (3) the detection of marker positions was stabilized by calculating the marker position for each frame. Tracking accuracy was examined with the original system and modified system for the following parameters: differences in the rotation matrix, maximum and minimum inter-reference point errors between CT-based and camera-based 3D data, and the average error for the three reference points.

RESULTS

In the original system, the average difference in rotation matrices was 5.51±2.68 mm. Average minimum and maximum errors were 1.10±0.61 and 15.53±12.51 mm, respectively. The average error of reference points was 6.26±4.49 mm. In the modified system, the average difference in rotation matrices was 4.22±1.73 mm. Average minimum and maximum errors were 0.79±0.49 and 1.94±0.87 mm, respectively. The average error of reference points was 1.41±0.58 mm. In the original system, once tracking failed, it was difficult to recover tracking accuracy. This resulted in a large maximum error in supination positions. These issues were resolved by the modified system. Significant improvements were achieved in maximum errors and average errors using the modified system (P<0.05).

CONCLUSION

AR technology using a 3D camera was developed. This system allows direct comparisons of 3D data from preoperative CT scans with 3D data acquired from the surgical field using a 3D camera. This method has the advantage of introducing AR into the surgical field without invasive markers.

Large-scale optical compression of free-space using an experimental three-lens spaceplate

Recently introduced, spaceplates achieve the propagation of light for a distance greater than their thickness. In this way, they compress optical space, reducing the required distance between optical elements in an imaging system. Here we introduce a spaceplate based on conventional optics in a 4-f arrangement, mimicking the transfer function of free-space in a thinner system - we term this device a three-lens spaceplate. It is broadband, polarization-independent, and can be used for meter-scale space compression. We experimentally measure compression ratios up to 15.6, replacing up to 4.4 meters of free-space, three orders of magnitude greater than current optical spaceplates. We demonstrate that three-lens spaceplates reduce the length of a full-color imaging system, albeit with reductions in resolution and contrast. We present theoretical limits on the numerical aperture and the compression ratio. Our design presents a simple, accessible, cost-effective method for optically compressing large amounts of space.

High-resolution 3D shape measurement with extended depth of field using fast chromatic focus stacking

Close-range 3D sensors based on the structured light principle have a constrained measuring range due to their depth of field (DOF). Focus stacking is a method to extend the DOF. The additional time to change the focus is a drawback in high-speed measurements. In our research, the method of chromatic focus stacking was applied to a high-speed 3D sensor with 180 fps frame rate. The extended DOF was evaluated by the distance-dependent 3D resolution derived from the 3D-MTF of a tilted edge. The conventional DOF of 14 mm was extended to 21 mm by stacking two foci at 455 and 520 nm wavelength. The 3D sensor allowed shape measurements with extended DOF within 44 ms.

Complex amplitude field reconstruction in atmospheric turbulence based on deep learning

In this paper, we use deep neural networks (DNNs) to simultaneously reconstruct the amplitude and phase information of the complex light field transmitted in atmospheric turbulence based on deep learning. The results of amplitude and phase reconstruction by four different training methods are compared comprehensively. The obtained results indicate that the training method that can more accurately reconstruct the complex amplitude field is to input the amplitude and phase pattern pairs into the neural network as two channels to train the model.

Three-Dimensional Reconstruction of Light Field Based on Phase Similarity

Light field imaging plays an increasingly important role in the field of three-dimensional (3D) reconstruction because of its ability to quickly obtain four-dimensional information (angle and space) of the scene. In this paper, a 3D reconstruction method of light field based on phase similarity is proposed to increase the accuracy of depth estimation and the scope of applicability of epipolar plane image (EPI). The calibration method of the light field camera was used to obtain the relationship between disparity and depth, and the projector calibration was removed to make the experimental procedure more flexible. Then, the disparity estimation algorithm based on phase similarity was designed to effectively improve the reliability and accuracy of disparity calculation, in which the phase information was used instead of the structure tensor, and the morphological processing method was used to denoise and optimize the disparity map. Finally, 3D reconstruction of the light field was realized by combining disparity information with the calibrated relationship. The experimental results showed that the reconstruction standard deviation of the two objects was 0.3179 mm and 0.3865 mm compared with the ground truth of the measured objects, respectively. Compared with the traditional EPI method, our method can not only make EPI perform well in a single scene or blurred texture situations but also maintain good reconstruction accuracy.

Multi-depth hologram generation using stochastic gradient descent algorithm with complex loss function

The stochastic gradient descent (SGD) method is useful in the phase-only hologram optimization process and can achieve a high-quality holographic display. However, for the current SGD solution in multi-depth hologram generation, the optimization time increases dramatically as the number of depth layers of object increases, leading to the SGD method nearly impractical in hologram generation of the complicated three-dimensional object. In this paper, the proposed method uses a complex loss function instead of an amplitude-only loss function in the SGD optimization process. This substitution ensures that the total loss function can be obtained through only one calculation, and the optimization time can be reduced hugely. Moreover, since both the amplitude and phase parts of the object are optimized, the proposed method can obtain a relatively accurate complex amplitude distribution. The defocus blur effect is therefore matched with the result from the complex amplitude reconstruction. Numerical simulations and optical experiments have validated the effectiveness of the proposed method.

Light Field View Synthesis via Aperture Disparity and Warping Confidence Map

This paper presents a learning-based approach to synthesize the view from an arbitrary camera position given a sparse set of images. A key challenge for this novel view synthesis arises from the reconstruction process, when the views from different input images may not be consistent due to obstruction in the light path. We overcome this by jointly modeling the epipolar property and occlusion in designing a convolutional neural network. We start by defining and computing the aperture disparity map, which approximates the parallax and measures the pixel-wise shift between two views. While this relates to free-space rendering and can fail near the object boundaries, we further develop a warping confidence map to address pixel occlusion in these challenging regions. The proposed method is evaluated on diverse real-world and synthetic light field scenes, and it shows better performance over several state-of-the-art techniques.

Light field reconstruction with randomly shot photographs

Light field imaging is a promising technique for recording and displaying three-dimensional (3D) scenes. Light field reconstruction using a conventional camera, instead of a lens-array-based plenoptic camera, is expected to have a higher resolution. However, in existing conventional-camera-based methods, the camera placement is restricted. In this paper, we propose a new flexible light field reconstruction method involving shooting using a conventional camera from random 3D positions and angles. The proposed method has the advantage of flexibility because it uses a conventional camera, and the shooting position and angle can be selected randomly. Such flexibility is expected to give it a wide range of applications.

Method of chromatic aberration elimination in holographic display based on zoomable liquid lens

In this paper, we propose a method of chromatic aberration elimination in holographic display based on a zoomable liquid lens. The liquid lens is filled with two immiscible liquids and developed by using the principle of electrowetting. The shape at the liquid-liquid interface changes with the voltage applied to the liquid lens, so the focal length can be adjusted by changing the voltage. By using the liquid lens in the holographic display system, the position of the reconstructed image can be controlled. When three color lasers illuminate the corresponding holograms and the focal length of the liquid lens changes accordingly, three color images can coincide in the same location clearly. The experimental results verify its feasibility.

Digital holographic phase imaging based on phase iteratively enhanced compressive sensing

Digital holography has been widely applied in quantitative phase imaging (QPI) for monolayer objects within a limited depth. For multilayer imaging, compressive sensing is employed to eliminate defocused images but with missing phase information. A phase iteratively enhanced compressive sensing (PIE-CS) algorithm is proposed to achieve phase imaging and eliminate defocused images simultaneously. Linear filtering is first applied to the off-axis hologram in Fourier domain, and an intermediate reconstructed complex image is obtained. A periodic phase mask is then superimposed on the intermediate reconstructed image to iteratively eliminate the defocused images and recover the object with phase information. The experimental recovery of amplitude and phase of a two-layer sample with as little as 7% random measurement is demonstrated. The average phase error of the PIE-CS algorithm is analyzed, and the results show the feasibility for QPI.