A naked eye 3D display and interaction system for medical education and training

Modeling and optimizing through plenoptic function for the dual lenticular lens-based directional autostereoscopic display system

We propose an autostereoscopic display system that ensures full resolution for multiple users by directional backlight and eye tracking technology. The steerable beam formed by directional backlight can be regarded as the result of sparsely sampling the light field in space. Therefore, we intuitively propose an optimization algorithm based on the characterization for the state of the steerable beams, which is computed in matrix form using the plenoptic function. This optimization algorithm aims to optimize the exit pupil quality and ultimately enhancing the viewing experience of stereoscopic display. Numerical simulations are conducted and the improvement in exit pupil quality achieved by the optimization scheme is verified. Furthermore, a prototype of the stereoscopic display that employs dual-lenticular lens sheets for the directional backlight has been constructed using the optimized optical parameters. It provides 9 independent exit pupils at the optimal viewing distance of 400 mm, with an exit pupil resolution of 1/30. The field of view is ±16.7°, the viewing distance range is 380 mm to 440 mm. At the optimal viewing distance 400 mm, the average crosstalk of the system is 3%, and the dynamic brightness uniformity across the entire viewing plane reaches 85%. The brightness uniformity of the display at each exit pupil is higher than 88%.

Aerial floating 3D display device with lenticular-type naked-eye 3D display and a crossed-mirror array

The depth feeling of a naked-eye 3D display is determined by the combination of the display and the lenticular lens, and because the parallax of the displayed image cannot be made stronger, providing a more stereoscopic effect is difficult. In this study, we propose a system that provides a clearer stereoscopic effect by enabling the naked-eye 3D display to be observed as an aerial stereoscopic image without the framework of the display. By comparing it with the conventional aerial display method, we demonstrate that it is possible to maintain the same level of resolution as the method without a crossed-mirror array and that natural stereoscopic view is possible.

Augmented reality-based autostereoscopic surgical visualization system for telesurgery

PURPOSE

The visualization of remote surgical scenes is the key to realizing the remote operation of surgical robots. However, current non-endoscopic surgical robot systems lack an effective visualization tool to offer sufficient surgical scene information and depth perception.

METHODS

We propose a novel autostereoscopic surgical visualization system integrating 3D intraoperative scene reconstruction, autostereoscopic 3D display, and augmented reality-based image fusion. The preoperative organ structure and the intraoperative surface point cloud are obtained from medical imaging and the RGB-D camera, respectively, and aligned by an automatic marker-free intraoperative registration algorithm. After registration, preoperative meshes with precalculated illumination and intraoperative textured point cloud are blended in real time. Finally, the fused image is shown on a 3D autostereoscopic display device to achieve depth perception.

RESULTS

A prototype of the autostereoscopic surgical visualization system was built. The system had a horizontal image resolution of 1.31 mm, a vertical image resolution of 0.82 mm, an average rendering rate of 33.1 FPS, an average registration rate of 20.5 FPS, and average registration errors of approximately 3 mm. A telesurgical robot prototype based on 3D autostereoscopic display was built. The quantitative evaluation experiments showed that our system achieved similar operational accuracy (1.79 ± 0.87 mm) as the conventional system (1.95 ± 0.71 mm), while having advantages in terms of completion time (with 34.11% reduction) and path length (with 35.87% reduction). Post-experimental questionnaires indicated that the system was user-friendly for novices and experts.

CONCLUSION

We propose a 3D surgical visualization system with augmented instruction and depth perception for telesurgery. The qualitative and quantitative evaluation results illustrate the accuracy and efficiency of the proposed system. Therefore, it shows great prospects in robotic surgery and telesurgery.

Roadmap on 3D integral imaging: sensing, processing, and display

This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field.