AR in VR: assessing surgical augmented reality visualizations in a steerable virtual reality environment

Applications and advances of immersive technology in cardiology

Different forms of immersive technology, such as Virtual Reality (VR) and Augmented Reality (AR), are getting increasingly invested in medicine. Advances in head-mounted display technology, processing, and rendering power have demonstrated the increasing utility of immersive technology in medicine and the healthcare environment. There are a growing number of publications on using immersive technology in cardiology. We reviewed the articles published within the last decade that reported case studies or research that uses or investigates the application of immersive technology in the broad field of cardiology - from education to preoperative planning and intraoperative guidance. We summarized the advantages and disadvantages of using AR and VR for various application categories. Our review highlights the need for a robust assessment of the effectiveness of the methods and discusses the technical limitations that hinder the complete integration of AR and VR in cardiology, including cost-effectiveness and regulatory compliance. Despite the limitations and gaps that have inhibited us from benefiting from immersive technologies' full potential in cardiology settings to date, its promising, impactful future for standard cardiovascular care is undoubted.

AR visualizations in laparoscopy: surgeon preferences and depth assessment of vascular anatomy

Introduction: This study compares five augmented reality (AR) vasculature visualization techniques in a mixed-reality laparoscopy simulator with 50 medical professionals and analyzes their impact on the surgeon. Material and methods: ​​The different visualization techniques' abilities to convey depth were measured using the participant's accuracy in an objective depth sorting task. Demographic data and subjective measures, such as the preference of each AR visualization technique and potential application areas, were collected with questionnaires. Results: Despite measuring differences in objective measurements across the visualization techniques, they were not statistically significant. In the subjective measures, however, 55% of the participants rated visualization technique II, 'Opaque with single-color Fresnel highlights', as their favorite. Participants felt that AR could be useful for various surgeries, especially complex surgeries (100%). Almost all participants agreed that AR could potentially improve surgical parameters, such as patient safety (88%), complication rate (84%), and identifying risk structures (96%). Conclusions: More studies are needed on the effect of different visualizations on task performance, as well as more sophisticated and effective visualization techniques for the operating room. With the findings of this study, we encourage the development of new study setups to advance surgical AR.

Comparative effectiveness of virtual reality (VR) vs 3D printed models of congenital heart disease in resident and nurse practitioner educational experience

BACKGROUND

Medical trainees frequently note that cardiac anatomy is difficult to conceive within a two dimensional framework. The specific anatomic defects and the subsequent pathophysiology in flow dynamics may become more apparent when framed in three dimensional models. Given the evidence of improved comprehension using such modeling, this study aimed to contribute further to that understanding by comparing Virtual Reality (VR) and 3D printed models (3DP) in medical education.

OBJECTIVES

We sought to systematically compare the perceived subjective effectiveness of Virtual Reality (VR) and 3D printed models (3DP) in the educational experience of residents and nurse practitioners.

METHODS

Trainees and practitioners underwent individual 15-minute teaching sessions in which features of a developmentally typical heart as well as a congenitally diseased heart were demonstrated using both Virtual Reality (VR) and 3D printed models (3DP). Participants then briefly explored each modality before filling out a short survey in which they identified which model (3DP or VR) they felt was more effective in enhancing their understanding of cardiac anatomy and associated pathophysiology. The survey included a binary summative assessment and a series of Likert scale questions addressing usefulness of each model type and degree of comfort with each modality.

RESULTS

Twenty-seven pediatric residents and 3 nurse practitioners explored models of a developmentally typical heart and tetralogy of Fallot pathology. Most participants had minimal prior exposure to VR (1.1 ± 0.4) or 3D printed models (2.1 ± 1.5). Participants endorsed a greater degree of understanding with VR models (8.5 ± 1) compared with 3D Printed models (6.3 ± 1.8) or traditional models of instruction (5.5 ± 1.5) p < 0.001. Most participants felt comfortable with modern technology (7.6 ± 2.1). 87% of participants preferred VR over 3DP.

CONCLUSIONS

Our study shows that, overall, VR was preferred over 3DP models by pediatric residents and nurse practitioners for understanding cardiac anatomy and pathophysiology.

In-Depth Review of Augmented Reality: Tracking Technologies, Development Tools, AR Displays, Collaborative AR, and Security Concerns

Augmented reality (AR) has gained enormous popularity and acceptance in the past few years. AR is indeed a combination of different immersive experiences and solutions that serve as integrated components to assemble and accelerate the augmented reality phenomena as a workable and marvelous adaptive solution for many realms. These solutions of AR include tracking as a means for keeping track of the point of reference to make virtual objects visible in a real scene. Similarly, display technologies combine the virtual and real world with the user's eye. Authoring tools provide platforms to develop AR applications by providing access to low-level libraries. The libraries can thereafter interact with the hardware of tracking sensors, cameras, and other technologies. In addition to this, advances in distributed computing and collaborative augmented reality also need stable solutions. The various participants can collaborate in an AR setting. The authors of this research have explored many solutions in this regard and present a comprehensive review to aid in doing research and improving different business transformations. However, during the course of this study, we identified that there is a lack of security solutions in various areas of collaborative AR (CAR), specifically in the area of distributed trust management in CAR. This research study also proposed a trusted CAR architecture with a use-case of tourism that can be used as a model for researchers with an interest in making secure AR-based remote communication sessions.

Phantom study on surgical performance in augmented reality laparoscopy

Purpose

Only a few studies have evaluated Augmented Reality (AR) in in vivo simulations compared to traditional laparoscopy; further research is especially needed regarding the most effective AR visualization technique. This pilot study aims to determine, under controlled conditions on a 3D-printed phantom, whether an AR laparoscope improves surgical outcomes over conventional laparoscopy without augmentation.

Methods

We selected six surgical residents at a similar level of training and had them perform a laparoscopic task. The participants repeated the experiment three times, using different 3D phantoms and visualizations: Floating AR, Occlusion AR, and without any AR visualization (Control). Surgical performance was determined using objective measurements. Subjective measures, such as task load and potential application areas, were collected with questionnaires.

Results

Differences in operative time, total touching time, and SurgTLX scores showed no statistical significance ($$p>0.05$$ p > 0.05 ). However, when assessing the invasiveness of the simulated intervention, the comparison revealed a statistically significant difference ($$p=0.009$$ p = 0.009 ). Participants felt AR could be useful for various surgeries, especially for liver, sigmoid, and pancreatic resections (100%). Almost all participants agreed that AR could potentially lead to improved surgical parameters, such as operative time (83%), complication rate (83%), and identifying risk structures (83%).

Conclusion

According to our results, AR may have great potential in visceral surgery and based on the objective measures of the study, may improve surgeons' performance in terms of an atraumatic approach. In this pilot study, participants consistently took more time to complete the task, had more contact with the vascular tree, were significantly more invasive, and scored higher on the SurgTLX survey than with AR.

HEARTBEAT4D: An Open-source Toolbox for Turning 4D Cardiac CT into VR/AR

Four-dimensional data sets are increasingly common in MRI and CT. While clinical visualization often focuses on individual temporal phases capturing the tissue(s) of interest, it may be possible to gain additional insight through exploring animated 3D reconstructions of physiological motion made possible by augmented or virtual reality representations of 4D patient imaging. Cardiac CT acquisitions can provide sufficient spatial resolution and temporal data to support advanced visualization, however, there are no open-source tools readily available to facilitate the transformation from raw medical images to dynamic and interactive augmented or virtual reality representations. To address this gap, we developed a workflow using free and open-source tools to process 4D cardiac CT imaging starting from raw DICOM data and ending with dynamic AR representations viewable on a phone, tablet, or computer. In addition to assembling the workflow using existing platforms (3D Slicer and Unity), we also contribute two new features: 1. custom software which can propagate a segmentation created for one cardiac phase to all others and export to surface files in a fully automated fashion, and 2. a user interface and linked code for the animation and interactive review of the surfaces in augmented reality. Validation of the surface-based areas demonstrated excellent correlation with radiologists' image-based areas (R > 0.99). While our tools were developed specifically for 4D cardiac CT, the open framework will allow it to serve as a blueprint for similar applications applied to 4D imaging of other tissues and using other modalities. We anticipate this and related workflows will be useful both clinically and for educational purposes.

A novel visualization system of using augmented reality in knee replacement surgery: Enhanced bidirectional maximum correntropy algorithm

BACKGROUND AND AIM

Image registration and alignment are the main limitations of augmented reality (AR)-based knee replacement surgery. This research aims to decrease the registration error, eliminate outcomes that are trapped in local minima to improve the alignment problems, handle the occlusion and maximize the overlapping parts.

METHODOLOGY

Markerless image registration method was used for AR-based knee replacement surgery to guide and visualize the surgical operation. While weight least square algorithm was used to enhance stereo camera-based tracking by filling border occlusion in right-to-left direction and non-border occlusion from left-to-right direction.

RESULTS

This study has improved video precision to 0.57-0.61 mm alignment error. Furthermore, with the use of bidirectional points, that is, forward and backward directional cloud point, the iteration on image registration was decreased. This has led to improve the processing time as well. The processing time of video frames was improved to 7.4-11.74 frames per second.

CONCLUSIONS

It seems clear that this proposed system has focused on overcoming the misalignment difficulty caused by the movement of patient and enhancing the AR visualization during knee replacement surgery. The proposed system was reliable and favourable which helps in eliminating alignment error by ascertaining the optimal rigid transformation between two cloud points and removing the outliers and non-Gaussian noise. The proposed AR system helps in accurate visualization and navigation of anatomy of knee such as femur, tibia, cartilage, blood vessels and so forth.