Augmented Reality Visualization for Image-Guided Surgery: A Validation Study Using a Three-Dimensional Printed Phantom

HoloDIEP-Faster and More Accurate Intraoperative DIEA Perforator Mapping Using a Novel Mixed Reality Tool

BACKGROUND

 Microsurgical breast reconstruction using abdominal tissue is a complex procedure, in part, due to variable vascular/perforator anatomy. Preoperative computed tomography angiography (CTA) has mitigated this challenge to some degree; yet it continues to pose certain challenges. The ability to map perforators with Mixed Reality has been demonstrated in case studies, but its accuracy has not been studied intraoperatively. Here, we compare the accuracy of "HoloDIEP" in identifying perforator location (vs. Doppler ultrasound) by using holographic 3D models derived from preoperative CTA.

METHODS

 Using a custom application on HoloLens, the deep inferior epigastric artery vascular tree was traced in 15 patients who underwent microsurgical breast reconstruction. Perforator markings were compared against the 3D model in a coordinate system centered on the umbilicus. Holographic- and Doppler-identified markings were compared using a perspective-corrected photo technique against the 3D model along with measurement of duration of perforator mapping for each technique.

RESULTS

 Vascular points in HoloDIEP skin markings were -0.97 ± 6.2 mm (perforators: -0.62 ± 6.13 mm) away from 3D-model ground-truth in radial length from the umbilicus at a true distance of 10.81 ± 6.14 mm (perforators: 11.40 ± 6.15 mm). Absolute difference in radial distance was twice as high for Doppler markings compared with Holo-markings (9.71 ± 6.16 and 4.02 ± 3.20 mm, respectively). Only in half of all cases (7/14), more than 50% of the Doppler-identified points were reasonably close (<30 mm) to 3D-model ground-truth. HoloDIEP was twice as fast as Doppler ultrasound (76.9s vs. 150.4 s per abdomen).

CONCLUSION

 HoloDIEP allows for faster and more accurate intraoperative perforator mapping than Doppler ultrasound.

Exploring the Practical Applications of Artificial Intelligence, Deep Learning, and Machine Learning in Maxillofacial Surgery: A Comprehensive Analysis of Published Works

Artificial intelligence (AI), deep learning (DL), and machine learning (ML) are computer, machine, and engineering systems that mimic human intelligence to devise procedures. These technologies also provide opportunities to advance diagnostics and planning in human medicine and dentistry. The purpose of this literature review was to ascertain the applicability and significance of AI and to highlight its uses in maxillofacial surgery. Our primary inclusion criterion was an original paper written in English focusing on the use of AI, DL, or ML in maxillofacial surgery. The sources were PubMed, Scopus, and Web of Science, and the queries were made on the 31 December 2023. The search strings used were "artificial intelligence maxillofacial surgery", "machine learning maxillofacial surgery", and "deep learning maxillofacial surgery". Following the removal of duplicates, the remaining search results were screened by three independent operators to minimize the risk of bias. A total of 324 publications from 1992 to 2023 were finally selected. These were calculated according to the year of publication with a continuous increase (excluding 2012 and 2013) and R2 = 0.9295. Generally, in orthognathic dentistry and maxillofacial surgery, AI and ML have gained popularity over the past few decades. When we included the keywords "planning in maxillofacial surgery" and "planning in orthognathic surgery", the number significantly increased to 7535 publications. The first publication appeared in 1965, with an increasing trend (excluding 2014-2018), with an R2 value of 0.8642. These technologies have been found to be useful in diagnosis and treatment planning in head and neck surgical oncology, cosmetic and aesthetic surgery, and oral pathology. In orthognathic surgery, they have been utilized for diagnosis, treatment planning, assessment of treatment needs, and cephalometric analyses, among other applications. This review confirms that the current use of AI and ML in maxillofacial surgery is focused mainly on evaluating digital diagnostic methods, especially radiology, treatment plans, and postoperative results. However, as these technologies become integrated into maxillofacial surgery and robotic surgery in the head and neck region, it is expected that they will be gradually utilized to plan and comprehensively evaluate the success of maxillofacial surgeries.

Novel Use of Virtual Reality and Augmented Reality in Temporomandibular Total Joint Replacement Using Stock Prosthesis

This technical innovation demonstrates the use of ImmersiveTouch Virtual Reality (VR) and Augmented Reality (AR)-guided total temporomandibular joint replacement (TJR) using Biomet stock prosthesis in 2 patients with condylar degeneration. TJR VR planning includes condylar resection, prosthesis selection and positioning, and interference identification. AR provides real-time guidance for osteotomies, placement of prostheses and fixation screws, occlusion verification, and flexibility to modify the surgical course. Radiographic analysis demonstrated high correspondence between the preoperative plan and postoperative result. The average differences in the positioning of the condylar and fossa prosthesis are 1.252 ± 0.269 mm and 1.393 ± 0.335 mm, respectively. The main challenges include a steep learning curve, intraoperative technical difficulties, added surgical time, and additional costs. In conclusion, the case report demonstrates the advantages of implementing AR and VR technology in TJR's using stock prostheses as a pilot study. Further clinical trials are needed prior to this innovation becoming a mainstream practice.

Surgical margins in head and neck squamous cell carcinoma: A narrative review

Head and neck squamous cell carcinoma (HNSCC), a prevalent and frequently recurring malignancy, often necessitates surgical intervention. The surgical margin (SM) plays a pivotal role in determining the postoperative treatment strategy and prognostic evaluation of HNSCC. Nonetheless, the process of clinical appraisal and assessment of the SMs remains a complex and indeterminate endeavor, thereby leading to potential difficulties for surgeons in defining the extent of resection. In this regard, we undertake a comprehensive review of the suggested surgical distance in varying circumstances, diverse methods of margin evaluation, and the delicate balance that must be maintained between tissue resection and preservation in head and neck surgical procedures. This review is intended to provide surgeons with pragmatic guidance in selecting the most suitable resection techniques, and in improving patients' quality of life by achieving optimal functional and aesthetic restoration.

Augmented Reality Head-Mounted Device and Dynamic Navigation System for Postremoval in Maxillary Molars

INTRODUCTION

This study evaluates the feasibility of an augmented reality (AR) head-mounted device (HMD) displaying a dynamic navigation system (DNS) in the surgical site for fiber postremoval in maxillary molars and compares it to the DNS technique.

METHODS

Fifty maxillary first molars were divided into 2 groups: AR HMD + DNS (n = 25) and DNS (n = 25). The palatal canal was restored with RelyX fiber post (3M ESPE) luted with RelyX Unicem (3M ESPE). A core buildup was performed using Paracore (Coltene/Whaledent). Cone beam computed tomography (CBCT) scans were taken before and after postremoval. The drilling trajectory and depth were planned under X-guide software (X-Nav Technologies, Lansdale, PA). For the AR HMD + DNS group, the AR HMD (Microsoft HoloLens 2) displayed the DNS in the surgical site. The three dimensional (3D) deviations (Global coronal deviation [GCD] and global apical deviation [GAD]) and angular deflection (AD) were calculated. The number of mishaps and operating time were recorded.

RESULTS

Fiber post was removed from all samples (50/50). The AR HMD + DNS was more accurate than DNS, showing significantly lower GCD and GAD deviations and AD (P < .05). No mishap was detected. The AR HMD + DNS was as efficient in time as DNS (P > .05).

CONCLUSIONS

Within the limitations of this in vitro study, the AR HMD can safely display DNS in the surgical site for fiber post-removal in maxillary molars. AR HMD improved the DNS accuracy. Both AR HMD + DNS and DNS were time-efficient for fiber postremoval in maxillary molars.

A System for Mixed-Reality Holographic Overlays of Real-Time Rendered 3D-Reconstructed Imaging Using a Video Pass-through Head-Mounted Display-A Pathway to Future Navigation in Chest Wall Surgery

Background: Three-dimensional reconstructions of state-of-the-art high-resolution imaging are progressively being used more for preprocedural assessment in thoracic surgery. It is a promising tool that aims to improve patient-specific treatment planning, for example, for minimally invasive or robotic-assisted lung resections. Increasingly available mixed-reality hardware based on video pass-through technology enables the projection of image data as a hologram onto the patient. We describe the novel method of real-time 3D surgical planning in a mixed-reality setting by presenting three representative cases utilizing volume rendering. Materials: A mixed-reality system was set up using a high-performance workstation running a video pass-through-based head-mounted display. Image data from computer tomography were imported and volume-rendered in real-time to be customized through live editing. The image-based hologram was projected onto the patient, highlighting the regions of interest. Results: Three oncological cases were selected to explore the potentials of the mixed-reality system. Two of them presented large tumor masses in the thoracic cavity, while a third case presented an unclear lesion of the chest wall. We aligned real-time rendered 3D holographic image data onto the patient allowing us to investigate the relationship between anatomical structures and their respective body position. Conclusions: The exploration of holographic overlay has proven to be promising in improving preprocedural surgical planning, particularly for complex oncological tasks in the thoracic surgical field. Further studies on outcome-related surgical planning and navigation should therefore be conducted. Ongoing technological progress of extended reality hardware and intelligent software features will most likely enhance applicability and the range of use in surgical fields within the near future.

Augmented Reality Integration in Skull Base Neurosurgery: A Systematic Review

Background and Objectives: To investigate the role of augmented reality (AR) in skull base (SB) neurosurgery. Materials and Methods: Utilizing PRISMA methodology, PubMed and Scopus databases were explored to extract data related to AR integration in SB surgery. Results: The majority of 19 included studies (42.1%) were conducted in the United States, with a focus on the last five years (77.8%). Categorization included phantom skull models (31.2%, n = 6), human cadavers (15.8%, n = 3), or human patients (52.6%, n = 10). Microscopic surgery was the predominant modality in 10 studies (52.6%). Of the 19 studies, surgical modality was specified in 18, with microscopic surgery being predominant (52.6%). Most studies used only CT as the data source (n = 9; 47.4%), and optical tracking was the prevalent tracking modality (n = 9; 47.3%). The Target Registration Error (TRE) spanned from 0.55 to 10.62 mm. Conclusion: Despite variations in Target Registration Error (TRE) values, the studies highlighted successful outcomes and minimal complications. Challenges, such as device practicality and data security, were acknowledged, but the application of low-cost AR devices suggests broader feasibility.

The use of virtual reality to assess the bony landmarks at the knee joint - The role of imaging modality and the assessor's experience

BACKGROUND

At present, extended reality technologies such as virtual reality (VR) have gained popularity in orthopedic surgery. The first aim of this study was to assess the precision of VR and other imaging modalities - computed tomography (CT), magnetic resonance imaging (MRI) - to localize bony landmarks near the knee joint. Secondly, the impact of the educational level of the assessor - medical master students, orthopedic residents, and orthopedic surgeons - on the precision with which landmarks near the knee joint could be localized was analyzed.

METHODS

We included a total of 77 participants: 62 medical master students, 10 orthopedic residents, and 5 orthopedic surgeons to analyze three cadaver legs. Every participant localized a series of sixteen bony landmarks on six different imaging modalities (CT, MRI, 3D-CT, 3D-MRI, VR-CT, VR-MRI).

RESULTS

Concerning the imaging modality, the inter- and intra-observer variability were lowest for 3D and VR, higher for MRI (respectively 7.6 mm and 6.9 mm), and highest for CT (respectively 9 mm and 8.7 m).Concerning the educational level of the assessor, inter- and intra-observer variability in VR were lowest for surgeons, (respectively 3.2 mm and 3.6 mm), higher for residents (respectively 5.9 mm and 6.5 mm) and medical students (respectively 5.9 mm and 5.8 mm).

CONCLUSIONS

VR can be considered a reliable imaging technique. Localization of landmarks tends to be more precise in VR and on 3D than on conventional CT and MRI images. Furthermore, orthopedic surgeons localize landmarks more precisely than orthopedic residents and medical students in VR.

Augmented Reality and 3-Dimensional Dynamic Navigation System Integration for Osteotomy and Root-end Resection

INTRODUCTION

Augmented reality (AR) superimposes high-definition computer-generated virtual content onto the existing environment, providing users with an enhanced perception of reality. This study investigates the feasibility of integrating an AR head-mounted device into a 3-dimensional dynamic navigation system (3D-DNS) for osteotomy and root-end resection (RER). It compares the accuracy and efficiency of AR + 3D-DNS to 3D-DNS for osteotomy and RER.

METHODS

Seventy-two tooth roots of 3D-printed surgical jaw models were divided into two groups: AR + 3D-DNS (n = 36) and 3D-DNS (n = 36). Cone-beam computed tomography scans were taken pre and postoperatively. The osteotomy and RER were virtually planned on X-guide software and delivered under 3D-DNS guidance. For the AR + 3D-DNS group, an AR head-mounted device (Microsoft HoloLens 2) was integrated into the 3D-DNS. The 2D- and 3D-deviations were calculated. The osteotomy and RER time and the number of procedural mishaps were recorded.

RESULTS

Osteotomy and RER were completed in all samples (72/72). AR + 3D-DNS was more accurate than 3D-DNS, showing lower 2D- and 3D-deviation values (P < .05). The AR + 3D-DNS was more efficient in time than 3D-DNS (P < .05). There was no significant difference in the number of mishaps (P > .05).

CONCLUSIONS

Within the limitations of this in vitro study, the integration of an AR head-mounted device to 3D-DNS is feasible for osteotomy and RER. AR improved the accuracy and time efficiency of 3D-DNS in osteotomy and RER. Head-mounted AR has the potential to be safely and reliably integrated into 3D-DNS for endodontic microsurgery.

The Metaverse for Healthcare: Trends, Applications, and Future Directions of Digital Therapeutics for Urology

In recent years, the emergence of digital therapeutics as a novel approach to managing conditions has garnered significant attention. This approach involves using evidence-based therapeutic interventions that are facilitated by high-quality software programs to treat, manage, or prevent medical conditions. The incorporation of digital therapeutics into the Metaverse has increased the feasibility of their implementation and application in all areas of medical services. In urology, substantial digital therapeutics are being produced and researched, including mobile apps, bladder devices, pelvic floor muscle trainers, smart toilet systems, mixed reality-guided training and surgery, and training and telemedicine for urological consultations. The purpose of this review article is to provide a comprehensive overview of the current impact of the Metaverse on the field of digital therapeutics and identify its current trends, applications, and future perspectives in the field of urology.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Visualization, registration and tracking techniques for augmented reality guided surgery: a review

Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories ofin situvisualization and nonin situvisualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.

Familiar size affects perception differently in virtual reality and the real world

The promise of virtual reality (VR) as a tool for perceptual and cognitive research rests on the assumption that perception in virtual environments generalizes to the real world. Here, we conducted two experiments to compare size and distance perception between VR and physical reality (Maltz et al. 2021 J. Vis. 21, 1-18). In experiment 1, we used VR to present dice and Rubik's cubes at their typical sizes or reversed sizes at distances that maintained a constant visual angle. After viewing the stimuli binocularly (to provide vergence and disparity information) or monocularly, participants manually estimated perceived size and distance. Unlike physical reality, where participants relied less on familiar size and more on presented size during binocular versus monocular viewing, in VR participants relied heavily on familiar size regardless of the availability of binocular cues. In experiment 2, we demonstrated that the effects in VR generalized to other stimuli and to a higher quality VR headset. These results suggest that the use of binocular cues and familiar size differs substantially between virtual and physical reality. A deeper understanding of perceptual differences is necessary before assuming that research outcomes from VR will generalize to the real world. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Augmented Reality to Assist Skin Paddle Harvesting in Osteomyocutaneous Fibular Flap Reconstructive Surgery: A Pilot Evaluation on a 3D-Printed Leg Phantom

Background

Augmented Reality (AR) represents an evolution of navigation-assisted surgery, providing surgeons with a virtual aid contextually merged with the real surgical field. We recently reported a case series of AR-assisted fibular flap harvesting for mandibular reconstruction. However, the registration accuracy between the real and the virtual content needs to be systematically evaluated before widely promoting this tool in clinical practice. In this paper, after description of the AR based protocol implemented for both tablet and HoloLens 2 smart glasses, we evaluated in a first test session the achievable registration accuracy with the two display solutions, and in a second test session the success rate in executing the AR-guided skin paddle incision task on a 3D printed leg phantom.

Methods

From a real computed tomography dataset, 3D virtual models of a human leg, including fibula, arteries and skin with planned paddle profile for harvesting, were obtained. All virtual models were imported into Unity software to develop a marker-less AR application suitable to be used both via tablet and via HoloLens 2 headset. The registration accuracy for both solutions was verified on a 3D printed leg phantom obtained from the virtual models, by repeatedly applying the tracking function and computing pose deviations between the AR-projected virtual skin paddle profile and the real one transferred to the phantom via a CAD/CAM cutting guide. The success rate in completing the AR-guided task of skin paddle harvesting was evaluated using CAD/CAM templates positioned on the phantom model surface.

Results

On average, the marker-less AR protocol showed comparable registration errors (ranging within 1-5 mm) for tablet-based and HoloLens-based solution. Registration accuracy seems to be quite sensitive to ambient light conditions. We found a good success rate in completing the AR-guided task within an error margin of 4 mm (97% and 100% for tablet and HoloLens, respectively). All subjects reported greater usability and ergonomics for HoloLens 2 solution.

Conclusions

Results revealed that the proposed marker-less AR based protocol may guarantee a registration error within 1-5 mm for assisting skin paddle harvesting in the clinical setting. Optimal lightening conditions and further improvement of marker-less tracking technologies have the potential to increase the efficiency and precision of this AR-assisted reconstructive surgery.