A new head-mounted display-based augmented reality system in neurosurgical oncology: a study on phantom

Intraoperative glioblastoma surgery-current challenges and clinical trials: An update

Surgical excision is an important part of the multimodal therapy strategy for patients with glioblastoma, a very aggressive and invasive brain tumor. While major advances in surgical methods and technology have been accomplished, numerous hurdles remain in the field of glioblastoma surgery. The purpose of this literature review is to offer a thorough overview of the current challenges in glioblastoma surgery. We reviewed the difficulties associated with tumor identification and visualization, resection extent, neurological function preservation, tumor margin evaluation, and inclusion of sophisticated imaging and navigation technology. Understanding and resolving these challenges is critical in order to improve surgical results and, ultimately, patient survival.

More than meets the eye: Augmented reality in surgical oncology

BACKGROUND AND OBJECTIVES

In the field of surgical oncology, there has been a desire for innovative techniques to improve tumor visualization, resection, and patient outcomes. Augmented reality (AR) technology superimposes digital content onto the real-world environment, enhancing the user's experience by blending digital and physical elements. A thorough examination of AR technology in surgical oncology has yet to be performed.

METHODS

A scoping review of intraoperative AR in surgical oncology was conducted according to the guidelines and recommendations of The Preferred Reporting Items for Systematic Review and Meta-analyzes Extension for Scoping Reviews (PRISMA-ScR) framework. All original articles examining the use of intraoperative AR during surgical management of cancer were included. Exclusion criteria included virtual reality applications only, preoperative use only, fluorescence, AR not specific to surgical oncology, and study design (reviews, commentaries, abstracts).

RESULTS

A total of 2735 articles were identified of which 83 were included. Most studies (52) were performed on animals or phantom models, while the remaining included patients. A total of 1112 intraoperative AR surgical cases were performed across the studies. The most common anatomic site was brain (20 articles), followed by liver (16), renal (9), and head and neck (8). AR was most often used for intraoperative navigation or anatomic visualization of tumors or critical structures but was also used to identify osteotomy or craniotomy planes.

CONCLUSIONS

AR technology has been applied across the field of surgical oncology to aid in localization and resection of tumors.

Enhancing Surgical Vision: Augmented Reality in Otolaryngology-Head and Neck Surgery

Augmented reality (AR) technology has become widely established in otolaryngology-head and neck surgery. Over the past 20 years, numerous AR systems have been investigated and validated across the subspecialties, both in cadaveric and in live surgical studies. AR displays projected through head-mounted devices, microscopes, and endoscopes, most commonly, have demonstrated utility in preoperative planning, intraoperative guidance, and improvement of surgical decision-making. Specifically, they have demonstrated feasibility in guiding tumor margin resections, identifying critical structures intraoperatively, and displaying patient-specific virtual models derived from preoperative imaging, with millimetric accuracy. This review summarizes both established and emerging AR technologies, detailing how their systems work, what features they offer, and their clinical impact across otolaryngology subspecialties. As AR technology continues to advance, its integration holds promise for enhancing surgical precision, simulation training, and ultimately, improving patient outcomes.

Evaluation Metrics for Augmented Reality in Neurosurgical Preoperative Planning, Surgical Navigation, and Surgical Treatment Guidance: A Systematic Review

BACKGROUND AND OBJECTIVE

Recent years have shown an advancement in the development of augmented reality (AR) technologies for preoperative visualization, surgical navigation, and intraoperative guidance for neurosurgery. However, proving added value for AR in clinical practice is challenging, partly because of a lack of standardized evaluation metrics. We performed a systematic review to provide an overview of the reported evaluation metrics for AR technologies in neurosurgical practice and to establish a foundation for assessment and comparison of such technologies.

METHODS

PubMed, Embase, and Cochrane were searched systematically for publications on assessment of AR for cranial neurosurgery on September 22, 2022. The findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

RESULTS

The systematic search yielded 830 publications; 114 were screened full text, and 80 were included for analysis. Among the included studies, 5% dealt with preoperative visualization using AR, with user perception as the most frequently reported metric. The majority (75%) researched AR technology for surgical navigation, with registration accuracy, clinical outcome, and time measurements as the most frequently reported metrics. In addition, 20% studied the use of AR for intraoperative guidance, with registration accuracy, task outcome, and user perception as the most frequently reported metrics.

CONCLUSION

For quality benchmarking of AR technologies in neurosurgery, evaluation metrics should be specific to the risk profile and clinical objectives of the technology. A key focus should be on using validated questionnaires to assess user perception; ensuring clear and unambiguous reporting of registration accuracy, precision, robustness, and system stability; and accurately measuring task performance in clinical studies. We provided an overview suggesting which evaluation metrics to use per AR application and innovation phase, aiming to improve the assessment of added value of AR for neurosurgical practice and to facilitate the integration in the clinical workflow.

Augmented reality simulation as training model of ventricular puncture: Evidence in the improvement of the quality of punctures

BACKGROUND

Ventricular puncture is a common procedure in neurosurgery and the first that resident must learn. Ongoing education is critical to improving patient outcomes. However, training at the expense of potential risk to patients warrants new and safer training methods for residents.

METHODS

An augmented reality (AR) simulator for the practice of ventricular punctures was designed. It consists of a navigation system with a virtual 3D projection of the anatomy over a 3D-printed patient model. Forty-eight participants from neurosurgery staff performed two free-hand ventricular punctures before and after a training session.

RESULTS

Participants achieved enhanced accuracy in reaching the target at the Monro foramen after practicing with the system. Additional metrics revealed significantly better trajectories after the training.

CONCLUSION

The study confirms the feasibility of AR as a training tool. This motivates future work towards standardising new educative methodologies in neurosurgery.

Alterations in Physical Demands During Virtual/Augmented Reality-Based Tasks: A Systematic Review

The digital world has recently experienced a swift rise in worldwide popularity due to Virtual (VR) and Augmented Reality (AR) devices. However, concrete evidence about the effects of VR/AR devices on the physical workload imposed on the human body is lacking. We reviewed 27 articles that evaluated the physical impact of VR/AR-based tasks on the users using biomechanical sensing equipment and subjective tools. Findings revealed that movement and muscle demands (neck and shoulder) varied in seven and five studies while using VR, while in four and three studies during AR use, respectively, compared to traditional methods. User discomfort was also found in seven VR and three AR studies. Outcomes indicate that interface and interaction design, precisely target locations (gestures, viewing), design of virtual elements, and device type (location of CG as in Head-Mounted Displays) influence these alterations in neck and shoulder regions. Recommendations based on the review include developing comfortable reach envelopes for gestures, improving wearability, and studying temporal effects of repetitive movements (such as effects on fatigue and stability). Finally, a guideline is provided to assist researchers in conducting effective evaluations. The presented findings from this review could benefit designers/evaluations working towards developing more effective VR/AR products.

Neuro-oncological augmented reality planning for intracranial tumor resection

Background

Before starting surgery for the resection of an intracranial tumor, its outlines are typically marked on the skin of the patient. This allows for the planning of the optimal skin incision, craniotomy, and angle of approach. Conventionally, the surgeon determines tumor borders using neuronavigation with a tracked pointer. However, interpretation errors can lead to important deviations, especially for deep-seated tumors, potentially resulting in a suboptimal approach with incomplete exposure. Augmented reality (AR) allows displaying of the tumor and critical structures directly on the patient, which can simplify and improve surgical preparation.

Methods

We developed an AR-based workflow for intracranial tumor resection planning deployed on the Microsoft HoloLens II, which exploits the built-in infrared-camera for tracking the patient. We initially performed a phantom study to assess the accuracy of the registration and tracking. Following this, we evaluated the AR-based planning step in a prospective clinical study for patients undergoing resection of a brain tumor. This planning step was performed by 12 surgeons and trainees with varying degrees of experience. After patient registration, tumor outlines were marked on the patient's skin by different investigators, consecutively using a conventional neuronavigation system and an AR-based system. Their performance in both registration and delineation was measured in terms of accuracy and duration and compared.

Results

During phantom testing, registration errors remained below 2.0 mm and 2.0° for both AR-based navigation and conventional neuronavigation, with no significant difference between both systems. In the prospective clinical trial, 20 patients underwent tumor resection planning. Registration accuracy was independent of user experience for both AR-based navigation and the commercial neuronavigation system. AR-guided tumor delineation was deemed superior in 65% of cases, equally good in 30% of cases, and inferior in 5% of cases when compared to the conventional navigation system. The overall planning time (AR = 119 ± 44 s, conventional = 187 ± 56 s) was significantly reduced through the adoption of the AR workflow (p < 0.001), with an average time reduction of 39%.

Conclusion

By providing a more intuitive visualization of relevant data to the surgeon, AR navigation provides an accurate method for tumor resection planning that is quicker and more intuitive than conventional neuronavigation. Further research should focus on intraoperative implementations.

Effect of binocular disparity on learning anatomy with stereoscopic augmented reality visualization: A double center randomized controlled trial

Binocular disparity provides one of the important depth cues within stereoscopic three-dimensional (3D) visualization technology. However, there is limited research on its effect on learning within a 3D augmented reality (AR) environment. This study evaluated the effect of binocular disparity on the acquisition of anatomical knowledge and perceived cognitive load in relation to visual-spatial abilities. In a double-center randomized controlled trial, first-year (bio)medical undergraduates studied lower extremity anatomy in an interactive 3D AR environment either with a stereoscopic 3D view (n = 32) or monoscopic 3D view (n = 34). Visual-spatial abilities were tested with a mental rotation test. Anatomical knowledge was assessed by a validated 30-item written test and 30-item specimen test. Cognitive load was measured by the NASA-TLX questionnaire. Students in the stereoscopic 3D and monoscopic 3D groups performed equally well in terms of percentage correct answers (written test: 47.9 ± 15.8 vs. 49.1 ± 18.3; P = 0.635; specimen test: 43.0 ± 17.9 vs. 46.3 ± 15.1; P = 0.429), and perceived cognitive load scores (6.2 ± 1.0 vs. 6.2 ± 1.3; P = 0.992). Regardless of intervention, visual-spatial abilities were positively associated with the specimen test scores (η² = 0.13, P = 0.003), perceived representativeness of the anatomy test questions (P = 0.010) and subjective improvement in anatomy knowledge (P < 0.001). In conclusion, binocular disparity does not improve learning anatomy. Motion parallax should be considered as another important depth cue that contributes to depth perception during learning in a stereoscopic 3D AR environment.

Visualization, navigation, augmentation. The ever-changing perspective of the neurosurgeon

Introduction

The evolution of neurosurgery coincides with the evolution of visualization and navigation. Augmented reality technologies, with their ability to bring digital information into the real environment, have the potential to provide a new, revolutionary perspective to the neurosurgeon.

Research question

To provide an overview on the historical and technical aspects of visualization and navigation in neurosurgery, and to provide a systematic review on augmented reality (AR) applications in neurosurgery.

Material and methods

We provided an overview on the main historical milestones and technical features of visualization and navigation tools in neurosurgery. We systematically searched PubMed and Scopus databases for AR applications in neurosurgery and specifically discussed their relationship with current visualization and navigation systems, as well as main limitations.

Results

The evolution of visualization in neurosurgery is embodied by four magnification systems: surgical loupes, endoscope, surgical microscope and more recently the exoscope, each presenting independent features in terms of magnification capabilities, eye-hand coordination and the possibility to implement additional functions. In regard to navigation, two independent systems have been developed: the frame-based and the frame-less systems. The most frequent application setting for AR is brain surgery (71.6%), specifically neuro-oncology (36.2%) and microscope-based (29.2%), even though in the majority of cases AR applications presented their own visualization supports (66%).

Discussion and conclusions

The evolution of visualization and navigation in neurosurgery allowed for the development of more precise instruments; the development and clinical validation of AR applications, have the potential to be the next breakthrough, making surgeries safer, as well as improving surgical experience and reducing costs.

Extended Reality in Neurosurgical Education: A Systematic Review

Surgical simulation practices have witnessed a rapid expansion as an invaluable approach to resident training in recent years. One emerging way of implementing simulation is the adoption of extended reality (XR) technologies, which enable trainees to hone their skills by allowing interaction with virtual 3D objects placed in either real-world imagery or virtual environments. The goal of the present systematic review is to survey and broach the topic of XR in neurosurgery, with a focus on education. Five databases were investigated, leading to the inclusion of 31 studies after a thorough reviewing process. Focusing on user performance (UP) and user experience (UX), the body of evidence provided by these 31 studies showed that this technology has, in fact, the potential of enhancing neurosurgical education through the use of a wide array of both objective and subjective metrics. Recent research on the topic has so far produced solid results, particularly showing improvements in young residents, compared to other groups and over time. In conclusion, this review not only aids to a better understanding of the use of XR in neurosurgical education, but also highlights the areas where further research is entailed while also providing valuable insight into future applications.

Exploitation of Emerging Technologies and Advanced Networks for a Smart Healthcare System

Current medical methods still confront numerous limitations and barriers to detect and fight against illnesses and disorders. The introduction of emerging technologies in the healthcare industry is anticipated to enable novel medical techniques for an efficient and effective smart healthcare system. Internet of Things (IoT), Wireless Sensor Networks (WSN), Big Data Analytics (BDA), and Cloud Computing (CC) can play a vital role in the instant detection of illnesses, diseases, viruses, or disorders. Complicated techniques such as Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL) could provide acceleration in drug and antibiotics discovery. Moreover, the integration of visualization techniques such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) with Tactile Internet (TI), can be applied from the medical staff to provide the most accurate diagnosis and treatment for the patients. A novel system architecture, which combines several future technologies, is proposed in this paper. The objective is to describe the integration of a mixture of emerging technologies in assistance with advanced networks to provide a smart healthcare system that may be established in hospitals or medical centers. Such a system will be able to deliver immediate and accurate data to the medical stuff in order to aim them in order to provide precise patient diagnosis and treatment.

When medical trainees encountering a performance difficulty: evidence from pupillary responses

BACKGROUND

Medical trainees are required to learn many procedures following instructions to improve their skills. This study aims to investigate the pupillary response of trainees when they encounter moment of performance difficulty (MPD) during skill learning. Detecting the moment of performance difficulty is essential for educators to assist trainees when they need it.

METHODS

Eye motions were recorded while trainees practiced the thoracostomy procedure in the simulation model. To make pupillary data comparable among trainees, we proposed the adjusted pupil size (APS) normalizing pupil dilation for each trainee in their entire procedure. APS variables including APS, maxAPS, minAPS, meanAPS, medianAPS, and max interval indices were compared between easy and difficult subtasks; the APSs were compared among the three different performance situations, the moment of normal performance (MNP), MPD, and moment of seeking help (MSH).

RESULTS

The mixed ANOVA revealed that the adjusted pupil size variables, such as the maxAPS, the minAPS, the meanAPS, and the medianAPS, had significant differences between performance situations. Compared to MPD and MNP, pupil size was reduced during MSH. Trainees displayed a smaller accumulative frequency of APS during difficult subtask when compared to easy subtasks.

CONCLUSIONS

Results from this project suggest that pupil responses can be a good behavioral indicator. This study is a part of our research aiming to create an artificial intelligent system for medical trainees with automatic detection of their performance difficulty and delivering instructional messages using augmented reality technology.

In Situ Visualization for 3D Ultrasound-Guided Interventions with Augmented Reality Headset

Augmented Reality (AR) headsets have become the most ergonomic and efficient visualization devices to support complex manual tasks performed under direct vision. Their ability to provide hands-free interaction with the augmented scene makes them perfect for manual procedures such as surgery. This study demonstrates the reliability of an AR head-mounted display (HMD), conceived for surgical guidance, in navigating in-depth high-precision manual tasks guided by a 3D ultrasound imaging system. The integration between the AR visualization system and the ultrasound imaging system provides the surgeon with real-time intra-operative information on unexposed soft tissues that are spatially registered with the surrounding anatomic structures. The efficacy of the AR guiding system was quantitatively assessed with an in vitro study simulating a biopsy intervention aimed at determining the level of accuracy achievable. In the experiments, 10 subjects were asked to perform the biopsy on four spherical lesions of decreasing sizes (10, 7, 5, and 3 mm). The experimental results showed that 80% of the subjects were able to successfully perform the biopsy on the 5 mm lesion, with a 2.5 mm system accuracy. The results confirmed that the proposed integrated system can be used for navigation during in-depth high-precision manual tasks.

Augmented Reality-Assisted Craniotomy for Parasagittal and Convexity En Plaque Meningiomas and Custom-Made Cranio-Plasty: A Preliminary Laboratory Report

BACKGROUND

This report discusses the utility of a wearable augmented reality platform in neurosurgery for parasagittal and convexity en plaque meningiomas with bone flap removal and custom-made cranioplasty.

METHODS

A real patient with en plaque cranial vault meningioma with diffuse and extensive dural involvement, extracranial extension into the calvarium, and homogeneous contrast enhancement on gadolinium-enhanced T1-weighted MRI, was selected for this case study. A patient-specific manikin was designed starting with the segmentation of the patient's preoperative MRI images to simulate a craniotomy procedure. Surgical planning was performed according to the segmented anatomy, and customized bone flaps were designed accordingly. During the surgical simulation stage, the VOSTARS head-mounted display was used to accurately display the planned craniotomy trajectory over the manikin skull. The precision of the craniotomy was assessed based on the evaluation of previously prepared custom-made bone flaps.

RESULTS

A bone flap with a radius 0.5 mm smaller than the radius of an ideal craniotomy fitted perfectly over the performed craniotomy, demonstrating an error of less than ±1 mm in the task execution. The results of this laboratory-based experiment suggest that the proposed augmented reality platform helps in simulating convexity en plaque meningioma resection and custom-made cranioplasty, as carefully planned in the preoperative phase.

CONCLUSIONS

Augmented reality head-mounted displays have the potential to be a useful adjunct in tumor surgical resection, cranial vault lesion craniotomy and also skull base surgery, but more study with large series is needed.

A High-Fidelity Hybrid Virtual Reality Simulator of Aneurysm Clipping Repair With Brain Sylvian Fissure Exploration for Vascular Neurosurgery Training

INTRODUCTION

Microsurgery clipping is one of the most challenging surgical interventions in neurosurgery. The opportunities to train residents are scarce, but the need for accumulating practice is mandatory. New simulating tools are needed for skill learning.

METHODS

The design, implementation, and assessment of a new hybrid aneurysm clipping simulator are presented. It consists of an ergonomic workstation with a patient head mannequin and a physics-based virtual reality simulation with bimanual haptic feedback. The simulator recreates scenarios of microsurgery from the patient fixation and the exploration of the brain lobes through Sylvian fissure and vascular structures to the aneurysm clipping. Skill metrics were introduced, including monitoring of gestures movements, exerted forces, tissue displacements, and precision in clipping.

RESULTS

Two experimental conditions were tested: (1) simple clipping without brain tissue exploration and (2) clipping the aneurysm with brain Sylvian fissure exploration. Differences in the bimanual gestures were observed between both conditions. The quantitative measurements of tissue displacement of the brain lobes exhibited more tissue retrieval for the surgical gestures of neurosurgeons. Appraisal with questionnaires showed positive scores by neurosurgeons in all items evaluating the usability and realism of the simulator.

CONCLUSIONS

The simulator was well accepted and feasible for training purposes. The analysis of the interactions with virtual tissues offers information to establish differential and common patterns between tested groups and thus useful metrics for skill evaluation of practitioners. Future work can lead to other tasks during the intervention and the inclusion of more clinical cases.

Holographic mixed-reality neuronavigation with a head-mounted device: technical feasibility and clinical application

OBJECTIVE

The authors aimed to evaluate the technical feasibility of a mixed-reality neuronavigation (MRN) system with a wearable head-mounted device (HMD) and to determine its clinical application and accuracy.

METHODS

A semiautomatic registration MRN system on HoloLens smart glasses was developed and tested for accuracy and feasibility. Thirty-seven patients with intracranial lesions were prospectively identified. For each patient, multimodal imaging-based holograms of lesions, markers, and surrounding eloquent structures were created and then imported to the MRN HMD. After a point-based registration, the holograms were projected onto the patient's head and observed through the HMD. The contour of the holograms was compared with standard neuronavigation (SN). The projection of the lesion boundaries perceived by the neurosurgeon on the patient's scalp was then marked with MRN and SN. The distance between the two contours generated by MRN and SN was measured so that the accuracy of MRN could be assessed.

RESULTS

MRN localization was achieved in all patients. The mean additional time required for MRN was 36.3 ± 6.3 minutes, in which the mean registration time was 2.6 ± 0.9 minutes. A trend toward a shorter time required for preparation was observed with the increase of neurosurgeon experience with the MRN system. The overall median deviation was 4.1 mm (IQR 3.0 mm-4.7 mm), and 81.1% of the lesions localized by MRN were found to be highly consistent with SN (deviation < 5.0 mm). There was a significant difference between the supine position and the prone position (3.7 ± 1.1 mm vs 5.4 ± 0.9 mm, p = 0.001). The magnitudes of deviation vectors did not correlate with lesion volume (p = 0.126) or depth (p = 0.128). There was no significant difference in additional operating time between different operators (37.4 ± 4.8 minutes vs 34.6 ± 4.8 minutes, p = 0.237) or in localization deviation (3.7 ± 1.0 mm vs 4.6 ± 1.5 mm, p = 0.070).

CONCLUSIONS

This study provided a complete set of a clinically applicable workflow on an easy-to-use MRN system using a wearable HMD, and has shown its technical feasibility and accuracy. Further development is required to improve the accuracy and clinical efficacy of this system.

Augmented reality navigation method for recontouring surgery of craniofacial fibrous dysplasia

The objective of this study is to introduce the application of augmented reality (AR) navigation system developed by the authors in recontouring surgery of craniofacial fibrous dysplasia. Five consecutive patients with craniofacial fibrous dysplasia were enrolled. Through three-dimensional (3D) simulation, a virtual plan was designed to reconstruct the normal anatomical contour of the deformed region. Surgical recontouring was achieved with the assistance of the AR navigation system. The accuracy of the surgical procedure was assessed by superimposing the post-operative 3D craniomaxillofacial model onto the virtual plan. The pre-operative preparation time and operation time were also counted. In all patients, AR navigation was performed successfully, with a mean ± SD of the errors of 1.442 ± 0.234 mm. The operative time of the patients ranged from 60 to 80 min. The pre-operative preparation time was 20 min for each patient. All the patients showed uneventful healing without any complications, in addition to satisfaction with the post-operative aesthetics. Using our AR navigation system in recontouring surgery can provide surgeons with a comprehensive and intuitive view of the recontouring border, as well as the depth, in real time. This method could improve the efficiency and safety of craniofacial fibrous dysplasia recontouring procedures.

Use of augmented reality navigation to optimise the surgical management of craniofacial fibrous dysplasia

The aim of this study was to apply an augmented reality (AR) navigation technique based on a head- mounted display in the treatment of craniofacial fibrous dysplasia and to explore the feasibility and the value of AR in craniofacial surgery. With preoperative planning and three-dimensional simulation, the normal anatomical contours of the deformed area were recreated by superimposing the unaffected side on to the affected side. We completed the recontouring procedures in real time with the aid of an AR navigation system. The surgical outcome was assessed by superimposing the postoperative computed tomographic images on to the preoperative virtual plan. The preparation and operation times were recorded. With intraoperative AR guidance, facial bone recontouring was performed uneventfully in all cases. The mean (SD) discrepancy between the actual surgical reduction and preoperative planning was 1.036 (0.081) mm (range: 0.913 (0.496) to 1.165 (0.498) mm). The operation time ranged from 50 to 80 minutes, with an average of 66.4 minutes. The preoperative preparation time ranged from 26 to 36 minutes, with a mean of 29.6 minutes. AR navigation-assisted facial bone recontouring is a valuable treatment modality in managing craniomaxillofacial fibrous dysplasia and shows benefits in improving the efficiency and safety of this complicated procedure.

The Wearable VOSTARS System for Augmented Reality-Guided Surgery: Preclinical Phantom Evaluation for High-Precision Maxillofacial Tasks

BACKGROUND

In the context of guided surgery, augmented reality (AR) represents a groundbreaking improvement. The Video and Optical See-Through Augmented Reality Surgical System (VOSTARS) is a new AR wearable head-mounted display (HMD), recently developed as an advanced navigation tool for maxillofacial and plastic surgery and other non-endoscopic surgeries. In this study, we report results of phantom tests with VOSTARS aimed to evaluate its feasibility and accuracy in performing maxillofacial surgical tasks.

METHODS

An early prototype of VOSTARS was used. Le Fort 1 osteotomy was selected as the experimental task to be performed under VOSTARS guidance. A dedicated set-up was prepared, including the design of a maxillofacial phantom, an ad hoc tracker anchored to the occlusal splint, and cutting templates for accuracy assessment. Both qualitative and quantitative assessments were carried out.

RESULTS

VOSTARS, used in combination with the designed maxilla tracker, showed excellent tracking robustness under operating room lighting. Accuracy tests showed that 100% of Le Fort 1 trajectories were traced with an accuracy of ±1.0 mm, and on average, 88% of the trajectory's length was within ±0.5 mm accuracy.

CONCLUSIONS

Our preliminary results suggest that the VOSTARS system can be a feasible and accurate solution for guiding maxillofacial surgical tasks, paving the way to its validation in clinical trials and for a wide spectrum of maxillofacial applications.

Stereotactic Neuro-Navigation Phantom Designs: A Systematic Review

Diverse stereotactic neuro-navigation systems are used daily in neurosurgery and novel systems are continuously being developed. Prior to clinical implementation of new surgical tools, methods or instruments, in vitro experiments on phantoms should be conducted. A stereotactic neuro-navigation phantom denotes a rigid or deformable structure resembling the cranium with the intracranial area. The use of phantoms is essential for the testing of complete procedures and their workflows, as well as for the final validation of the application accuracy. The aim of this study is to provide a systematic review of stereotactic neuro-navigation phantom designs, to identify their most relevant features, and to identify methodologies for measuring the target point error, the entry point error, and the angular error (α). The literature on phantom designs used for evaluating the accuracy of stereotactic neuro-navigation systems, i.e., robotic navigation systems, stereotactic frames, frameless navigation systems, and aiming devices, was searched. Eligible articles among the articles written in English in the period 2000-2020 were identified through the electronic databases PubMed, IEEE, Web of Science, and Scopus. The majority of phantom designs presented in those articles provide a suitable methodology for measuring the target point error, while there is a lack of objective measurements of the entry point error and angular error. We identified the need for a universal phantom design, which would be compatible with most common imaging techniques (e.g., computed tomography and magnetic resonance imaging) and suitable for simultaneous measurement of the target point, entry point, and angular errors.

Augmented reality in neurosurgical navigation: a survey

BACKGROUND

Neurosurgery has exceptionally high requirements for minimally invasive and safety. This survey attempts to analyze the practical application of AR in neurosurgical navigation. Also, this survey describes future trends in augmented reality neurosurgical navigation systems.

METHODS

In this survey, we searched related keywords "augmented reality", "virtual reality", "neurosurgery", "surgical simulation", "brain tumor surgery", "neurovascular surgery", "temporal bone surgery", and "spinal surgery" through Google Scholar, World Neurosurgery, PubMed and Science Direct. We collected 85 articles published over the past five years in areas related to this survey.

RESULTS

Detailed study has been conducted on the application of AR in neurosurgery and found that AR is constantly improving the overall efficiency of doctor training and treatment, which can help neurosurgeons learn and practice surgical procedures with zero risks.

CONCLUSIONS

Neurosurgical navigation is essential in neurosurgery. Despite certain technical limitations, it is still a necessary tool for the pursuit of maximum security and minimal intrusiveness. This article is protected by copyright. All rights reserved.

Surgical Strategy of Endoscopically assisted Periradicular Surgery Using Novel Head-mounted Display System

A novel head-mounted display offers high quality of endoscopic imagery in front of the eyes, irrespective of the head position. We present an application of the head-mounted display system as a personal integrated multi-image monitoring system in endoscopically assisted periradicular surgery. Our head-mounted display system displayed multiple forms of information as integrated, sharp, high-definition endoscope, biological monitor, and X-ray image (such as panoramic and computed tomography images) synchronously using a picture in picture. In addition, this system can cope with both the endoscopic field of view and the direct field of view. While monitoring the patient's general condition with a head-mounted display, the surgery was performed with endoscopic animation. We could also switch to the direct surgical field and endoscopic field of view smoothly without moving the head and without surgical interference. The availability of a head-mounted display system during endoscopically assisted periradicular surgery enabled the provision of a comfortable and appropriate surgical environment for the surgeon.

Mythri 1.0—Progress of an Indian Surgical Robot

Neurosurgical procedures are performed using operating microscopes. The technology of most microscopes has not changed much over the past 60 years. The National Institute of Mental Health and Neurosciences and International Institute of Information Technology based at Bengaluru have embarked on joint collaboration for developing robot for neurosurgical applications. As a working prototype, robotic microscope Mythri 1.0 has been developed. An overview of the development process, working, and features of the device is presented in the article.

Augmented reality guided osteotomy in hallux Valgus correction

Background

An optimal osteotomy angle avoids shortening of the first metatarsal bone after hallux valgus surgery and therefore reduces the risk of transfer-metatarsalgia. The purpose of the present ex-vivo study was to investigate whether augmented reality (AR) would improve accuracy of the distal osteotomy during hallux valgus surgery.

Methods

Distal osteotomies of the first metatarsals were performed on a foot model by two surgeons with different levels of surgical experience each with (AR, n = 15 × 2) or without (controls, n = 15 × 2) overlay of a hologram depicting an angle of osteotomy perpendicular to the second metatarsal. Subsequently, the deviation of the osteotomy angle in the transverse plane was analyzed.

Results

Overall, AR decreased the extent of deviation and the AR guided osteotomies were more accurate (4.9 ± 4.2°) compared to the freehand cuts (6.7 ± 6.1°) by tendency (p = 0.2). However, while the inexperienced surgeon performed more accurate osteotomies with AR with a mean angle of 6.4 ± 3.5° compared to freehand 10.5 ± 5.5° (p = 0.02), no significant difference was noticed for the experienced surgeon with an osteotomy angle of around 3° in both cases.

Conclusion

This pilot-study suggests that AR guided osteotomies can potentially improve accuracy during hallux valgus correction, particularly for less experienced surgeons.

Augmented Reality in Superficial Temporal Artery to Middle Cerebral Artery Bypass Surgery: Technical Note

BACKGROUND

Augmented reality (AR) applied to surgery refers to the virtual superimposition of computer-generated anatomical information on the surgical field. AR assistance in extracranial-intracranial (EC-IC) bypass revascularization surgery has been reported to be a helpful technical adjunct.

OBJECTIVE

To describe our experience of using AR in superficial temporal artery to middle cerebral artery (STA-MCA) bypass surgery with the additional implementation of new technical processes to improve the safety and efficacy of the procedure.

METHODS

Data sets from preoperative imaging were loaded and fused in a single 3-dimensional matrix using the neuronavigation system. Anatomical structures of interest (the STA, a selected M4 branch of the MCA, the middle meningeal artery [MMA], and the primary motor cortex [PMC]) were segmented. After the registration of the patient and the operating microscope, the structures of interest were projected into the eyepiece of the microscope and superimposed onto the patient's head, creating the AR surgical field.

RESULTS

AR was shown to be useful in patients undergoing EC-IC bypass revascularization, mostly during the following 4 surgical steps: (1) microsurgical dissection of the donor vessel (STA); (2) tailoring the craniotomy above the recipient vessel (M4 branch of the MCA); (3) tailoring the craniotomy to spare the MMA; and (4) tailoring the craniotomy and the anastomosis to spare the PMC.

CONCLUSION

AR assistance in EC-IC bypass revascularization is a versatile technical adjunct for helping surgeons to ensure the safety and efficacy of the procedure.

Wearable Augmented Reality Platform for Aiding Complex 3D Trajectory Tracing

Augmented reality (AR) Head-Mounted Displays (HMDs) are emerging as the most efficient output medium to support manual tasks performed under direct vision. Despite that, technological and human-factor limitations still hinder their routine use for aiding high-precision manual tasks in the peripersonal space. To overcome such limitations, in this work, we show the results of a user study aimed to validate qualitatively and quantitatively a recently developed AR platform specifically conceived for guiding complex 3D trajectory tracing tasks. The AR platform comprises a new-concept AR video see-through (VST) HMD and a dedicated software framework for the effective deployment of the AR application. In the experiments, the subjects were asked to perform 3D trajectory tracing tasks on 3D-printed replica of planar structures or more elaborated bony anatomies. The accuracy of the trajectories traced by the subjects was evaluated by using templates designed ad hoc to match the surface of the phantoms. The quantitative results suggest that the AR platform could be used to guide high-precision tasks: on average more than 94% of the traced trajectories stayed within an error margin lower than 1 mm. The results confirm that the proposed AR platform will boost the profitable adoption of AR HMDs to guide high precision manual tasks in the peripersonal space.

Ambiguity-Free Optical-Inertial Tracking for Augmented Reality Headsets

The increasing capability of computing power and mobile graphics has made possible the release of self-contained augmented reality (AR) headsets featuring efficient head-anchored tracking solutions. Ego motion estimation based on well-established infrared tracking of markers ensures sufficient accuracy and robustness. Unfortunately, wearable visible-light stereo cameras with short baseline and operating under uncontrolled lighting conditions suffer from tracking failures and ambiguities in pose estimation. To improve the accuracy of optical self-tracking and its resiliency to marker occlusions, degraded camera calibrations, and inconsistent lighting, in this work we propose a sensor fusion approach based on Kalman filtering that integrates optical tracking data with inertial tracking data when computing motion correlation. In order to measure improvements in AR overlay accuracy, experiments are performed with a custom-made AR headset designed for supporting complex manual tasks performed under direct vision. Experimental results show that the proposed solution improves the head-mounted display (HMD) tracking accuracy by one third and improves the robustness by also capturing the orientation of the target scene when some of the markers are occluded and when the optical tracking yields unstable and/or ambiguous results due to the limitations of using head-anchored stereo tracking cameras under uncontrollable lighting conditions.

Simultaneous Optimization of Patient-Image Registration and Hand-Eye Calibration for Accurate Augmented Reality in Surgery

OBJECTIVE

Augmented reality (AR) navigation using a position sensor in endoscopic surgeries relies on the quality of patient-image registration and hand-eye calibration. Conventional methods collect the necessary data to compute two output transformation matrices separately. However, the AR display setting during surgery generally differs from that during preoperative processes. Although conventional methods can identify optimal solutions under initial conditions, AR display errors are unavoidable during surgery owing to the inherent computational complexity of AR processes, such as error accumulation over successive matrix multiplications, and tracking errors of position sensor.

METHODS

We propose the simultaneous optimization of patient-image registration and hand-eye calibration in an AR environment before surgery. The relationship between the endoscope and a virtual object to overlay is first calculated using an endoscopic image, which also functions as a reference during optimization. After including the tracking information from the position sensor, patient-image registration and hand-eye calibration are optimized in terms of least-squares.

RESULTS

Experiments with synthetic data verify that the proposed method is less sensitive to computation and tracking errors. A phantom experiment with a position sensor is also conducted. The accuracy of the proposed method is significantly higher than that of the conventional method.

CONCLUSION

The AR accuracy of the proposed method is compared with those of the conventional ones, and the superiority of the proposed method is verified.

SIGNIFICANCE

This study demonstrates that the proposed method exhibits substantial potential for improving AR navigation accuracy.

Combination of CAD/CAM and Augmented Reality in Free Fibula Bone Harvest

Supplemental Digital Content is available in the text.

The CAD/CAM technology for mandibular reconstruction has improved the results in terms of outcomes in restoring mandibular complex defects. Augmented reality (AR) represents an evolution of the navigation-assisted surgery. This technology merges the images of the virtual planning with the anatomy of the patient, representing in this way an enhanced scene for the surgeon’s eye. AR can also display in a single scene additional information for the surgeon. Despite of classical navigation, this scenario can be obtained with marker-less registration method, without using reference points or fiducial markers. This technologic evolution together with the large use in our experience of CAD/CAM protocol for mandibular reconstruction we developed this feasibility study to evaluate the possibility of using a marker-less image registration system. Moreover, we tried to evaluate the overlaying of the virtual planning and its reproducibility using AR. We performed a case series of 3 consecutive patients who underwent mandibular reconstruction using AR-assisted fibular free flap harvesting applying our digital workflow. Once launched, the mobile app installed on our tablet, the registration is performed according to a shape recognition system of the leg of the patient, rendering in real time a superimposition of the anatomy of the bony, vascular, and skin of the patient and also the surgical planning of the reconstruction. AR-assisted fibular free flap harvesting was performed. We believe that AR can be a prospective improving technology for mandibular complex reconstruction.

CIGuide: in situ augmented reality laser guidance

PURPOSE : A robotic intraoperative laser guidance system with hybrid optic-magnetic tracking for skull base surgery is presented. It provides in situ augmented reality guidance for microscopic interventions at the lateral skull base with minimal mental and workload overhead on surgeons working without a monitor and dedicated pointing tools. METHODS : Three components were developed: a registration tool (Rhinospider), a hybrid magneto-optic-tracked robotic feedback control scheme and a modified robotic end-effector. Rhinospider optimizes registration of patient and preoperative CT data by excluding user errors in fiducial localization with magnetic tracking. The hybrid controller uses an integrated microscope HD camera for robotic control with a guidance beam shining on a dual plate setup avoiding magnetic field distortions. A robotic needle insertion platform (iSYS Medizintechnik GmbH, Austria) was modified to position a laser beam with high precision in a surgical scene compatible to microscopic surgery. RESULTS : System accuracy was evaluated quantitatively at various target positions on a phantom. The accuracy found is 1.2 mm ± 0.5 mm. Errors are primarily due to magnetic tracking. This application accuracy seems suitable for most surgical procedures in the lateral skull base. The system was evaluated quantitatively during a mastoidectomy of an anatomic head specimen and was judged useful by the surgeon. CONCLUSION : A hybrid robotic laser guidance system with direct visual feedback is proposed for navigated drilling and intraoperative structure localization. The system provides visual cues directly on/in the patient anatomy, reducing the standard limitations of AR visualizations like depth perception. The custom- built end-effector for the iSYS robot is transparent to using surgical microscopes and compatible with magnetic tracking. The cadaver experiment showed that guidance was accurate and that the end-effector is unobtrusive. This laser guidance has potential to aid the surgeon in finding the optimal mastoidectomy trajectory in more difficult interventions.

The use of video glasses improved learning of tonsillectomy and adenoidectomy surgery: A randomized controlled trial

OBJECTIVE

One of the most common challenges in surgical education for trainees is gaining practical experience through observing procedures in the operating room. Due to the nature of some procedures, a narrow surgical view severely limits the learning experience. Video glasses are new devices that offer the potential to project the primary surgeon's exact view to learners in real-time, allowing for an enhanced operative learning experience.

STUDY DESIGN

Single center randomized prospective trial.

SETTING

Tertiary care pediatric hospital.

PARTICIPANTS

Using block randomization, medical students and surgical residents observed either a tonsillectomy or adenoidectomy, either directly at table-side or by real-time video feed from the surgeon's video glasses projected to a screen in the operating room, in random order. Participants then completed a survey comparing aspects of their learning experience viewing the procedure through the video feed in comparison to direct observation.

MAIN OUTCOME MEASURES

Evaluating the hypothesis that video glasses provided an improved overall learning experience and a realistic simulation of the open surgical procedures tested.

RESULTS

23 trainees participated in the study. Survey results demonstrated that the overall learning experience with the use of video glasses was significantly improved when compared to direct visualization (average Visual Analog Scale (VAS) score 82/100 vs. 64/100, p = 0.021). Video glasses were shown to be superior when comparing the view of the surgical field (83/100 vs. 54/100 on VAS, p < 0.001) and the ability to identify anatomical structures (79/100 vs. 56/100 on VAS, p = 0.001). The ease of following surgical steps with video glasses was also shown to be better than by direct visualization (81/100 vs. 69/100 on VAS, p = 0.039). All participants stated that video glasses closely simulated the learning environment of the real-life open procedure.

CONCLUSION

This study showed that the use of video glasses was beneficial for surgical education and a realistic tool for learners at varying levels of training. Video glasses may significantly improve the learning experience for procedures with a narrow field of view.

Augmenting Microsoft's HoloLens with vuforia tracking for neuronavigation

Major hurdles for Microsoft's HoloLens as a tool in medicine have been accessing tracking data, as well as a relatively high-localisation error of the displayed information; cumulatively resulting in its limited use and minimal quantification. The following work investigates the augmentation of HoloLens with the proprietary image processing SDK Vuforia, allowing integration of data from its front-facing RGB camera to provide more spatially stable holograms for neuronavigational use. Continuous camera tracking was able to maintain hologram registration with a mean perceived drift of 1.41 mm, as well as a mean sub 2-mm surface point localisation accuracy of 53%, all while allowing the researcher to walk about a test area. This represents a 68% improvement for the later and a 34% improvement for the former compared with a typical HoloLens deployment used as a control. Both represent a significant improvement on hologram stability given the current state-of-the-art, and to the best of the authors knowledge are the first example of quantified measurements when augmenting hologram stability using data from the RGB sensor.

A low-cost multimodal head-mounted display system for neuroendoscopic surgery

BACKGROUND

With rapid advances in technology, wearable devices as head-mounted display (HMD) have been adopted for various uses in medical science, ranging from simply aiding in fitness to assisting surgery. We aimed to investigate the feasibility and practicability of a low-cost multimodal HMD system in neuroendoscopic surgery.

METHODS

A multimodal HMD system, mainly consisted of a HMD with two built-in displays, an action camera, and a laptop computer displaying reconstructed medical images, was developed to assist neuroendoscopic surgery. With this intensively integrated system, the neurosurgeon could freely switch between endoscopic image, three-dimensional (3D) reconstructed virtual endoscopy images, and surrounding environment images. Using a leap motion controller, the neurosurgeon could adjust or rotate the 3D virtual endoscopic images at a distance to better understand the positional relation between lesions and normal tissues at will.

RESULTS

A total of 21 consecutive patients with ventricular system diseases underwent neuroendoscopic surgery with the aid of this system. All operations were accomplished successfully, and no system-related complications occurred. The HMD was comfortable to wear and easy to operate. Screen resolution of the HMD was high enough for the neurosurgeon to operate carefully. With the system, the neurosurgeon might get a better comprehension on lesions by freely switching among images of different modalities. The system had a steep learning curve, which meant a quick increment of skill with it. Compared with commercially available surgical assistant instruments, this system was relatively low-cost.

CONCLUSIONS

The multimodal HMD system is feasible, practical, helpful, and relatively cost efficient in neuroendoscopic surgery.