Virtual reality and augmented reality in the management of intracranial tumors: A review

Pioneering Augmented and Mixed Reality in Cranial Surgery: The First Latin American Experience

INTRODUCTION

Augmented reality (AR) and mixed reality (MR) technologies have revolutionized cranial neurosurgery by overlaying digital information onto the surgical field, enhancing visualization, precision, and training. These technologies enable the real-time integration of preoperative imaging data, aiding in better decision-making and reducing operative risks. Despite challenges such as cost and specialized training needs, AR and MR offer significant benefits, including improved surgical outcomes and personalized surgical plans based on individual patient anatomy.

MATERIALS AND METHODS

This study describes three intracranial surgeries using AR and MR technologies at Hospital Ángeles Universidad, Mexico City, in 2023. Surgeries were performed with VisAR software 3 version and Microsoft HoloLens 2, transforming DICOM images into 3D models. Preoperative MRI and CT scans facilitated planning, and radiopaque tags ensured accurate image registration during surgery. Postoperative outcomes were assessed through clinical and imaging follow-up.

RESULTS

Three intracranial surgeries were performed with AR and MR assistance, resulting in successful outcomes with minimal postoperative complications. Case 1 achieved 80% tumor resection, Case 2 achieved near-total tumor resection, and Case 3 achieved complete lesion resection. All patients experienced significant symptom relief and favorable recoveries, demonstrating the precision and effectiveness of AR and MR in cranial surgery.

CONCLUSIONS

This study demonstrates the successful use of AR and MR in cranial surgery, enhancing precision and clinical outcomes. Despite challenges like training and costs, these technologies offer significant benefits. Future research should focus on long-term outcomes and broader applications to validate their efficacy and cost-effectiveness in neurosurgery.

Emergence of Precision Medicine Within Neurological Surgery: Promise and Opportunity

Within neurosurgery, it has always been important to individualize patient care. In recent years, however, technological advances have brought a new dimension to personalized care as developing methods, including next-generation sequencing, have enabled us to molecularly profile pathologies with increasing scale and resolution. In this review, the authors discuss the history and advances in precision medicine and neurosurgery, focusing both on neuro-oncology, as well as its extension to other neurosurgical subspecialties. They highlight the important roles of neurosurgeons in past work and future work, with the extension of tissue collection and precision medicine principles to additional sample types and disease indications.

Narrative review of patient-specific 3D visualization and reality technologies in skull base neurosurgery: enhancements in surgical training, planning, and navigation

Recent advances in medical imaging, computer vision, 3-dimensional (3D) modeling, and artificial intelligence (AI) integrated technologies paved the way for generating patient-specific, realistic 3D visualization of pathological anatomy in neurosurgical conditions. Immersive surgical simulations through augmented reality (AR), virtual reality (VR), mixed reality (MxR), extended reality (XR), and 3D printing applications further increased their utilization in current surgical practice and training. This narrative review investigates state-of-the-art studies, the limitations of these technologies, and future directions for them in the field of skull base surgery. We begin with a methodology summary to create accurate 3D models customized for each patient by combining several imaging modalities. Then, we explore how these models are employed in surgical planning simulations and real-time navigation systems in surgical procedures involving the anterior, middle, and posterior cranial skull bases, including endoscopic and open microsurgical operations. We also evaluate their influence on surgical decision-making, performance, and education. Accumulating evidence demonstrates that these technologies can enhance the visibility of the neuroanatomical structures situated at the cranial base and assist surgeons in preoperative planning and intraoperative navigation, thus showing great potential to improve surgical results and reduce complications. Maximum effectiveness can be achieved in approach selection, patient positioning, craniotomy placement, anti-target avoidance, and comprehension of spatial interrelationships of neurovascular structures. Finally, we present the obstacles and possible future paths for the broader implementation of these groundbreaking methods in neurosurgery, highlighting the importance of ongoing technological advancements and interdisciplinary collaboration to improve the accuracy and usefulness of 3D visualization and reality technologies in skull base surgeries.

The Integration of 3D Virtual Reality and 3D Printing Technology as Innovative Approaches to Preoperative Planning in Neuro-Oncology

Our study explores the integration of three-dimensional (3D) virtual reality (VR) and 3D printing in neurosurgical preoperative planning. Traditionally, surgeons relied on two-dimensional (2D) imaging for complex neuroanatomy analyses, requiring significant mental visualization. Fortunately, nowadays advanced technology enables the creation of detailed 3D models from patient scans, utilizing different software. Afterwards, these models can be experienced through VR systems, offering comprehensive preoperative rehearsal opportunities. Additionally, 3D models can be 3D printed for hands-on training, therefore enhancing surgical preparedness. This technological integration transforms the paradigm of neurosurgical planning, ensuring safer procedures.

Advances in artificial intelligence, robotics, augmented and virtual reality in neurosurgery

Neurosurgical practitioners undergo extensive and prolonged training to acquire diverse technical proficiencies, while neurosurgical procedures necessitate a substantial amount of pre-, post-, and intraoperative clinical data acquisition, making decisions, attention, and convalescence. The past decade witnessed an appreciable escalation in the significance of artificial intelligence (AI) in neurosurgery. AI holds significant potential in neurosurgery as it supplements the abilities of neurosurgeons to offer optimal interventional and non-interventional care to patients by improving prognostic and diagnostic outcomes in clinical therapy and assisting neurosurgeons in making decisions while surgical interventions to enhance patient outcomes. Other technologies including augmented reality, robotics, and virtual reality can assist and promote neurosurgical methods as well. Moreover, they play a significant role in generating, processing, as well as storing experimental and clinical data. Also, the usage of these technologies in neurosurgery is able to curtail the number of costs linked with surgical care and extend high-quality health care to a wider populace. This narrative review aims to integrate the results of articles that elucidate the role of the aforementioned technologies in neurosurgery.

Intra-operative applications of augmented reality in glioma surgery: a systematic review

Background

Augmented reality (AR) is increasingly being explored in neurosurgical practice. By visualizing patient-specific, three-dimensional (3D) models in real time, surgeons can improve their spatial understanding of complex anatomy and pathology, thereby optimizing intra-operative navigation, localization, and resection. Here, we aimed to capture applications of AR in glioma surgery, their current status and future potential.

Methods

A systematic review of the literature was conducted. This adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline. PubMed, Embase, and Scopus electronic databases were queried from inception to October 10, 2022. Leveraging the Population, Intervention, Comparison, Outcomes, and Study design (PICOS) framework, study eligibility was evaluated in the qualitative synthesis. Data regarding AR workflow, surgical application, and associated outcomes were then extracted. The quality of evidence was additionally examined, using hierarchical classes of evidence in neurosurgery.

Results

The search returned 77 articles. Forty were subject to title and abstract screening, while 25 proceeded to full text screening. Of these, 22 articles met eligibility criteria and were included in the final review. During abstraction, studies were classified as "development" or "intervention" based on primary aims. Overall, AR was qualitatively advantageous, due to enhanced visualization of gliomas and critical structures, frequently aiding in maximal safe resection. Non-rigid applications were also useful in disclosing and compensating for intra-operative brain shift. Irrespective, there was high variance in registration methods and measurements, which considerably impacted projection accuracy. Most studies were of low-level evidence, yielding heterogeneous results.

Conclusions

AR has increasing potential for glioma surgery, with capacity to positively influence the onco-functional balance. However, technical and design limitations are readily apparent. The field must consider the importance of consistency and replicability, as well as the level of evidence, to effectively converge on standard approaches that maximize patient benefit.

Views on Augmented Reality, Virtual Reality, and 3D Printing in Modern Medicine and Education: A Qualitative Exploration of Expert Opinion

Although an increased usage and development of 3D technologies is observed in healthcare over the last decades, full integration of these technologies remains challenging. The goal of this project is to qualitatively explore challenges, pearls, and pitfalls of AR/VR/3D printing applications usage in the medical field of a university medical center. Two rounds of face-to-face interviews were conducted using a semi-structured protocol. First an explorative round was held, interviewing medical specialists (8), PhD students (7), 3D technology specialists (5), and university teachers (3). In the second round, twenty employees in high executive functions of relevant departments were interviewed on seven statements that resulted from the first interviewing round. Data analysis was performed using direct content analyses. The first interviewing round resulted in challenges and opportunities in 3D technology usage that were grouped in 5 themes: aims of using AR/VR/3D printing (1), data acquisition (2), data management plans (3), software packages and segmentation tools (4), and output data and reaching end-user (5). The second interviewing round resulted in an overview of ideas and insights on centralization of knowledge, improving implementation of 3D technology in daily healthcare, reimbursement of 3D technologies, recommendations for further studies, and requirement of using certified software. An overview of challenges and opportunities of 3D technologies in healthcare was provided. Well-designed studies on clinical effectiveness, implementation and cost-effectiveness are warranted for further implementation into the clinical setting.

Augmented Reality Holographic Visualization System for Surgery Auxiliary Visualization: Proof of Concept for Surgical Training

An Augmented Reality (AR) system based on the holographic projection of the relevant anatomic structures is proposed for auxiliary visualization during surgeries. The current two-dimensional visualization systems require the surgeons to mentally extract the associated three-dimensional information during the interventions, which entails risks and complications. This work shows an AR holographic projection system for real-time three-dimensional representation of the relevant surgical information, thus overcoming this problem. As an initial proof of concept, the system is experimentally assessed as potential surgery training tool.Clinical Relevance- This work explores the potential of AR holographic projection systems for intraoperative assistance to the surgical team, starting from its possible use as surgery training and planning tool.

Surgeon's Eyes on the Relevant Surgical Target

The resolution of the naked eye has been a challenge for the neurosurgical endeavor since the very first attempts of cranial surgery, and advances have been achieved over the centuries, driven by a synergism between the application of emerging technology into the surgical environment and the expansion of the capabilities of neurosurgery. The understanding of the principles of the optical properties of lenses by Abbè (1840-1905) led to the introduction of loupes in the surgical practice, increasing the visual performance during macroscopic procedures. Modern neurosurgery began with the possibility of illumination and magnification of the surgical field as provided by the microscope. Pioneering contributions from Donaghy and Yasargil opened the way to the era of minimalism with reduction of operative corridors and surgical trauma through the adoption of the microsurgical technique. Almost at the same time, engineering mirabilia of Hopkins in terms of optics and lenses allowed for introduction of rigid and flexible endoscopes as a viable tool in neurosurgery. Nowadays, neurosurgeons are aware of and confident using effective and modern tools of visualization in their armamentarium. Herein we present a cogent review of the evolution of visualization tools in neurosurgery, with a special glimpse into the current development and future achievements.

Augmented and virtual reality usage in awake craniotomy: a systematic review

Augmented and virtual reality (AR, VR) are becoming promising tools in neurosurgery. AR and VR can reduce challenges associated with conventional approaches via the simulation and mimicry of specific environments of choice for surgeons. Awake craniotomy (AC) enables the resection of lesions from eloquent brain areas while monitoring higher cortical and subcortical functions. Evidence suggests that both surgeons and patients benefit from the various applications of AR and VR in AC. This paper investigates the application of AR and VR in AC and assesses its prospective utility in neurosurgery. A systematic review of the literature was performed using PubMed, Scopus, and Web of Science databases in accordance with the PRISMA guidelines. Our search results yielded 220 articles. A total of six articles consisting of 118 patients have been included in this review. VR was used in four papers, and the other two used AR. Tumour was the most common pathology in 108 patients, followed by vascular lesions in eight patients. VR was used for intraoperative mapping of language, vision, and social cognition, while AR was incorporated in preoperative training of white matter dissection and intraoperative visualisation and navigation. Overall, patients and surgeons were satisfied with the applications of AR and VR in their cases. AR and VR can be safely incorporated during AC to supplement, augment, or even replace conventional approaches in neurosurgery. Future investigations are required to assess the feasibility of AR and VR in various phases of AC.

The effect of virtual reality on temporal bone anatomy evaluation and performance

PURPOSE

There is only limited data on the application of virtual reality (VR) for the evaluation of temporal bone anatomy. The aim of the present study was to compare the VR environment to traditional cross-sectional viewing of computed tomography images in a simulated preoperative planning setting in novice and expert surgeons.

METHODS

A novice (n = 5) and an expert group (n = 5), based on their otosurgery experience, were created. The participants were asked to identify 24 anatomical landmarks, perform 11 distance measurements between surgically relevant anatomical structures and 10 fiducial markers on five cadaver temporal bones in both VR environment and cross-sectional viewings in PACS interface. The data on performance time and user-experience (i.e., subjective validation) were collected.

RESULTS

The novice group made significantly more errors (p < 0.001) and with significantly longer performance time (p = 0.001) in cross-sectional viewing than the expert group. In the VR environment, there was no significant differences (errors and time) between the groups. The performance of novices improved faster in the VR. The novices showed significantly faster task performance (p = 0.003) and a trend towards fewer errors (p = 0.054) in VR compared to cross-sectional viewing. No such difference between the methods were observed in the expert group. The mean overall scores of user-experience were significantly higher for VR than cross-sectional viewing in both groups (p < 0.001).

CONCLUSION

In the VR environment, novices performed the anatomical evaluation of temporal bone faster and with fewer errors than in the traditional cross-sectional viewing, which supports its efficiency for the evaluation of complex anatomy.

Augmented Reality in Surgery: A Scoping Review

Augmented reality (AR) is an innovative system that enhances the real world by superimposing virtual objects on reality. The aim of this study was to analyze the application of AR in medicine and which of its technical solutions are the most used. We carried out a scoping review of the articles published between 2019 and February 2022. The initial search yielded a total of 2649 articles. After applying filters, removing duplicates and screening, we included 34 articles in our analysis. The analysis of the articles highlighted that AR has been traditionally and mainly used in orthopedics in addition to maxillofacial surgery and oncology. Regarding the display application in AR, the Microsoft HoloLens Optical Viewer is the most used method. Moreover, for the tracking and registration phases, the marker-based method with a rigid registration remains the most used system. Overall, the results of this study suggested that AR is an innovative technology with numerous advantages, finding applications in several new surgery domains. Considering the available data, it is not possible to clearly identify all the fields of application and the best technologies regarding AR.

How effective is virtual reality technology in palliative care? A systematic review and meta-analysis

BACKGROUND

The efficacy of virtual reality for people living with a terminal illness is unclear.

AIM

To determine the feasibility and effectiveness of virtual reality use within a palliative care setting.

DESIGN

Systematic review and meta-analysis. PROSPERO (CRD42021240395).

DATA SOURCES

Medline, Embase, AMED, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials and Web of Science were searched from inception to March 2021. Search terms included 'virtual reality' and 'palliative care'. Eligibility: (1) adult (>18 years old) with a terminal illness (2) at least one virtual reality session and (3) feasibility data and/or at least one patient outcome reported. The ROB-2 and ROBINS tools assessed risk of bias. The Grading of Recommendations, Assessment, Development and Evaluations (GRADE) tool assessed the quality of the evidence. Standardised mean differences (Hedges's g) were calculated from the pre- and post-data. A DerSimonian-Laird random effects model meta-analysis was conducted.

RESULTS

Eight studies were included, of which five were in the meta-analysis. All studies had at least some concern for risk of bias. Virtual reality statistically significantly improved pain (p = 0.0363), tiredness (p = 0.0030), drowsiness (p = 0.0051), shortness of breath (p = 0.0284), depression (p = 0.0091) and psychological well-being (p = 0.0201). The quality of the evidence was graded as very low due to small sample sizes, non-randomisation methods and a lack of a comparator arm.

CONCLUSIONS

Virtual reality in palliative care is feasible and acceptable. However, limited sample sizes and very low-quality studies mean that the efficacy of virtual reality needs further research.

Virtual Reality in Neurosurgery: Beyond Neurosurgical Planning

Background: While several publications have focused on the intuitive role of augmented reality (AR) and virtual reality (VR) in neurosurgical planning, the aim of this review was to explore other avenues, where these technologies have significant utility and applicability. Methods: This review was conducted by searching PubMed, PubMed Central, Google Scholar, the Scopus database, the Web of Science Core Collection database, and the SciELO citation index, from 1989–2021. An example of a search strategy used in PubMed Central is: “Virtual reality” [All Fields] AND (“neurosurgical procedures” [MeSH Terms] OR (“neurosurgical” [All Fields] AND “procedures” [All Fields]) OR “neurosurgical procedures” [All Fields] OR “neurosurgery” [All Fields] OR “neurosurgery” [MeSH Terms]). Using this search strategy, we identified 487 (PubMed), 1097 (PubMed Central), and 275 citations (Web of Science Core Collection database). Results: Articles were found and reviewed showing numerous applications of VR/AR in neurosurgery. These applications included their utility as a supplement and augment for neuronavigation in the fields of diagnosis for complex vascular interventions, spine deformity correction, resident training, procedural practice, pain management, and rehabilitation of neurosurgical patients. These technologies have also shown promise in other area of neurosurgery, such as consent taking, training of ancillary personnel, and improving patient comfort during procedures, as well as a tool for training neurosurgeons in other advancements in the field, such as robotic neurosurgery. Conclusions: We present the first review of the immense possibilities of VR in neurosurgery, beyond merely planning for surgical procedures. The importance of VR and AR, especially in “social distancing” in neurosurgery training, for economically disadvantaged sections, for prevention of medicolegal claims and in pain management and rehabilitation, is promising and warrants further research.

New Approach to Accelerated Image Annotation by Leveraging Virtual Reality and Cloud Computing

Three-dimensional imaging is at the core of medical imaging and is becoming a standard in biological research. As a result, there is an increasing need to visualize, analyze and interact with data in a natural three-dimensional context. By combining stereoscopy and motion tracking, commercial virtual reality (VR) headsets provide a solution to this critical visualization challenge by allowing users to view volumetric image stacks in a highly intuitive fashion. While optimizing the visualization and interaction process in VR remains an active topic, one of the most pressing issue is how to utilize VR for annotation and analysis of data. Annotating data is often a required step for training machine learning algorithms. For example, enhancing the ability to annotate complex three-dimensional data in biological research as newly acquired data may come in limited quantities. Similarly, medical data annotation is often time-consuming and requires expert knowledge to identify structures of interest correctly. Moreover, simultaneous data analysis and visualization in VR is computationally demanding. Here, we introduce a new procedure to visualize, interact, annotate and analyze data by combining VR with cloud computing. VR is leveraged to provide natural interactions with volumetric representations of experimental imaging data. In parallel, cloud computing performs costly computations to accelerate the data annotation with minimal input required from the user. We demonstrate multiple proof-of-concept applications of our approach on volumetric fluorescent microscopy images of mouse neurons and tumor or organ annotations in medical images.

Augmented reality visualization in brain lesions: a prospective randomized controlled evaluation of its potential and current limitations in navigated microneurosurgery

Background

Augmented reality (AR) has the potential to support complex neurosurgical interventions by including visual information seamlessly. This study examines intraoperative visualization parameters and clinical impact of AR in brain tumor surgery.

Methods

Fifty-five intracranial lesions, operated either with AR-navigated microscope (n = 39) or conventional neuronavigation (n = 16) after randomization, have been included prospectively. Surgical resection time, duration/type/mode of AR, displayed objects (n, type), pointer-based navigation checks (n), usability of control, quality indicators, and overall surgical usefulness of AR have been assessed.

Results

AR display has been used in 44.4% of resection time. Predominant AR type was navigation view (75.7%), followed by target volumes (20.1%). Predominant AR mode was picture-in-picture (PiP) (72.5%), followed by 23.3% overlay display. In 43.6% of cases, vision of important anatomical structures has been partially or entirely blocked by AR information. A total of 7.7% of cases used MRI navigation only, 30.8% used one, 23.1% used two, and 38.5% used three or more object segmentations in AR navigation. A total of 66.7% of surgeons found AR visualization helpful in the individual surgical case. AR depth information and accuracy have been rated acceptable (median 3.0 vs. median 5.0 in conventional neuronavigation). The mean utilization of the navigation pointer was 2.6 × /resection hour (AR) vs. 9.7 × /resection hour (neuronavigation); navigation effort was significantly reduced in AR (P < 0.001).

Conclusions

The main benefit of HUD-based AR visualization in brain tumor surgery is the integrated continuous display allowing for pointer-less navigation. Navigation view (PiP) provides the highest usability while blocking the operative field less frequently. Visualization quality will benefit from improvements in registration accuracy and depth impression.

German clinical trials registration number.

DRKS00016955.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00701-021-05045-1.

Evaluation of augmented-reality based navigation for brain tumor surgery

To date, several researchers have introduced augmented reality navigation (ARN) into neurological surgery. While its application in brain tumor surgery seems promising, reports on its utility have been limited, thus warranting further evaluation. To clarify the stages and approaches in which ARN is useful and assess the effect of presurgical discussion with surgeons, we assessed usefulness using a hand-held ARN system we had developed, which displays three-dimensional (3D) virtual structures overlaid on a real-time image of the surgical field via a tablet PC monitor. The system was tested in 20 patients undergoing various procedures, with the first 10 consecutive cases being unselected and the following 10 cases being selected, for whom 3D models were prepared per the surgeons' request. Thereafter, the surgeons ranked its usefulness during each stage of surgery. Consequently, case selection and presurgical discussions with surgeons considerably improved the usefulness, with the "useful" gradings improving from 50% to 88% across all surgical stages. Accordingly, usefulness improved from 50% to 90%, 67% to 100%, and 40% to 80% during the skin incision and craniotomy, dura incision, and intradural procedure stages, respectively. ARN was useful for superficial tumor resection, but less so for deep-seated tumor resection, except when using the transcortical and interhemispheric approaches. In conclusion, a tablet-type ARN can be useful during skin incisions, craniotomy and dura incisions, superficial tumor resections, and transcortical and interhemispheric approaches for deep-seated tumors. Case selection and presurgical discussions with surgeons were essential for the efficacy of ARN.

Examining the benefits of extended reality in neurosurgery: A systematic review

While well-established in other surgical subspecialties, the benefits of extended reality, consisting of virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies, remains underexplored in neurosurgery despite its increasing utilization. To address this gap, we conducted a systematic review of the effects of extended reality (XR) in neurosurgery with an emphasis on the perioperative period, to provide a guide for future clinical optimization. Seven primary electronic databases were screened following guidelines outlined by PRISMA and the Institute of Medicine. Reported data related to outcomes in the perioperative period and resident training were all examined, and a focused analysis of studies reporting controlled, clinical outcomes was completed. After removal of duplicates, 2548 studies were screened with 116 studies reporting measurable effects of XR in neurosurgery. The majority (82%) included cranial based applications related to tumor surgery with 34% showing improved resection rates and functional outcomes. A rise in high-quality studies was identified from 2017 to 2020 compared to all previous years (p = 0.004). Primary users of the technology were: 56% neurosurgeon (n = 65), 28% residents (n = 33) and 5% patients (n = 6). A final synthesis was conducted on 10 controlled studies reporting patient outcomes. XR technologies have demonstrated benefits in preoperative planning and multimodal neuronavigation especially for tumor surgery. However, few studies have reported patient outcomes in a controlled design demonstrating a need for higher quality data. XR platforms offer several advantages to improve patient outcomes and specifically, the patient experience for neurosurgery.

Development of a New Image-Guided Neuronavigation System: Mixed-Reality Projection Mapping Is Accurate and Feasible

BACKGROUND

Image-guided systems improve the safety, functional outcome, and overall survival of neurosurgery but require extensive equipment.

OBJECTIVE

To develop an image-guided surgery system that combines the brain surface photographic texture (BSP-T) captured during surgery with 3-dimensional computer graphics (3DCG) using projection mapping.

METHODS

Patients who underwent initial surgery with brain tumors were prospectively enrolled. The texture of the 3DCG (3DCG-T) was obtained from 3DCG under similar conditions as those when capturing the brain surface photographs. The position and orientation at the time of 3DCG-T acquisition were used as the reference. The correct position and orientation of the BSP-T were obtained by aligning the BSP-T with the 3DCG-T using normalized mutual information. The BSP-T was combined with and displayed on the 3DCG using projection mapping. This mixed-reality projection mapping (MRPM) was used prospectively in 15 patients (mean age 46.6 yr, 6 males). The difference between the centerlines of surface blood vessels on the BSP-T and 3DCG constituted the target registration error (TRE) and was measured in 16 fields of the craniotomy area. We also measured the time required for image processing.

RESULTS

The TRE was measured at 158 locations in the 15 patients, with an average of 1.19 ± 0.14 mm (mean ± standard error). The average image processing time was 16.58 min.

CONCLUSION

Our MRPM method does not require extensive equipment while presenting information of patients' anatomy together with medical images in the same coordinate system. It has the potential to improve patient safety.

Augmented and virtual reality in spine surgery, current applications and future potentials

BACKGROUND CONTEXT

The field of artificial intelligence (AI) is rapidly advancing, especially with recent improvements in deep learning (DL) techniques. Augmented (AR) and virtual reality (VR) are finding their place in healthcare, and spine surgery is no exception. The unique capabilities and advantages of AR and VR devices include their low cost, flexible integration with other technologies, user-friendly features and their application in navigation systems, which makes them beneficial across different aspects of spine surgery. Despite the use of AR for pedicle screw placement, targeted cervical foraminotomy, bone biopsy, osteotomy planning, and percutaneous intervention, the current applications of AR and VR in spine surgery remain limited.

PURPOSE

The primary goal of this study was to provide the spine surgeons and clinical researchers with the general information about the current applications, future potentials, and accessibility of AR and VR systems in spine surgery.

STUDY DESIGN/SETTING

We reviewed titles of more than 250 journal papers from google scholar and PubMed with search words: augmented reality, virtual reality, spine surgery, and orthopaedic, out of which 89 related papers were selected for abstract review. Finally, full text of 67 papers were analyzed and reviewed.

METHODS

The papers were divided into four groups: technological papers, applications in surgery, applications in spine education and training, and general application in orthopaedic. A team of two reviewers performed paper reviews and a thorough web search to ensure the most updated state of the art in each of four group is captured in the review.

RESULTS

In this review we discuss the current state of the art in AR and VR hardware, their preoperative applications and surgical applications in spine surgery. Finally, we discuss the future potentials of AR and VR and their integration with AI, robotic surgery, gaming, and wearables.

CONCLUSIONS

AR and VR are promising technologies that will soon become part of standard of care in spine surgery.

Evaluation of the effect of standard neuronavigation and augmented reality on the integrity of the perifocal structures during a neurosurgical approach

OBJECTIVE

Intracranial minimally invasive procedures imply working in a restricted surgical corridor surrounded by critical structures, such as vessels and cranial nerves. Any damage to them may affect patient outcome. Neuronavigation systems may reduce the risk of such complications. In this study, the authors sought to compare standard neuronavigation (NV) and augmented reality (AR)-guided navigation with respect to the integrity of the perifocal structures during a neurosurgical approach using a novel model imitating intracranial vessels.

METHODS

A custom-made box, containing crisscrossing hard metal wires, a hidden nail at its bottom, and a wooden top, was scanned, fused, and referenced for the purpose of the study. The metal wires and an aneurysm clip applier were connected to a controller, which counted the number of contacts between them. Twenty-three naive participants were asked to 1) use NV to define an optimal entry point on the top, perform the smallest craniotomy possible on the wooden top, and to use a surgical microscope when placing a clip on the nail without touching the metal wires; and 2) use AR to preoperatively define an ideal trajectory, navigate the surgical microscope, and then perform the same task. The primary outcome was the number of contacts made between the metal wires and the clip applier. Secondary outcomes were craniotomy size, and trust in NV and AR to help avoid touching the metal wires, as assessed by a 9-level Likert scale.

RESULTS

The median number of contacts tended to be lower with the use of AR than with NV (AR, median 1 [Q1: 1, Q3: 2]; NV, median 3 [Q1: 1, Q3: 6]; p = 0.074). The size of the target-oriented craniotomy was significantly lower with the use of AR compared with NV (AR, median 4.91 cm2 [Q1: 4.71 cm2, Q3: 7.55 cm2]; and NV, median 9.62 cm2 [Q1: 7.07 cm2; Q3: 13.85 cm2]). Participants had more trust in AR than in NV (the differences posttest minus pretest were mean 0.9 [SD 1.2] and mean -0.3 [SD 0.2], respectively; p < 0.05).

CONCLUSIONS

The results of this study show a trend favoring the use of AR over NV with respect to reducing contact between a clip applier and the perifocal structures during a simulated clipping of an intracranial aneurysm. Target-guided craniotomies were smaller with the use of AR. AR may be used not only to localize surgical targets but also to prevent complications associated with damage to structures encountered during the surgical approach.

Augmented Reality in Medical Practice: From Spine Surgery to Remote Assistance

Background: While performing surgeries in the OR, surgeons and assistants often need to access several information regarding surgical planning and/or procedures related to the surgery itself, or the accessory equipment to perform certain operations. The accessibility of this information often relies on the physical presence of technical and medical specialists in the OR, which is increasingly difficult due to the number of limitations imposed by the COVID emergency to avoid overcrowded environments or external personnel. Here, we analyze several scenarios where we equipped OR personnel with augmented reality (AR) glasses, allowing a remote specialist to guide OR operations through voice and ad-hoc visuals, superimposed to the field of view of the operator wearing them.

Methods: This study is a preliminary case series of prospective collected data about the use of AR-assistance in spine surgery from January to July 2020. The technology has been used on a cohort of 12 patients affected by degenerative lumbar spine disease with lumbar sciatica co-morbidities. Surgeons and OR specialists were equipped with AR devices, customized with P2P videoconference commercial apps, or customized holographic apps. The devices were tested during surgeries for lumbar arthrodesis in a multicenter experience involving author's Institutions.

Findings: A total number of 12 lumbar arthrodesis have been performed while using the described AR technology, with application spanning from telementoring (3), teaching (2), surgical planning superimposition and interaction with the hologram using a custom application for Microsoft hololens (1). Surgeons wearing the AR goggles reported a positive feedback as for the ergonomy, wearability and comfort during the procedure; being able to visualize a 3D reconstruction during surgery was perceived as a straightforward benefit, allowing to speed-up procedures, thus limiting post-operational complications. The possibility of remotely interacting with a specialist on the glasses was a potent added value during COVID emergency, due to limited access of non-resident personnel in the OR.

Interpretation: By allowing surgeons to overlay digital medical content on actual surroundings, augmented reality surgery can be exploited easily in multiple scenarios by adapting commercially available or custom-made apps to several use cases. The possibility to observe directly the operatory theater through the eyes of the surgeon might be a game-changer, giving the chance to unexperienced surgeons to be virtually at the site of the operation, or allowing a remote experienced operator to guide wisely the unexperienced surgeon during a procedure.

Development of Innovative Neurosurgical Operation Support Method Using Mixed-Reality Computer Graphics

Background

In neurosurgery, it is important to inspect the spatial correspondence between the preoperative medical image (virtual space), and the intraoperative findings (real space) to improve the safety of the surgery. Navigation systems and related modalities have been reported as methods for matching this correspondence. However, because of the influence of the brain shift accompanying craniotomy, registration accuracy is reduced. In the present study, to overcome these issues, we developed a spatially accurate registration method of medical fusion 3-dimensional computer graphics and the intraoperative brain surface photograph, and its registration accuracy was measured.

Methods

The subjects included 16 patients with glioma. Nonrigid registration using the landmarks and thin-plate spline methods was performed for the fusion 3-dimensional computer graphics and the intraoperative brain surface photograph, termed mixed-reality computer graphics. Regarding the registration accuracy measurement, the target registration error was measured by two neurosurgeons, with 10 points for each case at the midpoint of the landmarks.

Results

The number of target registration error measurement points was 160 in the 16 cases. The target registration error was 0.72 ± 0.04 mm. Aligning the intraoperative brain surface photograph and the fusion 3-dimensional computer graphics required ∼10 minutes on average. The average number of landmarks used for alignment was 24.6.

Conclusions

Mixed-reality computer graphics enabled highly precise spatial alignment between the real space and virtual space. Mixed-reality computer graphics have the potential to improve the safety of the surgery by allowing complementary observation of brain surface photographs and fusion 3-dimensional computer graphics.

Evaluation of the precision of operative augmented reality compared to standard neuronavigation using a 3D-printed skull

OBJECTIVE

Augmented reality (AR) in cranial surgery allows direct projection of preregistered overlaid images in real time on the microscope surgical field. In this study, the authors aimed to compare the precision of AR-assisted navigation and standard pointer-based neuronavigation (NV) by using a 3D-printed skull in surgical conditions.

METHODS

A commercial standardized 3D-printed skull was scanned, fused, and referenced with an MR image and a CT scan of a patient with a 2 × 2-mm right frontal sinus defect. The defect was identified, registered, and integrated into NV. The target was physically marked on the 3D-printed skull replicating the right frontal sinus defect. Twenty-six subjects participated, 25 of whom had no prior NV or AR experience and 1 with little AR experience. The subjects were briefly trained in how to use NV, AR, and AR recalibration tools. Participants were asked to do the following: 1) "target the center of the defect in the 3D-printed skull with a navigation pointer, assisted only by NV orientation," and 2) "use the surgical microscope and AR to focus on the center of the projected object" under conventional surgical conditions. For the AR task, the number of recalibrations was recorded. Confidence regarding NV and AR precision were assessed prior to and after the experiment by using a 9-level Likert scale.

RESULTS

The median distance to target was statistically lower for AR than for NV (1 mm [Q1: 1 mm, Q3: 2 mm] vs 3 mm [Q1: 2 mm, Q3: 4 mm] [p < 0.001]). In the AR task, the median number of recalibrations was 4 (Q1: 4, Q3: 4.75). The number of recalibrations was significantly correlated with the precision (Spearman rho: -0.71, p < 0.05). The trust assessment after performing the experiment scored a median of 8 for AR and 5.5 for NV (p < 0.01).

CONCLUSIONS

This study shows for the first time the superiority of AR over NV in terms of precision. AR is easy to use. The number of recalibrations performed using reference structures increases the precision of the navigation. The confidence regarding precision increases with experience.

Comprehensive review of surgical microscopes: technology development and medical applications

SIGNIFICANCE

Surgical microscopes provide adjustable magnification, bright illumination, and clear visualization of the surgical field and have been increasingly used in operating rooms. State-of-the-art surgical microscopes are integrated with various imaging modalities, such as optical coherence tomography (OCT), fluorescence imaging, and augmented reality (AR) for image-guided surgery.

AIM

This comprehensive review is based on the literature of over 500 papers that cover the technology development and applications of surgical microscopy over the past century. The aim of this review is threefold: (i) providing a comprehensive technical overview of surgical microscopes, (ii) providing critical references for microscope selection and system development, and (iii) providing an overview of various medical applications.

APPROACH

More than 500 references were collected and reviewed. A timeline of important milestones during the evolution of surgical microscope is provided in this study. An in-depth technical overview of the optical system, mechanical system, illumination, visualization, and integration with advanced imaging modalities is provided. Various medical applications of surgical microscopes in neurosurgery and spine surgery, ophthalmic surgery, ear-nose-throat (ENT) surgery, endodontics, and plastic and reconstructive surgery are described.

RESULTS

Surgical microscopy has been significantly advanced in the technical aspects of high-end optics, bright and shadow-free illumination, stable and flexible mechanical design, and versatile visualization. New imaging modalities, such as hyperspectral imaging, OCT, fluorescence imaging, photoacoustic microscopy, and laser speckle contrast imaging, are being integrated with surgical microscopes. Advanced visualization and AR are being added to surgical microscopes as new features that are changing clinical practices in the operating room.

CONCLUSIONS

The combination of new imaging technologies and surgical microscopy will enable surgeons to perform challenging procedures and improve surgical outcomes. With advanced visualization and improved ergonomics, the surgical microscope has become a powerful tool in neurosurgery, spinal, ENT, ophthalmic, plastic and reconstructive surgeries.

A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery

Background: Research proves that the apprenticeship model, which is the gold standard for training surgical residents, is obsolete. For that reason, there is a continuing effort toward the development of high-fidelity surgical simulators to replace the apprenticeship model. Applying Virtual Reality Augmented Reality (AR) and Mixed Reality (MR) in surgical simulators increases the fidelity, level of immersion and overall experience of these simulators.Areas covered: The objective of this review is to provide a comprehensive overview of the application of VR, AR and MR for distinct surgical disciplines, including maxillofacial surgery and neurosurgery. The current developments in these areas, as well as potential future directions, are discussed.Expert opinion: The key components for incorporating VR into surgical simulators are visual and haptic rendering. These components ensure that the user is completely immersed in the virtual environment and can interact in the same way as in the physical world. The key components for the application of AR and MR into surgical simulators include the tracking system as well as the visual rendering. The advantages of these surgical simulators are the ability to perform user evaluations and increase the training frequency of surgical residents.

Virtual reality improves the accuracy of simulated preoperative planning in temporal bones: a feasibility and validation study

PURPOSE

Consumer-grade virtual reality (VR) has recently enabled various medical applications, but more evidence supporting their validity is needed. We investigated the accuracy of simulated surgical planning in a VR environment (VR) with temporal bones and compared it to conventional cross-sectional image viewing in picture archiving and communication system (PACS) interface.

METHODS

Five experienced otologic surgeons measured significant anatomic structures and fiducials on five fresh-frozen cadaveric temporal bones in VR and cross-sectional viewing. Primary image data were acquired by computed tomography. In total, 275 anatomical landmark measurements and 250 measurements of the distance between fiducials were obtained with both methods. Distance measurements between the fiducials were confirmed by physical measurement obtained by Vernier caliper. The experts evaluated the subjective validity of both methods on a 5-point Likert scale qualitative survey.

RESULTS

A strong correlation based on intraclass coefficient was found between the methods on both the anatomical (r > 0.900) and fiducial measurements (r > 0.916). Two-tailed paired t-test and Bland-Altman plots demonstrated high equivalences between the VR and cross-sectional viewing with mean differences of 1.9% (p = 0.396) and 0.472 mm (p = 0.065) for anatomical and fiducial measurements, respectively. Gross measurement errors due to the misidentification of fiducials occurred more frequently in the cross-sectional viewing. The mean face and content validity rating for VR were significantly better compared to cross-sectional viewing (total mean score 4.11 vs 3.39, p < 0.001).

CONCLUSION

Our study supports good accuracy and reliability of VR environment for simulated surgical planning in temporal bones compared to conventional cross-sectional visualization.

Augmented reality in neurosurgical navigation: a survey

BACKGROUND

Neurosurgery has exceptionally high requirements for minimally invasive and safety. This survey attempts to analyze the practical application of AR in neurosurgical navigation. Also, this survey describes future trends in augmented reality neurosurgical navigation systems.

METHODS

In this survey, we searched related keywords "augmented reality", "virtual reality", "neurosurgery", "surgical simulation", "brain tumor surgery", "neurovascular surgery", "temporal bone surgery", and "spinal surgery" through Google Scholar, World Neurosurgery, PubMed and Science Direct. We collected 85 articles published over the past five years in areas related to this survey.

RESULTS

Detailed study has been conducted on the application of AR in neurosurgery and found that AR is constantly improving the overall efficiency of doctor training and treatment, which can help neurosurgeons learn and practice surgical procedures with zero risks.

CONCLUSIONS

Neurosurgical navigation is essential in neurosurgery. Despite certain technical limitations, it is still a necessary tool for the pursuit of maximum security and minimal intrusiveness. This article is protected by copyright. All rights reserved.

Mini Fronto-Orbital Approach: "Window Opening" Towards the Superomedial Orbit - A Virtual Reality-Planned Anatomic Study

BACKGROUND

Surgical approaches to the orbit are challenging and require combined multispecialist skills. Considering its increasing relevance in neurosurgical practice, keyhole surgery could be also applied to this field. However, mastering a minimally invasive approach necessitates an extended learning curve. For this reason, virtual reality (VR) can be effectively used for planning and training in this demanding surgical technique.

OBJECTIVE

To validate the mini fronto-orbital (mFO) approach to the superomedial orbit, using VR planning and specimen dissections, conjugating the principles of skull base and keyhole neurosurgery.

METHODS

Three-dimensional measurements were performed thanks to Surgical Theater (Surgical Theater© LLC), and then, simulated craniotomies were implemented on cadaver specimens.

RESULTS

The mFO approach affords optimal exposure and operability in the target area and reduced risks of surrounding normal tissue injuries. The eyebrow skin incision, the minimal soft-tissue retraction, the limited temporalis muscle dissection and the single-piece craniotomy, as planned with VR, are the key elements of this minimally invasive approach. Furthermore, the "window-opening" cotton-tip intraorbital dissection technique, based on widening surgical corridors between neuromuscular bundles, provides a safe orientation and a deep access inside the orbit, thereby significantly limiting the risk of jeopardizing neurovascular structures.

CONCLUSION

The mFO approach associated to the window-opening dissection technique can be considered safe, effective, suitable, and convenient for treating lesions located in the superomedial orbital aspect, up to the orbital apex.

Mapping the intellectual structure of research on surgery with mixed reality: Bibliometric network analysis (2000-2019)

OBJECTIVE

The purpose of this study is to view research trends on surgery with mixed reality, and present the intellectual structure using bibliometric network analysis for the period 2000-2019.

METHODS

Analyses are implemented in the following four steps: (1) literature dataset acquisition from article database (Web of Science, Scopus, PubMed, and IEEE digital library), (2) dataset pre-processing and refinement, (3) network construction and visualization, and (4) analysis and interpretation. Descriptive analysis, bibliometric network analysis, and in-depth qualitative analysis were conducted.

RESULTS

The 14,591 keywords of 5897 abstracts data were ultimately used to ascertain the intellectual structure of research on surgery with mixed reality. The dynamics of the evolution of keywords in the structure throughout the four periods is summarized with four aspects: (a) maintaining a predominant utilization tool for training, (b) widening clinical application area, (c) reallocating the continuum of mixed reality, and (d) steering advanced imaging and simulation technology.

CONCLUSIONS

The results of this study can provide valuable insights into technology adoption and research trends of mixed reality in surgery. These findings can help clinicians to overview prospective medical research on surgery using mixed reality. Hospitals can also understand the periodical maturity of technology of mixed reality in surgery, and, therefore, these findings can suggest an academic landscape to make a decision in adopting new technologies in surgery.

Organic Haptics: Intersection of Materials Chemistry and Tactile Perception

The goal of the field of haptics is to create technologies that manipulate the sense of touch. In virtual and augmented reality, haptic devices are for touch what loudspeakers and RGB displays are for hearing and vision. Haptic systems that utilize micromotors or other miniaturized mechanical devices (e.g., for vibration and pneumatic actuation) produce interesting effects, but are quite far from reproducing the feeling of real materials. They are especially deficient in recapitulating surface properties: fine texture, friction, viscoelasticity, tack, and softness. The central argument of this Progress Report is that to reproduce the feel of everyday objects requires chemistry: molecular control over the properties of materials and ultimately design of materials which can change these properties in real time. Stimuli-responsive organic materials, such as polymers and composites, are a class of materials which can change their oxidation state, conductivity, shape, and rheological properties, and thus might be useful in future haptic technologies. Moreover, the use of such materials in research on tactile perception could help elucidate the limits of human tactile sensitivity. The work described represents the beginnings of this new area of inquiry, in which the defining approach is the marriage of materials science and psychology.

Virtual Reality in Neurosurgery: "Can You See It?"-A Review of the Current Applications and Future Potential

Virtual reality (VR) technology had its early development in the 1960s in the U.S. Air Force and has since evolved into a budding area of scientific research with many practical medical purposes. From medical education to resident training to the operating room, VR has provided tangible benefits to learners and trainees and has also improved surgery through enhanced preoperative planning and efficiency in the operating room. Neurosurgery is a particularly complex field of medicine, in which VR has blossomed into a tool with great usefulness and promise. In spinal surgery, VR simulation has allowed for the practice of innovative minimally invasive procedures. In cranial surgery, VR has excelled in helping neurosurgeons design unique patient-specific approaches to particularly challenging tumor excisions. In neurovascular surgery, VR has helped trainees practice and perfect procedures requiring high levels of dexterity to minimize intraoperative complications and patient radiation exposure. In peripheral nerve surgery, VR has allowed surgeons to gain increased practice and comfort with complex microsurgeries such as nerve decompression. Overall, VR continues to increase its potential in neurosurgery and is poised to benefit patients in a multitude of ways. Although cost-prohibiting, legal, and ethical challenges surrounding this technology must be considered, future research and more direct quantitative outcome comparisons between standard and VR-supplemented procedures would help provide more direction regarding the feasibility of widespread adoption of VR technology in neurosurgery.

Augmented Reality in Superficial Temporal Artery to Middle Cerebral Artery Bypass Surgery: Technical Note

BACKGROUND

Augmented reality (AR) applied to surgery refers to the virtual superimposition of computer-generated anatomical information on the surgical field. AR assistance in extracranial-intracranial (EC-IC) bypass revascularization surgery has been reported to be a helpful technical adjunct.

OBJECTIVE

To describe our experience of using AR in superficial temporal artery to middle cerebral artery (STA-MCA) bypass surgery with the additional implementation of new technical processes to improve the safety and efficacy of the procedure.

METHODS

Data sets from preoperative imaging were loaded and fused in a single 3-dimensional matrix using the neuronavigation system. Anatomical structures of interest (the STA, a selected M4 branch of the MCA, the middle meningeal artery [MMA], and the primary motor cortex [PMC]) were segmented. After the registration of the patient and the operating microscope, the structures of interest were projected into the eyepiece of the microscope and superimposed onto the patient's head, creating the AR surgical field.

RESULTS

AR was shown to be useful in patients undergoing EC-IC bypass revascularization, mostly during the following 4 surgical steps: (1) microsurgical dissection of the donor vessel (STA); (2) tailoring the craniotomy above the recipient vessel (M4 branch of the MCA); (3) tailoring the craniotomy to spare the MMA; and (4) tailoring the craniotomy and the anastomosis to spare the PMC.

CONCLUSION

AR assistance in EC-IC bypass revascularization is a versatile technical adjunct for helping surgeons to ensure the safety and efficacy of the procedure.