Use of augmented reality for image-guided spine procedures

Extended reality navigation for pain procedures: a narrative review

BACKGROUND

Extended reality (XR) technology, encompassing virtual reality, augmented reality, and mixed reality, has been widely studied for procedural navigation in surgical specialties. Similar to how ultrasound transformed regional anesthesia, XR has the potential to reshape how anesthesiologists and pain physicians perform procedures to relieve pain.

OBJECTIVE

This narrative review examines the clinical benefits of XR for navigation in various pain procedures. It defines key terms and concepts related to XR technology and explores characteristics of procedures that are most amenable to XR-based navigation. Finally, it suggests best practices for developing XR navigation systems and discusses the role of emerging technology in the future of XR in regional anesthesia and pain medicine.

EVIDENCE REVIEW

A search was performed across PubMed, Embase, and Cochrane Central Register of Controlled Trials for primary literature investigating the clinical benefits of XR navigation for pain procedures.

FINDINGS

Thirteen studies using XR for procedural navigation are included. The evidence includes randomized controlled trials, retrospective studies, and case series.

CONCLUSIONS

Early randomized controlled trials show potential for XR to improve procedural efficiency, but more comprehensive research is needed to determine if there are significant clinical benefits. Case reports demonstrate XR's utility in generating patient-specific navigation plans when difficult anatomy is encountered. Procedures that facilitate the generation and registration of XR images are most conducive to XR navigation, whereas those that rely on frequent re-imaging will continue to depend on traditional modes of navigation.

Virtual, Augmented, and Mixed Reality Applications for Surgical Rehearsal, Operative Execution, and Patient Education in Spine Surgery: A Scoping Review

Background and Objectives: Advances in virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies have resulted in their increased application across many medical specialties. VR's main application has been for teaching and preparatory roles, while AR has been mostly used as a surgical adjunct. The objective of this study is to discuss the various applications and prospects for VR, AR, and MR specifically as they relate to spine surgery. Materials and Methods: A systematic review was conducted to examine the current applications of VR, AR, and MR with a focus on spine surgery. A literature search of two electronic databases (PubMed and Scopus) was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). The study quality was assessed using the MERSQI score for educational research studies, QUACS for cadaveric studies, and the JBI critical appraisal tools for clinical studies. Results: A total of 228 articles were identified in the primary literature review. Following title/abstract screening and full-text review, 46 articles were included in the review. These articles comprised nine studies performed in artificial models, nine cadaveric studies, four clinical case studies, nineteen clinical case series, one clinical case-control study, and four clinical parallel control studies. Teaching applications utilizing holographic overlays are the most intensively studied aspect of AR/VR; the most simulated surgical procedure is pedicle screw placement. Conclusions: VR provides a reproducible and robust medium for surgical training through surgical simulations and for patient education through various platforms. Existing AR/MR platforms enhance the accuracy and precision of spine surgeries and show promise as a surgical adjunct.

Evaluating a cutting-edge augmented reality-supported navigation system for spinal instrumentation

OBJECTIVE

Dorsal instrumentation using pedicle screws is a standard treatment for multiple spinal pathologies, such as trauma, infection, or degenerative indications. Intraoperative three-dimensional (3D) imaging and navigated pedicle screw placement are used at multiple centers. For the present study, we evaluated a new navigation system enabling augmented reality (AR)-supported pedicle screw placement while integrating navigation cameras into the reference array and drill guide. The present study aimed to evaluate its clinical application regarding safety, efficacy, and accuracy.

METHODS

A total of 20 patients were operated on between 06/2021 and 01/2022 using the new technique for intraoperative navigation. Intraoperative data with a focus on accuracy and patient safety, including patient outcome, were analyzed. The accuracy of pedicle screw placement was evaluated by intraoperative CT imaging.

RESULTS

A median of 8 (4-18) pedicle screws were placed in each case. Percutaneous instrumentation was performed in 14 patients (70%). The duration of pedicle screw placement (duration scan-scan) was 56 ± 26 (30-107) min. Intraoperative screw revision was necessary for 3 of 180 pedicle screws (1.7%). Intraoperatively, no major complications occurred-one case of delay due to software issues and one case of difficult screw placement were reported.

CONCLUSION

The current study's results could confirm the use of the present AR-supported system for navigated pedicle screw placement for dorsal instrumentation in clinical routine. It provides a reliable and safe tool for 3D imaging-based pedicle screw placement, only requires a minimal intraoperative setup, and provides new opportunities by integrating AR.

Applications of Augmented Reality in Orthopaedic Spine Surgery

The application of augmented reality (AR) in surgical settings has primarily been as a navigation tool in the operating room because of its ease of use and minimal effect on surgical procedures. The surgeon can directly face the surgical field while viewing 3D anatomy virtually, thus reducing the need to look at an external display, such as a navigation system. Applications of AR are being explored in spine surgery. The basic principles of AR include data preparation, registration, tracking, and visualization. Current literature provides sufficient preclinical and clinical data evidence for the use of AR technology in spine surgery. AR systems are efficient assistive devices, providing greater accuracy for insertion points, more comfort for surgeons, and reduced operating time. AR technology also has beneficial applications in surgical training, education, and telementorship for spine surgery. However, costs associated with specially designed imaging equipment and physicians' comfort in using this technology continue to remain barriers to its adoption. As this technology evolves to a more widespread use, future applications will be directed by the cost-effectiveness of AR-assisted surgeries.

Improved Accuracy and Lowered Learning Curve of Ventricular Targeting Using Augmented Reality-Phantom and Cadaveric Model Testing

BACKGROUND

Augmented reality (AR) has demonstrated significant potential in neurosurgical cranial, spine, and teaching applications. External ventricular drain (EVD) placement remains a common procedure, but with error rates in targeting between 10% and 40%.

OBJECTIVE

To evaluate Novarad VisAR guidance system for the placement of EVDs in phantom and cadaveric models.

METHODS

Two synthetic ventricular phantom models and a third cadaver model underwent computerized tomography imaging and registration with the VisAR system (Novarad). Root mean square (RMS), angular error (γ), and Euclidian distance were measured by multiple methods for various standard EVD placements.

RESULTS

Computerized tomography measurements on a phantom model (0.5-mm targets showed a mean Euclidean distance error of 1.20 ± 0.98 mm and γ of 1.25° ± 1.02°. Eight participants placed EVDs in lateral and occipital burr holes using VisAR in a second phantom anatomic ventricular model (mean RMS: 3.9 ± 1.8 mm, γ: 3.95° ± 1.78°). There were no statistically significant differences in accuracy for postgraduate year level, prior AR experience, prior EVD experience, or experience with video games ( P > .05). In comparing EVDs placed with anatomic landmarks vs VisAR navigation in a cadaver, VisAR demonstrated significantly better RMS and γ, 7.47 ± 0.94 mm and 7.12° ± 0.97°, respectively ( P ≤ .05).

CONCLUSION

The novel VisAR AR system resulted in accurate placement of EVDs with a rapid learning curve, which may improve clinical treatment and patient safety. Future applications of VisAR can be expanded to other cranial procedures.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Remote Interactive Surgery Platform (RISP): Proof of Concept for an Augmented-Reality-Based Platform for Surgical Telementoring

The “Remote Interactive Surgery Platform” (RISP) is an augmented reality (AR)-based platform for surgical telementoring. It builds upon recent advances of mixed reality head-mounted displays (MR-HMD) and associated immersive visualization technologies to assist the surgeon during an operation. It enables an interactive, real-time collaboration with a remote consultant by sharing the operating surgeon’s field of view through the Microsoft (MS) HoloLens2 (HL2). Development of the RISP started during the Medical Augmented Reality Summer School 2021 and is currently still ongoing. It currently includes features such as three-dimensional annotations, bidirectional voice communication and interactive windows to display radiographs within the sterile field. This manuscript provides an overview of the RISP and preliminary results regarding its annotation accuracy and user experience measured with ten participants.

Visualization, registration and tracking techniques for augmented reality guided surgery: a review

Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories ofin situvisualization and nonin situvisualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.

A review of advances in image-guided orthopedic surgery

Orthopedic surgery remains technically demanding due to the complex anatomical structures and cumbersome surgical procedures. The introduction of image-guided orthopedic surgery (IGOS) has significantly decreased the surgical risk and improved the operation results. This review focuses on the application of recent advances in artificial intelligence (AI), deep learning (DL), augmented reality (AR) and robotics in image-guided spine surgery, joint arthroplasty, fracture reduction and bone tumor resection. For the pre-operative stage, key technologies of AI and DL based medical image segmentation, 3D visualization and surgical planning procedures are systematically reviewed. For the intra-operative stage, the development of novel image registration, surgical tool calibration and real-time navigation are reviewed. Furthermore, the combination of the surgical navigation system with AR and robotic technology is also discussed. Finally, the current issues and prospects of the IGOS system are discussed, with the goal of establishing a reference and providing guidance for surgeons, engineers, and researchers involved in the research and development of this area.

Systematic review of techniques used to validate the registration of augmented-reality images using a head-mounted device to navigate surgery

Augmented-reality (AR) head-mounted devices (HMD) allow the wearer to have digital images superposed on to their field of vision. They are being used to superpose annotations on to the surgical field akin to a navigation system. This review examines published validation studies on HMD-AR systems, their reported protocols, and outcomes. The aim was to establish commonalities and an acceptable registration outcome. Multiple databases were systematically searched for relevant articles between January 2015 and January 2021. Studies that examined the registration of AR content using a HMD to guide surgery were eligible for inclusion. The country of origin, year of publication, medical specialty, HMD device, software, and method of registration, were recorded. A meta-analysis of the mean registration error was conducted. A total of 4784 papers were identified, of which 23 met the inclusion criteria. They included studies using HoloLens (Microsoft) (n = 22) and nVisor ST60 (NVIS Inc) (n = 1). Sixty-six per cent of studies were in hard tissue specialties. Eleven studies reported registration errors using pattern markers (mean (SD) 2.6 (1.8) mm), and four reported registration errors using surface markers (mean (SD) 3.8 (3.7) mm). Three studies reported registration errors using manual alignment (mean (SD) 2.2 (1.3) mm). The majority of studies in this review used in-house software with a variety of registration methods and reported errors. The mean registration error calculated in this study can be considered as a minimum acceptable standard. It should be taken into consideration when procedural applications are selected.

The application of augmented reality-based navigation for accurate target acquisition of deep brain sites: advances in neurosurgical guidance

OBJECTIVE

The objective of this study is to quantify the navigational accuracy of an advanced augmented reality (AR)-based guidance system for neurological surgery, biopsy, and/or other minimally invasive neurological surgical procedures.

METHODS

Five burr holes were drilled through a plastic cranium, and 5 optical fiducials (AprilTags) printed with CT-visible ink were placed on the frontal, temporal, and parietal bones of a human skull model. Three 0.5-mm-diameter targets were mounted in the interior of the skull on nylon posts near the level of the tentorium cerebelli and the pituitary fossa. The skull was filled with ballistic gelatin to simulate brain tissue. A CT scan was taken and virtual needle tracts were annotated on the preoperative 3D workstation for the combination of 3 targets and 5 access holes (15 target tracts). The resulting annotated study was uploaded to and launched by VisAR software operating on the HoloLens 2 holographic visor by viewing an encrypted, printed QR code assigned to the study by the preoperative workstation. The DICOM images were converted to 3D holograms and registered to the skull by alignment of the holographic fiducials with the AprilTags attached to the skull. Five volunteers, familiar with the VisAR, used the software/visor combination to navigate an 18-gauge needle/trocar through the series of burr holes to the target, resulting in 70 data points (15 for 4 users and 10 for 1 user). After each attempt the needle was left in the skull, supported by the ballistic gelatin, and a high-resolution CT was taken. Radial error and angle of error were determined using vector coordinates. Summary statistics were calculated individually and collectively.

RESULTS

The combined angle of error of was 2.30° ± 1.28°. The mean radial error for users was 3.62 ± 1.71 mm. The mean target depth was 85.41 mm.

CONCLUSIONS

The mean radial error and angle of error with the associated variance measures demonstrates that VisAR navigation may have utility for guiding a small needle to neural lesions, or targets within an accuracy of 3.62 mm. These values are sufficiently accurate for the navigation of many neurological procedures such as ventriculostomy.

Augmented Reality Spine Surgery Navigation: Increasing Pedicle Screw Insertion Accuracy for Both Open and Minimally Invasive Spine Surgeries

STUDY DESIGN

Collectively, seven cadavers were instrumented with 124 thoracolumbar pedicle screws using VisAR augmented reality/guidance. Sixty-five screws were inserted into four donors using open dissection spine surgery. Fifty-nine screws were positioned in three donors with a minimally invasive spine surgery (MISS) procedure. For both open and MISS, VisAR was used exclusively for pedicle screw navigation.

OBJECTIVE

The objective of this study was to determine the accuracy of pedicle screw placement using VisAR for open spine and MISS procedures.

SUMMARY OF BACKGROUND DATA

Pedicle screw placement can be challenging depending on anatomical location and a surgeon's experience. AR may minimize fluoroscopy use and speed screw insertion.

METHODS

Prior to computed tomography (CT) a series of four image visible April Tag optical fiducials were attached to the backs' of the donors. Resulting images were used preoperatively for planned virtual pedicle screw pathways including entry point, trajectory, and depth. The study link was encrypted on a quick response (QR) code, printed, and viewed in the operating room (OR) by the surgeon using VisAR (HoloLens 2 headset). Viewing the code wirelessly uploads and launches the study, converting the DICOM data to holographic images which register to the fiducials on the donor's back. The annotated pathways for each pedicle were called up by voice command and the surgeon positioned each screw by aligning with the virtual guidance hologram.

RESULTS

Overall, 124 pedicle screws were inserted with VisAR navigation with 96% accuracy (Gertzbein-Robbins grades A and B). The combined angle of error was 2.4° and the distance error was 1.9 mm.

CONCLUSION

Augmented reality is a highly accurate, emerging technology for navigating both open and minimally invasive spine surgery techniques with off-the-shelf headset hardware.

LEVEL OF EVIDENCE

N/A.

Spine Surgery Assisted by Augmented Reality: Where Have We Been?

This present systematic review examines spine surgery literature supporting augmented reality (AR) technology and summarizes its current status in spinal surgery technology. Database search strategies were retrieved from PubMed, Web of Science, Cochrane Library, Embase, from the earliest records to April 1, 2021. Our review briefly examines the history of AR, and enumerates different device application workflows in a variety of spinal surgeries. We also sort out the pros and cons of current mainstream AR devices and the latest updates. A total of 45 articles are included in our review. The most prevalent surgical applications included are the augmented reality surgical navigation system and head-mounted display. The most popular application of AR is pedicle screw instrumentation in spine surgery, and the primary responsible surgical levels are thoracic and lumbar. AR guidance systems show high potential value in practical clinical applications for the spine. The overall number of cases in AR-related studies is still rare compared to traditional surgical-assisted techniques. These lack long-term clinical efficacy and robust surgical-related statistical data. Changing healthcare laws as well as the increasing prevalence of spinal surgery are generating critical data that determines the value of AR technology.

Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review

This article presents a systematic review of optical see-through head mounted display (OST-HMD) usage in augmented reality (AR) surgery applications from 2013 to 2020. Articles were categorised by: OST-HMD device, surgical speciality, surgical application context, visualisation content, experimental design and evaluation, accuracy and human factors of human-computer interaction. 91 articles fulfilled all inclusion criteria. Some clear trends emerge. The Microsoft HoloLens increasingly dominates the field, with orthopaedic surgery being the most popular application (28.6%). By far the most common surgical context is surgical guidance (n=58) and segmented preoperative models dominate visualisation (n=40). Experiments mainly involve phantoms (n=43) or system setup (n=21), with patient case studies ranking third (n=19), reflecting the comparative infancy of the field. Experiments cover issues from registration to perception with very different accuracy results. Human factors emerge as significant to OST-HMD utility. Some factors are addressed by the systems proposed, such as attention shift away from the surgical site and mental mapping of 2D images to 3D patient anatomy. Other persistent human factors remain or are caused by OST-HMD solutions, including ease of use, comfort and spatial perception issues. The significant upward trend in published articles is clear, but such devices are not yet established in the operating room and clinical studies showing benefit are lacking. A focused effort addressing technical registration and perceptual factors in the lab coupled with design that incorporates human factors considerations to solve clear clinical problems should ensure that the significant current research efforts will succeed.

Utility of Augmented Reality and Virtual Reality in Spine Surgery: A Systematic Review of the Literature

BACKGROUND

Augmented reality, virtual reality, and mixed reality (AR, VR, MR) are emerging technologies that are starting to be translated into clinical practice. There is limited data available about these tools being used in live surgery of the spine. The objective of this paper was to systematically collect, analyze, and interpret the existing data regarding AR, VR, and MR use in spine surgery on live people.

METHODS

A systematic review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (PRISMA). PubMed, PubMed Central, Cochrane Reviews, and Embase databases were searched. Combinations and variations of "augmented reality", "virtual reality", and "spine surgery" in both AND and OR configurations were used to gather relevant articles. References of included articles from the systematic review were also screened for possible inclusion as a part of manual review. Included studies were full text publications written in English that had any spine surgery on live persons with the use of virtual or augmented reality.

RESULTS

A total of 1566 unique articles were found, and fifteen full-text publications met criteria for this study. The total number of patients from all studies was 241 with a weighted average age of 50.37. Surgical procedures utilizing AR, VR, and/or MR were diverse and spanned from simple discectomies to intradural spinal tumor resection. All patients experienced improvement in their symptoms from clinical presentation. The highest complication rate mentioned in the articles was 6.1% and was for suboptimal pedicle screw placement. There were no complications that led to clinical sequelae.

CONCLUSIONS

The systematically collected, analyzed, and interpreted data of existing peer-reviewed full text articles showed favorable metrics regarding surgical efficacy, pedicle screw target accuracy, radiation exposure, clinical outcome, and disability and pain in patients with spinal pathology treated with the help of AR, VR, and/or MR.

Application of Artificial Intelligence in Medicine: An Overview

Artificial intelligence (AI) is a new technical discipline that uses computer technology to research and develop the theory, method, technique, and application system for the simulation, extension, and expansion of human intelligence. With the assistance of new AI technology, the traditional medical environment has changed a lot. For example, a patient's diagnosis based on radiological, pathological, endoscopic, ultrasonographic, and biochemical examinations has been effectively promoted with a higher accuracy and a lower human workload. The medical treatments during the perioperative period, including the preoperative preparation, surgical period, and postoperative recovery period, have been significantly enhanced with better surgical effects. In addition, AI technology has also played a crucial role in medical drug production, medical management, and medical education, taking them into a new direction. The purpose of this review is to introduce the application of AI in medicine and to provide an outlook of future trends.

Registration Techniques for Clinical Applications of Three-Dimensional Augmented Reality Devices

Many clinical procedures would benefit from direct and intuitive real-time visualization of anatomy, surgical plans, or other information crucial to the procedure. Three-dimensional augmented reality (3D-AR) is an emerging technology that has the potential to assist physicians with spatial reasoning during clinical interventions. The most intriguing applications of 3D-AR involve visualizations of anatomy or surgical plans that appear directly on the patient. However, commercially available 3D-AR devices have spatial localization errors that are too large for many clinical procedures. For this reason, a variety of approaches for improving 3D-AR registration accuracy have been explored. The focus of this review is on the methods, accuracy, and clinical applications of registering 3D-AR devices with the clinical environment. The works cited represent a variety of approaches for registering holograms to patients, including manual registration, computer vision-based registration, and registrations that incorporate external tracking systems. Evaluations of user accuracy when performing clinically relevant tasks suggest that accuracies of approximately 2 mm are feasible. 3D-AR device limitations due to the vergence-accommodation conflict or other factors attributable to the headset hardware add on the order of 1.5 mm of error compared to conventional guidance. Continued improvements to 3D-AR hardware will decrease these sources of error.

The Application of Virtual Reality in Cervical Spinal Surgery: A Review

In recent years, clinicians have used virtual reality (VR) to simulate real-world environments for medical purposes. The use of VR systems in the field of cervical spine surgery can lead to effective surgical training programs without causing harm to patients. Moreover, both imaging and VR can be used before surgery to assist preoperative surgical planning. VR devices have a variety of built-in motion sensors, therefore kinematic data can be recorded while users are wearing VR devices and performing some actions for the evaluation of cervical spine activity and exercise ability. Therapists have also applied VR to cervical spine rehabilitation and showed good results. At present, the application of VR systems in cervical spine surgery has great potential, but current research is insufficient. Here, we review the latest advancements in VR technology used in cervical spine surgery and discuss potential directions for future work.