Augmented Reality Based Navigation for Computer Assisted Hip Resurfacing: A Proof of Concept Study

Augmented reality guidance improves accuracy of orthopedic drilling procedures

In several orthopedic procedures, the accurate use of surgical power tools is critical to avoid damage to surrounding tissues. As such, various guidance techniques and safety measures were developed. Augmented reality (AR) guidance shows promise but requires validation. We evaluated a new approach using an inside-out infrared tracking solution for the HoloLens to compensate for its limited tracking performance. Eighteen participants with varying levels of experience (student, trainee, expert) each drilled twelve trajectories (six perpendicular, six oblique) in equidimensional wooden logs. Three different techniques were evaluated: freehand drilling; proprioception-guided drilling towards the contralateral index finger; and AR-guided drilling using a tracked drill and a virtual overlay of the log with predefined guidance vectors. The angular errors between planned and performed trajectories were compared using a mixed-design ANOVA. The results demonstrated that guidance technique (p < 0.001) and drilling direction (p < 0.001) significantly affected drilling accuracy, while experience (p = 0.75) did not. AR outperformed both other techniques, particularly for oblique trajectories (p < 0.001). For perpendicular trajectories, it only outperformed proprioception guidance (p = 0.04). Target plots revealed an important scatter perpendicular to the longitudinal axis of the log during freehand and proprioception-guided drilling, especially for oblique trajectories. This inaccuracy disappeared during AR-guided drilling. As such, we were able to conclude that AR guidance using inside-out infrared tracking reduced angular uncertainty during directional drilling, resulting in improved drilling accuracy. This improvement was particularly noticeable for complex trajectories and angles. The benefits of AR guidance were observed across all experience levels, highlighting its potential for orthopedic applications. We believe this study opens the way for the methodical evaluation of AR guidance in specific orthopedic use cases.

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the performance of the task. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers without the need to integrate any additional electronics into the HMD. The proposed framework can simultaneously track multiple tools without having previous knowledge of their geometry and only requires establishing a local network between the headset and a workstation. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09±0.06 mm on lateral translation, 0.42 ±0.32 mm on longitudinal translation and 0.80 ±0.39° for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, seven surgeons were provided with visual navigation and asked to perform 24 injections using the proposed framework. A second study with ten participants served to investigate the capabilities of the framework in the context of more general scenarios. Results from these studies provided comparable accuracy to those reported in the literature for AR-based navigation procedures.

Feasibility and Usability of Augmented Reality Technology in the Orthopaedic Operating Room

PURPOSE OF REVIEW

Augmented reality (AR) has gained popularity in various sectors, including gaming, entertainment, and healthcare. The desire for improved surgical navigation within orthopaedic surgery has led to the evaluation of the feasibility and usability of AR in the operating room (OR). However, the safe and effective use of AR technology in the OR necessitates a proper understanding of its capabilities and limitations. This review aims to describe the fundamental elements of AR, highlight limitations for use within the field of orthopaedic surgery, and discuss potential areas for development.

RECENT FINDINGS

To date, studies have demonstrated evidence that AR technology can be used to enhance navigation and performance in orthopaedic procedures. General hardware and software limitations of the technology include the registration process, ergonomics, and battery life. Other limitations are related to the human response factors such as inattentional blindness, which may lead to the inability to see complications within the surgical field. Furthermore, the prolonged use of AR can cause eye strain and headache due to phenomena such as the vergence-convergence conflict. AR technology may prove to be a better alternative to current orthopaedic surgery navigation systems. However, the current limitations should be mitigated to further improve the feasibility and usability of AR in the OR setting. It is important for both non-clinicians and clinicians to work in conjunction to guide the development of future iterations of AR technology and its implementation into the OR workflow.

Phantom study of augmented reality framework to assist epicardial punctures

Purpose

The objective of this study is to evaluate the accuracy of an augmented reality (AR) system in improving guidance, accuracy, and visualization during the subxiphoidal approach for epicardial ablation.

Approach

An AR application was developed to project real-time needle trajectories and patient-specific 3D organs using the Hololens 2. Additionally, needle tracking was implemented to offer real-time feedback to the operator, facilitating needle navigation. The AR application was evaluated through three different experiments: examining overlay accuracy, assessing puncture accuracy, and performing pre-clinical evaluations on a phantom.

Results

The results of the overlay accuracy assessment for the AR system yielded 2.36 ± 2.04    mm . Additionally, the puncture accuracy utilizing the AR system yielded 1.02 ± 2.41    mm . During the pre-clinical evaluation on the phantom, needle puncture with AR guidance showed 7.43 ± 2.73    mm , whereas needle puncture without AR guidance showed 22.62 ± 9.37    mm .

Conclusions

Overall, the AR platform has the potential to enhance the accuracy of percutaneous epicardial access for mapping and ablation of cardiac arrhythmias, thereby reducing complications and improving patient outcomes. The significance of this study lies in the potential of AR guidance to enhance the accuracy and safety of percutaneous epicardial access.

Exposure to Extended Reality and Artificial Intelligence-Based Manifestations: A Primer on the Future of Hip and Knee Arthroplasty

BACKGROUND

Software-infused services, from robot-assisted and wearable technologies to artificial intelligence (AI)-laden analytics, continue to augment clinical orthopaedics - namely hip and knee arthroplasty. Extended reality (XR) tools, which encompass augmented reality, virtual reality, and mixed reality technology, represent a new frontier for expanding surgical horizons to maximize technical education, expertise, and execution. The purpose of this review is to critically detail and evaluate the recent developments surrounding XR in the field of hip and knee arthroplasty and to address potential future applications as they relate to AI.

METHODS

In this narrative review surrounding XR, we discuss (1) definitions, (2) techniques, (3) studies, (4) current applications, and (5) future directions. We highlight XR subsets (augmented reality, virtual reality, and mixed reality) as they relate to AI in the increasingly digitized ecosystem within hip and knee arthroplasty.

RESULTS

A narrative review of the XR orthopaedic ecosystem with respect to XR developments is summarized with specific emphasis on hip and knee arthroplasty. The XR as a tool for education, preoperative planning, and surgical execution is discussed with future applications dependent upon AI to potentially obviate the need for robotic assistance and preoperative advanced imaging without sacrificing accuracy.

CONCLUSION

In a field where exposure is critical to clinical success, XR represents a novel stand-alone software-infused service that optimizes technical education, execution, and expertise but necessitates integration with AI and previously validated software solutions to offer opportunities that improve surgical precision with or without the use of robotics and computed tomography-based imaging.

Factors affecting augmented reality head-mounted device performance in real OR

PURPOSE

Over the last years, interest and efforts to implement augmented reality (AR) in orthopedic surgery through head-mounted devices (HMD) have increased. However, the majority of experiments were preclinical and within a controlled laboratory environment. The operating room (OR) is a more challenging environment with various confounding factors potentially affecting the performance of an AR-HMD. The aim of this study was to assess the performance of an AR-HMD in a real-life OR setting.

METHODS

An established AR application using the HoloLens 2 HMD was tested in an OR and in a laboratory by two users. The accuracy of the hologram overlay, the time to complete the trial, the number of rejected registration attempts, the delay in live overlay of the hologram, and the number of completely failed runs were recorded. Further, different OR setting parameters (light condition, setting up partitions, movement of personnel, and anchor placement) were modified and compared.

RESULTS

Time for full registration was higher with 48 s (IQR 24 s) in the OR versus 33 s (IQR 10 s) in the laboratory setting (p < 0.001). The other investigated parameters didn't differ significantly if an optimal OR setting was used. Within the OR, the strongest influence on performance of the AR-HMD was different light conditions with direct light illumination on the situs being the least favorable.

CONCLUSION

AR-HMDs are affected by different OR setups. Standardization measures for better AR-HMD performance include avoiding direct light illumination on the situs, setting up partitions, and minimizing the movement of personnel.

The Accuracy and Absolute Reliability of a Knee Surgery Assistance System Based on ArUco-Type Sensors

Recent advances allow the use of Augmented Reality (AR) for many medical procedures. AR via optical navigators to aid various knee surgery techniques (e.g., femoral and tibial osteotomies, ligament reconstructions or menisci transplants) is becoming increasingly frequent. Accuracy in these procedures is essential, but evaluations of this technology still need to be made. Our study aimed to evaluate the system's accuracy using an in vitro protocol. We hypothesised that the system's accuracy was equal to or less than 1 mm and 1° for distance and angular measurements, respectively. Our research was an in vitro laboratory with a 316 L steel model. Absolute reliability was assessed according to the Hopkins criteria by seven independent evaluators. Each observer measured the thirty palpation points and the trademarks to acquire direct angular measurements on three occasions separated by at least two weeks. The system's accuracy in assessing distances had a mean error of 1.203 mm and an uncertainty of 2.062, and for the angular values, a mean error of 0.778° and an uncertainty of 1.438. The intraclass correlation coefficient was for all intra-observer and inter-observers, almost perfect or perfect. The mean error for the distance's determination was statistically larger than 1 mm (1.203 mm) but with a trivial effect size. The mean error assessing angular values was statistically less than 1°. Our results are similar to those published by other authors in accuracy analyses of AR systems.

Acetabular Revision with McMinn Cup: Development and Application of a Patient-Specific Targeting Device

BACKGROUND

Surgeries of severe periacetabular bone defects (Paprosky ≥ 2B) are a major challenge in current practice. Although solutions are available for this serious clinical problem, they all have their disadvantages as well as their advantages. An alternative method of reconstructing such extensive defects was the use of a cup with a stem to solve these revision situations. As the instrumentation offered is typically designed for scenarios where a significant bone defect is not present, our unique technique has been developed for implantation in cases where reference points are missing. Our hypothesis was that a targeting device designed based on the CT scan of a patient's pelvis could facilitate the safe insertion of the guiding wire.

METHODS

Briefly, our surgical solution consists of a two-step operation. If periacetabular bone loss was found to be more significant during revision surgery, all implants were removed, and two titanium marker screws in the anterior iliac crest were percutaneously inserted. Next, by applying the metal artifact removal (MAR) algorithm, a CT scan of the pelvis was performed. Based on that, the dimensions and positioning of the cup to be inserted were determined, and a patient-specific 3D printed targeting device made of biocompatible material was created to safely insert the guidewire, which is essential to the implantation process.

RESULTS

In this study, medical, engineering, and technical tasks related to the design, the surgical technique, and experiences from 17 surgical cases between February 2018 and July 2021 are reported. There were no surgical complications in any cases. The implant had to be removed due to septic reasons (independently from the technique) in a single case, consistent with the septic statistics for this type of surgery. There was not any perforation of the linea terminalis of the pelvis due to the guiding method. The wound healing of patients was uneventful, and the implant was fixed securely. Following rehabilitation, the joints were able to bear weight again. After one to four years of follow-up, the patient satisfaction level was high, and the gait function of the patients improved a lot in all cases.

CONCLUSIONS

Our results show that CT-based virtual surgical planning and, based on it, the use of a patient-specific 3D printed aiming device is a reliable method for major hip surgeries with significant bone loss. This technique has also made it possible to perform these operations with minimal X-ray exposure.

Augmented reality and shoulder replacement: a state-of-the-art review article

Since its implementation, the rates of failure of total shoulder arthroplasty which may be due to malpositioning pushed to improve this surgery by creating new techniques and tools to help perioperatively. Augmented reality, a newly used tool in orthopedic surgery can help bypass this problem and reduce the rates of failure faced in shoulder replacement surgeries. Although this technology has revolutionized orthopedic surgery and helped improve the accuracy in shoulder prosthesis components positioning, it still has some limitations such as inaccurate over-imposition that should be addressed before it becomes of standard usage.

Nail it! vision-based drift correction for accurate mixed reality surgical guidance

PURPOSE

Mixed reality-guided surgery through head-mounted displays (HMDs) is gaining interest among surgeons. However, precise tracking of HMDs relative to the surgical environment is crucial for successful outcomes. Without fiducial markers, spatial tracking of the HMD suffers from millimeter- to centimeter-scale drift, resulting in misaligned visualization of registered overlays. Methods and workflows capable of automatically correcting for drift after patient registration are essential to assuring accurate execution of surgical plans.

METHODS

We present a mixed reality surgical navigation workflow that continuously corrects for drift after patient registration using only image-based methods. We demonstrate its feasibility and capabilities using the Microsoft HoloLens on glenoid pin placement in total shoulder arthroplasty. A phantom study was conducted involving five users with each user placing pins on six glenoids of different deformity, followed by a cadaver study by an attending surgeon.

RESULTS

In both studies, all users were satisfied with the registration overlay before drilling the pin. Postoperative CT scans showed 1.5 mm error in entry point deviation and 2.4[Formula: see text] error in pin orientation on average in the phantom study and 2.5 mm and 1.5[Formula: see text] in the cadaver study. A trained user takes around 90 s to complete the workflow. Our method also outperformed HoloLens native tracking in drift correction.

CONCLUSION

Our findings suggest that image-based drift correction can provide mixed reality environments precisely aligned with patient anatomy, enabling pin placement with consistently high accuracy. These techniques constitute a next step toward purely image-based mixed reality surgical guidance, without requiring patient markers or external tracking hardware.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Visualization, registration and tracking techniques for augmented reality guided surgery: a review

Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories ofin situvisualization and nonin situvisualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.

[Effectiveness of three-dimensional visible technique without fluoroscopy versus two-dimensional fluoroscopy in reduction of unstable pelvic fractures]

Objective

To compare the reduction qualities of three-dimensional visible technique without fluoroscopy and two-dimensional fluoroscopy for unstable pelvic fractures during operations.

Methods

The clinical data of 40 patients with unstable pelvic fractures, who met the selection criteria in three clinical centers between June 2021 and September 2022, were retrospectively analyzed. According to the reduction methods, the patients were divided into two groups. Twenty patients in trial group were treated with unlocking closed reduction system combined with three-dimensional visible technique without fluoroscopy; 20 patients in control group with unlocking closed reduction system under two-dimensional fluoroscopy. There was no significant difference in the gender, age, injury mechanism, Tile type of fracture, Injury Severity Score (ISS), and the time between injury to operation between the two groups ( P>0.05). The qualities of fracture reduction according to the Matta criteria, operative time, intraoperative blood loss, fracture reduction time, times of fluoroscopy, and System Usability Scale (SUS) score were recorded and compared.

Results

All operations were successfully completed in both groups. According to the Matta criteria, the qualities of fracture reduction were rated as excellent in 19 patients (95%) in trial group, which was better than that in the control group (13 cases, 65%), with a significant difference ( χ ²=3.906, P=0.048). The operative time and intraoperative blood loss had no significant differences between the two groups ( P>0.05). The fracture reduction time and times of fluoroscopy were significantly less in trial group than in control group ( P<0.05), and SUS score in trial group was significantly higher in trial group than in control group ( P<0.05).

Conclusion

Compared to using unlocking closed reduction system under two-dimensional fluoroscopy, three-dimensional visible technique without fluoroscopy can significantly improve the reduction quality of unstable pelvic fractures without prolonging the operative time, and is valuable to reduce iatrogenic radiation exposure for patients and medical workers.

Hemisphere Tabulation Method: An Ingenious Approach for Pose Evaluation of Instruments Using the Electromagnetic-Based Stereo Imaging Method

Drilling of a bone surface often occurs in clinical orthopaedic surgery. The position and orientation of the instrument are the most important factors in this process. Theoretically, some mechanical components may assist in orienting an instrument to certain bone shapes, such as the knee joint and caput femoris. However, the mechanical assisting component does not seem to work in some confined spaces where the bone shape is a free-form surface. In this paper, we propose an ingenious hemisphere tabulation method (HTM) for assessing the pose accuracy of an instrument. The acquisition and assessment of HTM is conducted based on an electromagnetic-based stereo imaging method using a custom-made optical measurement unit, and the operation steps of HTM are described in detail. Experimental results based on 50 tests show that the HTM can identify ideal poses and the evaluated pose of an instrument location on a hemisphere model. The mean error of pose localisation is 7.24 deg, with a range of 1.35 to 15.84 and a standard of 3.66 deg, which is more accurate than our previous method.

Next step trauma and orthopaedic surgery: integration of augmented reality for reduction and nail implantation of tibial fractures

INTRODUCTION

There is a tremendous scope of hardware and software development going on in augmented reality (AR), also in trauma and orthopaedic surgery. However, there are only a few systems available for intra-operative 3D imaging and guidance, most of them rely on peri- and intra-operative X-ray imaging. Especially in complex situations such as pelvic surgery or multifragmentary multilevel fractures, intra-operative 3D imaging and implant tracking systems have proven to be of great advantage for the outcome of the surgery and can help reduce X-ray exposure, at least for the surgical team (Ochs et al. in Injury 41:1297 1305, 2010). Yet, the current systems do not provide the ability to have a dynamic live view from the perspective of the surgeon. Our study describes a prototype AR-based system for live tracking which does not rely on X-rays.

MATERIALS AND METHODS

A protype live-view intra-operative guidance system using an AR head-mounted device (HMD) was developed and tested on the implantation of a medullary nail in a tibia fracture model. Software algorithms that allow live view and tracking of the implant, fracture fragments and soft tissue without the intra-operative use of X-rays were derived.

RESULTS

The implantation of a medullar tibia nail is possible while only relying on AR-guidance and live view without the intra-operative use of X-rays.

CONCLUSIONS

The current paper describes a feasibility study with a prototype of an intra-operative dynamic live tracking and imaging system that does not require intra-operative use of X-rays and dynamically adjust to the perspective of the surgeons due to an AR HMD. To our knowledge, the current literature does not describe any similar systems. This could be the next step in surgical imaging and education and a promising way to improve patient care.

Clinical applications of augmented reality in orthopaedic surgery: a comprehensive narrative review

PURPOSE

The development of augmented reality (AR) technology allows orthopaedic surgeons to incorporate and visualize surgical data, assisting the execution of both routine and complex surgical operations. Uniquely, AR technology allows a surgeon to view the surgical field and superimpose peri-operative imaging, anatomical landmarks, navigation guidance, and more, all in one view without the need for conjugate gaze between multiple screens. The aim of this literature review was to introduce the fundamental requirements for an augmented reality system and to assess the current applications, outcomes, and potential limitations to this technology.

METHODS

A literature search was performed using MEDLINE and Embase databases, by two independent reviewers, who then collaboratively synthesized and collated the results of the literature search into a narrative review focused on the applications of augmented reality in major orthopaedic sub-specialties.

RESULTS

Current technology requires that pre-operative patient data be acquired, and AR-compatible models constructed. Intra-operatively, to produce manipulatable virtual images into the user's view in real time, four major components are required including a camera, computer image processing technology, tracking tools, and an output screen. The user is provided with a heads-up display, which is a transparent display, enabling the user to look at both their natural view and the computer-generated images. Currently, high-quality evidence for clinical implementation of AR technology in the orthopaedic surgery operating room is lacking; however, growing in vitro literature highlights a multitude of potential applications, including increasing operative accuracy, improved biomechanical angular and alignment parameters, and potentially reduced operative time.

CONCLUSION

While the application of AR systems in surgery is currently in its infancy, we anticipate rapid and widespread implementation of this technology in various orthopaedic sub-specialties.

Systematic review of techniques used to validate the registration of augmented-reality images using a head-mounted device to navigate surgery

Augmented-reality (AR) head-mounted devices (HMD) allow the wearer to have digital images superposed on to their field of vision. They are being used to superpose annotations on to the surgical field akin to a navigation system. This review examines published validation studies on HMD-AR systems, their reported protocols, and outcomes. The aim was to establish commonalities and an acceptable registration outcome. Multiple databases were systematically searched for relevant articles between January 2015 and January 2021. Studies that examined the registration of AR content using a HMD to guide surgery were eligible for inclusion. The country of origin, year of publication, medical specialty, HMD device, software, and method of registration, were recorded. A meta-analysis of the mean registration error was conducted. A total of 4784 papers were identified, of which 23 met the inclusion criteria. They included studies using HoloLens (Microsoft) (n = 22) and nVisor ST60 (NVIS Inc) (n = 1). Sixty-six per cent of studies were in hard tissue specialties. Eleven studies reported registration errors using pattern markers (mean (SD) 2.6 (1.8) mm), and four reported registration errors using surface markers (mean (SD) 3.8 (3.7) mm). Three studies reported registration errors using manual alignment (mean (SD) 2.2 (1.3) mm). The majority of studies in this review used in-house software with a variety of registration methods and reported errors. The mean registration error calculated in this study can be considered as a minimum acceptable standard. It should be taken into consideration when procedural applications are selected.

Augmented reality calibration using feature triangulation iteration-based registration for surgical navigation

BACKGROUND

Marker-based augmented reality (AR) calibration methods for surgical navigation often require a second computed tomography scan of the patient, and their clinical application is limited due to high manufacturing costs and low accuracy.

METHODS

This work introduces a novel type of AR calibration framework that combines a Microsoft HoloLens device with a single camera registration module for surgical navigation. A camera is used to gather multi-view images of a patient for reconstruction in this framework. A shape feature matching-based search method is proposed to adjust the size of the reconstructed model. The double clustering-based 3D point cloud segmentation method and 3D line segment detection method are also proposed to extract the corner points of the image marker. The corner points are the registration data of the image marker. A feature triangulation iteration-based registration method is proposed to quickly and accurately calibrate the pose relationship between the image marker and the patient in the virtual and real space. The patient model after registration is wirelessly transmitted to the HoloLens device to display the AR scene.

RESULTS

The proposed approach was used to conduct accuracy verification experiments on the phantoms and volunteers, which were compared with six advanced AR calibration methods. The proposed method obtained average fusion errors of 0.70 ± 0.16 and 0.91 ± 0.13 mm in phantom and volunteer experiments, respectively. The fusion accuracy of the proposed method is the highest among all comparison methods. A volunteer liver puncture clinical simulation experiment was also conducted to show the clinical feasibility.

CONCLUSIONS

Our experiments proved the effectiveness of the proposed AR calibration method, and revealed a considerable potential for improving surgical performance.

USE OF COMPUTER NAVIGATION IN TOTAL HIP ARTHROPLASTY (LITERATURE REVIEW)

OBJECTIVE

The aim: Analyze the accuracy and ease of use of various computer navigations in total hip arthroplasty.

PATIENTS AND METHODS

Materials and methods: Data from about 50 literature sources for the last two decades have been analysed.

CONCLUSION

Conclusions: Analyzing the accuracy and ease of use of various computer navigations in total hip arthroplasty, we offer two the most promising for further study and improvement systems: a semi-active navigation system and augmented reality system in total hip arthroplasty.

Augmented Reality in Orthopedic Surgery and Its Application in Total Joint Arthroplasty: A Systematic Review

The development of augmented reality (AR) and its application in total joint arthroplasty aims at improving the accuracy and precision in implant components’ positioning, hopefully leading to increased outcomes and survivorship. However, this field is far from being thoroughly explored. We therefore performed a systematic review of the literature in order to examine the application, the results, and the different AR systems available in TJA. A systematic review of the literature according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines was performed. A comprehensive search of PubMed, MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews was conducted for English articles on the application of augmented reality in total joint arthroplasty using various combinations of keywords since the inception of the database to 31 March 2022. Accuracy was intended as the mean error from the targeted positioning angle and compared as mean values and standard deviations. In all, 14 articles met the inclusion criteria. Among them, four studies reported on the application of AR in total knee arthroplasty, six studies on total hip arthroplasty, three studies reported on reverse shoulder arthroplasty, and one study on total elbow arthroplasty. Nine of the included studies were preclinical (sawbones or cadaveric), while five of them reported results of AR’s clinical application. The main common feature was the high accuracy and precision when implant positioning was compared with preoperative targeted angles with errors ≤2 mm and/or ≤2°. Despite the promising results in terms of increased accuracy and precision, this technology is far from being widely adopted in daily clinical practice. However, the recent exponential growth in machine learning techniques and technologies may eventually lead to the resolution of the ongoing limitations including depth perception and their high complexity, favorably encouraging the widespread usage of AR systems.

Augmented Reality in Arthroplasty: An Overview of Clinical Applications, Benefits, and Limitations

Augmented reality (AR) is a natural extension of computer-assisted surgery whereby a computer-generated image is superimposed on the surgeon's field of vision to assist in the planning and execution of the procedure. This emerging technology shows great potential in the field of arthroplasty, improving efficiency, limb alignment, and implant position. AR has shown the capacity to build on computer navigation systems while providing more elaborate information in a streamlined workflow to the user. This review investigates the current uses of AR in the field of arthroplasty and discusses outcomes, limitations, and potential future directions.

Artificial intelligence in knee arthroplasty: current concept of the available clinical applications

Background

Artificial intelligence (AI) is defined as the study of algorithms that allow machines to reason and perform cognitive functions such as problem-solving, objects, images, word recognition, and decision-making. This study aimed to review the published articles and the comprehensive clinical relevance of AI-based tools used before, during, and after knee arthroplasty.

Methods

The search was conducted through PubMed, EMBASE, and MEDLINE databases from 2000 to 2021 using the 2009 Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocol (PRISMA).

Results

A total of 731 potential articles were reviewed, and 132 were included based on the inclusion criteria and exclusion criteria. Some steps of the knee arthroplasty procedure were assisted and improved by using AI-based tools. Before surgery, machine learning was used to aid surgeons in optimizing decision-making. During surgery, the robotic-assisted systems improved the accuracy of knee alignment, implant positioning, and ligamentous balance. After surgery, remote patient monitoring platforms helped to capture patients’ functional data.

Conclusion

In knee arthroplasty, the AI-based tools improve the decision-making process, surgical planning, accuracy, and repeatability of surgical procedures.

Review and Future/Potential Application of Mixed Reality Technology in Orthopaedic Oncology

In orthopaedic oncology, surgical planning and intraoperative execution errors may result in positive tumor resection margins that increase the risk of local recurrence and adversely affect patients’ survival. Computer navigation and 3D-printed resection guides have been reported to address surgical inaccuracy by replicating the surgical plans in complex cases. However, limitations include surgeons’ attention shift from the operative field to view the navigation monitor and expensive navigation facilities in computer navigation surgery. Practical concerns are lacking real-time visual feedback of preoperative images and the lead-time in manufacturing 3D-printed objects. Mixed Reality (MR) is a technology of merging real and virtual worlds to produce new environments with enhanced visualizations, where physical and digital objects coexist and allow users to interact with both in real-time. The unique MR features of enhanced medical images visualization and interaction with holograms allow surgeons real-time and on-demand medical information and remote assistance in their immediate working environment. Early application of MR technology has been reported in surgical procedures. Its role is unclear in orthopaedic oncology. This review aims to provide orthopaedic tumor surgeons with up-to-date knowledge of the emerging MR technology. The paper presents its essential features and clinical workflow, reviews the current literature and potential clinical applications, and discusses the limitations and future development in orthopaedic oncology. The emerging MR technology adds a new dimension to digital assistive tools with a more accessible and less costly alternative in orthopaedic oncology. The MR head-mounted display and hand-free control may achieve clinical point-of-care inside or outside the operating room and improve service efficiency and patient safety. However, lacking an accurate hologram-to-patient matching, an MR platform dedicated to orthopaedic oncology, and clinical results may hinder its wide adoption. Industry-academic partnerships are essential to advance the technology with its clinical role determined through future clinical studies.

Point your SmartPhone at the code above. If you have a QR code reader the video abstract will appear. Or use:

https://youtu.be/t4hl_Anh_kM

Augmented Reality: Mapping Methods and Tools for Enhancing the Human Role in Healthcare HMI

Background: Augmented Reality (AR) represents an innovative technology to improve data visualization and strengthen the human perception. Among Human–Machine Interaction (HMI), medicine can benefit most from the adoption of these digital technologies. In this perspective, the literature on orthopedic surgery techniques based on AR was evaluated, focusing on identifying the limitations and challenges of AR-based healthcare applications, to support the research and the development of further studies. Methods: Studies published from January 2018 to December 2021 were analyzed after a comprehensive search on PubMed, Google Scholar, Scopus, IEEE Xplore, Science Direct, and Wiley Online Library databases. In order to improve the review reporting, the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines were used. Results: Authors selected sixty-two articles meeting the inclusion criteria, which were categorized according to the purpose of the study (intraoperative, training, rehabilitation) and according to the surgical procedure used. Conclusions: AR has the potential to improve orthopedic training and practice by providing an increasingly human-centered clinical approach. Further research can be addressed by this review to cover problems related to hardware limitations, lack of accurate registration and tracking systems, and absence of security protocols.

Augmented Reality to Assist Skin Paddle Harvesting in Osteomyocutaneous Fibular Flap Reconstructive Surgery: A Pilot Evaluation on a 3D-Printed Leg Phantom

Background

Augmented Reality (AR) represents an evolution of navigation-assisted surgery, providing surgeons with a virtual aid contextually merged with the real surgical field. We recently reported a case series of AR-assisted fibular flap harvesting for mandibular reconstruction. However, the registration accuracy between the real and the virtual content needs to be systematically evaluated before widely promoting this tool in clinical practice. In this paper, after description of the AR based protocol implemented for both tablet and HoloLens 2 smart glasses, we evaluated in a first test session the achievable registration accuracy with the two display solutions, and in a second test session the success rate in executing the AR-guided skin paddle incision task on a 3D printed leg phantom.

Methods

From a real computed tomography dataset, 3D virtual models of a human leg, including fibula, arteries and skin with planned paddle profile for harvesting, were obtained. All virtual models were imported into Unity software to develop a marker-less AR application suitable to be used both via tablet and via HoloLens 2 headset. The registration accuracy for both solutions was verified on a 3D printed leg phantom obtained from the virtual models, by repeatedly applying the tracking function and computing pose deviations between the AR-projected virtual skin paddle profile and the real one transferred to the phantom via a CAD/CAM cutting guide. The success rate in completing the AR-guided task of skin paddle harvesting was evaluated using CAD/CAM templates positioned on the phantom model surface.

Results

On average, the marker-less AR protocol showed comparable registration errors (ranging within 1-5 mm) for tablet-based and HoloLens-based solution. Registration accuracy seems to be quite sensitive to ambient light conditions. We found a good success rate in completing the AR-guided task within an error margin of 4 mm (97% and 100% for tablet and HoloLens, respectively). All subjects reported greater usability and ergonomics for HoloLens 2 solution.

Conclusions

Results revealed that the proposed marker-less AR based protocol may guarantee a registration error within 1-5 mm for assisting skin paddle harvesting in the clinical setting. Optimal lightening conditions and further improvement of marker-less tracking technologies have the potential to increase the efficiency and precision of this AR-assisted reconstructive surgery.

Augmented Reality (AR) in Orthopedics: Current Applications and Future Directions

PURPOSE OF REVIEW

Imaging technologies (X-ray, CT, MRI, and ultrasound) have revolutionized orthopedic surgery, allowing for the more efficient diagnosis, monitoring, and treatment of musculoskeletal aliments. The current review investigates recent literature surrounding the impact of augmented reality (AR) imaging technologies on orthopedic surgery. In particular, it investigates the impact that AR technologies may have on provider cognitive burden, operative times, occupational radiation exposure, and surgical precision and outcomes.

RECENT FINDINGS

Many AR technologies have been shown to lower provider cognitive burden and reduce operative time and radiation exposure while improving surgical precision in pre-clinical cadaveric and sawbones models. So far, only a few platforms focusing on pedicle screw placement have been approved by the FDA. These technologies have been implemented clinically with mixed results when compared to traditional free-hand approaches. It remains to be seen if current AR technologies can deliver upon their multitude of promises, and the ability to do so seems contingent upon continued technological progress. Additionally, the impact of these platforms will likely be highly conditional on clinical indication and provider type. It remains unclear if AR will be broadly accepted and utilized or if it will be reserved for niche indications where it adds significant value. One thing is clear, orthopedics' high utilization of pre- and intra-operative imaging, combined with the relative ease of tracking rigid structures like bone as compared to soft tissues, has made it the clear beachhead market for AR technologies in medicine.

AUGMENTED REALITY NAVIGATION FRAMEWORK FOR TOTAL HIP ARTHROPLASTY SURGERY

In a total hip arthroplasty surgery, correctly implanting the artificial acetabulum and the femoral head is essential for a successful treatment. An augmented reality (AR) navigation framework is proposed in this paper to provide accurate surgical guidance in a total hip arthroplasty procedure. The AR framework consists of three parts: (1) preoperative surgical planning to generate virtual information for AR; (2) computer vision-based tracking for the real-time localization of both acetabular cup positioner and bony landmarks during surgery; (3) registration of a virtual object onto a real-world operative field to properly overlay the preoperative surgical planning data onto a three-dimensional (3D)-printed pelvis model. The cost-effective framework is designed with our clinical partner based on real surgical conditions. The bony landmarks are automatically detected and used for the registration between virtual and real objects. The AR performance is evaluated with a pelvis model, and it presents mean errors of 2.2[Formula: see text]mm and 0.8∘ in position and orientation, respectively, between real and virtual spaces. These small errors are within the tolerance of positive surgical outcomes.

Multi-Modality guidance based surgical navigation for percutaneous endoscopic transforaminal discectomy

OBJECTIVE

Fluoroscopic guidance is a critical step for the puncture procedure in percutaneous endoscopic transforaminal discectomy (PETD). However, two-dimensional observations of the three-dimensional anatomic structure suffer from the effects of projective simplification. To accurately assess the spatial relations between the patient vertebra tissues and puncture needle, a considerable number of fluoroscopic images from different orientations need to be acquired by the surgeons. This process significantly increases the radiation risk for both the patient and surgeons.

METHODS

In this paper, we propose an augmented reality (AR) surgical navigation system for PETD based on multi-modality information, which contains fluoroscopy, optical tracking, and depth camera. To register the fluoroscopic image with the intraoperative video, we design a lightweight non-invasive fiducial with markers and detect the markers based on the deep learning method. It can display the intraoperative video fused with the registered fluoroscopic images. We also present a self-adaptive calibration and transformation method between a 6-DOF optical tracking device and a depth camera, which are in different coordinate systems.

RESULTS

With the substantially reduced frequency of fluoroscopy imaging, the system can accurately track and superimpose the virtual puncture needle on fluoroscopy images in real-time. From operating theatre in vivo animal experiments, the results illustrate that the system average positioning accuracy can reach 1.98mm and the orientation accuracy can reach 1.19^∘. From the clinical validation results, the system significantly lower the frequency of fluoroscopy imaging (42.7%) and reduce the radiation risk for both the patient and surgeons.

CONCLUSION

Coupled with the user study, both the quantitative and qualitative results indicate that our navigation system has the potential to be highly useful in clinical practice. Compared with the existing navigation systems, which are usually equipped with a variety of large and high-cost medical equipments, such as O-arm, cone-beam CT, and robots, our navigation system does not need special equipment and can be implemented with common equipment in the operating room, such as C-arm, desktop, etc., even in small hospitals.

Augmented-Reality-Assisted K-Wire Placement for Glenoid Component Positioning in Reversed Shoulder Arthroplasty: A Proof-of-Concept Study

The accuracy of the implant's post-operative position and orientation in reverse shoulder arthroplasty is known to play a significant role in both clinical and functional outcomes. Whilst technologies such as navigation and robotics have demonstrated superior radiological outcomes in many fields of surgery, the impact of augmented reality (AR) assistance in the operating room is still unknown. Malposition of the glenoid component in shoulder arthroplasty is known to result in implant failure and early revision surgery. The use of AR has many promising advantages, including allowing the detailed study of patient-specific anatomy without the need for invasive procedures such as arthroscopy to interrogate the joint's articular surface. In addition, this technology has the potential to assist surgeons intraoperatively in aiding the guidance of surgical tools. It offers the prospect of increased component placement accuracy, reduced surgical procedure time, and improved radiological and functional outcomes, without recourse to the use of large navigation or robotic instruments, with their associated high overhead costs. This feasibility study describes the surgical workflow from a standardised CT protocol, via 3D reconstruction, 3D planning, and use of a commercial AR headset, to AR-assisted k-wire placement. Post-operative outcome was measured using a high-resolution laser scanner on the patient-specific 3D printed bone. In this proof-of-concept study, the discrepancy between the planned and the achieved glenoid entry point and guide-wire orientation was approximately 3 mm with a mean angulation error of 5°.

Evaluation of a Wearable AR Platform for Guiding Complex Craniotomies in Neurosurgery

Today, neuronavigation is widely used in daily clinical routine to perform safe and efficient surgery. Augmented reality (AR) interfaces can provide anatomical models and preoperative planning contextually blended with the real surgical scenario, overcoming the limitations of traditional neuronavigators. This study aims to demonstrate the reliability of a new-concept AR headset in navigating complex craniotomies. Moreover, we aim to prove the efficacy of a patient-specific template-based methodology for fast, non-invasive, and fully automatic planning-to-patient registration. The AR platform navigation performance was assessed with an in-vitro study whose goal was twofold: to measure the real-to-virtual 3D target visualization error (TVE), and assess the navigation accuracy through a user study involving 10 subjects in tracing a complex craniotomy. The feasibility of the template-based registration was preliminarily tested on a volunteer. The TVE mean and standard deviation were 1.3 and 0.6 mm. The results of the user study, over 30 traced craniotomies, showed that 97% of the trajectory length was traced within an error margin of 1.5 mm, and 92% within a margin of 1 mm. The in-vivo test confirmed the feasibility and reliability of the patient-specific template for registration. The proposed AR headset allows ergonomic and intuitive fruition of preoperative planning, and it can represent a valid option to support neurosurgical tasks.

Augmented reality in robotic assisted orthopaedic surgery: A pilot study

BACKGROUND

The research and development of augmented-reality (AR) technologies in surgical applications has seen an evolution of the traditional user-interfaces (UI) utilised by clinicians when conducting robot-assisted orthopaedic surgeries. The typical UI for such systems relies on surgeons managing 3D medical imaging data in the 2D space of a touchscreen monitor, located away from the operating site. Conversely, AR can provide a composite view overlaying the real surgical scene with co-located virtual holographic representations of medical data, leading to a more immersive and intuitive operator experience.

MATERIALS AND METHODS

This work explores the integration of AR within an orthopaedic setting by capturing and replicating the UI of an existing surgical robot within an AR head-mounted display worn by the clinician. The resulting mixed-reality workflow enabled users to simultaneously view the operating-site and real-time holographic operating informatics when carrying out a robot-assisted patellofemoral-arthroplasty (PFA). Ten surgeons were recruited to test the impact of the AR system on procedure completion time and operating surface roughness.

RESULTS AND DISCUSSION

The integration of AR did not appear to require subjects to significantly alter their surgical techniques, which was demonstrated by non-significant changes to the study's clinical metrics, with a statistically insignificant mean increase in operating time (+0.778 s, p = 0.488) and a statistically insignificant change in mean surface roughness (p = 0.274). Additionally, a post-operative survey indicated a positive consensus on the usability of the AR system without incurring noticeable physical distress such as eyestrain or fatigue.

CONCLUSIONS

Overall, these study results demonstrated a successful integration of AR technologies within the framework of an existing robot-assisted surgical platform with no significant negative effects in two quantitative metrics of surgical performance, and a positive outcome relating to user-centric and ergonomic evaluation criteria.

Opportunities and challenges of using augmented reality and heads-up display in orthopaedic surgery: A narrative review

BACKGROUND & AIM

Utilization of augmented reality (AR) and heads-up displays (HUD) to aid orthopaedic surgery has the potential to benefit surgeons and patients alike through improved accuracy, safety, and educational benefits. With the COVID-19 pandemic, the opportunity for adoption of novel technology is more relevant. The aims are to assess the technology available, to understand the current evidence regarding the benefit and to consider challenges to implementation in clinical practice.

METHODS & RESULTS

PRISMA guidelines were used to filter the literature. Of 1004 articles returned the following exclusion criteria were applied: 1) reviews/commentaries 2) unrelated to orthopaedic surgery 3) use of other AR wearables beyond visual aids leaving 42 papers for review.This review illustrates benefits including enhanced accuracy and reduced time of surgery, reduced radiation exposure and educational benefits.

CONCLUSION

Whilst there are obstacles to overcome, there are already reports of technology being used. As with all novel technologies, a greater understanding of the learning curve is crucial, in addition to shielding our patients from this learning curve. Improvements in usability and implementing surgeons' specific needs should increase uptake.

Augmented Reality, Virtual Reality and Artificial Intelligence in Orthopedic Surgery: A Systematic Review

Background: The application of virtual and augmented reality technologies to orthopaedic surgery training and practice aims to increase the safety and accuracy of procedures and reducing complications and costs. The purpose of this systematic review is to summarise the present literature on this topic while providing a detailed analysis of current flaws and benefits. Methods: A comprehensive search on the PubMed, Cochrane, CINAHL, and Embase database was conducted from inception to February 2021. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines were used to improve the reporting of the review. The Cochrane Risk of Bias Tool and the Methodological Index for Non-Randomized Studies (MINORS) was used to assess the quality and potential bias of the included randomized and non-randomized control trials, respectively. Results: Virtual reality has been proven revolutionary for both resident training and preoperative planning. Thanks to augmented reality, orthopaedic surgeons could carry out procedures faster and more accurately, improving overall safety. Artificial intelligence (AI) is a promising technology with limitless potential, but, nowadays, its use in orthopaedic surgery is limited to preoperative diagnosis. Conclusions: Extended reality technologies have the potential to reform orthopaedic training and practice, providing an opportunity for unidirectional growth towards a patient-centred approach.

Augmented Reality in Orthopedic Surgery Is Emerging from Proof of Concept Towards Clinical Studies: a Literature Review Explaining the Technology and Current State of the Art

PURPOSE OF REVIEW

Augmented reality (AR) is becoming increasingly popular in modern-day medicine. Computer-driven tools are progressively integrated into clinical and surgical procedures. The purpose of this review was to provide a comprehensive overview of the current technology and its challenges based on recent literature mainly focusing on clinical, cadaver, and innovative sawbone studies in the field of orthopedic surgery. The most relevant literature was selected according to clinical and innovational relevance and is summarized.

RECENT FINDINGS

Augmented reality applications in orthopedic surgery are increasingly reported. In this review, we summarize basic principles of AR including data preparation, visualization, and registration/tracking and present recently published clinical applications in the area of spine, osteotomies, arthroplasty, trauma, and orthopedic oncology. Higher accuracy in surgical execution, reduction of radiation exposure, and decreased surgery time are major findings presented in the literature. In light of the tremendous progress of technological developments in modern-day medicine and emerging numbers of research groups working on the implementation of AR in routine clinical procedures, we expect the AR technology soon to be implemented as standard devices in orthopedic surgery.

Use of augmented reality navigation to optimise the surgical management of craniofacial fibrous dysplasia

The aim of this study was to apply an augmented reality (AR) navigation technique based on a head- mounted display in the treatment of craniofacial fibrous dysplasia and to explore the feasibility and the value of AR in craniofacial surgery. With preoperative planning and three-dimensional simulation, the normal anatomical contours of the deformed area were recreated by superimposing the unaffected side on to the affected side. We completed the recontouring procedures in real time with the aid of an AR navigation system. The surgical outcome was assessed by superimposing the postoperative computed tomographic images on to the preoperative virtual plan. The preparation and operation times were recorded. With intraoperative AR guidance, facial bone recontouring was performed uneventfully in all cases. The mean (SD) discrepancy between the actual surgical reduction and preoperative planning was 1.036 (0.081) mm (range: 0.913 (0.496) to 1.165 (0.498) mm). The operation time ranged from 50 to 80 minutes, with an average of 66.4 minutes. The preoperative preparation time ranged from 26 to 36 minutes, with a mean of 29.6 minutes. AR navigation-assisted facial bone recontouring is a valuable treatment modality in managing craniomaxillofacial fibrous dysplasia and shows benefits in improving the efficiency and safety of this complicated procedure.

Towards quantitative and intuitive percutaneous tumor puncture via augmented virtual reality

In recent years, the radiofrequency ablation (RFA) therapy has become a widely accepted minimal invasive treatment for liver tumor patients. However, it is challenging for doctors to precisely and efficiently perform the percutaneous tumor punctures under free-breathing conditions. This is because the traditional RFA is based on the 2D CT Image information, the missing spatial and dynamic information is dependent on surgeons' experience. This paper presents a novel quantitative and intuitive surgical navigation modality for percutaneous respiratory tumor puncture via augmented virtual reality, which is to achieve the augmented visualization of the pre-operative virtual planning information precisely being overlaid on intra-operative surgical scenario. In the pre-operation stage, we first combine the signed distance field of feasible structures (like liver and tumor) where the puncture path can go through and unfeasible structures (like large vessels and ribs) where the needle is not allowed to go through to quantitatively generate the 3D feasible region for percutaneous puncture. Then we design three constraints according to the RFA specialists consensus to automatically determine the optimal puncture trajectory. In the intra-operative stage, we first propose a virtual-real alignment method to precisely superimpose the virtual information on surgical scenario. Then, a user-friendly collaborative holographic interface is designed for real-time 3D respiratory tumor puncture navigation, which can effectively assist surgeons fast and accurately locating the target step-by step. The validation of our system is performed on static abdominal phantom and in vivo beagle dogs with artificial lesion. Experimental results demonstrate that the accuracy of the proposed planning strategy is better than the manual planning sketched by experienced doctors. Besides, the proposed holographic navigation modality can effectively reduce the needle adjustment for precise puncture as well. Our system shows its clinical feasibility to provide the quantitative planning of optimal needle path and intuitive in situ holographic navigation for percutaneous tumor ablation without surgeons' experience-dependence and reduce the times of needle adjustment. The proposed augmented virtual reality navigation system can effectively improve the precision and reliability in percutaneous tumor ablation and has the potential to be used for other surgical navigation tasks.

Mixed reality as a novel tool for diagnostic and surgical navigation in orthopaedics

Purpose

This study presents a novel surgical navigation tool developed in mixed reality environment for orthopaedic surgery. Joint and skeletal deformities affect all age groups and greatly reduce the range of motion of the joints. These deformities are notoriously difficult to diagnose and to correct through surgery.

Method

We have developed a surgical tool which integrates surgical instrument tracking and augmented reality through a head mounted display. This allows the surgeon to visualise bones with the illusion of possessing “X-ray” vision. The studies presented below aim to assess the accuracy of the surgical navigation tool in tracking a location at the tip of the surgical instrument in holographic space.

Results

Results show that the average accuracy provided by the navigation tool is around 8 mm, and qualitative assessment by the orthopaedic surgeons provided positive feedback in terms of the capabilities for diagnostic use.

Conclusions

More improvements are necessary for the navigation tool to be accurate enough for surgical applications, however, this new tool has the potential to improve diagnostic accuracy and allow for safer and more precise surgeries, as well as provide for better learning conditions for orthopaedic surgeons in training.

Augmented Reality Based Surgical Navigation of Complex Pelvic Osteotomies—A Feasibility Study on Cadavers

Augmented reality (AR)-based surgical navigation may offer new possibilities for safe and accurate surgical execution of complex osteotomies. In this study we investigated the feasibility of navigating the periacetabular osteotomy of Ganz (PAO), known as one of the most complex orthopedic interventions, on two cadaveric pelves under realistic operating room conditions. Preoperative planning was conducted on computed tomography (CT)-reconstructed 3D models using an in-house developed software, which allowed creating cutting plane objects for planning of the osteotomies and reorientation of the acetabular fragment. An AR application was developed comprising point-based registration, motion compensation and guidance for osteotomies as well as fragment reorientation. Navigation accuracy was evaluated on CT-reconstructed 3D models, resulting in an error of 10.8 mm for osteotomy starting points and 5.4° for osteotomy directions. The reorientation errors were 6.7°, 7.0° and 0.9° for the x-, y- and z-axis, respectively. Average postoperative error of LCE angle was 4.5°. Our study demonstrated that the AR-based execution of complex osteotomies is feasible. Fragment realignment navigation needs further improvement, although it is more accurate than the state of the art in PAO surgery.

Registration Techniques for Clinical Applications of Three-Dimensional Augmented Reality Devices

Many clinical procedures would benefit from direct and intuitive real-time visualization of anatomy, surgical plans, or other information crucial to the procedure. Three-dimensional augmented reality (3D-AR) is an emerging technology that has the potential to assist physicians with spatial reasoning during clinical interventions. The most intriguing applications of 3D-AR involve visualizations of anatomy or surgical plans that appear directly on the patient. However, commercially available 3D-AR devices have spatial localization errors that are too large for many clinical procedures. For this reason, a variety of approaches for improving 3D-AR registration accuracy have been explored. The focus of this review is on the methods, accuracy, and clinical applications of registering 3D-AR devices with the clinical environment. The works cited represent a variety of approaches for registering holograms to patients, including manual registration, computer vision-based registration, and registrations that incorporate external tracking systems. Evaluations of user accuracy when performing clinically relevant tasks suggest that accuracies of approximately 2 mm are feasible. 3D-AR device limitations due to the vergence-accommodation conflict or other factors attributable to the headset hardware add on the order of 1.5 mm of error compared to conventional guidance. Continued improvements to 3D-AR hardware will decrease these sources of error.

Mixed Reality Interaction and Presentation Techniques for Medical Visualisations

Mixed, Augmented and Virtual reality technologies are burgeoning with new applications and use cases appearing rapidly. This chapter provides a brief overview of the fundamental display presentation methods; head-worn, hand-held and projector-based displays. We present a summary of visualisation methods that employ these technologies in the medical domain with a diverse range of examples presented including diagnostic and exploration, intervention and clinical, interaction and gestures, and education.

The role of virtual and augmented reality in orthopedic trauma surgery: From diagnosis to rehabilitation

Virtual and augmented reality have been used to assist and improve human capabilities in many fields. Most recent advances allow the usage of these technologies for personal and professional purposes. In particular, they have been progressively introduced in many medical procedures since the last century. Thanks to immersive training systems and a better comprehension of the ongoing procedure, their main objectives are to increase patient safety and decrease recovery time. The current and future possibilities of virtual and augmented reality in the context of bone fracture reduction are the main focus of this review. This medical procedure requires meticulous planning and a complex intervention in many cases, hence becoming a promising candidate to be benefited from this kind of technology. In this paper, we exhaustively analyze the impact of virtual and augmented reality to bone fracture healing, detailing each task from diagnosis to rehabilitation. Our primary goal is to introduce novel researchers to current trends applied to orthopedic trauma surgery, proposing new lines of research. To that end, we propose and evaluate a set of qualitative metrics to highlight the most promising challenges of virtual and augmented reality technologies in this context.

Applicability of augmented reality in orthopedic surgery - A systematic review

BACKGROUND

Computer-assisted solutions are changing surgical practice continuously. One of the most disruptive technologies among the computer-integrated surgical techniques is Augmented Reality (AR). While Augmented Reality is increasingly used in several medical specialties, its potential benefit in orthopedic surgery is not yet clear. The purpose of this article is to provide a systematic review of the current state of knowledge and the applicability of AR in orthopedic surgery.

METHODS

A systematic review of the current literature was performed to find the state of knowledge and applicability of AR in Orthopedic surgery. A systematic search of the following three databases was performed: "PubMed", "Cochrane Library" and "Web of Science". The systematic review followed the Preferred Reporting Items on Systematic Reviews and Meta-analysis (PRISMA) guidelines and it has been published and registered in the international prospective register of systematic reviews (PROSPERO).

RESULTS

31 studies and reports are included and classified into the following categories: Instrument / Implant Placement, Osteotomies, Tumor Surgery, Trauma, and Surgical Training and Education. Quality assessment could be performed in 18 studies. Among the clinical studies, there were six case series with an average score of 90% and one case report, which scored 81% according to the Joanna Briggs Institute Critical Appraisal Checklist (JBI CAC). The 11 cadaveric studies scored 81% according to the QUACS scale (Quality Appraisal for Cadaveric Studies).

CONCLUSION

This manuscript provides 1) a summary of the current state of knowledge and research of Augmented Reality in orthopedic surgery presented in the literature, and 2) a discussion by the authors presenting the key remarks required for seamless integration of Augmented Reality in the future surgical practice.

TRIAL REGISTRATION

PROSPERO registration number: CRD42019128569.

Optimization of virtual and real registration technology based on augmented reality in a surgical navigation system

Background

The traditional navigation interface was intended only for two-dimensional observation by doctors; thus, this interface does not display the total spatial information for the lesion area. Surgical navigation systems have become essential tools that enable for doctors to accurately and safely perform complex operations. The image navigation interface is separated from the operating area, and the doctor needs to switch the field of vision between the screen and the patient’s lesion area. In this paper, augmented reality (AR) technology was applied to spinal surgery to provide more intuitive information to surgeons. The accuracy of virtual and real registration was improved via research on AR technology. During the operation, the doctor could observe the AR image and the true shape of the internal spine through the skin.

Methods

To improve the accuracy of virtual and real registration, a virtual and real registration technique based on an improved identification method and robot-assisted method was proposed. The experimental method was optimized by using the improved identification method. X-ray images were used to verify the effectiveness of the puncture performed by the robot.

Results

The final experimental results show that the average accuracy of the virtual and real registration based on the general identification method was 9.73 ± 0.46 mm (range 8.90–10.23 mm). The average accuracy of the virtual and real registration based on the improved identification method was 3.54 ± 0.13 mm (range 3.36–3.73 mm). Compared with the virtual and real registration based on the general identification method, the accuracy was improved by approximately 65%. The highest accuracy of the virtual and real registration based on the robot-assisted method was 2.39 mm. The accuracy was improved by approximately 28.5% based on the improved identification method.

Conclusion

The experimental results show that the two optimized methods are highly very effective. The proposed AR navigation system has high accuracy and stability. This system may have value in future spinal surgeries.

Off-Line Camera-Based Calibration for Optical See-Through Head-Mounted Displays

In recent years, the entry into the market of self contained optical see-through headsets with integrated multi-sensor capabilities has led the way to innovative and technology driven augmented reality applications and has encouraged the adoption of these devices also across highly challenging medical and industrial settings. Despite this, the display calibration process of consumer level systems is still sub-optimal, particularly for those applications that require high accuracy in the spatial alignment between computer generated elements and a real-world scene. State-of-the-art manual and automated calibration procedures designed to estimate all the projection parameters are too complex for real application cases outside laboratory environments. This paper describes an off-line fast calibration procedure that only requires a camera to observe a planar pattern displayed on the see-through display. The camera that replaces the user’s eye must be placed within the eye-motion-box of the see-through display. The method exploits standard camera calibration and computer vision techniques to estimate the projection parameters of the display model for a generic position of the camera. At execution time, the projection parameters can then be refined through a planar homography that encapsulates the shift and scaling effect associated with the estimated relative translation from the old camera position to the current user’s eye position. Compared to classical SPAAM techniques that still rely on the human element and to other camera based calibration procedures, the proposed technique is flexible and easy to replicate in both laboratory environments and real-world settings.

Augmented reality in orthopaedics

Aims

Computer-based applications are increasingly being used by orthopaedic surgeons in their clinical practice. With the integration of technology in surgery, augmented reality (AR) may become an important tool for surgeons in the future. By superimposing a digital image on a user’s view of the physical world, this technology shows great promise in orthopaedics. The aim of this review is to investigate the current and potential uses of AR in orthopaedics.

Materials and Methods

A systematic review of the PubMed, MEDLINE, and Embase databases up to January 2019 using the keywords ‘orthopaedic’ OR ‘orthopedic AND augmented reality’ was performed by two independent reviewers.

Results

A total of 41 publications were included after screening. Applications were divided by subspecialty: spine (n = 15), trauma (n = 16), arthroplasty (n = 3), oncology (n = 3), and sports (n = 4). Out of these, 12 were clinical in nature. AR-based technologies have a wide variety of applications, including direct visualization of radiological images by overlaying them on the patient and intraoperative guidance using preoperative plans projected onto real anatomy, enabling hands-free real-time access to operating room resources, and promoting telemedicine and education.

Conclusion

There is an increasing interest in AR among orthopaedic surgeons. Although studies show similar or better outcomes with AR compared with traditional techniques, many challenges need to be addressed before this technology is ready for widespread use. Cite this article: Bone Joint J 2019;101-B:1479–1488

Co-localized augmented human and X-ray observers in collaborative surgical ecosystem

PURPOSE

Image-guided percutaneous interventions are safer alternatives to conventional orthopedic and trauma surgeries. To advance surgical tools in complex bony structures during these procedures with confidence, a large number of images is acquired. While image-guidance is the de facto standard to guarantee acceptable outcome, when these images are presented on monitors far from the surgical site the information content cannot be associated easily with the 3D patient anatomy.

METHODS

In this article, we propose a collaborative augmented reality (AR) surgical ecosystem to jointly co-localize the C-arm X-ray and surgeon viewer. The technical contributions of this work include (1) joint calibration of a visual tracker on a C-arm scanner and its X-ray source via a hand-eye calibration strategy, and (2) inside-out co-localization of human and X-ray observers in shared tracking and augmentation environments using vision-based simultaneous localization and mapping.

RESULTS

We present a thorough evaluation of the hand-eye calibration procedure. Results suggest convergence when using 50 pose pairs or more. The mean translation and rotation errors at convergence are 5.7 mm and [Formula: see text], respectively. Further, user-in-the-loop studies were conducted to estimate the end-to-end target augmentation error. The mean distance between landmarks in real and virtual environment was 10.8 mm.

CONCLUSIONS

The proposed AR solution provides a shared augmented experience between the human and X-ray viewer. The collaborative surgical AR system has the potential to simplify hand-eye coordination for surgeons or intuitively inform C-arm technologists for prospective X-ray view-point planning.