Augmented Reality in Orthopedic Surgery Is Emerging from Proof of Concept Towards Clinical Studies: a Literature Review Explaining the Technology and Current State of the Art

First-in-human trial using mixed-reality visualization for patient setup during breast or chest wall radiotherapy

BACKGROUND

The purpose of this study is to assess the feasibility of mixed-reality (MixR) visualization for patient setup in breast and chest wall radiotherapy (RT) by performing a first-in-human clinical trial comparing MixR with a 3-point alignment.

METHODS

IRB approval was granted for a study incorporating MixR during the setup process for patients undergoing proton (n = 10) or photon (n = 8) RT to the breast or chest wall. For each patient, MixR was utilized for five fractions and compared against another five fractions using 3-point alignment. During fractions with MixR, the patient was aligned by at least one therapist wearing a HoloLens 2 device who was able to guide the process by simultaneously and directly viewing the patient and a hologram of the patient's surface derived from their simulation CT scan. Alignment accuracy was quantified with cone-beam CT (CBCT) for photon treatments and CBCT plus kV/kV imaging for proton treatments. Registration time was tracked throughout the setup process as well as the amount of image guidance (IGRT) utilized for final alignment.

RESULTS

In the proton cohort, the mean 3D shift was 0.96 cm using 3-point alignment and 1.18 cm using MixR. An equivalence test indicated that the difference in registration accuracy between the two techniques was less than 0.5 cm. In the photon cohort, the mean 3D shift was 1.18 cm using 3-point alignment and 1.00 cm using MixR. An equivalence test indicated that the difference in registration accuracy was less than 0.3 cm. Minor differences were seen in registration time and the amount of IGRT utilization.

CONCLUSIONS

MixR for patient setup for breast cancer RT is possible at the level of accuracy and efficiency provided by a 3-point alignment. Further developments in marker tracking, feedback, and a better understanding of the perceptual challenges of MixR are needed to achieve a similar level of accuracy as provided by modern surface-guided radiotherapy (SGRT) systems.

TRIAL REGISTRATION

ClinicalTrials.gov, UFHPTI 2015-BR05: Improving Breast Radiotherapy Setup and Delivery Using Mixed-Reality Visualization, NCT05178927.

Efficacy of a computer vision-based system for exercise management in patients with knee osteoarthritis: a study protocol for a randomised controlled pilot trial

INTRODUCTION

This study aims to evaluate the efficacy of a computer vision system in guiding exercise management for patients with knee osteoarthritis (OA) by comparing functional improvement between a tele-rehabilitation versus an outpatient intervention program.

METHODS AND ANALYSIS

This is a prospective, single-blind, randomised controlled trial of 60 patients with knee OA who will be randomly assigned to exercise therapy (n=30) or control (n=30) . Both groups will receive treatment two times per week for 12 weeks. The primary outcome of the study will be assessed using the University of Western Ontario and McMaster University Osteoarthritis Index (WOMAC). The Knee Injury and Osteoarthritis Outcome Score will be assessed, as well as the visual analogue scale, quality of life score and physical fitness score. All observations will be collected at baseline and at weeks 4, 8 and 12 during the intervention period, as well as at weeks 4, 8, 12 and 24 during the follow-up visits after the end of the intervention.

ETHICS AND DISSEMINATION

This evaluator-blinded, prospective, randomised controlled study was approved by the Biomedical Ethics Review Committee of West China Hospital of Sichuan University.

TRIAL REGISTRATION NUMBER

ChiCTR2300070319.

Leveraging digital twins for improved orthopaedic evaluation and treatment

Purpose

The purpose of this article is to explore the potential of digital twin technologies in orthopaedics and to evaluate how their integration with artificial intelligence (AI) and deep learning (DL) can improve orthopaedic evaluation and treatment. This review addresses key applications of digital twins, including surgical planning, patient-specific outcome prediction, augmented reality-assisted surgery and simulation-based surgical training.

Methods

Existing studies on digital twins in various domains, including engineering, biomedical and orthopaedics are reviewed. We also reviewed advancements in AI and DL relevant to digital twins. We focused on identifying key benefits, challenges and future directions for the implementation of digital twins in orthopaedic practice.

Results

The review highlights that digital twins offer significant potential to revolutionise orthopaedic care by enabling precise surgical planning, real-time outcome prediction and enhanced training. Digital twins can model patient-specific anatomy using advanced imaging techniques and dynamically update with real-time data, providing valuable insights during surgery and postoperative care. However, challenges such as the need for large-scale data sets, technological limitations and integration issues must be addressed to fully realise these benefits.

Conclusion

Digital twins represent a promising frontier in orthopaedic research and practice, with the potential to improve patient outcomes and enhance surgical precision. To enable widespread adoption, future research must focus on overcoming current challenges and further refining the integration of digital twins with AI and DL technologies.

Level of Evidence

Level V.

iPhone-Based Cartilage Topography Scanning Yields Similar Results to Computed Tomography Scanning

Purpose

To investigate the feasibility and accuracy of 3-dimensional (3D) iPhone scans using commercially available applications compared with computed tomography (CT) for mapping chondral surface topography of the knee.

Methods

Ten cadaveric dysplastic trochleae, 16 patellae, and 24 distal femoral condyles (DFCs) underwent CT scans and 3D scans using 3 separate optical scanning applications on an iPhone X. The 3D surface models were compared by measuring surface-to-surface least distance distribution of overlapped models using a validated 3D-3D registration volume merge method. The absolute least mean square distances for the iPhone-generated models from each scanning application were calculated in comparison to CT models using a point-to-surface distance algorithm allowing regional "inside/outside" measurement of the absolute distance between models.

Results

Only 1 of the 3 scanning applications created models usable for quantitative analysis. Overall, there was a median absolute least mean square distance between the usable model and CT-generated models of 0.18 mm. The trochlea group had a significantly lower median absolute least mean square distance compared with the DFC group (0.14 mm [interquartile range, 0.13-0.17] vs 0.19 mm [0.17-0.25], P = .002). iPhone models were smaller compared with CT models (negative signed distances) for all trochleae, 83% of DFCs, and 69% of patellae.

Conclusions

In this study, we found minimal differences between a 3D iPhone scanning application and conventional CT scanning when analyzing surface topography.

Clinical Relevance

Emerging 3D iPhone scanning technology can create accurate, inexpensive, real-time 3D models of the intended target. Surface topography evaluation may be useful in graft selection during surgical procedures such as osteochondral allograft transplantation.

HoloDIEP-Faster and More Accurate Intraoperative DIEA Perforator Mapping Using a Novel Mixed Reality Tool

BACKGROUND

 Microsurgical breast reconstruction using abdominal tissue is a complex procedure, in part, due to variable vascular/perforator anatomy. Preoperative computed tomography angiography (CTA) has mitigated this challenge to some degree; yet it continues to pose certain challenges. The ability to map perforators with Mixed Reality has been demonstrated in case studies, but its accuracy has not been studied intraoperatively. Here, we compare the accuracy of "HoloDIEP" in identifying perforator location (vs. Doppler ultrasound) by using holographic 3D models derived from preoperative CTA.

METHODS

 Using a custom application on HoloLens, the deep inferior epigastric artery vascular tree was traced in 15 patients who underwent microsurgical breast reconstruction. Perforator markings were compared against the 3D model in a coordinate system centered on the umbilicus. Holographic- and Doppler-identified markings were compared using a perspective-corrected photo technique against the 3D model along with measurement of duration of perforator mapping for each technique.

RESULTS

 Vascular points in HoloDIEP skin markings were -0.97 ± 6.2 mm (perforators: -0.62 ± 6.13 mm) away from 3D-model ground-truth in radial length from the umbilicus at a true distance of 10.81 ± 6.14 mm (perforators: 11.40 ± 6.15 mm). Absolute difference in radial distance was twice as high for Doppler markings compared with Holo-markings (9.71 ± 6.16 and 4.02 ± 3.20 mm, respectively). Only in half of all cases (7/14), more than 50% of the Doppler-identified points were reasonably close (<30 mm) to 3D-model ground-truth. HoloDIEP was twice as fast as Doppler ultrasound (76.9s vs. 150.4 s per abdomen).

CONCLUSION

 HoloDIEP allows for faster and more accurate intraoperative perforator mapping than Doppler ultrasound.

Conventional vs Augmented Reality-Guided Lateral Calcaneal Lengthening Simulated in a Foot Bone Model

BACKGROUND

Acquired adult flatfoot deformity (AAFD) results in a loss of the medial longitudinal arch of the foot and dysfunction of the posteromedial soft tissues. Hintermann osteotomy (H-O) is often used to treat stage II AAFD. The procedure is challenging because of variations in the subtalar facets and limited intraoperative visibility. We aimed to assess the impact of augmented reality (AR) guidance on surgical accuracy and the facet violation rate.

METHODS

Sixty AR-guided and 60 conventional osteotomies were performed on foot bone models. For AR osteotomies, the ideal osteotomy plane was uploaded to a Microsoft HoloLens 1 headset and carried out in strict accordance with the superimposed holographic plane. The conventional osteotomies were performed relying solely on the anatomy of the calcaneal lateral column. The rate and severity of facet joint violation was measured, as well as accuracy of entry and exit points. The results were compared across AR-guided and conventional osteotomies, and between experienced and inexperienced surgeons.

RESULTS

Experienced surgeons showed significantly greater accuracy for the osteotomy entry point using AR, with the mean deviation of 1.6 ± 0.9 mm (95% CI 1.26, 1.93) compared to 2.3 ± 1.3 mm (95% CI 1.87, 2.79) in the conventional method (P = .035). The inexperienced had improved accuracy, although not statistically significant (P = .064), with the mean deviation of 2.0 ± 1.5 mm (95% CI 1.47, 2.55) using AR compared with 2.7 ± 1.6 mm (95% CI 2.18, 3.32) in the conventional method. AR helped the experienced surgeons avoid full violation of the posterior facet (P = .011). Inexperienced surgeons had a higher rate of middle and posterior facet injury with both methods (P = .005 and .021).

CONCLUSION

Application of AR guidance during H-O was associated with improved accuracy for experienced surgeons, demonstrated by a better accuracy of the osteotomy entry point. More crucially, AR guidance prevented full violation of the posterior facet in the experienced group. Further research is needed to address limitations and test this technology on cadaver feet. Ultimately, the use of AR in surgery has the potential to improve patient and surgeon safety while minimizing radiation exposure.

CLINICAL RELEVANCE

Subtalar facet injury during lateral column lengthening osteotomy represents a real problem in clinical orthopaedic practice. Because of limited intraoperative visibility and variable anatomy, it is hard to resolve this issue with conventional means. This study suggests the potential of augmented reality to improve the osteotomy accuracy.

STTAR: Surgical Tool Tracking Using Off-the-Shelf Augmented Reality Head-Mounted Displays

The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the performance of the task. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers without the need to integrate any additional electronics into the HMD. The proposed framework can simultaneously track multiple tools without having previous knowledge of their geometry and only requires establishing a local network between the headset and a workstation. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09±0.06 mm on lateral translation, 0.42 ±0.32 mm on longitudinal translation and 0.80 ±0.39° for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, seven surgeons were provided with visual navigation and asked to perform 24 injections using the proposed framework. A second study with ten participants served to investigate the capabilities of the framework in the context of more general scenarios. Results from these studies provided comparable accuracy to those reported in the literature for AR-based navigation procedures.

Using Immersive Virtual Reality to Classify Pediatric Thoracolumbar Spine Injuries

Objective This study aimed to assess the reliability and reproducibility of the AO Spine Thoracolumbar Injury Classification System by using virtual reality (VR). We hypothesized that VR is a highly reliable and reproducible method to classify traumatic spine injuries. Methods VR 3D models were created from CT scans of 26 pediatric patients with thoracolumbar spine injuries. Seven orthopedic trainees were educated on the VR platform and AO Spine Thoracolumbar Injury Classification System. Classifications were summarized by primary class and subclass for both rater readings performed two weeks apart with image order randomized. Intra-observer reproducibility was quantified by Fleiss's kappa (kF) for primary classifications and Krippendorff's alpha (aK) for subclassifications along with 95% confidence intervals (CIs) for each rater and across all raters. Inter-observer reliability was quantified by kF for primary classifications and aK for subclassifications along with 95% CIs across all raters for the first read, the second read, and all reads combined. The interpretations were as follows: 0-0.2: slight; 0.2-0.4: fair; 0.4-0.6: moderate; 0.6-0.8: substantial; and >0.8: almost perfect agreement. Results A total of 364 classifications were submitted by seven raters. Intra-observer reproducibility ranged from moderate (kF=0.55) to almost perfect (kF=0.94) for primary classifications and from substantial (aK=0.68) to almost perfect (aK=0.91) for subclassifications. Reproducibility was substantial across all raters for the primary class (kF=0.71; 95% CI=0.61-9.82) and subclass (aK=0.79; 95% CI=0.69-0.86). Inter-observer reliability was substantial (kF=0.63; 95% CI=0.57-0.69) for the first read, moderate (kF=0.58; 95% CI=0.52-0.64) for the second read, and substantial (kF=0.61; 95% CI=0.56-0.65) for all reads for primary classifications. For subclassifications, inter-observer reliability was substantial (aK=0.74; 95% CI=0.58-0.83) for the first read, second read (aK=0.70; 95% CI=0.53-0.80), and all reads (aK=0.72; 95% CI=0.60-0.79). Conclusions Based on our findings, VR is a reliable and reproducible method for the classification of pediatric spine trauma, besides its ability to function as an educational tool for trainees. Further research is needed to evaluate its application for other spine conditions.

Metaverse, virtual reality and augmented reality in total shoulder arthroplasty: a systematic review

PURPOSE

This systematic review aims to provide an overview of the current knowledge on the role of the metaverse, augmented reality, and virtual reality in reverse shoulder arthroplasty.

METHODS

A systematic review was performed using the PRISMA guidelines. A comprehensive review of the applications of the metaverse, augmented reality, and virtual reality in in-vivo intraoperative navigation, in the training of orthopedic residents, and in the latest innovations proposed in ex-vivo studies was conducted.

RESULTS

A total of 22 articles were included in the review. Data on navigated shoulder arthroplasty was extracted from 14 articles: seven hundred ninety-three patients treated with intraoperative navigated rTSA or aTSA were included. Also, three randomized control trials (RCTs) reported outcomes on a total of fifty-three orthopedics surgical residents and doctors receiving VR-based training for rTSA, which were also included in the review. Three studies reporting the latest VR and AR-based rTSA applications and two proof of concept studies were also included in the review.

CONCLUSIONS

The metaverse, augmented reality, and virtual reality present immense potential for the future of orthopedic surgery. As these technologies advance, it is crucial to conduct additional research, foster development, and seamlessly integrate them into surgical education to fully harness their capabilities and transform the field. This evolution promises enhanced accuracy, expanded training opportunities, and improved surgical planning capabilities.

Feasibility and Usability of Augmented Reality Technology in the Orthopaedic Operating Room

PURPOSE OF REVIEW

Augmented reality (AR) has gained popularity in various sectors, including gaming, entertainment, and healthcare. The desire for improved surgical navigation within orthopaedic surgery has led to the evaluation of the feasibility and usability of AR in the operating room (OR). However, the safe and effective use of AR technology in the OR necessitates a proper understanding of its capabilities and limitations. This review aims to describe the fundamental elements of AR, highlight limitations for use within the field of orthopaedic surgery, and discuss potential areas for development.

RECENT FINDINGS

To date, studies have demonstrated evidence that AR technology can be used to enhance navigation and performance in orthopaedic procedures. General hardware and software limitations of the technology include the registration process, ergonomics, and battery life. Other limitations are related to the human response factors such as inattentional blindness, which may lead to the inability to see complications within the surgical field. Furthermore, the prolonged use of AR can cause eye strain and headache due to phenomena such as the vergence-convergence conflict. AR technology may prove to be a better alternative to current orthopaedic surgery navigation systems. However, the current limitations should be mitigated to further improve the feasibility and usability of AR in the OR setting. It is important for both non-clinicians and clinicians to work in conjunction to guide the development of future iterations of AR technology and its implementation into the OR workflow.

An Image-Based Augmented Reality System for Achieving Accurate Bone Resection in Total Knee Arthroplasty

Background and objective With the steady advancement of computer-assisted surgical techniques, the importance of assessing and researching technology related to total knee arthroplasty (TKA) procedures has increased. Augmented reality (AR), a recently proposed next-generation technology, is expected to enhance the precision of orthopedic surgery by providing a more efficient and cost-effective approach. However, the accuracy of image-based AR in TKA surgery has not been established. Therefore, this study aimed to determine whether accurate bone resection can be achieved in TKA surgery using image-based AR. Methods In this study, we replaced traditional CT imaging and reconstructions for creating a bone 3D model by direct 3D scanning of the femur and tibia. The preoperative planning involved identifying anatomical landmarks and determining the surgical details. During surgery, markers were employed to create a local coordinate system for an AR-assisted surgical system using a Polaris camera. This approach helped minimize discrepancies between the 3D model and actual positioning, ensuring accurate alignment. Results The AR-assisted surgery using the image method resulted in fewer errors [average error: 0.32 mm; standard deviation (SD): 0.143] between the bone resection depth of the preoperative surgical plan and the bone model test results. Conclusions Our findings demonstrated the accuracy of bone resectioning by using image-based AR-assisted navigation for TKA surgery. Image-based AR-assisted navigation in TKA surgery is a valuable tool not only for enhancing accuracy by using smart glasses and sensors but also for improving the efficiency of the procedure. Therefore, we anticipate that image-based AR-assisted navigation in TKA surgery will gain wide acceptance in practice.

Distributed visual positioning for surgical instrument tracking

In clinical operations, it is crucial for surgeons to know the location of the surgical instrument. Traditional positioning systems have difficulty dealing with camera occlusion, marker occlusion, and environmental interference. To address these issues, we propose a distributed visual positioning system for surgical instrument tracking in surgery. First, we design the marker pattern with a black and white triangular grid and dot that can be adapted to various instrument surfaces and improve the marker location accuracy of the feature. The cross-points in the marker are the features that each feature has a unique ID. Furthermore, we proposed detection and identification for the position-sensing marker to realize the accurate location and identification of features. Second, we introduce multi Perspective-n-Point (mPnP) method, which fuses feature coordinates from all cameras to deduce the final result directly by the intrinsic and extrinsic parameters. This method provides a reliable initial value for the Bundle Adjustment algorithms. During instrument tracking, we assess the motion state of the instrument and select either dynamic or static Kalman filtering to mitigate any jitter in the instrument's movement. The core algorithms comparison experiment indicates our positioning algorithm has a lower reprojection error comparison to the mainstream algorithms. A series of quantitative experiments showed that the proposed system positioning error is below 0.207 mm, and the run time is below 118.842 ms. The results demonstrate the tremendous clinical application potential of our system providing accurate positioning of instruments promoting the efficiency and safety of clinical surgery.

Spatial Fidelity of Microvascular Perforating Vessels as Perceived by Augmented Reality Virtual Projections

BACKGROUND

Autologous breast reconstruction yields improved long-term aesthetic results but requires increased resources of practitioners and hospital systems. Innovations in radiographic imaging have been increasingly used to improve the efficiency and success of free flap harvest. Augmented reality affords the opportunity to superimpose relevant imaging on a surgeon's native field of view, potentially facilitating dissection of anatomically variable structures. To validate the spatial fidelity of augmented reality projections of deep inferior epigastric perforator flap-relevant anatomy, comparisons of three-dimensional (3D) models and their virtual renderings were performed by four independent observers. Measured discrepancies between the real and holographic models were evaluated.

METHODS

The 3D-printed models of deep inferior epigastric perforator flap-relevant anatomy were fabricated from computed tomographic angiography data from 19 de-identified patients. The corresponding computed tomographic angiography data were similarly formatted for the Microsoft HoloLens to generate corresponding projections. Anatomic points were initially measured on 3D models, after which the corresponding points were measured on the HoloLens projections from two separate vantage points (V1 and V2). Statistical analyses, including generalized linear modeling, were performed to characterize spatial fidelity regarding translation, rotation, and scale of holographic projections.

RESULTS

Among all participants, the median translational displacement at corresponding points was 9.0 mm between the real-3D model and V1, 12.1 mm between the 3D model and V2, and 13.5 mm between V1 and V2.

CONCLUSION

Corresponding points, including topography of perforating vessels, for the purposes of breast reconstruction can be identified within millimeters, but there remain multiple independent contributors of error, most notably the participant and location at which the projection is perceived.

Point-of-Care Orthopedic Oncology Device Development

BACKGROUND

The triad of 3D design, 3D printing, and xReality technologies is explored and exploited to collaboratively realize patient-specific products in a timely manner with an emphasis on designs with meta-(bio)materials.

METHODS

A case study on pelvic reconstruction after oncological resection (osteosarcoma) was selected and conducted to evaluate the applicability and performance of an inter-epistemic workflow and the feasibility and potential of 3D technologies for modeling, optimizing, and materializing individualized orthopedic devices at the point of care (PoC).

RESULTS

Image-based diagnosis and treatment at the PoC can be readily deployed to develop orthopedic devices for pre-operative planning, training, intra-operative navigation, and bone substitution.

CONCLUSIONS

Inter-epistemic symbiosis between orthopedic surgeons and (bio)mechanical engineers at the PoC, fostered by appropriate quality management systems and end-to-end workflows under suitable scientifically amalgamated synergies, could maximize the potential benefits. However, increased awareness is recommended to explore and exploit the full potential of 3D technologies at the PoC to deliver medical devices with greater customization, innovation in design, cost-effectiveness, and high quality.

Mixed Reality and Artificial Intelligence: A Holistic Approach to Multimodal Visualization and Extended Interaction in Knee Osteotomy

OBJECTIVE

Recent advancements in augmented reality led to planning and navigation systems for orthopedic surgery. However little is known about mixed reality (MR) in orthopedics. Furthermore, artificial intelligence (AI) has the potential to boost the capabilities of MR by enabling automation and personalization. The purpose of this work is to assess Holoknee prototype, based on AI and MR for multimodal data visualization and surgical planning in knee osteotomy, developed to run on the HoloLens 2 headset.

METHODS

Two preclinical test sessions were performed with 11 participants (eight surgeons, two residents, and one medical student) executing three times six tasks, corresponding to a number of holographic data interactions and preoperative planning steps. At the end of each session, participants answered a questionnaire on user perception and usability.

RESULTS

During the second trial, the participants were faster in all tasks than in the first one, while in the third one, the time of execution decreased only for two tasks ("Patient selection" and "Scrolling through radiograph") with respect to the second attempt, but without statistically significant difference (respectively [Formula: see text] = 0.14 and [Formula: see text] = 0.13, [Formula: see text]). All subjects strongly agreed that MR can be used effectively for surgical training, whereas 10 (90.9%) strongly agreed that it can be used effectively for preoperative planning. Six (54.5%) agreed and two of them (18.2%) strongly agreed that it can be used effectively for intraoperative guidance.

DISCUSSION/CONCLUSION

In this work, we presented Holoknee, the first holistic application of AI and MR for surgical planning for knee osteotomy. It reported promising results on its potential translation to surgical training, preoperative planning, and surgical guidance. Clinical and Translational Impact Statement - Holoknee can be helpful to support surgeons in the preoperative planning of knee osteotomy. It has the potential to impact positively the training of the future generation of residents and aid surgeons in the intraoperative stage.

Mixed-reality technology for clinical communication: objective assessment of the HoloLens 2 as a clinical communication device in a simulated on-call scenario

OBJECTIVE

Specialty on-call clinicians cover large areas and complex workloads. This study aimed to assess clinical communication using the mixed-reality HoloLens 2 device within a simulated on-call scenario.

METHOD

This study was structured as a randomised, within-participant, controlled study. Thirty ENT trainees used either the HoloLens 2 or a traditional telephone to communicate a clinical case to a consultant. The quality of the clinical communication was scored objectively and subjectively.

RESULTS

Clinical communication using the HoloLens 2 scored statistically higher than telephone (n = 30) (11.9 of 15 vs 10.2 of 15; p = 0.001). Subjectively, consultants judged more communication episodes to be inadequate when using the telephone (7 of 30) versus the HoloLens 2 (0 of 30) (p = 0.01). Qualitative feedback indicates that the HoloLens 2 was easy to use and would add value during an on-call scenario with remote consultant supervision.

CONCLUSION

This study demonstrated the benefit that mixed-reality devices, such as the HoloLens 2 can bring to clinical communication through increasing the accuracy of communication and confidence of the users.

Factors affecting augmented reality head-mounted device performance in real OR

PURPOSE

Over the last years, interest and efforts to implement augmented reality (AR) in orthopedic surgery through head-mounted devices (HMD) have increased. However, the majority of experiments were preclinical and within a controlled laboratory environment. The operating room (OR) is a more challenging environment with various confounding factors potentially affecting the performance of an AR-HMD. The aim of this study was to assess the performance of an AR-HMD in a real-life OR setting.

METHODS

An established AR application using the HoloLens 2 HMD was tested in an OR and in a laboratory by two users. The accuracy of the hologram overlay, the time to complete the trial, the number of rejected registration attempts, the delay in live overlay of the hologram, and the number of completely failed runs were recorded. Further, different OR setting parameters (light condition, setting up partitions, movement of personnel, and anchor placement) were modified and compared.

RESULTS

Time for full registration was higher with 48 s (IQR 24 s) in the OR versus 33 s (IQR 10 s) in the laboratory setting (p < 0.001). The other investigated parameters didn't differ significantly if an optimal OR setting was used. Within the OR, the strongest influence on performance of the AR-HMD was different light conditions with direct light illumination on the situs being the least favorable.

CONCLUSION

AR-HMDs are affected by different OR setups. Standardization measures for better AR-HMD performance include avoiding direct light illumination on the situs, setting up partitions, and minimizing the movement of personnel.

The Accuracy and Absolute Reliability of a Knee Surgery Assistance System Based on ArUco-Type Sensors

Recent advances allow the use of Augmented Reality (AR) for many medical procedures. AR via optical navigators to aid various knee surgery techniques (e.g., femoral and tibial osteotomies, ligament reconstructions or menisci transplants) is becoming increasingly frequent. Accuracy in these procedures is essential, but evaluations of this technology still need to be made. Our study aimed to evaluate the system's accuracy using an in vitro protocol. We hypothesised that the system's accuracy was equal to or less than 1 mm and 1° for distance and angular measurements, respectively. Our research was an in vitro laboratory with a 316 L steel model. Absolute reliability was assessed according to the Hopkins criteria by seven independent evaluators. Each observer measured the thirty palpation points and the trademarks to acquire direct angular measurements on three occasions separated by at least two weeks. The system's accuracy in assessing distances had a mean error of 1.203 mm and an uncertainty of 2.062, and for the angular values, a mean error of 0.778° and an uncertainty of 1.438. The intraclass correlation coefficient was for all intra-observer and inter-observers, almost perfect or perfect. The mean error for the distance's determination was statistically larger than 1 mm (1.203 mm) but with a trivial effect size. The mean error assessing angular values was statistically less than 1°. Our results are similar to those published by other authors in accuracy analyses of AR systems.

Augmented reality in ultrasound-guided regional anaesthesia: useful tool or expensive toy?

Use of augmented reality is increasingly applied in medical education and practice. The main advantage of this technology is the display of relevant information in the visual field of multiple operators. Here we provide a critical analysis of the potential application of augmented reality in regional anaesthesia.

Applications of Augmented Reality in Orthopaedic Spine Surgery

The application of augmented reality (AR) in surgical settings has primarily been as a navigation tool in the operating room because of its ease of use and minimal effect on surgical procedures. The surgeon can directly face the surgical field while viewing 3D anatomy virtually, thus reducing the need to look at an external display, such as a navigation system. Applications of AR are being explored in spine surgery. The basic principles of AR include data preparation, registration, tracking, and visualization. Current literature provides sufficient preclinical and clinical data evidence for the use of AR technology in spine surgery. AR systems are efficient assistive devices, providing greater accuracy for insertion points, more comfort for surgeons, and reduced operating time. AR technology also has beneficial applications in surgical training, education, and telementorship for spine surgery. However, costs associated with specially designed imaging equipment and physicians' comfort in using this technology continue to remain barriers to its adoption. As this technology evolves to a more widespread use, future applications will be directed by the cost-effectiveness of AR-assisted surgeries.

Augmented Reality in Minimally Invasive Spinal Surgery: A Narrative Review of Available Technology

INTRODUCTION

Spine surgery has undergone significant changes in approach and technique. With the adoption of intraoperative navigation, minimally invasive spinal surgery (MISS) has arguably become the gold standard. Augmented reality (AR) has now emerged as a front-runner in anatomical visualization and narrower operative corridors. In effect, AR is poised to revolutionize surgical training and operative outcomes. Our study examines the current literature on AR-assisted MISS, synthesizes findings, and creates a narrative highlighting the history and future of AR in spine surgery.

MATERIAL AND METHODS

Relevant literature was gathered using the PubMed (Medline) database from 1975 to 2023. Pedicle screw placement models were the primary intervention in AR. These were compared to the outcomes of traditional MISS RESULTS: We found that AR devices on the market show promising clinical outcomes in preoperative training and intraoperative use. Three prominent systems were as follows: XVision, HoloLens, and ImmersiveTouch. In the studies, surgeons, residents, and medical students had opportunities to operate AR systems, showcasing their educational potential across each phase of learning. Specifically, one facet described training with cadaver models to gauge accuracy in pedicle screw placement. AR-MISS exceeded free-hand methods without unique complications or contraindications.

CONCLUSIONS

While still in its infancy, AR has already proven beneficial for educational training and intraoperative MISS applications. We believe that with continued research and advancement of this technology, AR is poised to become a dominant player within the fundamentals of surgical education and MISS operative technique.

Artifical intelligence use in orthopedics: an ethical point of view

Artificial intelligence (AI) is increasingly being utilized in orthopedics practice. Ethical concerns have arisen alongside marked improvements and widespread utilization of AI. Patient privacy, consent, data protection, cybersecurity, data safety and monitoring, bias, and accountability are some of the ethical concerns.

Leveraging the Academic Artificial Intelligence Silecosystem to Advance the Community Oncology Enterprise

Over the last 75 years, artificial intelligence has evolved from a theoretical concept and novel paradigm describing the role that computers might play in our society to a tool with which we daily engage. In this review, we describe AI in terms of its constituent elements, the synthesis of which we refer to as the AI Silecosystem. Herein, we provide an historical perspective of the evolution of the AI Silecosystem, conceptualized and summarized as a Kuhnian paradigm. This manuscript focuses on the role that the AI Silecosystem plays in oncology and its emerging importance in the care of the community oncology patient. We observe that this important role arises out of a unique alliance between the academic oncology enterprise and community oncology practices. We provide evidence of this alliance by illustrating the practical establishment of the AI Silecosystem at the City of Hope Comprehensive Cancer Center and its team utilization by community oncology providers.

Extended reality for procedural planning and guidance in structural heart disease - a review of the state-of-the-art

Extended reality (XR), which encompasses virtual, augmented and mixed reality, is an emerging medical imaging display platform which enables intuitive and immersive interaction in a three-dimensional space. This technology holds the potential to enhance understanding of complex spatial relationships when planning and guiding cardiac procedures in congenital and structural heart disease moving beyond conventional 2D and 3D image displays. A systematic review of the literature demonstrates a rapid increase in publications describing adoption of this technology. At least 33 XR systems have been described, with many demonstrating proof of concept, but with no specific mention of regulatory approval including some prospective studies. Validation remains limited, and true clinical benefit difficult to measure. This review describes and critically appraises the range of XR technologies and its applications for procedural planning and guidance in structural heart disease while discussing the challenges that need to be overcome in future studies to achieve safe and effective clinical adoption.

Integration of Square Fiducial Markers in Patient-Specific Instrumentation and Their Applicability in Knee Surgery

Computer technologies play a crucial role in orthopaedic surgery and are essential in personalising different treatments. Recent advances allow the usage of augmented reality (AR) for many orthopaedic procedures, which include different types of knee surgery. AR assigns the interaction between virtual environments and the physical world, allowing both to intermingle (AR superimposes information on real objects in real-time) through an optical device and allows personalising different processes for each patient. This article aims to describe the integration of fiducial markers in planning knee surgeries and to perform a narrative description of the latest publications on AR applications in knee surgery. Augmented reality-assisted knee surgery is an emerging set of techniques that can increase accuracy, efficiency, and safety and decrease the radiation exposure (in some surgical procedures, such as osteotomies) of other conventional methods. Initial clinical experience with AR projection based on ArUco-type artificial marker sensors has shown promising results and received positive operator feedback. Once initial clinical safety and efficacy have been demonstrated, the continued experience should be studied to validate this technology and generate further innovation in this rapidly evolving field.

Learning how to perform ultrasound-guided interventions with and without augmented reality visualization: a randomized study

OBJECTIVES

Augmented reality (AR), which entails overlay of in situ images onto the anatomy, may be a promising technique for assisting image-guided interventions. The purpose of this study was to investigate and compare the learning experience and performance of untrained operators in puncture of soft tissue lesions, when using AR ultrasound (AR US) compared with standard US (sUS).

METHODS

Forty-four medical students (28 women, 16 men) who had completed a basic US course, but had no experience with AR US, were asked to perform US-guided biopsies with both sUS and AR US, with a randomized selection of the initial modality. The experimental setup aimed to simulate biopsies of superficial soft tissue lesions, such as for example breast masses in clinical practice, by use of a turkey breast containing olives. Time to puncture(s) and success (yes/no) of the biopsies was documented. All participants completed questionnaires about their coordinative skills and their experience during the training.

RESULTS

Despite having no experience with the AR technique, time to puncture did not differ significantly between AR US and sUS (median [range]: 17.0 s [6-60] and 14.5 s [5-41], p = 0.16), nor were there any gender-related differences (p = 0.22 and p = 0.50). AR US was considered by 79.5% of the operators to be the more enjoyable means of learning and performing US-guided biopsies. Further, a more favorable learning curve was achieved using AR US.

CONCLUSIONS

Students considered AR US to be the preferable and more enjoyable modality for learning how to obtain soft tissue biopsies; however, they did not perform the biopsies faster than when using sUS.

KEY POINTS

• Performance of standard and augmented reality US-guided biopsies was comparable • A more favorable learning curve was achieved using augmented reality US. • Augmented reality US was the preferred technique and was considered more enjoyable.

Translation of Medical AR Research into Clinical Practice

Translational research is aimed at turning discoveries from basic science into results that advance patient treatment. The translation of technical solutions into clinical use is a complex, iterative process that involves different stages of design, development, and validation, such as the identification of unmet clinical needs, technical conception, development, verification and validation, regulatory matters, and ethics. For this reason, many promising technical developments at the interface of technology, informatics, and medicine remain research prototypes without finding their way into clinical practice. Augmented reality is a technology that is now making its breakthrough into patient care, even though it has been available for decades. In this work, we explain the translational process for Medical AR devices and present associated challenges and opportunities. To the best knowledge of the authors, this concept paper is the first to present a guideline for the translation of medical AR research into clinical practice.

Next step trauma and orthopaedic surgery: integration of augmented reality for reduction and nail implantation of tibial fractures

INTRODUCTION

There is a tremendous scope of hardware and software development going on in augmented reality (AR), also in trauma and orthopaedic surgery. However, there are only a few systems available for intra-operative 3D imaging and guidance, most of them rely on peri- and intra-operative X-ray imaging. Especially in complex situations such as pelvic surgery or multifragmentary multilevel fractures, intra-operative 3D imaging and implant tracking systems have proven to be of great advantage for the outcome of the surgery and can help reduce X-ray exposure, at least for the surgical team (Ochs et al. in Injury 41:1297 1305, 2010). Yet, the current systems do not provide the ability to have a dynamic live view from the perspective of the surgeon. Our study describes a prototype AR-based system for live tracking which does not rely on X-rays.

MATERIALS AND METHODS

A protype live-view intra-operative guidance system using an AR head-mounted device (HMD) was developed and tested on the implantation of a medullary nail in a tibia fracture model. Software algorithms that allow live view and tracking of the implant, fracture fragments and soft tissue without the intra-operative use of X-rays were derived.

RESULTS

The implantation of a medullar tibia nail is possible while only relying on AR-guidance and live view without the intra-operative use of X-rays.

CONCLUSIONS

The current paper describes a feasibility study with a prototype of an intra-operative dynamic live tracking and imaging system that does not require intra-operative use of X-rays and dynamically adjust to the perspective of the surgeons due to an AR HMD. To our knowledge, the current literature does not describe any similar systems. This could be the next step in surgical imaging and education and a promising way to improve patient care.

Augmented Reality Navigation Can Achieve Accurate Coronal Component Alignment During Total Knee Arthroplasty

Background Computer-navigated knee arthroplasty has been shown to improve accuracy over conventional instruments. The next generation of computer assistance is being developed using augmented reality. The accuracy of augmented reality navigation has not been established. Methods From April 2021 to October 2021, a prospective, consecutive series of 20 patients underwent total knee arthroplasty utilising an augmented reality-assisted navigation system (ARAN). The coronal and sagittal alignment of the femoral and tibial bone cuts was measured using the ARAN and the final position of the components was measured on postoperative CT scans. The absolute difference between the measurements was recorded to determine the accuracy of the ARAN. Results Two cases were excluded due to segmentation errors, leaving 18 cases for analysis. The ARAN produced a mean absolute error of 1.4°, 2.0°, 1.1° and 1.6° for the femoral coronal, femoral sagittal, tibial coronal and tibial sagittal alignments, respectively. No outliers (absolute error of >3°) were identified in femoral coronal or tibial coronal alignment measurements. Three outliers were identified in tibial sagittal alignment, with all cases demonstrating less tibial slope (by 3.1°, 3.3° and 4°). Five outliers were identified in femoral sagittal alignment and in all cases, the component was more extended (3.1°, 3.2°, 3.2°, 3.4° and 3.9°). The mean operative time significantly decreased from the first nine augmented reality cases to the final nine cases by 11 minutes (p<0.05). There was no difference in the accuracy between the early and late ARAN cases. Conclusion Augmented reality navigation can achieve accurate alignment of total knee arthroplasty with a low rate of component malposition in the coronal plane. Acceptable and consistent accuracy can be achieved from the initial adoption of this technique, however, some sagittal outliers were identified and there is a clear learning curve with respect to operating time. The level of evidence was IV.

Augmented Reality-Assisted Navigation System for Transforaminal Epidural Injection

Purpose

Multiple studies have attempted to demonstrate the benefits of augmented reality (AR)-assisted navigation systems in surgery. Lumbosacral transforaminal epidural injection is an effective treatment commonly used in patients with radiculopathy due to spinal degenerative pathologies. However, few studies have applied AR-assisted navigation systems to this procedure. The study aimed to investigate the safety and effectiveness of an AR-assisted navigation system for transforaminal epidural injection.

Patients and Methods

Through a real-time tracking system and a wireless network to the head-mounted display, computed tomography images of the spine and the path of a spinal needle to the target were visualized on a torso phantom with respiration movements installed. From L1/L2 to L5/S1, needle insertions were performed using an AR-assisted system on the left side of the phantom, and the conventional method was performed on the right side.

Results

The procedure duration was approximately three times shorter, and the number of radiographs required was reduced in the experimental group compared to the control group. The distance from the needle tips to the target areas in the plan showed no significant difference between the two groups. (AR group 1.7 ± 2.3mm, control group 3.2 ± 2.8mm, P value 0.067).

Conclusion

An AR-assisted navigation system may be used to reduce the time required for spinal interventions and ensure the safety of patients and physicians in view of radiation exposure. Further studies are essential to apply AR-assisted navigation systems to spine interventions.

Development of Intraoperative Plantar Pressure Measurement System Considering Weight Bearing Axis and Center of Pressure

To prevent postoperative complications in corrective surgery for foot deformities such as hallux valgus and pes planus, it is critical to quantitatively predict the postoperative standing-position plantar pressure distribution during the operation. The authors have previously proposed an intraoperative plantar pressure measurement system (IPPM) that allows for the measurement of a supine patient’s plantar pressure distribution that is equivalent to that in the standing position. This system consists of an IPPM device comprising of a force plate and pressure distribution sensor, an optical three-dimensional position measurement device, a navigation monitor, and a PC. The plantar pressure distribution in the standing position is reproduced by navigating the operator, as he or she presses the IPPM device against the patient’s sole so that the weight-bearing axis (floor reaction force vector) and femoral head center are as close to each other as possible. However, in our previous study, the reproducibility of the standing position plantar pressure distribution was insufficient. Therefore, in the present study, we add a navigational function that can be used to bring the centers of pressure in the standing position and under measurement, as well as to correct the IPPM’s self-weight in the measured force. The improved device was used in an experiment with nine healthy subjects, and the similarity of the plantar pressure distribution in the standing and supine positions was evaluated using normalized cross-correlation, yielding an average of 0.90. Furthermore, in an evaluation experiment with ten orthopedic surgeons, it was observed that using the system reproduced the plantar pressure distribution significantly better than when the system was not used. These results indicate that the present system can predict the plantar pressure distribution in the standing position. We believe that this system can contribute to reducing complications after foot surgery.

Mixed Reality Interfaces for Achieving Desired Views with Robotic X-ray Systems

Robotic X-ray C-arm imaging systems can precisely achieve any position and orientation relative to the patient. Informing the system, however, what pose exactly corresponds to a desired view is challenging. Currently these systems are operated by the surgeon using joysticks, but this interaction paradigm is not necessarily effective because users may be unable to efficiently actuate more than a single axis of the system simultaneously. Moreover, novel robotic imaging systems, such as the Brainlab Loop-X, allow for independent source and detector movements, adding even more complexity. To address this challenge, we consider complementary interfaces for the surgeon to command robotic X-ray systems effectively. Specifically, we consider three interaction paradigms: (1) the use of a pointer to specify the principal ray of the desired view relative to the anatomy, (2) the same pointer, but combined with a mixed reality environment to synchronously render digitally reconstructed radiographs from the tool's pose, and (3) the same mixed reality environment but with a virtual X-ray source instead of the pointer. Initial human-in-the-loop evaluation with an attending trauma surgeon indicates that mixed reality interfaces for robotic X-ray system control are promising and may contribute to substantially reducing the number of X-ray images acquired solely during "fluoro hunting" for the desired view or standard plane.

The Role of Augmented Reality in the Next Phase of Surgical Education

Concomitant with such a shift toward competency-based curricula, there has been increasing adoption of surgical simulation coupled with virtual, mixed, and augmented reality. These technologies have become more commonplace across multiple surgical disciplines, in domains such as preoperative planning, surgical education, and intraoperative navigation. However, there is a relative paucity of literature pertaining to the application of this technology to plastic surgery education. This review outlines the advantages of mixed and augmented reality in the pursuit of an ideal simulation environment, their benefits for the education of plastic surgery trainees, and their role in standardized assessments. In addition, we offer practical solutions to commonly encountered problems with this technology. Augmented reality has tremendous untapped potential in the next phase of plastic surgery education, and we outline steps toward broader implementation to enhance the learning environment for our trainees and to improve patient outcomes.

Preoperative templating in orthopaedic fracture surgery: The past, present and future

Preoperative planning in orthopaedic fracture surgery corroborates with the goal of establishing the best possible surgical result and ensuring a functioning limb for the patient. From placing sketches on overhead projector paper and measuring lengths from anatomical landmarks, ways of preoperative planning have evolved rapidly over the last 100 years. Today, preoperative planning includes methods such as advanced 3-Dimensional (3D) printed models and software programs incorporating entire libraries of osteosynthesis materials that can be shaped and rotated to fit a patient's specific anatomy. Relevant literature was evaluated to review the development of preoperative templating from the past and present, in order to assess its impact on the future of osteosynthesis.We identified studies on 3D-imaging, computer-assisted systems, and 3D-printed fractured bones and drill guides. The use of some of these systems resulted in a reduction in operation time, blood loss, perioperative fluoroscopy and hospital stay, as well as better placement of osteosynthesis material. Only few studies have identified differences in patient morbidity and mortality. Future techniques of preoperative templating are on the rise and the potential is vast. The cost-effectiveness and usefulness of certain methods need to be evaluated further, but the benefit of preoperative templating has the potential of being revolutionary, with the possibility of radical advances within orthopaedic surgery.

Trends of Augmented Reality for Agri-Food Applications

Recent years have witnessed an increasing interest in deploying state-of-the-art augmented reality (AR) head-mounted displays (HMDs) for agri-food applications. The benefits of AR HMDs to agri-food industry stakeholders (e.g., food suppliers, retail/food service) have received growing attention and recognition. AR HMDs enable users to make healthier dietary choices, experience novel changes in their perception of taste, enhance the cooking and food shopping experience, improve productivity at work and enhance the implementation of precision farming. Therefore, although development costs are still high, the case for integration of AR in food chains appears to be compelling. This review will present the most recent developments of AR HMDs for agri-food relevant applications. The summarized applications can be clustered into different themes: (1) dietary and food nutrition assessment; (2) food sensory science; (3) changing the eating environment; (4) retail food chain applications; (5) enhancing the cooking experience; (6) food-related training and learning; and (7) food production and precision farming. Limitations of current practices will be highlighted, along with some proposed applications.

Augmented Reality in Surgery: A Scoping Review

Augmented reality (AR) is an innovative system that enhances the real world by superimposing virtual objects on reality. The aim of this study was to analyze the application of AR in medicine and which of its technical solutions are the most used. We carried out a scoping review of the articles published between 2019 and February 2022. The initial search yielded a total of 2649 articles. After applying filters, removing duplicates and screening, we included 34 articles in our analysis. The analysis of the articles highlighted that AR has been traditionally and mainly used in orthopedics in addition to maxillofacial surgery and oncology. Regarding the display application in AR, the Microsoft HoloLens Optical Viewer is the most used method. Moreover, for the tracking and registration phases, the marker-based method with a rigid registration remains the most used system. Overall, the results of this study suggested that AR is an innovative technology with numerous advantages, finding applications in several new surgery domains. Considering the available data, it is not possible to clearly identify all the fields of application and the best technologies regarding AR.

Surgical Treatment of Bone Sarcoma

Bone sarcomas are rare primary malignant mesenchymal bone tumors. The three main entities are osteosarcoma, chondrosarcoma, and Ewing sarcoma. While prognosis has improved for affected patients over the past decades, bone sarcomas are still critical conditions that require an interdisciplinary diagnostic and therapeutic approach. While radiotherapy plays a role especially in Ewing sarcoma and chemotherapy in Ewing sarcoma and osteosarcoma, surgery remains the main pillar of treatment in all three entities. After complete tumor resection, the created bone defects need to be reconstructed. Possible strategies are implantation of allografts or autografts including vascularized bone grafts (e.g., of the fibula). Around the knee joint, rotationplasty can be performed or, as an alternative, the implantation of (expandable) megaprostheses can be performed. Challenges still associated with the implantation of foreign materials are aseptic loosening and infection. Future improvements may come with advances in 3D printing of individualized resection blades/implants, thus also securing safe tumor resection margins while at the same time shortening the required surgical time. Faster osseointegration and lower infection rates may possibly be achieved through more elaborate implant surface structures.

What is Significant in Modern Augmented Reality: A Systematic Analysis of Existing Reviews

Augmented reality (AR) is a field of technology that has evolved drastically during the last decades, due to its vast range of applications in everyday life. The aim of this paper is to provide researchers with an overview of what has been surveyed since 2010 in terms of AR application areas as well as in terms of its technical aspects, and to discuss the extent to which both application areas and technical aspects have been covered, as well as to examine whether one can extract useful evidence of what aspects have not been covered adequately and whether it is possible to define common taxonomy criteria for performing AR reviews in the future. To this end, a search with inclusion and exclusion criteria has been performed in the Scopus database, producing a representative set of 47 reviews, covering the years from 2010 onwards. A proper taxonomy of the results is introduced, and the findings reveal, among others, the lack of AR application reviews covering all suggested criteria.

Augmented Reality in Arthroplasty: An Overview of Clinical Applications, Benefits, and Limitations

Augmented reality (AR) is a natural extension of computer-assisted surgery whereby a computer-generated image is superimposed on the surgeon's field of vision to assist in the planning and execution of the procedure. This emerging technology shows great potential in the field of arthroplasty, improving efficiency, limb alignment, and implant position. AR has shown the capacity to build on computer navigation systems while providing more elaborate information in a streamlined workflow to the user. This review investigates the current uses of AR in the field of arthroplasty and discusses outcomes, limitations, and potential future directions.

Augmented Reality: Mapping Methods and Tools for Enhancing the Human Role in Healthcare HMI

Background: Augmented Reality (AR) represents an innovative technology to improve data visualization and strengthen the human perception. Among Human–Machine Interaction (HMI), medicine can benefit most from the adoption of these digital technologies. In this perspective, the literature on orthopedic surgery techniques based on AR was evaluated, focusing on identifying the limitations and challenges of AR-based healthcare applications, to support the research and the development of further studies. Methods: Studies published from January 2018 to December 2021 were analyzed after a comprehensive search on PubMed, Google Scholar, Scopus, IEEE Xplore, Science Direct, and Wiley Online Library databases. In order to improve the review reporting, the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines were used. Results: Authors selected sixty-two articles meeting the inclusion criteria, which were categorized according to the purpose of the study (intraoperative, training, rehabilitation) and according to the surgical procedure used. Conclusions: AR has the potential to improve orthopedic training and practice by providing an increasingly human-centered clinical approach. Further research can be addressed by this review to cover problems related to hardware limitations, lack of accurate registration and tracking systems, and absence of security protocols.

Multi-Stage Platform for (Semi-)Automatic Planning in Reconstructive Orthopedic Surgery

Intricate lesions of the musculoskeletal system require reconstructive orthopedic surgery to restore the correct biomechanics. Careful pre-operative planning of the surgical steps on 2D image data is an essential tool to increase the precision and safety of these operations. However, the plan’s effectiveness in the intra-operative workflow is challenged by unpredictable patient and device positioning and complex registration protocols. Here, we develop and analyze a multi-stage algorithm that combines deep learning-based anatomical feature detection and geometric post-processing to enable accurate pre- and intra-operative surgery planning on 2D X-ray images. The algorithm allows granular control over each element of the planning geometry, enabling real-time adjustments directly in the operating room (OR). In the method evaluation of three ligament reconstruction tasks effect on the knee joint, we found high spatial precision in drilling point localization (ε<2.9mm) and low angulation errors for k-wire instrumentation (ε<0.75∘) on 38 diagnostic radiographs. Comparable precision was demonstrated in 15 complex intra-operative trauma cases suffering from strong implant overlap and multi-anatomy exposure. Furthermore, we found that the diverse feature detection tasks can be efficiently solved with a multi-task network topology, improving precision over the single-task case. Our platform will help overcome the limitations of current clinical practice and foster surgical plan generation and adjustment directly in the OR, ultimately motivating the development of novel 2D planning guidelines.

"Augmented reality" applications in urology: a systematic review

INTRODUCTION

Augmented reality (AR) applied to surgical procedures refers to the superimposition of preoperative or intra-operative images onto the operative field. Augmented reality has been increasingly used in myriad surgical specialties including Urology. The following study reviews advances in the use of AR for improvements in urologic outcomes.

EVIDENCE ACQUISITION

We identified all descriptive, validity, prospective randomized/nonrandomized trials and retrospective comparative/noncomparative studies about the use of AR in Urology up until March 2021. The MEDLINE, Scopus, and Web of Science databases were used for literature search. We conducted the study selection according to the PRISMA (Preferred Reporting Items for Systematic Reviews and meta-analysis statement) guidelines. We limited included studies to only those using AR, excluding all that used virtual reality technology.

EVIDENCE SYNTHESIS

A total of 60 studies were identified and included in the present analysis. Overall, 19 studies were descriptive/validity/phantom studies for specific AR methodologies, 4 studies were case reports, and 37 studies included clinical prospective/retrospective comparative studies.

CONCLUSIONS

Advances in AR have led to increasing registration accuracy as well as increased ability to identify anatomic landmarks and improve outcomes during Urologic procedures such as RARP and robot-assisted partial nephrectomy.

Immersive technologies for total knee arthroplasty surgical education

The need to adapt surgical curricula to meet the demands of an increasingly restrictive training environment is rising. Modern constraints of surgical trainees including work-hour restrictions and concerns surrounding patient safety have created an opportunity to supplement traditional teaching methods with developing immersive technologies including virtual and augmented reality. Virtual reality (VR) and augmented reality (AR) have been preliminarily investigated as it relates to total joint arthroplasty. The purpose of this article is to discuss VR and AR as it applies to modern total knee replacement (TKR) surgical education.

Augmented Reality-Assisted Femoral Bone Resection in Total Knee Arthroplasty

An augmented reality (AR)-based navigation system allows visualization of the center of the femoral head and femoral mechanical axis superimposed on the surgical field during total knee arthroplasty (TKA) and may help surgeons to improve the accuracy of distal femoral resection.

The use of augmented reality for limb and component alignment in total knee arthroplasty: systematic review of the literature and clinical pilot study

PURPOSE

A systematic review of the literature has been carried out to assess the actual evidence of the use of augmented reality in total knee arthroplasty (TKA). We then conducted a pilot clinical study to examine the accuracy of the Knee + augmented reality navigation in performing TKA. The present augmented reality (AR) system allows the surgeon to view the tibial and femur axis superimposed on the surgical field through the smart glasses. It provides real-time information during surgery and intraoperative feedback.

METHODS

A systematic review of the PubMed, MEDLINE, and Embase databases up to May 2021 using the keywords "augmented reality", "knee arthroplasty", "computer assisted surgery", "navigation knee arthroplasty" was performed by two independent reviewers. We performed five TKAs using the Knee + system. Patients were 4 females, with mean age of 76.4 years old (range 73-79) and mean Body Max Index (BMI) of 31.9 kg/m2 (range 27-35). The axial alignment of the limb and the orientation of the components were evaluated on standardized pre and postoperative full leg length weight-bearing radiographs, anteroposterior radiographs, and lateral radiographs of the knee. The time of tourniquet was recorded. The perception of motion sickness was assessed by Virtual Reality Sickness Questionnaire (VRSQ) subjected to surgeon immediately after surgery.

RESULTS

After duplicate removal, a total of 31 abstracts were found. However, only two studies concerned knee arthroplasty. Unfortunately, both were preclinical studies. Knee + system is able to perform a cutting error of less than 1° of difference about coronal alignment of femur and tibia and less than 2° about flexion/extension of femur and posterior tibial slope. The absolute differences between the values obtained during surgery and the measurement of varus femur, varus tibia, posterior slope, and femur flexion angle on post-operative radiographs were 0.6° ± 1.34°, 0.8° ± 0.84°, 0.8° ± 1.79°, and 0.4 mm ± 0.55 mm, respectively.

CONCLUSIONS

On light of our preliminary results, the Knee + system is accurate and effective to perform TKA. The translation from pilot study to high-level prospective studies is warranted to assess accuracy and cost-effective analysis compared to conventional techniques.

LEVEL OF EVIDENCE

IV.

Head-Mounted Augmented Reality Platform for Markerless Orthopaedic Navigation

Visual augmented reality (AR) has the potential to improve the accuracy, efficiency and reproducibility of computer-assisted orthopaedic surgery (CAOS). AR Head-mounted displays (HMDs) further allow non-eye-shift target observation and egocentric view. Recently, a markerless tracking and registration (MTR) algorithm was proposed to avoid the artificial markers that are conventionally pinned into the target anatomy for tracking, as their use prolongs surgical workflow, introduces human-induced errors, and necessitates additional surgical invasion in patients. However, such an MTR-based method has neither been explored for surgical applications nor integrated into current AR HMDs, making the ergonomic HMD-based markerless AR CAOS navigation hard to achieve. To these aims, we present a versatile, device-agnostic and accurate HMD-based AR platform. Our software platform, supporting both video see-through (VST) and optical see-through (OST) modes, integrates two proposed fast calibration procedures using a specially designed calibration tool. According to the camera-based evaluation, our AR platform achieves a display error of 6.31 2.55 arcmin for VST and 7.72 3.73 arcmin for OST. A proof-of-concept markerless surgical navigation system to assist in femoral bone drilling was then developed based on the platform and Microsoft HoloLens 1. According to the user study, both VST and OST markerless navigation systems are reliable, with the OST system providing the best usability. The measured navigation error is 4.90 1.04 mm, 5.96 2.22 for VST system and 4.36 0.80 mm, 5.65 1.42 for OST system.

Augmented Reality in Industry 4.0 and Future Innovation Programs

Augmented Reality (AR) is worldwide recognized as one of the leading technologies of the 21st century and one of the pillars of the new industrial revolution envisaged by the Industry 4.0 international program. Several papers describe, in detail, specific applications of Augmented Reality developed to test its potentiality in a variety of fields. However, there is a lack of sources detailing the current limits of this technology in the event of its introduction in a real working environment where everyday tasks could be carried out by operators using an AR-based approach. A literature analysis to detect AR strength and weakness has been carried out, and a set of case studies has been implemented by authors to find the limits of current AR technologies in industrial applications outside the laboratory-protected environment. The outcome of this paper is that, even though Augmented Reality is a well-consolidated computer graphic technique in research applications, several improvements both from a software and hardware point of view should be introduced before its introduction in industrial operations. The originality of this paper lies in the detection of guidelines to improve the Augmented Reality potentialities in factories and industries.

Computer-assisted open reduction internal fixation of intraarticular radius fractures navigated with patient-specific instrumentation, a prospective case series

BACKGROUND

Intra-articular fractures are associated with posttraumatic arthritis if inappropriately treated. Exact reduction of the joint congruency is the main factor to avoid the development of arthrosis. Aim of this study was to evaluate feasibility of computer-assisted surgical planning and 3D-printed patient-specific instrumentation (PSI) for treatment of distal intraarticular radius fractures.

METHOD

7 Patients who suffered a distal intraarticular radius fracture were enrolled in this prospective case series. Preoperative CT-scan was recorded, whereupon a 3D model was computed for surgical planning and design of PSI for surgical navigation. Postoperative accuracy and joint congruency were assessed. Patients were followed-up 3, 6 and 12 months postoperatively.

RESULTS

Mean follow-up was 16 months. Over all range of motion was restored and flexion, extension and pronation showed significant recovery, p < 0.05. Biggest intraarticular joint step-off and gap reduced from average 2.49 (± 1.04) to 0.8 mm (± 0.44), p < 0.05 and 6.12 mm (± 1.04) to 2.21 mm (± 1.16), p < 0.05. Average grip strength restored (3-16 months) from 20.33 kg (± 7.12) to 39.3 kg (± 19.55) p < 0.05, 100% of the healthy contralateral side. 3D-accuracy for guided fragments was 2.07 mm (± 0.64) and 8.59° (± 2.9) and 2.33 mm (± 0.69) and 12.86° (± 7.13), p > 0.05 for fragments reduced with ligamentotaxis.

CONCLUSION

Computer-assisted and PSI navigated intraarticular radius fracture treatment is feasible, safe and accurate. The benefits of this method, however, do not outstand the additional effort.

LEVEL OF EVIDENCE

IV.