Mixed Reality in Visceral Surgery: Development of a Suitable Workflow and Evaluation of Intraoperative Use-cases

A decade of progress: bringing mixed reality image-guided surgery systems in the operating room

Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.

The use of mixed reality in the preoperative planning of colorectal surgery: Preliminary experience with a narrative review

New advanced technologies have recently been developed and preliminarily applied to surgery, including virtual reality (VR), augmented reality (AR) and mixed reality (MR). We retrospectively review all colorectal cases in which we used holographic 3D reconstruction from February 2020 to December 2022. This innovative approach was used to identify vascular anomalies, pinpoint tumor locations, evaluate infiltration into neighboring organs and devise surgical plans for both training and educating trainee assistants. We have also provided a state-of-the-art analysis, briefly highlighting what has been stated by the scientific literature to date. VR facilitates training and anatomical assessments, while AR enhances training and laparoscopic performance evaluations. MR, powered by HoloLens, enriches anatomic recognition, navigation, and visualization. Successful implementation was observed in 10 colorectal cancer cases, showcasing the effectiveness of MR in improving preoperative planning and its intraoperative application. This technology holds significant promise for advancing colorectal surgery by elevating safety and reliability standards.

Intraoperative liver deformation and organ motion caused by ventilation, laparotomy, and pneumoperitoneum in a porcine model for image-guided liver surgery

BACKGROUND

Image-guidance promises to make complex situations in liver interventions safer. Clinical success is limited by intraoperative organ motion due to ventilation and surgical manipulation. The aim was to assess influence of different ventilatory and operative states on liver motion in an experimental model.

METHODS

Liver motion due to ventilation (expiration, middle, and full inspiration) and operative state (native, laparotomy, and pneumoperitoneum) was assessed in a live porcine model (n = 10). Computed tomography (CT)-scans were taken for each pig for each possible combination of factors. Liver motion was measured by the vectors between predefined landmarks along the hepatic vein tree between CT scans after image segmentation.

RESULTS

Liver position changed significantly with ventilation. Peripheral regions of the liver showed significantly higher motion (maximal Euclidean motion 17.9 ± 2.7 mm) than central regions (maximal Euclidean motion 12.6 ± 2.1 mm, p < 0.001) across all operative states. The total average motion measured 11.6 ± 0.7 mm (p < 0.001). Between the operative states, the position of the liver changed the most from native state to pneumoperitoneum (14.6 ± 0.9 mm, p < 0.001). From native state to laparotomy comparatively, the displacement averaged 9.8 ± 1.2 mm (p < 0.001). With pneumoperitoneum, the breath-dependent liver motion was significantly reduced when compared to other modalities. Liver motion due to ventilation was 7.7 ± 0.6 mm during pneumoperitoneum, 13.9 ± 1.1 mm with laparotomy, and 13.5 ± 1.4 mm in the native state (p < 0.001 in all cases).

CONCLUSIONS

Ventilation and application of pneumoperitoneum caused significant changes in liver position. Liver motion was reduced but clearly measurable during pneumoperitoneum. Intraoperative guidance/navigation systems should therefore account for ventilation and intraoperative changes of liver position and peripheral deformation.

Collaborative Virtual Reality Real-Time 3D Image Editing for Chest Wall Resections and Reconstruction Planning

The integration of extended reality (XR) technologies into health care procedures presents transformative opportunities, particularly in surgical processes. This study delves into the utilization of virtual reality (VR) for preoperative planning related to chest wall resections in thoracic surgery. Leveraging the capabilities of 3-dimensional (3D) imaging, real-time visualization, and collaborative VR environments, surgeons gain enhanced anatomical insights and can develop predictive surgical strategies. Two clinical cases highlighted the effectiveness of this approach, showcasing the potential for personalized and intricate surgical planning. The setup provides an immersive, dynamic representation of real patient data, enabling collaboration among teams from separate locations. While VR offers enhanced interactive and visualization capabilities, preliminary evidence suggests it may support more refined preoperative strategies, potentially influence postoperative outcomes, and optimize resource management. However, its comparative advantage over traditional methods needs further empirical validation. Emphasizing the potential of XR, this exploration suggests its broad implications in thoracic surgery, especially when dealing with complex cases requiring multidisciplinary collaboration in the immersive virtual space, often referred to as the metaverse. This innovative approach necessitates further examination, marking a shift toward future surgical preparations. In this article, we sought to demonstrate the technique of an immersive real-time volume-rendered collaborative VR-planning tool using exemplary case studies in chest wall surgery.

Mixed reality in primary retroperitoneal tumour surgery: Evaluation of preoperative and intraoperative application value

OBJECTIVE

To evaluate the feasibility and application value of mixed reality technology (MR) in Primary retroperitoneal tumour (PRT) surgery.

METHODS

From 276 patients who underwent PRT resection at the First Affiliated Hospital of Xi'an Jiaotong University, we screened 46 patients who underwent MR-assisted retroperitoneal tumour resection and 46 patients who underwent tumour resection without MR assistance. The intraoperative and postoperative recovery of the patients in both groups were compared, and the reliability and validity of the application of MR were further examined using the Likert scale.

RESULTS

There was a significant difference in the mean intraoperative bleeding volume between the two groups, but it was reduced in the MR group. The results of the Likert scale showed higher scores in the MR group than non-MR group.

CONCLUSIONS

MR can be used to assist PRT resection and has great potential to improve the rate of complete retroperitoneal tumour resection.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Visualization, registration and tracking techniques for augmented reality guided surgery: a review

Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories ofin situvisualization and nonin situvisualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.

User's image perception improved strategy and application of augmented reality systems in smart medical care: A review

BACKGROUND

Augmented reality (AR) is a new human-computer interaction technology that combines virtual reality, computer vision, and computer networks. With the rapid advancement of the medical field towards intelligence and data visualisation, AR systems are becoming increasingly popular in the medical field because they can provide doctors with clear enough medical images and accurate image navigation in practical applications. However, it has been discovered that different display types of AR systems have different effects on doctors' perception of the image after virtual-real fusion during the actual medical application. If doctors cannot correctly perceive the image, they may be unable to correctly match the virtual information with the real world, which will have a significant impact on their ability to recognise complex structures.

METHODS

This paper uses Citespace, a literature analysis tool, to visualise and analyse the research hotspots when AR systems are used in the medical field.

RESULTS

A visual analysis of the 1163 articles retrieved from the Web of Science Core Collection database reveals that display technology and visualisation technology are the key research directions of AR systems at the moment.

CONCLUSION

This paper categorises AR systems based on their display principles, reviews current image perception optimisation schemes for various types of systems, and analyses and compares different display types of AR systems based on their practical applications in the field of smart medical care so that doctors can select the appropriate display types based on different application scenarios. Finally, the future development direction of AR display technology is anticipated in order for AR technology to be more effectively applied in the field of smart medical care. The advancement of display technology for AR systems is critical for their use in the medical field, and the advantages and disadvantages of various display types should be considered in different application scenarios to select the best AR system.

Mixed Reality in Modern Surgical and Interventional Practice: Narrative Review of the Literature

BACKGROUND

Mixed reality (MR) and its potential applications have gained increasing interest within the medical community over the recent years. The ability to integrate virtual objects into a real-world environment within a single video-see-through display is a topic that sparks imagination. Given these characteristics, MR could facilitate preoperative and preinterventional planning, provide intraoperative and intrainterventional guidance, and aid in education and training, thereby improving the skills and merits of surgeons and residents alike.

OBJECTIVE

In this narrative review, we provide a broad overview of the different applications of MR within the entire spectrum of surgical and interventional practice and elucidate on potential future directions.

METHODS

A targeted literature search within the PubMed, Embase, and Cochrane databases was performed regarding the application of MR within surgical and interventional practice. Studies were included if they met the criteria for technological readiness level 5, and as such, had to be validated in a relevant environment.

RESULTS

A total of 57 studies were included and divided into studies regarding preoperative and interventional planning, intraoperative and interventional guidance, as well as training and education.

CONCLUSIONS

The overall experience with MR is positive. The main benefits of MR seem to be related to improved efficiency. Limitations primarily seem to be related to constraints associated with head-mounted display. Future directions should be aimed at improving head-mounted display technology as well as incorporation of MR within surgical microscopes, robots, and design of trials to prove superiority.

Mixed reality navigation training system for liver surgery based on a high-definition human cross-sectional anatomy data set

OBJECTIVES

This study aims to use the three-dimensional (3D) mixed-reality model of liver, entailing complex intrahepatic systems and to deeply study the anatomical structures and to promote the training, diagnosis and treatment of liver diseases.

METHODS

Vascular perfusion human specimens were used for thin-layer frozen milling to obtain liver cross-sections. The 104-megapixel-high-definition cross sectional data set was established and registered to achieve structure identification and manual segmentation. The digital model was reconstructed and data was used to print a 3D hepatic model. The model was combined with HoloLens mixed reality technology to reflect the complex relationships of intrahepatic systems. We simulated 3D patient specific anatomy for identification and preoperative planning, conducted a questionnaire survey, and evaluated the results.

RESULTS

The 3D digital model and 1:1 transparent and colored model of liver established truly reflected intrahepatic vessels and their complex relationships. The reconstructed model imported into HoloLens could be accurately matched with the 3D model. Only 7.7% participants could identify accessory hepatic veins. The depth and spatial-relationship of intrahepatic structures were better understandable for 92%. The 100%, 84.6%, 69% and 84% believed the 3D models were useful in planning, safer surgical paths, reducing intraoperative complications and training of young surgeons respectively.

CONCLUSIONS

A detailed 3D model can be reconstructed using the higher quality cross-sectional anatomical data set. When combined with 3D printing and HoloLens technology, a novel hybrid-reality navigation-training system for liver surgery is created. Mixed Reality training is a worthy alternative to provide 3D information to clinicians and its possible application in surgery. This conclusion was obtained based on a questionnaire and evaluation. Surgeons with extensive experience in surgical operations perceived in the questionnaire that this technology might be useful in liver surgery, would help in precise preoperative planning, accurate intraoperative identification, and reduction of hepatic injury.

Combining physical and virtual worlds for motor-cognitive training interventions: Position paper with guidelines on technology classification in movement-related research

Efficient movements require intact motor and cognitive function. There is a growing literature on motor-cognitive interventions to improve the overall quality of life of healthy or diseased older people. For such interventions, novel technological advances are crucial not only in terms of motivation but also to improve the user experience in a multi-stimuli world, usually offered as a mixture of real and virtual environments. This article provides a classification system for movement-related research dealing with motor-cognitive interventions performed in different extents of a virtual environment. The classification is divided into three categories: (a) type of digital device with the associated degree of immersiveness provided; (b) presence or absence of a human-computer interaction; and (c) activity engagement during training, defined by activity >1.5 Metabolic Equivalent of task. Since virtual reality (VR) often categorizes different technologies under the same term, we propose a taxonomy of digital devices ranging from computer monitors and projectors to head-mounted VR technology. All immersive technologies that have developed rapidly in recent years are grouped under the umbrella term Extended Reality (XR). These include augmented reality (AR), mixed reality (MR), and VR, as well as all technologies that have yet to be developed. This technology has potential not only for gaming and entertainment, but also for research, motor-cognitive training programs, rehabilitation, telemedicine, etc. This position paper provides definitions, recommendations, and guidelines for future movement-related interventions based on digital devices, human-computer interactions, and physical engagement to use terms more consistently and contribute to a clearer understanding of their implications.

Application of Augmented Reality to Maxillary Resections: A Three-Dimensional Approach to Maxillofacial Oncologic Surgery

In the relevant global context, although virtual reality, augmented reality, and mixed reality have been emerging methodologies for several years, only now have technological and scientific advances made them suitable for revolutionizing clinical care and medical settings through the provision of advanced features and improved healthcare services. Over the past fifteen years, tools and applications using augmented reality (AR) have been designed and tested in the context of various surgical and medical disciplines, including maxillofacial surgery. The purpose of this paper is to show how a marker-less AR guidance system using the Microsoft^® HoloLens 2 can be applied in mandible and maxillary demolition surgery to guide maxillary osteotomies. We describe three mandibular and maxillary oncologic resections performed during 2021 using AR support. In these three patients, we applied a marker-less tracking method based on recognition of the patient's facial profile. The surgeon, using HoloLens 2 smart glasses, could see the virtual surgical planning superimposed on the patient's anatomy. We showed that performing osteotomies under AR guidance is feasible and viable, as demonstrated by comparison with osteotomies performed using CAD-CAM cutting guides. This technology has advantages and disadvantages. However, further research is needed to improve the stability and robustness of the marker-less tracking method applied to patient face recognition.

Selecting the Best Approach for the Treatment of Multiple Non-Metastatic Hepatocellular Carcinoma

According to the Barcelona Clinic Liver Cancer (BCLC) staging system, the optimal strategy for patients with multiple HCC within the Milan Criteria is liver transplantation (LT). However, LT cannot be offered to all the patients due to organ shortages and long waiting lists, as well as because of the advanced disease carrying a high risk of poor outcomes. For early stages, liver resection (LR) or thermal ablation (TA) can be proposed, while trans-arterial chemoembolization (TACE) still remains the treatment of choice for intermediate stages (BCLC-B). Asian guidelines and the National Comprehensive Cancer Network suggest LR for resectable multinodular HCCs, even beyond Milan criteria. In this scenario, a growing body of evidence shows better outcomes after surgical resection when compared with TACE. Trans-arterial radioembolization (TARE) and stereotaxic body radiation therapy (SBRT) can also play an important role in this setting. Furthermore, the role of minimally invasive liver surgery (MILS) specifically for patients with multiple HCC is still not clear. This review aims to summarize current knowledge about the best therapeutical strategy for multiple HCC while focusing on the role of minimally invasive surgery and on the most attractive future perspectives.

Leberchirurgie 4.0 - OP-Planung, Volumetrie, Navigation und Virtuelle Realität

Durch die Optimierung der konservativen Behandlung, die Verbesserung der bildgebenden Verfahren und die Weiterentwicklung der Operationstechniken haben sich das operative Spektrum sowie der Maßstab für die Resektabilität in Bezug auf die Leberchirurgie in den letzten Jahrzehnten deutlich verändert.

Dank zahlreicher technischer Entwicklungen, insbesondere der 3-dimensionalen Segmentierung, kann heutzutage die präoperative Planung und die Orientierung während der Operation selbst, vor allem bei komplexen Eingriffen, unter Berücksichtigung der patientenspezifischen Anatomie erleichtert werden.

Neue Technologien wie 3-D-Druck, virtuelle und augmentierte Realität bieten zusätzliche Darstellungsmöglichkeiten für die individuelle Anatomie. Verschiedene intraoperative Navigationsmöglichkeiten sollen die präoperative Planung im Operationssaal verfügbar machen, um so die Patientensicherheit zu erhöhen.

Dieser Übersichtsartikel soll einen Überblick über den gegenwärtigen Stand der verfügbaren Technologien sowie einen Ausblick in den Operationssaal der Zukunft geben.

Case Report: Application of Mixed Reality Combined With A Surgical Template for Precise Periapical Surgery

Objective

The etiology of apical diseases is diverse, and most are due to incomplete root canal therapy. The common clinical manifestations include gingival abscess, fistula and bone destruction. The currently existing limitation of procedures is that surgeons cannot visually evaluate the surgical areas. We sought to combine mixed reality (MR) technology with a 3-dimensional (3D) printed surgical template to achieve visualization in apical surgery. Notably, no reports have described this application.

Methods

We created visual 3D (V3D) files and transferred them into the HoloLens system. We explained the surgical therapy plan to the patient using a mixed reality head-mounted display (MR-HMD). Then, the 3D information was preliminarily matched with the operative area, and the optimal surgical approach was determined by combining this information with 3D surgical guide plate technology.

Results

We successfully developed a suitable surgical workflow and confirmed the optimal surgical approach from the buccal side. We completely exposed the apical lesion and removed the inflammatory granulation tissue.

Conclusion

We are the first group to use the MR technique in apical surgery. We integrated the MR technique with a 3D surgical template to successfully accomplish the surgery. Desirable outcomes using minimally invasive therapy could be achieved with the MR technique.

Vibrotactile feedback in virtual motor learning: A systematic review

Vibrotactile feedback can be effectively applied to motor (physical) learning in virtual environments, as it can provide task-intrinsic and augmented feedback to users, assisting them in enhancing their motor performance. This review investigates current uses of vibrotactile feedback systems in motor learning applications built upon virtual environments by systematically synthesizing 24 peer-reviewed studies. We aim to understand: (1) the current state of the science of using real-time vibrotactile feedback in virtual environments for aiding the acquisition (or improvement) of motor skills, (2) the effectiveness of using vibrotactile feedback in such applications, and (3) research gaps and opportunities in current technology. We used the Sensing-Analysis-Assessment-Intervention framework to assess the scientific literature in our review. The review identifies several research gaps in current studies, as well as potential design considerations that can improve vibrotactile feedback systems in virtual motor learning applications, including the selection and placement of feedback devices and feedback designs.

Mixed Reality Technology in Total Knee Arthroplasty: An Updated Review With a Preliminary Case Report

Background

Augmented reality and mixed reality have been used to help surgeons perform complex surgeries. With the development of technology, mixed reality (MR) technology has been used to improve the success rate of complex hip arthroplasty due to its unique advantages. At present, there are few reports on the application of MR technology in total knee arthroplasty. We presented a case of total knee arthroplasty with the help of mixed reality technology.

Case Presentation

We presented a case of a 71-year-old woman who was diagnosed with bilateral knee osteoarthritis with varus deformity, especially on the right side. After admission, the right total knee arthroplasty was performed with the assistance of MR technology. Before the operation, the three-dimensional virtual model of the knee joint of the patient was reconstructed for condition analysis, operation plan formulation, and operation simulation. During the operation, the three-dimensional virtual images of the femur and tibia coincided with the real body of the patient, showing the osteotomy plane designed before the operation, which can accurately guide the completion of osteotomy and prosthesis implantation.

Conclusions

As far as we know, this is the first report on total knee arthroplasty under the guidance of mixed reality technology.

Leaping the Boundaries in Laparoscopic Liver Surgery for Hepatocellular Carcinoma

Simple Summary

Recent advances in surgical techniques and perioperative management lead to a redefinition of the actual frontiers of Laparoscopic Liver Resection (LLR) by including patients with more advanced disease. Nonetheless, because of both underlying liver conditions and technical difficulty, LLR for Hepatocellular Carcinoma (HCC) is still considered as a challenging procedure. Specific concerns exist about LLR in cirrhotic patients, posterosuperior segments, giant and multiple tumors, as well as repeat resections. This review focuses on the specific limits of this approach in HCC patients in order to put into practice all the pre- and intra-operative precautions to overcome their boundaries, making this technique the standard of care within high-volume hepatobiliary centers.

Abstract

The minimally invasive approach for hepatocellular carcinoma (HCC) had a slower diffusion compared to other surgical fields, mainly due to inherent peculiarities regarding the risks of uncontrollable bleeding, oncological inadequacy, and the need for both laparoscopic and liver major skills. Recently, laparoscopic liver resection (LLR) has been associated with an improved postoperative course, including reduced postoperative decompensation, intraoperative blood losses, length of hospitalization, and unaltered oncological outcomes, leading to its adoption within international guidelines. However, LLR for HCC still faces several limitations, mainly linked to the impaired function of underlying parenchyma, tumor size and numbers, and difficult tumor position. The aim of this review is to highlight the state of the art and future perspectives of LLR for HCC, focusing on key points for overcoming currents limitations and pushing the boundaries in minimally invasive liver surgery (MILS).

Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review

This article presents a systematic review of optical see-through head mounted display (OST-HMD) usage in augmented reality (AR) surgery applications from 2013 to 2020. Articles were categorised by: OST-HMD device, surgical speciality, surgical application context, visualisation content, experimental design and evaluation, accuracy and human factors of human-computer interaction. 91 articles fulfilled all inclusion criteria. Some clear trends emerge. The Microsoft HoloLens increasingly dominates the field, with orthopaedic surgery being the most popular application (28.6%). By far the most common surgical context is surgical guidance (n=58) and segmented preoperative models dominate visualisation (n=40). Experiments mainly involve phantoms (n=43) or system setup (n=21), with patient case studies ranking third (n=19), reflecting the comparative infancy of the field. Experiments cover issues from registration to perception with very different accuracy results. Human factors emerge as significant to OST-HMD utility. Some factors are addressed by the systems proposed, such as attention shift away from the surgical site and mental mapping of 2D images to 3D patient anatomy. Other persistent human factors remain or are caused by OST-HMD solutions, including ease of use, comfort and spatial perception issues. The significant upward trend in published articles is clear, but such devices are not yet established in the operating room and clinical studies showing benefit are lacking. A focused effort addressing technical registration and perceptual factors in the lab coupled with design that incorporates human factors considerations to solve clear clinical problems should ensure that the significant current research efforts will succeed.

Virtual Reality for Surgical Planning – Evaluation Based on Two Liver Tumor Resections

Purpose

For complex cases, preoperative surgical planning is a standard procedure to ensure patient safety and keep the surgery time to a minimum. Based on the available information, such as MRI or CT images, and prior anatomical knowledge the surgeons create their own mental 3D model of the organ of interest. This is challenging, requires years of training and an inherent uncertainty remains even for experienced surgeons.

Goal

Virtual reality (VR) is by nature excellent in showing spatial relationships through its stereoscopic displays. Therefore, it is well suited to be used to support the understanding of individual anatomy of patient-specific 3D organ models generated from MRI or CT data. Utilizing this potential, we developed a VR surgical planning tool that provides a 3D view of the medical data for better spatial understanding and natural interaction with the data in 3D space. Following a user-centered design process, in this first user study, we focus on usability, usefulness, and target audience feedback. Thereby, we also investigate the individual impact the tool and the 3D presentation of the organ have on the understanding of the 3D structures for the surgical team.

Methods

We employed the VR prototype for surgical planning using a standard VR setup to two real cases of patients with liver tumors who were scheduled for surgery at a University Hospital for Visceral Surgery. Surgeons (N = 4) used the VR prototype before the surgery to plan the procedure in addition to their regular planning process. We used semi-structured interviews before and after the surgery to explore the benefits and pitfalls of VR surgical planning.

Results

The participants used on average 14.3 min (SD = 3.59) to plan the cases in VR. The reported usability was good. Results from the interviews and observations suggest that planning in VR can be very beneficial for surgeons. They reported an improved spatial understanding of the individual anatomical structures and better identification of anatomical variants. Additionally, as the surgeons mentioned an improved recall of the information and better identification of surgical relevant structures, the VR tool has the potential to improve the surgery and patient safety.

Trends in the Use of Augmented Reality, Virtual Reality, and Mixed Reality in Surgical Research: a Global Bibliometric and Visualized Analysis

There have been many major developments in the use of augmented reality (AR), virtual reality (VR), and mixed reality (MR) technologies in the context of global surgical research, yet few reports on the trends in this field have been published to date. This study was therefore designed to explore these worldwide trends in this clinically important field. Relevant studies published from 1 January 2009 through 13 October 2020 were retrieved from the Science Citation Index-Expanded (SCI-E) tool of the Web of Science database. Bibliometric techniques were then used to analyze the resultant data, with visual bibliographic coupling, co-authorship, co-citation, co-occurrence, and publication trend analyses subsequently being conducted with GraphPad Prism 8 and with the visualization of similarities (VOS) software tool. There is no patient and public involved. In total, 6221 relevant studies were incorporated into this analysis. At a high level, clear global annual increases in the number of publications in this field were observed. The USA made the greatest contributions to this field over the studied period, with the highest H-index value, the most citations, and the greatest total link strength for analyzed publications. The country with the highest number of average citations per publication was Scotland. The Surgical Endoscopy And Other Interventional Techniques journal contributed the greatest number of publications in this field. The University of London was the institution that produced the greatest volume of research in this field. Overall, studies could be broadly classified into five clusters: Neurological Research, Surgical Techniques, Technological Products, Rehabilitative Medicine, and Clinical Therapy. The trends detected in the present analysis suggest that the number of global publications pertaining to the use of AR, VR, and MR techniques in surgical research is likely to increase in the coming years. Particular attention should be paid to emerging trends in related fields including MR, extended reality, head-mounted displays, navigation, and holographic images.

Applications of Mixed Reality Technology in Orthopedics Surgery: A Pilot Study

Objective: The aim of this study is to explore the potential of mixed reality (MR) technology in the visualization of orthopedic surgery.

Methods: The visualization system with MR technology is widely used in orthopedic surgery. The system is composed of a 3D imaging workstation, a cloud platform, and an MR space station. An intelligent segmentation algorithm is adopted on the 3D imaging workstation to create a 3D anatomical model with zooming and rotation effects. This model is then exploited for efficient 3D reconstruction of data for computerized tomography (CT) and magnetic resonance imaging (MRI). Additionally, the model can be uploaded to the cloud platform for physical parameter tuning, model positioning, rendering and high-dimensional display. Using Microsoft’s HoloLens glasses in combination with the MR system, we project and view 3D holograms in real time under different clinical scenarios. After each procedure, nine surgeons completed a Likert-scale questionnaire on communication and understanding, spatial awareness and effectiveness of MR technology use. In addition to that, the National Aeronautics and Space Administration Task Load Index (NASA-TLX) is also used to evaluate the workload of MR hologram support.

Results: 1) MR holograms can clearly show the 3D structures of bone fractures, which improves the understanding of different fracture types and the design of treatment plans; 2) Holograms with three-dimensional lifelike dynamic features provide an intuitive communication tool among doctors and also between doctors and patients; 3) During surgeries, a full lesion hologram can be obtained and blended in real time with a patient’s virtual 3D digital model in order to give surgeons superior visual guidance through novel high-dimensional “perspectives” of the surgical area; 4) Hologram-based magnetic navigation improves the accuracy and safety of the screw placement in orthopaedics surgeries; 5) The combination of mixed reality cloud platform and telemedicine system based on 5G provides a new technology platform for telesurgery collaboration. Results of qualitative study encourage the usage of MR technology for orthopaedics surgery. Analysis of the Likert-scale questionnaire shows that MR adds significant value to understanding and communication, spatial awareness, learning and effectiveness. Based on the NASA TLX-scale questionnaire results, mixed reality scored significantly lower under the “mental,” “temporal,” “performance,” and “frustration” categories compared to usual 2D.

Conclusion: The integration of MR technology in orthopaedic surgery reduces the dependence on surgeons’ experience and provides personalized 3D visualization models for accurate diagnosis and treatment of orthopaedic abnormalities. This integration is clearly one of the prominent future development directions in medical surgery.

Accuracy of Mixed Reality Combined With Surgical Navigation Assisted Oral and Maxillofacial Tumor Resection

OBJECTIVE

To evaluate the feasibility and accuracy of mixed reality combined with surgical navigation in oral and maxillofacial tumor surgery.

METHODS

Retrospective analysis of data of seven patients with oral and maxillofacial tumors who underwent surgery between January 2019 and January 2021 using a combination of mixed reality and surgical navigation. Virtual surgical planning and navigation plan were based on preoperative CT datasets. Through IGT-Link port, mixed reality workstation was synchronized with surgical navigation, and surgical planning data were transferred to the mixed reality workstation. Osteotomy lines were marked with the aid of both surgical navigation and mixed reality images visualized through HoloLens. Frozen section examination was used to ensure negative surgical margins. Postoperative CT datasets were obtained 1 week after the surgery, and chromatographic analysis of virtual osteotomies and actual osteotomies was carried out. Patients received standard oncological postoperative follow-up.

RESULTS

Of the seven patients, four had maxillary tumors and three had mandibular tumors. There were total of 13 osteotomy planes. Mean deviation between the planned osteotomy plane and the actual osteotomy plane was 1.68 ± 0.92 mm; the maximum deviation was 3.46 mm. Chromatographic analysis showed error of ≤3 mm for 80.16% of the points. Mean deviations of maxillary and mandibular osteotomy lines were approximate (1.60 ± 0.93 mm vs. 1.86 ± 0.93 mm). While five patients had benign tumors, two had malignant tumors. Mean deviations of osteotomy lines was comparable between patients with benign and malignant tumors (1.48 ± 0.74 mm vs. 2.18 ± 0.77 mm). Intraoperative frozen pathology confirmed negative resection margins in all cases. No tumor recurrence or complications occurred during mean follow-up of 15.7 months (range, 6-26 months).

CONCLUSION

The combination of mixed reality technology and surgical navigation appears to be feasible, safe, and effective for tumor resection in the oral and maxillofacial region.

Artificial intelligence assisted display in thoracic surgery: development and possibilities

In this golden age of rapid development of artificial intelligence (AI), researchers and surgeons realized that AI could contribute to healthcare in all aspects, especially in surgery. The popularity of low-dose computed tomography (LDCT) and the improvement of the video-assisted thoracoscopic surgery (VATS) not only bring opportunities for thoracic surgery but also bring challenges on the way forward. Preoperatively localizing lung nodules precisely, intraoperatively identifying anatomical structures accurately, and avoiding complications requires a visual display of individuals’ specific anatomy for surgical simulation and assistance. With the advance of AI-assisted display technologies, including 3D reconstruction/3D printing, virtual reality (VR), augmented reality (AR), and mixed reality (MR), computer tomography (CT) imaging in thoracic surgery has been fully utilized for transforming 2D images to 3D model, which facilitates surgical teaching, planning, and simulation. AI-assisted display based on surgical videos is a new surgical application, which is still in its infancy. Notably, it has potential applications in thoracic surgery education, surgical quality evaluation, intraoperative assistance, and postoperative analysis. In this review, we illustrated the current AI-assisted display applications based on CT in thoracic surgery; focused on the emerging AI applications in thoracic surgery based on surgical videos by reviewing its relevant researches in other surgical fields and anticipate its potential development in thoracic surgery.

Virtual reality and 3D printing improve preoperative visualization of 3D liver reconstructions-results from a preclinical comparison of presentation modalities and user's preference

Background

Preoperative three-dimensional (3D) reconstructions for liver surgery planning have been shown to be effective in reduction of blood loss and operation time. However, the role of the ‘presentation modality’ is not well investigated. We present the first study to compare 3D PDFs, 3D printed models (PR) and virtual reality (VR) 3D models with regard to anatomical orientation and personal preferences in a high volume liver surgery center.

Methods

Thirty participants, 10 medical students, 10 residents, 5 fellows and 5 hepatopancreatobiliary (HPB) experts, assigned the tumor-bearing segments of 20 different patient’s individual liver reconstructions. Liver models were presented in a random order in all modalities. Time needed to specify the tumor location was recorded. In addition, a score was calculated factoring in correct, wrong and missing segment assignments. Furthermore, standardized test/questionnaires for spatial thinking and seeing, vegetative side effects and usability were completed.

Results

Participants named significantly more correct segments in VR (P=0.040) or PR (P=0.036) compared to PDF. Tumor assignment was significantly shorter with 3D PR models compared to 3D PDF (P<0.001) or VR application (P<0.001). Regardless of the modality, HPB experts were significantly faster (24±8 vs. 35±11 sec; P=0.014) and more often correct (0.87±0.12 vs. 0.83±0.15; P<0.001) than medical students. Test results for spatial thinking and seeing had no influence on time but on correctness of tumor assignment. Regarding usability and user experience the VR application achieved the highest scores without causing significant vegetative symptoms and was also the most preferred method (n=22, 73.3%) because of the multiple functions like scaling and change of transparency. Ninety percent (n=27) stated that this application can positively influence the operation planning.

Conclusions

3D PR models and 3D VR models enable a better and partially faster anatomical orientation than reconstructions presented as 3D PDFs. User’s preferred the VR application over the PR models and PDF. A prospective trial is needed to evaluate the different presentation modalities regarding intra- and postoperative outcomes.

Development of a patient-oriented Hololens application to illustrate the function of medication after myocardial infarction

Aims

Statin treatment is one of the hallmarks of secondary prevention after myocardial infarction. Adherence to statins tends to be difficult and can be improved by patient education. Novel technologies such as mixed reality (MR) expand the possibilities to support this process. To assess if an MR medication-application supports patient education focused on function of statins after myocardial infarction.

Methods and results

A human-centred design-approach was used to develop an MR statin tool for Microsoft HoloLens™. Twenty-two myocardial infarction patients were enrolled; 12 tested the application, 10 patients were controls. Clinical, demographic, and qualitative data were obtained. All patients performed a test on statin knowledge. To test if patients with a higher tendency to become involved in virtual environments affected test outcome in the intervention group, validated Presence- and Immersive Tendency Questionnaires (PQ and ITQ) were used. Twenty-two myocardial infarction patients (ST-elevation myocardial infarction, 18/22, 82%) completed the study. Ten out of 12 (83%) patients in the intervention group improved their statin knowledge by using the MR application (median 8 points, IQR 8). Test improvement was mainly the result of increased understanding of statin mechanisms in the body and secondary preventive effects. A high tendency to get involved and focused in virtual environments was moderately positive correlated with better test improvement (r = 0.57, P < 0.05). The median post-test score in the control group was poor (median 6 points, IQR 4).

Conclusions

An MR statin education application can be applied effectively in myocardial infarction patients to explain statin function and importance.

Evaluation of how users interface with holographic augmented reality surgical scenes: Interactive planning MR-Guided prostate biopsies

BACKGROUND

User interfaces play a vital role in the planning and execution of an interventional procedure. The objective of this study is to investigate the effect of using different user interfaces for planning transrectal robot-assisted MR-guided prostate biopsy (MRgPBx) in an augmented reality (AR) environment.

METHOD

End-user studies were conducted by simulating an MRgPBx system with end- and side-firing modes. The information from the system to the operator was rendered on HoloLens as an output interface. Joystick, mouse/keyboard, and holographic menus were used as input interfaces to the system.

RESULTS

The studies indicated that using a joystick improved the interactive capacity and enabled operator to plan MRgPBx in less time. It efficiently captures the operator's commands to manipulate the augmented environment representing the state of MRgPBx system.

CONCLUSIONS

The study demonstrates an alternative to conventional input interfaces to interact and manipulate an AR environment within the context of MRgPBx planning.

A three-dimensional printing navigational template combined with mixed reality technique for localizing pulmonary nodules

OBJECTIVES

Localizing non-palpable pulmonary nodules is challenging for thoracic surgeons. Here, we investigated the accuracy of three-dimensional (3D) printing technology combined with mixed reality (MR) for localizing ground glass opacity-dominant pulmonary nodules.

METHODS

In this single-arm study, we prospectively enrolled patients with small pulmonary nodules (<2 cm) that required accurate localization. A 3D-printing physical navigational template was designed based on the reconstruction of computed tomography images, and a 3D model was generated through the MR glasses. We set the deviation distance as the primary end point for efficacy evaluation. Clinicopathological and surgical data were obtained for further analysis.

RESULTS

Sixteen patients with 17 non-palpable pulmonary nodules were enrolled in this study. Sixteen nodules were localized successfully (16/17; 94.1%) using this novel approach with a median deviation of 9 mm. The mean time required for localization was 25 ± 5.2 min. For the nodules in the upper/middle and lower lobes, the median deviation was 6 mm (range, 0-12.0) and 16 mm (range, 15.0-20.0), respectively. The deviation difference between the groups was significant (Z = -2.957, P = 0.003). The pathological evaluation of resection margins was negative.

CONCLUSIONS

The 3D printing navigational template combined with MR can be a feasible approach for localizing pulmonary nodules.

Using virtual 3D-models in surgical planning: workflow of an immersive virtual reality application in liver surgery

Purpose

Three-dimensional (3D) surgical planning is widely accepted in liver surgery. Currently, the 3D reconstructions are usually presented as 3D PDF data on regular monitors. 3D-printed liver models are sometimes used for education and planning.

Methods

We developed an immersive virtual reality (VR) application that enables the presentation of preoperative 3D models. The 3D reconstructions are exported as STL files and easily imported into the application, which creates the virtual model automatically. The presentation is possible in “OpenVR”-ready VR headsets. To interact with the 3D liver model, VR controllers are used. Scaling is possible, as well as changing the opacity from invisible over transparent to fully opaque. In addition, the surgeon can draw potential resection lines on the surface of the liver. All these functions can be used in a single or multi-user mode.

Results

Five highly experienced HPB surgeons of our department evaluated the VR application after using it for the very first time and considered it helpful according to the “System Usability Scale” (SUS) with a score of 76.6%. Especially with the subitem “necessary learning effort,” it was shown that the application is easy to use.

Conclusion

We introduce an immersive, interactive presentation of medical volume data for preoperative 3D liver surgery planning. The application is easy to use and may have advantages over 3D PDF and 3D print in preoperative liver surgery planning. Prospective trials are needed to evaluate the optimal presentation mode of 3D liver models.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00423-021-02127-7.

Augmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System

INTRODUCTION

Intraoperative navigation during liver resection remains difficult and requires high radiologic skills because liver anatomy is complex and patient-specific. Augmented reality (AR) during open liver surgery could be helpful to guide hepatectomies and optimize resection margins but faces many challenges when large parenchymal deformations take place. We aimed to experiment a new vision-based AR to assess its clinical feasibility and anatomical accuracy.

PATIENTS AND METHODS

Based on preoperative CT scan 3-D segmentations, we applied a non-rigid registration method, integrating a physics-based elastic model of the liver, computed in real time using an efficient finite element method. To fit the actual deformations, the model was driven by data provided by a single RGB-D camera. Five livers were considered in this experiment. In vivo AR was performed during hepatectomy (n = 4), with manual handling of the livers resulting in large realistic deformations. Ex vivo experiment (n = 1) consisted in repeated CT scans of explanted whole organ carrying internal metallic landmarks, in fixed deformations, and allowed us to analyze our estimated deformations and quantify spatial errors.

RESULTS

In vivo AR tests were successfully achieved in all patients with a fast and agile setup installation (< 10 min) and real-time overlay of the virtual anatomy onto the surgical field displayed on an external screen. In addition, an ex vivo quantification demonstrated a 7.9 mm root mean square error for the registration of internal landmarks.

CONCLUSION

These first experiments of a markerless AR provided promising results, requiring very little equipment and setup time, yet providing real-time AR with satisfactory 3D accuracy. These results must be confirmed in a larger prospective study to definitively assess the impact of such minimally invasive technology on pathological margins and oncological outcomes.

Experiences With Three-dimensional Printing in Complex Liver Surgery

OBJECTIVE

We present a series of cases where we used 3D printing in planning of complex liver surgery.

BACKGROUND

In liver surgery, three-dimensional reconstruction of the liver anatomy, in particular of vascular structures, has shown to be helpful in operation planning. So far, 3D printing has been used for medical applications only rarely.

METHODS AND PATIENTS

From December 2017 to December 2019, in 10 cases where surgery was assumed to be challenging operation planning was performed using full size 3D prints in addition to standard 3 phase CT scans. Models included transparent parenchyma, hepatic veins, vena cava, portal vein, hepatic artery, (biliary tree if requested), and tumors. In 7/10 cases vascular reconstructions were needed during the procedure. Nonstructured feedback of the surgical team revealed that the major benefit was visualization of the critical areas of vascular reconstruction, the expected dimensions of tangential vascular infiltration and the planning of reconstruction. In the multifocal tumors, 3D prints were considered to be helpful for intraoperative orientation to detect metastases and to improve planning of the resection.

CONCLUSIONS

In complex liver surgery with potential need for vascular reconstructions operation planning may be optimized using a 3D printed liver model. Prospective studies are needed to evaluate the clinical impact of 3D printing in liver surgery compared to other 3D visualizations.

Role of artificial intelligence in hepatobiliary and pancreatic surgery

Over the past decade, enhanced preoperative imaging and visualization, improved delineation of the complex anatomical structures of the liver and pancreas, and intra-operative technological advances have helped deliver the liver and pancreatic surgery with increased safety and better postoperative outcomes. Artificial intelligence (AI) has a major role to play in 3D visualization, virtual simulation, augmented reality that helps in the training of surgeons and the future delivery of conventional, laparoscopic, and robotic hepatobiliary and pancreatic (HPB) surgery; artificial neural networks and machine learning has the potential to revolutionize individualized patient care during the preoperative imaging, and postoperative surveillance. In this paper, we reviewed the existing evidence and outlined the potential for applying AI in the perioperative care of patients undergoing HPB surgery.

A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery

Background: Research proves that the apprenticeship model, which is the gold standard for training surgical residents, is obsolete. For that reason, there is a continuing effort toward the development of high-fidelity surgical simulators to replace the apprenticeship model. Applying Virtual Reality Augmented Reality (AR) and Mixed Reality (MR) in surgical simulators increases the fidelity, level of immersion and overall experience of these simulators.Areas covered: The objective of this review is to provide a comprehensive overview of the application of VR, AR and MR for distinct surgical disciplines, including maxillofacial surgery and neurosurgery. The current developments in these areas, as well as potential future directions, are discussed.Expert opinion: The key components for incorporating VR into surgical simulators are visual and haptic rendering. These components ensure that the user is completely immersed in the virtual environment and can interact in the same way as in the physical world. The key components for the application of AR and MR into surgical simulators include the tracking system as well as the visual rendering. The advantages of these surgical simulators are the ability to perform user evaluations and increase the training frequency of surgical residents.

Mixed-reality hologram for diagnosis and surgical planning of double outlet of the right ventricle: a pilot study

AIM

To evaluate the mixed-reality (MR) hologram, a novel technology based on two-dimensional images, which simulates three-dimensional (3D) images and provides a dynamic and interactive alternative, for its usefulness in the diagnosis and surgical planning of double outlet of the right ventricle (DORV).

MATERIALS AND METHODS

Thirty-four patients who were suspected of DORV based on ultrasound findings underwent cardiac computed tomography angiography (CTA). The patients were assigned randomly to the MR holographic guidance (MRHG) group or the control group. For the patients in the MRHG group, the CTA images were converted into Standard Template Library (STL) files after segmentation, 3D reconstruction, colourisation, and transparentisation, and then exported for MR holographic visualisation. The CTA images of the patients in the control group were analysed using routine 3D reconstruction only. Diagnostic accuracy and surgical planning were compared between the two groups based on visualisation at surgery.

RESULTS

In the MRHG group, the 3D hologram observation was in concordance with the actual anatomical findings, and the DORV type was classified accurately in all patients. The diagnostic accuracy for the malformation was 95.5% in the MRHG group and 89.7% in the control group, but the difference was not significant (p=0.3). All the procedures were exactly the same as planned based on the 3D MR holographic model. The surgical planning time was shorter for the MRHG group (51.65 ± 11.11 min) than that for the control group (65.71 ± 18.07 min, p<0.05).

CONCLUSION

MR 3D holograms may provide a clear and deeper anatomical perception of DORV and improve surgical planning.

Mixed reality holograms for heart surgery planning: first user experience in congenital heart disease

AIMS

Proof of concept and feasibility study for preoperative diagnostic use of mixed reality (MR) holograms of individual 3D heart models from standard cardiac computed tomography angiograms (CTA) images. Optimal repair for complex congenital heart disease poses high demands on 3D anatomical imagination. Three-dimensional printed heart models are increasingly used for improved morphological understanding during surgical and interventional planning. Holograms are a dynamic and interactive alternative, probably with wider applications.

METHODS AND RESULTS

A 3D heart model was segmented from CTA images in a patient with double outlet right ventricle and transposition of the great arteries (DORV-TGA). The hologram was visualized in the wearable MR platform HoloLens® for 36 paediatric heart team members who filled out a diagnostic and quality rating questionnaire. Morphological and diagnostic output from the hologram was assessed and the 3D experience was evaluated. Locally developed app tools such as hologram rotation, scaling, and cutting were rated. Anatomy identification and diagnostic output was high as well as rating of 3D experience. Younger and female users rated the app tools higher.

CONCLUSION

This preliminary study demonstrates that MR holograms as surgical planning tool for congenital heart disease may have a high diagnostic value and contribute to understanding complex morphology. The first users experience of the hologram presentation was found to be very positive, with a preference among the female and the younger users. There is potential for improvement of the hologram manipulation tools.

The Short-Term Outcomes of Gastric Cancer Patients Based on a Proposal for a Novel Classification of Perigastric Arteries

PURPOSE

To establish a novel classification of perigastric arteries by computerized tomography angiography (CTA) and discuss its influence in patients' short-term clinical outcomes.

METHODS

The clinical data were analyzed retrospectively from 680 gastric cancer patients. The types of the perigastric artery were classified according to CTA image and we compared the short-term clinical outcomes.

RESULTS

The perigastric arteries can be divided into seven categories. Type I, trifurcation of the celiac trunk (CT) (294/343, 85.7%); type II, hepatosplenic trunk, left gastric artery (LGA) arising from the abdominal aorta (8/343, 2.3%); type III, hepatogastric trunk, splenic artery arising from the superior mesenteric artery (SMA) (2/343, 0.6%); type IV, celiacomesenteric trunk (5/343, 1.5%); type V, common hepatic artery (CHA) arising from the SMA, gastrosplenic trunk (11/343, 3.2%); type VI, aberrant (accessory or replaced) left hepatic artery arising from LGA (21/343, 6.1%); and type VII, CHA arising from LGA (2/343, 0.6%). The number of retrieved LNs in the CTA group was significantly higher than that in the non-CTA group. However, the operation time, estimated blood loss, intraoperative vascular injury, and medical cost of the CTA group were significantly less than those in the non-CTA group. Of note, in patients with BMI ≥ 25.0, higher LNs retrieval and less vascular injury were still present in the CTA group, which was of vital importance in clinical practice. Furthermore, the CTA group displayed shorter hospital stay (LOS).

CONCLUSION

We established a new perigastric artery classification. Application of the classification can improve the short-term clinical outcomes of patients.

Value of the surgeon's sightline on hologram registration and targeting in mixed reality

Purpose

Mixed reality (MR) is being evaluated as a visual tool for surgical navigation. Current literature presents unclear results on intraoperative accuracy using the Microsoft HoloLens 1^®. This study aims to assess the impact of the surgeon’s sightline in an inside-out marker-based MR navigation system for open surgery.

Methods

Surgeons at Akershus University Hospital tested this system. A custom-made phantom was used, containing 18 wire target crosses within its inner walls. A CT scan was obtained in order to segment all wire targets into a single 3D-model (hologram). An in-house software application (CTrue), developed for the Microsoft HoloLens 1, uploaded 3D-models and automatically registered the 3D-model with the phantom. Based on the surgeon’s sightline while registering and targeting (free sightline /F/or a strictly perpendicular sightline /P/), 4 scenarios were developed (FF-PF-FP-PP). Target error distance (TED) was obtained in three different working axes-(XYZ).

Results

Six surgeons (5 males, age 29–62) were enrolled. A total of 864 measurements were collected in 4 scenarios, twice. Scenario PP showed the smallest TED in XYZ-axes mean = 2.98 mm ± SD 1.33; 2.28 mm ± SD 1.45; 2.78 mm ± SD 1.91, respectively. Scenario FF showed the largest TED in XYZ-axes with mean = 10.03 mm ± SD 3.19; 6.36 mm ± SD 3.36; 16.11 mm ± SD 8.91, respectively. Multiple comparison tests, grouped in scenarios and axes, showed that the majority of scenario comparisons had significantly different TED values (p < 0.05). Y-axis always presented the smallest TED regardless of scenario tested.

Conclusion

A strictly perpendicular working sightline in relation to the 3D-model achieves the best accuracy results. Shortcomings in this technology, as an intraoperative visual cue, can be overcome by sightline correction. Incidentally, this is the preferred working angle for open surgery.

[Extended reality technologies for support of surgical workflows]

Current developments in the field of extended reality (XR) could prove useful in the optimization of surgical workflows, time effectiveness and postoperative outcome. Although still primarily a subject of research, the state of XR technologies is rapidly improving and approaching feasibility for a broad clinical application. Surgical fields of application of XR technologies are currently primarily training, preoperative planning and intraoperative assistance. For all three areas, products already exist (some clinically approved) and technical feasibility studies have been conducted. In teaching, the use of XR can already be assessed as fundamentally practical and meaningful but still needs to be evaluated in large multicenter studies. In preoperative planning XR can also offer advantages, although technical limitations often impede routine use; however, for cases of intraoperative use informative evaluation studies are mostly lacking, so that an assessment is not yet possible in a meaningful way. Furthermore, there is a lack of assessments regarding cost-effectiveness in all three areas. The XR technologies enable proven advantages in surgical workflows despite the lack of high-quality evaluation with respect to the practical and clinical use of XR. New concepts for effective interaction with XR media also need to be developed. In the future, further research progress and technical developments in the field can be expected.

Validation and Precision of Mixed Reality Technology in Baha Attract Implant Surgery

OBJECTIVE

To test the feasibility of image-guided Baha Attract implant surgery with mixed reality (MR) in the form of the HoloLens to visualize critical structures and facilitate precise Baha implant placement.

METHODS

A cadaveric case study of bilateral Baha Attract implant approaches was conducted using Star Atlas MR three-dimensional (3D) medical interaction system guidance at the Otolaryngology Department of PUMCH, Beijing, China. The accuracy of visual surface registration was determined by the target registration error (TRE) between the predefined points on the preoperative 3D holographic Baha Attract implant model and the postoperatively reconstructed 3D model.

RESULTS

Bilateral Baha Attract implantation was completed successfully for all four cadaveric heads using the Star Atlas MR 3D medical interaction system with the HoloLens. The preoperative 3D digital model characteristics (including bone quality and thickness and avoidance of cranial vessels, air cells, and cranial sutures) corresponded well with the 3D model of the actual implantation reconstructed postoperatively. The median TRE of our system was 2.97 mm (ranging from 1.98 to 4.58 mm) in terms of distance and 2.76 degrees (ranging from 0.59 to 6.4 degrees) in terms of angle.

CONCLUSIONS

Applying MR technology in the form of the HoloLens in Baha Attract implant surgery is feasible and could improve the accuracy of the surgery. The described MR system for Baha Attract implantation has the potential to improve the surgeon's confidence, as well as the surgical safety, efficiency, and precision.

Mapping the intellectual structure of research on surgery with mixed reality: Bibliometric network analysis (2000-2019)

OBJECTIVE

The purpose of this study is to view research trends on surgery with mixed reality, and present the intellectual structure using bibliometric network analysis for the period 2000-2019.

METHODS

Analyses are implemented in the following four steps: (1) literature dataset acquisition from article database (Web of Science, Scopus, PubMed, and IEEE digital library), (2) dataset pre-processing and refinement, (3) network construction and visualization, and (4) analysis and interpretation. Descriptive analysis, bibliometric network analysis, and in-depth qualitative analysis were conducted.

RESULTS

The 14,591 keywords of 5897 abstracts data were ultimately used to ascertain the intellectual structure of research on surgery with mixed reality. The dynamics of the evolution of keywords in the structure throughout the four periods is summarized with four aspects: (a) maintaining a predominant utilization tool for training, (b) widening clinical application area, (c) reallocating the continuum of mixed reality, and (d) steering advanced imaging and simulation technology.

CONCLUSIONS

The results of this study can provide valuable insights into technology adoption and research trends of mixed reality in surgery. These findings can help clinicians to overview prospective medical research on surgery using mixed reality. Hospitals can also understand the periodical maturity of technology of mixed reality in surgery, and, therefore, these findings can suggest an academic landscape to make a decision in adopting new technologies in surgery.

Ultrasound in augmented reality: a mixed-methods evaluation of head-mounted displays in image-guided interventions

Purpose

Augmented reality (AR) and head-mounted displays (HMD) in medical practice are current research topics. A commonly proposed use case of AR-HMDs is to display data in image-guided interventions. Although technical feasibility has been thoroughly shown, effects of AR-HMDs on interventions are not yet well researched, hampering clinical applicability. Therefore, the goal of this study is to better understand the benefits and limitations of this technology in ultrasound-guided interventions.

Methods

We used an AR-HMD system (based on the first-generation Microsoft Hololens) which overlays live ultrasound images spatially correctly at the location of the ultrasound transducer. We chose ultrasound-guided needle placements as a representative task for image-guided interventions. To examine the effects of the AR-HMD, we used mixed methods and conducted two studies in a lab setting: (1) In a randomized crossover study, we asked participants to place needles into a training model and evaluated task duration and accuracy with the AR-HMD as compared to the standard procedure without visual overlay and (2) in a qualitative study, we analyzed the user experience with AR-HMD using think-aloud protocols during ultrasound examinations and semi-structured interviews after the task.

Results

Participants (n = 20) placed needles more accurately (mean error of 7.4 mm vs. 4.9 mm, p = 0.022) but not significantly faster (mean task duration of 74.4 s vs. 66.4 s, p = 0.211) with the AR-HMD. All participants in the qualitative study (n = 6) reported limitations of and unfamiliarity with the AR-HMD, yet all but one also clearly noted benefits and/or that they would like to test the technology in practice.

Conclusion

We present additional, though still preliminary, evidence that AR-HMDs provide benefits in image-guided procedures. Our data also contribute insights into potential causes underlying the benefits, such as improved spatial perception. Still, more comprehensive studies are needed to ascertain benefits for clinical applications and to clarify mechanisms underlying these benefits.

Electronic supplementary material

The online version of this article (10.1007/s11548-020-02236-6) contains supplementary material, which is available to authorized users.