HoloUS: Augmented reality visualization of live ultrasound images using HoloLens for ultrasound-guided procedures

A decade of progress: bringing mixed reality image-guided surgery systems in the operating room

Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.

Holo-Stroke: Assessing for Immersive Stroke Care Through Stroke Hologram Teleportation

Background: Augmented reality enables the wearer to see both their physical environment and virtual objects. Holograms could allow 3D video of providers to be transmitted to distant sites, allowing patients to interact with virtual providers as if they are in the same physical space. Our aim was to determine if Tele-Stroke augmented with Holo-Stroke, compared with Tele-Stroke alone, could improve satisfaction and perception of immersion for the patient. Methods: Kinect cameras positioned at 90-degree intervals around the hub practitioner were used. Cameras streamed real-time optical video to a unity point-cloud program where the data were stitched together in a 360-degree view. The resultant hologram was positioned in 3D space and was visible through the head-mounted display by the patient. Radiology images were shared in Tele-Stroke and via hologram. Likert satisfaction questions were administered. Wilcoxon signed-rank testing was used. Results: Each of the 30 neurology clinic participants scored both Tele-Stroke and Holo-Stroke. Out of these, 29 patients completed the assessments (1 failure owing to computer reboot). Average age was 52 years, with 53.3% of the patients being female, 70.0% being White, and 13.3% being Hispanic. Likert scale score median "Overall" was 32 Tele-Stroke versus 48 Holo-Stroke (p < 0.00001), "Immersion" was 5 versus 10 (p < 0.00001), "Beneficial Technique" was 6 versus 10 (p < 0.00001), and "Ability to See Images" was 5 versus 10 (p < 0.00001). Discussion: Holo-Stroke 3D holographic Tele-Stroke exams resulted in feasibility, satisfaction, and high perception of immersion for the patient. Patients were enthusiastic for the more immersive, personal discussion with their provider and a robust way to experience radiology images. Though further assessments are needed, Holo-Stroke can help the provider "be there, not just see there!"

Navigate biopsy with ultrasound under augmented reality device: Towards higher system performance

PURPOSE

Biopsies play a crucial role in determining the classification and staging of tumors. Ultrasound is frequently used in this procedure to provide real-time anatomical information. Using augmented reality (AR), surgeons can visualize ultrasound data and spatial navigation information seamlessly integrated with real tissues. This innovation facilitates faster and more precise biopsy operations.

METHODS

We have developed an augmented reality biopsy navigation system characterized by low display latency and high accuracy. Ultrasound data is initially read by an image capture card and streamed to Unity via net communication. In Unity, navigation information is rendered and transmitted to the HoloLens 2 device using holographic remoting. Concurrently, a retro-reflective tool tracking method is implemented on the HoloLens 2, enabling the simultaneous tracking of the ultrasound probe and biopsy needle. Distinct navigation information is provided during in-plane and out-of-plane punctuation. To evaluate the effectiveness of our system, we conducted a study involving ten participants, assessing puncture accuracy and biopsy time in comparison to traditional methods.

RESULTS

Ultrasound image was streamed from the ultrasound device to augmented reality headset with 122.49±11.61ms latency, while only 16.22±11.25ms was taken after data acquisition from image capture card. Navigation accuracy reached 1.23±0.68mm in the image plane and 0.95±0.70mm outside the image plane, within a depth range of 200 millimeters. Remarkably, the utilization of our system led to 98% and 95% success rate in out-of-plane and in-plane biopsy, among ten participants with little ultrasound experience.

CONCLUSION

To sum up, this paper introduces an AR-based ultrasound biopsy navigation system characterized by high navigation accuracy and minimal latency. The system provides distinct visualization contents during in-plane and out-of-plane operations according to their different characteristics. Use case study in this paper proved that our system can help young surgeons perform biopsy faster and more accurately.

Current Trends and Beyond Conventional Approaches: Advancements in Breast Cancer Surgery through Three-Dimensional Imaging, Virtual Reality, Augmented Reality, and the Emerging Metaverse

Breast cancer stands as the most prevalent cancer globally, necessitating comprehensive care. A multidisciplinary approach proves crucial for precise diagnosis and treatment, ultimately leading to effective disease management. While surgical interventions continue to evolve and remain integral for curative treatment, imaging assumes a fundamental role in breast cancer detection. Advanced imaging techniques not only facilitate improved diagnosis but also contribute significantly to the overall enhancement of breast cancer management. This review article aims to provide an overview of innovative technologies such as virtual reality, augmented reality, and three-dimensional imaging, utilized in the medical field to elevate the diagnosis and treatment of breast cancer. Additionally, the article delves into an emerging technology known as the metaverse, still under development. Through the analysis of impactful research and comparison of their findings, this study offers valuable insights into the advantages of each innovative technique. The goal is to provide physicians, surgeons, and radiologists with information on how to enhance breast cancer management.

Real-time integration between Microsoft HoloLens 2 and 3D Slicer with demonstration in pedicle screw placement planning

PURPOSE

Up to date, there has been a lack of software infrastructure to connect 3D Slicer to any augmented reality (AR) device. This work describes a novel connection approach using Microsoft HoloLens 2 and OpenIGTLink, with a demonstration in pedicle screw placement planning.

METHODS

We developed an AR application in Unity that is wirelessly rendered onto Microsoft HoloLens 2 using Holographic Remoting. Simultaneously, Unity connects to 3D Slicer using the OpenIGTLink communication protocol. Geometrical transform and image messages are transferred between both platforms in real time. Through the AR glasses, a user visualizes a patient's computed tomography overlaid onto virtual 3D models showing anatomical structures. We technically evaluated the system by measuring message transference latency between the platforms. Its functionality was assessed in pedicle screw placement planning. Six volunteers planned pedicle screws' position and orientation with the AR system and on a 2D desktop planner. We compared the placement accuracy of each screw with both methods. Finally, we administered a questionnaire to all participants to assess their experience with the AR system.

RESULTS

The latency in message exchange is sufficiently low to enable real-time communication between the platforms. The AR method was non-inferior to the 2D desktop planner, with a mean error of 2.1 ± 1.4 mm. Moreover, 98% of the screw placements performed with the AR system were successful, according to the Gertzbein-Robbins scale. The average questionnaire outcomes were 4.5/5.

CONCLUSIONS

Real-time communication between Microsoft HoloLens 2 and 3D Slicer is feasible and supports accurate planning for pedicle screw placement.

Assessing the Effect of Augmented Reality on Procedural Outcomes During Ultrasound-Guided Vascular Access

OBJECTIVE

Augmented reality devices are increasingly accepted in health care, though most applications involve education and pre-operative planning. A novel augmented reality ultrasound application, HoloUS, was developed for the Microsoft HoloLens 2 to project real-time ultrasound images directly into the user's field of view. In this work, we assessed the effect of using HoloUS on vascular access procedural outcomes.

METHODS

A single-center user study was completed with participants with (N = 22) and without (N = 12) experience performing ultrasound-guided vascular access. Users completed a venipuncture and aspiration task a total of four times: three times on study day 1, and once on study day 2 between 2 and 4 weeks later. Users were randomized to use conventional ultrasound during either their first or second task and the HoloUS application at all other times. Task completion time, numbers of needle re-directions, head adjustments and needle visualization rates were recorded.

RESULTS

For expert users, task completion time was significantly faster using HoloUS (11.5 s, interquartile range [IQR] = 6.5-23.5 s vs. 18.5 s, IQR = 11.0-36.5 s; p = 0.04). The number of head adjustments was significantly lower using the HoloUS app (1.0, IQR = 0.0-1.0 vs. 3.0, IQR = 1.0-5.0; p < 0.0001). No significant differences were identified in other measured outcomes.

CONCLUSION

This is the first investigation of augmented reality-based ultrasound-guided vascular access using the second-generation HoloLens. It demonstrates equivalent procedural efficiency and accuracy, with favorable usability, ergonomics and user independence when compared with traditional ultrasound techniques.

Development and Evaluation of a Mixed-Reality Tele-ultrasound System

OBJECTIVE

The objective of this feasibility study was to develop and assess a tele-ultrasound system that would enable an expert sonographer (situated at the remote site) to provide real-time guidance to an operator (situated at the imaging site) using a mixed-reality environment.

METHODS

An architecture along with the operational workflow of the system is designed and a prototype is developed that enables guidance in form of audiovisual cues. The visual cues comprise holograms (of the ultrasound images and ultrasound probe) and is rendered to the operator using a head-mounted display device. The position and orientation of the ultrasound probe's hologram are remotely controlled by the expert sonographer and guide the placement of a physical ultrasound probe at the imaging site. The developed prototype was evaluated for its performance on a network. In addition, a user study (with 12 participants) was conducted to assess the operator's ability to align the probe under different guidance modes.

RESULTS

The network performance revealed the view of the imaging site and ultrasound images were transferred to the remote site in 233 ± 42 and 158 ± 38 ms, respectively. The expert sonographer was able to transfer, to the imaging site, data related to position and orientation of the ultrasound probe's hologram in 78 ± 13 ms. The user study indicated that the audiovisual cues are sufficient for an operator to position and orient a physical probe for accurate depiction of the targeted tissue (p < 0.001). The probe's placement translational and rotational errors were 1.4 ± 0.6 mm and 5.4 ± 2.2º.

CONCLUSION

The work illustrates the feasibility of using a mixed-reality environment for effective communication between an expert sonographer (ultrasound physician) and an operator. Further studies are required to determine its applicability in a clinical setting during tele-ultrasound.

3D Virtual modelling, 3D printing and extended reality for planning of implant procedure of short-term and long-term mechanical circulatory support devices and heart transplantation

Introduction

The use of three-dimensional (3D) reconstruction and printing technology, together with extended reality applied to advanced heart failure adult patients with complex anatomy, is rapidly spreading in clinical practice. We report practical experience with application to acute and chronic heart failure: planning and performing mechanical circulatory device insertion or heart transplantation.

Methods

From November 2019 until February 2022, 53 3D virtual biomodels were produced for intervention planning (using Virtual/Augmented Reality and/or 3D printing), following a specific segmentation and preprocessing workflow for biomodelling, in patients with advanced heart failure due to structural heart disease or cardiomyopathies. Four of those patients were complex cases requiring mechanical circulatory support implant procedures in our center.

Results

One short-term and three long-term ventricular assist device system were successfully clinically implanted after application of this technique. In other two cases with extremely high procedural risk, visualized after application of this multimodality imaging, heart transplantation was elected.

Conclusion

3D printing based planning and virtual procedure simulation, are of great importance to select appropriate candidates for mechanical circulatory support in case of complex patient anatomy and may help to diminish periprocedural complications. Extended reality represents a perspective tool in planification of complex surgical procedures or ventricular assist device insertion in this setting.

Developing virtual and augmented reality applications for science, technology, engineering and math education

The Rhode Island IDeA Network of Biomedical Research Excellence Molecular Informatics Core at the University of Rhode Island Information Technology Services Innovative Learning Technologies developed virtual and augmented reality applications to teach concepts in biomedical science, including pharmacology, medicinal chemistry, cell culture and nanotechnology. The apps were developed as full virtual reality/augmented reality and 3D gaming versions, which do not require virtual reality headsets. Development challenges included creating intuitive user interfaces, text-to-voice functionality, visualization of molecules and implementing complex science concepts. In-app quizzes are used to assess the user's understanding of topics, and user feedback was collected for several apps to improve the experience. The apps were positively reviewed by users and are being implemented into the curriculum at the University of Rhode Island.

How to improve the efficiency and the safety of real-time ultrasound-guided central venous catheterization in 2023: a narrative review

Central venous catheterization (CVC) is a frequent procedure, practiced by intensivists, anesthesiologists and advanced practice nurses in intensive care units and operative rooms. To reduce CVC-associated morbidity, it is essential to strive for best practices, based on the latest evidence. This narrative review aims to synthesize current knowledge on evidence-based best practices for CVC that improve the use and feasibility of real-time ultrasound-guided insertion procedures. Optimization of the vein puncture technique and the development of new technologies are discussed to reinforce the use of the subclavian vein catheterization as first choice. The search for alternative site of insertions, without increasing infectious and thrombotic risks, deserves further research.

Augmented reality during parotid surgery: real-life evaluation of voice control of a head mounted display

PURPOSE

Augmented Reality can improve surgical planning and performance in parotid surgery. For easier application we implemented a voice control manual for our augmented reality system. The aim of the study was to evaluate the feasibility of the voice control in real-life situations.

METHODS

We used the HoloLens 1^® (Microsoft Corporation) with a special speech recognition software for parotid surgery. The evaluation took place in a audiometry cubicle and during real surgical procedures. Voice commands were used to display various 3D structures of the patient with the HoloLens 1^®. Commands had different variations (male/female, 65 dB SPL)/louder, various structures).

RESULTS

In silence, 100% of commands were recognized. If the volume of the operation room (OR) background noise exceeds 42 dB, the recognition rate decreases significantly, and it drops below 40% at > 60 dB SPL. With constant speech volume at 65 dB SPL male speakers had a significant better recognition rate than female speakers (p = 0.046). Higher speech volumes can compensate this effect. The recognition rate depends on the type of background noise. Mixed OR noise (52 dB(A)) reduced the detection rate significantly compared to single suction noise at 52 dB(A) (p ≤ 0.00001). The recognition rate was significantly better in the OR than in the audio cubicle (p = 0.00013 both genders, 0.0086 female, and 0.0036 male).

CONCLUSIONS

The recognition rate of voice commands can be enhanced by increasing the speech volume and by singularizing ambient noises. The detection rate depends on the loudness of the OR noise. Male voices are understood significantly better than female voices.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Augmenting Image-Guided Procedures through In Situ Visualization of 3D Ultrasound via a Head-Mounted Display

Medical ultrasound (US) is a commonly used modality for image-guided procedures. Recent research systems providing an in situ visualization of 2D US images via an augmented reality (AR) head-mounted display (HMD) were shown to be advantageous over conventional imaging through reduced task completion times and improved accuracy. In this work, we continue in the direction of recent developments by describing the first AR HMD application visualizing real-time volumetric (3D) US in situ for guiding vascular punctures. We evaluated the application on a technical level as well as in a mixed-methods user study with a qualitative prestudy and a quantitative main study, simulating a vascular puncture. Participants completed the puncture task significantly faster when using 3D US AR mode compared to 2D US AR, with a decrease of 28.4% in time. However, no significant differences were observed regarding the success rate of vascular puncture (2D US AR-50% vs. 3D US AR-72%). On the technical side, the system offers a low latency of 49.90 ± 12.92 ms and a satisfactory frame rate of 60 Hz. Our work shows the feasibility of a system that visualizes real-time 3D US data via an AR HMD, and our experiments show, furthermore, that this may offer additional benefits in US-guided tasks (i.e., reduced task completion time) over 2D US images viewed in AR by offering a vividly volumetric visualization.

Augmented Reality-Assisted Ultrasound Breast Biopsy

Breast cancer is the most prevalent cancer in the world and the fifth-leading cause of cancer-related death. Treatment is effective in the early stages. Thus, a need to screen considerable portions of the population is crucial. When the screening procedure uncovers a suspect lesion, a biopsy is performed to assess its potential for malignancy. This procedure is usually performed using real-time Ultrasound (US) imaging. This work proposes a visualization system for US breast biopsy. It consists of an application running on AR glasses that interact with a computer application. The AR glasses track the position of QR codes mounted on an US probe and a biopsy needle. US images are shown in the user's field of view with enhanced lesion visualization and needle trajectory. To validate the system, latency of the transmission of US images was evaluated. Usability assessment compared our proposed prototype with a traditional approach with different users. It showed that needle alignment was more precise, with 92.67 ± 2.32° in our prototype versus 89.99 ± 37.49° in a traditional system. The users also reached the lesion more accurately. Overall, the proposed solution presents promising results, and the use of AR glasses as a tracking and visualization device exhibited good performance.

A multimodal user interface for touchless control of robotic ultrasound

PURPOSE

Past research contained the investigation and development of robotic ultrasound. In this context, interfaces which allow for interaction with the robotic system are of paramount importance. Few researchers have addressed the issue of developing non-tactile interaction approaches, although they could be beneficial for maintaining sterility during medical procedures. Interaction could be supported by multimodality, which has the potential to enable intuitive and natural interaction. To assess the feasibility of multimodal interaction for non-tactile control of a co-located robotic ultrasound system, a novel human-robot interaction concept was developed.

METHODS

The medical use case of needle-based interventions under hybrid computed tomography and ultrasound imaging was analyzed by interviewing four radiologists. From the resulting workflow, interaction tasks were derived which include human-robot interaction. Based on this, characteristics of a multimodal, touchless human-robot interface were elaborated, suitable interaction modalities were identified, and a corresponding interface was developed, which was thereafter evaluated in a user study with eight participants.

RESULTS

The implemented interface includes voice commands, combined with hand gesture control for discrete control and navigation interaction of the robotic US probe, respectively. The interaction concept was evaluated by the users in the form of a quantitative questionnaire with a average usability. Qualitative analysis of interview results revealed user satisfaction with the implemented interaction methods and potential improvements to the system.

CONCLUSION

A multimodal, touchless interaction concept for a robotic US for the use case of needle-based procedures in interventional radiology was developed, incorporating combined voice and hand gesture control. Future steps will include the integration of a solution for the missing haptic feedback and the evaluation of its clinical suitability.

Ultrasound-Guided Percutaneous Arteriovenous Fistula Creation Simulation Training in a Lifelike Flow Model

Objectives: To assess the feasibility and training effect of simulation training for ultrasound-guided percutaneous arteriovenous fistula (pAVF) creation in a lifelike flow model. Methods: Twenty vascular trainees and specialists were shown an instructional video on creating a pAVF in a dedicated flow model and then randomized to a study or control group. The procedure was divided into five clearly defined steps. Two observers rated the performance on each step, and the time to perform the exercise was recorded. The study group participants underwent supervised hands-on training on the model before performing a second rated pAVF creation. All participants subsequently completed a feedback questionnaire. Results: After supervised simulation training, the study groups participants increased their mean performance rating from 2.2 ± 0.9 to 3.2 ± 0.7. A mean of 3.8 ± 0.8 procedure steps was accomplished independently (control group 2.1 ± 1.4; p < 0.05). The time taken to perform the procedure was 15.6 ± 3.8 min in the study group (control group 27.2 ± 7.3, p < 0.05). The participants with previous experience in ultrasound-guided vascular procedures (n = 5) achieved higher overall mean scores 3.0 ± 0.8 and accomplished more steps without assistance (2.0 ± 1.0) during the simulation training compared to their inexperienced peers (1.5 ± 0.3 and 0.8 ± 0.4, respectively). The feedback questionnaire revealed that the study group participants strongly agreed (n = 7) or agreed (n = 3) that training on the simulation model improved their skills regarding catheter handling. Conclusions: The study group participants increased their overall performance after training on the simulator. More experienced attendees performed better from the beginning, indicating the model to be lifelike and a potential skill assessment tool. Simulation training for pAVF creation using a lifelike model may be an intermediate step between acquiring ultrasound and theoretical pAVF skills and procedure guidance in theatre. However, this type of training is limited by its reliance on the simulator quality, demonstration devices and costs.

Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery

We conducted a systematic review of recent literature to understand the current challenges in the use of optical see-through head-mounted displays (OST-HMDs) for augmented reality (AR) assisted surgery. Using Google Scholar, 57 relevant articles from 1 January 2021 through 18 March 2022 were identified. Selected articles were then categorized based on a taxonomy that described the required components of an effective AR-based navigation system: data, processing, overlay, view, and validation. Our findings indicated a focus on orthopedic (n=20) and maxillofacial surgeries (n=8). For preoperative input data, computed tomography (CT) (n=34), and surface rendered models (n=39) were most commonly used to represent image information. Virtual content was commonly directly superimposed with the target site (n=47); this was achieved by surface tracking of fiducials (n=30), external tracking (n=16), or manual placement (n=11). Microsoft HoloLens devices (n=24 in 2021, n=7 in 2022) were the most frequently used OST-HMDs; gestures and/or voice (n=32) served as the preferred interaction paradigm. Though promising system accuracy in the order of 2–5 mm has been demonstrated in phantom models, several human factors and technical challenges—perception, ease of use, context, interaction, and occlusion—remain to be addressed prior to widespread adoption of OST-HMD led surgical navigation.

Ultrasound training simulator using augmented reality glasses: an accuracy and precision assessment study

Ultrasound (US) imaging despite being safe, cost-effective, and radiation-free, presents poor quality and artifacts, requiring specific medical training in US probe handling and image evaluation. The use of simulators to train physicians has proven its effectiveness, but most of them require specific facilities and hardware. In the last few years, augmented reality has gained relevance to simulate real scenarios which can avoid large setups and broaden medical training to more physicians. This work proposes a new framework for the training of US images acquisition. It consists of a custom-made application that runs on AR glasses (Microsoft HoloLens 2) and interacts with a US simulator application. The AR glasses track the orientation of a QR code mounted on a US probe, communicating its orientation with the US simulator application. This allows the physician to interact with a US probe seeing in real-time the US image in the physician's field of view. The QR code tracking assessment of the AR glasses was conducted by measuring the orientation accuracy and precision when compared with the measures of an electromagnetic tracking device (i.e., NDI Aurora). The proposed solution presented a good performance, rendering the US image in AR glasses with real-time feedback. Moreover, it can track the QR code on the US probe with an accuracy of 0.755°, and a precision of 0.018°. Overall, the proposed framework presents promising results and the use of AR glasses as a tracking device reached a good performance. Clinical Relevance- Simulation is a relevant tool to train physicians, especially in US imaging. AR glasses can broaden the training to less trained physicians by reducing the need for complex setups. This technology allows the implementation of a more natural user interface, which can be relevant in scenarios where good coordination between the eyes and hands of the physician is necessary (i.e., Biopsies).

UltrARsound: in situ visualization of live ultrasound images using HoloLens 2

Purpose

Augmented Reality (AR) has the potential to simplify ultrasound (US) examinations which usually require a skilled and experienced sonographer to mentally align narrow 2D cross-sectional US images in the 3D anatomy of the patient. This work describes and evaluates a novel approach to track retroreflective spheres attached to the US probe using an inside-out technique with the AR glasses HoloLens 2. Finally, live US images are displayed in situ on the imaged anatomy.

Methods

The Unity application UltrARsound performs spatial tracking of the US probe and attached retroreflective markers using the depth camera integrated into the AR glasses—thus eliminating the need for an external tracking system. Additionally, a Kalman filter is implemented to improve the noisy measurements of the camera. US images are streamed wirelessly via the PLUS toolkit to HoloLens 2. The technical evaluation comprises static and dynamic tracking accuracy, frequency and latency of displayed images.

Results

Tracking is performed with a median accuracy of 1.98 mm/1.81\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^\circ $$\end{document}∘ for the static setting when using the Kalman filter. In a dynamic scenario, the median error was 2.81 mm/1.70\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^\circ $$\end{document}∘. The tracking frequency is currently limited to 20 Hz. 83% of the displayed US images had a latency lower than 16 ms.

Conclusions

In this work, we showed that spatial tracking of retroreflective spheres with the depth camera of HoloLens 2 is feasible, achieving a promising accuracy for in situ visualization of live US images. For tracking, no additional hardware nor modifications to HoloLens 2 are required making it a cheap and easy-to-use approach. Moreover, a minimal latency of displayed images enables a real-time perception for the sonographer.