1
|
Asadi Z, Asadi M, Kazemipour N, Léger É, Kersten-Oertel M. A decade of progress: bringing mixed reality image-guided surgery systems in the operating room. Comput Assist Surg (Abingdon) 2024; 29:2355897. [PMID: 38794834 DOI: 10.1080/24699322.2024.2355897] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/26/2024] Open
Abstract
Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.
Collapse
Affiliation(s)
- Zahra Asadi
- Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
| | - Mehrdad Asadi
- Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
| | - Negar Kazemipour
- Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
| | - Étienne Léger
- Montréal Neurological Institute & Hospital (MNI/H), Montréal, Canada
- McGill University, Montréal, Canada
| | - Marta Kersten-Oertel
- Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
| |
Collapse
|
2
|
Weibel N, Alwood B, Ramesh V, Liu W, Meyer DM, McQuaid T, Germain ES, Meyer BC. Holo-Stroke: Assessing for Immersive Stroke Care Through Stroke Hologram Teleportation. Telemed J E Health 2024. [PMID: 38995868 DOI: 10.1089/tmj.2024.0229] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/14/2024] Open
Abstract
Background: Augmented reality enables the wearer to see both their physical environment and virtual objects. Holograms could allow 3D video of providers to be transmitted to distant sites, allowing patients to interact with virtual providers as if they are in the same physical space. Our aim was to determine if Tele-Stroke augmented with Holo-Stroke, compared with Tele-Stroke alone, could improve satisfaction and perception of immersion for the patient. Methods: Kinect cameras positioned at 90-degree intervals around the hub practitioner were used. Cameras streamed real-time optical video to a unity point-cloud program where the data were stitched together in a 360-degree view. The resultant hologram was positioned in 3D space and was visible through the head-mounted display by the patient. Radiology images were shared in Tele-Stroke and via hologram. Likert satisfaction questions were administered. Wilcoxon signed-rank testing was used. Results: Each of the 30 neurology clinic participants scored both Tele-Stroke and Holo-Stroke. Out of these, 29 patients completed the assessments (1 failure owing to computer reboot). Average age was 52 years, with 53.3% of the patients being female, 70.0% being White, and 13.3% being Hispanic. Likert scale score median "Overall" was 32 Tele-Stroke versus 48 Holo-Stroke (p < 0.00001), "Immersion" was 5 versus 10 (p < 0.00001), "Beneficial Technique" was 6 versus 10 (p < 0.00001), and "Ability to See Images" was 5 versus 10 (p < 0.00001). Discussion: Holo-Stroke 3D holographic Tele-Stroke exams resulted in feasibility, satisfaction, and high perception of immersion for the patient. Patients were enthusiastic for the more immersive, personal discussion with their provider and a robust way to experience radiology images. Though further assessments are needed, Holo-Stroke can help the provider "be there, not just see there!"
Collapse
Affiliation(s)
- Nadir Weibel
- Department of Computer Science and Engineering & Design Lab, University of California, San Diego, California, USA
| | - Ben Alwood
- College of Medicine, University of Florida, Jacksonville, Florida, USA
| | - Vishwajith Ramesh
- Department of Computer Science and Engineering & Design Lab, University of California, San Diego, California, USA
| | - Weichen Liu
- Department of Mathematics, University of California, San Diego, California, USA
| | - Dawn M Meyer
- Department of Neurosciences, University of California, San Diego, California, USA
| | - Teri McQuaid
- Department of Neurosciences, University of California, San Diego, California, USA
| | - Emily St Germain
- Telehealth Program, University of California, San Diego, California, USA
| | - Brett C Meyer
- Department of Neurosciences, University of California, San Diego, California, USA
| |
Collapse
|
3
|
Li H, Yan W, Zhao J, Ji Y, Qian L, Ding H, Zhao Z, Wang G. Navigate biopsy with ultrasound under augmented reality device: Towards higher system performance. Comput Biol Med 2024; 174:108453. [PMID: 38636327 DOI: 10.1016/j.compbiomed.2024.108453] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Revised: 03/11/2024] [Accepted: 04/07/2024] [Indexed: 04/20/2024]
Abstract
PURPOSE Biopsies play a crucial role in determining the classification and staging of tumors. Ultrasound is frequently used in this procedure to provide real-time anatomical information. Using augmented reality (AR), surgeons can visualize ultrasound data and spatial navigation information seamlessly integrated with real tissues. This innovation facilitates faster and more precise biopsy operations. METHODS We have developed an augmented reality biopsy navigation system characterized by low display latency and high accuracy. Ultrasound data is initially read by an image capture card and streamed to Unity via net communication. In Unity, navigation information is rendered and transmitted to the HoloLens 2 device using holographic remoting. Concurrently, a retro-reflective tool tracking method is implemented on the HoloLens 2, enabling the simultaneous tracking of the ultrasound probe and biopsy needle. Distinct navigation information is provided during in-plane and out-of-plane punctuation. To evaluate the effectiveness of our system, we conducted a study involving ten participants, assessing puncture accuracy and biopsy time in comparison to traditional methods. RESULTS Ultrasound image was streamed from the ultrasound device to augmented reality headset with 122.49±11.61ms latency, while only 16.22±11.25ms was taken after data acquisition from image capture card. Navigation accuracy reached 1.23±0.68mm in the image plane and 0.95±0.70mm outside the image plane, within a depth range of 200 millimeters. Remarkably, the utilization of our system led to 98% and 95% success rate in out-of-plane and in-plane biopsy, among ten participants with little ultrasound experience. CONCLUSION To sum up, this paper introduces an AR-based ultrasound biopsy navigation system characterized by high navigation accuracy and minimal latency. The system provides distinct visualization contents during in-plane and out-of-plane operations according to their different characteristics. Use case study in this paper proved that our system can help young surgeons perform biopsy faster and more accurately.
Collapse
Affiliation(s)
- Haowei Li
- Biomedical Engineering, Tsinghua University, Shuang Qing Road, Beijing, 100084, Beijing, China.
| | - Wenqing Yan
- School of Medicine, Tsinghua University, Shuang Qing Road, Beijing, 100084, Beijing, China.
| | - Jiasheng Zhao
- Biomedical Engineering, Tsinghua University, Shuang Qing Road, Beijing, 100084, Beijing, China.
| | - Yuqi Ji
- School of Medicine, Tsinghua University, Shuang Qing Road, Beijing, 100084, Beijing, China.
| | - Long Qian
- Medivis Inc., 920 Broadway, New York, 10010, NY, USA.
| | - Hui Ding
- Biomedical Engineering, Tsinghua University, Shuang Qing Road, Beijing, 100084, Beijing, China.
| | - Zhe Zhao
- School of Clinical Medicine, Tsinghua University, Shuang Qing Road, Beijing, 100084, Beijing, China; Orthopedics & Sports Medicine Center, Beijing Tsinghua Changgung Hospital, Li Tang Road, Beijing, 100043, Beijing, China.
| | - Guangzhi Wang
- Biomedical Engineering, Tsinghua University, Shuang Qing Road, Beijing, 100084, Beijing, China.
| |
Collapse
|
4
|
Żydowicz WM, Skokowski J, Marano L, Polom K. Current Trends and Beyond Conventional Approaches: Advancements in Breast Cancer Surgery through Three-Dimensional Imaging, Virtual Reality, Augmented Reality, and the Emerging Metaverse. J Clin Med 2024; 13:915. [PMID: 38337610 PMCID: PMC10856583 DOI: 10.3390/jcm13030915] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2024] [Revised: 01/25/2024] [Accepted: 02/03/2024] [Indexed: 02/12/2024] Open
Abstract
Breast cancer stands as the most prevalent cancer globally, necessitating comprehensive care. A multidisciplinary approach proves crucial for precise diagnosis and treatment, ultimately leading to effective disease management. While surgical interventions continue to evolve and remain integral for curative treatment, imaging assumes a fundamental role in breast cancer detection. Advanced imaging techniques not only facilitate improved diagnosis but also contribute significantly to the overall enhancement of breast cancer management. This review article aims to provide an overview of innovative technologies such as virtual reality, augmented reality, and three-dimensional imaging, utilized in the medical field to elevate the diagnosis and treatment of breast cancer. Additionally, the article delves into an emerging technology known as the metaverse, still under development. Through the analysis of impactful research and comparison of their findings, this study offers valuable insights into the advantages of each innovative technique. The goal is to provide physicians, surgeons, and radiologists with information on how to enhance breast cancer management.
Collapse
Affiliation(s)
- Weronika Magdalena Żydowicz
- Department of General Surgery and Surgical Oncology, “Saint Wojciech” Hospital, “Nicolaus Copernicus” Health Center, Jana Pawła II 50, 80-462 Gdańsk, Poland; (W.M.Ż.); (J.S.)
| | - Jaroslaw Skokowski
- Department of General Surgery and Surgical Oncology, “Saint Wojciech” Hospital, “Nicolaus Copernicus” Health Center, Jana Pawła II 50, 80-462 Gdańsk, Poland; (W.M.Ż.); (J.S.)
- Department of Medicine, Academy of Applied Medical and Social Sciences, Akademia Medycznych I Spolecznych Nauk Stosowanych (AMiSNS), 2 Lotnicza Street, 82-300 Elbląg, Poland;
| | - Luigi Marano
- Department of General Surgery and Surgical Oncology, “Saint Wojciech” Hospital, “Nicolaus Copernicus” Health Center, Jana Pawła II 50, 80-462 Gdańsk, Poland; (W.M.Ż.); (J.S.)
- Department of Medicine, Academy of Applied Medical and Social Sciences, Akademia Medycznych I Spolecznych Nauk Stosowanych (AMiSNS), 2 Lotnicza Street, 82-300 Elbląg, Poland;
| | - Karol Polom
- Department of Medicine, Academy of Applied Medical and Social Sciences, Akademia Medycznych I Spolecznych Nauk Stosowanych (AMiSNS), 2 Lotnicza Street, 82-300 Elbląg, Poland;
- Department of Gastrointestinal Surgical Oncology, Greater Poland Cancer Centre, Garbary 15, 61-866 Poznan, Poland
| |
Collapse
|
5
|
Pose-Díez-de-la-Lastra A, Ungi T, Morton D, Fichtinger G, Pascau J. Real-time integration between Microsoft HoloLens 2 and 3D Slicer with demonstration in pedicle screw placement planning. Int J Comput Assist Radiol Surg 2023; 18:2023-2032. [PMID: 37310561 PMCID: PMC10589185 DOI: 10.1007/s11548-023-02977-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2023] [Accepted: 05/23/2023] [Indexed: 06/14/2023]
Abstract
PURPOSE Up to date, there has been a lack of software infrastructure to connect 3D Slicer to any augmented reality (AR) device. This work describes a novel connection approach using Microsoft HoloLens 2 and OpenIGTLink, with a demonstration in pedicle screw placement planning. METHODS We developed an AR application in Unity that is wirelessly rendered onto Microsoft HoloLens 2 using Holographic Remoting. Simultaneously, Unity connects to 3D Slicer using the OpenIGTLink communication protocol. Geometrical transform and image messages are transferred between both platforms in real time. Through the AR glasses, a user visualizes a patient's computed tomography overlaid onto virtual 3D models showing anatomical structures. We technically evaluated the system by measuring message transference latency between the platforms. Its functionality was assessed in pedicle screw placement planning. Six volunteers planned pedicle screws' position and orientation with the AR system and on a 2D desktop planner. We compared the placement accuracy of each screw with both methods. Finally, we administered a questionnaire to all participants to assess their experience with the AR system. RESULTS The latency in message exchange is sufficiently low to enable real-time communication between the platforms. The AR method was non-inferior to the 2D desktop planner, with a mean error of 2.1 ± 1.4 mm. Moreover, 98% of the screw placements performed with the AR system were successful, according to the Gertzbein-Robbins scale. The average questionnaire outcomes were 4.5/5. CONCLUSIONS Real-time communication between Microsoft HoloLens 2 and 3D Slicer is feasible and supports accurate planning for pedicle screw placement.
Collapse
Affiliation(s)
| | - Tamas Ungi
- Laboratory for Percutaneous Surgery, School of Computing, Queen's University, Kingston, ON, K7M2N8, Canada
| | - David Morton
- Laboratory for Percutaneous Surgery, School of Computing, Queen's University, Kingston, ON, K7M2N8, Canada
| | - Gabor Fichtinger
- Laboratory for Percutaneous Surgery, School of Computing, Queen's University, Kingston, ON, K7M2N8, Canada
| | - Javier Pascau
- Departamento de Bioingeniería, Universidad Carlos III de Madrid, 28911, Leganés, Spain
| |
Collapse
|
6
|
Saruwatari MS, Nguyen TN, Talari HF, Matisoff AJ, Sharma KV, Donoho KG, Basu S, Dwivedi P, Bost JE, Shekhar R. Assessing the Effect of Augmented Reality on Procedural Outcomes During Ultrasound-Guided Vascular Access. ULTRASOUND IN MEDICINE & BIOLOGY 2023; 49:2346-2353. [PMID: 37573178 PMCID: PMC10658651 DOI: 10.1016/j.ultrasmedbio.2023.07.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Revised: 06/16/2023] [Accepted: 07/11/2023] [Indexed: 08/14/2023]
Abstract
OBJECTIVE Augmented reality devices are increasingly accepted in health care, though most applications involve education and pre-operative planning. A novel augmented reality ultrasound application, HoloUS, was developed for the Microsoft HoloLens 2 to project real-time ultrasound images directly into the user's field of view. In this work, we assessed the effect of using HoloUS on vascular access procedural outcomes. METHODS A single-center user study was completed with participants with (N = 22) and without (N = 12) experience performing ultrasound-guided vascular access. Users completed a venipuncture and aspiration task a total of four times: three times on study day 1, and once on study day 2 between 2 and 4 weeks later. Users were randomized to use conventional ultrasound during either their first or second task and the HoloUS application at all other times. Task completion time, numbers of needle re-directions, head adjustments and needle visualization rates were recorded. RESULTS For expert users, task completion time was significantly faster using HoloUS (11.5 s, interquartile range [IQR] = 6.5-23.5 s vs. 18.5 s, IQR = 11.0-36.5 s; p = 0.04). The number of head adjustments was significantly lower using the HoloUS app (1.0, IQR = 0.0-1.0 vs. 3.0, IQR = 1.0-5.0; p < 0.0001). No significant differences were identified in other measured outcomes. CONCLUSION This is the first investigation of augmented reality-based ultrasound-guided vascular access using the second-generation HoloLens. It demonstrates equivalent procedural efficiency and accuracy, with favorable usability, ergonomics and user independence when compared with traditional ultrasound techniques.
Collapse
Affiliation(s)
- Michele S Saruwatari
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA; Department of Surgery, MedStar Georgetown University Hospital and Washington Hospital Center, Washington, DC, USA
| | | | - Hadi Fooladi Talari
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - Andrew J Matisoff
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA; George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| | - Karun V Sharma
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA; George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| | - Kelsey G Donoho
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA; George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| | - Sonali Basu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA; George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| | - Pallavi Dwivedi
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - James E Bost
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA; George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA; IGI Technologies, Silver Spring, MD, USA; George Washington University School of Medicine and Health Sciences, Washington, DC, USA.
| |
Collapse
|
7
|
Shabir D, Anjum A, Hamza H, Padhan J, Al-Ansari A, Yaacoub E, Mohammed A, Navkar NV. Development and Evaluation of a Mixed-Reality Tele-ultrasound System. ULTRASOUND IN MEDICINE & BIOLOGY 2023; 49:1867-1874. [PMID: 37263893 DOI: 10.1016/j.ultrasmedbio.2023.04.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Revised: 02/25/2023] [Accepted: 04/28/2023] [Indexed: 06/03/2023]
Abstract
OBJECTIVE The objective of this feasibility study was to develop and assess a tele-ultrasound system that would enable an expert sonographer (situated at the remote site) to provide real-time guidance to an operator (situated at the imaging site) using a mixed-reality environment. METHODS An architecture along with the operational workflow of the system is designed and a prototype is developed that enables guidance in form of audiovisual cues. The visual cues comprise holograms (of the ultrasound images and ultrasound probe) and is rendered to the operator using a head-mounted display device. The position and orientation of the ultrasound probe's hologram are remotely controlled by the expert sonographer and guide the placement of a physical ultrasound probe at the imaging site. The developed prototype was evaluated for its performance on a network. In addition, a user study (with 12 participants) was conducted to assess the operator's ability to align the probe under different guidance modes. RESULTS The network performance revealed the view of the imaging site and ultrasound images were transferred to the remote site in 233 ± 42 and 158 ± 38 ms, respectively. The expert sonographer was able to transfer, to the imaging site, data related to position and orientation of the ultrasound probe's hologram in 78 ± 13 ms. The user study indicated that the audiovisual cues are sufficient for an operator to position and orient a physical probe for accurate depiction of the targeted tissue (p < 0.001). The probe's placement translational and rotational errors were 1.4 ± 0.6 mm and 5.4 ± 2.2º. CONCLUSION The work illustrates the feasibility of using a mixed-reality environment for effective communication between an expert sonographer (ultrasound physician) and an operator. Further studies are required to determine its applicability in a clinical setting during tele-ultrasound.
Collapse
Affiliation(s)
- Dehlela Shabir
- Department of Surgery, Hamad Medical Corporation, Doha, Qatar
| | - Arshak Anjum
- Department of Computer Science and Engineering, Qatar University, Doha, Qatar
| | - Hawa Hamza
- Department of Surgery, Hamad Medical Corporation, Doha, Qatar
| | | | | | - Elias Yaacoub
- Department of Computer Science and Engineering, Qatar University, Doha, Qatar
| | - Amr Mohammed
- Department of Computer Science and Engineering, Qatar University, Doha, Qatar
| | - Nikhil V Navkar
- Department of Surgery, Hamad Medical Corporation, Doha, Qatar.
| |
Collapse
|
8
|
Stepanenko A, Perez LM, Ferre JC, Ybarra Falcón C, Pérez de la Sota E, San Roman JA, Redondo Diéguez A, Baladron C. 3D Virtual modelling, 3D printing and extended reality for planning of implant procedure of short-term and long-term mechanical circulatory support devices and heart transplantation. Front Cardiovasc Med 2023; 10:1191705. [PMID: 37663417 PMCID: PMC10473250 DOI: 10.3389/fcvm.2023.1191705] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Accepted: 06/19/2023] [Indexed: 09/05/2023] Open
Abstract
Introduction The use of three-dimensional (3D) reconstruction and printing technology, together with extended reality applied to advanced heart failure adult patients with complex anatomy, is rapidly spreading in clinical practice. We report practical experience with application to acute and chronic heart failure: planning and performing mechanical circulatory device insertion or heart transplantation. Methods From November 2019 until February 2022, 53 3D virtual biomodels were produced for intervention planning (using Virtual/Augmented Reality and/or 3D printing), following a specific segmentation and preprocessing workflow for biomodelling, in patients with advanced heart failure due to structural heart disease or cardiomyopathies. Four of those patients were complex cases requiring mechanical circulatory support implant procedures in our center. Results One short-term and three long-term ventricular assist device system were successfully clinically implanted after application of this technique. In other two cases with extremely high procedural risk, visualized after application of this multimodality imaging, heart transplantation was elected. Conclusion 3D printing based planning and virtual procedure simulation, are of great importance to select appropriate candidates for mechanical circulatory support in case of complex patient anatomy and may help to diminish periprocedural complications. Extended reality represents a perspective tool in planification of complex surgical procedures or ventricular assist device insertion in this setting.
Collapse
Affiliation(s)
- Alexander Stepanenko
- Cardiology Department, Hospital Clínico Universitario de Valladolid, Valladolid, Spain
- Centro de Investigación Biomédica en Red de Enfermedades Cardiovasculares (CIBERCV), Madrid, Spain
| | - Laura Maroto Perez
- Cardiovascular Surgery, Hospital Recoletas Campo Grande, Valladolid, Spain
| | - Jordi Candela Ferre
- Cardiology Department, Hospital Clínico Universitario de Valladolid, Valladolid, Spain
| | | | | | - Jose Alberto San Roman
- Cardiology Department, Hospital Clínico Universitario de Valladolid, Valladolid, Spain
- Centro de Investigación Biomédica en Red de Enfermedades Cardiovasculares (CIBERCV), Madrid, Spain
- Vall3DLab, Hospital Clínico Universitario de Valladolid, Valladolid, Spain
| | - Alfredo Redondo Diéguez
- Vall3DLab, Hospital Clínico Universitario de Valladolid, Valladolid, Spain
- Cardiology Department, University Hospital of Santiago, Santiago de Compostela, Spain
| | - Carlos Baladron
- Cardiology Department, Hospital Clínico Universitario de Valladolid, Valladolid, Spain
- Centro de Investigación Biomédica en Red de Enfermedades Cardiovasculares (CIBERCV), Madrid, Spain
- Vall3DLab, Hospital Clínico Universitario de Valladolid, Valladolid, Spain
| |
Collapse
|
9
|
Hemme CL, Carley R, Norton A, Ghumman M, Nguyen H, Ivone R, Menon JU, Shen J, Bertin M, King R, Leibovitz E, Bergstrom R, Cho B. Developing virtual and augmented reality applications for science, technology, engineering and math education. Biotechniques 2023; 75:343-352. [PMID: 37291856 PMCID: PMC10505987 DOI: 10.2144/btn-2023-0029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2023] [Accepted: 05/30/2023] [Indexed: 06/10/2023] Open
Abstract
The Rhode Island IDeA Network of Biomedical Research Excellence Molecular Informatics Core at the University of Rhode Island Information Technology Services Innovative Learning Technologies developed virtual and augmented reality applications to teach concepts in biomedical science, including pharmacology, medicinal chemistry, cell culture and nanotechnology. The apps were developed as full virtual reality/augmented reality and 3D gaming versions, which do not require virtual reality headsets. Development challenges included creating intuitive user interfaces, text-to-voice functionality, visualization of molecules and implementing complex science concepts. In-app quizzes are used to assess the user's understanding of topics, and user feedback was collected for several apps to improve the experience. The apps were positively reviewed by users and are being implemented into the curriculum at the University of Rhode Island.
Collapse
Affiliation(s)
- Christopher L Hemme
- Rhode Island IDeA Network of Biomedical Research Excellence (RI-INBRE)
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
| | - Rachel Carley
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
| | - Arielle Norton
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
| | - Moez Ghumman
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
| | - Hannah Nguyen
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
| | - Ryan Ivone
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
| | - Jyothi U Menon
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
- College of Engineering, University of Rhode Island, Kingston, RI 02881, USA
| | - Jie Shen
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
- College of Engineering, University of Rhode Island, Kingston, RI 02881, USA
| | - Matthew Bertin
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
| | - Roberta King
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
| | | | - Roy Bergstrom
- Information Technology Services, Innovative Learning Technologies Program, University of Rhode Island, Kingston, RI 02881, USA
| | - Bongsup Cho
- Rhode Island IDeA Network of Biomedical Research Excellence (RI-INBRE)
- College of Pharmacy, University of Rhode Island, Kingston, RI 02881, USA
| |
Collapse
|
10
|
Boulet N, Muller L, Rickard CM, Lefrant JY, Roger C. How to improve the efficiency and the safety of real-time ultrasound-guided central venous catheterization in 2023: a narrative review. Ann Intensive Care 2023; 13:46. [PMID: 37227571 PMCID: PMC10212873 DOI: 10.1186/s13613-023-01141-w] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Accepted: 05/15/2023] [Indexed: 05/26/2023] Open
Abstract
Central venous catheterization (CVC) is a frequent procedure, practiced by intensivists, anesthesiologists and advanced practice nurses in intensive care units and operative rooms. To reduce CVC-associated morbidity, it is essential to strive for best practices, based on the latest evidence. This narrative review aims to synthesize current knowledge on evidence-based best practices for CVC that improve the use and feasibility of real-time ultrasound-guided insertion procedures. Optimization of the vein puncture technique and the development of new technologies are discussed to reinforce the use of the subclavian vein catheterization as first choice. The search for alternative site of insertions, without increasing infectious and thrombotic risks, deserves further research.
Collapse
Affiliation(s)
- Nicolas Boulet
- Department of Anesthesiology, Critical Care, Intensive Care, Pain and Emergency Medicine, Nîmes University Hospital, Place du Professeur Debré, Gard, 30900, Nîmes, France.
- IMAGINE, UR-UM 103, University of Montpellier, Nîmes University Hospital, Nîmes, France.
| | - Laurent Muller
- Department of Anesthesiology, Critical Care, Intensive Care, Pain and Emergency Medicine, Nîmes University Hospital, Place du Professeur Debré, Gard, 30900, Nîmes, France
- IMAGINE, UR-UM 103, University of Montpellier, Nîmes University Hospital, Nîmes, France
| | - Claire M Rickard
- School of Nursing, Midwifery, and Social Work & Herston Infectious Diseases Institute, The University of Queensland & Royal Brisbane and Women's Hospital, Brisbane, QLD, Australia
- Alliance for Vascular Access Teaching and Research, Menzies Health Institute Queensland, Griffith University, Brisbane, QLD, Australia
| | - Jean-Yves Lefrant
- Department of Anesthesiology, Critical Care, Intensive Care, Pain and Emergency Medicine, Nîmes University Hospital, Place du Professeur Debré, Gard, 30900, Nîmes, France
- IMAGINE, UR-UM 103, University of Montpellier, Nîmes University Hospital, Nîmes, France
| | - Claire Roger
- Department of Anesthesiology, Critical Care, Intensive Care, Pain and Emergency Medicine, Nîmes University Hospital, Place du Professeur Debré, Gard, 30900, Nîmes, France
- IMAGINE, UR-UM 103, University of Montpellier, Nîmes University Hospital, Nîmes, France
| |
Collapse
|
11
|
Augmented reality during parotid surgery: real-life evaluation of voice control of a head mounted display. Eur Arch Otorhinolaryngol 2023; 280:2043-2049. [PMID: 36269364 PMCID: PMC9988782 DOI: 10.1007/s00405-022-07699-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Accepted: 10/08/2022] [Indexed: 11/03/2022]
Abstract
PURPOSE Augmented Reality can improve surgical planning and performance in parotid surgery. For easier application we implemented a voice control manual for our augmented reality system. The aim of the study was to evaluate the feasibility of the voice control in real-life situations. METHODS We used the HoloLens 1® (Microsoft Corporation) with a special speech recognition software for parotid surgery. The evaluation took place in a audiometry cubicle and during real surgical procedures. Voice commands were used to display various 3D structures of the patient with the HoloLens 1®. Commands had different variations (male/female, 65 dB SPL)/louder, various structures). RESULTS In silence, 100% of commands were recognized. If the volume of the operation room (OR) background noise exceeds 42 dB, the recognition rate decreases significantly, and it drops below 40% at > 60 dB SPL. With constant speech volume at 65 dB SPL male speakers had a significant better recognition rate than female speakers (p = 0.046). Higher speech volumes can compensate this effect. The recognition rate depends on the type of background noise. Mixed OR noise (52 dB(A)) reduced the detection rate significantly compared to single suction noise at 52 dB(A) (p ≤ 0.00001). The recognition rate was significantly better in the OR than in the audio cubicle (p = 0.00013 both genders, 0.0086 female, and 0.0036 male). CONCLUSIONS The recognition rate of voice commands can be enhanced by increasing the speech volume and by singularizing ambient noises. The detection rate depends on the loudness of the OR noise. Male voices are understood significantly better than female voices.
Collapse
|
12
|
Gsaxner C, Li J, Pepe A, Jin Y, Kleesiek J, Schmalstieg D, Egger J. The HoloLens in medicine: A systematic review and taxonomy. Med Image Anal 2023; 85:102757. [PMID: 36706637 DOI: 10.1016/j.media.2023.102757] [Citation(s) in RCA: 20] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2022] [Revised: 01/05/2023] [Accepted: 01/18/2023] [Indexed: 01/22/2023]
Abstract
The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.
Collapse
Affiliation(s)
- Christina Gsaxner
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; BioTechMed, 8010 Graz, Austria.
| | - Jianning Li
- Institute of AI in Medicine, University Medicine Essen, 45131 Essen, Germany; Cancer Research Center Cologne Essen, University Medicine Essen, 45147 Essen, Germany
| | - Antonio Pepe
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; BioTechMed, 8010 Graz, Austria
| | - Yuan Jin
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; Research Center for Connected Healthcare Big Data, Zhejiang Lab, Hangzhou, 311121 Zhejiang, China
| | - Jens Kleesiek
- Institute of AI in Medicine, University Medicine Essen, 45131 Essen, Germany; Cancer Research Center Cologne Essen, University Medicine Essen, 45147 Essen, Germany
| | - Dieter Schmalstieg
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; BioTechMed, 8010 Graz, Austria
| | - Jan Egger
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; Institute of AI in Medicine, University Medicine Essen, 45131 Essen, Germany; BioTechMed, 8010 Graz, Austria; Cancer Research Center Cologne Essen, University Medicine Essen, 45147 Essen, Germany
| |
Collapse
|
13
|
von Haxthausen F, Rüger C, Sieren MM, Kloeckner R, Ernst F. Augmenting Image-Guided Procedures through In Situ Visualization of 3D Ultrasound via a Head-Mounted Display. SENSORS (BASEL, SWITZERLAND) 2023; 23:2168. [PMID: 36850766 PMCID: PMC9961663 DOI: 10.3390/s23042168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Revised: 02/09/2023] [Accepted: 02/10/2023] [Indexed: 06/18/2023]
Abstract
Medical ultrasound (US) is a commonly used modality for image-guided procedures. Recent research systems providing an in situ visualization of 2D US images via an augmented reality (AR) head-mounted display (HMD) were shown to be advantageous over conventional imaging through reduced task completion times and improved accuracy. In this work, we continue in the direction of recent developments by describing the first AR HMD application visualizing real-time volumetric (3D) US in situ for guiding vascular punctures. We evaluated the application on a technical level as well as in a mixed-methods user study with a qualitative prestudy and a quantitative main study, simulating a vascular puncture. Participants completed the puncture task significantly faster when using 3D US AR mode compared to 2D US AR, with a decrease of 28.4% in time. However, no significant differences were observed regarding the success rate of vascular puncture (2D US AR-50% vs. 3D US AR-72%). On the technical side, the system offers a low latency of 49.90 ± 12.92 ms and a satisfactory frame rate of 60 Hz. Our work shows the feasibility of a system that visualizes real-time 3D US data via an AR HMD, and our experiments show, furthermore, that this may offer additional benefits in US-guided tasks (i.e., reduced task completion time) over 2D US images viewed in AR by offering a vividly volumetric visualization.
Collapse
Affiliation(s)
- Felix von Haxthausen
- Institute for Robotics and Cognitive Systems, University of Lübeck, 23562 Lübeck, Germany
| | - Christoph Rüger
- Department of Surgery, Campus Charité Mitte, Campus Virchow-Klinikum, Experimental Surgery, Charité–Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin Institute of Health, 10117 Berlin, Germany
| | - Malte Maria Sieren
- Department of Radiology and Nuclear Medicine, University Hospital Schleswig-Holstein Campus Lübeck, 23569 Lübeck, Germany
- Institute of Interventional Radiology, University Hospital Schleswig-Holstein Campus Lübeck, 23569 Lübeck, Germany
| | - Roman Kloeckner
- Institute of Interventional Radiology, University Hospital Schleswig-Holstein Campus Lübeck, 23569 Lübeck, Germany
| | - Floris Ernst
- Institute for Robotics and Cognitive Systems, University of Lübeck, 23562 Lübeck, Germany
| |
Collapse
|
14
|
Costa N, Ferreira L, de Araújo ARVF, Oliveira B, Torres HR, Morais P, Alves V, Vilaça JL. Augmented Reality-Assisted Ultrasound Breast Biopsy. SENSORS (BASEL, SWITZERLAND) 2023; 23:1838. [PMID: 36850436 PMCID: PMC9961993 DOI: 10.3390/s23041838] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Revised: 01/17/2023] [Accepted: 02/04/2023] [Indexed: 06/18/2023]
Abstract
Breast cancer is the most prevalent cancer in the world and the fifth-leading cause of cancer-related death. Treatment is effective in the early stages. Thus, a need to screen considerable portions of the population is crucial. When the screening procedure uncovers a suspect lesion, a biopsy is performed to assess its potential for malignancy. This procedure is usually performed using real-time Ultrasound (US) imaging. This work proposes a visualization system for US breast biopsy. It consists of an application running on AR glasses that interact with a computer application. The AR glasses track the position of QR codes mounted on an US probe and a biopsy needle. US images are shown in the user's field of view with enhanced lesion visualization and needle trajectory. To validate the system, latency of the transmission of US images was evaluated. Usability assessment compared our proposed prototype with a traditional approach with different users. It showed that needle alignment was more precise, with 92.67 ± 2.32° in our prototype versus 89.99 ± 37.49° in a traditional system. The users also reached the lesion more accurately. Overall, the proposed solution presents promising results, and the use of AR glasses as a tracking and visualization device exhibited good performance.
Collapse
Affiliation(s)
- Nuno Costa
- 2Ai—School of Technology, IPCA, 4750-810 Barcelos, Portugal
- Algoritmi Center, School of Engineering, University of Minho, 4800-058 Guimaraes, Portugal
- LASI—Associate Laboratory of Intelligent Systems, 4800-058 Guimaraes, Portugal
| | - Luís Ferreira
- 2Ai—School of Technology, IPCA, 4750-810 Barcelos, Portugal
| | - Augusto R. V. F. de Araújo
- 2Ai—School of Technology, IPCA, 4750-810 Barcelos, Portugal
- Institute of Computing, Universidade Federal Fluminense (UFF), Niteroi 24210-310, Brazil
| | - Bruno Oliveira
- 2Ai—School of Technology, IPCA, 4750-810 Barcelos, Portugal
- Algoritmi Center, School of Engineering, University of Minho, 4800-058 Guimaraes, Portugal
- LASI—Associate Laboratory of Intelligent Systems, 4800-058 Guimaraes, Portugal
- Life and Health Sciences Research Institute (ICVS), School of Medicine, University of Minho, 4710-057 Braga, Portugal
- ICVS/3B’s—PT Government Associate Laboratory, 4710-057 Braga/Guimaraes, Portugal
| | - Helena R. Torres
- 2Ai—School of Technology, IPCA, 4750-810 Barcelos, Portugal
- Algoritmi Center, School of Engineering, University of Minho, 4800-058 Guimaraes, Portugal
- LASI—Associate Laboratory of Intelligent Systems, 4800-058 Guimaraes, Portugal
- Life and Health Sciences Research Institute (ICVS), School of Medicine, University of Minho, 4710-057 Braga, Portugal
- ICVS/3B’s—PT Government Associate Laboratory, 4710-057 Braga/Guimaraes, Portugal
| | - Pedro Morais
- 2Ai—School of Technology, IPCA, 4750-810 Barcelos, Portugal
| | - Victor Alves
- Algoritmi Center, School of Engineering, University of Minho, 4800-058 Guimaraes, Portugal
- LASI—Associate Laboratory of Intelligent Systems, 4800-058 Guimaraes, Portugal
| | - João L. Vilaça
- 2Ai—School of Technology, IPCA, 4750-810 Barcelos, Portugal
| |
Collapse
|
15
|
A multimodal user interface for touchless control of robotic ultrasound. Int J Comput Assist Radiol Surg 2022:10.1007/s11548-022-02810-0. [PMID: 36565368 PMCID: PMC10363039 DOI: 10.1007/s11548-022-02810-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2022] [Accepted: 12/07/2022] [Indexed: 01/05/2023]
Abstract
PURPOSE Past research contained the investigation and development of robotic ultrasound. In this context, interfaces which allow for interaction with the robotic system are of paramount importance. Few researchers have addressed the issue of developing non-tactile interaction approaches, although they could be beneficial for maintaining sterility during medical procedures. Interaction could be supported by multimodality, which has the potential to enable intuitive and natural interaction. To assess the feasibility of multimodal interaction for non-tactile control of a co-located robotic ultrasound system, a novel human-robot interaction concept was developed. METHODS The medical use case of needle-based interventions under hybrid computed tomography and ultrasound imaging was analyzed by interviewing four radiologists. From the resulting workflow, interaction tasks were derived which include human-robot interaction. Based on this, characteristics of a multimodal, touchless human-robot interface were elaborated, suitable interaction modalities were identified, and a corresponding interface was developed, which was thereafter evaluated in a user study with eight participants. RESULTS The implemented interface includes voice commands, combined with hand gesture control for discrete control and navigation interaction of the robotic US probe, respectively. The interaction concept was evaluated by the users in the form of a quantitative questionnaire with a average usability. Qualitative analysis of interview results revealed user satisfaction with the implemented interaction methods and potential improvements to the system. CONCLUSION A multimodal, touchless interaction concept for a robotic US for the use case of needle-based procedures in interventional radiology was developed, incorporating combined voice and hand gesture control. Future steps will include the integration of a solution for the missing haptic feedback and the evaluation of its clinical suitability.
Collapse
|
16
|
Isaak A, Wolff T, Zdoroveac A, Taher F, Gürke L, Richarz S, Akifi S. Ultrasound-Guided Percutaneous Arteriovenous Fistula Creation Simulation Training in a Lifelike Flow Model. Bioengineering (Basel) 2022; 9:bioengineering9110659. [PMID: 36354570 PMCID: PMC9687548 DOI: 10.3390/bioengineering9110659] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Revised: 10/30/2022] [Accepted: 11/01/2022] [Indexed: 11/09/2022] Open
Abstract
Objectives: To assess the feasibility and training effect of simulation training for ultrasound-guided percutaneous arteriovenous fistula (pAVF) creation in a lifelike flow model. Methods: Twenty vascular trainees and specialists were shown an instructional video on creating a pAVF in a dedicated flow model and then randomized to a study or control group. The procedure was divided into five clearly defined steps. Two observers rated the performance on each step, and the time to perform the exercise was recorded. The study group participants underwent supervised hands-on training on the model before performing a second rated pAVF creation. All participants subsequently completed a feedback questionnaire. Results: After supervised simulation training, the study groups participants increased their mean performance rating from 2.2 ± 0.9 to 3.2 ± 0.7. A mean of 3.8 ± 0.8 procedure steps was accomplished independently (control group 2.1 ± 1.4; p < 0.05). The time taken to perform the procedure was 15.6 ± 3.8 min in the study group (control group 27.2 ± 7.3, p < 0.05). The participants with previous experience in ultrasound-guided vascular procedures (n = 5) achieved higher overall mean scores 3.0 ± 0.8 and accomplished more steps without assistance (2.0 ± 1.0) during the simulation training compared to their inexperienced peers (1.5 ± 0.3 and 0.8 ± 0.4, respectively). The feedback questionnaire revealed that the study group participants strongly agreed (n = 7) or agreed (n = 3) that training on the simulation model improved their skills regarding catheter handling. Conclusions: The study group participants increased their overall performance after training on the simulator. More experienced attendees performed better from the beginning, indicating the model to be lifelike and a potential skill assessment tool. Simulation training for pAVF creation using a lifelike model may be an intermediate step between acquiring ultrasound and theoretical pAVF skills and procedure guidance in theatre. However, this type of training is limited by its reliance on the simulator quality, demonstration devices and costs.
Collapse
Affiliation(s)
- Andrej Isaak
- Department of Vascular and Endovascular Surgery, University Hospital Basel, Spitalstrasse 21, 4031 Basel, Switzerland
- Vascular and Endovascular Surgery, Cantonal Hospital Aarau, 5001 Aarau, Switzerland
- Correspondence: ; Tel.: +41-62-838-45-13
| | - Thomas Wolff
- Department of Vascular and Endovascular Surgery, University Hospital Basel, Spitalstrasse 21, 4031 Basel, Switzerland
| | - Andrei Zdoroveac
- Vascular and Endovascular Surgery, Cantonal Hospital Aarau, 5001 Aarau, Switzerland
| | - Fadi Taher
- Vascular and Endovascular Surgery, Klinik Ottakring, Montlearstrasse 37, 1160 Wien, Austria
| | - Lorenz Gürke
- Department of Vascular and Endovascular Surgery, University Hospital Basel, Spitalstrasse 21, 4031 Basel, Switzerland
- Vascular and Endovascular Surgery, Cantonal Hospital Aarau, 5001 Aarau, Switzerland
| | - Sabine Richarz
- Department of Vascular and Endovascular Surgery, University Hospital Basel, Spitalstrasse 21, 4031 Basel, Switzerland
| | - Shuaib Akifi
- Vascular and Endovascular Surgery, Cantonal Hospital Aarau, 5001 Aarau, Switzerland
| |
Collapse
|
17
|
Doughty M, Ghugre NR, Wright GA. Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery. J Imaging 2022; 8:jimaging8070203. [PMID: 35877647 PMCID: PMC9318659 DOI: 10.3390/jimaging8070203] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2022] [Revised: 07/15/2022] [Accepted: 07/18/2022] [Indexed: 02/01/2023] Open
Abstract
We conducted a systematic review of recent literature to understand the current challenges in the use of optical see-through head-mounted displays (OST-HMDs) for augmented reality (AR) assisted surgery. Using Google Scholar, 57 relevant articles from 1 January 2021 through 18 March 2022 were identified. Selected articles were then categorized based on a taxonomy that described the required components of an effective AR-based navigation system: data, processing, overlay, view, and validation. Our findings indicated a focus on orthopedic (n=20) and maxillofacial surgeries (n=8). For preoperative input data, computed tomography (CT) (n=34), and surface rendered models (n=39) were most commonly used to represent image information. Virtual content was commonly directly superimposed with the target site (n=47); this was achieved by surface tracking of fiducials (n=30), external tracking (n=16), or manual placement (n=11). Microsoft HoloLens devices (n=24 in 2021, n=7 in 2022) were the most frequently used OST-HMDs; gestures and/or voice (n=32) served as the preferred interaction paradigm. Though promising system accuracy in the order of 2–5 mm has been demonstrated in phantom models, several human factors and technical challenges—perception, ease of use, context, interaction, and occlusion—remain to be addressed prior to widespread adoption of OST-HMD led surgical navigation.
Collapse
Affiliation(s)
- Mitchell Doughty
- Department of Medical Biophysics, University of Toronto, Toronto, ON M5S 1A1, Canada; (N.R.G.); (G.A.W.)
- Schulich Heart Program, Sunnybrook Health Sciences Centre, Toronto, ON M4N 3M5, Canada
- Correspondence:
| | - Nilesh R. Ghugre
- Department of Medical Biophysics, University of Toronto, Toronto, ON M5S 1A1, Canada; (N.R.G.); (G.A.W.)
- Schulich Heart Program, Sunnybrook Health Sciences Centre, Toronto, ON M4N 3M5, Canada
- Physical Sciences Platform, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada
| | - Graham A. Wright
- Department of Medical Biophysics, University of Toronto, Toronto, ON M5S 1A1, Canada; (N.R.G.); (G.A.W.)
- Schulich Heart Program, Sunnybrook Health Sciences Centre, Toronto, ON M4N 3M5, Canada
- Physical Sciences Platform, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada
| |
Collapse
|
18
|
Costa JN, Gomes-Fonseca J, Valente S, Ferreira L, Oliveira B, Torres HR, Morais P, Alves V, Vilaca JL. Ultrasound training simulator using augmented reality glasses: an accuracy and precision assessment study. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:4461-4464. [PMID: 36086196 DOI: 10.1109/embc48229.2022.9871406] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Ultrasound (US) imaging despite being safe, cost-effective, and radiation-free, presents poor quality and artifacts, requiring specific medical training in US probe handling and image evaluation. The use of simulators to train physicians has proven its effectiveness, but most of them require specific facilities and hardware. In the last few years, augmented reality has gained relevance to simulate real scenarios which can avoid large setups and broaden medical training to more physicians. This work proposes a new framework for the training of US images acquisition. It consists of a custom-made application that runs on AR glasses (Microsoft HoloLens 2) and interacts with a US simulator application. The AR glasses track the orientation of a QR code mounted on a US probe, communicating its orientation with the US simulator application. This allows the physician to interact with a US probe seeing in real-time the US image in the physician's field of view. The QR code tracking assessment of the AR glasses was conducted by measuring the orientation accuracy and precision when compared with the measures of an electromagnetic tracking device (i.e., NDI Aurora). The proposed solution presented a good performance, rendering the US image in AR glasses with real-time feedback. Moreover, it can track the QR code on the US probe with an accuracy of 0.755°, and a precision of 0.018°. Overall, the proposed framework presents promising results and the use of AR glasses as a tracking device reached a good performance. Clinical Relevance- Simulation is a relevant tool to train physicians, especially in US imaging. AR glasses can broaden the training to less trained physicians by reducing the need for complex setups. This technology allows the implementation of a more natural user interface, which can be relevant in scenarios where good coordination between the eyes and hands of the physician is necessary (i.e., Biopsies).
Collapse
|
19
|
von Haxthausen F, Moreta-Martinez R, Pose Díez de la Lastra A, Pascau J, Ernst F. UltrARsound: in situ visualization of live ultrasound images using HoloLens 2. Int J Comput Assist Radiol Surg 2022; 17:2081-2091. [PMID: 35776399 PMCID: PMC9515035 DOI: 10.1007/s11548-022-02695-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2022] [Accepted: 05/31/2022] [Indexed: 11/24/2022]
Abstract
Purpose Augmented Reality (AR) has the potential to simplify ultrasound (US) examinations which usually require a skilled and experienced sonographer to mentally align narrow 2D cross-sectional US images in the 3D anatomy of the patient. This work describes and evaluates a novel approach to track retroreflective spheres attached to the US probe using an inside-out technique with the AR glasses HoloLens 2. Finally, live US images are displayed in situ on the imaged anatomy. Methods The Unity application UltrARsound performs spatial tracking of the US probe and attached retroreflective markers using the depth camera integrated into the AR glasses—thus eliminating the need for an external tracking system. Additionally, a Kalman filter is implemented to improve the noisy measurements of the camera. US images are streamed wirelessly via the PLUS toolkit to HoloLens 2. The technical evaluation comprises static and dynamic tracking accuracy, frequency and latency of displayed images. Results Tracking is performed with a median accuracy of 1.98 mm/1.81\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$^\circ $$\end{document}∘ for the static setting when using the Kalman filter. In a dynamic scenario, the median error was 2.81 mm/1.70\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$^\circ $$\end{document}∘. The tracking frequency is currently limited to 20 Hz. 83% of the displayed US images had a latency lower than 16 ms. Conclusions In this work, we showed that spatial tracking of retroreflective spheres with the depth camera of HoloLens 2 is feasible, achieving a promising accuracy for in situ visualization of live US images. For tracking, no additional hardware nor modifications to HoloLens 2 are required making it a cheap and easy-to-use approach. Moreover, a minimal latency of displayed images enables a real-time perception for the sonographer.
Collapse
Affiliation(s)
- Felix von Haxthausen
- Departamento de Bioingeniería e Ingeniería Aeroespacial, Universidad Carlos III de Madrid, Avda. Universidad 30, 28911, Leganés, Spain. .,Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562, Lübeck, Schleswig-Holstein, Germany.
| | - Rafael Moreta-Martinez
- Departamento de Bioingeniería e Ingeniería Aeroespacial, Universidad Carlos III de Madrid, Avda. Universidad 30, 28911, Leganés, Spain
| | - Alicia Pose Díez de la Lastra
- Departamento de Bioingeniería e Ingeniería Aeroespacial, Universidad Carlos III de Madrid, Avda. Universidad 30, 28911, Leganés, Spain
| | - Javier Pascau
- Departamento de Bioingeniería e Ingeniería Aeroespacial, Universidad Carlos III de Madrid, Avda. Universidad 30, 28911, Leganés, Spain
| | - Floris Ernst
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562, Lübeck, Schleswig-Holstein, Germany
| |
Collapse
|