1
|
Asadi Z, Asadi M, Kazemipour N, Léger É, Kersten-Oertel M. A decade of progress: bringing mixed reality image-guided surgery systems in the operating room. Comput Assist Surg (Abingdon) 2024; 29:2355897. [PMID: 38794834 DOI: 10.1080/24699322.2024.2355897] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/26/2024] Open
Abstract
Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.
Collapse
Affiliation(s)
- Zahra Asadi
- Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
| | - Mehrdad Asadi
- Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
| | - Negar Kazemipour
- Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
| | - Étienne Léger
- Montréal Neurological Institute & Hospital (MNI/H), Montréal, Canada
- McGill University, Montréal, Canada
| | - Marta Kersten-Oertel
- Department of Computer Science and Software Engineering, Concordia University, Montréal, Canada
| |
Collapse
|
2
|
Van Ravestyn A, Frantz T, Vandemeulebroucke J, Jansen B, Duerinck J, Scheerlinck T. Determination of rotation center and diameter of femoral heads using off-the-shelf augmented reality hardware for navigation. Sci Rep 2024; 14:15458. [PMID: 38965266 PMCID: PMC11224340 DOI: 10.1038/s41598-024-64957-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2024] [Accepted: 06/14/2024] [Indexed: 07/06/2024] Open
Abstract
In total hip arthroplasty (THA), determining the center of rotation (COR) and diameter of the hip joint (acetabulum and femoral head) is essential to restore patient biomechanics. This study investigates on-the-fly determination of hip COR and size, using off-the-shelf augmented reality (AR) hardware. An AR head-mounted device (HMD) was configured with inside-out infrared tracking enabling the determination of surface coordinates using a handheld stylus. Two investigators examined 10 prosthetic femoral heads and cups, and 10 human femurs. The HMD calculated the diameter and COR through sphere fitting. Results were compared to data obtained from either verified prosthetic geometry or post-hoc CT analysis. Repeated single-observer measurements showed a mean diameter error of 0.63 mm ± 0.48 mm for the prosthetic heads and 0.54 mm ± 0.39 mm for the cups. Inter-observer comparison yielded mean diameter errors of 0.28 mm ± 0.71 mm and 1.82 mm ± 1.42 mm for the heads and cups, respectively. Cadaver testing found a mean COR error of 3.09 mm ± 1.18 mm and a diameter error of 1.10 mm ± 0.90 mm. Intra- and inter-observer reliability averaged below 2 mm. AR-based surface mapping using HMD proved accurate and reliable in determining the diameter of THA components with promise in identifying COR and diameter of osteoarthritic femoral heads.
Collapse
Affiliation(s)
- Antoine Van Ravestyn
- Department of Orthopedic Surgery and Traumatology, Universitair Ziekenhuis Brussel (UZ Brussel), Laarbeeklaan 101, 1090, Brussels, Belgium.
- Vrije Universiteit Brussel (VUB), Research Group BEFY-ORTHO, Laarbeeklaan 103, 1090, Brussels, Belgium.
| | - Taylor Frantz
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Pleinlaan 2, 1050, Brussels, Belgium
- IMEC, Kapeldreef 75, 3001, Leuven, Belgium
| | - Jef Vandemeulebroucke
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Pleinlaan 2, 1050, Brussels, Belgium
- IMEC, Kapeldreef 75, 3001, Leuven, Belgium
- Department of Radiology, Universitair Ziekenhuis Brussel (UZ Brussel), Laarbeeklaan 101, 1090, Brussels, Belgium
| | - Bart Jansen
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Pleinlaan 2, 1050, Brussels, Belgium
- IMEC, Kapeldreef 75, 3001, Leuven, Belgium
| | - Johnny Duerinck
- Department of Neurosurgery, Universitair Ziekenhuis Brussel (UZ Brussel), Laarbeeklaan 101, 1090, Brussels, Belgium
| | - Thierry Scheerlinck
- Department of Orthopedic Surgery and Traumatology, Universitair Ziekenhuis Brussel (UZ Brussel), Laarbeeklaan 101, 1090, Brussels, Belgium
- Vrije Universiteit Brussel (VUB), Research Group BEFY-ORTHO, Laarbeeklaan 103, 1090, Brussels, Belgium
| |
Collapse
|
3
|
Gu Z, Bao X, Zhu Y, Wang Q, Gao W, Wang D, Tian Y, Li Y. Rejecting small! Accepting larger? Optimizing the font size of Chinese characters basing legibility and visual fatigue in OST-HMDs. ERGONOMICS 2024:1-14. [PMID: 38934640 DOI: 10.1080/00140139.2024.2369205] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/21/2023] [Accepted: 06/11/2024] [Indexed: 06/28/2024]
Abstract
Font size is highly related to the legibility and visual fatigue in OST-HMDs, but the effects of font size on these factors remain further explored. An experiment was conducted to investigate the effects of a wider range of Chinese character font size (0.32°-1°) on legibility and visual fatigue, as well as to determine the optimal font size. Results showed that 0.32° had the worst legibility, but there was no continuous improvement as font size increased. A larger font size was found to be beneficial in reducing visual fatigue until it reached 0.95°, beyond which visual fatigue would relatively increase. Font size smaller than 0.32° should be rejected while a larger font size does not always provide more benefits. Considering legibility, visual fatigue and efficiency of text presentation, 0.84° is a relatively optimal Chinese character font size.
Collapse
Affiliation(s)
- Zhengyin Gu
- Department of Psychology, Zhejiang Sci-Tech University, Hangzhou, China
| | - Xinle Bao
- Department of Psychology, Zhejiang Sci-Tech University, Hangzhou, China
| | - Ying Zhu
- Hangzhou ROBAM Appliances Co., Ltd, Hangzhou, China
| | - Qijun Wang
- Department of Psychology, Zhejiang Sci-Tech University, Hangzhou, China
| | - Wei Gao
- Department of Psychology, Zhejiang Sci-Tech University, Hangzhou, China
| | - Duming Wang
- Department of Psychology, Zhejiang Sci-Tech University, Hangzhou, China
| | - Yu Tian
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China
| | - You Li
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China
| |
Collapse
|
4
|
Bamps K, Bertels J, Minten L, Puvrez A, Coudyzer W, De Buck S, Ector J. Phantom study of augmented reality framework to assist epicardial punctures. J Med Imaging (Bellingham) 2024; 11:035002. [PMID: 38817712 PMCID: PMC11135927 DOI: 10.1117/1.jmi.11.3.035002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Revised: 04/24/2024] [Accepted: 05/15/2024] [Indexed: 06/01/2024] Open
Abstract
Purpose The objective of this study is to evaluate the accuracy of an augmented reality (AR) system in improving guidance, accuracy, and visualization during the subxiphoidal approach for epicardial ablation. Approach An AR application was developed to project real-time needle trajectories and patient-specific 3D organs using the Hololens 2. Additionally, needle tracking was implemented to offer real-time feedback to the operator, facilitating needle navigation. The AR application was evaluated through three different experiments: examining overlay accuracy, assessing puncture accuracy, and performing pre-clinical evaluations on a phantom. Results The results of the overlay accuracy assessment for the AR system yielded 2.36 ± 2.04 mm . Additionally, the puncture accuracy utilizing the AR system yielded 1.02 ± 2.41 mm . During the pre-clinical evaluation on the phantom, needle puncture with AR guidance showed 7.43 ± 2.73 mm , whereas needle puncture without AR guidance showed 22.62 ± 9.37 mm . Conclusions Overall, the AR platform has the potential to enhance the accuracy of percutaneous epicardial access for mapping and ablation of cardiac arrhythmias, thereby reducing complications and improving patient outcomes. The significance of this study lies in the potential of AR guidance to enhance the accuracy and safety of percutaneous epicardial access.
Collapse
Affiliation(s)
- Kobe Bamps
- KU Leuven, Department of Cardiovascular Sciences, Leuven, Belgium
- KU Leuven, ESAT-PSI, Leuven, Belgium
| | | | - Lennert Minten
- KU Leuven, Department of Cardiovascular Sciences, Leuven, Belgium
| | - Alexis Puvrez
- KU Leuven, Department of Cardiovascular Sciences, Leuven, Belgium
| | | | - Stijn De Buck
- KU Leuven, ESAT-PSI, Leuven, Belgium
- KU Leuven, Department of Imaging and Pathology, Leuven, Belgium
| | - Joris Ector
- KU Leuven, Department of Cardiovascular Sciences, Leuven, Belgium
| |
Collapse
|
5
|
Arensmeyer J, Bedetti B, Schnorr P, Buermann J, Zalepugas D, Schmidt J, Feodorovici P. A System for Mixed-Reality Holographic Overlays of Real-Time Rendered 3D-Reconstructed Imaging Using a Video Pass-through Head-Mounted Display-A Pathway to Future Navigation in Chest Wall Surgery. J Clin Med 2024; 13:2080. [PMID: 38610849 PMCID: PMC11012529 DOI: 10.3390/jcm13072080] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2024] [Revised: 03/26/2024] [Accepted: 03/27/2024] [Indexed: 04/14/2024] Open
Abstract
Background: Three-dimensional reconstructions of state-of-the-art high-resolution imaging are progressively being used more for preprocedural assessment in thoracic surgery. It is a promising tool that aims to improve patient-specific treatment planning, for example, for minimally invasive or robotic-assisted lung resections. Increasingly available mixed-reality hardware based on video pass-through technology enables the projection of image data as a hologram onto the patient. We describe the novel method of real-time 3D surgical planning in a mixed-reality setting by presenting three representative cases utilizing volume rendering. Materials: A mixed-reality system was set up using a high-performance workstation running a video pass-through-based head-mounted display. Image data from computer tomography were imported and volume-rendered in real-time to be customized through live editing. The image-based hologram was projected onto the patient, highlighting the regions of interest. Results: Three oncological cases were selected to explore the potentials of the mixed-reality system. Two of them presented large tumor masses in the thoracic cavity, while a third case presented an unclear lesion of the chest wall. We aligned real-time rendered 3D holographic image data onto the patient allowing us to investigate the relationship between anatomical structures and their respective body position. Conclusions: The exploration of holographic overlay has proven to be promising in improving preprocedural surgical planning, particularly for complex oncological tasks in the thoracic surgical field. Further studies on outcome-related surgical planning and navigation should therefore be conducted. Ongoing technological progress of extended reality hardware and intelligent software features will most likely enhance applicability and the range of use in surgical fields within the near future.
Collapse
Affiliation(s)
- Jan Arensmeyer
- Division of Thoracic Surgery, Department of General, Thoracic and Vascular Surgery, University Hospital Bonn, 53127 Bonn, Germany (P.F.)
- Bonn Surgical Technology Center (BOSTER), University Hospital Bonn, 53227 Bonn, Germany
| | - Benedetta Bedetti
- Division of Thoracic Surgery, Department of General, Thoracic and Vascular Surgery, University Hospital Bonn, 53127 Bonn, Germany (P.F.)
- Department of Thoracic Surgery, Helios Hospital Bonn/Rhein-Sieg, 53123 Bonn, Germany
| | - Philipp Schnorr
- Division of Thoracic Surgery, Department of General, Thoracic and Vascular Surgery, University Hospital Bonn, 53127 Bonn, Germany (P.F.)
- Department of Thoracic Surgery, Helios Hospital Bonn/Rhein-Sieg, 53123 Bonn, Germany
| | - Jens Buermann
- Division of Thoracic Surgery, Department of General, Thoracic and Vascular Surgery, University Hospital Bonn, 53127 Bonn, Germany (P.F.)
- Department of Thoracic Surgery, Helios Hospital Bonn/Rhein-Sieg, 53123 Bonn, Germany
| | - Donatas Zalepugas
- Division of Thoracic Surgery, Department of General, Thoracic and Vascular Surgery, University Hospital Bonn, 53127 Bonn, Germany (P.F.)
- Department of Thoracic Surgery, Helios Hospital Bonn/Rhein-Sieg, 53123 Bonn, Germany
| | - Joachim Schmidt
- Division of Thoracic Surgery, Department of General, Thoracic and Vascular Surgery, University Hospital Bonn, 53127 Bonn, Germany (P.F.)
- Bonn Surgical Technology Center (BOSTER), University Hospital Bonn, 53227 Bonn, Germany
- Department of Thoracic Surgery, Helios Hospital Bonn/Rhein-Sieg, 53123 Bonn, Germany
| | - Philipp Feodorovici
- Division of Thoracic Surgery, Department of General, Thoracic and Vascular Surgery, University Hospital Bonn, 53127 Bonn, Germany (P.F.)
- Bonn Surgical Technology Center (BOSTER), University Hospital Bonn, 53227 Bonn, Germany
| |
Collapse
|
6
|
Morley CT, Arreola DM, Qian L, Lynn AL, Veigulis ZP, Osborne TF. Mixed Reality Surgical Navigation System; Positional Accuracy Based on Food and Drug Administration Standard. Surg Innov 2024; 31:48-57. [PMID: 38019844 PMCID: PMC10773158 DOI: 10.1177/15533506231217620] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2023]
Abstract
BACKGROUND Computer assisted surgical navigation systems are designed to improve outcomes by providing clinicians with procedural guidance information. The use of new technologies, such as mixed reality, offers the potential for more intuitive, efficient, and accurate procedural guidance. The goal of this study is to assess the positional accuracy and consistency of a clinical mixed reality system that utilizes commercially available wireless head-mounted displays (HMDs), custom software, and localization instruments. METHODS Independent teams using the second-generation Microsoft HoloLens© hardware, Medivis SurgicalAR© software, and localization instruments, tested the accuracy of the combined system at different institutions, times, and locations. The ASTM F2554-18 consensus standard for computer-assisted surgical systems, as recognized by the U.S. FDA, was utilized to measure the performance. 288 tests were performed. RESULTS The system demonstrated consistent results, with an average accuracy performance that was better than one millimeter (.75 ± SD .37 mm). CONCLUSION Independently acquired positional tracking accuracies exceed conventional in-market surgical navigation tracking systems and FDA standards. Importantly, the performance was achieved at two different institutions, using an international testing standard, and with a system that included a commercially available off-the-shelf wireless head mounted display and software.
Collapse
Affiliation(s)
| | - David M. Arreola
- US Department of Veterans Affairs, Palo Alto Healthcare System, Palo Alto, CA, USA
| | | | | | - Zachary P. Veigulis
- US Department of Veterans Affairs, Palo Alto Healthcare System, Palo Alto, CA, USA
- Department of Business Analytics, Tippie College of Business, University of Iowa, Iowa, IA, USA
| | - Thomas F. Osborne
- US Department of Veterans Affairs, Palo Alto Healthcare System, Palo Alto, CA, USA
- Department of Radiology, Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
7
|
Ali JT, Yang G, Green CA, Reed BL, Madani A, Ponsky TA, Hazey J, Rothenberg SS, Schlachta CM, Oleynikov D, Szoka N. Defining digital surgery: a SAGES white paper. Surg Endosc 2024; 38:475-487. [PMID: 38180541 DOI: 10.1007/s00464-023-10551-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2023] [Accepted: 10/17/2023] [Indexed: 01/06/2024]
Abstract
BACKGROUND Digital surgery is a new paradigm within the surgical innovation space that is rapidly advancing and encompasses multiple areas. METHODS This white paper from the SAGES Digital Surgery Working Group outlines the scope of digital surgery, defines key terms, and analyzes the challenges and opportunities surrounding this disruptive technology. RESULTS In its simplest form, digital surgery inserts a computer interface between surgeon and patient. We divide the digital surgery space into the following elements: advanced visualization, enhanced instrumentation, data capture, data analytics with artificial intelligence/machine learning, connectivity via telepresence, and robotic surgical platforms. We will define each area, describe specific terminology, review current advances as well as discuss limitations and opportunities for future growth. CONCLUSION Digital Surgery will continue to evolve and has great potential to bring value to all levels of the healthcare system. The surgical community has an essential role in understanding, developing, and guiding this emerging field.
Collapse
Affiliation(s)
- Jawad T Ali
- University of Texas at Austin, Austin, TX, USA
| | - Gene Yang
- University at Buffalo, Buffalo, NY, USA
| | | | | | - Amin Madani
- University of Toronto, Toronto, ON, Canada
- Surgical Artificial Intelligence Research Academy, University Health Network, Toronto, ON, Canada
| | - Todd A Ponsky
- Cincinnati Children's Hospital Medical Center, Cincinnati, OH, USA
| | | | | | | | - Dmitry Oleynikov
- Monmouth Medical Center, Robert Wood Johnson Barnabas Health, Rutgers School of Medicine, Long Branch, NJ, USA
| | - Nova Szoka
- Department of Surgery, West Virginia University, Suite 7500 HSS, PO Box 9238, Morgantown, WV, 26506-9238, USA.
| |
Collapse
|
8
|
Volmer B, Liu JS, Matthews B, Bornkessel-Schlesewsky I, Feiner S, Thomas BH. Multi-Level Precues for Guiding Tasks Within and Between Workspaces in Spatial Augmented Reality. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2023; 29:4449-4459. [PMID: 37874709 DOI: 10.1109/tvcg.2023.3320246] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/26/2023]
Abstract
We explore Spatial Augmented Reality (SAR) precues (predictive cues) for procedural tasks within and between workspaces and for visualizing multiple upcoming steps in advance. We designed precues based on several factors: cue type, color transparency, and multi-level (number of precues). Precues were evaluated in a procedural task requiring the user to press buttons in three surrounding workspaces. Participants performed fastest in conditions where tasks were linked with line cues with different levels of color transparency. Precue performance was also affected by whether the next task was in the same workspace or a different one.
Collapse
|
9
|
Schneider M, Kunz C, Wirtz CR, Mathis-Ullrich F, Pala A, Hlavac M. Augmented Reality-Assisted versus Freehand Ventriculostomy in a Head Model. J Neurol Surg A Cent Eur Neurosurg 2023; 84:562-569. [PMID: 37402395 DOI: 10.1055/s-0042-1759827] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/06/2023]
Abstract
BACKGROUND Ventriculostomy (VST) is a frequent neurosurgical procedure. Freehand catheter placement represents the standard current practice. However, multiple attempts are often required. We present augmented reality (AR) headset guided VST with in-house developed head models. We conducted a proof of concept study in which we tested AR-guided as well as freehand VST. Repeated AR punctures were conducted to investigate if a learning curve can be derived. METHODS Five custom-made 3D-printed head models, each holding an anatomically different ventricular system, were filled with agarose gel. Eleven surgeons placed two AR-guided as well as two freehand ventricular drains per head. A subgroup of four surgeons did a total of three series of AR-guided punctures each to test for a learning curve. A Microsoft HoloLens served as the hardware platform. The marker-based tracking did not require rigid head fixation. Catheter tip position was evaluated in computed tomography scans. RESULTS Marker-tracking, image segmentation, and holographic display worked satisfactorily. In freehand VST, a success rate of 72.7% was achieved, which was higher than under AR guidance (68.2%, difference not statistically significant). Repeated AR-guided punctures increased the success rate from 65 to 95%. We assume a steep learning curve as repeated AR-guided punctures led to an increase in successful attempts. Overall user experience showed positive feedback. CONCLUSIONS We achieved promising results that encourage the continued development and technical improvement. However, several more developmental steps have to be taken before an application in humans can be considered. In the future, AR headset-based holograms have the potential to serve as a compact navigational help inside and outside the operating room.
Collapse
Affiliation(s)
- Max Schneider
- Department of Neurosurgery, University Hospital Ulm, Ulm, Germany
| | - Christian Kunz
- Institute for Anthropomatics and Robotics - Health Robotics and Automation (HERA), KIT, Karlsruhe, Germany
| | | | - Franziska Mathis-Ullrich
- Institute for Anthropomatics and Robotics - Health Robotics and Automation (HERA), KIT, Karlsruhe, Germany
| | - Andrej Pala
- Department of Neurosurgery, University Hospital Ulm, Ulm, Germany
| | - Michal Hlavac
- Department of Neurosurgery, University Hospital Ulm, Ulm, Germany
| |
Collapse
|
10
|
Hayasaka T, Kawano K, Onodera Y, Suzuki H, Nakane M, Kanoto M, Kawamae K. Comparison of accuracy between augmented reality/mixed reality techniques and conventional techniques for epidural anesthesia using a practice phantom model kit. BMC Anesthesiol 2023; 23:171. [PMID: 37210521 DOI: 10.1186/s12871-023-02133-w] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Accepted: 05/10/2023] [Indexed: 05/22/2023] Open
Abstract
BACKGROUND This study used an epidural anesthesia practice kit (model) to evaluate the accuracy of epidural anesthesia using standard techniques (blind) and augmented/mixed reality technology and whether visualization using augmented/mixed reality technology would facilitate epidural anesthesia. METHODS This study was conducted at the Yamagata University Hospital (Yamagata, Japan) between February and June 2022. Thirty medical students with no experience in epidural anesthesia were randomly divided into augmented reality (-), augmented reality (+), and semi-augmented reality groups, with 10 students in each group. Epidural anesthesia was performed using the paramedian approach with an epidural anesthesia practice kit. The augmented reality (-) group performed epidural anesthesia without HoloLens2Ⓡ and the augmented reality (+) group with HoloLens2Ⓡ. The semi-augmented reality group performed epidural anesthesia without HoloLens2Ⓡ after 30 s of image construction of the spine using HoloLens2Ⓡ. The epidural space puncture point distance between the ideal insertion needle and participant's insertion needle was compared. RESULTS Four medical students in the augmented reality (-), zero in the augmented reality (+), and one in the semi-augmented reality groups failed to insert the needle into the epidural space. The epidural space puncture point distance for the augmented reality (-), augmented reality (+), and semi-augmented reality groups were 8.7 (5.7-14.3) mm, 3.5 (1.8-8.0) mm (P = 0.017), and 4.9 (3.2-5.9) mm (P = 0.027), respectively; a significant difference was observed between the two groups. CONCLUSIONS Augmented/mixed reality technology has the potential to contribute significantly to the improvement of epidural anesthesia techniques.
Collapse
Affiliation(s)
- Tatsuya Hayasaka
- Department of Anesthesiology, Yamagata University Hospital, 2-2-2 Iidanishi, Yamagata City, Yamagata, 990-9585, Japan.
| | - Kazuharu Kawano
- Department of Medicine, Yamagata University School of Medicine, Yamagata, Japan
| | - Yu Onodera
- Critical Care Center, Yamagata University Hospital, Yamagata, Japan
| | - Hiroto Suzuki
- Critical Care Center, Yamagata University Hospital, Yamagata, Japan
| | - Masaki Nakane
- Department of Emergency and Critical Care Medicine, Yamagata University Hospital, Yamagata, Japan
| | - Masafumi Kanoto
- Department of Radiology, Division of Diagnostic Radiology, Yamagata University Hospital, Yamagata, Japan
| | - Kaneyuki Kawamae
- Department of Anesthesiology, Yamagata University Hospital, 2-2-2 Iidanishi, Yamagata City, Yamagata, 990-9585, Japan
| |
Collapse
|
11
|
Lui JT, Dahm V, Chen JM, Lin VY, Irish JC, Le TN, Chan HHL. Using augmented reality to guide bone conduction device implantation. Sci Rep 2023; 13:7182. [PMID: 37137995 PMCID: PMC10156678 DOI: 10.1038/s41598-023-33523-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2022] [Accepted: 04/14/2023] [Indexed: 05/05/2023] Open
Abstract
Exact placement of bone conduction implants requires avoidance of critical structures. Existing guidance technologies for intraoperative placement have lacked widespread adoption given accessibility challenges and significant cognitive loading. The purpose of this study is to examine the application of augmented reality (AR) guided surgery on accuracy, duration, and ease on bone conduction implantation. Five surgeons surgically implanted two different types of conduction implants on cadaveric specimens with and without AR projection. Pre- and postoperative computer tomography scans were superimposed to calculate centre-to-centre distances and angular accuracies. Wilcoxon signed-rank testing was used to compare centre-to-centre (C-C) and angular accuracies between the control and experimental arms. Additionally, projection accuracy was derived from the distance between the bony fiducials and the projected fiducials using image guidance coordinates. Both operative time (4.3 ± 1.2 min. vs. 6.6 ± 3.5 min., p = 0.030) and centre-to-centre distances surgery (1.9 ± 1.6 mm vs. 9.0 ± 5.3 mm, p < 0.001) were significantly less in augmented reality guided surgery. The difference in angular accuracy, however, was not significantly different. The overall average distance between the bony fiducial markings and the AR projected fiducials was 1.7 ± 0.6 mm. With direct intraoperative reference, AR-guided surgery enhances bone conduction implant placement while reduces operative time when compared to conventional surgical planning.
Collapse
Affiliation(s)
- Justin T Lui
- Section of Otolaryngology-Head & Neck Surgery, Department of Surgery, Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - Valerie Dahm
- Department of Otolaryngology-Head & Neck Surgery, Temerty School of Medicine, University of Toronto, Toronto, Canada
| | - Joseph M Chen
- Department of Otolaryngology-Head & Neck Surgery, Temerty School of Medicine, University of Toronto, Toronto, Canada
| | - Vincent Y Lin
- Department of Otolaryngology-Head & Neck Surgery, Temerty School of Medicine, University of Toronto, Toronto, Canada
| | - Jonathan C Irish
- Department of Otolaryngology-Head & Neck Surgery, Temerty School of Medicine, University of Toronto, Toronto, Canada
- Guided Therapeutics (GTx) Program, Techna Research Institute, University Health Network, Toronto, Canada
| | - Trung N Le
- Department of Otolaryngology-Head & Neck Surgery, Temerty School of Medicine, University of Toronto, Toronto, Canada
| | - Harley H L Chan
- Guided Therapeutics (GTx) Program, Techna Research Institute, University Health Network, Toronto, Canada.
- Techna Institute for the Advancement of Technology for Health, University Health Network, 100 College Street, Room 7-207, MaRS Building, Princess Margaret Cancer Research Tower, 7th Floor (STTARR), Toronto, ON, M5G 1P5, Canada.
| |
Collapse
|
12
|
Gsaxner C, Li J, Pepe A, Jin Y, Kleesiek J, Schmalstieg D, Egger J. The HoloLens in medicine: A systematic review and taxonomy. Med Image Anal 2023; 85:102757. [PMID: 36706637 DOI: 10.1016/j.media.2023.102757] [Citation(s) in RCA: 20] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2022] [Revised: 01/05/2023] [Accepted: 01/18/2023] [Indexed: 01/22/2023]
Abstract
The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.
Collapse
Affiliation(s)
- Christina Gsaxner
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; BioTechMed, 8010 Graz, Austria.
| | - Jianning Li
- Institute of AI in Medicine, University Medicine Essen, 45131 Essen, Germany; Cancer Research Center Cologne Essen, University Medicine Essen, 45147 Essen, Germany
| | - Antonio Pepe
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; BioTechMed, 8010 Graz, Austria
| | - Yuan Jin
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; Research Center for Connected Healthcare Big Data, Zhejiang Lab, Hangzhou, 311121 Zhejiang, China
| | - Jens Kleesiek
- Institute of AI in Medicine, University Medicine Essen, 45131 Essen, Germany; Cancer Research Center Cologne Essen, University Medicine Essen, 45147 Essen, Germany
| | - Dieter Schmalstieg
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; BioTechMed, 8010 Graz, Austria
| | - Jan Egger
- Institute of Computer Graphics and Vision, Graz University of Technology, 8010 Graz, Austria; Institute of AI in Medicine, University Medicine Essen, 45131 Essen, Germany; BioTechMed, 8010 Graz, Austria; Cancer Research Center Cologne Essen, University Medicine Essen, 45147 Essen, Germany
| |
Collapse
|
13
|
Mamone V, Ferrari V, D’Amato R, Condino S, Cattari N, Cutolo F. Head-Mounted Projector for Manual Precision Tasks: Performance Assessment. SENSORS (BASEL, SWITZERLAND) 2023; 23:3494. [PMID: 37050554 PMCID: PMC10098766 DOI: 10.3390/s23073494] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Revised: 03/22/2023] [Accepted: 03/24/2023] [Indexed: 06/19/2023]
Abstract
The growing interest in augmented reality applications has led to an in-depth look at the performance of head-mounted displays and their testing in numerous domains. Other devices for augmenting the real world with virtual information are presented less frequently and usually focus on the description of the device rather than on its performance analysis. This is the case of projected augmented reality, which, compared to head-worn AR displays, offers the advantages of being simultaneously accessible by multiple users whilst preserving user awareness of the environment and feeling of immersion. This work provides a general evaluation of a custom-made head-mounted projector for the aid of precision manual tasks through an experimental protocol designed for investigating spatial and temporal registration and their combination. The results of the tests show that the accuracy (0.6±0.1 mm of spatial registration error) and motion-to-photon latency (113±12 ms) make the proposed solution suitable for guiding precision tasks.
Collapse
Affiliation(s)
- Virginia Mamone
- EndoCAS Center for Computer-Assisted Surgery, University of Pisa, 56124 Pisa, Italy (S.C.); (N.C.); (F.C.)
- Azienda Ospedaliero Universitaria Pisana, 56126 Pisa, Italy
| | - Vincenzo Ferrari
- EndoCAS Center for Computer-Assisted Surgery, University of Pisa, 56124 Pisa, Italy (S.C.); (N.C.); (F.C.)
- Information Engineering Department, University of Pisa, 56126 Pisa, Italy
| | - Renzo D’Amato
- EndoCAS Center for Computer-Assisted Surgery, University of Pisa, 56124 Pisa, Italy (S.C.); (N.C.); (F.C.)
- Information Engineering Department, University of Pisa, 56126 Pisa, Italy
| | - Sara Condino
- EndoCAS Center for Computer-Assisted Surgery, University of Pisa, 56124 Pisa, Italy (S.C.); (N.C.); (F.C.)
- Information Engineering Department, University of Pisa, 56126 Pisa, Italy
| | - Nadia Cattari
- EndoCAS Center for Computer-Assisted Surgery, University of Pisa, 56124 Pisa, Italy (S.C.); (N.C.); (F.C.)
- Information Engineering Department, University of Pisa, 56126 Pisa, Italy
| | - Fabrizio Cutolo
- EndoCAS Center for Computer-Assisted Surgery, University of Pisa, 56124 Pisa, Italy (S.C.); (N.C.); (F.C.)
- Information Engineering Department, University of Pisa, 56126 Pisa, Italy
| |
Collapse
|
14
|
Zari G, Condino S, Cutolo F, Ferrari V. Magic Leap 1 versus Microsoft HoloLens 2 for the Visualization of 3D Content Obtained from Radiological Images. SENSORS (BASEL, SWITZERLAND) 2023; 23:3040. [PMID: 36991751 PMCID: PMC10054537 DOI: 10.3390/s23063040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/24/2023] [Revised: 03/01/2023] [Accepted: 03/08/2023] [Indexed: 06/19/2023]
Abstract
The adoption of extended reality solutions is growing rapidly in the healthcare world. Augmented reality (AR) and virtual reality (VR) interfaces can bring advantages in various medical-health sectors; it is thus not surprising that the medical MR market is among the fastest-growing ones. The present study reports on a comparison between two of the most popular MR head-mounted displays, Magic Leap 1 and Microsoft HoloLens 2, for the visualization of 3D medical imaging data. We evaluate the functionalities and performance of both devices through a user-study in which surgeons and residents assessed the visualization of 3D computer-generated anatomical models. The digital content is obtained through a dedicated medical imaging suite (Verima imaging suite) developed by the Italian start-up company (Witapp s.r.l.). According to our performance analysis in terms of frame rate, there are no significant differences between the two devices. The surgical staff expressed a clear preference for Magic Leap 1, particularly for the better visualization quality and the ease of interaction with the 3D virtual content. Nonetheless, even though the results of the questionnaire were slightly more positive for Magic Leap 1, the spatial understanding of the 3D anatomical model in terms of depth relations and spatial arrangement was positively evaluated for both devices.
Collapse
Affiliation(s)
- Giulia Zari
- Information Engineering Department, University of Pisa, Via Girolamo Caruso, 16, 56122 Pisa, Italy; (G.Z.); (S.C.); (V.F.)
| | - Sara Condino
- Information Engineering Department, University of Pisa, Via Girolamo Caruso, 16, 56122 Pisa, Italy; (G.Z.); (S.C.); (V.F.)
- EndoCAS Center, Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, 56126 Pisa, Italy
| | - Fabrizio Cutolo
- Information Engineering Department, University of Pisa, Via Girolamo Caruso, 16, 56122 Pisa, Italy; (G.Z.); (S.C.); (V.F.)
- EndoCAS Center, Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, 56126 Pisa, Italy
| | - Vincenzo Ferrari
- Information Engineering Department, University of Pisa, Via Girolamo Caruso, 16, 56122 Pisa, Italy; (G.Z.); (S.C.); (V.F.)
- EndoCAS Center, Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, 56126 Pisa, Italy
| |
Collapse
|
15
|
Ma L, Huang T, Wang J, Liao H. Visualization, registration and tracking techniques for augmented reality guided surgery: a review. Phys Med Biol 2023; 68. [PMID: 36580681 DOI: 10.1088/1361-6560/acaf23] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2022] [Accepted: 12/29/2022] [Indexed: 12/31/2022]
Abstract
Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories ofin situvisualization and nonin situvisualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.
Collapse
Affiliation(s)
- Longfei Ma
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, People's Republic of China
| | - Tianqi Huang
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, People's Republic of China
| | - Jie Wang
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, People's Republic of China
| | - Hongen Liao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, People's Republic of China
| |
Collapse
|
16
|
Shimizu T, Oba T, Ito KI. The Advantage of Using an Optical See-Through Head-Mounted Display in Ultrasonography-Guided Needle Biopsy Procedures: A Prospective Randomized Study. J Clin Med 2023; 12:jcm12020512. [PMID: 36675443 PMCID: PMC9865023 DOI: 10.3390/jcm12020512] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2022] [Revised: 12/26/2022] [Accepted: 01/05/2023] [Indexed: 01/11/2023] Open
Abstract
An optical see-through head-mounted display (OST-HMD) can potentially improve the safety and accuracy of ultrasonography (US)-guided fine-needle aspiration. We aimed to evaluate the usefulness of an OST-HMD in US-guided needle-puncture procedures. We conducted a prospective randomized controlled study in which we compared the accuracy and safety of the US-guided needle puncture procedure and the stress on the practitioner when using OST-HMD versus standard US display (SUD). Inexperienced medical students were enrolled and randomly divided into two groups. A breast phantom was used to evaluate the required time and accuracy of the US-guided needle puncture. Practitioner stress was quantified using a visual analog scale (VAS). When the procedure was performed for the first time, the time required to reach the target lesion at a shallow depth was significantly shorter in the OST-HMD group (39.8 ± 39.9 s) than in the SUD group (71.0 ± 81.0 s) (p = 0.01). Using the OST-HMD significantly reduced the unintentional puncture of a non-target lesion (p = 0.01). Furthermore, the stress felt by the practitioners when capturing the image of the target lesion (p < 0.001), inserting and advancing the needle more deeply (p < 0.001), and puncturing the target lesion (p < 0.001) was significantly reduced in the OST-HMD group compared with that in the SUD group. Use of OST-HMD may improve the accuracy and safety of US-guided needle puncture procedures and may reduce practitioner stress during the procedure.
Collapse
|
17
|
Wenk N, Penalver-Andres J, Buetler KA, Nef T, Müri RM, Marchal-Crespo L. Effect of immersive visualization technologies on cognitive load, motivation, usability, and embodiment. VIRTUAL REALITY 2023; 27:307-331. [PMID: 36915633 PMCID: PMC9998603 DOI: 10.1007/s10055-021-00565-8] [Citation(s) in RCA: 15] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/31/2020] [Accepted: 07/22/2021] [Indexed: 05/09/2023]
Abstract
Virtual reality (VR) is a promising tool to promote motor (re)learning in healthy users and brain-injured patients. However, in current VR-based motor training, movements of the users performed in a three-dimensional space are usually visualized on computer screens, televisions, or projection systems, which lack depth cues (2D screen), and thus, display information using only monocular depth cues. The reduced depth cues and the visuospatial transformation from the movements performed in a three-dimensional space to their two-dimensional indirect visualization on the 2D screen may add cognitive load, reducing VR usability, especially in users suffering from cognitive impairments. These 2D screens might further reduce the learning outcomes if they limit users' motivation and embodiment, factors previously associated with better motor performance. The goal of this study was to evaluate the potential benefits of more immersive technologies using head-mounted displays (HMDs). As a first step towards potential clinical implementation, we ran an experiment with 20 healthy participants who simultaneously performed a 3D motor reaching and a cognitive counting task using: (1) (immersive) VR (IVR) HMD, (2) augmented reality (AR) HMD, and (3) computer screen (2D screen). In a previous analysis, we reported improved movement quality when movements were visualized with IVR than with a 2D screen. Here, we present results from the analysis of questionnaires to evaluate whether the visualization technology impacted users' cognitive load, motivation, technology usability, and embodiment. Reports on cognitive load did not differ across visualization technologies. However, IVR was more motivating and usable than AR and the 2D screen. Both IVR and AR rea ched higher embodiment level than the 2D screen. Our results support our previous finding that IVR HMDs seem to be more suitable than the common 2D screens employed in VR-based therapy when training 3D movements. For AR, it is still unknown whether the absence of benefit over the 2D screen is due to the visualization technology per se or to technical limitations specific to the device.
Collapse
Affiliation(s)
- N. Wenk
- Motor Learning and Neurorehabilitation Laboratory, ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
| | - J. Penalver-Andres
- Motor Learning and Neurorehabilitation Laboratory, ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
| | - K. A. Buetler
- Motor Learning and Neurorehabilitation Laboratory, ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
| | - T. Nef
- Gerontechnology & Rehabilitation, ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
| | - R. M. Müri
- Gerontechnology & Rehabilitation, ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
- Department of Neurology, University Neurorehabilitation, University Hospital Bern (Inselspital), University of Bern, Bern, Switzerland
| | - L. Marchal-Crespo
- Motor Learning and Neurorehabilitation Laboratory, ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
- Department of Cognitive Robotics, Delft University of Technology, Delft, The Netherlands
| |
Collapse
|
18
|
Chegini S, Edwards E, McGurk M, Clarkson M, Schilling C. Systematic review of techniques used to validate the registration of augmented-reality images using a head-mounted device to navigate surgery. Br J Oral Maxillofac Surg 2023; 61:19-27. [PMID: 36513525 DOI: 10.1016/j.bjoms.2022.08.007] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 07/31/2022] [Accepted: 08/17/2022] [Indexed: 12/14/2022]
Abstract
Augmented-reality (AR) head-mounted devices (HMD) allow the wearer to have digital images superposed on to their field of vision. They are being used to superpose annotations on to the surgical field akin to a navigation system. This review examines published validation studies on HMD-AR systems, their reported protocols, and outcomes. The aim was to establish commonalities and an acceptable registration outcome. Multiple databases were systematically searched for relevant articles between January 2015 and January 2021. Studies that examined the registration of AR content using a HMD to guide surgery were eligible for inclusion. The country of origin, year of publication, medical specialty, HMD device, software, and method of registration, were recorded. A meta-analysis of the mean registration error was conducted. A total of 4784 papers were identified, of which 23 met the inclusion criteria. They included studies using HoloLens (Microsoft) (n = 22) and nVisor ST60 (NVIS Inc) (n = 1). Sixty-six per cent of studies were in hard tissue specialties. Eleven studies reported registration errors using pattern markers (mean (SD) 2.6 (1.8) mm), and four reported registration errors using surface markers (mean (SD) 3.8 (3.7) mm). Three studies reported registration errors using manual alignment (mean (SD) 2.2 (1.3) mm). The majority of studies in this review used in-house software with a variety of registration methods and reported errors. The mean registration error calculated in this study can be considered as a minimum acceptable standard. It should be taken into consideration when procedural applications are selected.
Collapse
Affiliation(s)
- Soudeh Chegini
- University College London, Charles Bell House, 43-45 Foley St, London W1W 7TY, United Kingdom.
| | - Eddie Edwards
- University College London, Charles Bell House, 43-45 Foley St, London W1W 7TY, United Kingdom
| | - Mark McGurk
- University College London, Charles Bell House, 43-45 Foley St, London W1W 7TY, United Kingdom
| | - Matthew Clarkson
- University College London, Charles Bell House, 43-45 Foley St, London W1W 7TY, United Kingdom
| | - Clare Schilling
- University College London, Charles Bell House, 43-45 Foley St, London W1W 7TY, United Kingdom
| |
Collapse
|
19
|
Birlo M, Edwards PJE, Yoo S, Dromey B, Vasconcelos F, Clarkson MJ, Stoyanov D. CAL-Tutor: A HoloLens 2 Application for Training in Obstetric Sonography and User Motion Data Recording. J Imaging 2022; 9:6. [PMID: 36662104 PMCID: PMC9860994 DOI: 10.3390/jimaging9010006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Revised: 11/30/2022] [Accepted: 12/20/2022] [Indexed: 12/30/2022] Open
Abstract
Obstetric ultrasound (US) training teaches the relationship between foetal anatomy and the viewed US slice to enable navigation to standardised anatomical planes (head, abdomen and femur) where diagnostic measurements are taken. This process is difficult to learn, and results in considerable inter-operator variability. We propose the CAL-Tutor system for US training based on a US scanner and phantom, where a model of both the baby and the US slice are displayed to the trainee in its physical location using the HoloLens 2. The intention is that AR guidance will shorten the learning curve for US trainees and improve spatial awareness. In addition to the AR guidance, we also record many data streams to assess user motion and the learning process. The HoloLens 2 provides eye gaze, head and hand position, ARToolkit and NDI Aurora tracking gives the US probe positions and an external camera records the overall scene. These data can provide a rich source for further analysis, such as distinguishing expert from novice motion. We have demonstrated the system in a sample of engineers. Feedback suggests that the system helps novice users navigate the US probe to the standard plane. The data capture is successful and initial data visualisations show that meaningful information about user behaviour can be captured. Initial feedback is encouraging and shows improved user assessment where AR guidance is provided.
Collapse
Affiliation(s)
- Manuel Birlo
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), Charles Bell House, 43–45 Foley Street, London W1W 7TY, UK
| | - Philip J. Eddie Edwards
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), Charles Bell House, 43–45 Foley Street, London W1W 7TY, UK
| | - Soojeong Yoo
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), Charles Bell House, 43–45 Foley Street, London W1W 7TY, UK
- UCL Interaction Centre (UCLIC), University College London, 66-72 Gower Street, London WC1E 6EA, UK
| | - Brian Dromey
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), Charles Bell House, 43–45 Foley Street, London W1W 7TY, UK
- UCL EGA Institute for Women’s Health, Medical School Building, 74 Huntley Street, London WC1E 6AU, UK
| | - Francisco Vasconcelos
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), Charles Bell House, 43–45 Foley Street, London W1W 7TY, UK
| | - Matthew J. Clarkson
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), Charles Bell House, 43–45 Foley Street, London W1W 7TY, UK
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), Charles Bell House, 43–45 Foley Street, London W1W 7TY, UK
| |
Collapse
|
20
|
Minimally invasive and invasive liver surgery based on augmented reality training: a review of the literature. J Robot Surg 2022; 17:753-763. [DOI: 10.1007/s11701-022-01499-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2022] [Accepted: 11/14/2022] [Indexed: 11/29/2022]
|
21
|
Yu K, Zacharis K, Eck U, Navab N. Projective Bisector Mirror (PBM): Concept and Rationale. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2022; 28:3694-3704. [PMID: 36048998 DOI: 10.1109/tvcg.2022.3203108] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Our world is full of cameras, whether they are installed in the environment or integrated into mobile devices such as mobile phones or head-mounted displays. Displaying external camera views in our egocentric view with a picture-in-picture approach allows us to understand their view; however, it would not allow us to correlate their viewpoint with our perceived reality. We introduce Projective Bisector Mirrors for visualizing a camera view comprehensibly in the egocentric view of an observer with the metaphor of a virtual mirror. Our concept projects the image of a capturing camera onto the bisecting plane between the capture and the observer camera. We present extensive mathematical descriptions of this novel paradigm for multi-view visualization, discuss the effects of tracking errors and provide concrete implementation for multiple exemplary use-cases.
Collapse
|
22
|
Kim YJ, Kumaran R, Sayyad E, Milner A, Bullock T, Giesbrecht B, Hollerer T. Investigating Search Among Physical and Virtual Objects Under Different Lighting Conditions. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2022; 28:3788-3798. [PMID: 36048996 DOI: 10.1109/tvcg.2022.3203093] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
By situating computer-generated content in the physical world, mobile augmented reality (AR) can support many tasks that involve effective search and inspection of physical environments. Currently, there is limited information regarding the viability of using AR in realistic wide-area outdoor environments and how AR experiences affect human behavior in these environments. Here, we conducted a wide-area outdoor AR user study ($n=48$) using a commercially available AR headset (Microsoft Hololens 2) to compare (1) user interactions with physical and virtual objects in the environment (2) the effects of different lighting conditions on user behavior and AR experience and (3) the impact of varying cognitive load on AR task performance. Participants engaged in a treasure hunt task where they searched for and classified virtual target items (green "gems") in an augmented outdoor courtyard scene populated with physical and virtual objects. Cognitive load was manipulated so that in half the search trials users were required to monitor an audio stream and respond to specific target sounds. Walking paths, head orientation and eye gaze information were measured, and users were queried about their memory of encountered objects and provided feedback on the experience. Key findings included (1) Participants self-reported significantly lower comfort in the ambient natural light condition, with virtual objects more visible and participants more likely to walk into physical objects at night; (2) recall for physical objects was worse than for virtual objects, (3) participants discovered more gems hidden behind virtual objects than physical objects, implying higher attention on virtual objects and (4) dual-tasking modified search behavior. These results suggest there are important technical, perceptual and cognitive factors that must be considered if the full potential of "anywhere and anytime mobile AR" is to be realized.
Collapse
|
23
|
Guinet AL, Bouyer G, Otmane S, Desailly E. Visual Feedback in Augmented Reality to Walk at Predefined Speed Cross-Sectional Study Including Children With Cerebral Palsy. IEEE Trans Neural Syst Rehabil Eng 2022; 30:2322-2331. [PMID: 35951576 DOI: 10.1109/tnsre.2022.3198243] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
In an augmented reality environment, the range of possible real-time visual feedback is extensive. This study aimed to compare the impact of six scenarios in augmented reality combining four visual feedback characteristics on achieving a target walking speed. The six scenarios have been developed for Microsoft Hololens augmented reality headset. The four feedback characteristics that we have varied were: Color; Spatial anchoring; Speed of the feedback, and Persistence. Each characteristic could have different values (for example, the color could be unicolor, bicolor, or gradient). Participants had to walk for two consecutive walking trials for each scenario: at their maximal speed and an intermediate speed. Mean speed, percentage of time spent above or around target speed, and time to reach target speed were compared between scenarios using mixed linear models. A total of 25 children with disabilities have been included. The feasibility and user experience were excellent. Mean speed during scenario 6, which displayed feedback with gradient color, attached to the world, with a speed relative to the player equal to his speed, and that disappeared over time, was significantly higher than other scenarios and control (p =0.003). Participants spent 80.98% of time above target speed during scenario 6. This scenario mixed the best combination of feedback characteristics to exceed the target walking speed (p=0.0058). Scenarios 5 and 6, which shared the same feedback characteristics for spatial anchoring (world-locked) and feedback speed (equal to the player speed), decreased the time to reach the target speed (p=0.019). Delivering multi-modal feedback has been recognized as more effective for improving motor performance. Therefore, our results showed that not all visual feedback had the same impact on performance. Further studies are required to test the weight of each feedback characteristic and their possible interactions inside each scenario. This study was registered in the ClinicalTrials.gov database (NCT04460833).
Collapse
|
24
|
Multicenter assessment of augmented reality registration methods for image-guided interventions. Radiol Med 2022; 127:857-865. [DOI: 10.1007/s11547-022-01515-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2021] [Accepted: 06/13/2022] [Indexed: 10/17/2022]
|
25
|
Doughty M, Ghugre NR, Wright GA. Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery. J Imaging 2022; 8:jimaging8070203. [PMID: 35877647 PMCID: PMC9318659 DOI: 10.3390/jimaging8070203] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2022] [Revised: 07/15/2022] [Accepted: 07/18/2022] [Indexed: 02/01/2023] Open
Abstract
We conducted a systematic review of recent literature to understand the current challenges in the use of optical see-through head-mounted displays (OST-HMDs) for augmented reality (AR) assisted surgery. Using Google Scholar, 57 relevant articles from 1 January 2021 through 18 March 2022 were identified. Selected articles were then categorized based on a taxonomy that described the required components of an effective AR-based navigation system: data, processing, overlay, view, and validation. Our findings indicated a focus on orthopedic (n=20) and maxillofacial surgeries (n=8). For preoperative input data, computed tomography (CT) (n=34), and surface rendered models (n=39) were most commonly used to represent image information. Virtual content was commonly directly superimposed with the target site (n=47); this was achieved by surface tracking of fiducials (n=30), external tracking (n=16), or manual placement (n=11). Microsoft HoloLens devices (n=24 in 2021, n=7 in 2022) were the most frequently used OST-HMDs; gestures and/or voice (n=32) served as the preferred interaction paradigm. Though promising system accuracy in the order of 2–5 mm has been demonstrated in phantom models, several human factors and technical challenges—perception, ease of use, context, interaction, and occlusion—remain to be addressed prior to widespread adoption of OST-HMD led surgical navigation.
Collapse
Affiliation(s)
- Mitchell Doughty
- Department of Medical Biophysics, University of Toronto, Toronto, ON M5S 1A1, Canada; (N.R.G.); (G.A.W.)
- Schulich Heart Program, Sunnybrook Health Sciences Centre, Toronto, ON M4N 3M5, Canada
- Correspondence:
| | - Nilesh R. Ghugre
- Department of Medical Biophysics, University of Toronto, Toronto, ON M5S 1A1, Canada; (N.R.G.); (G.A.W.)
- Schulich Heart Program, Sunnybrook Health Sciences Centre, Toronto, ON M4N 3M5, Canada
- Physical Sciences Platform, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada
| | - Graham A. Wright
- Department of Medical Biophysics, University of Toronto, Toronto, ON M5S 1A1, Canada; (N.R.G.); (G.A.W.)
- Schulich Heart Program, Sunnybrook Health Sciences Centre, Toronto, ON M4N 3M5, Canada
- Physical Sciences Platform, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada
| |
Collapse
|
26
|
Domeneghetti D, Carbone M, Cutolo F, Ferrari V. A Rendering Engine for Integral Imaging in Augmented Reality Guided Surgery . ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:2693-2696. [PMID: 36086410 DOI: 10.1109/embc48229.2022.9871806] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
In the field of image-guided surgery, Augmented Reality wearable displays are a widely studied and documented technology for their ability to provide egocentric vision together with the overlap between real and virtual content. In particular, optical see-through (OST) displays have the advantage of maintaining visual perception of the real world. However, OST displays suffer from vergeance-accomodation conflict when virtual content is superimposed on real world. Furthermore, the calibration methods required to achieve geometric consistency between real and virtual are inherently error-prone. One of the solutions, already studied, to these problems is to use of integral imaging displays. In this paper we present an easy and straightforward real-time rendering strategy implemented in modern OpenGL to show the 3D image of a virtual object on a wearable OST display deploying the integral imaging approach. Clinical Relevance- The algorithm proposed open the way towards more effective AR surgical navigation in terms of comfort of the AR experience and accuracy of the AR guidance.
Collapse
|
27
|
Beams R, Brown E, Cheng WC, Joyner JS, Kim AS, Kontson K, Amiras D, Baeuerle T, Greenleaf W, Grossmann RJ, Gupta A, Hamilton C, Hua H, Huynh TT, Leuze C, Murthi SB, Penczek J, Silva J, Spiegel B, Varshney A, Badano A. Evaluation Challenges for the Application of Extended Reality Devices in Medicine. J Digit Imaging 2022; 35:1409-1418. [PMID: 35469355 PMCID: PMC9582055 DOI: 10.1007/s10278-022-00622-x] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2021] [Revised: 02/04/2022] [Accepted: 03/08/2022] [Indexed: 12/24/2022] Open
Abstract
Augmented and virtual reality devices are being actively investigated and implemented for a wide range of medical uses. However, significant gaps in the evaluation of these medical devices and applications hinder their regulatory evaluation. Addressing these gaps is critical to demonstrating the devices' safety and effectiveness. We outline the key technical and clinical evaluation challenges discussed during the US Food and Drug Administration's public workshop, "Medical Extended Reality: Toward Best Evaluation Practices for Virtual and Augmented Reality in Medicine" and future directions for evaluation method development. Evaluation challenges were categorized into several key technical and clinical areas. Finally, we highlight current efforts in the standards communities and illustrate connections between the evaluation challenges and the intended uses of the medical extended reality (MXR) devices. Participants concluded that additional research is needed to assess the safety and effectiveness of MXR devices across the use cases.
Collapse
Affiliation(s)
- Ryan Beams
- Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, MD, USA.
| | - Ellenor Brown
- Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, MD, USA
| | - Wei-Chung Cheng
- Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, MD, USA
| | - Janell S Joyner
- Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, MD, USA
| | - Andrea S Kim
- Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, MD, USA
| | - Kimberly Kontson
- Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, MD, USA
| | - Dimitri Amiras
- Department of Imaging, Imperial College Healthcare NHS Trust, London, UK
| | | | - Walter Greenleaf
- Stanford University Virtual Human Interaction Lab, Stanford University, Stanford, CA, USA
| | | | | | | | - Hong Hua
- James C. Wyant College of Optical Sciences, University of Arizona, Tucson, AZ, USA
| | | | - Christoph Leuze
- Department of Radiology, Stanford University, Stanford, CA, USA
| | - Sarah B Murthi
- R Adams Cowley Shock Trauma Center, University of Maryland Baltimore, Baltimore, MD, USA
| | - John Penczek
- NIST, Boulder, CO, USA.,University of Colorado, Boulder, CO, USA
| | - Jennifer Silva
- SentiAR, Inc., St Louis, MT, USA.,School of Medicine, Division of Pediatric Cardiology, Washington University, St Louis, MO, USA
| | - Brennan Spiegel
- Department of Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Amitabh Varshney
- Department of Computer Science, University of Maryland, College Park, MD, USA
| | - Aldo Badano
- Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, MD, USA
| |
Collapse
|
28
|
Projected cutting guides using an augmented reality system to improve surgical margins in maxillectomies: A preclinical study. Oral Oncol 2022; 127:105775. [DOI: 10.1016/j.oraloncology.2022.105775] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2022] [Revised: 02/03/2022] [Accepted: 02/13/2022] [Indexed: 11/21/2022]
|
29
|
Birlo M, Edwards PJE, Clarkson M, Stoyanov D. Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review. Med Image Anal 2022; 77:102361. [PMID: 35168103 PMCID: PMC10466024 DOI: 10.1016/j.media.2022.102361] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Revised: 11/17/2021] [Accepted: 01/10/2022] [Indexed: 12/11/2022]
Abstract
This article presents a systematic review of optical see-through head mounted display (OST-HMD) usage in augmented reality (AR) surgery applications from 2013 to 2020. Articles were categorised by: OST-HMD device, surgical speciality, surgical application context, visualisation content, experimental design and evaluation, accuracy and human factors of human-computer interaction. 91 articles fulfilled all inclusion criteria. Some clear trends emerge. The Microsoft HoloLens increasingly dominates the field, with orthopaedic surgery being the most popular application (28.6%). By far the most common surgical context is surgical guidance (n=58) and segmented preoperative models dominate visualisation (n=40). Experiments mainly involve phantoms (n=43) or system setup (n=21), with patient case studies ranking third (n=19), reflecting the comparative infancy of the field. Experiments cover issues from registration to perception with very different accuracy results. Human factors emerge as significant to OST-HMD utility. Some factors are addressed by the systems proposed, such as attention shift away from the surgical site and mental mapping of 2D images to 3D patient anatomy. Other persistent human factors remain or are caused by OST-HMD solutions, including ease of use, comfort and spatial perception issues. The significant upward trend in published articles is clear, but such devices are not yet established in the operating room and clinical studies showing benefit are lacking. A focused effort addressing technical registration and perceptual factors in the lab coupled with design that incorporates human factors considerations to solve clear clinical problems should ensure that the significant current research efforts will succeed.
Collapse
Affiliation(s)
- Manuel Birlo
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), Charles Bell House, 43-45 Foley Street, London W1W 7TS, UK.
| | - P J Eddie Edwards
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), Charles Bell House, 43-45 Foley Street, London W1W 7TS, UK
| | - Matthew Clarkson
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), Charles Bell House, 43-45 Foley Street, London W1W 7TS, UK
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), Charles Bell House, 43-45 Foley Street, London W1W 7TS, UK
| |
Collapse
|
30
|
Marklin RW, Toll AM, Bauman EH, Simmins JJ, LaDisa JF, Cooper R. Do Head-Mounted Augmented Reality Devices Affect Muscle Activity and Eye Strain of Utility Workers Who Do Procedural Work? Studies of Operators and Manhole Workers. HUMAN FACTORS 2022; 64:305-323. [PMID: 32830567 PMCID: PMC8935469 DOI: 10.1177/0018720820943710] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/24/2020] [Accepted: 06/29/2020] [Indexed: 06/11/2023]
Abstract
OBJECTIVE The objective was to determine the effect of two head-mounted display (HMD) augmented reality (AR) devices on muscle activity and eye strain of electric utility workers. The AR devices were the Microsoft HoloLens and RealWear HMT-1. BACKGROUND The HoloLens is an optical see-through device. The HMT-1 has a small display that is mounted to the side of one eye of the user. METHOD Twelve power plant operators and 13 manhole workers conducted their normal procedural tasks on-site in three conditions: HoloLens, HMT-1, and "No AR" (regular method). Duration of test trials ranged up to 30 s for operators and up to 10 min for manhole workers. Mean and peak values of surface electromyographic (sEMG) signals from eight neck muscles were measured. A small eye camera measured blink rate of the right eye. RESULTS In general, there were no differences in sEMG activity between the AR and "No AR" conditions for both groups of workers. For the manhole workers, the HoloLens blink rate was 8 to 11 blinks per min lower than the HMT-1 in two tasks and 6.5 fewer than "No AR" in one task. Subjective assessment of the two AR devices did not vary in general. CONCLUSION The decrease in blink rate with the HoloLens may expose utility manhole workers to risk of eye strain or dry-eye syndrome. APPLICATION HMD AR devices should be tested thoroughly with respect to risk of eye strain before deployment by manhole workers for long-duration procedural work.
Collapse
Affiliation(s)
| | | | - Eric H. Bauman
- EPRI (Electric Power Research Institute), Palo Alto, California, USA
| | | | | | | |
Collapse
|
31
|
Ferrari V, Cattari N, Fontana U, Cutolo F. Parallax Free Registration for Augmented Reality Optical See-Through Displays in the Peripersonal Space. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2022; 28:1608-1618. [PMID: 32881688 DOI: 10.1109/tvcg.2020.3021534] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Egocentric augmented reality (AR) interfaces are quickly becoming a key asset for assisting high precision activities in the peripersonal space in several application fields. In these applications, accurate and robust registration of computer-generated information to the real scene is hard to achieve with traditional Optical See-Through (OST) displays given that it relies on the accurate calibration of the combined eye-display projection model. The calibration is required to efficiently estimate the projection parameters of the pinhole model that encapsulate the optical features of the display and whose values vary according to the position of the user's eye. In this article, we describe an approach that prevents any parallax-related AR misregistration at a pre-defined working distance in OST displays with infinity focus; our strategy relies on the use of a magnifier placed in front of the OST display, and features a proper parameterization of the virtual rendering camera achieved through a dedicated calibration procedure that accounts for the contribution of the magnifier. We model the registration error due to the viewpoint parallax outside the ideal working distance. Finally, we validate our strategy on a OST display, and we show that sub-millimetric registration accuracy can be achieved for working distances of ±100 mm around the focal length of the magnifier.
Collapse
|
32
|
Lu L, Wang H, Liu P, Liu R, Zhang J, Xie Y, Liu S, Huo T, Xie M, Wu X, Ye Z. Applications of Mixed Reality Technology in Orthopedics Surgery: A Pilot Study. Front Bioeng Biotechnol 2022; 10:740507. [PMID: 35273954 PMCID: PMC8902164 DOI: 10.3389/fbioe.2022.740507] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2021] [Accepted: 01/21/2022] [Indexed: 12/28/2022] Open
Abstract
Objective: The aim of this study is to explore the potential of mixed reality (MR) technology in the visualization of orthopedic surgery. Methods: The visualization system with MR technology is widely used in orthopedic surgery. The system is composed of a 3D imaging workstation, a cloud platform, and an MR space station. An intelligent segmentation algorithm is adopted on the 3D imaging workstation to create a 3D anatomical model with zooming and rotation effects. This model is then exploited for efficient 3D reconstruction of data for computerized tomography (CT) and magnetic resonance imaging (MRI). Additionally, the model can be uploaded to the cloud platform for physical parameter tuning, model positioning, rendering and high-dimensional display. Using Microsoft’s HoloLens glasses in combination with the MR system, we project and view 3D holograms in real time under different clinical scenarios. After each procedure, nine surgeons completed a Likert-scale questionnaire on communication and understanding, spatial awareness and effectiveness of MR technology use. In addition to that, the National Aeronautics and Space Administration Task Load Index (NASA-TLX) is also used to evaluate the workload of MR hologram support. Results: 1) MR holograms can clearly show the 3D structures of bone fractures, which improves the understanding of different fracture types and the design of treatment plans; 2) Holograms with three-dimensional lifelike dynamic features provide an intuitive communication tool among doctors and also between doctors and patients; 3) During surgeries, a full lesion hologram can be obtained and blended in real time with a patient’s virtual 3D digital model in order to give surgeons superior visual guidance through novel high-dimensional “perspectives” of the surgical area; 4) Hologram-based magnetic navigation improves the accuracy and safety of the screw placement in orthopaedics surgeries; 5) The combination of mixed reality cloud platform and telemedicine system based on 5G provides a new technology platform for telesurgery collaboration. Results of qualitative study encourage the usage of MR technology for orthopaedics surgery. Analysis of the Likert-scale questionnaire shows that MR adds significant value to understanding and communication, spatial awareness, learning and effectiveness. Based on the NASA TLX-scale questionnaire results, mixed reality scored significantly lower under the “mental,” “temporal,” “performance,” and “frustration” categories compared to usual 2D. Conclusion: The integration of MR technology in orthopaedic surgery reduces the dependence on surgeons’ experience and provides personalized 3D visualization models for accurate diagnosis and treatment of orthopaedic abnormalities. This integration is clearly one of the prominent future development directions in medical surgery.
Collapse
Affiliation(s)
- Lin Lu
- Department of Orthopaedics Surgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Intelligent Medical Laboratory, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Honglin Wang
- Department of Orthopaedics Surgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Intelligent Medical Laboratory, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Pengran Liu
- Department of Orthopaedics Surgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Intelligent Medical Laboratory, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Rong Liu
- Department of Orthopaedic Surgery, Puren Hospital of Wuhan, Wuhan University of Science and Technology, Wuhan, China
| | - Jiayao Zhang
- Department of Orthopaedics Surgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Intelligent Medical Laboratory, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Yi Xie
- Department of Orthopaedics Surgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Intelligent Medical Laboratory, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Songxiang Liu
- Department of Orthopaedics Surgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Intelligent Medical Laboratory, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Tongtong Huo
- Intelligent Medical Laboratory, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Mao Xie
- Department of Orthopaedics Surgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Xinghuo Wu
- Department of Orthopaedics Surgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Intelligent Medical Laboratory, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- *Correspondence: Xinghuo Wu, ; Zhewei Ye,
| | - Zhewei Ye
- Department of Orthopaedics Surgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- Intelligent Medical Laboratory, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
- *Correspondence: Xinghuo Wu, ; Zhewei Ye,
| |
Collapse
|
33
|
Tanwani J, Alam F, Matava C, Choi S, McHardy P, Singer O, Cheong G, Wiegelmann J. Development of a Head-Mounted Holographic Needle Guidance System for Enhanced Ultrasound-Guided Neuraxial Anesthesia: System Development and Observational Evaluation (Preprint). JMIR Form Res 2022; 6:e36931. [PMID: 35737430 PMCID: PMC9264121 DOI: 10.2196/36931] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2022] [Revised: 05/12/2022] [Accepted: 05/15/2022] [Indexed: 11/13/2022] Open
Abstract
Background Objective Methods Results Conclusions Trial Registration
Collapse
Affiliation(s)
- Jaya Tanwani
- Department of Anesthesiology and Pain Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Fahad Alam
- Department of Anesthesiology and Pain Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Clyde Matava
- Department of Anesthesia, The Hospital for Sick Children, University of Toronto, Toronto, ON, Canada
| | - Stephen Choi
- Department of Anesthesiology and Pain Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Paul McHardy
- Department of Anesthesiology and Pain Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Oskar Singer
- Department of Anesthesiology and Pain Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Geraldine Cheong
- Department of Anesthesia, Khoo Teck Puat Hospital, Singapore, Singapore
| | - Julian Wiegelmann
- Department of Anesthesiology and Pain Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
34
|
Architecture of a Hybrid Video/Optical See-through Head-Mounted Display-Based Augmented Reality Surgical Navigation Platform. INFORMATION 2022. [DOI: 10.3390/info13020081] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022] Open
Abstract
In the context of image-guided surgery, augmented reality (AR) represents a ground-breaking enticing improvement, mostly when paired with wearability in the case of open surgery. Commercially available AR head-mounted displays (HMDs), designed for general purposes, are increasingly used outside their indications to develop surgical guidance applications with the ambition to demonstrate the potential of AR in surgery. The applications proposed in the literature underline the hunger for AR-guidance in the surgical room together with the limitations that hinder commercial HMDs from being the answer to such a need. The medical domain demands specifically developed devices that address, together with ergonomics, the achievement of surgical accuracy objectives and compliance with medical device regulations. In the framework of an EU Horizon2020 project, a hybrid video and optical see-through augmented reality headset paired with a software architecture, both specifically designed to be seamlessly integrated into the surgical workflow, has been developed. In this paper, the overall architecture of the system is described. The developed AR HMD surgical navigation platform was positively tested on seven patients to aid the surgeon while performing Le Fort 1 osteotomy in cranio-maxillofacial surgery, demonstrating the value of the hybrid approach and the safety and usability of the navigation platform.
Collapse
|
35
|
Bussink T, Maal T, Meulstee J, Xi T. Augmented reality guided condylectomy. Br J Oral Maxillofac Surg 2022; 60:991-993. [DOI: 10.1016/j.bjoms.2022.01.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2022] [Revised: 01/21/2022] [Accepted: 01/26/2022] [Indexed: 10/19/2022]
|
36
|
Doughty M, Ghugre NR. Head-Mounted Display-Based Augmented Reality for Image-Guided Media Delivery to the Heart: A Preliminary Investigation of Perceptual Accuracy. J Imaging 2022; 8:jimaging8020033. [PMID: 35200735 PMCID: PMC8878166 DOI: 10.3390/jimaging8020033] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2021] [Revised: 01/25/2022] [Accepted: 01/28/2022] [Indexed: 01/14/2023] Open
Abstract
By aligning virtual augmentations with real objects, optical see-through head-mounted display (OST-HMD)-based augmented reality (AR) can enhance user-task performance. Our goal was to compare the perceptual accuracy of several visualization paradigms involving an adjacent monitor, or the Microsoft HoloLens 2 OST-HMD, in a targeted task, as well as to assess the feasibility of displaying imaging-derived virtual models aligned with the injured porcine heart. With 10 participants, we performed a user study to quantify and compare the accuracy, speed, and subjective workload of each paradigm in the completion of a point-and-trace task that simulated surgical targeting. To demonstrate the clinical potential of our system, we assessed its use for the visualization of magnetic resonance imaging (MRI)-based anatomical models, aligned with the surgically exposed heart in a motion-arrested open-chest porcine model. Using the HoloLens 2 with alignment of the ground truth target and our display calibration method, users were able to achieve submillimeter accuracy (0.98 mm) and required 1.42 min for calibration in the point-and-trace task. In the porcine study, we observed good spatial agreement between the MRI-models and target surgical site. The use of an OST-HMD led to improved perceptual accuracy and task-completion times in a simulated targeting task.
Collapse
Affiliation(s)
- Mitchell Doughty
- Department of Medical Biophysics, University of Toronto, Toronto, ON M5S 1A1, Canada;
- Schulich Heart Program, Sunnybrook Health Sciences Centre, Toronto, ON M4N 3M5, Canada
- Correspondence:
| | - Nilesh R. Ghugre
- Department of Medical Biophysics, University of Toronto, Toronto, ON M5S 1A1, Canada;
- Schulich Heart Program, Sunnybrook Health Sciences Centre, Toronto, ON M4N 3M5, Canada
- Physical Sciences Platform, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada
| |
Collapse
|
37
|
Cercenelli L, Babini F, Badiali G, Battaglia S, Tarsitano A, Marchetti C, Marcelli E. Augmented Reality to Assist Skin Paddle Harvesting in Osteomyocutaneous Fibular Flap Reconstructive Surgery: A Pilot Evaluation on a 3D-Printed Leg Phantom. Front Oncol 2022; 11:804748. [PMID: 35071009 PMCID: PMC8770836 DOI: 10.3389/fonc.2021.804748] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2021] [Accepted: 12/10/2021] [Indexed: 11/13/2022] Open
Abstract
Background Augmented Reality (AR) represents an evolution of navigation-assisted surgery, providing surgeons with a virtual aid contextually merged with the real surgical field. We recently reported a case series of AR-assisted fibular flap harvesting for mandibular reconstruction. However, the registration accuracy between the real and the virtual content needs to be systematically evaluated before widely promoting this tool in clinical practice. In this paper, after description of the AR based protocol implemented for both tablet and HoloLens 2 smart glasses, we evaluated in a first test session the achievable registration accuracy with the two display solutions, and in a second test session the success rate in executing the AR-guided skin paddle incision task on a 3D printed leg phantom. Methods From a real computed tomography dataset, 3D virtual models of a human leg, including fibula, arteries and skin with planned paddle profile for harvesting, were obtained. All virtual models were imported into Unity software to develop a marker-less AR application suitable to be used both via tablet and via HoloLens 2 headset. The registration accuracy for both solutions was verified on a 3D printed leg phantom obtained from the virtual models, by repeatedly applying the tracking function and computing pose deviations between the AR-projected virtual skin paddle profile and the real one transferred to the phantom via a CAD/CAM cutting guide. The success rate in completing the AR-guided task of skin paddle harvesting was evaluated using CAD/CAM templates positioned on the phantom model surface. Results On average, the marker-less AR protocol showed comparable registration errors (ranging within 1-5 mm) for tablet-based and HoloLens-based solution. Registration accuracy seems to be quite sensitive to ambient light conditions. We found a good success rate in completing the AR-guided task within an error margin of 4 mm (97% and 100% for tablet and HoloLens, respectively). All subjects reported greater usability and ergonomics for HoloLens 2 solution. Conclusions Results revealed that the proposed marker-less AR based protocol may guarantee a registration error within 1-5 mm for assisting skin paddle harvesting in the clinical setting. Optimal lightening conditions and further improvement of marker-less tracking technologies have the potential to increase the efficiency and precision of this AR-assisted reconstructive surgery.
Collapse
Affiliation(s)
- Laura Cercenelli
- eDIMES Lab - Laboratory of Bioengineering, Department of Experimental, Diagnostic and Specialty Medicine, University of Bologna, Bologna, Italy
| | - Federico Babini
- eDIMES Lab - Laboratory of Bioengineering, Department of Experimental, Diagnostic and Specialty Medicine, University of Bologna, Bologna, Italy
| | - Giovanni Badiali
- Maxillofacial Surgery Unit, Head and Neck Department, IRCCS Azienda Ospedaliera Universitaria di Bologna, Department of Biomedical and Neuromotor Sciences, Alma Mater Studiorum University of Bologna, Bologna, Italy
| | - Salvatore Battaglia
- Maxillofacial Surgery Unit, Policlinico San Marco University Hospital, University of Catania, Catania, Italy
| | - Achille Tarsitano
- Maxillofacial Surgery Unit, Head and Neck Department, IRCCS Azienda Ospedaliera Universitaria di Bologna, Department of Biomedical and Neuromotor Sciences, Alma Mater Studiorum University of Bologna, Bologna, Italy
| | - Claudio Marchetti
- Maxillofacial Surgery Unit, Head and Neck Department, IRCCS Azienda Ospedaliera Universitaria di Bologna, Department of Biomedical and Neuromotor Sciences, Alma Mater Studiorum University of Bologna, Bologna, Italy
| | - Emanuela Marcelli
- eDIMES Lab - Laboratory of Bioengineering, Department of Experimental, Diagnostic and Specialty Medicine, University of Bologna, Bologna, Italy
| |
Collapse
|
38
|
Rahimov C, Aliyev D, Rahimov N, Farzaliyev I. Mixed reality in the reconstruction of orbital floor: An experimental and clinical evaluative study. Ann Maxillofac Surg 2022; 12:46-53. [PMID: 36199454 PMCID: PMC9527844 DOI: 10.4103/ams.ams_141_21] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2021] [Revised: 06/22/2022] [Accepted: 07/21/2022] [Indexed: 11/04/2022] Open
Abstract
Introduction: Materials and Methods: Results: Discussion:
Collapse
|
39
|
In Situ Visualization for 3D Ultrasound-Guided Interventions with Augmented Reality Headset. Bioengineering (Basel) 2021; 8:bioengineering8100131. [PMID: 34677204 PMCID: PMC8533537 DOI: 10.3390/bioengineering8100131] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2021] [Revised: 09/16/2021] [Accepted: 09/21/2021] [Indexed: 12/03/2022] Open
Abstract
Augmented Reality (AR) headsets have become the most ergonomic and efficient visualization devices to support complex manual tasks performed under direct vision. Their ability to provide hands-free interaction with the augmented scene makes them perfect for manual procedures such as surgery. This study demonstrates the reliability of an AR head-mounted display (HMD), conceived for surgical guidance, in navigating in-depth high-precision manual tasks guided by a 3D ultrasound imaging system. The integration between the AR visualization system and the ultrasound imaging system provides the surgeon with real-time intra-operative information on unexposed soft tissues that are spatially registered with the surrounding anatomic structures. The efficacy of the AR guiding system was quantitatively assessed with an in vitro study simulating a biopsy intervention aimed at determining the level of accuracy achievable. In the experiments, 10 subjects were asked to perform the biopsy on four spherical lesions of decreasing sizes (10, 7, 5, and 3 mm). The experimental results showed that 80% of the subjects were able to successfully perform the biopsy on the 5 mm lesion, with a 2.5 mm system accuracy. The results confirmed that the proposed integrated system can be used for navigation during in-depth high-precision manual tasks.
Collapse
|
40
|
Intraoperative Use of Mixed Reality Technology in Median Neck and Branchial Cyst Excision. FUTURE INTERNET 2021. [DOI: 10.3390/fi13080214] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
The paper considers the possibilities, prospects, and drawbacks of the mixed reality (MR) technology application using mixed reality smartglasses Microsoft HoloLens 2. The main challenge was to find and develop an approach that would allow surgeons to conduct operations using mixed reality on a large scale, reducing the preparation time required for the procedure and without having to create custom solutions for each patient. Research was conducted in three clinical cases: two median neck and one branchial cyst excisions. In each case, we applied a unique approach of hologram positioning in space based on mixed reality markers. As a result, we listed a series of positive and negative aspects related to MR surgery, along with proposed solutions for using MR in surgery on a daily basis.
Collapse
|
41
|
Can Liquid Lenses Increase Depth of Field in Head Mounted Video See-Through Devices? J Imaging 2021; 7:jimaging7080138. [PMID: 34460773 PMCID: PMC8404927 DOI: 10.3390/jimaging7080138] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2021] [Revised: 08/02/2021] [Accepted: 08/03/2021] [Indexed: 02/05/2023] Open
Abstract
Wearable Video See-Through (VST) devices for Augmented Reality (AR) and for obtaining a Magnified View are taking hold in the medical and surgical fields. However, these devices are not yet usable in daily clinical practice, due to focusing problems and a limited depth of field. This study investigates the use of liquid-lens optics to create an autofocus system for wearable VST visors. The autofocus system is based on a Time of Flight (TOF) distance sensor and an active autofocus control system. The integrated autofocus system in the wearable VST viewers showed good potential in terms of providing rapid focus at various distances and a magnified view.
Collapse
|
42
|
Pladere T, Luguzis A, Zabels R, Smukulis R, Barkovska V, Krauze L, Konosonoka V, Svede A, Krumina G. When virtual and real worlds coexist: Visualization and visual system affect spatial performance in augmented reality. J Vis 2021; 21:17. [PMID: 34388233 PMCID: PMC8363769 DOI: 10.1167/jov.21.8.17] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2021] [Accepted: 07/16/2021] [Indexed: 11/24/2022] Open
Abstract
New visualization approaches are being actively developed aiming to mitigate the effect of vergence-accommodation conflict in stereoscopic augmented reality; however, high interindividual variability in spatial performance makes it difficult to predict user gain. To address this issue, we investigated the effects of consistent and inconsistent binocular and focus cues on perceptual matching in the stereoscopic environment of augmented reality using a head-mounted display that was driven in multifocal and single focal plane modes. Participants matched the distance of a real object with images projected at three viewing distances, concordant with the display focal planes when driven in the multifocal mode. As a result, consistency of depth cues facilitated faster perceptual judgments on spatial relations. Moreover, the individuals with mild binocular and accommodative disorders benefited from the visualization of information on the focal planes corresponding to image planes more than individuals with normal vision, which was reflected in performance accuracy. Because symptoms and complaints may be absent when the functionality of the sensorimotor system is reduced, the results indicate the need for a detailed assessment of visual functions in research on spatial performance. This study highlights that the development of a visualization system that reduces visual stress and improves user performance should be a priority for the successful implementation of augmented reality displays.
Collapse
Affiliation(s)
- Tatjana Pladere
- Department of Optometry and Vision Science, Faculty of Physics, Mathematics and Optometry, University of Latvia, Riga, Latvia
| | - Artis Luguzis
- Department of Optometry and Vision Science, Faculty of Physics, Mathematics and Optometry, University of Latvia, Riga, Latvia
- Laboratory of Statistical Research and Data Analysis, Faculty of Physics, Mathematics and Optometry, University of Latvia, Riga, Latvia
| | | | | | - Viktorija Barkovska
- Department of Optometry and Vision Science, Faculty of Physics, Mathematics and Optometry, University of Latvia, Riga, Latvia
| | - Linda Krauze
- Department of Optometry and Vision Science, Faculty of Physics, Mathematics and Optometry, University of Latvia, Riga, Latvia
| | - Vita Konosonoka
- Department of Optometry and Vision Science, Faculty of Physics, Mathematics and Optometry, University of Latvia, Riga, Latvia
| | - Aiga Svede
- Department of Optometry and Vision Science, Faculty of Physics, Mathematics and Optometry, University of Latvia, Riga, Latvia
| | - Gunta Krumina
- Department of Optometry and Vision Science, Faculty of Physics, Mathematics and Optometry, University of Latvia, Riga, Latvia
| |
Collapse
|
43
|
Winnand P, Ayoub N, Redick T, Gesenhues J, Heitzer M, Peters F, Raith S, Abel D, Hölzle F, Modabber A. Navigation of iliac crest graft harvest using markerless augmented reality and cutting guide technology: A pilot study. Int J Med Robot 2021; 18:e2318. [PMID: 34328700 DOI: 10.1002/rcs.2318] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Revised: 07/27/2021] [Accepted: 03/04/2021] [Indexed: 11/08/2022]
Abstract
BACKGROUND Defects of the facial skeleton often require complex reconstruction with vascularized grafts. This trial elucidated the usability, visual perception and accuracy of a markerless augmented reality (AR)-guided navigation for harvesting iliac crest transplants. METHODS Random CT scans were used to virtually plan two common transplant configurations on 10 iliac crest models, each printed four times. The transplants were harvested using projected AR and cutting guides. The duration and accuracies of the angulation, distance and volume between the planned and executed osteotomies were measured. RESULTS AR was characterized by the efficient use of time and accurate rendition of preoperatively planned geometries. However, vertical osteotomies and complex anatomical settings displayed significant inferiority of AR guidance compared to cutting guides. CONCLUSIONS This study demonstrated the usability of a markerless AR setup for harvesting iliac crest transplants. The visual perception and accuracy of the AR-guided osteotomies constituted remaining weaknesses against cutting guide technology.
Collapse
Affiliation(s)
- Philipp Winnand
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
| | - Nassim Ayoub
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
| | - Tim Redick
- Institute of Automatic Control, RWTH Aachen University, Aachen, Germany
| | - Jonas Gesenhues
- Institute of Automatic Control, RWTH Aachen University, Aachen, Germany
| | - Marius Heitzer
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
| | - Florian Peters
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
| | - Stefan Raith
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
| | - Dirk Abel
- Institute of Automatic Control, RWTH Aachen University, Aachen, Germany
| | - Frank Hölzle
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
| | - Ali Modabber
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
| |
Collapse
|
44
|
Evaluation of a Wearable AR Platform for Guiding Complex Craniotomies in Neurosurgery. Ann Biomed Eng 2021; 49:2590-2605. [PMID: 34297263 DOI: 10.1007/s10439-021-02834-8] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2021] [Accepted: 07/12/2021] [Indexed: 10/20/2022]
Abstract
Today, neuronavigation is widely used in daily clinical routine to perform safe and efficient surgery. Augmented reality (AR) interfaces can provide anatomical models and preoperative planning contextually blended with the real surgical scenario, overcoming the limitations of traditional neuronavigators. This study aims to demonstrate the reliability of a new-concept AR headset in navigating complex craniotomies. Moreover, we aim to prove the efficacy of a patient-specific template-based methodology for fast, non-invasive, and fully automatic planning-to-patient registration. The AR platform navigation performance was assessed with an in-vitro study whose goal was twofold: to measure the real-to-virtual 3D target visualization error (TVE), and assess the navigation accuracy through a user study involving 10 subjects in tracing a complex craniotomy. The feasibility of the template-based registration was preliminarily tested on a volunteer. The TVE mean and standard deviation were 1.3 and 0.6 mm. The results of the user study, over 30 traced craniotomies, showed that 97% of the trajectory length was traced within an error margin of 1.5 mm, and 92% within a margin of 1 mm. The in-vivo test confirmed the feasibility and reliability of the patient-specific template for registration. The proposed AR headset allows ergonomic and intuitive fruition of preoperative planning, and it can represent a valid option to support neurosurgical tasks.
Collapse
|
45
|
Tarutani K, Takaki H, Igeta M, Fujiwara M, Okamura A, Horio F, Toudou Y, Nakajima S, Kagawa K, Tanooka M, Yamakado K. Development and Accuracy Evaluation of Augmented Reality-based Patient Positioning System in Radiotherapy: A Phantom Study. In Vivo 2021; 35:2081-2087. [PMID: 34182483 DOI: 10.21873/invivo.12477] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2021] [Revised: 05/27/2021] [Accepted: 05/28/2021] [Indexed: 01/04/2023]
Abstract
BACKGROUND/AIM To develop and evaluate the accuracy of augmented reality (AR)-based patient positioning systems in radiotherapy. MATERIALS AND METHODS AR head-mounted displays (AR-HMDs), which virtually superimpose a three-dimensional (3D) image generated by the digital imaging and communications in medicine (DICOM) data, have been developed. The AR-based positioning feasibility was evaluated. Then, the setup errors of three translational axes directions and rotation angles between the AR and the conventional laser-based positioning were compared. RESULTS The AR-based pelvic phantom positioning was feasible. The setup errors of AR-based positioning were comparable to laser-based positioning in all translational axis directions and rotation angles. The time necessary for AR-based positioning was significantly longer than that for laser-based positioning (171.0 s vs. 47.5 s, p<0.001). CONCLUSION AR-based positioning for radiotherapy was feasible, and showed comparable positioning errors to those of conventional line-based positioning; however, a markedly longer setup time was necessary.
Collapse
Affiliation(s)
- Kazuo Tarutani
- Department of Radiology, Hyogo College of Medicine, Hyogo, Japan.,Japan Organization of Occupational Health and Safety Kansai Rousai Hospital, Hyogo, Japan
| | - Haruyuki Takaki
- Department of Radiology, Hyogo College of Medicine, Hyogo, Japan;
| | - Masataka Igeta
- Department of Biostatistics, Hyogo College of Medicine, Hyogo, Japan
| | | | - Ayako Okamura
- Japan Organization of Occupational Health and Safety Kansai Rousai Hospital, Hyogo, Japan
| | - Futo Horio
- Kobe Digital Labo Incorporated, Hyogo, Japan
| | - Yuki Toudou
- Japan Organization of Occupational Health and Safety Kansai Rousai Hospital, Hyogo, Japan
| | - Satoshi Nakajima
- Japan Organization of Occupational Health and Safety Kansai Rousai Hospital, Hyogo, Japan
| | - Kazufumi Kagawa
- Japan Organization of Occupational Health and Safety Kansai Rousai Hospital, Hyogo, Japan
| | - Masao Tanooka
- Department of Radiotherapy, Takarazuka City Hospital, Hyogo, Japan
| | | |
Collapse
|
46
|
Augmented reality in the operating room: a clinical feasibility study. BMC Musculoskelet Disord 2021; 22:451. [PMID: 34006234 PMCID: PMC8132365 DOI: 10.1186/s12891-021-04339-w] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/05/2020] [Accepted: 05/06/2021] [Indexed: 11/20/2022] Open
Abstract
Background Augmented Reality (AR) is a rapidly emerging technology finding growing acceptance and application in different fields of surgery. Various studies have been performed evaluating the precision and accuracy of AR guided navigation. This study investigates the feasibility of a commercially available AR head mounted device during orthopedic surgery. Methods Thirteen orthopedic surgeons from a Swiss university clinic performed 25 orthopedic surgical procedures wearing a holographic AR headset (HoloLens, Microsoft, Redmond, WA, USA) providing complementary three-dimensional, patient specific anatomic information. The surgeon’s experience of using the device during surgery was recorded using a standardized 58-item questionnaire grading different aspects on a 100-point scale with anchor statements. Results Surgeons were generally satisfied with image quality (85 ± 17 points) and accuracy of the virtual objects (84 ± 19 point). Wearing the AR device was rated as fairly comfortable (79 ± 13 points). Functionality of voice commands (68 ± 20 points) and gestures (66 ± 20 points) provided less favorable results. The greatest potential in the use of the AR device was found for surgical correction of deformities (87 ± 15 points). Overall, surgeons were satisfied with the application of this novel technology (78 ± 20 points) and future access to it was demanded (75 ± 22 points). Conclusion AR is a rapidly evolving technology with large potential in different surgical settings, offering the opportunity to provide a compact, low cost alternative requiring a minimum of infrastructure compared to conventional navigation systems. While surgeons where generally satisfied with image quality of the here tested head mounted AR device, some technical and ergonomic shortcomings were pointed out. This study serves as a proof of concept for the use of an AR head mounted device in a real-world sterile setting in orthopedic surgery. Supplementary Information The online version contains supplementary material available at 10.1186/s12891-021-04339-w.
Collapse
|
47
|
Penczek J, Boynton PA, Beams R, Sriram RD. Measurement Challenges for Medical Image Display Devices. J Digit Imaging 2021; 34:458-472. [PMID: 33846889 DOI: 10.1007/s10278-021-00438-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2020] [Revised: 01/27/2021] [Accepted: 02/24/2021] [Indexed: 12/25/2022] Open
Abstract
Visual information is a critical component in the evaluation and communication of patient medical information. As display technologies have evolved, the medical community has sought to take advantage of advances in wider color gamuts, greater display portability, and more immersive imagery. These image quality enhancements have shown improvements in the quality of healthcare through greater efficiency, higher diagnostic accuracy, added functionality, enhanced training, and better health records. However, the display technology advances typically introduce greater complexity in the image workflow and display evaluation. This paper highlights some of the optical measurement challenges created by these new display technologies and offers possible pathways to address them.
Collapse
Affiliation(s)
- J Penczek
- National Institute of Standards and Technology, Boulder and University of Colorado, CO, 80305, Boulder, USA.
| | - P A Boynton
- National Institute of Standards and Technology, Gaithersburg, MD, 20899, USA
| | - R Beams
- Food and Drug Administration, Silver Springs, MD, 20993, USA
| | - R D Sriram
- National Institute of Standards and Technology, Gaithersburg, MD, 20899, USA
| |
Collapse
|
48
|
Lareyre F, Chaudhuri A, Adam C, Carrier M, Mialhe C, Raffort J. Applications of Head-Mounted Displays and Smart Glasses in Vascular Surgery. Ann Vasc Surg 2021; 75:497-512. [PMID: 33823254 DOI: 10.1016/j.avsg.2021.02.033] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2021] [Revised: 02/22/2021] [Accepted: 02/25/2021] [Indexed: 12/11/2022]
Abstract
OBJECTIVES Advances in virtual, augmented and mixed reality have led to the development of wearable technologies including head mounted displays (HMD) and smart glasses. While there is a growing interest on their potential applications in health, only a few studies have addressed so far their use in vascular surgery. The aim of this review was to summarize the fundamental notions associated with these technologies and to discuss potential applications and current limits for their use in vascular surgery. METHODS A comprehensive literature review was performed to introduce the fundamental concepts and provide an overview of applications of HMD and smart glasses in surgery. RESULTS HMD and smart glasses demonstrated a potential interest for the education of surgeons including anatomical teaching, surgical training, teaching and telementoring. Applications for pre-surgical planning have been developed in general and cardiac surgery and could be transposed for a use in vascular surgery. The use of wearable technologies in the operating room has also been investigated in both general and cardiovascular surgery and demonstrated its potential interest for image-guided surgery and data collection. CONCLUSION Studies performed so far represent a proof of concept of the interest of HMD and smart glasses in vascular surgery for education of surgeons and for surgical practice. Although these technologies exhibited encouraging results for applications in vascular surgery, technical improvements and further clinical research in large series are required before hoping using them in daily clinical practice.
Collapse
Affiliation(s)
- Fabien Lareyre
- Department of Vascular Surgery, Hospital of Antibes-Juan-les-Pins, France; Université Côte d'Azur, CHU, Inserm U1065, C3M, Nice, France.
| | - Arindam Chaudhuri
- Bedfordshire-Milton Keynes Vascular Centre, Bedfordshire Hospitals NHS Foundation Trust, Bedford, UK
| | - Cédric Adam
- Laboratory of Applied Mathematics and Computer Science (MICS), CentraleSupélec, Université Paris-Saclay, France
| | - Marion Carrier
- Laboratory of Applied Mathematics and Computer Science (MICS), CentraleSupélec, Université Paris-Saclay, France
| | - Claude Mialhe
- Cardiovascular Surgery Unit, Cardio Thoracic Centre of Monaco, Monaco
| | - Juliette Raffort
- Université Côte d'Azur, CHU, Inserm U1065, C3M, Nice, France; Clinical Chemistry Laboratory, University Hospital of Nice, France
| |
Collapse
|
49
|
Teatini A, Kumar RP, Elle OJ, Wiig O. Mixed reality as a novel tool for diagnostic and surgical navigation in orthopaedics. Int J Comput Assist Radiol Surg 2021; 16:407-414. [PMID: 33555563 PMCID: PMC7946663 DOI: 10.1007/s11548-020-02302-z] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2020] [Accepted: 12/14/2020] [Indexed: 12/15/2022]
Abstract
Purpose This study presents a novel surgical navigation tool developed in mixed reality environment for orthopaedic surgery. Joint and skeletal deformities affect all age groups and greatly reduce the range of motion of the joints. These deformities are notoriously difficult to diagnose and to correct through surgery. Method We have developed a surgical tool which integrates surgical instrument tracking and augmented reality through a head mounted display. This allows the surgeon to visualise bones with the illusion of possessing “X-ray” vision. The studies presented below aim to assess the accuracy of the surgical navigation tool in tracking a location at the tip of the surgical instrument in holographic space. Results Results show that the average accuracy provided by the navigation tool is around 8 mm, and qualitative assessment by the orthopaedic surgeons provided positive feedback in terms of the capabilities for diagnostic use. Conclusions More improvements are necessary for the navigation tool to be accurate enough for surgical applications, however, this new tool has the potential to improve diagnostic accuracy and allow for safer and more precise surgeries, as well as provide for better learning conditions for orthopaedic surgeons in training.
Collapse
Affiliation(s)
- Andrea Teatini
- The Intervention Centre, Oslo University Hospital, Oslo, Norway.
- Department of Informatics, University of Oslo, Oslo, Norway.
| | - Rahul P Kumar
- The Intervention Centre, Oslo University Hospital, Oslo, Norway
| | - Ole Jakob Elle
- The Intervention Centre, Oslo University Hospital, Oslo, Norway
- Department of Informatics, University of Oslo, Oslo, Norway
| | - Ola Wiig
- Department of Orthopaedic Surgery, Oslo University Hospital, Oslo, Norway
| |
Collapse
|
50
|
Hybrid Spine Simulator Prototype for X-ray Free Pedicle Screws Fixation Training. APPLIED SCIENCES-BASEL 2021. [DOI: 10.3390/app11031038] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Simulation for surgical training is increasingly being considered a valuable addition to traditional teaching methods. 3D-printed physical simulators can be used for preoperative planning and rehearsal in spine surgery to improve surgical workflows and postoperative patient outcomes. This paper proposes an innovative strategy to build a hybrid simulation platform for training of pedicle screws fixation: the proposed method combines 3D-printed patient-specific spine models with augmented reality functionalities and virtual X-ray visualization, thus avoiding any exposure to harmful radiation during the simulation. Software functionalities are implemented by using a low-cost tracking strategy based on fiducial marker detection. Quantitative tests demonstrate the accuracy of the method to track the vertebral model and surgical tools, and to coherently visualize them in either the augmented reality or virtual fluoroscopic modalities. The obtained results encourage further research and clinical validation towards the use of the simulator as an effective tool for training in pedicle screws insertion in lumbar vertebrae.
Collapse
|