1
|
Li P, Xu B, Zhang X, Fang D, Zhang J. Design and development of a personalized virtual reality-based training system for vascular intervention surgery. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 249:108142. [PMID: 38547688 DOI: 10.1016/j.cmpb.2024.108142] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Revised: 02/17/2024] [Accepted: 03/20/2024] [Indexed: 04/21/2024]
Abstract
BACKGROUND AND OBJECTIVES Virtual training has emerged as an exceptionally effective approach for training healthcare practitioners in the field of vascular intervention surgery. By providing a simulated environment and blood vessel model that enables repeated practice, virtual training facilitates the acquisition of surgical skills in a safe and efficient manner for trainees. However, the current state of research in this area is characterized by limitations in the fidelity of blood vessel and guidewire models, which restricts the effectiveness of training. Additionally, existing approaches lack the necessary real-time responsiveness and precision, while the blood vessel models suffer from incompleteness and a lack of scientific rigor. METHODS To address these challenges, this paper integrates position-based dynamics (PBD) and its extensions, shape matching, and Cosserat elastic rods. By combining these approaches within a unified particle framework, accurate and realistic deformation simulation of personalized blood vessel and guidewire models is achieved, thereby enhancing the training experience. Furthermore, a multi-level progressive continuous collision detection method, leveraging spatial hashing, is proposed to improve the accuracy and efficiency of collision detection. RESULTS Our proposed blood vessel model demonstrated acceptable performance with the reduced deformation simulation response times of 7 ms, improving the real-time capability at least of 43.75 %. Experimental validation confirmed that the guidewire model proposed in this paper can dynamically adjust the density of its elastic rods to alter the degree of bending and torsion. It also exhibited a deformation process comparable to that of real guidewires, with an average response time of 6 ms. In the interaction of blood vessel and guidewire models, the simulator blood vessel model used for coronary vascular intervention training exhibited an average response time of 15.42 ms, with a frame rate of approximately 64 FPS. CONCLUSIONS The method presented in this paper achieves deformation simulation of both vascular and guidewire models, demonstrating sufficient real-time performance and accuracy. The interaction efficiency between vascular and guidewire models is enhanced through the unified simulation framework and collision detection. Furthermore, it can be integrated with virtual training scenarios within the system, making it suitable for developing more advanced vascular interventional surgery training systems.
Collapse
Affiliation(s)
- Pan Li
- Tianjin Key Lab of Integrated Design and On-line Monitoring for Light Industry & Food Machinery and Equipment, College of Mechanical Engineering, Tianjin University of Science & Technology, Tianjin 300222, China.
| | - Boxuan Xu
- Tianjin Key Lab of Integrated Design and On-line Monitoring for Light Industry & Food Machinery and Equipment, College of Mechanical Engineering, Tianjin University of Science & Technology, Tianjin 300222, China
| | - Xinxin Zhang
- Tianjin Key Lab of Integrated Design and On-line Monitoring for Light Industry & Food Machinery and Equipment, College of Mechanical Engineering, Tianjin University of Science & Technology, Tianjin 300222, China
| | - Delei Fang
- Tianjin Key Lab of Integrated Design and On-line Monitoring for Light Industry & Food Machinery and Equipment, College of Mechanical Engineering, Tianjin University of Science & Technology, Tianjin 300222, China
| | - Junxia Zhang
- Tianjin Key Lab of Integrated Design and On-line Monitoring for Light Industry & Food Machinery and Equipment, College of Mechanical Engineering, Tianjin University of Science & Technology, Tianjin 300222, China
| |
Collapse
|
2
|
Kos TM, Colombo E, Bartels LW, Robe PA, van Doormaal TPC. Evaluation Metrics for Augmented Reality in Neurosurgical Preoperative Planning, Surgical Navigation, and Surgical Treatment Guidance: A Systematic Review. Oper Neurosurg (Hagerstown) 2023; 26:01787389-990000000-01007. [PMID: 38146941 PMCID: PMC11008635 DOI: 10.1227/ons.0000000000001009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 10/10/2023] [Indexed: 12/27/2023] Open
Abstract
BACKGROUND AND OBJECTIVE Recent years have shown an advancement in the development of augmented reality (AR) technologies for preoperative visualization, surgical navigation, and intraoperative guidance for neurosurgery. However, proving added value for AR in clinical practice is challenging, partly because of a lack of standardized evaluation metrics. We performed a systematic review to provide an overview of the reported evaluation metrics for AR technologies in neurosurgical practice and to establish a foundation for assessment and comparison of such technologies. METHODS PubMed, Embase, and Cochrane were searched systematically for publications on assessment of AR for cranial neurosurgery on September 22, 2022. The findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. RESULTS The systematic search yielded 830 publications; 114 were screened full text, and 80 were included for analysis. Among the included studies, 5% dealt with preoperative visualization using AR, with user perception as the most frequently reported metric. The majority (75%) researched AR technology for surgical navigation, with registration accuracy, clinical outcome, and time measurements as the most frequently reported metrics. In addition, 20% studied the use of AR for intraoperative guidance, with registration accuracy, task outcome, and user perception as the most frequently reported metrics. CONCLUSION For quality benchmarking of AR technologies in neurosurgery, evaluation metrics should be specific to the risk profile and clinical objectives of the technology. A key focus should be on using validated questionnaires to assess user perception; ensuring clear and unambiguous reporting of registration accuracy, precision, robustness, and system stability; and accurately measuring task performance in clinical studies. We provided an overview suggesting which evaluation metrics to use per AR application and innovation phase, aiming to improve the assessment of added value of AR for neurosurgical practice and to facilitate the integration in the clinical workflow.
Collapse
Affiliation(s)
- Tessa M. Kos
- Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands
- Department of Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Elisa Colombo
- Department of Neurosurgery, Clinical Neuroscience Center, Universitätsspital Zürich, Zurich, The Netherlands
| | - L. Wilbert Bartels
- Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Pierre A. Robe
- Department of Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Tristan P. C. van Doormaal
- Department of Neurosurgery, Clinical Neuroscience Center, Universitätsspital Zürich, Zurich, The Netherlands
- Department of Neurosurgery, University Medical Center Utrecht, Utrecht, The Netherlands
| |
Collapse
|
3
|
Chen Z, Marzullo A, Alberti D, Lievore E, Fontana M, De Cobelli O, Musi G, Ferrigno G, De Momi E. FRSR: Framework for real-time scene reconstruction in robot-assisted minimally invasive surgery. Comput Biol Med 2023; 163:107121. [PMID: 37311383 DOI: 10.1016/j.compbiomed.2023.107121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2023] [Revised: 05/12/2023] [Accepted: 05/30/2023] [Indexed: 06/15/2023]
Abstract
3D reconstruction of the intra-operative scenes provides precise position information which is the foundation of various safety related applications in robot-assisted surgery, such as augmented reality. Herein, a framework integrated into a known surgical system is proposed to enhance the safety of robotic surgery. In this paper, we present a scene reconstruction framework to restore the 3D information of the surgical site in real time. In particular, a lightweight encoder-decoder network is designed to perform disparity estimation, which is the key component of the scene reconstruction framework. The stereo endoscope of da Vinci Research Kit (dVRK) is adopted to explore the feasibility of the proposed approach, and it provides the possibility for the migration to other Robot Operating System (ROS) based robot platforms due to the strong independence on hardware. The framework is evaluated using three different scenarios, including a public dataset (3018 pairs of endoscopic images), the scene from the dVRK endoscope in our lab as well as a self-made clinical dataset captured from an oncology hospital. Experimental results show that the proposed framework can reconstruct 3D surgical scenes in real time (25 FPS), and achieve high accuracy (2.69 ± 1.48 mm in MAE, 5.47 ± 1.34 mm in RMSE and 0.41 ± 0.23 in SRE, respectively). It demonstrates that our framework can reconstruct intra-operative scenes with high reliability of both accuracy and speed, and the validation of clinical data also shows its potential in surgery. This work enhances the state of art in 3D intra-operative scene reconstruction based on medical robot platforms. The clinical dataset has been released to promote the development of scene reconstruction in the medical image community.
Collapse
Affiliation(s)
- Ziyang Chen
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, 20133, Italy.
| | - Aldo Marzullo
- Department of Mathematics and Computer Science, University of Calabria, Rende, 87036, Italy
| | - Davide Alberti
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, 20133, Italy
| | - Elena Lievore
- Department of Urology, European Institute of Oncology, IRCCS, Milan, 20141, Italy
| | - Matteo Fontana
- Department of Urology, European Institute of Oncology, IRCCS, Milan, 20141, Italy
| | - Ottavio De Cobelli
- Department of Urology, European Institute of Oncology, IRCCS, Milan, 20141, Italy; Department of Oncology and Onco-haematology, Faculty of Medicine and Surgery, University of Milan, Milan, 20122, Italy
| | - Gennaro Musi
- Department of Urology, European Institute of Oncology, IRCCS, Milan, 20141, Italy; Department of Oncology and Onco-haematology, Faculty of Medicine and Surgery, University of Milan, Milan, 20122, Italy
| | - Giancarlo Ferrigno
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, 20133, Italy
| | - Elena De Momi
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, 20133, Italy; Department of Urology, European Institute of Oncology, IRCCS, Milan, 20141, Italy
| |
Collapse
|
4
|
Brockmeyer P, Wiechens B, Schliephake H. The Role of Augmented Reality in the Advancement of Minimally Invasive Surgery Procedures: A Scoping Review. Bioengineering (Basel) 2023; 10:bioengineering10040501. [PMID: 37106688 PMCID: PMC10136262 DOI: 10.3390/bioengineering10040501] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Revised: 04/13/2023] [Accepted: 04/20/2023] [Indexed: 04/29/2023] Open
Abstract
The purpose of this review was to analyze the evidence on the role of augmented reality (AR) in the improvement of minimally invasive surgical (MIS) procedures. A scoping literature search of the PubMed and ScienceDirect databases was performed to identify articles published in the last five years that addressed the direct impact of AR technology on MIS procedures or that addressed an area of education or clinical care that could potentially be used for MIS development. A total of 359 studies were screened and 31 articles were reviewed in depth and categorized into three main groups: Navigation, education and training, and user-environment interfaces. A comparison of studies within the different application groups showed that AR technology can be useful in various disciplines to advance the development of MIS. Although AR-guided navigation systems do not yet offer a precision advantage, benefits include improved ergonomics and visualization, as well as reduced surgical time and blood loss. Benefits can also be seen in improved education and training conditions and improved user-environment interfaces that can indirectly influence MIS procedures. However, there are still technical challenges that need to be addressed to demonstrate added value to patient care and should be evaluated in clinical trials with sufficient patient numbers or even in systematic reviews or meta-analyses.
Collapse
Affiliation(s)
- Phillipp Brockmeyer
- Department of Oral and Maxillofacial Surgery, University Medical Center Goettingen, D-37075 Goettingen, Germany
| | - Bernhard Wiechens
- Department of Orthodontics, University Medical Center Goettingen, D-37075 Goettingen, Germany
| | - Henning Schliephake
- Department of Oral and Maxillofacial Surgery, University Medical Center Goettingen, D-37075 Goettingen, Germany
| |
Collapse
|
5
|
Palumbo A. Microsoft HoloLens 2 in Medical and Healthcare Context: State of the Art and Future Prospects. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22207709. [PMID: 36298059 PMCID: PMC9611914 DOI: 10.3390/s22207709] [Citation(s) in RCA: 33] [Impact Index Per Article: 16.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/07/2022] [Revised: 09/29/2022] [Accepted: 10/07/2022] [Indexed: 05/08/2023]
Abstract
In the world reference context, although virtual reality, augmented reality and mixed reality have been emerging methodologies for several years, only today technological and scientific advances have made them suitable to revolutionize clinical care and medical contexts through the provision of enhanced functionalities and improved health services. This systematic review provides the state-of-the-art applications of the Microsoft® HoloLens 2 in a medical and healthcare context. Focusing on the potential that this technology has in providing digitally supported clinical care, also but not only in relation to the COVID-19 pandemic, studies that proved the applicability and feasibility of HoloLens 2 in a medical and healthcare scenario were considered. The review presents a thorough examination of the different studies conducted since 2019, focusing on HoloLens 2 medical sub-field applications, device functionalities provided to users, software/platform/framework used, as well as the study validation. The results provided in this paper could highlight the potential and limitations of the HoloLens 2-based innovative solutions and bring focus to emerging research topics, such as telemedicine, remote control and motor rehabilitation.
Collapse
Affiliation(s)
- Arrigo Palumbo
- Department of Medical and Surgical Sciences, Magna Græcia University, 88100 Catanzaro, Italy
| |
Collapse
|
6
|
Zhou Z, Yang Z, Jiang S, Zhuo J, Zhu T, Ma S. Augmented reality surgical navigation system based on the spatial drift compensation method for glioma resection surgery. Med Phys 2022; 49:3963-3979. [PMID: 35383964 DOI: 10.1002/mp.15650] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2021] [Revised: 03/11/2022] [Accepted: 03/28/2022] [Indexed: 11/12/2022] Open
Abstract
BACKGROUND The number of patients who suffer from glioma has been increasing, and this malignancy is a serious threat to human health. The mainstream treatment for glioma is surgical resection; therefore, accurate resection can improve postoperative patient recovery. PURPOSE Many studies have investigated surgical navigation guided by mixed reality, with good outcomes. However, the limitations of mixed reality, such as spatial drift caused by environmental changes, limit its clinical application. Therefore, we present a mixed reality surgical navigation system for glioma resection. Preoperative information can be fused precisely with the real patient with the spatial compensation method to achieve clinically suitable accuracy. METHODS A head-mounted device was used to display virtual information, and a markerless spatial registration method was applied to precisely align the virtual anatomy with the real patient preoperatively. High-accuracy preoperative and intraoperative movement and spatial drift compensation methods were used to increase the positional accuracy of the mixed reality-guided glioma resection system when the patient's head is fixed to the bed frame. Several experiments were designed to validate the accuracy and efficacy of this system. RESULTS Phantom experiments were performed to test the efficacy and accuracy of this system under ideal conditions, and clinical tests were conducted to assess the performance of this system in clinical application. The accuracy of spatial registration was 1.18 mm in the phantom experiments and 1.86 mm in the clinical application. CONCLUSIONS Herein, we present a mixed reality-based multimodality fused surgical navigation system for assisting surgeons in intuitively identifying the glioma boundary intraoperatively. The experimental results indicate that this system has suitable accuracy and efficacy for clinical usage. This article is protected by copyright. All rights reserved.
Collapse
Affiliation(s)
- Zeyang Zhou
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Zhiyong Yang
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Shan Jiang
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China.,Centre for advanced Mechanisms and Robotics, Tianjin University, Tianjin, 300350, China
| | - Jie Zhuo
- Department of Neurosurgery, Tianjin Huanhu hospital, Tianjin, 300200, China
| | - Tao Zhu
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Shixing Ma
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| |
Collapse
|