1
|
Javaheri H, Ghamarnejad O, Bade R, Lukowicz P, Karolus J, Stavrou GA. Beyond the visible: preliminary evaluation of the first wearable augmented reality assistance system for pancreatic surgery. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03131-0. [PMID: 38849631 DOI: 10.1007/s11548-024-03131-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2024] [Accepted: 03/27/2024] [Indexed: 06/09/2024]
Abstract
PURPOSE The retroperitoneal nature of the pancreas, marked by minimal intraoperative organ shifts and deformations, makes augmented reality (AR)-based systems highly promising for pancreatic surgery. This study presents preliminary data from a prospective study aiming to develop the first wearable AR assistance system, ARAS, for pancreatic surgery and evaluating its usability, accuracy, and effectiveness in enhancing the perioperative outcomes of patients. METHODS We developed ARAS as a two-phase system for a wearable AR device to aid surgeons in planning and operation. This system was used to visualize and register patient-specific 3D anatomical models during the surgery. The location and precision of the registered 3D anatomy were evaluated by assessing the arterial pulse and employing Doppler and duplex ultrasonography. The usability, accuracy, and effectiveness of ARAS were assessed using a five-point Likert scale questionnaire. RESULTS Perioperative outcomes of five patients underwent various pancreatic resections with ARAS are presented. Surgeons rated ARAS as excellent for preoperative planning. All structures were accurately identified without any noteworthy errors. Only tumor identification decreased after the preparation phase, especially in patients who underwent pancreaticoduodenectomy because of the extensive mobilization of peripancreatic structures. No perioperative complications related to ARAS were observed. CONCLUSIONS ARAS shows promise in enhancing surgical precision during pancreatic procedures. Its efficacy in preoperative planning and intraoperative vascular identification positions it as a valuable tool for pancreatic surgery and a potential educational resource for future surgical residents.
Collapse
Affiliation(s)
- Hamraz Javaheri
- German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany
| | - Omid Ghamarnejad
- Department of General, Visceral, and Oncological Surgery, Klinikum Saarbrücken, Winterberg 1, 66119, Saarbrücken, Germany
| | | | - Paul Lukowicz
- German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany
- University of Kaiserslautern-Landau, Kaiserslautern, Germany
| | - Jakob Karolus
- German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany.
- University of Kaiserslautern-Landau, Kaiserslautern, Germany.
| | - Gregor Alexander Stavrou
- Department of General, Visceral, and Oncological Surgery, Klinikum Saarbrücken, Winterberg 1, 66119, Saarbrücken, Germany
| |
Collapse
|
2
|
Brollo PP, Bresadola V. Enhancing visualization and guidance in general surgery: a comprehensive and narrative review of the current cutting-edge technologies and future perspectives. J Gastrointest Surg 2024; 28:179-185. [PMID: 38445941 DOI: 10.1016/j.gassur.2023.12.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/09/2023] [Revised: 11/25/2023] [Accepted: 12/08/2023] [Indexed: 03/07/2024]
Abstract
BACKGROUND In the last decade, there has been a great effort in developing new technologies to enhance surgical visualization and guidance. This comprehensive and narrative review aimed to provide a wide and extensive overview of the current state of the art on this topic and their near-future perspectives linked to the development of artificial intelligence (AI), by focusing on the most recent and relevant literature. METHODS A comprehensive and narrative review of the literature was performed by searching specific terms on PubMed/MEDLINE, Scopus, and Embase databases, assessing the current state of the art on this topic. RESULTS Fluorescence-guided surgery, contrast-enhanced ultrasound (CEUS), ultra-high frequency ultrasound (UHFUS), photoacoustic imaging (PAI), and augmented reality (AR) are boosting the field of image-guided techniques as the rapid development of AI in surgery is promising a more automated decision-making and surgical movements in the operating room. CONCLUSION Fluorescence-guided surgery, CEUS, UHFUS, PAI, and AR are becoming crucial to give surgeons a new level of information during the intervention, with the right timing and sequence, and represent the future of surgery. As many more controlled studies are needed to validate the employment of these technologies, the next generation of surgeons must become more familiar with the basics of AI to better incorporate new tools into the daily surgical practice of the future.
Collapse
Affiliation(s)
- Pier Paolo Brollo
- Department of Medicine, General Surgery Department and Simulation Center, Academic Hospital of Udine, University of Udine, Udine, Italy; General Surgical Oncology Department, Istituto di Ricovero e Cura a Carattere Scientifico Centro di Riferimento Oncologico di Aviano (Istituto Nazionale Tumori), Aviano, Italy.
| | - Vittorio Bresadola
- Department of Medicine, General Surgery Department and Simulation Center, Academic Hospital of Udine, University of Udine, Udine, Italy
| |
Collapse
|
3
|
Tamayo-Estebaranz N, Viñas MJ, Arrieta-Blanco P, Zubizarreta-Macho Á, Aragoneses-Lamas JM. Is Augmented Reality Technology Effective in Locating the Apex of Teeth Undergoing Apicoectomy Procedures? J Pers Med 2024; 14:73. [PMID: 38248774 PMCID: PMC10820688 DOI: 10.3390/jpm14010073] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2023] [Revised: 12/12/2023] [Accepted: 01/04/2024] [Indexed: 01/23/2024] Open
Abstract
This study seeks to assess the accuracy of apical location using an augmented reality (AR) device with a free-hand method. Sixty (60) osteotomy site preparations were randomly assigned to one of two study groups: A. AR device (AR) (n = 30), and B. conventional free-hand method (FHM) (n = 30). Preoperative CBCT scans and intraoral scans were taken and uploaded to specialized implant-planning software to virtually plan preparations for the apical location osteotomy sites. The planning software was then used to automatically segment the teeth in each experimental model for their complete visualization using the AR device. A CBCT scan was carried out postoperatively after conducting the apical location procedures. The subsequent datasets were imported into therapeutic software to analyze the coronal, apical, and angular deviations. The Mann-Whitney non-parametric test was used. There were no statistically significant differences identified at the coronal (p = 0.1335), apical (p = 0.2401), and angular deviations (p = 0.4849) between the AR and FHM study groups. The augmented reality technique did not show a statistically significant accuracy of osteotomies for apical location when compared with the conventional free-hand method.
Collapse
Affiliation(s)
- Nuria Tamayo-Estebaranz
- Department of Medicine and Medical Specialties, Faculty of Health Sciences, Universidad Alcalá de Henares, 28691 Madrid, Spain;
- Faculty of Dentistry, Universidad Alfonso X el Sabio, 28691 Madrid, Spain; (M.J.V.); (P.A.-B.); or (J.M.A.-L.)
| | - María José Viñas
- Faculty of Dentistry, Universidad Alfonso X el Sabio, 28691 Madrid, Spain; (M.J.V.); (P.A.-B.); or (J.M.A.-L.)
| | - Patricia Arrieta-Blanco
- Faculty of Dentistry, Universidad Alfonso X el Sabio, 28691 Madrid, Spain; (M.J.V.); (P.A.-B.); or (J.M.A.-L.)
| | - Álvaro Zubizarreta-Macho
- Faculty of Dentistry, Universidad Alfonso X el Sabio, 28691 Madrid, Spain; (M.J.V.); (P.A.-B.); or (J.M.A.-L.)
- Department of Surgery, Faculty of Medicine and Dentistry, University of Salamanca, 37008 Salamanca, Spain
| | - Juan Manuel Aragoneses-Lamas
- Faculty of Dentistry, Universidad Alfonso X el Sabio, 28691 Madrid, Spain; (M.J.V.); (P.A.-B.); or (J.M.A.-L.)
- Department of Dentistry, Universidad Federico Henríquez y Carvajal, Santo Domingo 11114, Dominican Republic
| |
Collapse
|
4
|
Ruggiero F, Cercenelli L, Emiliani N, Badiali G, Bevini M, Zucchelli M, Marcelli E, Tarsitano A. Preclinical Application of Augmented Reality in Pediatric Craniofacial Surgery: An Accuracy Study. J Clin Med 2023; 12:jcm12072693. [PMID: 37048777 PMCID: PMC10095377 DOI: 10.3390/jcm12072693] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2023] [Revised: 03/29/2023] [Accepted: 03/31/2023] [Indexed: 04/08/2023] Open
Abstract
Background: Augmented reality (AR) allows the overlapping and integration of virtual information with the real environment. The camera of the AR device reads the object and integrates the virtual data. It has been widely applied to medical and surgical sciences in recent years and has the potential to enhance intraoperative navigation. Materials and methods: In this study, the authors aim to assess the accuracy of AR guidance when using the commercial HoloLens 2 head-mounted display (HMD) in pediatric craniofacial surgery. The Authors selected fronto-orbital remodeling (FOR) as the procedure to test (specifically, frontal osteotomy and nasal osteotomy were considered). Six people (three surgeons and three engineers) were recruited to perform the osteotomies on a 3D printed stereolithographic model under the guidance of AR. By means of calibrated CAD/CAM cutting guides with different grooves, the authors measured the accuracy of the osteotomies that were performed. We tested accuracy levels of ±1.5 mm, ±1 mm, and ±0.5 mm. Results: With the HoloLens 2, the majority of the individuals involved were able to successfully trace the trajectories of the frontal and nasal osteotomies with an accuracy level of ±1.5 mm. Additionally, 80% were able to achieve an accuracy level of ±1 mm when performing a nasal osteotomy, and 52% were able to achieve an accuracy level of ±1 mm when performing a frontal osteotomy, while 61% were able to achieve an accuracy level of ±0.5 mm when performing a nasal osteotomy, and 33% were able to achieve an accuracy level of ±0.5 mm when performing a frontal osteotomy. Conclusions: despite this being an in vitro study, the authors reported encouraging results for the prospective use of AR on actual patients.
Collapse
Affiliation(s)
- Federica Ruggiero
- Department of Biomedical and Neuromotor Science, University of Bologna, 40138 Bologna, Italy
- Maxillo-Facial Surgery Unit, AUSL Bologna, 40124 Bologna, Italy
| | - Laura Cercenelli
- Laboratory of Bioengineering—eDIMES Lab, Department of Medical and Surgical Sciences (DIMEC), University of Bologna, 40138 Bologna, Italy
| | - Nicolas Emiliani
- Laboratory of Bioengineering—eDIMES Lab, Department of Medical and Surgical Sciences (DIMEC), University of Bologna, 40138 Bologna, Italy
| | - Giovanni Badiali
- Department of Biomedical and Neuromotor Science, University of Bologna, 40138 Bologna, Italy
- Oral and Maxillo-Facial Surgery Unit, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Via Albertoni 15, 40138 Bologna, Italy
| | - Mirko Bevini
- Department of Biomedical and Neuromotor Science, University of Bologna, 40138 Bologna, Italy
- Oral and Maxillo-Facial Surgery Unit, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Via Albertoni 15, 40138 Bologna, Italy
| | - Mino Zucchelli
- Pediatric Neurosurgery, IRCCS Istituto delle Scienze Neurologiche di Bologna, Via Altura 3, 40138 Bologna, Italy
| | - Emanuela Marcelli
- Laboratory of Bioengineering—eDIMES Lab, Department of Medical and Surgical Sciences (DIMEC), University of Bologna, 40138 Bologna, Italy
| | - Achille Tarsitano
- Department of Biomedical and Neuromotor Science, University of Bologna, 40138 Bologna, Italy
- Oral and Maxillo-Facial Surgery Unit, IRCCS Azienda Ospedaliero-Universitaria di Bologna, Via Albertoni 15, 40138 Bologna, Italy
| |
Collapse
|
5
|
Ma L, Huang T, Wang J, Liao H. Visualization, registration and tracking techniques for augmented reality guided surgery: a review. Phys Med Biol 2023; 68. [PMID: 36580681 DOI: 10.1088/1361-6560/acaf23] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2022] [Accepted: 12/29/2022] [Indexed: 12/31/2022]
Abstract
Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories ofin situvisualization and nonin situvisualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.
Collapse
Affiliation(s)
- Longfei Ma
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, People's Republic of China
| | - Tianqi Huang
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, People's Republic of China
| | - Jie Wang
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, People's Republic of China
| | - Hongen Liao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, 100084, People's Republic of China
| |
Collapse
|
6
|
A novel motionless calibration method for augmented reality surgery navigation system based on optical tracker. Heliyon 2022; 8:e12115. [PMID: 36590529 PMCID: PMC9801086 DOI: 10.1016/j.heliyon.2022.e12115] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2022] [Revised: 09/25/2022] [Accepted: 11/28/2022] [Indexed: 12/13/2022] Open
Abstract
Augmented reality (AR) surgery navigation systems display the pre-operation planned virtual model at the accurate position in the real surgical scene to assist the operation. Accurate calibration of the mapping relationship between the virtual coordinate and the real world is the key to the virtual-real fusion effect. Former calibration methods require the doctor user to conduct complex manual procedures before usage. This paper introduces a novel motionless virtual-real calibration method. The method only requires to take a mixed reality image containing both virtual and real marker balls using the built-in forward camera of the AR glasses. The mapping relationship between the virtual and real spaces is calculated by using the camera coordinate system as a transformation medium. The composition and working process of the AR navigation system is introduced, and then the mathematical principle of the calibration is designed. The feasibility of the proposed calibration scheme is verified with a verification experiment, and the average registration accuracy of the scheme is around 5.80mm, which is of same level of formerly reported methods. The proposed method is convenient and rapid to implement, and the calibration accuracy is not dependent on the user experience. Further, it can potentially realize the real-time update of the registration transformation matrix, which can improve the AR fusion accuracy when the AR glasses moves. This motionless calibration method has great potential to be applied in future clinical navigation research.
Collapse
|
7
|
Chiou SY, Zhang ZY, Liu HL, Yan JL, Wei KC, Chen PY. Augmented Reality Surgical Navigation System for External Ventricular Drain. Healthcare (Basel) 2022; 10:healthcare10101815. [PMID: 36292263 PMCID: PMC9601392 DOI: 10.3390/healthcare10101815] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2022] [Revised: 09/14/2022] [Accepted: 09/19/2022] [Indexed: 12/02/2022] Open
Abstract
Augmented reality surgery systems are playing an increasing role in the operating room, but applying such systems to neurosurgery presents particular challenges. In addition to using augmented reality technology to display the position of the surgical target position in 3D in real time, the application must also display the scalpel entry point and scalpel orientation, with accurate superposition on the patient. To improve the intuitiveness, efficiency, and accuracy of extra-ventricular drain surgery, this paper proposes an augmented reality surgical navigation system which accurately superimposes the surgical target position, scalpel entry point, and scalpel direction on a patient’s head and displays this data on a tablet. The accuracy of the optical measurement system (NDI Polaris Vicra) was first independently tested, and then complemented by the design of functions to help the surgeon quickly identify the surgical target position and determine the preferred entry point. A tablet PC was used to display the superimposed images of the surgical target, entry point, and scalpel on top of the patient, allowing for correct scalpel orientation. Digital imaging and communications in medicine (DICOM) results for the patient’s computed tomography were used to create a phantom and its associated AR model. This model was then imported into the application, which was then executed on the tablet. In the preoperative phase, the technician first spent 5–7 min to superimpose the virtual image of the head and the scalpel. The surgeon then took 2 min to identify the intended target position and entry point position on the tablet, which then dynamically displayed the superimposed image of the head, target position, entry point position, and scalpel (including the scalpel tip and scalpel orientation). Multiple experiments were successfully conducted on the phantom, along with six practical trials of clinical neurosurgical EVD. In the 2D-plane-superposition model, the optical measurement system (NDI Polaris Vicra) provided highly accurate visualization (2.01 ± 1.12 mm). In hospital-based clinical trials, the average technician preparation time was 6 min, while the surgeon required an average of 3.5 min to set the target and entry-point positions and accurately overlay the orientation with an NDI surgical stick. In the preparation phase, the average time required for the DICOM-formatted image processing and program import was 120 ± 30 min. The accuracy of the designed augmented reality optical surgical navigation system met clinical requirements, and can provide a visual and intuitive guide for neurosurgeons. The surgeon can use the tablet application to obtain real-time DICOM-formatted images of the patient, change the position of the surgical entry point, and instantly obtain an updated surgical path and surgical angle. The proposed design can be used as the basis for various augmented reality brain surgery navigation systems in the future.
Collapse
Affiliation(s)
- Shin-Yan Chiou
- Department of Electrical Engineering, College of Engineering, Chang Gung University, Kwei-Shan, Taoyuan 333, Taiwan
- Department of Nuclear Medicine, Linkou Chang Gung Memorial Hospital, Taoyuan 333, Taiwan
- Department of Neurosurgery, Keelung Chang Gung Memorial Hospital, Keelung 204, Taiwan
| | - Zhi-Yue Zhang
- Department of Electrical Engineering, College of Engineering, Chang Gung University, Kwei-Shan, Taoyuan 333, Taiwan
| | - Hao-Li Liu
- Department of Electrical Engineering, National Taiwan University, Taipei 106, Taiwan
| | - Jiun-Lin Yan
- Department of Neurosurgery, Keelung Chang Gung Memorial Hospital, Keelung 204, Taiwan
| | - Kuo-Chen Wei
- Department of Neurosurgery, New Taipei City TuCheng Hospital, New Taipei City 236, Taiwan
| | - Pin-Yuan Chen
- Department of Neurosurgery, Keelung Chang Gung Memorial Hospital, Keelung 204, Taiwan
- School of Medicine, Chang Gung University, Kwei-Shan, Taoyuan 333, Taiwan
- Correspondence: ; Tel.: +886-2-2431-3131
| |
Collapse
|
8
|
Kokko MA, Van Citters DW, Seigne JD, Halter RJ. A particle filter approach to dynamic kidney pose estimation in robotic surgical exposure. Int J Comput Assist Radiol Surg 2022; 17:1079-1089. [PMID: 35511394 DOI: 10.1007/s11548-022-02638-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2021] [Accepted: 04/08/2022] [Indexed: 11/27/2022]
Abstract
PURPOSE Traditional soft tissue registration methods require direct intraoperative visualization of a significant portion of the target anatomy in order to produce acceptable surface alignment. Image guidance is therefore generally not available during the robotic exposure of structures like the kidneys which are not immediately visualized upon entry into the abdomen. This paper proposes guiding surgical exposure with an iterative state estimator that assimilates small visual cues into an a priori anatomical model as exposure progresses, thereby evolving pose estimates for the occluded structures of interest. METHODS Intraoperative surface observations of a right kidney are simulated using endoscope tracking and preoperative tomography from a representative robotic partial nephrectomy case. Clinically relevant random perturbations of the true kidney pose are corrected using this sequence of observations in a particle filter framework to estimate an optimal similarity transform for fitting a patient-specific kidney model at each step. The temporal response of registration error is compared against that of serial rigid coherent point drift (CPD) in both static and simulated dynamic surgical fields, and for varying levels of observation persistence. RESULTS In the static case, both particle filtering and persistent CPD achieved sub-5 mm accuracy, with CPD processing observations 75% faster. Particle filtering outperformed CPD in the dynamic case under equivalent computation times due to the former requiring only minimal persistence. CONCLUSION This proof-of-concept simulation study suggests that Bayesian state estimation may provide a viable pathway to image guidance for surgical exposure in the abdomen, especially in the presence of dynamic intraoperative tissue displacement and deformation.
Collapse
Affiliation(s)
- Michael A Kokko
- Thayer School of Engineering, Dartmouth College, 15 Thayer Drive, Hanover, NH, 03755, USA.
| | - Douglas W Van Citters
- Thayer School of Engineering, Dartmouth College, 15 Thayer Drive, Hanover, NH, 03755, USA
| | - John D Seigne
- Dartmouth-Hitchcock Medical Center, Section of Urology, 1 Medical Center Drive, Lebanon, NH, 03756, USA.,Geisel School of Medicine, Dartmouth College, 1 Rope Ferry Road, Hanover, NH, 03755, USA
| | - Ryan J Halter
- Thayer School of Engineering, Dartmouth College, 15 Thayer Drive, Hanover, NH, 03755, USA.,Geisel School of Medicine, Dartmouth College, 1 Rope Ferry Road, Hanover, NH, 03755, USA
| |
Collapse
|
9
|
Birlo M, Edwards PJE, Clarkson M, Stoyanov D. Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review. Med Image Anal 2022; 77:102361. [PMID: 35168103 PMCID: PMC10466024 DOI: 10.1016/j.media.2022.102361] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Revised: 11/17/2021] [Accepted: 01/10/2022] [Indexed: 12/11/2022]
Abstract
This article presents a systematic review of optical see-through head mounted display (OST-HMD) usage in augmented reality (AR) surgery applications from 2013 to 2020. Articles were categorised by: OST-HMD device, surgical speciality, surgical application context, visualisation content, experimental design and evaluation, accuracy and human factors of human-computer interaction. 91 articles fulfilled all inclusion criteria. Some clear trends emerge. The Microsoft HoloLens increasingly dominates the field, with orthopaedic surgery being the most popular application (28.6%). By far the most common surgical context is surgical guidance (n=58) and segmented preoperative models dominate visualisation (n=40). Experiments mainly involve phantoms (n=43) or system setup (n=21), with patient case studies ranking third (n=19), reflecting the comparative infancy of the field. Experiments cover issues from registration to perception with very different accuracy results. Human factors emerge as significant to OST-HMD utility. Some factors are addressed by the systems proposed, such as attention shift away from the surgical site and mental mapping of 2D images to 3D patient anatomy. Other persistent human factors remain or are caused by OST-HMD solutions, including ease of use, comfort and spatial perception issues. The significant upward trend in published articles is clear, but such devices are not yet established in the operating room and clinical studies showing benefit are lacking. A focused effort addressing technical registration and perceptual factors in the lab coupled with design that incorporates human factors considerations to solve clear clinical problems should ensure that the significant current research efforts will succeed.
Collapse
Affiliation(s)
- Manuel Birlo
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), Charles Bell House, 43-45 Foley Street, London W1W 7TS, UK.
| | - P J Eddie Edwards
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), Charles Bell House, 43-45 Foley Street, London W1W 7TS, UK
| | - Matthew Clarkson
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), Charles Bell House, 43-45 Foley Street, London W1W 7TS, UK
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), Charles Bell House, 43-45 Foley Street, London W1W 7TS, UK
| |
Collapse
|
10
|
Giannone F, Felli E, Cherkaoui Z, Mascagni P, Pessaux P. Augmented Reality and Image-Guided Robotic Liver Surgery. Cancers (Basel) 2021; 13:cancers13246268. [PMID: 34944887 PMCID: PMC8699460 DOI: 10.3390/cancers13246268] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2021] [Revised: 12/07/2021] [Accepted: 12/10/2021] [Indexed: 12/13/2022] Open
Abstract
Artificial intelligence makes surgical resection easier and safer, and, at the same time, can improve oncological results. The robotic system fits perfectly with these more or less diffused technologies, and it seems that this benefit is mutual. In liver surgery, robotic systems help surgeons to localize tumors and improve surgical results with well-defined preoperative planning or increased intraoperative detection. Furthermore, they can balance the absence of tactile feedback and help recognize intrahepatic biliary or vascular structures during parenchymal transection. Some of these systems are well known and are already widely diffused in open and laparoscopic hepatectomies, such as indocyanine green fluorescence or ultrasound-guided resections, whereas other tools, such as Augmented Reality, are far from being standardized because of the high complexity and elevated costs. In this paper, we review all the experiences in the literature on the use of artificial intelligence systems in robotic liver resections, describing all their practical applications and their weaknesses.
Collapse
Affiliation(s)
- Fabio Giannone
- Department of Visceral and Digestive Surgery, University Hospital of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France; (F.G.); (E.F.); (Z.C.)
- Institute of Viral and Liver Disease, Inserm U1110, University of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France
- University Hospital Institute (IHU), Institute of Image-Guided Surgery, University of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France;
| | - Emanuele Felli
- Department of Visceral and Digestive Surgery, University Hospital of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France; (F.G.); (E.F.); (Z.C.)
- Institute of Viral and Liver Disease, Inserm U1110, University of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France
- University Hospital Institute (IHU), Institute of Image-Guided Surgery, University of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France;
| | - Zineb Cherkaoui
- Department of Visceral and Digestive Surgery, University Hospital of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France; (F.G.); (E.F.); (Z.C.)
- Institute of Viral and Liver Disease, Inserm U1110, University of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France
| | - Pietro Mascagni
- University Hospital Institute (IHU), Institute of Image-Guided Surgery, University of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France;
| | - Patrick Pessaux
- Department of Visceral and Digestive Surgery, University Hospital of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France; (F.G.); (E.F.); (Z.C.)
- Institute of Viral and Liver Disease, Inserm U1110, University of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France
- University Hospital Institute (IHU), Institute of Image-Guided Surgery, University of Strasbourg, 1 Place de l’Hôpital, 67100 Strasbourg, France;
- Correspondence: ; Tel.: +33-369-550-552
| |
Collapse
|
11
|
Wang Y, Cao D, Chen SL, Li YM, Zheng YW, Ohkohchi N. Current trends in three-dimensional visualization and real-time navigation as well as robot-assisted technologies in hepatobiliary surgery. World J Gastrointest Surg 2021; 13:904-922. [PMID: 34621469 PMCID: PMC8462083 DOI: 10.4240/wjgs.v13.i9.904] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/14/2021] [Revised: 04/19/2021] [Accepted: 08/02/2021] [Indexed: 02/06/2023] Open
Abstract
With the continuous development of digital medicine, minimally invasive precision and safety have become the primary development trends in hepatobiliary surgery. Due to the specificity and complexity of hepatobiliary surgery, traditional preoperative imaging techniques such as computed tomography and magnetic resonance imaging cannot meet the need for identification of fine anatomical regions. Imaging-based three-dimensional (3D) reconstruction, virtual simulation of surgery and 3D printing optimize the surgical plan through preoperative assessment, improving the controllability and safety of intraoperative operations, and in difficult-to-reach areas of the posterior and superior liver, assistive robots reproduce the surgeon’s natural movements with stable cameras, reducing natural vibrations. Electromagnetic navigation in abdominal surgery solves the problem of conventional surgery still relying on direct visual observation or preoperative image assessment. We summarize and compare these recent trends in digital medical solutions for the future development and refinement of digital medicine in hepatobiliary surgery.
Collapse
Affiliation(s)
- Yun Wang
- Institute of Regenerative Medicine, and Affiliated Hospital of Jiangsu University, Jiangsu University, Zhenjiang 212001, Jiangsu Province, China
| | - Di Cao
- Institute of Regenerative Medicine, and Affiliated Hospital of Jiangsu University, Jiangsu University, Zhenjiang 212001, Jiangsu Province, China
| | - Si-Lin Chen
- Institute of Regenerative Medicine, and Affiliated Hospital of Jiangsu University, Jiangsu University, Zhenjiang 212001, Jiangsu Province, China
| | - Yu-Mei Li
- Institute of Regenerative Medicine, and Affiliated Hospital of Jiangsu University, Jiangsu University, Zhenjiang 212001, Jiangsu Province, China
| | - Yun-Wen Zheng
- Institute of Regenerative Medicine, and Affiliated Hospital of Jiangsu University, Jiangsu University, Zhenjiang 212001, Jiangsu Province, China
- Department of Gastrointestinal and Hepato-Biliary-Pancreatic Surgery, Faculty of Medicine, University of Tsukuba, Tsukuba 305-8575, Ibaraki, Japan
- Guangdong Provincial Key Laboratory of Large Animal Models for Biomedicine, and School of Biotechnology and Heath Sciences, Wuyi University, Jiangmen 529020, Guangdong Province, China
- School of Medicine, Yokohama City University, Yokohama 234-0006, Kanagawa, Japan
| | - Nobuhiro Ohkohchi
- Department of Gastrointestinal and Hepato-Biliary-Pancreatic Surgery, Faculty of Medicine, University of Tsukuba, Tsukuba 305-8575, Ibaraki, Japan
| |
Collapse
|
12
|
Kanehira M, Okamoto T, Abe K, Yasuda J, Onda S, Futagawa Y, Ikegami T, Suzuki N, Hattori A. Development of recognised position-guided navigation system. Int J Med Robot 2021; 17:e2322. [PMID: 34405536 DOI: 10.1002/rcs.2322] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2020] [Revised: 08/16/2021] [Accepted: 08/16/2021] [Indexed: 11/11/2022]
Abstract
BACKGROUND Previously, we developed an image-guided navigation system (IG-NS) incorporating augmented reality technology. Nevertheless, the system could still only aid the operator by presenting imagery and was short of achieving the goal of developing a real navigation system. Therefore, we developed a recognised position-guided navigation system (RP-NS) and herein reported the functionality and usefulness of this system in a phantom model for clinical applications. METHODS We developed RP-NS which was reconstructed by adding the positional recognition and instruction functions with the cautions by displaying the images on the monitor with a voice to the IG-NS. We evaluated accuracy of positional recognition and instruction functions using phantom model. By utilising the chronological recording of the tip position of the surgical apparatus, the surgical precision of the operators was assessed. Finally, the feasibility of improvements in surgical precision using this system was evaluated. RESULTS The RP-NS indicated an accuracy of the position recognition functions with an error of 2.7 mm. The surgeons could perform partial hepatectomies within mean value of 7.5% error as compared with calculated volume according to the instruction. Improvements in surgical precision using this system were obtained on the surgeons with different levels. CONCLUSIONS The RP-NS was highly effective as a navigation system owing to precise positional recognition and adequate instruction functions. Therefore, these results indicate that the use of this system may complement differences in proficiency, and numerically evaluate surgical skills and analyse tendencies of surgeons.
Collapse
Affiliation(s)
- Masaru Kanehira
- Department of Surgery, The Jikei University Daisan Hospital, Komae, Japan
| | - Tomoyoshi Okamoto
- Department of Surgery, The Jikei University Daisan Hospital, Komae, Japan
| | - Kyohei Abe
- Department of Surgery, The Jikei University Daisan Hospital, Komae, Japan
| | - Jungo Yasuda
- Department of Surgery, The Jikei University School of Medicine, Minato-ku, Japan
| | - Shinji Onda
- Department of Surgery, The Jikei University School of Medicine, Minato-ku, Japan
| | - Yasuro Futagawa
- Department of Surgery, The Jikei University Daisan Hospital, Komae, Japan
| | - Toru Ikegami
- Department of Surgery, The Jikei University School of Medicine, Minato-ku, Japan
| | - Naoki Suzuki
- Institute for High Dimensional Medical Imaging, The Jikei University School of Medicine, Minato-ku, Japan
| | - Asaki Hattori
- Institute for High Dimensional Medical Imaging, The Jikei University School of Medicine, Minato-ku, Japan
| |
Collapse
|
13
|
Schneider C, Allam M, Stoyanov D, Hawkes DJ, Gurusamy K, Davidson BR. Performance of image guided navigation in laparoscopic liver surgery - A systematic review. Surg Oncol 2021; 38:101637. [PMID: 34358880 DOI: 10.1016/j.suronc.2021.101637] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2021] [Revised: 07/04/2021] [Accepted: 07/24/2021] [Indexed: 02/07/2023]
Abstract
BACKGROUND Compared to open surgery, minimally invasive liver resection has improved short term outcomes. It is however technically more challenging. Navigated image guidance systems (IGS) are being developed to overcome these challenges. The aim of this systematic review is to provide an overview of their current capabilities and limitations. METHODS Medline, Embase and Cochrane databases were searched using free text terms and corresponding controlled vocabulary. Titles and abstracts of retrieved articles were screened for inclusion criteria. Due to the heterogeneity of the retrieved data it was not possible to conduct a meta-analysis. Therefore results are presented in tabulated and narrative format. RESULTS Out of 2015 articles, 17 pre-clinical and 33 clinical papers met inclusion criteria. Data from 24 articles that reported on accuracy indicates that in recent years navigation accuracy has been in the range of 8-15 mm. Due to discrepancies in evaluation methods it is difficult to compare accuracy metrics between different systems. Surgeon feedback suggests that current state of the art IGS may be useful as a supplementary navigation tool, especially in small liver lesions that are difficult to locate. They are however not able to reliably localise all relevant anatomical structures. Only one article investigated IGS impact on clinical outcomes. CONCLUSIONS Further improvements in navigation accuracy are needed to enable reliable visualisation of tumour margins with the precision required for oncological resections. To enhance comparability between different IGS it is crucial to find a consensus on the assessment of navigation accuracy as a minimum reporting standard.
Collapse
Affiliation(s)
- C Schneider
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK.
| | - M Allam
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK; General surgery Department, Tanta University, Egypt
| | - D Stoyanov
- Department of Computer Science, University College London, London, UK; Centre for Medical Image Computing (CMIC), University College London, London, UK
| | - D J Hawkes
- Centre for Medical Image Computing (CMIC), University College London, London, UK; Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK
| | - K Gurusamy
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK
| | - B R Davidson
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK
| |
Collapse
|
14
|
Zhao Z, Poyhonen J, Chen Cai X, Sophie Woodley Hooper F, Ma Y, Hu Y, Ren H, Song W, Tsz Ho Tse Z. Augmented reality technology in image-guided therapy: State-of-the-art review. Proc Inst Mech Eng H 2021; 235:1386-1398. [PMID: 34304631 PMCID: PMC8573682 DOI: 10.1177/09544119211034357] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
Abstract
Image-guided therapies have been on the rise in recent years as they can achieve higher accuracy and are less invasive than traditional methods. By combining augmented reality technology with image-guided therapy, more organs, and tissues can be observed by surgeons to improve surgical accuracy. In this review, 233 publications (dated from 2015 to 2020) on the design and application of augmented reality-based systems for image-guided therapy, including both research prototypes and commercial products, were considered for review. Based on their functions and applications. Sixteen studies were selected. The engineering specifications and applications were analyzed and summarized for each study. Finally, future directions and existing challenges in the field were summarized and discussed.
Collapse
Affiliation(s)
- Zhuo Zhao
- School of Electrical and Computer Engineering, University of Georgia, Athens, GA, USA
| | - Jasmin Poyhonen
- Department of Electronic Engineering, University of York, York, UK
| | - Xin Chen Cai
- Department of Electronic Engineering, University of York, York, UK
| | | | - Yangmyung Ma
- Hull York Medical School, University of York, York, UK
| | - Yihua Hu
- Department of Electronic Engineering, University of York, York, UK
| | - Hongliang Ren
- Department of Electronic Engineering The Chinese University of Hong Kong (CUHK), Hong Kong, China
| | - Wenzhan Song
- School of Electrical and Computer Engineering, University of Georgia, Athens, GA, USA
| | - Zion Tsz Ho Tse
- Department of Electronic Engineering, University of York, York, UK
| |
Collapse
|
15
|
Liu K, Gao Y, Abdelrehem A, Zhang L, Chen X, Xie L, Wang X. Augmented reality navigation method for recontouring surgery of craniofacial fibrous dysplasia. Sci Rep 2021; 11:10043. [PMID: 33976233 PMCID: PMC8113548 DOI: 10.1038/s41598-021-88860-x] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2020] [Accepted: 04/13/2021] [Indexed: 11/09/2022] Open
Abstract
The objective of this study is to introduce the application of augmented reality (AR) navigation system developed by the authors in recontouring surgery of craniofacial fibrous dysplasia. Five consecutive patients with craniofacial fibrous dysplasia were enrolled. Through three-dimensional (3D) simulation, a virtual plan was designed to reconstruct the normal anatomical contour of the deformed region. Surgical recontouring was achieved with the assistance of the AR navigation system. The accuracy of the surgical procedure was assessed by superimposing the post-operative 3D craniomaxillofacial model onto the virtual plan. The pre-operative preparation time and operation time were also counted. In all patients, AR navigation was performed successfully, with a mean ± SD of the errors of 1.442 ± 0.234 mm. The operative time of the patients ranged from 60 to 80 min. The pre-operative preparation time was 20 min for each patient. All the patients showed uneventful healing without any complications, in addition to satisfaction with the post-operative aesthetics. Using our AR navigation system in recontouring surgery can provide surgeons with a comprehensive and intuitive view of the recontouring border, as well as the depth, in real time. This method could improve the efficiency and safety of craniofacial fibrous dysplasia recontouring procedures.
Collapse
Affiliation(s)
- Kai Liu
- Department of Oral and Craniomaxillofacial Surgery, Shanghai 9Th People's Hospital, Shanghai Jiaotong University College of Medicine, Shanghai, China.,Shanghai Key Laboratory of Stomatology, Shanghai, China
| | - Yuan Gao
- Institute of Forming Technology and Equipment, Shanghai JiaoTong University, Shanghai, China
| | - Ahmed Abdelrehem
- Department of Craniomaxillofacial and Plastic Surgery, Faculty of Dentistry, Alexandria University, Alexandria, Egypt
| | - Lei Zhang
- Department of Oral and Craniomaxillofacial Surgery, Shanghai 9Th People's Hospital, Shanghai Jiaotong University College of Medicine, Shanghai, China.,Shanghai Key Laboratory of Stomatology, Shanghai, China
| | - Xi Chen
- Department of Oral and Craniomaxillofacial Surgery, Shanghai 9Th People's Hospital, Shanghai Jiaotong University College of Medicine, Shanghai, China
| | - Le Xie
- Institute of Forming Technology and Equipment, Shanghai JiaoTong University, Shanghai, China. .,Institute of Medical Robot, Shanghai JiaoTong University, Shanghai, China. .,Quanzhou Normal University, Fujian, China.
| | - Xudong Wang
- Department of Oral and Craniomaxillofacial Surgery, Shanghai 9Th People's Hospital, Shanghai Jiaotong University College of Medicine, Shanghai, China. .,Shanghai Key Laboratory of Stomatology, Shanghai, China.
| |
Collapse
|
16
|
Use of augmented reality navigation to optimise the surgical management of craniofacial fibrous dysplasia. Br J Oral Maxillofac Surg 2021; 60:162-167. [PMID: 34930644 DOI: 10.1016/j.bjoms.2021.03.011] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2021] [Accepted: 03/25/2021] [Indexed: 11/20/2022]
Abstract
The aim of this study was to apply an augmented reality (AR) navigation technique based on a head- mounted display in the treatment of craniofacial fibrous dysplasia and to explore the feasibility and the value of AR in craniofacial surgery. With preoperative planning and three-dimensional simulation, the normal anatomical contours of the deformed area were recreated by superimposing the unaffected side on to the affected side. We completed the recontouring procedures in real time with the aid of an AR navigation system. The surgical outcome was assessed by superimposing the postoperative computed tomographic images on to the preoperative virtual plan. The preparation and operation times were recorded. With intraoperative AR guidance, facial bone recontouring was performed uneventfully in all cases. The mean (SD) discrepancy between the actual surgical reduction and preoperative planning was 1.036 (0.081) mm (range: 0.913 (0.496) to 1.165 (0.498) mm). The operation time ranged from 50 to 80 minutes, with an average of 66.4 minutes. The preoperative preparation time ranged from 26 to 36 minutes, with a mean of 29.6 minutes. AR navigation-assisted facial bone recontouring is a valuable treatment modality in managing craniomaxillofacial fibrous dysplasia and shows benefits in improving the efficiency and safety of this complicated procedure.
Collapse
|
17
|
Augmented Reality-Assisted Pancreaticoduodenectomy with Superior Mesenteric Vein Resection and Reconstruction. Gastroenterol Res Pract 2021; 2021:9621323. [PMID: 33815500 PMCID: PMC7990556 DOI: 10.1155/2021/9621323] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/06/2020] [Revised: 02/22/2021] [Accepted: 02/27/2021] [Indexed: 11/17/2022] Open
Abstract
Introduction Pancreaticoduodenectomy (PD) with superior mesenteric vein (SMV) reconstruction are often required to achieve complete (R0) resection for pancreatic head cancer (PHC) with tumor invasion of the SMV. Augmented reality (AR) technology can be used to assist in determining the extent of SMV involvement by superimposing virtual 3-dimensional (3D) images of the pancreas and regional vasculature on the surgical field. Materials and Methods Three patients with PHC and tumor invasion of the SMV underwent AR-assisted PD with SMV resection and reconstruction following preoperative computed tomography scanning. Preoperative imaging data were used to reconstruct 3D images of anatomical structures, including the tumor, portal vein (PV), SMV, and splenic vein (SV). Using AR software installed on a smart phone, the reconstructed 3D images were superimposed on the surgical field as viewed in a smart phone display to provide intermittent navigational assistance to the surgeon in identifying the boundaries of PHC tumor invasion for resection of the vessels involved. Result All patients successfully completed the operation. Intraoperative AR applications displayed virtual images of the pancreas, SMV, bile duct, common hepatic artery (CHA), and superior mesenteric artery (SMA). Two patients required end-to-end anastomosis for reconstruction of the SMV. One patient required allogenic vascular bypass to reconstruct the SMV-PV juncture with concomitant reconstruction of the SV-SMV confluence by end-to-side anastomosis of the SV and bypass vessel. Postoperative pathology confirmed R0 resections for all patients. Conclusion AR navigation technology based on preoperative CT image data can assist surgeons performing PD with SMV resection and reconstruction.
Collapse
|
18
|
Singh A, Ferry D, Ramakrishnan A, Balasubramanian S. Using Virtual Reality in Biomedical Engineering Education. J Biomech Eng 2020; 142:111013. [PMID: 32747925 PMCID: PMC7580657 DOI: 10.1115/1.4048005] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2020] [Revised: 07/23/2020] [Indexed: 11/08/2022]
Abstract
This study explored virtual reality (VR) as an educational tool to offer immersive and experiential learning environments to biomedical engineering (BME) students. VR and traditional two-dimensional (2D) videos were created and used to teach required communication skills to BME students' while working with clinical partners in healthcare settings. The videos of interdisciplinary teams (engineering and nursing students) tackling medical device-related problems, similar to those commonly observed in healthcare settings, were shown to BME students. Student surveys indicated that, through VR videos, they felt more immersed in real-world clinical scenarios while learning about the clinical problems, each team-member's areas of expertise, their roles and responsibilities, and how an interdisciplinary team operated collectively to solve a problem in the presented settings. Students with a prior in-person immersion experience, in the presented settings, reported VR videos to serve as a possible alternative to in-person immersion and a useful tool for their preparedness for real-world clinical immersion. We concluded that VR holds promise as an educational tool to offer simulated clinical scenarios that are effective in training BME students for interprofessional collaborations.
Collapse
Affiliation(s)
- Anita Singh
- Biomedical Engineering, School of Engineering, Widener University, Chester, PA 19063
| | - Dawn Ferry
- School of Nursing, Widener University, Chester, PA 19013
| | - Arun Ramakrishnan
- College of Nursing and Health Professions, Drexel University, Philadelphia, PA 19102
| | - Sriram Balasubramanian
- School of Biomedical Engineering, Sciences and Health Systems, Drexel University, Philadelphia, PA 19104
| |
Collapse
|
19
|
Chu LC, Park S, Kawamoto S, Yuille AL, Hruban RH, Fishman EK. Pancreatic Cancer Imaging: A New Look at an Old Problem. Curr Probl Diagn Radiol 2020; 50:540-550. [PMID: 32988674 DOI: 10.1067/j.cpradiol.2020.08.002] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2020] [Accepted: 08/21/2020] [Indexed: 12/18/2022]
Abstract
Computed tomography is the most commonly used imaging modality to detect and stage pancreatic cancer. Previous advances in pancreatic cancer imaging have focused on optimizing image acquisition parameters and reporting standards. However, current state-of-the-art imaging approaches still misdiagnose some potentially curable pancreatic cancers and do not provide prognostic information or inform optimal management strategies beyond stage. Several recent developments in pancreatic cancer imaging, including artificial intelligence and advanced visualization techniques, are rapidly changing the field. The purpose of this article is to review how these recent advances have the potential to revolutionize pancreatic cancer imaging.
Collapse
Affiliation(s)
- Linda C Chu
- The Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins University School of Medicine, Baltimore, MD.
| | - Seyoun Park
- The Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Satomi Kawamoto
- The Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Alan L Yuille
- Department of Computer Science, Johns Hopkins University, Baltimore, MD
| | - Ralph H Hruban
- Sol Goldman Pancreatic Cancer Research Center, Department of Pathology, Johns Hopkins University School of Medicine, Baltimore, MD
| | - Elliot K Fishman
- The Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins University School of Medicine, Baltimore, MD
| |
Collapse
|
20
|
Augmented Reality: A Systematic Review of Its Benefits and Challenges in E-learning Contexts. APPLIED SCIENCES-BASEL 2020. [DOI: 10.3390/app10165660] [Citation(s) in RCA: 28] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
Abstract
Augmented reality (AR) has received increasing attention in the research literature as a fundamental pedagogical tool that can enhance learning at most educational levels. In academic contexts, this technology permits the superimposing of three-dimensional images onto the real environment. Although AR has been found to enhance learning in the academic environment, no systematic review of it has been conducted to identify, evaluate, and summarize empirical findings on its advantages and challenges in e-learning contexts. Hence, a systematic review of the research literature was conducted on the use of AR in e-learning contexts, with a focus on the key benefits and challenges related to its adoption and implementation. Electronic searches on databases, including Springer, Science Direct, EBSCO, and Google Scholar, were performed to retrieve relevant journal articles; 28 studies were included after they were screened using the inclusion and exclusion criteria. The key benefits of using AR in e-learning included support of kinesthetic (tactile) learning, collaborative learning, distance/remote learning, learner-centered learning, and creative learning. Studies also reported that AR enhanced students’ engagement, motivation, attention/focus, interactivity, verbal participation, concentration, knowledge retention, and spatial abilities, as well as information accessibility. The findings suggest that challenges associated with AR in e-learning include information and cognitive overload, lack of experience in using the technology, resistance from teachers, complex technology, costly technology, and technical issues, such as connectivity problems.
Collapse
|
21
|
Hu MH, Chiang CC, Wang ML, Wu NY, Lee PY. Clinical feasibility of the augmented reality computer-assisted spine surgery system for percutaneous vertebroplasty. EUROPEAN SPINE JOURNAL : OFFICIAL PUBLICATION OF THE EUROPEAN SPINE SOCIETY, THE EUROPEAN SPINAL DEFORMITY SOCIETY, AND THE EUROPEAN SECTION OF THE CERVICAL SPINE RESEARCH SOCIETY 2020; 29:1590-1596. [DOI: 10.1007/s00586-020-06417-4] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/10/2019] [Revised: 02/16/2020] [Accepted: 04/11/2020] [Indexed: 12/15/2022]
|
22
|
Verhey JT, Haglin JM, Verhey EM, Hartigan DE. Virtual, augmented, and mixed reality applications in orthopedic surgery. Int J Med Robot 2020; 16:e2067. [DOI: 10.1002/rcs.2067] [Citation(s) in RCA: 56] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2019] [Revised: 12/11/2019] [Accepted: 12/11/2019] [Indexed: 12/18/2022]
Affiliation(s)
| | | | | | - David E. Hartigan
- Mayo Clinic Alix School of Medicine Scottsdale Arizona
- Department of Orthopedic SurgeryMayo Clinic Phoenix Arizona
| |
Collapse
|
23
|
Fairén M, Moyés J, Insa E. VR4Health: Personalized teaching and learning anatomy using VR. J Med Syst 2020; 44:94. [PMID: 32193612 PMCID: PMC7082407 DOI: 10.1007/s10916-020-01550-5] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2020] [Accepted: 02/18/2020] [Indexed: 11/25/2022]
Abstract
Virtual Reality (VR) is being integrated into many different areas of our lives, from industrial engineering to video-games, and also including teaching and education. We have several examples where VR has been used to engage students and facilitate their 3D spatial understanding, but can VR help also teachers? What are the benefits teachers can obtain on using VR applications? In this paper we present an application (VR4Health) designed to allow students to directly inspect 3D models of several human organs by using Virtual Reality systems. The application is designed to be used in an HMD device autonomously as a self-learning tool and also reports information to teachers in order that he/she becomes aware of what the students do and can redirect his/her work to the concrete necessities of the student. We evaluate both the students’ and the teachers’ perception by doing an experiment and asking them to fill-in a questionnaire at the end of the experiment.
Collapse
Affiliation(s)
- M Fairén
- ViRVIG Group, Universitat Politècnica de Catalunya, Barcelona, Spain
| | - J Moyés
- ViRVIG Group, Universitat Politècnica de Catalunya, Barcelona, Spain
| | - E Insa
- GIEES Group, Campus Docent Sant Joan de Déu-Fundació privada, Barcelona, Spain.
| |
Collapse
|
24
|
Schneider C, Thompson S, Totz J, Song Y, Allam M, Sodergren MH, Desjardins AE, Barratt D, Ourselin S, Gurusamy K, Stoyanov D, Clarkson MJ, Hawkes DJ, Davidson BR. Comparison of manual and semi-automatic registration in augmented reality image-guided liver surgery: a clinical feasibility study. Surg Endosc 2020; 34:4702-4711. [PMID: 32780240 PMCID: PMC7524854 DOI: 10.1007/s00464-020-07807-x] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2020] [Accepted: 07/10/2020] [Indexed: 02/06/2023]
Abstract
BACKGROUND The laparoscopic approach to liver resection may reduce morbidity and hospital stay. However, uptake has been slow due to concerns about patient safety and oncological radicality. Image guidance systems may improve patient safety by enabling 3D visualisation of critical intra- and extrahepatic structures. Current systems suffer from non-intuitive visualisation and a complicated setup process. A novel image guidance system (SmartLiver), offering augmented reality visualisation and semi-automatic registration has been developed to address these issues. A clinical feasibility study evaluated the performance and usability of SmartLiver with either manual or semi-automatic registration. METHODS Intraoperative image guidance data were recorded and analysed in patients undergoing laparoscopic liver resection or cancer staging. Stereoscopic surface reconstruction and iterative closest point matching facilitated semi-automatic registration. The primary endpoint was defined as successful registration as determined by the operating surgeon. Secondary endpoints were system usability as assessed by a surgeon questionnaire and comparison of manual vs. semi-automatic registration accuracy. Since SmartLiver is still in development no attempt was made to evaluate its impact on perioperative outcomes. RESULTS The primary endpoint was achieved in 16 out of 18 patients. Initially semi-automatic registration failed because the IGS could not distinguish the liver surface from surrounding structures. Implementation of a deep learning algorithm enabled the IGS to overcome this issue and facilitate semi-automatic registration. Mean registration accuracy was 10.9 ± 4.2 mm (manual) vs. 13.9 ± 4.4 mm (semi-automatic) (Mean difference - 3 mm; p = 0.158). Surgeon feedback was positive about IGS handling and improved intraoperative orientation but also highlighted the need for a simpler setup process and better integration with laparoscopic ultrasound. CONCLUSION The technical feasibility of using SmartLiver intraoperatively has been demonstrated. With further improvements semi-automatic registration may enhance user friendliness and workflow of SmartLiver. Manual and semi-automatic registration accuracy were comparable but evaluation on a larger patient cohort is required to confirm these findings.
Collapse
Affiliation(s)
- C. Schneider
- Division of Surgery & Interventional Science, Royal Free Campus, University College London, Pond Street, London, NW3 2QG UK
| | - S. Thompson
- Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Centre for Medical Image Computing (CMIC), University College London, London, UK ,Department of Medical Physics and Bioengineering, University College London, London, UK
| | - J. Totz
- Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Centre for Medical Image Computing (CMIC), University College London, London, UK ,Department of Medical Physics and Bioengineering, University College London, London, UK
| | - Y. Song
- Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Centre for Medical Image Computing (CMIC), University College London, London, UK ,Department of Medical Physics and Bioengineering, University College London, London, UK
| | - M. Allam
- Division of Surgery & Interventional Science, Royal Free Campus, University College London, Pond Street, London, NW3 2QG UK
| | - M. H. Sodergren
- Centre for Medical Image Computing (CMIC), University College London, London, UK
| | - A. E. Desjardins
- Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Department of Medical Physics and Bioengineering, University College London, London, UK
| | - D. Barratt
- Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Centre for Medical Image Computing (CMIC), University College London, London, UK ,Department of Medical Physics and Bioengineering, University College London, London, UK
| | - S. Ourselin
- Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Centre for Medical Image Computing (CMIC), University College London, London, UK ,Department of Medical Physics and Bioengineering, University College London, London, UK
| | - K. Gurusamy
- Division of Surgery & Interventional Science, Royal Free Campus, University College London, Pond Street, London, NW3 2QG UK ,Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Department of Hepatopancreatobiliary and Liver Transplant Surgery, Royal Free Hospital, London, UK
| | - D. Stoyanov
- Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Centre for Medical Image Computing (CMIC), University College London, London, UK ,Department of Computer Science, University College London, London, UK
| | - M. J. Clarkson
- Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Centre for Medical Image Computing (CMIC), University College London, London, UK ,Department of Medical Physics and Bioengineering, University College London, London, UK
| | - D. J. Hawkes
- Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Centre for Medical Image Computing (CMIC), University College London, London, UK ,Department of Medical Physics and Bioengineering, University College London, London, UK
| | - B. R. Davidson
- Division of Surgery & Interventional Science, Royal Free Campus, University College London, Pond Street, London, NW3 2QG UK ,Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK ,Department of Hepatopancreatobiliary and Liver Transplant Surgery, Royal Free Hospital, London, UK
| |
Collapse
|
25
|
The Preliminary Application of Augmented Reality in Unilateral Orbitozygomatic Maxillary Complex Fractures Treatment. J Craniofac Surg 2020; 31:542-548. [DOI: 10.1097/scs.0000000000006238] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022] Open
|
26
|
Lin L, Fan B, Yu Z, Xu L, Yuan J, Wu J, Wei M. Application of computer-assisted navigation in mandibular angle osteotomy. J Int Med Res 2019; 47:3160-3170. [PMID: 31204540 PMCID: PMC6683906 DOI: 10.1177/0300060519850722] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Objective To compare the effectiveness, accuracy, and surgical safety of a navigation technique with those of a traditional technique for intraoperative mandibular angle osteotomy. Methods Forty-three postsurgical patients with mandibular angle hypertrophy who were admitted to our Department from June 2014 to June 2017 were retrospectively reviewed. Of these patients, 23 underwent mandibular angle osteotomy using computer-assisted navigation (navigation group), and 20 underwent osteotomy using a traditional technique (traditional group). Postoperative computed tomography images were analyzed by three-dimensional software. Each patient’s facial proportion indices were measured using Mimics 19.0 software, and statistical comparisons and analyses were performed preoperatively and postoperatively. Results The postoperative facial contour morphology and facial proportion were improved in both groups; the navigation group showed greater improvement. The difference between the predicted and postoperative values was smaller in the navigation group than traditional group. The postoperative shape of the mandibular angle sample was similar to the preoperative predicted shape in the navigation group. No complications occurred in the navigation group, but paresthesia occurred in 17% of patients in the traditional group. Conclusions Mandibular angle osteotomy aided with computer-assisted navigation is more effective, accurate, and safe than the traditional technique and represents a promising clinical approach.
Collapse
Affiliation(s)
- Liqin Lin
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Bokai Fan
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Zheyuan Yu
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Liang Xu
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Jie Yuan
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Jun Wu
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Min Wei
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| |
Collapse
|
27
|
Jiang T, Zhu M, Chai G, Li Q. Precision of a Novel Craniofacial Surgical Navigation System Based on Augmented Reality Using an Occlusal Splint as a Registration Strategy. Sci Rep 2019; 9:501. [PMID: 30679507 PMCID: PMC6345963 DOI: 10.1038/s41598-018-36457-2] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2017] [Accepted: 11/14/2018] [Indexed: 11/30/2022] Open
Abstract
The authors have developed a novel augmented reality (AR)-based navigation system (NS) for craniofacial surgery. In this study, the authors aimed to measure the precision of the system and further analyze the primary influencing factors of the precision. The drilling of holes into the mandibles of ten beagle dogs was performed under the AR-based NS, and the precision was analyzed by comparing the deviation between the preoperational plan and the surgical outcome. The AR-based NS was successfully applied to quickly and precisely drill holes in the mandibles. The mean positional deviation between the preoperative design and intraoperative navigation was 1.29 ± 0.70 mm for the entry points and 2.47 ± 0.66 mm for the end points, and the angular deviation was 1.32° ± 1.17°. The precision linearly decreased with the distance from the marker. In conclusion, the precision of this system could satisfy clinical requirements, and this system may serve as a helpful tool for improving the precision in craniofacial surgery.
Collapse
Affiliation(s)
- Taoran Jiang
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Zhizaoju Road 639, Shanghai, 200011, People's Republic of China
| | - Ming Zhu
- Department of Plastic and Reconstructive Surgery, Zhongshan Hospital, Fudan University, No. 180 Feng Lin Road, Shanghai, 200032, People's Republic of China
| | - Gang Chai
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Zhizaoju Road 639, Shanghai, 200011, People's Republic of China
| | - Qingfeng Li
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People's Hospital, Shanghai Jiao Tong University School of Medicine, Zhizaoju Road 639, Shanghai, 200011, People's Republic of China.
| |
Collapse
|
28
|
Witowski J, Wake N, Grochowska A, Sun Z, Budzyński A, Major P, Popiela TJ, Pędziwiatr M. Investigating accuracy of 3D printed liver models with computed tomography. Quant Imaging Med Surg 2019; 9:43-52. [PMID: 30788245 DOI: 10.21037/qims.2018.09.16] [Citation(s) in RCA: 32] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022]
Abstract
Background The aim of this study was to evaluate the accuracy of three-dimensional (3D) printed liver models developed by a cost-effective approach for establishing validity of using these models in a clinical setting. Methods Fifteen patients undergoing laparoscopic liver resection in a single surgical department were included. Patient-specific, 1-1 scale 3D printed liver models including the liver, tumor, and vasculature were created from contrast-enhanced computed tomography (CT) images using a cost-effective approach. The 3D models were subsequently CT scanned, 3D image post-processing was performed, and these 3D computer models (MCT) were compared to the original 3D models created from the original patient images (PCT). 3D computer models of each type were co-registered using a point set registration method. 3D volume measurements of the liver and lesions were calculated and compared for each set. In addition, Hausdorff distances were calculated and surface quality was compared by generated heatmaps. Results The median liver volume in MCT was 1,281.84 [interquartile range (IQR) =296.86] cm3, and 1,448.03 (IQR =413.23) cm3 in PCT. Analysis of differences between surfaces showed that the median value of mean Hausdorff distances for liver parenchyma was 1.92 mm. Bland-Altman plots revealed no significant bias in liver volume and diameters of hepatic veins and tumor location. Median errors of all measured vessel diameters were smaller than CT slice height. There was a slight trend towards undersizing anatomical structures, although those errors are most likely due to source imaging. Conclusions We have confirmed the accuracy of 3D printed liver models created by using the low-cost method. 3D models are useful tools for pre-operative planning and intra-operative guidance. Future research in this field should continue to move towards clinical trials for assessment of the impact of these models on pre-surgical planning decisions and perioperative outcomes.
Collapse
Affiliation(s)
- Jan Witowski
- 2nd Department of General Surgery, Jagiellonian University Medical College, Kraków, Poland.,Centre for Research, Training and Innovation in Surgery (CERTAIN Surgery), Kraków, Poland
| | - Nicole Wake
- Center for Advanced Imaging Innovation and Research (CAI2R) and Bernard and Irene Schwartz Center for Biomedical Imaging, Department of Radiology, NYU Langone Health, NYU School of Medicine, New York, NY, USA
| | - Anna Grochowska
- Chair of Radiology, Jagiellonian University Medical College, Kraków, Poland
| | - Zhonghua Sun
- Discipline of Medical Radiation Sciences, School of Molecular and Life Sciences, Curtin University, Perth, Australia
| | - Andrzej Budzyński
- 2nd Department of General Surgery, Jagiellonian University Medical College, Kraków, Poland.,Centre for Research, Training and Innovation in Surgery (CERTAIN Surgery), Kraków, Poland
| | - Piotr Major
- 2nd Department of General Surgery, Jagiellonian University Medical College, Kraków, Poland.,Centre for Research, Training and Innovation in Surgery (CERTAIN Surgery), Kraków, Poland
| | | | - Michał Pędziwiatr
- 2nd Department of General Surgery, Jagiellonian University Medical College, Kraków, Poland.,Centre for Research, Training and Innovation in Surgery (CERTAIN Surgery), Kraków, Poland
| |
Collapse
|
29
|
Lee D, Kong HJ, Kim D, Yi JW, Chai YJ, Lee KE, Kim HC. Preliminary study on application of augmented reality visualization in robotic thyroid surgery. Ann Surg Treat Res 2018; 95:297-302. [PMID: 30505820 PMCID: PMC6255749 DOI: 10.4174/astr.2018.95.6.297] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2018] [Revised: 06/01/2018] [Accepted: 06/12/2018] [Indexed: 01/24/2023] Open
Abstract
Purpose Increased robotic surgery is attended by increased reports of complications, largely due to limited operative view and lack of tactile sense. These kinds of obstacles, which seldom occur in open surgery, are challenging for beginner surgeons. To enhance robotic surgery safety, we created an augmented reality (AR) model of the organs around the thyroid glands, and tested the AR model applicability in robotic thyroidectomy. Methods We created AR images of the thyroid gland, common carotid arteries, trachea, and esophagus using preoperative CT images of a thyroid carcinoma patient. For a preliminary test, we overlaid the AR images on a 3-dimensional printed model at five different angles and evaluated its accuracy using Dice similarity coefficient. We then overlaid the AR images on the real-time operative images during robotic thyroidectomy. Results The Dice similarity coefficients ranged from 0.984 to 0.9908, and the mean of the five different angles was 0.987. During the entire process of robotic thyroidectomy, the AR images were successfully overlaid on the real-time operative images using manual registration. Conclusion We successfully demonstrated the use of AR on the operative field during robotic thyroidectomy. Although there are currently limitations, the use of AR in robotic surgery will become more practical as the technology advances and may contribute to the enhancement of surgical safety.
Collapse
Affiliation(s)
- Dongheon Lee
- Interdisciplinary Program, Bioengineering Major, Graduate School, Seoul National University, Seoul, Korea
| | - Hyoun-Joong Kong
- Department of Biomedical Engineering, Chungnam National University Hospital, Chungnam National University College of Medicine, Daejeon, Korea
| | - Donguk Kim
- Department of Biomedical Engineering, Seoul National University Hospital, Seoul, Korea
| | - Jin Wook Yi
- Department of Surgery, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Korea
| | - Young Jun Chai
- Department of Surgery, Seoul Metropolitan Government - Seoul National University Boramae Medical Center, Seoul, Korea
| | - Kyu Eun Lee
- Department of Surgery, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Korea
| | - Hee Chan Kim
- Department of Biomedical Engineering, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Korea.,Institute of Medical & Biological Engineering, Medical Research Center, Seoul National University College of Medicine, Seoul, Korea
| |
Collapse
|
30
|
Chu Y, Li X, Yang X, Ai D, Huang Y, Song H, Jiang Y, Wang Y, Chen X, Yang J. Perception enhancement using importance-driven hybrid rendering for augmented reality based endoscopic surgical navigation. BIOMEDICAL OPTICS EXPRESS 2018; 9:5205-5226. [PMID: 30460123 PMCID: PMC6238941 DOI: 10.1364/boe.9.005205] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/31/2018] [Revised: 09/20/2018] [Accepted: 09/22/2018] [Indexed: 06/09/2023]
Abstract
Misleading depth perception may greatly affect the correct identification of complex structures in image-guided surgery. In this study, we propose a novel importance-driven hybrid rendering method to enhance perception for navigated endoscopic surgery. First, the volume structures are enhanced using gradient-based shading to reduce the color information in low-priority regions and improve the distinctions between complicated structures. Second, an importance sorting method based on the order-independent transparency rendering is introduced to intensify the perception of multiple surfaces. Third, volume data are adaptively truncated and emphasized with respect to the perspective orientation and the illustration of critical information for viewing range extension. Various experimental results prove that with the combination of volume and surface rendering, our method can effectively improve the depth distinction of multiple objects both in simulated and clinical scenes. Our importance-driven surface rendering method demonstrates improved average performance and statistical significance as rated by 15 participants (five clinicians and ten non-clinicians) on a five-point Likert scale. Further, the average frame rate of hybrid rendering with thin-layer sectioning reaches 42 fps. Given that the process of the hybrid rendering is fully automatic, it can be utilized in real-time surgical navigation to improve the rendering efficiency and information validity.
Collapse
Affiliation(s)
- Yakui Chu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Xu Li
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Xilin Yang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Danni Ai
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Yong Huang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Hong Song
- School of Software, Beijing Institute of Technology, Beijing 100081, China
| | - Yurong Jiang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Yongtian Wang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Xiaohong Chen
- Department of Otolaryngology, Head and Neck Surgery, Beijing Tongren Hospital, Beijing 100730, China
- Co-corresponding authors
| | - Jian Yang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
- Co-corresponding authors
| |
Collapse
|
31
|
Jakubovic R, Guha D, Gupta S, Lu M, Jivraj J, Standish BA, Leung MK, Mariampillai A, Lee K, Siegler P, Skowron P, Farooq H, Nguyen N, Alarcon J, Deorajh R, Ramjist J, Ford M, Howard P, Phan N, Costa LD, Heyn C, Tan G, George R, Cadotte DW, Mainprize T, Yee A, Yang VXD. High Speed, High Density Intraoperative 3D Optical Topographical Imaging with Efficient Registration to MRI and CT for Craniospinal Surgical Navigation. Sci Rep 2018; 8:14894. [PMID: 30291261 PMCID: PMC6173775 DOI: 10.1038/s41598-018-32424-z] [Citation(s) in RCA: 26] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2018] [Accepted: 09/05/2018] [Indexed: 11/09/2022] Open
Abstract
Intraoperative image-guided surgical navigation for craniospinal procedures has significantly improved accuracy by providing an avenue for the surgeon to visualize underlying internal structures corresponding to the exposed surface anatomy. Despite the obvious benefits of surgical navigation, surgeon adoption remains relatively low due to long setup and registration times, steep learning curves, and workflow disruptions. We introduce an experimental navigation system utilizing optical topographical imaging (OTI) to acquire the 3D surface anatomy of the surgical cavity, enabling visualization of internal structures relative to exposed surface anatomy from registered preoperative images. Our OTI approach includes near instantaneous and accurate optical measurement of >250,000 surface points, computed at >52,000 points-per-second for considerably faster patient registration than commercially available benchmark systems without compromising spatial accuracy. Our experience of 171 human craniospinal surgical procedures, demonstrated significant workflow improvement (41 s vs. 258 s and 794 s, p < 0.05) relative to benchmark navigation systems without compromising surgical accuracy. Our advancements provide the cornerstone for widespread adoption of image guidance technologies for faster and safer surgeries without intraoperative CT or MRI scans. This work represents a major workflow improvement for navigated craniospinal procedures with possible extension to other image-guided applications.
Collapse
Affiliation(s)
- Raphael Jakubovic
- Department of Biomedical Physics, Ryerson University, Toronto, ON, Canada.,Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada
| | - Daipayan Guha
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Division of Neurosurgery, Department of Surgery, University of Toronto, Toronto, ON, Canada.,Institute of Medical Science, School of Graduate Studies, University of Toronto, Toronto, ON, Canada
| | - Shaurya Gupta
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Division of Neurosurgery, Department of Surgery, University of Toronto, Toronto, ON, Canada
| | - Michael Lu
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada
| | - Jamil Jivraj
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada
| | - Beau A Standish
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada
| | - Michael K Leung
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada
| | - Adrian Mariampillai
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada
| | - Kenneth Lee
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada
| | - Peter Siegler
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada
| | - Patryk Skowron
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada
| | - Hamza Farooq
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada
| | - Nhu Nguyen
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada
| | - Joseph Alarcon
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada
| | - Ryan Deorajh
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada
| | - Joel Ramjist
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada
| | - Michael Ford
- Division of Orthopedic Surgery, Department of Surgery, University of Toronto, Toronto, ON, Canada
| | - Peter Howard
- Division of Neuroradiology, Department of Medical Imaging, Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Department of Medical Imaging, University of Toronto, Toronto, ON, Canada
| | - Nicolas Phan
- Division of Neurosurgery, Department of Surgery, University of Toronto, Toronto, ON, Canada
| | - Leo da Costa
- Division of Neurosurgery, Department of Surgery, University of Toronto, Toronto, ON, Canada
| | - Chris Heyn
- Division of Neuroradiology, Department of Medical Imaging, Sunnybrook Health Sciences Centre, Toronto, ON, Canada.,Department of Medical Imaging, University of Toronto, Toronto, ON, Canada
| | - Gamaliel Tan
- Jurong Health, Ng Teng Fong General Hospital, Singapore, Singapore
| | - Rajeesh George
- Jurong Health, Ng Teng Fong General Hospital, Singapore, Singapore
| | - David W Cadotte
- Spine Program and Division of Neurosurgery, Department of Clinical Neurosciences, Department of Radiology, University of Calgary, Calgary, Canada.,Hotchkiss Brain Institute, Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - Todd Mainprize
- Division of Neurosurgery, Department of Surgery, University of Toronto, Toronto, ON, Canada
| | - Albert Yee
- Division of Orthopedic Surgery, Department of Surgery, University of Toronto, Toronto, ON, Canada
| | - Victor X D Yang
- Biophotonics and Bioengineering Laboratory, Ryerson University Sunnybrook Health Sciences Centre, Toronto, ON, Canada. .,Division of Neurosurgery, Department of Surgery, University of Toronto, Toronto, ON, Canada. .,Institute of Medical Science, School of Graduate Studies, University of Toronto, Toronto, ON, Canada. .,Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada.
| |
Collapse
|
32
|
Abstract
Augmentation reality technology offers virtual information in addition to that of the real environment and thus opens new possibilities in various fields. The medical applications of augmentation reality are generally concentrated on surgery types, including neurosurgery, laparoscopic surgery and plastic surgery. Augmentation reality technology is also widely used in medical education and training. In dentistry, oral and maxillofacial surgery is the primary area of use, where dental implant placement and orthognathic surgery are the most frequent applications. Recent technological advancements are enabling new applications of restorative dentistry, orthodontics and endodontics. This review briefly summarizes the history, definitions, features, and components of augmented reality technology and discusses its applications and future perspectives in dentistry.
Collapse
Affiliation(s)
- Ho-Beom Kwon
- Department of Prosthodontics, School of Dentistry, Seoul National University and Dental Research Institute, Seoul, Korea
| | - Young-Seok Park
- Department of Oral Medicine and Oral Diagnosis, School of Dentistry, Seoul National University and Dental Research Institute, Seoul, Korea
| | - Jung-Suk Han
- Department of Prosthodontics, School of Dentistry, Seoul National University and Dental Research Institute, Seoul, Korea
| |
Collapse
|
33
|
Zhu M, Liu F, Zhou C, Lin L, Zhang Y, Chai G, Xie L, Qi F, Li Q. Does intraoperative navigation improve the accuracy of mandibular angle osteotomy: Comparison between augmented reality navigation, individualised templates and free-hand techniques. J Plast Reconstr Aesthet Surg 2018; 71:1188-1195. [DOI: 10.1016/j.bjps.2018.03.018] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2017] [Revised: 03/06/2018] [Accepted: 03/30/2018] [Indexed: 01/14/2023]
|
34
|
Penza V, Ciullo AS, Moccia S, Mattos LS, De Momi E. EndoAbS dataset: Endoscopic abdominal stereo image dataset for benchmarking 3D stereo reconstruction algorithms. Int J Med Robot 2018; 14:e1926. [DOI: 10.1002/rcs.1926] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2017] [Revised: 05/01/2018] [Accepted: 05/02/2018] [Indexed: 11/05/2022]
Affiliation(s)
- Veronica Penza
- Department of Advanced Robotics; Istituto Italiano di Tecnologia; 16163 Genova Italy
- Department of Electronics Information and Bioengineering; Politecnico di Milano; 20133 Milano Italy
| | - Andrea S. Ciullo
- Department of Electronics Information and Bioengineering; Politecnico di Milano; 20133 Milano Italy
| | - Sara Moccia
- Department of Advanced Robotics; Istituto Italiano di Tecnologia; 16163 Genova Italy
- Department of Electronics Information and Bioengineering; Politecnico di Milano; 20133 Milano Italy
| | - Leonardo S. Mattos
- Department of Advanced Robotics; Istituto Italiano di Tecnologia; 16163 Genova Italy
| | - Elena De Momi
- Department of Electronics Information and Bioengineering; Politecnico di Milano; 20133 Milano Italy
| |
Collapse
|
35
|
A Novel Augmented Reality-Based Navigation System in Perforator Flap Transplantation - A Feasibility Study. Ann Plast Surg 2018; 79:192-196. [PMID: 28509695 DOI: 10.1097/sap.0000000000001078] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
BACKGROUND In perforator flap transplantation, dissection of the perforator is an important but difficult procedure because of the high variability in vascular anatomy. Preoperative imaging techniques could provide substantial information about vascular anatomy; however, it cannot provide direct guidance for surgeons during the operation. In this study, a navigation system (NS) was established to overlie a vascular map on surgical sites to further provide a direct guide for perforator flap transplantation. METHODS The NS was established based on computed tomographic angiography and augmented reality techniques. A virtual vascular map was reconstructed according to computed tomographic angiography data and projected onto real patient images using ARToolKit software. Additionally, a screw-fixation marker holder was created to facilitate registration. With the use of a tracking and display system, we conducted the NS on an animal model and measured the system error on a rapid prototyping model. RESULTS The NS assistance allowed for correct identification, as well as a safe and precise dissection of the perforator. The mean value of the system error was determined to be 3.474 ± 1.546 mm. CONCLUSIONS Augmented reality-based NS can provide precise navigation information by directly displaying a 3-dimensional individual anatomical virtual model onto the operative field in real time. It will allow rapid identification and safe dissection of a perforator in free flap transplantation surgery.
Collapse
|
36
|
Wilhelm D, Vogel T, Ostler D, Marahrens N, Kohn N, Koller S, Friess H, Kranzfelder M. Enhanced Visualization: From Intraoperative Tissue Differentiation to Augmented Reality. Visc Med 2018; 34:52-59. [PMID: 29594170 DOI: 10.1159/000485940] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022] Open
Abstract
Background Optimal visualization of the operative field and methods that additionally provide supportive optical information form the basis for target-directed and successful surgery. This article strives to give an overview of current enhanced visualization techniques in visceral surgery and to highlight future developments. Methods The article was written as a comprehensive review on this topic and is based on a MEDLINE search and ongoing research from our own group and from other working groups. Results Various techniques for enhanced visualization are described comprising augmented reality, unspecific and targeted staining methods, and optical modalities such as narrow-band imaging. All facilitate our surgical performance; however, due to missing randomized controlled studies for most of the innovations reported on, the available evidence is low. Conclusion Many new visualization technologies are emerging with the aim to improve our perception of the surgical field leading to less invasive, target-oriented, and elegant treatment forms that are of significant benefit to our patients.
Collapse
Affiliation(s)
- Dirk Wilhelm
- Department of Surgery, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany.,Working Group for Minimally Invasive Interdisciplinary Therapeutical Intervention, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Thomas Vogel
- Department of Surgery, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany.,Working Group for Minimally Invasive Interdisciplinary Therapeutical Intervention, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Daniel Ostler
- Working Group for Minimally Invasive Interdisciplinary Therapeutical Intervention, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Nils Marahrens
- Working Group for Minimally Invasive Interdisciplinary Therapeutical Intervention, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Nils Kohn
- Working Group for Minimally Invasive Interdisciplinary Therapeutical Intervention, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Sebastian Koller
- Working Group for Minimally Invasive Interdisciplinary Therapeutical Intervention, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Helmut Friess
- Department of Surgery, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Michael Kranzfelder
- Department of Surgery, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany.,Working Group for Minimally Invasive Interdisciplinary Therapeutical Intervention, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| |
Collapse
|
37
|
Detmer FJ, Hettig J, Schindele D, Schostak M, Hansen C. Virtual and Augmented Reality Systems for Renal Interventions: A Systematic Review. IEEE Rev Biomed Eng 2017; 10:78-94. [PMID: 28885161 DOI: 10.1109/rbme.2017.2749527] [Citation(s) in RCA: 28] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
PURPOSE Many virtual and augmented reality systems have been proposed to support renal interventions. This paper reviews such systems employed in the treatment of renal cell carcinoma and renal stones. METHODS A systematic literature search was performed. Inclusion criteria were virtual and augmented reality systems for radical or partial nephrectomy and renal stone treatment, excluding systems solely developed or evaluated for training purposes. RESULTS In total, 52 research papers were identified and analyzed. Most of the identified literature (87%) deals with systems for renal cell carcinoma treatment. About 44% of the systems have already been employed in clinical practice, but only 20% in studies with ten or more patients. Main challenges remaining for future research include the consideration of organ movement and deformation, human factor issues, and the conduction of large clinical studies. CONCLUSION Augmented and virtual reality systems have the potential to improve safety and outcomes of renal interventions. In the last ten years, many technical advances have led to more sophisticated systems, which are already applied in clinical practice. Further research is required to cope with current limitations of virtual and augmented reality assistance in clinical environments.
Collapse
|
38
|
Recent Development of Augmented Reality in Surgery: A Review. JOURNAL OF HEALTHCARE ENGINEERING 2017; 2017:4574172. [PMID: 29065604 PMCID: PMC5585624 DOI: 10.1155/2017/4574172] [Citation(s) in RCA: 138] [Impact Index Per Article: 19.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/12/2017] [Accepted: 07/03/2017] [Indexed: 12/11/2022]
Abstract
Introduction The development augmented reality devices allow physicians to incorporate data visualization into diagnostic and treatment procedures to improve work efficiency, safety, and cost and to enhance surgical training. However, the awareness of possibilities of augmented reality is generally low. This review evaluates whether augmented reality can presently improve the results of surgical procedures. Methods We performed a review of available literature dating from 2010 to November 2016 by searching PubMed and Scopus using the terms “augmented reality” and “surgery.” Results. The initial search yielded 808 studies. After removing duplicates and including only journal articles, a total of 417 studies were identified. By reading of abstracts, 91 relevant studies were chosen to be included. 11 references were gathered by cross-referencing. A total of 102 studies were included in this review. Conclusions The present literature suggest an increasing interest of surgeons regarding employing augmented reality into surgery leading to improved safety and efficacy of surgical procedures. Many studies showed that the performance of newly devised augmented reality systems is comparable to traditional techniques. However, several problems need to be addressed before augmented reality is implemented into the routine practice.
Collapse
|
39
|
Chu Y, Yang J, Ma S, Ai D, Li W, Song H, Li L, Chen D, Chen L, Wang Y. Registration and fusion quantification of augmented reality based nasal endoscopic surgery. Med Image Anal 2017; 42:241-256. [PMID: 28881251 DOI: 10.1016/j.media.2017.08.003] [Citation(s) in RCA: 34] [Impact Index Per Article: 4.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2016] [Revised: 06/10/2017] [Accepted: 08/02/2017] [Indexed: 11/24/2022]
Abstract
This paper quantifies the registration and fusion display errors of augmented reality-based nasal endoscopic surgery (ARNES). We comparatively investigated the spatial calibration process for front-end endoscopy and redefined the accuracy level of a calibrated endoscope by using a calibration tool with improved structural reliability. We also studied how registration accuracy was combined with the number and distribution of the deployed fiducial points (FPs) for positioning and the measured registration time. A physically integrated ARNES prototype was customarily configured for performance evaluation in skull base tumor resection surgery with an innovative approach of dynamic endoscopic vision expansion. As advised by surgical experts in otolaryngology, we proposed a hierarchical rendering scheme to properly adapt the fused images with the required visual sensation. By constraining the rendered sight in a known depth and radius, the visual focus of the surgeon can be induced only on the anticipated critical anatomies and vessel structures to avoid misguidance. Furthermore, error analysis was conducted to examine the feasibility of hybrid optical tracking based on point cloud, which was proposed in our previous work as an in-surgery registration solution. Measured results indicated that the error of target registration for ARNES can be reduced to 0.77 ± 0.07 mm. For initial registration, our results suggest that a trade-off for a new minimal time of registration can be reached when the distribution of five FPs is considered. For in-surgery registration, our findings reveal that the intrinsic registration error is a major cause of performance loss. Rigid model and cadaver experiments confirmed that the scenic integration and display fluency of ARNES are smooth, as demonstrated by three clinical trials that surpassed practicality.
Collapse
Affiliation(s)
- Yakui Chu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Jian Yang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China.
| | - Shaodong Ma
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Danni Ai
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Wenjie Li
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| | - Hong Song
- School of Software, Beijing Institute of Technology, Beijing 100081, China
| | - Liang Li
- Department of Otolaryngology-Head and Neck Surgery, Chinese PLA General Hospital, Beijing 100853, China
| | - Duanduan Chen
- School of Life Science, Beijing Institute of Technology, Beijing 100081, China
| | - Lei Chen
- Department of Otolaryngology-Head and Neck Surgery, Chinese PLA General Hospital, Beijing 100853, China
| | - Yongtian Wang
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Electronics, Beijing Institute of Technology, Beijing 100081, China
| |
Collapse
|
40
|
Intraoperative Evaluation of Body Surface Improvement by an Augmented Reality System That a Clinician Can Modify. PLASTIC AND RECONSTRUCTIVE SURGERY-GLOBAL OPEN 2017; 5:e1432. [PMID: 28894655 PMCID: PMC5585428 DOI: 10.1097/gox.0000000000001432] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2017] [Accepted: 06/09/2017] [Indexed: 01/01/2023]
Abstract
BACKGROUND Augmented reality (AR) technology that can combine computer-generated images with a real scene has been reported in the medical field recently. We devised the AR system for evaluation of improvements of the body surface, which is important for plastic surgery. METHODS We constructed an AR system that is easy to modify by combining existing devices and free software. We superimposed the 3-dimensional images of the body surface and the bone (obtained from VECTRA H1 and CT) onto the actual surgical field by Moverio BT-200 smart glasses and evaluated improvements of the body surface in 8 cases. RESULTS In all cases, the 3D image was successfully projected on the surgical field. Improvement of the display method of the 3D image made it easier to distinguish the different shapes in the 3D image and surgical field, making comparison easier. In a patient with fibrous dysplasia, the symmetrized body surface image was useful for confirming improvement of the real body surface. In a patient with complex facial fracture, the simulated bone image was useful as a reference for reduction. In a patient with an osteoma of the forehead, simultaneously displayed images of the body surface and the bone made it easier to understand these positional relationships. CONCLUSIONS This study confirmed that AR technology is helpful for evaluation of the body surface in several clinical applications. Our findings are not only useful for body surface evaluation but also for effective utilization of AR technology in the field of plastic surgery.
Collapse
|
41
|
The status of augmented reality in laparoscopic surgery as of 2016. Med Image Anal 2017; 37:66-90. [DOI: 10.1016/j.media.2017.01.007] [Citation(s) in RCA: 183] [Impact Index Per Article: 26.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2016] [Revised: 01/16/2017] [Accepted: 01/23/2017] [Indexed: 12/27/2022]
|
42
|
Zhu M, Liu F, Chai G, Pan JJ, Jiang T, Lin L, Xin Y, Zhang Y, Li Q. A novel augmented reality system for displaying inferior alveolar nerve bundles in maxillofacial surgery. Sci Rep 2017; 7:42365. [PMID: 28198442 PMCID: PMC5309766 DOI: 10.1038/srep42365] [Citation(s) in RCA: 58] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2016] [Accepted: 01/09/2017] [Indexed: 11/23/2022] Open
Abstract
Augmented reality systems can combine virtual images with a real environment to ensure accurate surgery with lower risk. This study aimed to develop a novel registration and tracking technique to establish a navigation system based on augmented reality for maxillofacial surgery. Specifically, a virtual image is reconstructed from CT data using 3D software. The real environment is tracked by the augmented reality (AR) software. The novel registration strategy that we created uses an occlusal splint compounded with a fiducial marker (OSM) to establish a relationship between the virtual image and the real object. After the fiducial marker is recognized, the virtual image is superimposed onto the real environment, forming the “integrated image” on semi-transparent glass. Via the registration process, the integral image, which combines the virtual image with the real scene, is successfully presented on the semi-transparent helmet. The position error of this navigation system is 0.96 ± 0.51 mm. This augmented reality system was applied in the clinic and good surgical outcomes were obtained. The augmented reality system that we established for maxillofacial surgery has the advantages of easy manipulation and high accuracy, which can improve surgical outcomes. Thus, this system exhibits significant potential in clinical applications.
Collapse
Affiliation(s)
- Ming Zhu
- Shanghai Ninth People's Hospital affiliated to Shanghai Jiao Tong University, Shanghai, China
| | - Fei Liu
- Shanghai Ninth People's Hospital affiliated to Shanghai Jiao Tong University, Shanghai, China
| | - Gang Chai
- Shanghai Ninth People's Hospital affiliated to Shanghai Jiao Tong University, Shanghai, China
| | - Jun J Pan
- State Key Laboratory of Virtual Reality Technology and Systems Beihang University, Beijing, China
| | - Taoran Jiang
- Shanghai Ninth People's Hospital affiliated to Shanghai Jiao Tong University, Shanghai, China
| | - Li Lin
- Shanghai Ninth People's Hospital affiliated to Shanghai Jiao Tong University, Shanghai, China
| | - Yu Xin
- Shanghai Ninth People's Hospital affiliated to Shanghai Jiao Tong University, Shanghai, China
| | - Yan Zhang
- Shanghai Ninth People's Hospital affiliated to Shanghai Jiao Tong University, Shanghai, China
| | - Qingfeng Li
- Shanghai Ninth People's Hospital affiliated to Shanghai Jiao Tong University, Shanghai, China
| |
Collapse
|
43
|
Bond L, Schulz B, VanMeter T, Martin R. Intra-operative navigation of a 3-dimensional needle localization system for precision of irreversible electroporation needles in locally advanced pancreatic cancer. Eur J Surg Oncol 2017; 43:337-343. [DOI: 10.1016/j.ejso.2016.09.009] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2016] [Revised: 08/29/2016] [Accepted: 09/06/2016] [Indexed: 12/18/2022] Open
|
44
|
New simple image overlay system using a tablet PC for pinpoint identification of the appropriate site for anastomosis in peripheral arterial reconstruction. Surg Today 2016; 46:1387-1393. [PMID: 26988854 DOI: 10.1007/s00595-016-1326-4] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2015] [Accepted: 02/12/2016] [Indexed: 10/22/2022]
Abstract
PURPOSE To evaluate the accuracy and utility of a new image overlay system using a tablet PC for patients undergoing peripheral arterial reconstruction. METHODS Eleven limbs treated with distal bypass surgery were studied. Three-dimensional images obtained by processing a preoperative contrast-enhanced computed tomography scan were superimposed onto the back-camera images of a tablet PC. We used this system to pinpoint a planned distal anastomotic site preoperatively and to make a precise incision directly above it during surgery. We used a branch artery near the distal anastomotic site as a reference point and the accuracy of the system was validated by comparing its results with the intraoperative findings. The precision of the system was also compared with that of a preoperative ultrasonographic examination. RESULTS Both the image overlay system and ultrasonography (US) accurately identified the target branch artery in all except one limb. In that limb, which had a very small reference branch artery, preoperative US wrongly identified another branch, whereas the image overlay system located the target branch with an error of 10 mm. CONCLUSIONS Our image overlay system was easy to use and allowed us to precisely identify a target artery preoperatively. Therefore, this system could be helpful for pinpointing the most accurate incision site during surgery.
Collapse
|
45
|
Nitschke P, Bork U, Plodeck V, Podlesek D, Sobottka SB, Schackert G, Weitz J, Kirsch M. [Importance of preoperative and intraoperative imaging for operative strategies]. Chirurg 2016; 87:179-88. [PMID: 26939896 DOI: 10.1007/s00104-016-0163-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
Recent advances in preoperative and postoperative imaging have an increasing influence on surgical decision-making and make more complex surgical interventions possible. This improves the possibilities for frequently occurring challenges and promoting improved functional and oncological outcome. This manuscript reviews the role of preoperative and intraoperative imaging in surgery. Various techniques are explained based on examples from hepatobiliary surgery and neurosurgery, in particular real-time procedures, such as the online use of augmented reality and in vivo fluorescence, as well as new and promising optical techniques including imaging of intrinsic signals and vibrational spectroscopy.
Collapse
Affiliation(s)
- P Nitschke
- Klinik und Poliklinik für Viszeral-, Thorax- und Gefäßchirurgie, Carl Gustav Carus Universitätsklinikum Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
| | - U Bork
- Klinik und Poliklinik für Viszeral-, Thorax- und Gefäßchirurgie, Carl Gustav Carus Universitätsklinikum Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
| | - V Plodeck
- Institut für Radiologie, Carl Gustav Carus Universitätsklinikum Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
| | - D Podlesek
- Klinik und Poliklinik für Neurochirurgie und Experimental Neuroimaging Laboratory, Carl Gustav Carus Universitätsklinikum Dresden, Technische Universität Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
| | - S B Sobottka
- Klinik und Poliklinik für Neurochirurgie und Experimental Neuroimaging Laboratory, Carl Gustav Carus Universitätsklinikum Dresden, Technische Universität Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
| | - G Schackert
- Klinik und Poliklinik für Neurochirurgie und Experimental Neuroimaging Laboratory, Carl Gustav Carus Universitätsklinikum Dresden, Technische Universität Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
| | - J Weitz
- Klinik und Poliklinik für Viszeral-, Thorax- und Gefäßchirurgie, Carl Gustav Carus Universitätsklinikum Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland
| | - M Kirsch
- Klinik und Poliklinik für Neurochirurgie und Experimental Neuroimaging Laboratory, Carl Gustav Carus Universitätsklinikum Dresden, Technische Universität Dresden, Fetscherstr. 74, 01307, Dresden, Deutschland.
| |
Collapse
|
46
|
Augmented Reality for Assistance of Total Knee Replacement. JOURNAL OF ELECTRICAL AND COMPUTER ENGINEERING 2016. [DOI: 10.1155/2016/9358369] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
The aim of this work was the development of a surgical assistance system based on augmented reality to support joint replacement procedures and implantation of prosthetic knee. Images of the scene were captured in order to detect the visual markers located on the lateral surface of the patient’s leg for overlapping the 3D models of the prosthesis and the joint, as well as the tool used by the medical specialist. With the marker identification, it was possible to compute its position and orientation for locating the virtual models, obtaining a monitoring system for giving accurate information about the procedure. Also it can be used as training platform for surgeons, without having volunteers or patients for performing real surgeries; instead they can train in a virtual environment. The results have shown an efficient system in terms of cost-benefit relation, taking into account the materials used for developing the system; nevertheless, the accuracy of the algorithm decreases according to the distance between the markers.
Collapse
|
47
|
Mahmud N, Cohen J, Tsourides K, Berzin TM. Computer vision and augmented reality in gastrointestinal endoscopy. Gastroenterol Rep (Oxf) 2015; 3:179-84. [PMID: 26133175 PMCID: PMC4527270 DOI: 10.1093/gastro/gov027] [Citation(s) in RCA: 40] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/30/2015] [Accepted: 06/07/2015] [Indexed: 02/06/2023] Open
Abstract
Augmented reality (AR) is an environment-enhancing technology, widely applied in the computer sciences, which has only recently begun to permeate the medical field. Gastrointestinal endoscopy—which relies on the integration of high-definition video data with pathologic correlates—requires endoscopists to assimilate and process a tremendous amount of data in real time. We believe that AR is well positioned to provide computer-guided assistance with a wide variety of endoscopic applications, beginning with polyp detection. In this article, we review the principles of AR, describe its potential integration into an endoscopy set-up, and envisage a series of novel uses. With close collaboration between physicians and computer scientists, AR promises to contribute significant improvements to the field of endoscopy.
Collapse
Affiliation(s)
- Nadim Mahmud
- Department of Internal Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston MA, USA
| | - Jonah Cohen
- The Center for Advanced Endoscopy, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston MA, USA
| | - Kleovoulos Tsourides
- Brain and Cognitive Sciences, Massachusetts Institute of Technology, Boston MA, USA
| | - Tyler M Berzin
- The Center for Advanced Endoscopy, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston MA, USA
| |
Collapse
|
48
|
Murakami K, Naka S, Shiomi H, Akabori H, Kurumi Y, Morikawa S, Tani T. Initial experiences with MR Image-guided laparoscopic microwave coagulation therapy for hepatic tumors. Surg Today 2014; 45:1173-8. [DOI: 10.1007/s00595-014-1042-x] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2014] [Accepted: 09/15/2014] [Indexed: 02/06/2023]
|