1
|
Alam I, Garg K, Kumar AG, Raheja A, Shah H, Pandey K, Sharma R, Mishra S, Tandon V, Singh M, Ahmad FU, Suri A, Kale SS. Beyond Traditional Training: Exploring the Benefits of Virtual Reality Simulator in Lumbar Pedicle Screw Insertion - A Randomized Controlled Trial. World Neurosurg 2024; 189:e61-e68. [PMID: 38825310 DOI: 10.1016/j.wneu.2024.05.163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2024] [Revised: 05/26/2024] [Accepted: 05/26/2024] [Indexed: 06/04/2024]
Abstract
INTRODUCTION This study compares the effectiveness of virtual reality simulators (VRS) and a saw bone model for learning lumbar pedicle screw insertion (LPSI) in neurosurgery. METHODS A single-center, cross-sectional, randomized controlled laboratory investigation was conducted involving residents and fellows from a tertiary care referral hospital. Participants were divided into two groups (A and B). Group A performed 3 LPSI tasks: the first on a saw bone model, the second on VRS, and the third on another saw bone model. Group B completed 2 LPSI tasks: the first on a saw bone model and the second on another saw bone model. The accuracy of LPSI was evaluated through noncontrast computed tomography scans for the saw bone models, while the in-built application of VRS was utilized to check for accuracy of screw placement using the simulator. RESULTS The study included 38 participants (19 in each group). Group A participants showed reduced mean entry point error (0.11 mm, P 0.024), increased mean purchase length (4.66 cm, P 0.007), and no cortical breaches (P 0.031) when placing the second saw bone model screw. Similar improvements were observed among group A participants in PGY 1-3 while placing the second saw bone model screws. CONCLUSIONS Virtual reality simulators (VRS) prove to be an invaluable tool for teaching complex neurosurgical skills, such as LPSI, to trainees. This technology investment can enhance the learning curve while maintaining patient safety.
Collapse
Affiliation(s)
- Intekhab Alam
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Kanwaljeet Garg
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Akshay Ganesh Kumar
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Amol Raheja
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Het Shah
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Kushagra Pandey
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Ravi Sharma
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Shashwat Mishra
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Vivek Tandon
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India.
| | - Manmohan Singh
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Faiz U Ahmad
- Department of Neurosurgery, Emory University School of Medicine, Atlanta, Georgia
| | - Ashish Suri
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| | - Shashank Sharad Kale
- Department of Neurosurgery, All India Institute of Medical Sciences, New Delhi, India
| |
Collapse
|
2
|
Laterza V, Marchegiani F, Aisoni F, Ammendola M, Schena CA, Lavazza L, Ravaioli C, Carra MC, Costa V, De Franceschi A, De Simone B, de’Angelis N. Smart Operating Room in Digestive Surgery: A Narrative Review. Healthcare (Basel) 2024; 12:1530. [PMID: 39120233 PMCID: PMC11311806 DOI: 10.3390/healthcare12151530] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2024] [Revised: 07/24/2024] [Accepted: 07/29/2024] [Indexed: 08/10/2024] Open
Abstract
The introduction of new technologies in current digestive surgical practice is progressively reshaping the operating room, defining the fourth surgical revolution. The implementation of black boxes and control towers aims at streamlining workflow and reducing surgical error by early identification and analysis, while augmented reality and artificial intelligence augment surgeons' perceptual and technical skills by superimposing three-dimensional models to real-time surgical images. Moreover, the operating room architecture is transitioning toward an integrated digital environment to improve efficiency and, ultimately, patients' outcomes. This narrative review describes the most recent evidence regarding the role of these technologies in transforming the current digestive surgical practice, underlining their potential benefits and drawbacks in terms of efficiency and patients' outcomes, as an attempt to foresee the digestive surgical practice of tomorrow.
Collapse
Affiliation(s)
- Vito Laterza
- Department of Digestive Surgical Oncology and Liver Transplantation, University Hospital of Besançon, 3 Boulevard Alexandre Fleming, 25000 Besancon, France;
| | - Francesco Marchegiani
- Unit of Colorectal and Digestive Surgery, DIGEST Department, Beaujon University Hospital, AP-HP, University of Paris Cité, Clichy, 92110 Paris, France
| | - Filippo Aisoni
- Unit of Emergency Surgery, Department of Surgery, Ferrara University Hospital, 44124 Ferrara, Italy;
| | - Michele Ammendola
- Digestive Surgery Unit, Health of Science Department, University Hospital “R.Dulbecco”, 88100 Catanzaro, Italy;
| | - Carlo Alberto Schena
- Unit of Robotic and Minimally Invasive Surgery, Department of Surgery, Ferrara University Hospital, 44124 Ferrara, Italy; (C.A.S.); (N.d.)
| | - Luca Lavazza
- Hospital Network Coordinator of Azienda Ospedaliero, Universitaria and Azienda USL di Ferrara, 44121 Ferrara, Italy;
| | - Cinzia Ravaioli
- Azienda Ospedaliero, Universitaria di Ferrara, 44121 Ferrara, Italy;
| | - Maria Clotilde Carra
- Rothschild Hospital (AP-HP), 75012 Paris, France;
- INSERM-Sorbonne Paris Cité, Epidemiology and Statistics Research Centre, 75004 Paris, France
| | - Vittore Costa
- Unit of Orthopedics, Humanitas Hospital, 24125 Bergamo, Italy;
| | | | - Belinda De Simone
- Department of Emergency Surgery, Academic Hospital of Villeneuve St Georges, 91560 Villeneuve St. Georges, France;
| | - Nicola de’Angelis
- Unit of Robotic and Minimally Invasive Surgery, Department of Surgery, Ferrara University Hospital, 44124 Ferrara, Italy; (C.A.S.); (N.d.)
- Department of Translational Medicine, University of Ferrara, 44121 Ferrara, Italy
| |
Collapse
|
3
|
Pilz da Cunha G, van Delden OM, Kazemier G, Vahrmeijer AL, Bonjer HJ, Meijerink MR, Swijnenburg RJ. Hybrid operating room applications for precision hepatobiliary surgery: A narrative review. J Surg Oncol 2024; 129:1265-1273. [PMID: 38567691 DOI: 10.1002/jso.27634] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Accepted: 03/17/2024] [Indexed: 06/04/2024]
Abstract
This review summarizes the key applications of a hybrid operating room (HOR) in hepatobiliary surgery and explores the advantages, limitations, and future directions of its utilization. A comprehensive literature search was conducted in PubMed to identify articles reporting on the utilization of HORs in liver surgery. So far, the HOR has been limitedly applied in hepatobiliary surgery. It can offer an optimal environment for combining radiological and surgical interventions and for performing image-guided surgical navigation.
Collapse
Affiliation(s)
- Gabriela Pilz da Cunha
- Department of Surgery, Amsterdam UMC Location, University of Amsterdam, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Treatment and Quality of Life, Amsterdam, The Netherlands
| | - Otto M van Delden
- Department of Radiology, Amsterdam UMC Location Amsterdam Medical Center, Amsterdam, The Netherlands
| | - Geert Kazemier
- Cancer Center Amsterdam, Treatment and Quality of Life, Amsterdam, The Netherlands
- Department of Surgery, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Alexander L Vahrmeijer
- Department of Surgical Oncology, Leiden University Medical Center, Leiden, The Netherlands
| | - H Jaap Bonjer
- Cancer Center Amsterdam, Treatment and Quality of Life, Amsterdam, The Netherlands
- Department of Surgery, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Martijn R Meijerink
- Department of Radiology, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| | - Rutger-Jan Swijnenburg
- Cancer Center Amsterdam, Treatment and Quality of Life, Amsterdam, The Netherlands
- Department of Surgery, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
| |
Collapse
|
4
|
A 3D Image Registration Method for Laparoscopic Liver Surgery Navigation. ELECTRONICS 2022. [DOI: 10.3390/electronics11111670] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
Abstract
At present, laparoscopic augmented reality (AR) navigation has been applied to minimally invasive abdominal surgery, which can help doctors to see the location of blood vessels and tumors in organs, so as to perform precise surgery operations. Image registration is the process of optimally mapping one or more images to the target image, and it is also the core of laparoscopic AR navigation. The key is how to shorten the registration time and optimize the registration accuracy. We have studied the three-dimensional (3D) image registration technology in laparoscopic liver surgery navigation and proposed a new registration method combining rough registration and fine registration. First, the adaptive fireworks algorithm (AFWA) is applied to rough registration, and then the optimized iterative closest point (ICP) algorithm is applied to fine registration. We proposed a method that is validated by the computed tomography (CT) dataset 3D-IRCADb-01. Experimental results show that our method is superior to other registration methods based on stochastic optimization algorithms in terms of registration time and accuracy.
Collapse
|
5
|
Lee JJ, Klepcha M, Wong M, Dang PN, Sadrameli SS, Britz GW. The First Pilot Study of an Interactive, 360° Augmented Reality Visualization Platform for Neurosurgical Patient Education: A Case Series. Oper Neurosurg (Hagerstown) 2022; 23:53-59. [DOI: 10.1227/ons.0000000000000186] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2021] [Accepted: 01/09/2022] [Indexed: 11/19/2022] Open
|
6
|
Wendler T, van Leeuwen FWB, Navab N, van Oosterom MN. How molecular imaging will enable robotic precision surgery : The role of artificial intelligence, augmented reality, and navigation. Eur J Nucl Med Mol Imaging 2021; 48:4201-4224. [PMID: 34185136 PMCID: PMC8566413 DOI: 10.1007/s00259-021-05445-6] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2021] [Accepted: 06/01/2021] [Indexed: 02/08/2023]
Abstract
Molecular imaging is one of the pillars of precision surgery. Its applications range from early diagnostics to therapy planning, execution, and the accurate assessment of outcomes. In particular, molecular imaging solutions are in high demand in minimally invasive surgical strategies, such as the substantially increasing field of robotic surgery. This review aims at connecting the molecular imaging and nuclear medicine community to the rapidly expanding armory of surgical medical devices. Such devices entail technologies ranging from artificial intelligence and computer-aided visualization technologies (software) to innovative molecular imaging modalities and surgical navigation (hardware). We discuss technologies based on their role at different steps of the surgical workflow, i.e., from surgical decision and planning, over to target localization and excision guidance, all the way to (back table) surgical verification. This provides a glimpse of how innovations from the technology fields can realize an exciting future for the molecular imaging and surgery communities.
Collapse
Affiliation(s)
- Thomas Wendler
- Chair for Computer Aided Medical Procedures and Augmented Reality, Technische Universität München, Boltzmannstr. 3, 85748 Garching bei München, Germany
| | - Fijs W. B. van Leeuwen
- Department of Radiology, Interventional Molecular Imaging Laboratory, Leiden University Medical Center, Leiden, The Netherlands
- Department of Urology, The Netherlands Cancer Institute - Antonie van Leeuwenhoek Hospital, Amsterdam, The Netherlands
- Orsi Academy, Melle, Belgium
| | - Nassir Navab
- Chair for Computer Aided Medical Procedures and Augmented Reality, Technische Universität München, Boltzmannstr. 3, 85748 Garching bei München, Germany
- Chair for Computer Aided Medical Procedures Laboratory for Computational Sensing + Robotics, Johns-Hopkins University, Baltimore, MD USA
| | - Matthias N. van Oosterom
- Department of Radiology, Interventional Molecular Imaging Laboratory, Leiden University Medical Center, Leiden, The Netherlands
- Department of Urology, The Netherlands Cancer Institute - Antonie van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| |
Collapse
|
7
|
Augmented and virtual reality in spine surgery, current applications and future potentials. Spine J 2021; 21:1617-1625. [PMID: 33774210 DOI: 10.1016/j.spinee.2021.03.018] [Citation(s) in RCA: 67] [Impact Index Per Article: 22.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Accepted: 03/17/2021] [Indexed: 02/03/2023]
Abstract
BACKGROUND CONTEXT The field of artificial intelligence (AI) is rapidly advancing, especially with recent improvements in deep learning (DL) techniques. Augmented (AR) and virtual reality (VR) are finding their place in healthcare, and spine surgery is no exception. The unique capabilities and advantages of AR and VR devices include their low cost, flexible integration with other technologies, user-friendly features and their application in navigation systems, which makes them beneficial across different aspects of spine surgery. Despite the use of AR for pedicle screw placement, targeted cervical foraminotomy, bone biopsy, osteotomy planning, and percutaneous intervention, the current applications of AR and VR in spine surgery remain limited. PURPOSE The primary goal of this study was to provide the spine surgeons and clinical researchers with the general information about the current applications, future potentials, and accessibility of AR and VR systems in spine surgery. STUDY DESIGN/SETTING We reviewed titles of more than 250 journal papers from google scholar and PubMed with search words: augmented reality, virtual reality, spine surgery, and orthopaedic, out of which 89 related papers were selected for abstract review. Finally, full text of 67 papers were analyzed and reviewed. METHODS The papers were divided into four groups: technological papers, applications in surgery, applications in spine education and training, and general application in orthopaedic. A team of two reviewers performed paper reviews and a thorough web search to ensure the most updated state of the art in each of four group is captured in the review. RESULTS In this review we discuss the current state of the art in AR and VR hardware, their preoperative applications and surgical applications in spine surgery. Finally, we discuss the future potentials of AR and VR and their integration with AI, robotic surgery, gaming, and wearables. CONCLUSIONS AR and VR are promising technologies that will soon become part of standard of care in spine surgery.
Collapse
|
8
|
Schneider C, Allam M, Stoyanov D, Hawkes DJ, Gurusamy K, Davidson BR. Performance of image guided navigation in laparoscopic liver surgery - A systematic review. Surg Oncol 2021; 38:101637. [PMID: 34358880 DOI: 10.1016/j.suronc.2021.101637] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2021] [Revised: 07/04/2021] [Accepted: 07/24/2021] [Indexed: 02/07/2023]
Abstract
BACKGROUND Compared to open surgery, minimally invasive liver resection has improved short term outcomes. It is however technically more challenging. Navigated image guidance systems (IGS) are being developed to overcome these challenges. The aim of this systematic review is to provide an overview of their current capabilities and limitations. METHODS Medline, Embase and Cochrane databases were searched using free text terms and corresponding controlled vocabulary. Titles and abstracts of retrieved articles were screened for inclusion criteria. Due to the heterogeneity of the retrieved data it was not possible to conduct a meta-analysis. Therefore results are presented in tabulated and narrative format. RESULTS Out of 2015 articles, 17 pre-clinical and 33 clinical papers met inclusion criteria. Data from 24 articles that reported on accuracy indicates that in recent years navigation accuracy has been in the range of 8-15 mm. Due to discrepancies in evaluation methods it is difficult to compare accuracy metrics between different systems. Surgeon feedback suggests that current state of the art IGS may be useful as a supplementary navigation tool, especially in small liver lesions that are difficult to locate. They are however not able to reliably localise all relevant anatomical structures. Only one article investigated IGS impact on clinical outcomes. CONCLUSIONS Further improvements in navigation accuracy are needed to enable reliable visualisation of tumour margins with the precision required for oncological resections. To enhance comparability between different IGS it is crucial to find a consensus on the assessment of navigation accuracy as a minimum reporting standard.
Collapse
Affiliation(s)
- C Schneider
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK.
| | - M Allam
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK; General surgery Department, Tanta University, Egypt
| | - D Stoyanov
- Department of Computer Science, University College London, London, UK; Centre for Medical Image Computing (CMIC), University College London, London, UK
| | - D J Hawkes
- Centre for Medical Image Computing (CMIC), University College London, London, UK; Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK
| | - K Gurusamy
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK
| | - B R Davidson
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK
| |
Collapse
|
9
|
Müller PC, Haslebacher C, Steinemann DC, Müller-Stich BP, Hackert T, Peterhans M, Eigl B. Image-guided minimally invasive endopancreatic surgery using a computer-assisted navigation system. Surg Endosc 2020; 35:1610-1617. [PMID: 32253555 DOI: 10.1007/s00464-020-07540-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2020] [Accepted: 03/31/2020] [Indexed: 12/16/2022]
Abstract
BACKGROUND Minimally invasive endopancreatic surgery (EPS), performing a pancreatic resection from inside the pancreatic duct, has been proposed as an experimental alternative to duodenum-preserving pancreatic head resection in benign diseases such as chronic pancreatitis, but is complicated by difficult spatial orientation when trying to reach structures of interest. This study assessed the feasibility and potential benefits of image-guided EPS using a computer-assisted navigation system in artificial pancreas silicon model. METHODS A surgical navigation system displayed a 3D reconstruction of the original computed tomography (CT) scan and the endoscope in relation to a selected target structure. In a first step, different surface landmark (LM)-based and intraparenchymal LM-based approaches for image-to-physical space registration were evaluated. The accuracy of registration was measured as fiducial registration error (FRE). Subsequently, intrapancreatic lesions (n = 8) that were visible on preoperative imaging, but not on the endoscopic view, were targeted with a computer-assisted, image-guided endopancreatic resection technique in pancreas silicon models. After each experiment, a CT scan was obtained for measurement of the shortest distance from the resection cavity to the centre of the lesion. RESULTS Intraparenchymal LM registration [FRE 2.24 mm (1.40-2.85)] was more accurate than surface LM registration [FRE 3.46 mm (2.25-4.85); p = 0.035], but not more accurate than combined registration of intraparenchymal and surface LM [FRE 2.46 mm (1.60-3.35); p = 0.052]. Using image-guided EPS, six of seven lesions were successfully targeted. The median distance from the resection cavity to the centre of the lesion on CT was 1.52 mm (0-2.4). In one pancreas, a lesion could not be resected due to the fragility of the pancreas model. CONCLUSION Image-guided minimally invasive EPS using a computer-assisted navigation system enabled successful targeting of pancreatic lesions that were invisible on the endoscopic image, but detectable on preoperative imaging. In the clinical setting, this tool could facilitate complex minimally invasive and robotic pancreatic procedures.
Collapse
Affiliation(s)
- Philip C Müller
- Department of Visceral and Transplant Surgery, University Hospital Zurich, Rämistrasse 100, 8091, Zurich, Switzerland.
| | | | - Daniel C Steinemann
- Department of Surgery, Clarunis, University Hospital Basel, Basel, Switzerland
| | - Beat P Müller-Stich
- Department of General, Visceral and Transplantation Surgery, University Hospital Heidelberg, Heidelberg, Germany
| | - Thilo Hackert
- Department of General, Visceral and Transplantation Surgery, University Hospital Heidelberg, Heidelberg, Germany
| | | | - Benjamin Eigl
- CAScination AG, Bern, Switzerland.,ARTORG Center for Computer-Aided Surgery, University of Bern, Bern, Switzerland
| |
Collapse
|
10
|
Quero G, Lapergola A, Soler L, Shahbaz M, Hostettler A, Collins T, Marescaux J, Mutter D, Diana M, Pessaux P. Virtual and Augmented Reality in Oncologic Liver Surgery. Surg Oncol Clin N Am 2018; 28:31-44. [PMID: 30414680 DOI: 10.1016/j.soc.2018.08.002] [Citation(s) in RCA: 50] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
Virtual reality (VR) and augmented reality (AR) in complex surgery are evolving technologies enabling improved preoperative planning and intraoperative navigation. The basis of these technologies is a computer-based generation of a patient-specific 3-dimensional model from Digital Imaging and Communications in Medicine (DICOM) data. This article provides a state-of-the- art overview on the clinical use of this technology with a specific focus on hepatic surgery. Although VR and AR are still in an evolving stage with only some clinical application today, these technologies have the potential to become a key factor in improving preoperative and intraoperative decision making.
Collapse
Affiliation(s)
- Giuseppe Quero
- IHU-Strasbourg, Institute of Image-Guided Surgery, 1 Place de l'Hôpital, Strasbourg 67091, France
| | - Alfonso Lapergola
- IRCAD, Research Institute Against Cancer of the Digestive System, 1 Place de l'Hôpital, Strasbourg 67091, France
| | - Luc Soler
- IRCAD, Research Institute Against Cancer of the Digestive System, 1 Place de l'Hôpital, Strasbourg 67091, France
| | - Muhammad Shahbaz
- IHU-Strasbourg, Institute of Image-Guided Surgery, 1 Place de l'Hôpital, Strasbourg 67091, France
| | - Alexandre Hostettler
- IRCAD, Research Institute Against Cancer of the Digestive System, 1 Place de l'Hôpital, Strasbourg 67091, France
| | - Toby Collins
- IRCAD, Research Institute Against Cancer of the Digestive System, 1 Place de l'Hôpital, Strasbourg 67091, France
| | - Jacques Marescaux
- IHU-Strasbourg, Institute of Image-Guided Surgery, 1 Place de l'Hôpital, Strasbourg 67091, France; IRCAD, Research Institute Against Cancer of the Digestive System, 1 Place de l'Hôpital, Strasbourg 67091, France
| | - Didier Mutter
- Department of General, Digestive and Endocrine Surgery, University Hospital of Strasbourg, 1 Place de l'Hôpital, Strasbourg 67091, France
| | - Michele Diana
- IHU-Strasbourg, Institute of Image-Guided Surgery, 1 Place de l'Hôpital, Strasbourg 67091, France; IRCAD, Research Institute Against Cancer of the Digestive System, 1 Place de l'Hôpital, Strasbourg 67091, France; Department of General, Digestive and Endocrine Surgery, University Hospital of Strasbourg, 1 Place de l'Hôpital, Strasbourg 67091, France
| | - Patrick Pessaux
- IHU-Strasbourg, Institute of Image-Guided Surgery, 1 Place de l'Hôpital, Strasbourg 67091, France; IRCAD, Research Institute Against Cancer of the Digestive System, 1 Place de l'Hôpital, Strasbourg 67091, France; Department of General, Digestive and Endocrine Surgery, University Hospital of Strasbourg, 1 Place de l'Hôpital, Strasbourg 67091, France.
| |
Collapse
|
11
|
A Novel Augmented Reality-Based Navigation System in Perforator Flap Transplantation - A Feasibility Study. Ann Plast Surg 2018; 79:192-196. [PMID: 28509695 DOI: 10.1097/sap.0000000000001078] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
BACKGROUND In perforator flap transplantation, dissection of the perforator is an important but difficult procedure because of the high variability in vascular anatomy. Preoperative imaging techniques could provide substantial information about vascular anatomy; however, it cannot provide direct guidance for surgeons during the operation. In this study, a navigation system (NS) was established to overlie a vascular map on surgical sites to further provide a direct guide for perforator flap transplantation. METHODS The NS was established based on computed tomographic angiography and augmented reality techniques. A virtual vascular map was reconstructed according to computed tomographic angiography data and projected onto real patient images using ARToolKit software. Additionally, a screw-fixation marker holder was created to facilitate registration. With the use of a tracking and display system, we conducted the NS on an animal model and measured the system error on a rapid prototyping model. RESULTS The NS assistance allowed for correct identification, as well as a safe and precise dissection of the perforator. The mean value of the system error was determined to be 3.474 ± 1.546 mm. CONCLUSIONS Augmented reality-based NS can provide precise navigation information by directly displaying a 3-dimensional individual anatomical virtual model onto the operative field in real time. It will allow rapid identification and safe dissection of a perforator in free flap transplantation surgery.
Collapse
|
12
|
Kenngott HG, Preukschas AA, Wagner M, Nickel F, Müller M, Bellemann N, Stock C, Fangerau M, Radeleff B, Kauczor HU, Meinzer HP, Maier-Hein L, Müller-Stich BP. Mobile, real-time, and point-of-care augmented reality is robust, accurate, and feasible: a prospective pilot study. Surg Endosc 2018; 32:2958-2967. [PMID: 29602988 DOI: 10.1007/s00464-018-6151-y] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2017] [Accepted: 03/21/2018] [Indexed: 11/28/2022]
Abstract
BACKGROUND Augmented reality (AR) systems are currently being explored by a broad spectrum of industries, mainly for improving point-of-care access to data and images. Especially in surgery and especially for timely decisions in emergency cases, a fast and comprehensive access to images at the patient bedside is mandatory. Currently, imaging data are accessed at a distance from the patient both in time and space, i.e., at a specific workstation. Mobile technology and 3-dimensional (3D) visualization of radiological imaging data promise to overcome these restrictions by making bedside AR feasible. METHODS In this project, AR was realized in a surgical setting by fusing a 3D-representation of structures of interest with live camera images on a tablet computer using marker-based registration. The intent of this study was to focus on a thorough evaluation of AR. Feasibility, robustness, and accuracy were thus evaluated consecutively in a phantom model and a porcine model. Additionally feasibility was evaluated in one male volunteer. RESULTS In the phantom model (n = 10), AR visualization was feasible in 84% of the visualization space with high accuracy (mean reprojection error ± standard deviation (SD): 2.8 ± 2.7 mm; 95th percentile = 6.7 mm). In a porcine model (n = 5), AR visualization was feasible in 79% with high accuracy (mean reprojection error ± SD: 3.5 ± 3.0 mm; 95th percentile = 9.5 mm). Furthermore, AR was successfully used and proved feasible within a male volunteer. CONCLUSIONS Mobile, real-time, and point-of-care AR for clinical purposes proved feasible, robust, and accurate in the phantom, animal, and single-trial human model shown in this study. Consequently, AR following similar implementation proved robust and accurate enough to be evaluated in clinical trials assessing accuracy, robustness in clinical reality, as well as integration into the clinical workflow. If these further studies prove successful, AR might revolutionize data access at patient bedside.
Collapse
Affiliation(s)
- Hannes Götz Kenngott
- Department of General, Visceral and Transplantation Surgery, Heidelberg University, Im Neuenheimer Feld 110, 69120, Heidelberg, Germany
| | - Anas Amin Preukschas
- Department of General, Visceral and Transplantation Surgery, Heidelberg University, Im Neuenheimer Feld 110, 69120, Heidelberg, Germany
| | - Martin Wagner
- Department of General, Visceral and Transplantation Surgery, Heidelberg University, Im Neuenheimer Feld 110, 69120, Heidelberg, Germany
| | - Felix Nickel
- Department of General, Visceral and Transplantation Surgery, Heidelberg University, Im Neuenheimer Feld 110, 69120, Heidelberg, Germany
| | - Michael Müller
- Division of Medical and Biological Informatics, German Cancer Research Center, Heidelberg, Germany
| | - Nadine Bellemann
- Department of Diagnostic and Interventional Radiology, Heidelberg University, Heidelberg, Germany
| | - Christian Stock
- Institute for Medical Biometry and Informatics, Heidelberg University, Heidelberg, Germany
| | - Markus Fangerau
- Department of Diagnostic and Interventional Radiology, Heidelberg University, Heidelberg, Germany
| | - Boris Radeleff
- Department of Diagnostic and Interventional Radiology, Heidelberg University, Heidelberg, Germany
| | - Hans-Ulrich Kauczor
- Department of Diagnostic and Interventional Radiology, Heidelberg University, Heidelberg, Germany
| | - Hans-Peter Meinzer
- Division of Medical and Biological Informatics, German Cancer Research Center, Heidelberg, Germany
| | - Lena Maier-Hein
- Division of Medical and Biological Informatics, German Cancer Research Center, Heidelberg, Germany
| | - Beat Peter Müller-Stich
- Department of General, Visceral and Transplantation Surgery, Heidelberg University, Im Neuenheimer Feld 110, 69120, Heidelberg, Germany.
| |
Collapse
|
13
|
Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: A review of current methods. Hepatobiliary Pancreat Dis Int 2018; 17:101-112. [PMID: 29567047 DOI: 10.1016/j.hbpd.2018.02.002] [Citation(s) in RCA: 61] [Impact Index Per Article: 10.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/31/2017] [Accepted: 11/16/2017] [Indexed: 02/05/2023]
Abstract
BACKGROUND Augmented reality (AR) technology is used to reconstruct three-dimensional (3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes. DATA SOURCES The keywords "augmented reality", "liver", "laparoscopic" and "hepatectomy" were used for searching publications in the PubMed database. The primary source of literatures was from peer-reviewed journals up to December 2016. Additional articles were identified by manual search of references found in the key articles. RESULTS In general, AR technology mainly includes 3D reconstruction, display, registration as well as tracking techniques and has recently been adopted gradually for liver surgeries including laparoscopy and laparotomy with video-based AR assisted laparoscopic resection as the main technical application. By applying AR technology, blood vessels and tumor structures in the liver can be displayed during surgery, which permits precise navigation during complex surgical procedures. Liver transformation and registration errors during surgery were the main factors that limit the application of AR technology. CONCLUSIONS With recent advances, AR technologies have the potential to improve hepatobiliary surgical procedures. However, additional clinical studies will be required to evaluate AR as a tool for reducing postoperative morbidity and mortality and for the improvement of long-term clinical outcomes. Future research is needed in the fusion of multiple imaging modalities, improving biomechanical liver modeling, and enhancing image data processing and tracking technologies to increase the accuracy of current AR methods.
Collapse
|
14
|
Kim D, Kim N, Lee S, Seo JB. A fast and robust level set motion-assisted deformable registration method for volumetric CT guided lung intervention. Biocybern Biomed Eng 2018. [DOI: 10.1016/j.bbe.2018.04.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
|
15
|
Peterlík I, Courtecuisse H, Rohling R, Abolmaesumi P, Nguan C, Cotin S, Salcudean S. Fast elastic registration of soft tissues under large deformations. Med Image Anal 2017; 45:24-40. [PMID: 29414434 DOI: 10.1016/j.media.2017.12.006] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2016] [Revised: 12/07/2017] [Accepted: 12/07/2017] [Indexed: 12/21/2022]
Abstract
A fast and accurate fusion of intra-operative images with a pre-operative data is a key component of computer-aided interventions which aim at improving the outcomes of the intervention while reducing the patient's discomfort. In this paper, we focus on the problematic of the intra-operative navigation during abdominal surgery, which requires an accurate registration of tissues undergoing large deformations. Such a scenario occurs in the case of partial hepatectomy: to facilitate the access to the pathology, e.g. a tumor located in the posterior part of the right lobe, the surgery is performed on a patient in lateral position. Due to the change in patient's position, the resection plan based on the pre-operative CT scan acquired in the supine position must be updated to account for the deformations. We suppose that an imaging modality, such as the cone-beam CT, provides the information about the intra-operative shape of an organ, however, due to the reduced radiation dose and contrast, the actual locations of the internal structures necessary to update the planning are not available. To this end, we propose a method allowing for fast registration of the pre-operative data represented by a detailed 3D model of the liver and its internal structure and the actual configuration given by the organ surface extracted from the intra-operative image. The algorithm behind the method combines the iterative closest point technique with a biomechanical model based on a co-rotational formulation of linear elasticity which accounts for large deformations of the tissue. The performance, robustness and accuracy of the method is quantitatively assessed on a control semi-synthetic dataset with known ground truth and a real dataset composed of nine pairs of abdominal CT scans acquired in supine and flank positions. It is shown that the proposed surface-matching method is capable of reducing the target registration error evaluated of the internal structures of the organ from more than 40 mm to less then 10 mm. Moreover, the control data is used to demonstrate the compatibility of the method with intra-operative clinical scenario, while the real datasets are utilized to study the impact of parametrization on the accuracy of the method. The method is also compared to a state-of-the art intensity-based registration technique in terms of accuracy and performance.
Collapse
Affiliation(s)
- Igor Peterlík
- MIMESIS, Inria Nancy, France; ICube, University of Strasbourg, CNRS, Strasbourg, France; Institute of Computer Science, Masaryk University, Brno, Czech Republic.
| | - Hadrien Courtecuisse
- ICube, University of Strasbourg, CNRS, Strasbourg, France; MIMESIS, Inria Nancy, France
| | - Robert Rohling
- Department of Electrical Engineering, University of British Columbia, Vancouver, BC, Canada
| | - Purang Abolmaesumi
- Department of Electrical Engineering, University of British Columbia, Vancouver, BC, Canada
| | - Christopher Nguan
- Urology Department, Vancouver General Hospital, Vancouver, BC, Canada
| | - Stéphane Cotin
- MIMESIS, Inria Nancy, France; ICube, University of Strasbourg, CNRS, Strasbourg, France
| | - Septimiu Salcudean
- Department of Electrical Engineering, University of British Columbia, Vancouver, BC, Canada
| |
Collapse
|
16
|
Tang R, Ma L, Xiang C, Wang X, Li A, Liao H, Dong J. Augmented reality navigation in open surgery for hilar cholangiocarcinoma resection with hemihepatectomy using video-based in situ three-dimensional anatomical modeling: A case report. Medicine (Baltimore) 2017; 96:e8083. [PMID: 28906410 PMCID: PMC5604679 DOI: 10.1097/md.0000000000008083] [Citation(s) in RCA: 24] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/22/2022] Open
Abstract
RATIONALE Patients who undergo hilar cholangiocarcinoma (HCAC) resection with concomitant hepatectomy have a high risk of postoperative morbidity and mortality due to surgical trauma to the hepatic and biliary vasculature. PATIENT CONCERNS A 58-year-old Chinese man with yellowing skin and sclera, abdominal distension, pruritus, and anorexia for approximately 3 weeks. DIAGNOSES Magnetic resonance cholangiopancreatography and enhanced computed tomography (CT) scanning revealed a mass over the biliary tree at the porta hepatis, which diagnosed to be s a hilar cholangiocarcinoma. INTERVENTION Three-dimensional (3D) images of the patient's hepatic and biliary structures were reconstructed preoperatively from CT data, and the 3D images were used for preoperative planning and augmented reality (AR)-assisted intraoperative navigation during open HCAC resection with hemihepatectomy. A 3D-printed model of the patient's biliary structures was also used intraoperatively as a visual reference. OUTCOMES No serious postoperative complications occurred, and the patient was tumor-free at the 9-month follow-up examination based on CT results. LESSONS AR-assisted preoperative planning and intraoperative navigation might be beneficial in other patients with HCAC patients to reduce postoperative complications and ensure disease-free survival. In our postoperative analysis, we also found that, when the3D images were superimposed 3D-printed model using a see-through integral video graphy display device, our senses of depth perception and motion parallax were improved, compared with that which we had experienced intraoperatively using the videobased AR display system.
Collapse
Affiliation(s)
- Rui Tang
- Department of Hepatopancreatobiliary Surgery, Beijing Tsinghua Changgung Hospital
| | - Longfei Ma
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China
| | - Canhong Xiang
- Department of Hepatopancreatobiliary Surgery, Beijing Tsinghua Changgung Hospital
| | - Xuedong Wang
- Department of Hepatopancreatobiliary Surgery, Beijing Tsinghua Changgung Hospital
| | - Ang Li
- Department of Hepatopancreatobiliary Surgery, Beijing Tsinghua Changgung Hospital
| | - Hongen Liao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China
| | - Jiahong Dong
- Department of Hepatopancreatobiliary Surgery, Beijing Tsinghua Changgung Hospital
| |
Collapse
|
17
|
Recent Development of Augmented Reality in Surgery: A Review. JOURNAL OF HEALTHCARE ENGINEERING 2017; 2017:4574172. [PMID: 29065604 PMCID: PMC5585624 DOI: 10.1155/2017/4574172] [Citation(s) in RCA: 138] [Impact Index Per Article: 19.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/12/2017] [Accepted: 07/03/2017] [Indexed: 12/11/2022]
Abstract
Introduction The development augmented reality devices allow physicians to incorporate data visualization into diagnostic and treatment procedures to improve work efficiency, safety, and cost and to enhance surgical training. However, the awareness of possibilities of augmented reality is generally low. This review evaluates whether augmented reality can presently improve the results of surgical procedures. Methods We performed a review of available literature dating from 2010 to November 2016 by searching PubMed and Scopus using the terms “augmented reality” and “surgery.” Results. The initial search yielded 808 studies. After removing duplicates and including only journal articles, a total of 417 studies were identified. By reading of abstracts, 91 relevant studies were chosen to be included. 11 references were gathered by cross-referencing. A total of 102 studies were included in this review. Conclusions The present literature suggest an increasing interest of surgeons regarding employing augmented reality into surgery leading to improved safety and efficacy of surgical procedures. Many studies showed that the performance of newly devised augmented reality systems is comparable to traditional techniques. However, several problems need to be addressed before augmented reality is implemented into the routine practice.
Collapse
|
18
|
Liu X, Rice CE, Shekhar R. Fast calibration of electromagnetically tracked oblique-viewing rigid endoscopes. Int J Comput Assist Radiol Surg 2017. [PMID: 28623479 DOI: 10.1007/s11548-017-1623-4] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
PURPOSE The oblique-viewing (i.e., angled) rigid endoscope is a commonly used tool in conventional endoscopic surgeries. The relative rotation between its two moveable parts, the telescope and the camera head, creates a rotation offset between the actual and the projection of an object in the camera image. A calibration method tailored to compensate such offset is needed. METHODS We developed a fast calibration method for oblique-viewing rigid endoscopes suitable for clinical use. In contrast to prior approaches based on optical tracking, we used electromagnetic (EM) tracking as the external tracking hardware to improve compactness and practicality. Two EM sensors were mounted on the telescope and the camera head, respectively, with considerations to minimize EM tracking errors. Single-image calibration was incorporated into the method, and a sterilizable plate, laser-marked with the calibration pattern, was also developed. Furthermore, we proposed a general algorithm to estimate the rotation center in the camera image. Formulas for updating the camera matrix in terms of clockwise and counterclockwise rotations were also developed. RESULTS The proposed calibration method was validated using a conventional [Formula: see text], 5-mm laparoscope. Freehand calibrations were performed using the proposed method, and the calibration time averaged 2 min and 8 s. The calibration accuracy was evaluated in a simulated clinical setting with several surgical tools present in the magnetic field of EM tracking. The root-mean-square re-projection error averaged 4.9 pixel (range 2.4-8.5 pixel, with image resolution of [Formula: see text] for rotation angles ranged from [Formula: see text] to [Formula: see text]. CONCLUSIONS We developed a method for fast and accurate calibration of oblique-viewing rigid endoscopes. The method was also designed to be performed in the operating room and will therefore support clinical translation of many emerging endoscopic computer-assisted surgical systems.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC, 20010, USA
| | - Christina E Rice
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC, 20010, USA.,Department of Mechanical and Aerospace Engineering, Princeton University, Princeton, NJ, 08544, USA
| | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC, 20010, USA.
| |
Collapse
|
19
|
The status of augmented reality in laparoscopic surgery as of 2016. Med Image Anal 2017; 37:66-90. [DOI: 10.1016/j.media.2017.01.007] [Citation(s) in RCA: 183] [Impact Index Per Article: 26.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2016] [Revised: 01/16/2017] [Accepted: 01/23/2017] [Indexed: 12/27/2022]
|
20
|
Robust augmented reality registration method for localization of solid organs' tumors using CT-derived virtual biomechanical model and fluorescent fiducials. Surg Endosc 2016; 31:2863-2871. [PMID: 27796600 DOI: 10.1007/s00464-016-5297-8] [Citation(s) in RCA: 32] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2016] [Accepted: 10/14/2016] [Indexed: 12/11/2022]
Abstract
BACKGROUND Augmented reality (AR) is the fusion of computer-generated and real-time images. AR can be used in surgery as a navigation tool, by creating a patient-specific virtual model through 3D software manipulation of DICOM imaging (e.g., CT scan). The virtual model can be superimposed to real-time images enabling transparency visualization of internal anatomy and accurate localization of tumors. However, the 3D model is rigid and does not take into account inner structures' deformations. We present a concept of automated AR registration, while the organs undergo deformation during surgical manipulation, based on finite element modeling (FEM) coupled with optical imaging of fluorescent surface fiducials. METHODS Two 10 × 1 mm wires (pseudo-tumors) and six 10 × 0.9 mm fluorescent fiducials were placed in ex vivo porcine kidneys (n = 10). Biomechanical FEM-based models were generated from CT scan. Kidneys were deformed and the shape changes were identified by tracking the fiducials, using a near-infrared optical system. The changes were registered automatically with the virtual model, which was deformed accordingly. Accuracy of prediction of pseudo-tumors' location was evaluated with a CT scan in the deformed status (ground truth). In vivo: fluorescent fiducials were inserted under ultrasound guidance in the kidney of one pig, followed by a CT scan. The FEM-based virtual model was superimposed on laparoscopic images by automatic registration of the fiducials. RESULTS Biomechanical models were successfully generated and accurately superimposed on optical images. The mean measured distance between the estimated tumor by biomechanical propagation and the scanned tumor (ground truth) was 0.84 ± 0.42 mm. All fiducials were successfully placed in in vivo kidney and well visualized in near-infrared mode enabling accurate automatic registration of the virtual model on the laparoscopic images. CONCLUSIONS Our preliminary experiments showed the potential of a biomechanical model with fluorescent fiducials to propagate the deformation of solid organs' surface to their inner structures including tumors with good accuracy and automatized robust tracking.
Collapse
|
21
|
Liu X, Kang S, Plishker W, Zaki G, Kane TD, Shekhar R. Laparoscopic stereoscopic augmented reality: toward a clinically viable electromagnetic tracking solution. J Med Imaging (Bellingham) 2016; 3:045001. [PMID: 27752522 DOI: 10.1117/1.jmi.3.4.045001] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2016] [Accepted: 09/08/2016] [Indexed: 11/14/2022] Open
Abstract
The purpose of this work was to develop a clinically viable laparoscopic augmented reality (AR) system employing stereoscopic (3-D) vision, laparoscopic ultrasound (LUS), and electromagnetic (EM) tracking to achieve image registration. We investigated clinically feasible solutions to mount the EM sensors on the 3-D laparoscope and the LUS probe. This led to a solution of integrating an externally attached EM sensor near the imaging tip of the LUS probe, only slightly increasing the overall diameter of the probe. Likewise, a solution for mounting an EM sensor on the handle of the 3-D laparoscope was proposed. The spatial image-to-video registration accuracy of the AR system was measured to be [Formula: see text] and [Formula: see text] for the left- and right-eye channels, respectively. The AR system contributed 58-ms latency to stereoscopic visualization. We further performed an animal experiment to demonstrate the use of the system as a visualization approach for laparoscopic procedures. In conclusion, we have developed an integrated, compact, and EM tracking-based stereoscopic AR visualization system, which has the potential for clinical use. The system has been demonstrated to achieve clinically acceptable accuracy and latency. This work is a critical step toward clinical translation of AR visualization for laparoscopic procedures.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation , Children's National Health System, 111 Michigan Avenue NW, Washington, DC 20010, United States
| | - Sukryool Kang
- Sheikh Zayed Institute for Pediatric Surgical Innovation , Children's National Health System, 111 Michigan Avenue NW, Washington, DC 20010, United States
| | - William Plishker
- IGI Technologies, Inc. , 387 Technology Drive #3110D, College Park, Maryland 20742, United States
| | - George Zaki
- IGI Technologies, Inc. , 387 Technology Drive #3110D, College Park, Maryland 20742, United States
| | - Timothy D Kane
- Sheikh Zayed Institute for Pediatric Surgical Innovation , Children's National Health System, 111 Michigan Avenue NW, Washington, DC 20010, United States
| | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC 20010, United States; IGI Technologies, Inc., 387 Technology Drive #3110D, College Park, Maryland 20742, United States
| |
Collapse
|
22
|
Simpfendörfer T, Gasch C, Hatiboglu G, Müller M, Maier-Hein L, Hohenfellner M, Teber D. Intraoperative Computed Tomography Imaging for Navigated Laparoscopic Renal Surgery: First Clinical Experience. J Endourol 2016; 30:1105-1111. [PMID: 27530774 DOI: 10.1089/end.2016.0385] [Citation(s) in RCA: 25] [Impact Index Per Article: 3.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/20/2023] Open
Abstract
INTRODUCTION Laparoscopic partial nephrectomy (LPN) remains challenging in endophytic and complex kidney tumors as the clear understanding of tumor location and spreading depends on a precise analysis of available imaging. The purpose of this study was to investigate navigated kidney surgery using intraoperative cone-beam computed tomography (CBCT) images in conjunction with a previously proposed method for augmented reality (AR) guidance for safe LPN. MATERIALS AND METHODS The concept proposed is based on using an intraoperative CBCT scan for (1) marker-based AR guidance for fast and reliable tumor access and (2) enhancement of real-time fluoroscopy images for accurate tumor resection. Workflow and accuracy of the system were assessed using a porcine kidney model. Ten patients with complex or endophytic tumor localization and R.E.N.A.L. Nephrometry Score of at least nine scheduled for LPN were included in this study. Patients received an intraoperative CBCT after marker placement. Defining the resection line was assisted by AR. In addition, fluoroscopy imaging for depth perception was used for assistance during dissection. Feasibility and performance were assessed by histopathological results, peri- and postoperative data. RESULTS Surgery was performed successfully and negative margins were found in all cases. Segmental branches of the renal artery shifted as much as 10 mm in the vertical and 11 mm in the sagittal axis intraoperatively compared to preoperative imaging. Fluoroscopy to intraoperative computed tomography image fusion enabled enhanced depth perception during dissection in all cases. Radiation dose area product was 4.8 mGym2. CONCLUSIONS The application of the navigation system is feasible and allows for safe and direct access to complex or endophytic renal masses. Radiation limits the application to selected indications.
Collapse
Affiliation(s)
| | - Claudia Gasch
- 1 Department of Urology, University of Heidelberg , Heidelberg, Germany
| | - Gencay Hatiboglu
- 1 Department of Urology, University of Heidelberg , Heidelberg, Germany
| | | | - Lena Maier-Hein
- 3 Junior Group Computer-Assisted Interventions, German Cancer Research Center (DKFZ) , Heidelberg, Germany
| | | | - Dogu Teber
- 1 Department of Urology, University of Heidelberg , Heidelberg, Germany
| |
Collapse
|
23
|
Abstract
Minimally invasive surgery is slowly taking over as the preferred operative approach for colorectal diseases. However, many of the procedures remain technically difficult. This article will give an overview of the state of minimally invasive surgery and the many advances that have been made over the last two decades. Specifically, we discuss the introduction of the robotic platform and some of its benefits and limitations. We also describe some newer techniques related to robotics.
Collapse
Affiliation(s)
- Matthew Whealon
- Department of Surgery, University of California, Irvine, Orange, California
| | - Alessio Vinci
- Department of Surgery, University of California, Irvine, Orange, California
| | - Alessio Pigazzi
- Department of Surgery, University of California, Irvine, Orange, California
| |
Collapse
|
24
|
Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, Pessaux P. Augmented Reality Guidance for the Resection of Missing Colorectal Liver Metastases: An Initial Experience. World J Surg 2016; 40:419-26. [PMID: 26316112 DOI: 10.1007/s00268-015-3229-8] [Citation(s) in RCA: 58] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
BACKGROUND Modern chemotherapy achieves the shrinking of colorectal cancer liver metastases (CRLM) to such extent that they may disappear from radiological imaging. Disappearing CRLM rarely represents a complete pathological remission and have an important risk of recurrence. Augmented reality (AR) consists in the fusion of real-time patient images with a computer-generated 3D virtual patient model created from pre-operative medical imaging. The aim of this prospective pilot study is to investigate the potential of AR navigation as a tool to help locate and surgically resect missing CRLM. METHODS A 3D virtual anatomical model was created from thoracoabdominal CT-scans using customary software (VR RENDER(®), IRCAD). The virtual model was superimposed to the operative field using an Exoscope (VITOM(®), Karl Storz, Tüttlingen, Germany). Virtual and real images were manually registered in real-time using a video mixer, based on external anatomical landmarks with an estimated accuracy of 5 mm. This modality was tested in three patients, with four missing CRLM that had sizes from 12 to 24 mm, undergoing laparotomy after receiving pre-operative oxaliplatin-based chemotherapy. RESULTS AR display and fine registration was performed within 6 min. AR helped detect all four missing CRLM, and guided their resection. In all cases the planned security margin of 1 cm was clear and resections were confirmed to be R0 by pathology. There was no postoperative major morbidity or mortality. No local recurrence occurred in the follow-up period of 6-22 months. CONCLUSIONS This initial experience suggests that AR may be a helpful navigation tool for the resection of missing CRLM.
Collapse
Affiliation(s)
- Dimitrios Ntourakis
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France.
| | - Ricardo Memeo
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France
| | - Luc Soler
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France
| | - Jacques Marescaux
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France
| | - Didier Mutter
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France
| | - Patrick Pessaux
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France.
| |
Collapse
|
25
|
Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, Pessaux P. Augmented Reality Guidance for the Resection of Missing Colorectal Liver Metastases: An Initial Experience. World J Surg 2016. [PMID: 26316112 DOI: 10.1007/-s00268-015-3229-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
BACKGROUND Modern chemotherapy achieves the shrinking of colorectal cancer liver metastases (CRLM) to such extent that they may disappear from radiological imaging. Disappearing CRLM rarely represents a complete pathological remission and have an important risk of recurrence. Augmented reality (AR) consists in the fusion of real-time patient images with a computer-generated 3D virtual patient model created from pre-operative medical imaging. The aim of this prospective pilot study is to investigate the potential of AR navigation as a tool to help locate and surgically resect missing CRLM. METHODS A 3D virtual anatomical model was created from thoracoabdominal CT-scans using customary software (VR RENDER(®), IRCAD). The virtual model was superimposed to the operative field using an Exoscope (VITOM(®), Karl Storz, Tüttlingen, Germany). Virtual and real images were manually registered in real-time using a video mixer, based on external anatomical landmarks with an estimated accuracy of 5 mm. This modality was tested in three patients, with four missing CRLM that had sizes from 12 to 24 mm, undergoing laparotomy after receiving pre-operative oxaliplatin-based chemotherapy. RESULTS AR display and fine registration was performed within 6 min. AR helped detect all four missing CRLM, and guided their resection. In all cases the planned security margin of 1 cm was clear and resections were confirmed to be R0 by pathology. There was no postoperative major morbidity or mortality. No local recurrence occurred in the follow-up period of 6-22 months. CONCLUSIONS This initial experience suggests that AR may be a helpful navigation tool for the resection of missing CRLM.
Collapse
Affiliation(s)
- Dimitrios Ntourakis
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France.
| | - Ricardo Memeo
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France
| | - Luc Soler
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France
| | - Jacques Marescaux
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France
| | - Didier Mutter
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France
| | - Patrick Pessaux
- IRCAD-IHU, University of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France.
| |
Collapse
|
26
|
On-demand calibration and evaluation for electromagnetically tracked laparoscope in augmented reality visualization. Int J Comput Assist Radiol Surg 2016; 11:1163-71. [PMID: 27250853 DOI: 10.1007/s11548-016-1406-3] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2016] [Accepted: 03/24/2016] [Indexed: 10/21/2022]
Abstract
PURPOSE Common camera calibration methods employed in current laparoscopic augmented reality systems require the acquisition of multiple images of an entire checkerboard pattern from various poses. This lengthy procedure prevents performing laparoscope calibration in the operating room (OR). The purpose of this work was to develop a fast calibration method for electromagnetically (EM) tracked laparoscopes, such that the calibration can be performed in the OR on demand. METHODS We designed a mechanical tracking mount to uniquely and snugly position an EM sensor to an appropriate location on a conventional laparoscope. A tool named fCalib was developed to calibrate intrinsic camera parameters, distortion coefficients, and extrinsic parameters (transformation between the scope lens coordinate system and the EM sensor coordinate system) using a single image that shows an arbitrary portion of a special target pattern. For quick evaluation of calibration results in the OR, we integrated a tube phantom with fCalib prototype and overlaid a virtual representation of the tube on the live video scene. RESULTS We compared spatial target registration error between the common OpenCV method and the fCalib method in a laboratory setting. In addition, we compared the calibration re-projection error between the EM tracking-based fCalib and the optical tracking-based fCalib in a clinical setting. Our results suggest that the proposed method is comparable to the OpenCV method. However, changing the environment, e.g., inserting or removing surgical tools, might affect re-projection accuracy for the EM tracking-based approach. Computational time of the fCalib method averaged 14.0 s (range 3.5 s-22.7 s). CONCLUSIONS We developed and validated a prototype for fast calibration and evaluation of EM tracked conventional (forward viewing) laparoscopes. The calibration method achieved acceptable accuracy and was relatively fast and easy to be performed in the OR on demand.
Collapse
|
27
|
Automatic localization of endoscope in intraoperative CT image: A simple approach to augmented reality guidance in laparoscopic surgery. Med Image Anal 2016; 30:130-143. [DOI: 10.1016/j.media.2016.01.008] [Citation(s) in RCA: 37] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2014] [Revised: 04/17/2015] [Accepted: 01/04/2016] [Indexed: 11/23/2022]
|
28
|
Dixon BJ, Chan H, Daly MJ, Qiu J, Vescan A, Witterick IJ, Irish JC. Three-dimensional virtual navigation versus conventional image guidance: A randomized controlled trial. Laryngoscope 2016; 126:1510-5. [PMID: 27075606 DOI: 10.1002/lary.25882] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2015] [Revised: 12/12/2015] [Accepted: 12/29/2015] [Indexed: 11/10/2022]
Abstract
OBJECTIVES/HYPOTHESIS Providing image guidance in a 3-dimensional (3D) format, visually more in keeping with the operative field, could potentially reduce workload and lead to faster and more accurate navigation. We wished to assess a 3D virtual-view surgical navigation prototype in comparison to a traditional 2D system. METHODS Thirty-seven otolaryngology surgeons and trainees completed a randomized crossover navigation exercise on a cadaver model. Each subject identified three sinonasal landmarks with 3D virtual (3DV) image guidance and three landmarks with conventional cross-sectional computed tomography (CT) image guidance. Subjects were randomized with regard to which side and display type was tested initially. Accuracy, task completion time, and task workload were recorded. RESULTS Display type did not influence accuracy (P > 0.2) or efficiency (P > 0.3) for any of the six landmarks investigated. Pooled landmark data revealed a trend of improved accuracy in the 3DV group by 0.44 millimeters (95% confidence interval [0.00-0.88]). High-volume surgeons were significantly faster (P < 0.01) and had reduced workload scores in all domains (P < 0.01), but they were no more accurate (P > 0.28). CONCLUSION Real-time 3D image guidance did not influence accuracy, efficiency, or task workload when compared to conventional triplanar image guidance. The subtle pooled accuracy advantage for the 3DV view is unlikely to be of clinical significance. Experience level was strongly correlated to task completion time and workload but did not influence accuracy. LEVEL OF EVIDENCE N/A. Laryngoscope, 126:1510-1515, 2016.
Collapse
Affiliation(s)
- Benjamin J Dixon
- Department of Surgery, University of Melbourne, St Vincent's Hospital and Peter MacCallum Cancer Centre, Melbourne, Australia
| | - Harley Chan
- Ontario Cancer Institute, University Health Network, Toronto, Ontario, Canada
| | - Michael J Daly
- Ontario Cancer Institute, University Health Network, Toronto, Ontario, Canada.,Institute of Medical Science, Princess Margaret Hospital, University Health Network, Toronto, Ontario, Canada
| | - Jimmy Qiu
- Ontario Cancer Institute, University Health Network, Toronto, Ontario, Canada
| | - Allan Vescan
- Department of Otolaryngology-Head and Neck Surgery, Mount Sinai Hospital, University of Toronto, Toronto, Ontario, Canada
| | - Ian J Witterick
- Departments of Surgical Oncology, University Health Network, Toronto, Ontario, Canada.,Otolaryngology-Head and Neck Surgery, Ontario Cancer Institute, University Health Network, Toronto, Ontario, Canada.,Department of Otolaryngology-Head and Neck Surgery, Mount Sinai Hospital, University of Toronto, Toronto, Ontario, Canada
| | - Jonathan C Irish
- Departments of Surgical Oncology, University Health Network, Toronto, Ontario, Canada.,Otolaryngology-Head and Neck Surgery, Ontario Cancer Institute, University Health Network, Toronto, Ontario, Canada
| |
Collapse
|
29
|
Fusaglia M, Tinguely P, Banz V, Weber S, Lu H. A Novel Ultrasound-Based Registration for Image-Guided Laparoscopic Liver Ablation. Surg Innov 2016; 23:397-406. [PMID: 26969718 DOI: 10.1177/1553350616637691] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/11/2023]
Abstract
Background Patient-to-image registration is a core process of image-guided surgery (IGS) systems. We present a novel registration approach for application in laparoscopic liver surgery, which reconstructs in real time an intraoperative volume of the underlying intrahepatic vessels through an ultrasound (US) sweep process. Methods An existing IGS system for an open liver procedure was adapted, with suitable instrument tracking for laparoscopic equipment. Registration accuracy was evaluated on a realistic phantom by computing the target registration error (TRE) for 5 intrahepatic tumors. The registration work flow was evaluated by computing the time required for performing the registration. Additionally, a scheme for intraoperative accuracy assessment by visual overlay of the US image with preoperative image data was evaluated. Results The proposed registration method achieved an average TRE of 7.2 mm in the left lobe and 9.7 mm in the right lobe. The average time required for performing the registration was 12 minutes. A positive correlation was found between the intraoperative accuracy assessment and the obtained TREs. Conclusions The registration accuracy of the proposed method is adequate for laparoscopic intrahepatic tumor targeting. The presented approach is feasible and fast and may, therefore, not be disruptive to the current surgical work flow.
Collapse
|
30
|
Azagury DE, Dua MM, Barrese JC, Henderson JM, Buchs NC, Ris F, Cloyd JM, Martinie JB, Razzaque S, Nicolau S, Soler L, Marescaux J, Visser BC. Image-guided surgery. Curr Probl Surg 2015; 52:476-520. [PMID: 26683419 DOI: 10.1067/j.cpsurg.2015.10.001] [Citation(s) in RCA: 21] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2015] [Accepted: 10/01/2015] [Indexed: 12/11/2022]
Affiliation(s)
- Dan E Azagury
- Department of Surgery, Stanford University School of Medicine, Stanford, CA
| | - Monica M Dua
- Department of Surgery, Stanford University School of Medicine, Stanford, CA
| | - James C Barrese
- Department of Neurosurgery, Stanford University School of Medicine, Stanford, CA
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University School of Medicine, Stanford, CA
| | - Nicolas C Buchs
- Department of Surgery, University Hospital of Geneva, Clinic for Visceral and Transplantation Surgery, Geneva, Switzerland
| | - Frederic Ris
- Department of Surgery, University Hospital of Geneva, Clinic for Visceral and Transplantation Surgery, Geneva, Switzerland
| | - Jordan M Cloyd
- Department of Surgery, Stanford University School of Medicine, Stanford, CA
| | - John B Martinie
- Department of Surgery, Carolinas Healthcare System, Charlotte, NC
| | - Sharif Razzaque
- Department of Surgery, Carolinas Healthcare System, Charlotte, NC
| | - Stéphane Nicolau
- IRCAD (Research Institute Against Digestive Cancer), Strasbourg, France
| | - Luc Soler
- IRCAD (Research Institute Against Digestive Cancer), Strasbourg, France
| | - Jacques Marescaux
- IRCAD (Research Institute Against Digestive Cancer), Strasbourg, France
| | - Brendan C Visser
- Department of Surgery, Stanford University School of Medicine, Stanford, CA.
| |
Collapse
|
31
|
Electromagnetic organ tracking allows for real-time compensation of tissue shift in image-guided laparoscopic rectal surgery: results of a phantom study. Surg Endosc 2015; 30:495-503. [DOI: 10.1007/s00464-015-4231-9] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2014] [Accepted: 04/20/2015] [Indexed: 02/01/2023]
|
32
|
Diana M, Marescaux J. Robotic surgery. Br J Surg 2015; 102:e15-28. [PMID: 25627128 DOI: 10.1002/bjs.9711] [Citation(s) in RCA: 141] [Impact Index Per Article: 15.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2014] [Accepted: 10/20/2014] [Indexed: 12/14/2022]
Abstract
BACKGROUND Proficiency in minimally invasive surgery requires intensive and continuous training, as it is technically challenging for unnatural visual and haptic perceptions. Robotic and computer sciences are producing innovations to augment the surgeon's skills to achieve accuracy and high precision during complex surgery. This article reviews the current use of robotically assisted surgery, focusing on technology as well as main applications in digestive surgery, and future perspectives. METHODS The PubMed database was interrogated to retrieve evidence-based data on surgical applications. Internal and external consulting with key opinion leaders, renowned robotics laboratories and robotic platform manufacturers was used to produce state-of-the art business intelligence around robotically assisted surgery. RESULTS Selected digestive procedures (oesophagectomy, gastric bypass, pancreatic and liver resections, rectal resection for cancer) might benefit from robotic assistance, although the current level of evidence is insufficient to support widespread adoption. The surgical robotic market is growing, and a variety of projects have recently been launched at both academic and corporate levels to develop lightweight, miniaturized surgical robotic prototypes. CONCLUSION The magnified view, and improved ergonomics and dexterity offered by robotic platforms, might facilitate the uptake of minimally invasive procedures. Image guidance to complement robotically assisted procedures, through the concepts of augmented reality, could well represent a major revolution to increase safety and deal with difficulties associated with the new minimally invasive approaches.
Collapse
Affiliation(s)
- M Diana
- Research Institute Against Cancer of the Digestive System (IRCAD), European Institute of TeleSurgery (EITS) and International Institute for Image-Guided Surgery (IHU Strasbourg), Strasbourg, France
| | | |
Collapse
|
33
|
Okamoto T, Onda S, Yasuda J, Yanaga K, Suzuki N, Hattori A. Navigation surgery using an augmented reality for pancreatectomy. Dig Surg 2015; 32:117-23. [PMID: 25766302 DOI: 10.1159/000371860] [Citation(s) in RCA: 25] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/30/2014] [Accepted: 12/31/2014] [Indexed: 12/18/2022]
Abstract
AIM The aim of this study was to evaluate the utility of navigation surgery using augmented reality technology (AR-based NS) for pancreatectomy. METHODS The 3D reconstructed images from CT were created by segmentation. The initial registration was performed by using the optical location sensor. The reconstructed images were superimposed onto the real organs in the monitor display. Of the 19 patients who had undergone hepatobiliary and pancreatic surgery using AR-based NS, the accuracy, visualization ability, and utility of our system were assessed in five cases with pancreatectomy. RESULTS The position of each organ in the surface-rendering image corresponded almost to that of the actual organ. Reference to the display image allowed for safe dissection while preserving the adjacent vessels or organs. The locations of the lesions and resection line on the targeted organ were overlaid on the operating field. The initial mean registration error was improved to approximately 5 mm by our refinements. However, several problems such as registration accuracy, portability and cost still remain. CONCLUSION AR-based NS contributed to accurate and effective surgical resection in pancreatectomy. The pancreas appears to be a suitable organ for further investigations. This technology is promising to improve surgical quality, training, and education.
Collapse
Affiliation(s)
- Tomoyoshi Okamoto
- Department of Surgery, The Jikei University Daisan Hospital, Tokyo, Japan
| | | | | | | | | | | |
Collapse
|
34
|
Next step in minimally invasive surgery: hybrid image-guided surgery. J Pediatr Surg 2015; 50:30-6. [PMID: 25598089 DOI: 10.1016/j.jpedsurg.2014.10.022] [Citation(s) in RCA: 46] [Impact Index Per Article: 5.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/14/2014] [Accepted: 10/06/2014] [Indexed: 02/07/2023]
Abstract
Surgery, interventional radiology, and advanced endoscopy have all developed minimally invasive techniques to effectively treat a variety of diseases with positive impact on patients' postoperative outcomes. However, those techniques are challenging and require extensive training. Robotics and computer sciences can help facilitate minimally invasive approaches. Furthermore, surgery, advanced endoscopy, and interventional radiology could converge towards a new hybrid specialty, hybrid image-guided minimally invasive therapies, in which the three fundamental disciplines could complement one another to maximize the positive effects and reduce the iatrogenic footprint on patients. The present manuscript describes the fundamental steps of this new paradigm shift in surgical therapies that, in our opinion, will be the next revolutionary step in minimally invasive approaches.
Collapse
|
35
|
Wang J, Suenaga H, Liao H, Hoshi K, Yang L, Kobayashi E, Sakuma I. Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation. Comput Med Imaging Graph 2014; 40:147-59. [PMID: 25465067 DOI: 10.1016/j.compmedimag.2014.11.003] [Citation(s) in RCA: 50] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2014] [Revised: 09/03/2014] [Accepted: 11/03/2014] [Indexed: 11/26/2022]
Abstract
Autostereoscopic 3D image overlay for augmented reality (AR) based surgical navigation has been studied and reported many times. For the purpose of surgical overlay, the 3D image is expected to have the same geometric shape as the original organ, and can be transformed to a specified location for image overlay. However, how to generate a 3D image with high geometric fidelity and quantitative evaluation of 3D image's geometric accuracy have not been addressed. This paper proposes a graphics processing unit (GPU) based computer-generated integral imaging pipeline for real-time autostereoscopic 3D display, and an automatic closed-loop 3D image calibration paradigm for displaying undistorted 3D images. Based on the proposed methods, a novel AR device for 3D image surgical overlay is presented, which mainly consists of a 3D display, an AR window, a stereo camera for 3D measurement, and a workstation for information processing. The evaluation on the 3D image rendering performance with 2560×1600 elemental image resolution shows the rendering speeds of 50-60 frames per second (fps) for surface models, and 5-8 fps for large medical volumes. The evaluation of the undistorted 3D image after the calibration yields sub-millimeter geometric accuracy. A phantom experiment simulating oral and maxillofacial surgery was also performed to evaluate the proposed AR overlay device in terms of the image registration accuracy, 3D image overlay accuracy, and the visual effects of the overlay. The experimental results show satisfactory image registration and image overlay accuracy, and confirm the system usability.
Collapse
Affiliation(s)
- Junchen Wang
- Graduate School of Engineering, The University of Tokyo, Tokyo, Japan.
| | - Hideyuki Suenaga
- Department of Oral-Maxillofacial Surgery, Dentistry and Orthodontics, The University of Tokyo Hospital, Tokyo, Japan
| | - Hongen Liao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China.
| | - Kazuto Hoshi
- Department of Oral-Maxillofacial Surgery, Dentistry and Orthodontics, The University of Tokyo Hospital, Tokyo, Japan
| | - Liangjing Yang
- Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
| | - Etsuko Kobayashi
- Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
| | - Ichiro Sakuma
- Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
36
|
Pessaux P, Diana M, Soler L, Piardi T, Mutter D, Marescaux J. Towards cybernetic surgery: robotic and augmented reality-assisted liver segmentectomy. Langenbecks Arch Surg 2014; 400:381-5. [PMID: 25392120 DOI: 10.1007/s00423-014-1256-9] [Citation(s) in RCA: 89] [Impact Index Per Article: 8.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2013] [Accepted: 10/20/2014] [Indexed: 12/12/2022]
Abstract
BACKGROUND Augmented reality (AR) in surgery consists in the fusion of synthetic computer-generated images (3D virtual model) obtained from medical imaging preoperative workup and real-time patient images in order to visualize unapparent anatomical details. The 3D model could be used for a preoperative planning of the procedure. The potential of AR navigation as a tool to improve safety of the surgical dissection is outlined for robotic hepatectomy. MATERIALS AND METHODS Three patients underwent a fully robotic and AR-assisted hepatic segmentectomy. The 3D virtual anatomical model was obtained using a thoracoabdominal CT scan with a customary software (VR-RENDER®, IRCAD). The model was then processed using a VR-RENDER® plug-in application, the Virtual Surgical Planning (VSP®, IRCAD), to delineate surgical resection planes including the elective ligature of vascular structures. Deformations associated with pneumoperitoneum were also simulated. The virtual model was superimposed to the operative field. A computer scientist manually registered virtual and real images using a video mixer (MX 70; Panasonic, Secaucus, NJ) in real time. RESULTS Two totally robotic AR segmentectomy V and one segmentectomy VI were performed. AR allowed for the precise and safe recognition of all major vascular structures during the procedure. Total time required to obtain AR was 8 min (range 6-10 min). Each registration (alignment of the vascular anatomy) required a few seconds. Hepatic pedicle clamping was never performed. At the end of the procedure, the remnant liver was correctly vascularized. Resection margins were negative in all cases. The postoperative period was uneventful without perioperative transfusion. CONCLUSIONS AR is a valuable navigation tool which may enhance the ability to achieve safe surgical resection during robotic hepatectomy.
Collapse
Affiliation(s)
- Patrick Pessaux
- Institut Hospitalo-Universitaire de Strasbourg (IHU) Digestive and Endocrine Surgery, Nouvel Hôpital Civil, University of Strasbourg, 1, Place de l'Hôpital, 67091, Strasbourg, France,
| | | | | | | | | | | |
Collapse
|
37
|
Okamoto T, Onda S, Yanaga K, Suzuki N, Hattori A. Clinical application of navigation surgery using augmented reality in the abdominal field. Surg Today 2014; 45:397-406. [PMID: 24898629 DOI: 10.1007/s00595-014-0946-9] [Citation(s) in RCA: 47] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2013] [Accepted: 01/23/2014] [Indexed: 12/20/2022]
Abstract
This article presents general principles and recent advancements in the clinical application of augmented reality-based navigation surgery (AR based NS) for abdominal procedures and includes a description of our clinical trial and subsequent outcomes. Moreover, current problems and future aspects are discussed. The development of AR-based NS in the abdomen is delayed compared with another field because of the problem of intraoperative organ deformations or the existence of established modalities. Although there are a few reports on the clinical use of AR-based NS for digestive surgery, sophisticated technologies in urology have often been reported. However, the rapid widespread use of video- or robot assisted surgeries requires this technology. We have worked to develop a system of AR-based NS for hepatobiliary and pancreatic surgery. Then we developed a short rigid scope that enables surgeons to obtain 3D view. We recently focused on pancreatic surgery, because intraoperative organ shifting is minimal. The position of each organ in overlaid image almost corresponded with that of the actual organ with about 5 mm of mean registration errors. Intraoperative information generated from this system provided us with useful navigation. However, AR-based NS has several problems to overcome such as organ deformity, evaluation of utility, portability or cost.
Collapse
Affiliation(s)
- Tomoyoshi Okamoto
- Department of Surgery, The Jikei University Daisan Hospital, 4-11-1 Izumihoncho, Komae-shi, Tokyo, Japan,
| | | | | | | | | |
Collapse
|
38
|
Soler L, Nicolau S, Pessaux P, Mutter D, Marescaux J. Real-time 3D image reconstruction guidance in liver resection surgery. Hepatobiliary Surg Nutr 2014; 3:73-81. [PMID: 24812598 DOI: 10.3978/j.issn.2304-3881.2014.02.03] [Citation(s) in RCA: 33] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/08/2014] [Accepted: 02/20/2014] [Indexed: 12/26/2022]
Abstract
BACKGROUND Minimally invasive surgery represents one of the main evolutions of surgical techniques. However, minimally invasive surgery adds difficulty that can be reduced through computer technology. METHODS From a patient's medical image [US, computed tomography (CT) or MRI], we have developed an Augmented Reality (AR) system that increases the surgeon's intraoperative vision by providing a virtual transparency of the patient. AR is based on two major processes: 3D modeling and visualization of anatomical or pathological structures appearing in the medical image, and the registration of this visualization onto the real patient. We have thus developed a new online service, named Visible Patient, providing efficient 3D modeling of patients. We have then developed several 3D visualization and surgical planning software tools to combine direct volume rendering and surface rendering. Finally, we have developed two registration techniques, one interactive and one automatic providing intraoperative augmented reality view. RESULTS From January 2009 to June 2013, 769 clinical cases have been modeled by the Visible Patient service. Moreover, three clinical validations have been realized demonstrating the accuracy of 3D models and their great benefit, potentially increasing surgical eligibility in liver surgery (20% of cases). From these 3D models, more than 50 interactive AR-assisted surgical procedures have been realized illustrating the potential clinical benefit of such assistance to gain safety, but also current limits that automatic augmented reality will overcome. CONCLUSIONS Virtual patient modeling should be mandatory for certain interventions that have now to be defined, such as liver surgery. Augmented reality is clearly the next step of the new surgical instrumentation but remains currently limited due to the complexity of organ deformations during surgery. Intraoperative medical imaging used in new generation of automated augmented reality should solve this issue thanks to the development of Hybrid OR.
Collapse
Affiliation(s)
- Luc Soler
- 1 IRCAD France, 2 IHU Strasbourg, 1 place de l'hôpital, 67091 Strasbourg, France
| | - Stephane Nicolau
- 1 IRCAD France, 2 IHU Strasbourg, 1 place de l'hôpital, 67091 Strasbourg, France
| | - Patrick Pessaux
- 1 IRCAD France, 2 IHU Strasbourg, 1 place de l'hôpital, 67091 Strasbourg, France
| | - Didier Mutter
- 1 IRCAD France, 2 IHU Strasbourg, 1 place de l'hôpital, 67091 Strasbourg, France
| | - Jacques Marescaux
- 1 IRCAD France, 2 IHU Strasbourg, 1 place de l'hôpital, 67091 Strasbourg, France
| |
Collapse
|
39
|
Robotic duodenopancreatectomy assisted with augmented reality and real-time fluorescence guidance. Surg Endosc 2014; 28:2493-8. [PMID: 24609700 DOI: 10.1007/s00464-014-3465-2] [Citation(s) in RCA: 42] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2013] [Accepted: 01/24/2014] [Indexed: 02/06/2023]
Abstract
BACKGROUND The minimally invasive surgeon cannot use 'sense of touch' to orientate surgical resection, identifying important structures (vessels, tumors, etc.) by manual palpation. Robotic research has provided technology to facilitate laparoscopic surgery; however, robotics has yet to solve the lack of tactile feedback inherent to keyhole surgery. Misinterpretation of the vascular supply and tumor location may increase the risk of intraoperative bleeding and worsen dissection with positive resection margins. METHODS Augmented reality (AR) consists of the fusion of synthetic computer-generated images (three-dimensional virtual model) obtained from medical imaging preoperative work-up and real-time patient images with the aim of visualizing unapparent anatomical details. RESULTS In this article, we review the most common modalities used to achieve surgical navigation through AR, along with a report of a case of robotic duodenopancreatectomy using AR guidance complemented with the use of fluorescence guidance. CONCLUSIONS The presentation of this complex and high-technology case of robotic duodenopancreatectomy, and the overview of current technology that has made it possible to use AR in the operating room, highlights the needs for further evolution and the windows of opportunity to create a new paradigm in surgical practice.
Collapse
|
40
|
Kang X, Azizian M, Wilson E, Wu K, Martin AD, Kane TD, Peters CA, Cleary K, Shekhar R. Stereoscopic augmented reality for laparoscopic surgery. Surg Endosc 2014; 28:2227-35. [PMID: 24488352 DOI: 10.1007/s00464-014-3433-x] [Citation(s) in RCA: 51] [Impact Index Per Article: 5.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2013] [Accepted: 01/10/2014] [Indexed: 02/08/2023]
Abstract
BACKGROUND Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field. METHODS The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine. RESULTS The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy. CONCLUSIONS We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.
Collapse
Affiliation(s)
- Xin Kang
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Medical Center, 111 Michigan Avenue NW, Washington, DC, 20010, USA,
| | | | | | | | | | | | | | | | | |
Collapse
|
41
|
Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 2013; 17:974-96. [DOI: 10.1016/j.media.2013.04.003] [Citation(s) in RCA: 182] [Impact Index Per Article: 16.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2012] [Revised: 04/05/2013] [Accepted: 04/12/2013] [Indexed: 12/16/2022]
|
42
|
Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging. Surg Endosc 2013; 28:933-40. [PMID: 24178862 DOI: 10.1007/s00464-013-3249-0] [Citation(s) in RCA: 69] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2013] [Accepted: 10/04/2013] [Indexed: 01/11/2023]
Abstract
BACKGROUND Laparoscopic liver surgery is particularly challenging owing to restricted access, risk of bleeding, and lack of haptic feedback. Navigation systems have the potential to improve information on the exact position of intrahepatic tumors, and thus facilitate oncological resection. This study aims to evaluate the feasibility of a commercially available augmented reality (AR) guidance system employing intraoperative robotic C-arm cone-beam computed tomography (CBCT) for laparoscopic liver surgery. METHODS A human liver-like phantom with 16 target fiducials was used to evaluate the Syngo iPilot(®) AR system. Subsequently, the system was used for the laparoscopic resection of a hepatocellular carcinoma in segment 7 of a 50-year-old male patient. RESULTS In the phantom experiment, the AR system showed a mean target registration error of 0.96 ± 0.52 mm, with a maximum error of 2.49 mm. The patient successfully underwent the operation and showed no postoperative complications. CONCLUSION The use of intraoperative CBCT and AR for laparoscopic liver resection is feasible and could be considered an option for future liver surgery in complex cases.
Collapse
|
43
|
Marzano E, Piardi T, Soler L, Diana M, Mutter D, Marescaux J, Pessaux P. Augmented reality-guided artery-first pancreatico-duodenectomy. J Gastrointest Surg 2013; 17:1980-3. [PMID: 23943389 DOI: 10.1007/s11605-013-2307-1] [Citation(s) in RCA: 45] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/20/2012] [Accepted: 07/29/2013] [Indexed: 01/31/2023]
Abstract
BACKGROUND Augmented Reality (AR) in surgery consists in the fusion of synthetic computer-generated images (3D virtual model) obtained from medical imaging preoperative work-up and real-time patient images with the aim to visualize unapparent anatomical details. The potential of AR navigation as a tool to improve safety of the surgical dissection is presented in a case of pancreatico-duodenectomy (PD). METHODS A 77-year-old male patient underwent an AR-assisted PD. The 3D virtual anatomical model was obtained from thoraco-abdominal CT scan using customary software (VR-RENDER®, IRCAD). The virtual model was superimposed to the operative field using an Exoscope (VITOM®, Karl Storz, Tüttlingen, Germany) as well as different visible landmarks (inferior vena cava, left renal vein, aorta, superior mesenteric vein, inferior margin of the pancreas). A computer scientist manually registered virtual and real images using a video mixer (MX 70; Panasonic, Secaucus, NJ) in real time. Dissection of the superior mesenteric artery and the hanging maneuver were performed under AR guidance along the hanging plane. RESULTS AR allowed for precise and safe recognition of all the important vascular structures. Operative time was 360 min. AR display and fine registration was performed within 6 min. The postoperative course was uneventful. The pathology was positive for ampullary adenocarcinoma; the final stage was pT1N0 (0/43 retrieved lymph nodes) with clear surgical margins. CONCLUSIONS AR is a valuable navigation tool that can enhance the ability to achieve a safe surgical resection during PD.
Collapse
Affiliation(s)
- Ettore Marzano
- IRCAD-IHU University Hospital of Strasbourg, 1 place de l'Hôpital, 67091, Strasbourg, France
| | | | | | | | | | | | | |
Collapse
|
44
|
Utility of augmented reality system in hepatobiliary surgery. JOURNAL OF HEPATO-BILIARY-PANCREATIC SCIENCES 2013; 20:249-53. [PMID: 22399157 DOI: 10.1007/s00534-012-0504-z] [Citation(s) in RCA: 28] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
BACKGROUND/PURPOSE The aim of this study was to evaluate the utility of an image display system for augmented reality in hepatobiliary surgery under laparotomy. METHODS An overlay display of organs, vessels, or tumor was obtained using a video see-through system as a display system developed at our institute. Registration between visceral organs and the surface-rendering image reconstructed by preoperative computed tomography (CT) was carried out with an optical location sensor. Using this system, we performed laparotomy for a patient with benign biliary stricture, a patient with gallbladder carcinoma, and a patient with hepatocellular carcinoma. RESULTS The operative procedures performed consisted of choledochojejunostomy, right hepatectomy, and microwave coagulation therapy. All the operations were carried out safely using images of the site of tumor, preserved organs, and resection aspect overlaid onto the operation field images observed on the monitors. The position of each organ in the overlaid image closely corresponded with that of the actual organ. Intraoperative information generated from this system provided us with useful navigation. However, several problems such as registration error and lack of depth knowledge were noted. CONCLUSION The image display system appeared to be useful in performing hepatobiliary surgery under laparotomy. Further improvement of the system with individualized function for each operation will be essential, with feedback from clinical trials in the future.
Collapse
|
45
|
Kersten-Oertel M, Jannin P, Collins DL. The state of the art of visualization in mixed reality image guided surgery. Comput Med Imaging Graph 2013; 37:98-112. [PMID: 23490236 DOI: 10.1016/j.compmedimag.2013.01.009] [Citation(s) in RCA: 106] [Impact Index Per Article: 9.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2012] [Revised: 01/04/2013] [Accepted: 01/23/2013] [Indexed: 11/26/2022]
Abstract
This paper presents a review of the state of the art of visualization in mixed reality image guided surgery (IGS). We used the DVV (data, visualization processing, view) taxonomy to classify a large unbiased selection of publications in the field. The goal of this work was not only to give an overview of current visualization methods and techniques in IGS but more importantly to analyze the current trends and solutions used in the domain. In surveying the current landscape of mixed reality IGS systems, we identified a strong need to assess which of the many possible data sets should be visualized at particular surgical steps, to focus on novel visualization processing techniques and interface solutions, and to evaluate new systems.
Collapse
Affiliation(s)
- Marta Kersten-Oertel
- Department of Biomedical Engineering, McGill University, McConnell Brain Imaging Center, Montreal Neurological Institute, Montréal, Canada.
| | | | | |
Collapse
|
46
|
Dixon BJ, Daly MJ, Chan H, Vescan AD, Witterick IJ, Irish JC. Surgeons blinded by enhanced navigation: the effect of augmented reality on attention. Surg Endosc 2012; 27:454-61. [PMID: 22833264 DOI: 10.1007/s00464-012-2457-3] [Citation(s) in RCA: 56] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2012] [Accepted: 06/11/2012] [Indexed: 01/09/2023]
Abstract
BACKGROUND Advanced image-guidance systems allowing presentation of three-dimensional navigational data in real time are being developed enthusiastically for many medical procedures. Other industries, including aviation and the military, have noted that shifting attention toward such compelling assistance has detrimental effects. Using the detection rate of unexpected findings, we assess whether inattentional blindness is significant in a surgical context and evaluate the impact of on-screen navigational cuing with augmented reality. METHODS Surgeons and trainees performed an endoscopic navigation exercise on a cadaveric specimen. The subjects were randomized to either a standard endoscopic view (control) or an AR view consisting of an endoscopic video fused with anatomic contours. Two unexpected findings were presented in close proximity to the target point: one critical complication and one foreign body (screw). Task completion time, accuracy, and recognition of findings were recorded. RESULTS Detection of the complication was 0/15 in the AR group versus 7/17 in the control group (p = 0.008). Detection of the screw was 1/15 (AR) and 7/17 (control) (p = 0.041). Recognition of either finding was 12/17 for the control group and 1/15 for the AR group (p < 0.001). Accuracy was greater for the AR group than for the control group, with the median distance from the target point measuring respectively 2.10 mm (interquartile range [IQR], 1.29-2.37) and 4.13 (IQR, 3.11-7.39) (p < 0.001). CONCLUSION Inattentional blindness was evident in both groups. Although more accurate, the AR group was less likely to identify significant unexpected findings clearly within view. Advanced navigational displays may increase precision, but strategies to mitigate attentional costs need further investigation to allow safe implementation.
Collapse
Affiliation(s)
- Benjamin J Dixon
- Department of Surgical Oncology, Princess Margaret Hospital, University Health Network, University of Toronto, 610 University Avenue, 3-954, Toronto, ON M5G 2M9, Canada.
| | | | | | | | | | | |
Collapse
|
47
|
Weidert S, Wang L, von der Heide A, Navab N, Euler E. [Intraoperative augmented reality visualization. Current state of development and initial experiences with the CamC]. Unfallchirurg 2012; 115:209-13. [PMID: 22406917 DOI: 10.1007/s00113-011-2121-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
The intraoperative application of augmented reality (AR) has so far mainly taken place in the field of endoscopy. Here, the camera image of the endoscope was augmented by computer graphics derived mostly from preoperative imaging. Due to the complex setup and operation of the devices, they have not yet become part of routine clinical practice. The Camera Augmented Mobile C-arm (CamC) that extends a classic C-arm by a video camera and mirror construction is characterized by its uncomplicated handling. It combines its video live stream geometrically correct with the acquired X-ray. The clinical application of the device in 43 cases showed the strengths of the device in positioning for X-ray acquisition, incision placement, K-wire placement, and instrument guidance. With its new function and the easy integration into the OR workflow of any procedure that requires X-ray imaging, the CamC has the potential to become the first widely used AR technology for orthopedic and trauma surgery.
Collapse
Affiliation(s)
- S Weidert
- Chirurgische Klinik und Poliklinik, Ludwig-Maximilians-Universität München, Nussbaumstraße 20, 80336, München, Deutschland.
| | | | | | | | | |
Collapse
|
48
|
Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion. JOURNAL OF HEPATO-BILIARY-PANCREATIC SCIENCES 2011; 18:506-9. [PMID: 21487758 DOI: 10.1007/s00534-011-0385-6] [Citation(s) in RCA: 88] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
BACKGROUND New technologies can considerably improve preoperative planning, enhance the surgeon's skill and simplify the approach to complex procedures. Augmented reality techniques, robot assisted operations and computer assisted navigation tools will become increasingly important in surgery and in residents' education. METHODS We obtained 3D reconstructions from simple spiral computed tomography (CT) slides using OsiriX, an open source processing software package dedicated to DICOM images. These images were then projected on the patient's body with a beamer fixed to the operating table to enhance spatial perception during surgical intervention (augmented reality). RESULTS Changing a window's deepness level allowed the surgeon to navigate through the patient's anatomy, highlighting regions of interest and marked pathologies. We used image overlay navigation for laparoscopic operations such cholecystectomy, abdominal exploration, distal pancreas resection and robotic liver resection. CONCLUSIONS Augmented reality techniques will transform the behaviour of surgeons, making surgical interventions easier, faster and probably safer. These new techniques will also renew methods of surgical teaching, facilitating transmission of knowledge and skill to young surgeons.
Collapse
|
49
|
Abstract
Minimally invasive surgery represents one of the main evolutions of surgical techniques aimed at providing a greater benefit to the patient. However, minimally invasive surgery increases the operative difficulty since the depth perception is usually dramatically reduced, the field of view is limited and the sense of touch is transmitted by an instrument. However, these drawbacks can currently be reduced by computer technology guiding the surgical gesture. Indeed, from a patient's medical image (US, CT or MRI), Augmented Reality (AR) can increase the surgeon's intra-operative vision by providing a virtual transparency of the patient. AR is based on two main processes: the 3D visualization of the anatomical or pathological structures appearing in the medical image, and the registration of this visualization on the real patient. 3D visualization can be performed directly from the medical image without the need for a pre-processing step thanks to volume rendering. But better results are obtained with surface rendering after organ and pathology delineations and 3D modelling. Registration can be performed interactively or automatically. Several interactive systems have been developed and applied to humans, demonstrating the benefit of AR in surgical oncology. It also shows the current limited interactivity due to soft organ movements and interaction between surgeon instruments and organs. If the current automatic AR systems show the feasibility of such system, it is still relying on specific and expensive equipment which is not available in clinical routine. Moreover, they are not robust enough due to the high complexity of developing a real-time registration taking organ deformation and human movement into account. However, the latest results of automatic AR systems are extremely encouraging and show that it will become a standard requirement for future computer-assisted surgical oncology. In this article, we will explain the concept of AR and its principles. Then, we will review the existing interactive and automatic AR systems in digestive surgical oncology, highlighting their benefits and limitations. Finally, we will discuss the future evolutions and the issues that still have to be tackled so that this technology can be seamlessly integrated in the operating room.
Collapse
|