1
|
Long Z, Chi Y, Zhang X, Li H, Yang D, Li D, Jiang Z. An EM-Tracked Approach for Calibrating the 3D Pose of Flexible Endoscopes. Ann Biomed Eng 2024; 52:1435-1447. [PMID: 38402316 DOI: 10.1007/s10439-024-03469-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Accepted: 02/06/2024] [Indexed: 02/26/2024]
Abstract
Flexible endoscopes are ideal instruments for visualizing and diagnosing the inner surfaces of organs via a minimally invasive incision. Calibrating a flexible endoscope is a troublesome yet inevitable process in image-based tools tracking. Aiming to simplify the calibration process, we propose an electromagnetic (EM)-tracked calibration approach that does not require any predefined poses of the EM sensor. A three-stage calibration protocol was presented in an extensor. First, the orientation of the endoscope tube was derived by conducting a circular rotation of the endoscope around its axis utilizing a pair of tightly bearing stands. Second, the 3D position of the endoscope tip was acquired by having the tip come into contact with a flat plane. Third, the pose model of the bending section was derived and transformed into the local coordinate system of the EM sensor attached to the endoscope handle. To assess the accuracy of the proposed calibration approach, two experiments were designed and performed. Experimental results indicate accuracies of 0.09 ± 0.06 deg and 0.03 ± 0.19 deg in the estimation of the endoscope tube orientation and 0.52 ± 0.29, 0.33 ± 0.11, and 0.29 ± 0.17 mm in the x, y, and z estimations of the endoscope tip position, respectively. The proposed approach is accurate and easy to operate, does not require the employment of custom calibration markers, and can be used not only in surgical training systems but also in the endoscopic-based tools tracking.
Collapse
Affiliation(s)
- Zhongjie Long
- Key Laboratory of the Ministry of Education for Modern Measurement & Control Technology, and School of Electromechanical Engineering, Beijing Information Science & Technology University, Beijing, 100192, China.
| | - Yongting Chi
- Key Laboratory of the Ministry of Education for Modern Measurement & Control Technology, and School of Electromechanical Engineering, Beijing Information Science & Technology University, Beijing, 100192, China
| | - Xianbo Zhang
- Key Laboratory of the Ministry of Education for Modern Measurement & Control Technology, and School of Electromechanical Engineering, Beijing Information Science & Technology University, Beijing, 100192, China
| | - Hongbing Li
- Department of Instrument Science and Engineering, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China
| | - Dejin Yang
- Beijing Jishuitan Hospital, Capital Medical School, 4th Clinical College of Peking University, Beijing, 100035, China
| | - Diangeng Li
- Department of Nephrology, Beijing Chao-yang Hospital, Capital Medical University, Beijing, 100020, China
| | - Zhouxiang Jiang
- Key Laboratory of the Ministry of Education for Modern Measurement & Control Technology, and School of Electromechanical Engineering, Beijing Information Science & Technology University, Beijing, 100192, China
| |
Collapse
|
2
|
El Chemaly T, Athayde Neves C, Leuze C, Hargreaves B, H Blevins N. Stereoscopic calibration for augmented reality visualization in microscopic surgery. Int J Comput Assist Radiol Surg 2023; 18:2033-2041. [PMID: 37450175 DOI: 10.1007/s11548-023-02980-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2023] [Accepted: 05/26/2023] [Indexed: 07/18/2023]
Abstract
PURPOSE Middle and inner ear procedures target hearing loss, infections, and tumors of the temporal bone and lateral skull base. Despite the advances in surgical techniques, these procedures remain challenging due to limited haptic and visual feedback. Augmented reality (AR) may improve operative safety by allowing the 3D visualization of anatomical structures from preoperative computed tomography (CT) scans on real intraoperative microscope video feed. The purpose of this work was to develop a real-time CT-augmented stereo microscope system using camera calibration and electromagnetic (EM) tracking. METHODS A 3D printed and electromagnetically tracked calibration board was used to compute the intrinsic and extrinsic parameters of the surgical stereo microscope. These parameters were used to establish a transformation between the EM tracker coordinate system and the stereo microscope image space such that any tracked 3D point can be projected onto the left and right images of the microscope video stream. This allowed the augmentation of the microscope feed of a 3D printed temporal bone with its corresponding CT-derived virtual model. Finally, the calibration board was also used for evaluating the accuracy of the calibration. RESULTS We evaluated the accuracy of the system by calculating the registration error (RE) in 2D and 3D in a microsurgical laboratory setting. Our calibration workflow achieved a RE of 0.11 ± 0.06 mm in 2D and 0.98 ± 0.13 mm in 3D. In addition, we overlaid a 3D CT model on the microscope feed of a 3D resin printed model of a segmented temporal bone. The system exhibited small latency and good registration accuracy. CONCLUSION We present the calibration of an electromagnetically tracked surgical stereo microscope for augmented reality visualization. The calibration method achieved accuracy within a range suitable for otologic procedures. The AR process introduces enhanced visualization of the surgical field while allowing depth perception.
Collapse
Affiliation(s)
- Trishia El Chemaly
- Department of Bioengineering, Stanford University, Stanford, CA, USA.
- Department of Otolaryngology, Stanford School of Medicine, Stanford, CA, USA.
- Department of Radiology, Stanford School of Medicine, Stanford, CA, USA.
| | - Caio Athayde Neves
- Department of Otolaryngology, Stanford School of Medicine, Stanford, CA, USA
- Faculty of Medicine, University of Brasília, Brasília, Brazil
| | - Christoph Leuze
- Department of Radiology, Stanford School of Medicine, Stanford, CA, USA
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
| | - Brian Hargreaves
- Department of Bioengineering, Stanford University, Stanford, CA, USA
- Department of Radiology, Stanford School of Medicine, Stanford, CA, USA
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
| | - Nikolas H Blevins
- Department of Otolaryngology, Stanford School of Medicine, Stanford, CA, USA
| |
Collapse
|
3
|
Dowrick T, Xiao G, Nikitichev D, Dursun E, van Berkel N, Allam M, Koo B, Ramalhinho J, Thompson S, Gurusamy K, Blandford A, Stoyanov D, Davidson BR, Clarkson MJ. Evaluation of a calibration rig for stereo laparoscopes. Med Phys 2023; 50:2695-2704. [PMID: 36779419 PMCID: PMC10614700 DOI: 10.1002/mp.16310] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2022] [Revised: 02/01/2023] [Accepted: 02/01/2023] [Indexed: 02/14/2023] Open
Abstract
BACKGROUND Accurate camera and hand-eye calibration are essential to ensure high-quality results in image-guided surgery applications. The process must also be able to be undertaken by a nonexpert user in a surgical setting. PURPOSE This work seeks to identify a suitable method for tracked stereo laparoscope calibration within theater. METHODS A custom calibration rig, to enable rapid calibration in a surgical setting, was designed. The rig was compared against freehand calibration. Stereo reprojection, stereo reconstruction, tracked stereo reprojection, and tracked stereo reconstruction error metrics were used to evaluate calibration quality. RESULTS Use of the calibration rig reduced mean errors: reprojection (1.47 mm [SD 0.13] vs. 3.14 mm [SD 2.11], p-value 1e-8), reconstruction (1.37 px [SD 0.10] vs. 10.10 px [SD 4.54], p-value 6e-7), and tracked reconstruction (1.38 mm [SD 0.10] vs. 12.64 mm [SD 4.34], p-value 1e-6) compared with freehand calibration. The use of a ChArUco pattern yielded slightly lower reprojection errors, while a dot grid produced lower reconstruction errors and was more robust under strong global illumination. CONCLUSION The use of the calibration rig results in a statistically significant decrease in calibration error metrics, versus freehand calibration, and represents the preferred approach for use in the operating theater.
Collapse
Affiliation(s)
- Thomas Dowrick
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | - Guofang Xiao
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | - Daniil Nikitichev
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | - Eren Dursun
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | - Niels van Berkel
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | - Moustafa Allam
- Royal Free CampusUCL Medical SchoolRoyal Free HospitalLondonUK
| | - Bongjin Koo
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | - Joao Ramalhinho
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | - Stephen Thompson
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | | | - Ann Blandford
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | - Danail Stoyanov
- Wellcome EPSRC Centre for Interventional and Surgical SciencesUCLLondonUK
| | | | | |
Collapse
|
4
|
Spinczyk D. Measuring Respiratory Motion for Supporting the Minimally Invasive Destruction of Liver Tumors. SENSORS (BASEL, SWITZERLAND) 2022; 22:6446. [PMID: 36080904 PMCID: PMC9460029 DOI: 10.3390/s22176446] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Revised: 08/19/2022] [Accepted: 08/25/2022] [Indexed: 06/15/2023]
Abstract
OBJECTIVE Destroying liver tumors is a challenge for contemporary interventional radiology. The aim of this work is to compare different techniques used for the measurement of respiratory motion, as this is the main hurdle to the effective implementation of this therapy. METHODS Laparoscopic stereoscopic reconstruction of point displacements on the surface of the liver, observation of breathing using external markers placed on the surface of the abdominal cavity, and methods for registration of the surface of the abdominal cavity during breathing were implemented and evaluated. RESULTS The following accuracies were obtained: above 4 mm and 0.5 mm, and below 8 mm for laparoscopic, skin markers, and skin surface registration methods, respectively. CONCLUSIONS The clinical techniques and accompanying imaging modalities employed to destroy liver tumors, as well as the advantages and limitations of the proposed methods, are presented. Further directions for their development are also indicated.
Collapse
Affiliation(s)
- Dominik Spinczyk
- Faculty of Biomedical Engineering, Silesian University of Technology, 40 Roosevelta, 41-800 Zabrze, Poland
| |
Collapse
|
5
|
Liu X, Plishker W, Shekhar R. Hybrid electromagnetic-ArUco tracking of laparoscopic ultrasound transducer in laparoscopic video. J Med Imaging (Bellingham) 2021; 8:015001. [PMID: 33585664 PMCID: PMC7857492 DOI: 10.1117/1.jmi.8.1.015001] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2020] [Accepted: 01/12/2021] [Indexed: 11/14/2022] Open
Abstract
Purpose: The purpose of this work was to develop a new method of tracking a laparoscopic ultrasound (LUS) transducer in laparoscopic video by combining the hardware [e.g., electromagnetic (EM)] and the computer vision-based (e.g., ArUco) tracking methods. Approach: We developed a special tracking mount for the imaging tip of the LUS transducer. The mount incorporated an EM sensor and an ArUco pattern registered to it. The hybrid method used ArUco tracking for ArUco-success frames (i.e., frames where ArUco succeeds in detecting the pattern) and used corrected EM tracking for the ArUco-failure frames. The corrected EM tracking result was obtained by applying correction matrices to the original EM tracking result. The correction matrices were calculated in previous ArUco-success frames by comparing the ArUco result and the original EM tracking result. Results: We performed phantom and animal studies to evaluate the performance of our hybrid tracking method. The corrected EM tracking results showed significant improvements over the original EM tracking results. In the animal study, 59.2% frames were ArUco-success frames. For the ArUco-failure frames, mean reprojection errors for the original EM tracking method and for the corrected EM tracking method were 30.8 pixel and 10.3 pixel, respectively. Conclusions: The new hybrid method is more reliable than using ArUco tracking alone and more accurate and practical than using EM tracking alone for tracking the LUS transducer in the laparoscope camera image. The proposed method has the potential to significantly improve tracking performance for LUS-based augmented reality applications.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, United States
| | | | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, United States.,IGI Technologies, Inc., Silver Spring, Maryland, United States
| |
Collapse
|
6
|
Özgüner O, Shkurti T, Huang S, Hao R, Jackson RC, Newman WS, Çavuşoğlu MC. Camera-Robot Calibration for the da Vinci® Robotic Surgery System. IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING : A PUBLICATION OF THE IEEE ROBOTICS AND AUTOMATION SOCIETY 2020; 17:2154-2161. [PMID: 33746640 PMCID: PMC7978174 DOI: 10.1109/tase.2020.2986503] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
The development of autonomous or semi-autonomous surgical robots stands to improve the performance of existing teleoperated equipment, but requires fine hand-eye calibration between the free-moving endoscopic camera and patient-side manipulator arms (PSMs). A novel method of solving this problem for the da Vinci® robotic surgical system and kinematically similar systems is presented. First, a series of image-processing and optical-tracking operations are performed to compute the coordinate transformation between the endoscopic camera view frame and an optical-tracking marker permanently affixed to the camera body. Then, the kinematic properties of the PSM are exploited to compute the coordinate transformation between the kinematic base frame of the PSM and an optical marker permanently affixed thereto. Using these transformations, it is then possible to compute the spatial relationship between the PSM and the endoscopic camera using only one tracker snapshot of the two markers. The effectiveness of this calibration is demonstrated by successfully guiding the PSM end effector to points of interest identified through the camera. Additional tests on a surgical task, namely grasping a surgical needle, are also performed to validate the proposed method. The resulting visually-guided robot positioning accuracy is better than the earlier hand-eye calibration results reported in the literature for the da Vinci® system, while supporting intraoperative update of the calibration and requiring only devices that are already commonly used in the surgical environment.
Collapse
Affiliation(s)
- Orhan Özgüner
- Department of Electrical Engineering and Computer Science, Case Western Reserve University, Cleveland, OH
| | - Thomas Shkurti
- Department of Electrical Engineering and Computer Science, Case Western Reserve University, Cleveland, OH
| | - Siqi Huang
- Department of Electrical Engineering and Computer Science, Case Western Reserve University, Cleveland, OH
| | - Ran Hao
- Department of Electrical Engineering and Computer Science, Case Western Reserve University, Cleveland, OH
| | - Russell C Jackson
- Department of Electrical Engineering and Computer Science, Case Western Reserve University, Cleveland, OH
| | - Wyatt S Newman
- Department of Electrical Engineering and Computer Science, Case Western Reserve University, Cleveland, OH
| | - M Cenk Çavuşoğlu
- Department of Electrical Engineering and Computer Science, Case Western Reserve University, Cleveland, OH
| |
Collapse
|
7
|
Liu X, Plishker W, Kane TD, Geller DA, Lau LW, Tashiro J, Sharma K, Shekhar R. Preclinical evaluation of ultrasound-augmented needle navigation for laparoscopic liver ablation. Int J Comput Assist Radiol Surg 2020; 15:803-810. [PMID: 32323211 DOI: 10.1007/s11548-020-02164-5] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2019] [Accepted: 04/06/2020] [Indexed: 12/17/2022]
Abstract
PURPOSE For laparoscopic ablation to be successful, accurate placement of the needle to the tumor is essential. Laparoscopic ultrasound is an essential tool to guide needle placement, but the ultrasound image is generally presented separately from the laparoscopic image. We aim to evaluate an augmented reality (AR) system which combines laparoscopic ultrasound image, laparoscope video, and the needle trajectory in a unified view. METHODS We created a tissue phantom made of gelatin. Artificial tumors represented by plastic spheres were secured in the gelatin at various depths. The top point of the sphere surface was our target, and its 3D coordinates were known. The participants were invited to perform needle placement with and without AR guidance. Once the participant reported that the needle tip had reached the target, the needle tip location was recorded and compared to the ground truth location of the target, and the difference was the target localization error (TLE). The time of the needle placement was also recorded. We further tested the technical feasibility of the AR system in vivo on a 40-kg swine. RESULTS The AR guidance system was evaluated by two experienced surgeons and two surgical fellows. The users performed needle placement on a total of 26 targets, 13 with AR and 13 without (i.e., the conventional approach). The average TLE for the conventional and the AR approaches was 14.9 mm and 11.1 mm, respectively. The average needle placement time needed for the conventional and AR approaches was 59.4 s and 22.9 s, respectively. For the animal study, ultrasound image and needle trajectory were successfully fused with the laparoscopic video in real time and presented on a single screen for the surgeons. CONCLUSION By providing projected needle trajectory, we believe our AR system can assist the surgeon with more efficient and precise needle placement.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | | | - Timothy D Kane
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - David A Geller
- Department of Surgery, University of Pittsburgh Medical Center, Pittsburgh, PA, USA
| | - Lung W Lau
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - Jun Tashiro
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - Karun Sharma
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA.
- IGI Technologies, Inc., College Park, MD, USA.
| |
Collapse
|
8
|
Xiao G, Bonmati E, Thompson S, Evans J, Hipwell J, Nikitichev D, Gurusamy K, Ourselin S, Hawkes DJ, Davidson B, Clarkson MJ. Electromagnetic tracking in image-guided laparoscopic surgery: Comparison with optical tracking and feasibility study of a combined laparoscope and laparoscopic ultrasound system. Med Phys 2018; 45:5094-5104. [PMID: 30247765 PMCID: PMC6282846 DOI: 10.1002/mp.13210] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2018] [Revised: 09/07/2018] [Accepted: 09/07/2018] [Indexed: 11/23/2022] Open
Abstract
PURPOSE In image-guided laparoscopy, optical tracking is commonly employed, but electromagnetic (EM) systems have been proposed in the literature. In this paper, we provide a thorough comparison of EM and optical tracking systems for use in image-guided laparoscopic surgery and a feasibility study of a combined, EM-tracked laparoscope and laparoscopic ultrasound (LUS) image guidance system. METHODS We first assess the tracking accuracy of a laparoscope with two optical trackers tracking retroreflective markers mounted on the shaft and an EM tracker with the sensor embedded at the proximal end, using a standard evaluation plate. We then use a stylus to test the precision of position measurement and accuracy of distance measurement of the trackers. Finally, we assess the accuracy of an image guidance system comprised of an EM-tracked laparoscope and an EM-tracked LUS probe. RESULTS In the experiment using a standard evaluation plate, the two optical trackers show less jitter in position and orientation measurement than the EM tracker. Also, the optical trackers demonstrate better consistency of orientation measurement within the test volume. However, their accuracy of measuring relative positions decreases significantly with longer distances whereas the EM tracker's performance is stable; at 50 mm distance, the RMS errors for the two optical trackers are 0.210 and 0.233 mm, respectively, and it is 0.214 mm for the EM tracker; at 250 mm distance, the RMS errors for the two optical trackers become 1.031 and 1.178 mm, respectively, while it is 0.367 mm for the EM tracker. In the experiment using the stylus, the two optical trackers have RMS errors of 1.278 and 1.555 mm in localizing the stylus tip, and it is 1.117 mm for the EM tracker. Our prototype of a combined, EM-tracked laparoscope and LUS system using representative calibration methods showed a RMS point localization error of 3.0 mm for the laparoscope and 1.3 mm for the LUS probe, the lager error of the former being predominantly due to the triangulation error when using a narrow-baseline stereo laparoscope. CONCLUSIONS The errors incurred by optical trackers, due to the lever-arm effect and variation in tracking accuracy in the depth direction, would make EM-tracked solutions preferable if the EM sensor is placed at the proximal end of the laparoscope.
Collapse
Affiliation(s)
- Guofang Xiao
- Wellcome/EPSRC Center for Interventional and Surgical SciencesUniversity College LondonLondonUK
- Center for Medical Image ComputingUniversity College LondonLondonUK
- Department of Medical Physics and Biomedical EngineeringUniversity College LondonLondonUK
| | - Ester Bonmati
- Center for Medical Image ComputingUniversity College LondonLondonUK
- Department of Medical Physics and Biomedical EngineeringUniversity College LondonLondonUK
| | - Stephen Thompson
- Wellcome/EPSRC Center for Interventional and Surgical SciencesUniversity College LondonLondonUK
- Center for Medical Image ComputingUniversity College LondonLondonUK
- Department of Medical Physics and Biomedical EngineeringUniversity College LondonLondonUK
| | - Joe Evans
- Department of Medical Physics and Biomedical EngineeringUniversity College LondonLondonUK
| | - John Hipwell
- Center for Medical Image ComputingUniversity College LondonLondonUK
- Department of Medical Physics and Biomedical EngineeringUniversity College LondonLondonUK
| | - Daniil Nikitichev
- Wellcome/EPSRC Center for Interventional and Surgical SciencesUniversity College LondonLondonUK
- Center for Medical Image ComputingUniversity College LondonLondonUK
- Department of Medical Physics and Biomedical EngineeringUniversity College LondonLondonUK
| | - Kurinchi Gurusamy
- Division of Surgery and Interventional ScienceUniversity College LondonLondonUK
| | - Sébastien Ourselin
- Wellcome/EPSRC Center for Interventional and Surgical SciencesUniversity College LondonLondonUK
- Center for Medical Image ComputingUniversity College LondonLondonUK
- Department of Medical Physics and Biomedical EngineeringUniversity College LondonLondonUK
| | - David J. Hawkes
- Wellcome/EPSRC Center for Interventional and Surgical SciencesUniversity College LondonLondonUK
- Center for Medical Image ComputingUniversity College LondonLondonUK
- Department of Medical Physics and Biomedical EngineeringUniversity College LondonLondonUK
| | - Brian Davidson
- Wellcome/EPSRC Center for Interventional and Surgical SciencesUniversity College LondonLondonUK
- Division of Surgery and Interventional ScienceUniversity College LondonLondonUK
| | - Matthew J. Clarkson
- Wellcome/EPSRC Center for Interventional and Surgical SciencesUniversity College LondonLondonUK
- Center for Medical Image ComputingUniversity College LondonLondonUK
- Department of Medical Physics and Biomedical EngineeringUniversity College LondonLondonUK
| |
Collapse
|
9
|
Lau LW, Liu X, Plishker W, Sharma K, Shekhar R, Kane TD. Laparoscopic Liver Resection with Augmented Reality: A Preclinical Experience. J Laparoendosc Adv Surg Tech A 2018; 29:88-93. [PMID: 30192172 DOI: 10.1089/lap.2018.0183] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022] Open
Abstract
INTRODUCTION Intraoperative imaging, such as ultrasound, provides subsurface anatomical information not seen by standard laparoscopy. Currently, information from the two modalities may only be integrated in the surgeon's mind, an often distracting and inefficient task. The desire to improve intraoperative efficiency has guided the development of a novel, augmented reality (AR) laparoscopic system that integrates, in real time, laparoscopic ultrasound (LUS) images with the laparoscopic video. This study shows the initial application of this system for laparoscopic hepatic wedge resection in a porcine model. MATERIALS AND METHODS The AR system consists of a standard laparoscopy setup, LUS scanner, electromagnetic tracking system, and a laptop computer for image fusion. Two liver lesions created in a 40-kg swine by radiofrequency ablation (RFA) were resected using the novel AR system and under standard laparoscopy. RESULTS Anatomical details from the LUS were successfully fused with the laparoscopic video in real time and presented on a single screen for the surgeons. The RFA lesions created were 2.5 and 1 cm in diameter. The 2.5 cm lesion was resected under AR guidance, taking about 7 minutes until completion, while the 1 cm lesion required 3 minutes using standard laparoscopy and ultrasound. Resection margins of both lesions grossly showed noncoagulated liver parenchyma, indicating a negative-margin resection. CONCLUSIONS The use of our AR system in laparoscopic hepatic wedge resection in a swine provided real-time integration of ultrasound image with standard laparoscopy. With more experience and testing, this system can be used for other laparoscopic procedures.
Collapse
Affiliation(s)
- Lung W Lau
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia.,2 Department of Surgery, University Hospitals Cleveland Medical Center, Cleveland, Ohio
| | - Xinyang Liu
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia
| | | | - Karun Sharma
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia
| | - Raj Shekhar
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia.,3 IGI Technologies, Inc., College Park, Maryland
| | - Timothy D Kane
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia
| |
Collapse
|
10
|
Liu X, Rice CE, Shekhar R. Fast calibration of electromagnetically tracked oblique-viewing rigid endoscopes. Int J Comput Assist Radiol Surg 2017. [PMID: 28623479 DOI: 10.1007/s11548-017-1623-4] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
PURPOSE The oblique-viewing (i.e., angled) rigid endoscope is a commonly used tool in conventional endoscopic surgeries. The relative rotation between its two moveable parts, the telescope and the camera head, creates a rotation offset between the actual and the projection of an object in the camera image. A calibration method tailored to compensate such offset is needed. METHODS We developed a fast calibration method for oblique-viewing rigid endoscopes suitable for clinical use. In contrast to prior approaches based on optical tracking, we used electromagnetic (EM) tracking as the external tracking hardware to improve compactness and practicality. Two EM sensors were mounted on the telescope and the camera head, respectively, with considerations to minimize EM tracking errors. Single-image calibration was incorporated into the method, and a sterilizable plate, laser-marked with the calibration pattern, was also developed. Furthermore, we proposed a general algorithm to estimate the rotation center in the camera image. Formulas for updating the camera matrix in terms of clockwise and counterclockwise rotations were also developed. RESULTS The proposed calibration method was validated using a conventional [Formula: see text], 5-mm laparoscope. Freehand calibrations were performed using the proposed method, and the calibration time averaged 2 min and 8 s. The calibration accuracy was evaluated in a simulated clinical setting with several surgical tools present in the magnetic field of EM tracking. The root-mean-square re-projection error averaged 4.9 pixel (range 2.4-8.5 pixel, with image resolution of [Formula: see text] for rotation angles ranged from [Formula: see text] to [Formula: see text]. CONCLUSIONS We developed a method for fast and accurate calibration of oblique-viewing rigid endoscopes. The method was also designed to be performed in the operating room and will therefore support clinical translation of many emerging endoscopic computer-assisted surgical systems.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC, 20010, USA
| | - Christina E Rice
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC, 20010, USA.,Department of Mechanical and Aerospace Engineering, Princeton University, Princeton, NJ, 08544, USA
| | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC, 20010, USA.
| |
Collapse
|
11
|
Jaeger HA, Franz AM, O’Donoghue K, Seitel A, Trauzettel F, Maier-Hein L, Cantillon-Murphy P. Anser EMT: the first open-source electromagnetic tracking platform for image-guided interventions. Int J Comput Assist Radiol Surg 2017; 12:1059-1067. [DOI: 10.1007/s11548-017-1568-7] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2017] [Accepted: 03/15/2017] [Indexed: 10/19/2022]
|
12
|
Liu X, Kang S, Plishker W, Zaki G, Kane TD, Shekhar R. Laparoscopic stereoscopic augmented reality: toward a clinically viable electromagnetic tracking solution. J Med Imaging (Bellingham) 2016; 3:045001. [PMID: 27752522 DOI: 10.1117/1.jmi.3.4.045001] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2016] [Accepted: 09/08/2016] [Indexed: 11/14/2022] Open
Abstract
The purpose of this work was to develop a clinically viable laparoscopic augmented reality (AR) system employing stereoscopic (3-D) vision, laparoscopic ultrasound (LUS), and electromagnetic (EM) tracking to achieve image registration. We investigated clinically feasible solutions to mount the EM sensors on the 3-D laparoscope and the LUS probe. This led to a solution of integrating an externally attached EM sensor near the imaging tip of the LUS probe, only slightly increasing the overall diameter of the probe. Likewise, a solution for mounting an EM sensor on the handle of the 3-D laparoscope was proposed. The spatial image-to-video registration accuracy of the AR system was measured to be [Formula: see text] and [Formula: see text] for the left- and right-eye channels, respectively. The AR system contributed 58-ms latency to stereoscopic visualization. We further performed an animal experiment to demonstrate the use of the system as a visualization approach for laparoscopic procedures. In conclusion, we have developed an integrated, compact, and EM tracking-based stereoscopic AR visualization system, which has the potential for clinical use. The system has been demonstrated to achieve clinically acceptable accuracy and latency. This work is a critical step toward clinical translation of AR visualization for laparoscopic procedures.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation , Children's National Health System, 111 Michigan Avenue NW, Washington, DC 20010, United States
| | - Sukryool Kang
- Sheikh Zayed Institute for Pediatric Surgical Innovation , Children's National Health System, 111 Michigan Avenue NW, Washington, DC 20010, United States
| | - William Plishker
- IGI Technologies, Inc. , 387 Technology Drive #3110D, College Park, Maryland 20742, United States
| | - George Zaki
- IGI Technologies, Inc. , 387 Technology Drive #3110D, College Park, Maryland 20742, United States
| | - Timothy D Kane
- Sheikh Zayed Institute for Pediatric Surgical Innovation , Children's National Health System, 111 Michigan Avenue NW, Washington, DC 20010, United States
| | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC 20010, United States; IGI Technologies, Inc., 387 Technology Drive #3110D, College Park, Maryland 20742, United States
| |
Collapse
|