1
|
Higgins E, Crowley D, van den Bosch C, Cantillon-Murphy P. Distortion-Free Magnetic Tracking of Metal Instruments in Image-Guided Interventions. SENSORS (BASEL, SWITZERLAND) 2024; 24:5364. [PMID: 39205058 PMCID: PMC11360588 DOI: 10.3390/s24165364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/18/2024] [Revised: 08/13/2024] [Accepted: 08/16/2024] [Indexed: 09/04/2024]
Abstract
Electromagnetic tracking (EMT) can benefit image-guided interventions in cases where line of sight is unavailable. However, EMT can suffer from electromagnetic distortion in the presence of metal instruments. Metal instruments are widely used in laparoscopic surgery, ENT surgery, arthroscopy and many other clinical applications. In this work, we investigate the feasibility of tracking such metal instruments by placing the inductive sensor within the instrument shaft. We propose a magnetostatic model of the field within the instrument, and verify the results experimentally for frequencies from 6 kHz to 60 kHz. The impact of the instrument's dimensions, conductivity and transmitting field frequency is quantified for ranges representative of typical metal instruments used in image-guided interventions. We then performed tracking using the open-source Anser EMT system and quantify the error caused by the presence of the rod as a function of the frequency of the eight emitting coils for the system. The work clearly demonstrates why smaller tool diameters (less than 8 mm) are less susceptible to distortion, as well as identifying optimal frequencies (1 kHz to 2 kHz) for transmitter design to minimise for distortion in larger instruments.
Collapse
Affiliation(s)
- Eoin Higgins
- Tyndall National Institute, Lee Maltings, Dyke Parade, T12 R5CP Cork, Ireland; (D.C.); (P.C.-M.)
- School of Engineering, University College Cork, College Road, T12 K8AF Cork, Ireland
| | - Daragh Crowley
- Tyndall National Institute, Lee Maltings, Dyke Parade, T12 R5CP Cork, Ireland; (D.C.); (P.C.-M.)
- School of Engineering, University College Cork, College Road, T12 K8AF Cork, Ireland
| | - Christian van den Bosch
- Tyndall National Institute, Lee Maltings, Dyke Parade, T12 R5CP Cork, Ireland; (D.C.); (P.C.-M.)
- School of Engineering, University College Cork, College Road, T12 K8AF Cork, Ireland
| | - Pádraig Cantillon-Murphy
- Tyndall National Institute, Lee Maltings, Dyke Parade, T12 R5CP Cork, Ireland; (D.C.); (P.C.-M.)
- School of Engineering, University College Cork, College Road, T12 K8AF Cork, Ireland
| |
Collapse
|
2
|
Pruitt K, Johnson B, Gahan J, Ma L, Fei B. A High-Speed Hyperspectral Laparoscopic Imaging System. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2023; 12466:1246608. [PMID: 38524190 PMCID: PMC10961180 DOI: 10.1117/12.2653922] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/26/2024]
Abstract
Minimally invasive surgery (MIS) has expanded broadly in the field of abdominal and pelvic surgery. Laparoscopic and robotic surgery has improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery across various abdominal procedures. The goal of this study is to establish the feasibility of implementing high-speed hyperspectral imaging into a standard laparoscopic setup and exploring its benefit to common intracorporeal procedures. A hyperspectral laparoscopic imaging system was constructed using a customized hyperspectral camera alongside a standard rigid laparoscope and was validated for both spectral and spatial accuracy. Demosaicing methods were investigated for improved full-resolution visualization. Hyperspectral cameras with different spectral ranges were considered and compared with one another alongside two different light sources to determine the most effective configuration. Finally, different porcine tissues were imaged ex-vivo to test the capabilities of the system and spectral footprints of the various tissues were extracted. The tissue was also imaged in a phantom to simulate the system's use in MIS. The results demonstrated a hyperspectral laparoscopic imaging system that could provide quantitative, diagnostic information while not disrupting normal workflow nor adding excessive weight to the laparoscopic setup. The high-speed hyperspectral laparoscopic imaging system can have immediate applications in image-guided surgery.
Collapse
Affiliation(s)
- Kelden Pruitt
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
| | - Brett Johnson
- University of Texas Southwestern Medical Center, Department of Urology, Dallas, TX
| | - Jeffrey Gahan
- University of Texas Southwestern Medical Center, Department of Urology, Dallas, TX
| | - Ling Ma
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
| | - Baowei Fei
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- University of Texas Southwestern Medical Center, Department of Radiology, Dallas, TX
| |
Collapse
|
3
|
Shen X(S, Gao J, Li M, Zhou C, Hu S, He M, Zhuang W. Toward immersive communications in 6G. FRONTIERS IN COMPUTER SCIENCE 2023. [DOI: 10.3389/fcomp.2022.1068478] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/12/2023] Open
Abstract
The sixth generation (6G) networks are expected to enable immersive communications and bridge the physical and the virtual worlds. Integrating extended reality, holography, and haptics, immersive communications will revolutionize how people work, entertain, and communicate by enabling lifelike interactions. However, the unprecedented demand for data transmission rate and the stringent requirements on latency and reliability create challenges for 6G networks to support immersive communications. In this survey article, we present the prospect of immersive communications and investigate emerging solutions to the corresponding challenges for 6G. First, we introduce use cases of immersive communications, in the fields of entertainment, education, and healthcare. Second, we present the concepts of immersive communications, including extended reality, haptic communication, and holographic communication, their basic implementation procedures, and their requirements on networks in terms of transmission rate, latency, and reliability. Third, we summarize the potential solutions to addressing the challenges from the aspects of communication, computing, and networking. Finally, we discuss future research directions and conclude this study.
Collapse
|
4
|
Zhu T, Jiang S, Yang Z, Zhou Z, Li Y, Ma S, Zhuo J. A neuroendoscopic navigation system based on dual-mode augmented reality for minimally invasive surgical treatment of hypertensive intracerebral hemorrhage. Comput Biol Med 2022; 140:105091. [PMID: 34872012 DOI: 10.1016/j.compbiomed.2021.105091] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2021] [Revised: 11/23/2021] [Accepted: 11/26/2021] [Indexed: 01/01/2023]
Abstract
BACKGROUND AND OBJECTIVE Hypertensive intracerebral hemorrhage is characterized by a high rate of morbidity, mortality, disability and recurrence. Neuroendoscopy has been utilized for treatment as an advanced technology. However, traditional neuroendoscopy allows professionals to see only tissue surfaces, and the field of vision is limited, which cannot provide spatial guidance. In this study, an AR-based neuroendoscopic navigation system is proposed to assist surgeons in locating and clearing hematoma. METHODS The neuroendoscope can be registered through the vector closed loop algorithm. The single-shot method is designed to register medical images with patients precisely. Real-time AR is realized based on video stream fusion. Dual-mode AR navigation is proposed to provide comprehensive guidance from catheter implantation to hematoma removal. A series of experiments is designed to validate the accuracy and significance of this system. RESULTS The average root mean square error of the registration between medical images and patients is 0.784 mm, and the variance is 0.1426 mm. The pixel mismatching degrees are less than 1% in different AR modes. In catheter implantation experiments, the average error of distance is 1.28 mm, and the variance is 0.43 mm, while the average error of angles is 1.34°, and the variance is 0.45°. Comparative experiments are also conducted to evaluate the feasibility of this system. CONCLUSION This system can provide stereo images with depth information fused with patients to guide surgeons to locate targets and remove hematoma. It has been validated to have high accuracy and feasibility.
Collapse
Affiliation(s)
- Tao Zhu
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Shan Jiang
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China.
| | - Zhiyong Yang
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Zeyang Zhou
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Yuhua Li
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Shixing Ma
- School of Mechanical Engineering, Tianjin University, Tianjin, 300350, China
| | - Jie Zhuo
- Department of Neurosurgery, Tianjin Huanhu Hospital, Tianjin, 300200, China
| |
Collapse
|
5
|
HoloUS: Augmented reality visualization of live ultrasound images using HoloLens for ultrasound-guided procedures. Int J Comput Assist Radiol Surg 2021; 17:385-391. [PMID: 34817764 DOI: 10.1007/s11548-021-02526-7] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2021] [Accepted: 10/20/2021] [Indexed: 10/19/2022]
Abstract
PURPOSE Microsoft HoloLens is a pair of augmented reality (AR) smart glasses that could improve the intraprocedural visualization of ultrasound-guided procedures. With the wearable HoloLens headset, an ultrasound image can be virtually rendered and registered with the ultrasound transducer and placed directly in the practitioner's field of view. METHODS A custom application, called HoloUS, was developed using the HoloLens and a portable ultrasound machine connected through a wireless network. A custom 3D-printed case with an AR-pattern for the ultrasound transducer permitted ultrasound image tracking and registration. Voice controls on the HoloLens supported the scaling and movement of the ultrasound image as desired. The ultrasound images were streamed and displayed in real-time. A user study was performed to assess the effectiveness of the HoloLens as an alternative display platform. Novices and experts were timed on tasks involving targeting simulated vessels using a needle in a custom phantom. RESULTS Technical characterization of the HoloUS app was conducted using frame rate, tracking accuracy, and latency as performance metrics. The app ran at 25 frames/s, had an 80-ms latency, and could track the transducer with an average reprojection error of 0.0435 pixels. With AR visualization, the novices' times improved by 17% but the experts' times decreased slightly by 5%, which may reflect the experts' training and experience bias. CONCLUSION The HoloUS application was found to enhance user experience and simplify hand-eye coordination. By eliminating the need to alternately observe the patient and the ultrasound images presented on a separate monitor, the proposed AR application has the potential to improve efficiency and effectiveness of ultrasound-guided procedures.
Collapse
|
6
|
Yang A, Gao F. Effect of dexmedetomidine combined with propofol on stress response, hemodynamics, and postoperative complications in patients undergoing laparoscopic cholecystectomy. Am J Transl Res 2021; 13:11824-11832. [PMID: 34786111 PMCID: PMC8581921] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2020] [Accepted: 02/19/2021] [Indexed: 06/13/2023]
Abstract
OBJECTIVE This research was designed to probe the effect of dexmedetomidine combined with propofol on stress response, hemodynamics, and postoperative complications in patients undergoing laparoscopic cholecystectomy. METHODS Altogether 144 patients who underwent laparoscopic cholecystectomy in the Beibei Traditional Chinese Medical Hospital, the Sixth People's Hospital of Chongqing from January 2018 to July 2020 were research subjects. The control group (CG) (n=68) received propofol continuous pumping sedation, while the research group (RG) (n=76) was given dexmedetomidine combined with propofol continuous pumping sedation. The quality of postoperative recovery and incidence of postoperative complications of the two groups were observed and compared. The hemodynamic indexes (HR, SpO2 and MAP) before anesthesia induction (T0), tracheal intubation (T1), at the commencement of operation (T2), at the end of operation (T3), and extubation (T4) were observed. The stress response indexes (Cortisol, ACTH and norepinephrine, NE) were monitored, and the scores of pain, anxiety and cognitive dysfunction before and after operation were evaluated. RESULTS Compared with the CG, the incidence of postoperative complications in the RG was lower, and the quality of postoperative recovery (time of breathing recovery, eye opening, consciousness and extubation) was better. Besides, the hemodynamic indexes of the RG were more stable, and the levels of stress indexes Cortisol, ACTH, and NE were lower. The RG had VAS and SAS scores that were lower, and MMSE scores were higher. CONCLUSION Dexmedetomidine combined with propofol can effectively alleviate the stress response of patients undergoing laparoscopic cholecystectomy, stabilize perioperative hemodynamics, and reduce postoperative complications.
Collapse
Affiliation(s)
- Anbo Yang
- Department of Anesthesiology, Beibei Traditional Chinese Medical HospitalChongqing 400700, China
| | - Feng Gao
- Department of Anesthesiology, The Sixth People’s Hospital of ChongqingChongqing 400060, China
| |
Collapse
|
7
|
Zhao Z, Poyhonen J, Chen Cai X, Sophie Woodley Hooper F, Ma Y, Hu Y, Ren H, Song W, Tsz Ho Tse Z. Augmented reality technology in image-guided therapy: State-of-the-art review. Proc Inst Mech Eng H 2021; 235:1386-1398. [PMID: 34304631 PMCID: PMC8573682 DOI: 10.1177/09544119211034357] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
Abstract
Image-guided therapies have been on the rise in recent years as they can achieve higher accuracy and are less invasive than traditional methods. By combining augmented reality technology with image-guided therapy, more organs, and tissues can be observed by surgeons to improve surgical accuracy. In this review, 233 publications (dated from 2015 to 2020) on the design and application of augmented reality-based systems for image-guided therapy, including both research prototypes and commercial products, were considered for review. Based on their functions and applications. Sixteen studies were selected. The engineering specifications and applications were analyzed and summarized for each study. Finally, future directions and existing challenges in the field were summarized and discussed.
Collapse
Affiliation(s)
- Zhuo Zhao
- School of Electrical and Computer Engineering, University of Georgia, Athens, GA, USA
| | - Jasmin Poyhonen
- Department of Electronic Engineering, University of York, York, UK
| | - Xin Chen Cai
- Department of Electronic Engineering, University of York, York, UK
| | | | - Yangmyung Ma
- Hull York Medical School, University of York, York, UK
| | - Yihua Hu
- Department of Electronic Engineering, University of York, York, UK
| | - Hongliang Ren
- Department of Electronic Engineering The Chinese University of Hong Kong (CUHK), Hong Kong, China
| | - Wenzhan Song
- School of Electrical and Computer Engineering, University of Georgia, Athens, GA, USA
| | - Zion Tsz Ho Tse
- Department of Electronic Engineering, University of York, York, UK
| |
Collapse
|
8
|
Iqbal H, Tatti F, Rodriguez Y Baena F. Augmented reality in robotic assisted orthopaedic surgery: A pilot study. J Biomed Inform 2021; 120:103841. [PMID: 34146717 DOI: 10.1016/j.jbi.2021.103841] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2020] [Revised: 06/11/2021] [Accepted: 06/14/2021] [Indexed: 01/18/2023]
Abstract
BACKGROUND The research and development of augmented-reality (AR) technologies in surgical applications has seen an evolution of the traditional user-interfaces (UI) utilised by clinicians when conducting robot-assisted orthopaedic surgeries. The typical UI for such systems relies on surgeons managing 3D medical imaging data in the 2D space of a touchscreen monitor, located away from the operating site. Conversely, AR can provide a composite view overlaying the real surgical scene with co-located virtual holographic representations of medical data, leading to a more immersive and intuitive operator experience. MATERIALS AND METHODS This work explores the integration of AR within an orthopaedic setting by capturing and replicating the UI of an existing surgical robot within an AR head-mounted display worn by the clinician. The resulting mixed-reality workflow enabled users to simultaneously view the operating-site and real-time holographic operating informatics when carrying out a robot-assisted patellofemoral-arthroplasty (PFA). Ten surgeons were recruited to test the impact of the AR system on procedure completion time and operating surface roughness. RESULTS AND DISCUSSION The integration of AR did not appear to require subjects to significantly alter their surgical techniques, which was demonstrated by non-significant changes to the study's clinical metrics, with a statistically insignificant mean increase in operating time (+0.778 s, p = 0.488) and a statistically insignificant change in mean surface roughness (p = 0.274). Additionally, a post-operative survey indicated a positive consensus on the usability of the AR system without incurring noticeable physical distress such as eyestrain or fatigue. CONCLUSIONS Overall, these study results demonstrated a successful integration of AR technologies within the framework of an existing robot-assisted surgical platform with no significant negative effects in two quantitative metrics of surgical performance, and a positive outcome relating to user-centric and ergonomic evaluation criteria.
Collapse
Affiliation(s)
- Hisham Iqbal
- Mechatronics in Medicine Laboratory, Imperial College London, London, UK.
| | - Fabio Tatti
- Mechatronics in Medicine Laboratory, Imperial College London, London, UK
| | | |
Collapse
|
9
|
Liu X, Plishker W, Shekhar R. Hybrid electromagnetic-ArUco tracking of laparoscopic ultrasound transducer in laparoscopic video. J Med Imaging (Bellingham) 2021; 8:015001. [PMID: 33585664 PMCID: PMC7857492 DOI: 10.1117/1.jmi.8.1.015001] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2020] [Accepted: 01/12/2021] [Indexed: 11/14/2022] Open
Abstract
Purpose: The purpose of this work was to develop a new method of tracking a laparoscopic ultrasound (LUS) transducer in laparoscopic video by combining the hardware [e.g., electromagnetic (EM)] and the computer vision-based (e.g., ArUco) tracking methods. Approach: We developed a special tracking mount for the imaging tip of the LUS transducer. The mount incorporated an EM sensor and an ArUco pattern registered to it. The hybrid method used ArUco tracking for ArUco-success frames (i.e., frames where ArUco succeeds in detecting the pattern) and used corrected EM tracking for the ArUco-failure frames. The corrected EM tracking result was obtained by applying correction matrices to the original EM tracking result. The correction matrices were calculated in previous ArUco-success frames by comparing the ArUco result and the original EM tracking result. Results: We performed phantom and animal studies to evaluate the performance of our hybrid tracking method. The corrected EM tracking results showed significant improvements over the original EM tracking results. In the animal study, 59.2% frames were ArUco-success frames. For the ArUco-failure frames, mean reprojection errors for the original EM tracking method and for the corrected EM tracking method were 30.8 pixel and 10.3 pixel, respectively. Conclusions: The new hybrid method is more reliable than using ArUco tracking alone and more accurate and practical than using EM tracking alone for tracking the LUS transducer in the laparoscope camera image. The proposed method has the potential to significantly improve tracking performance for LUS-based augmented reality applications.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, United States
| | | | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, United States.,IGI Technologies, Inc., Silver Spring, Maryland, United States
| |
Collapse
|
10
|
Advances and Trends in Pediatric Minimally Invasive Surgery. J Clin Med 2020; 9:jcm9123999. [PMID: 33321836 PMCID: PMC7764454 DOI: 10.3390/jcm9123999] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2020] [Revised: 11/28/2020] [Accepted: 12/03/2020] [Indexed: 12/11/2022] Open
Abstract
As many meta-analyses comparing pediatric minimally invasive to open surgery can be found in the literature, the aim of this review is to summarize the current state of minimally invasive pediatric surgery and specifically focus on the trends and developments which we expect in the upcoming years. Print and electronic databases were systematically searched for specific keywords, and cross-link searches with references found in the literature were added. Full-text articles were obtained, and eligibility criteria were applied independently. Pediatric minimally invasive surgery is a wide field, ranging from minimally invasive fetal surgery over microlaparoscopy in newborns to robotic surgery in adolescents. New techniques and devices, like natural orifice transluminal endoscopic surgery (NOTES), single-incision and endoscopic surgery, as well as the artificial uterus as a backup for surgery in preterm fetuses, all contribute to the development of less invasive procedures for children. In spite of all promising technical developments which will definitely change the way pediatric surgeons will perform minimally invasive procedures in the upcoming years, one must bear in mind that only hard data of prospective randomized controlled and double-blind trials can validate whether these techniques and devices really improve the surgical outcome of our patients.
Collapse
|
11
|
Liu X, Plishker W, Kane TD, Geller DA, Lau LW, Tashiro J, Sharma K, Shekhar R. Preclinical evaluation of ultrasound-augmented needle navigation for laparoscopic liver ablation. Int J Comput Assist Radiol Surg 2020; 15:803-810. [PMID: 32323211 DOI: 10.1007/s11548-020-02164-5] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2019] [Accepted: 04/06/2020] [Indexed: 12/17/2022]
Abstract
PURPOSE For laparoscopic ablation to be successful, accurate placement of the needle to the tumor is essential. Laparoscopic ultrasound is an essential tool to guide needle placement, but the ultrasound image is generally presented separately from the laparoscopic image. We aim to evaluate an augmented reality (AR) system which combines laparoscopic ultrasound image, laparoscope video, and the needle trajectory in a unified view. METHODS We created a tissue phantom made of gelatin. Artificial tumors represented by plastic spheres were secured in the gelatin at various depths. The top point of the sphere surface was our target, and its 3D coordinates were known. The participants were invited to perform needle placement with and without AR guidance. Once the participant reported that the needle tip had reached the target, the needle tip location was recorded and compared to the ground truth location of the target, and the difference was the target localization error (TLE). The time of the needle placement was also recorded. We further tested the technical feasibility of the AR system in vivo on a 40-kg swine. RESULTS The AR guidance system was evaluated by two experienced surgeons and two surgical fellows. The users performed needle placement on a total of 26 targets, 13 with AR and 13 without (i.e., the conventional approach). The average TLE for the conventional and the AR approaches was 14.9 mm and 11.1 mm, respectively. The average needle placement time needed for the conventional and AR approaches was 59.4 s and 22.9 s, respectively. For the animal study, ultrasound image and needle trajectory were successfully fused with the laparoscopic video in real time and presented on a single screen for the surgeons. CONCLUSION By providing projected needle trajectory, we believe our AR system can assist the surgeon with more efficient and precise needle placement.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | | | - Timothy D Kane
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - David A Geller
- Department of Surgery, University of Pittsburgh Medical Center, Pittsburgh, PA, USA
| | - Lung W Lau
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - Jun Tashiro
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - Karun Sharma
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA
| | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Hospital, Washington, DC, USA.
- IGI Technologies, Inc., College Park, MD, USA.
| |
Collapse
|
12
|
Luo H, Yin D, Zhang S, Xiao D, He B, Meng F, Zhang Y, Cai W, He S, Zhang W, Hu Q, Guo H, Liang S, Zhou S, Liu S, Sun L, Guo X, Fang C, Liu L, Jia F. Augmented reality navigation for liver resection with a stereoscopic laparoscope. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2020; 187:105099. [PMID: 31601442 DOI: 10.1016/j.cmpb.2019.105099] [Citation(s) in RCA: 37] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/10/2019] [Revised: 08/14/2019] [Accepted: 09/27/2019] [Indexed: 06/10/2023]
Abstract
OBJECTIVE Understanding the three-dimensional (3D) spatial position and orientation of vessels and tumor(s) is vital in laparoscopic liver resection procedures. Augmented reality (AR) techniques can help surgeons see the patient's internal anatomy in conjunction with laparoscopic video images. METHOD In this paper, we present an AR-assisted navigation system for liver resection based on a rigid stereoscopic laparoscope. The stereo image pairs from the laparoscope are used by an unsupervised convolutional network (CNN) framework to estimate depth and generate an intraoperative 3D liver surface. Meanwhile, 3D models of the patient's surgical field are segmented from preoperative CT images using V-Net architecture for volumetric image data in an end-to-end predictive style. A globally optimal iterative closest point (Go-ICP) algorithm is adopted to register the pre- and intraoperative models into a unified coordinate space; then, the preoperative 3D models are superimposed on the live laparoscopic images to provide the surgeon with detailed information about the subsurface of the patient's anatomy, including tumors, their resection margins and vessels. RESULTS The proposed navigation system is tested on four laboratory ex vivo porcine livers and five operating theatre in vivo porcine experiments to validate its accuracy. The ex vivo and in vivo reprojection errors (RPE) are 6.04 ± 1.85 mm and 8.73 ± 2.43 mm, respectively. CONCLUSION AND SIGNIFICANCE Both the qualitative and quantitative results indicate that our AR-assisted navigation system shows promise and has the potential to be highly useful in clinical practice.
Collapse
Affiliation(s)
- Huoling Luo
- Research Lab for Medical Imaging and Digital Surgery, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China; Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences, Shenzhen, China
| | - Dalong Yin
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China; Department of Hepatobiliary Surgery, Shengli Hospital Affiliated to University of Science and Technology of China, Hefei, China
| | - Shugeng Zhang
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China; Department of Hepatobiliary Surgery, Shengli Hospital Affiliated to University of Science and Technology of China, Hefei, China
| | - Deqiang Xiao
- Research Lab for Medical Imaging and Digital Surgery, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China; Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences, Shenzhen, China
| | - Baochun He
- Research Lab for Medical Imaging and Digital Surgery, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Fanzheng Meng
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Yanfang Zhang
- Department of Interventional Radiology, Shenzhen People's Hospital, Shenzhen, China
| | - Wei Cai
- Department of Hepatobiliary Surgery, Zhujiang Hospital, Southern Medical University, Guangzhou, China
| | - Shenghao He
- Research Lab for Medical Imaging and Digital Surgery, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Wenyu Zhang
- Department of Hepatobiliary Surgery, Zhujiang Hospital, Southern Medical University, Guangzhou, China
| | - Qingmao Hu
- Research Lab for Medical Imaging and Digital Surgery, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China; Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences, Shenzhen, China
| | - Hongrui Guo
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Shuhang Liang
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Shuo Zhou
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Shuxun Liu
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Linmao Sun
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Xiao Guo
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Chihua Fang
- Department of Hepatobiliary Surgery, Zhujiang Hospital, Southern Medical University, Guangzhou, China
| | - Lianxin Liu
- Department of Hepatobiliary Surgery, First Affiliated Hospital of Harbin Medical University, Harbin, China; Department of Hepatobiliary Surgery, Shengli Hospital Affiliated to University of Science and Technology of China, Hefei, China.
| | - Fucang Jia
- Research Lab for Medical Imaging and Digital Surgery, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China; Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences, Shenzhen, China.
| |
Collapse
|
13
|
Lee EJ, Plishker W, Liu X, Bhattacharyya SS, Shekhar R. Weakly supervised segmentation for real-time surgical tool tracking. Healthc Technol Lett 2019; 6:231-236. [PMID: 32038863 PMCID: PMC6952260 DOI: 10.1049/htl.2019.0083] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2019] [Accepted: 10/02/2019] [Indexed: 12/14/2022] Open
Abstract
Surgical tool tracking has a variety of applications in different surgical scenarios. Electromagnetic (EM) tracking can be utilised for tool tracking, but the accuracy is often limited by magnetic interference. Vision-based methods have also been suggested; however, tracking robustness is limited by specular reflection, occlusions, and blurriness observed in the endoscopic image. Recently, deep learning-based methods have shown competitive performance on segmentation and tracking of surgical tools. The main bottleneck of these methods lies in acquiring a sufficient amount of pixel-wise, annotated training data, which demands substantial labour costs. To tackle this issue, the authors propose a weakly supervised method for surgical tool segmentation and tracking based on hybrid sensor systems. They first generate semantic labellings using EM tracking and laparoscopic image processing concurrently. They then train a light-weight deep segmentation network to obtain a binary segmentation mask that enables tool tracking. To the authors' knowledge, the proposed method is the first to integrate EM tracking and laparoscopic image processing for generation of training labels. They demonstrate that their framework achieves accurate, automatic tool segmentation (i.e. without any manual labelling of the surgical tool to be tracked) and robust tool tracking in laparoscopic image sequences.
Collapse
Affiliation(s)
- Eung-Joo Lee
- Department of Electrical and Computer Engineering and the Institute for Advanced Computer Studies, University of Maryland, College Park, MD, USA.,IGI Technologies, Inc., College Park, MD, USA
| | | | - Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, the Children's National Medical Center, Washington DC, USA
| | - Shuvra S Bhattacharyya
- Department of Electrical and Computer Engineering and the Institute for Advanced Computer Studies, University of Maryland, College Park, MD, USA
| | - Raj Shekhar
- IGI Technologies, Inc., College Park, MD, USA.,Sheikh Zayed Institute for Pediatric Surgical Innovation, the Children's National Medical Center, Washington DC, USA
| |
Collapse
|
14
|
Liu X, Kane TD, Shekhar R. GPS Laparoscopic Ultrasound: Embedding an Electromagnetic Sensor in a Laparoscopic Ultrasound Transducer. ULTRASOUND IN MEDICINE & BIOLOGY 2019; 45:989-997. [PMID: 30709691 DOI: 10.1016/j.ultrasmedbio.2018.11.014] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/10/2018] [Revised: 11/21/2018] [Accepted: 11/29/2018] [Indexed: 06/09/2023]
Abstract
Tracking the location and orientation of a laparoscopic ultrasound (LUS) transducer is a prerequisite in many surgical visualization and navigation applications. Electromagnetic (EM) tracking is a preferred method to track an LUS transducer with an articulating imaging tip. The conventional approach to integrating EM tracking with LUS is to attach an EM sensor on the outer surface of the imaging tip (external setup), which is not ideal for routine clinical use. In this work, we embedded an EM sensor inside a standard LUS transducer. We found that ultrasound image quality and the four-way articulation function of the transducer were not affected by this sensor integration. Furthermore, we found that the tracking accuracy of our integrated transducer was comparable to that of the external setup. An animal study conducted using the developed transducer suggests that an internally embedded EM sensor is a clinically more viable approach, and may be the future of tracking an articulating LUS transducer.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Medical Center, Washington, DC, USA
| | - Timothy D Kane
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Medical Center, Washington, DC, USA
| | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Medical Center, Washington, DC, USA.
| |
Collapse
|
15
|
Lau LW, Liu X, Plishker W, Sharma K, Shekhar R, Kane TD. Laparoscopic Liver Resection with Augmented Reality: A Preclinical Experience. J Laparoendosc Adv Surg Tech A 2018; 29:88-93. [PMID: 30192172 DOI: 10.1089/lap.2018.0183] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022] Open
Abstract
INTRODUCTION Intraoperative imaging, such as ultrasound, provides subsurface anatomical information not seen by standard laparoscopy. Currently, information from the two modalities may only be integrated in the surgeon's mind, an often distracting and inefficient task. The desire to improve intraoperative efficiency has guided the development of a novel, augmented reality (AR) laparoscopic system that integrates, in real time, laparoscopic ultrasound (LUS) images with the laparoscopic video. This study shows the initial application of this system for laparoscopic hepatic wedge resection in a porcine model. MATERIALS AND METHODS The AR system consists of a standard laparoscopy setup, LUS scanner, electromagnetic tracking system, and a laptop computer for image fusion. Two liver lesions created in a 40-kg swine by radiofrequency ablation (RFA) were resected using the novel AR system and under standard laparoscopy. RESULTS Anatomical details from the LUS were successfully fused with the laparoscopic video in real time and presented on a single screen for the surgeons. The RFA lesions created were 2.5 and 1 cm in diameter. The 2.5 cm lesion was resected under AR guidance, taking about 7 minutes until completion, while the 1 cm lesion required 3 minutes using standard laparoscopy and ultrasound. Resection margins of both lesions grossly showed noncoagulated liver parenchyma, indicating a negative-margin resection. CONCLUSIONS The use of our AR system in laparoscopic hepatic wedge resection in a swine provided real-time integration of ultrasound image with standard laparoscopy. With more experience and testing, this system can be used for other laparoscopic procedures.
Collapse
Affiliation(s)
- Lung W Lau
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia.,2 Department of Surgery, University Hospitals Cleveland Medical Center, Cleveland, Ohio
| | - Xinyang Liu
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia
| | | | - Karun Sharma
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia
| | - Raj Shekhar
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia.,3 IGI Technologies, Inc., College Park, Maryland
| | - Timothy D Kane
- 1 Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, Washington, District of Columbia
| |
Collapse
|
16
|
Liu X, Rice CE, Shekhar R. Fast calibration of electromagnetically tracked oblique-viewing rigid endoscopes. Int J Comput Assist Radiol Surg 2017. [PMID: 28623479 DOI: 10.1007/s11548-017-1623-4] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
PURPOSE The oblique-viewing (i.e., angled) rigid endoscope is a commonly used tool in conventional endoscopic surgeries. The relative rotation between its two moveable parts, the telescope and the camera head, creates a rotation offset between the actual and the projection of an object in the camera image. A calibration method tailored to compensate such offset is needed. METHODS We developed a fast calibration method for oblique-viewing rigid endoscopes suitable for clinical use. In contrast to prior approaches based on optical tracking, we used electromagnetic (EM) tracking as the external tracking hardware to improve compactness and practicality. Two EM sensors were mounted on the telescope and the camera head, respectively, with considerations to minimize EM tracking errors. Single-image calibration was incorporated into the method, and a sterilizable plate, laser-marked with the calibration pattern, was also developed. Furthermore, we proposed a general algorithm to estimate the rotation center in the camera image. Formulas for updating the camera matrix in terms of clockwise and counterclockwise rotations were also developed. RESULTS The proposed calibration method was validated using a conventional [Formula: see text], 5-mm laparoscope. Freehand calibrations were performed using the proposed method, and the calibration time averaged 2 min and 8 s. The calibration accuracy was evaluated in a simulated clinical setting with several surgical tools present in the magnetic field of EM tracking. The root-mean-square re-projection error averaged 4.9 pixel (range 2.4-8.5 pixel, with image resolution of [Formula: see text] for rotation angles ranged from [Formula: see text] to [Formula: see text]. CONCLUSIONS We developed a method for fast and accurate calibration of oblique-viewing rigid endoscopes. The method was also designed to be performed in the operating room and will therefore support clinical translation of many emerging endoscopic computer-assisted surgical systems.
Collapse
Affiliation(s)
- Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC, 20010, USA
| | - Christina E Rice
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC, 20010, USA.,Department of Mechanical and Aerospace Engineering, Princeton University, Princeton, NJ, 08544, USA
| | - Raj Shekhar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System, 111 Michigan Avenue NW, Washington, DC, 20010, USA.
| |
Collapse
|