1
|
Bano S, Casella A, Vasconcelos F, Qayyum A, Benzinou A, Mazher M, Meriaudeau F, Lena C, Cintorrino IA, De Paolis GR, Biagioli J, Grechishnikova D, Jiao J, Bai B, Qiao Y, Bhattarai B, Gaire RR, Subedi R, Vazquez E, Płotka S, Lisowska A, Sitek A, Attilakos G, Wimalasundera R, David AL, Paladini D, Deprest J, De Momi E, Mattos LS, Moccia S, Stoyanov D. Placental vessel segmentation and registration in fetoscopy: Literature review and MICCAI FetReg2021 challenge findings. Med Image Anal 2024; 92:103066. [PMID: 38141453 PMCID: PMC11162867 DOI: 10.1016/j.media.2023.103066] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2022] [Revised: 11/27/2023] [Accepted: 12/19/2023] [Indexed: 12/25/2023]
Abstract
Fetoscopy laser photocoagulation is a widely adopted procedure for treating Twin-to-Twin Transfusion Syndrome (TTTS). The procedure involves photocoagulation pathological anastomoses to restore a physiological blood exchange among twins. The procedure is particularly challenging, from the surgeon's side, due to the limited field of view, poor manoeuvrability of the fetoscope, poor visibility due to amniotic fluid turbidity, and variability in illumination. These challenges may lead to increased surgery time and incomplete ablation of pathological anastomoses, resulting in persistent TTTS. Computer-assisted intervention (CAI) can provide TTTS surgeons with decision support and context awareness by identifying key structures in the scene and expanding the fetoscopic field of view through video mosaicking. Research in this domain has been hampered by the lack of high-quality data to design, develop and test CAI algorithms. Through the Fetoscopic Placental Vessel Segmentation and Registration (FetReg2021) challenge, which was organized as part of the MICCAI2021 Endoscopic Vision (EndoVis) challenge, we released the first large-scale multi-center TTTS dataset for the development of generalized and robust semantic segmentation and video mosaicking algorithms with a focus on creating drift-free mosaics from long duration fetoscopy videos. For this challenge, we released a dataset of 2060 images, pixel-annotated for vessels, tool, fetus and background classes, from 18 in-vivo TTTS fetoscopy procedures and 18 short video clips of an average length of 411 frames for developing placental scene segmentation and frame registration for mosaicking techniques. Seven teams participated in this challenge and their model performance was assessed on an unseen test dataset of 658 pixel-annotated images from 6 fetoscopic procedures and 6 short clips. For the segmentation task, overall baseline performed was the top performing (aggregated mIoU of 0.6763) and was the best on the vessel class (mIoU of 0.5817) while team RREB was the best on the tool (mIoU of 0.6335) and fetus (mIoU of 0.5178) classes. For the registration task, overall the baseline performed better than team SANO with an overall mean 5-frame SSIM of 0.9348. Qualitatively, it was observed that team SANO performed better in planar scenarios, while baseline was better in non-planner scenarios. The detailed analysis showed that no single team outperformed on all 6 test fetoscopic videos. The challenge provided an opportunity to create generalized solutions for fetoscopic scene understanding and mosaicking. In this paper, we present the findings of the FetReg2021 challenge, alongside reporting a detailed literature review for CAI in TTTS fetoscopy. Through this challenge, its analysis and the release of multi-center fetoscopic data, we provide a benchmark for future research in this field.
Collapse
Affiliation(s)
- Sophia Bano
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, UK.
| | - Alessandro Casella
- Department of Advanced Robotics, Istituto Italiano di Tecnologia, Italy; Department of Electronics, Information and Bioengineering, Politecnico di Milano, Italy
| | - Francisco Vasconcelos
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, UK
| | | | | | - Moona Mazher
- Department of Computer Engineering and Mathematics, University Rovira i Virgili, Spain
| | | | - Chiara Lena
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Italy
| | | | - Gaia Romana De Paolis
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Italy
| | - Jessica Biagioli
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Italy
| | | | | | - Bizhe Bai
- Medical Computer Vision and Robotics Group, Department of Mathematical and Computational Sciences, University of Toronto, Canada
| | - Yanyan Qiao
- Shanghai MicroPort MedBot (Group) Co., Ltd, China
| | - Binod Bhattarai
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, UK
| | | | - Ronast Subedi
- NepAL Applied Mathematics and Informatics Institute for Research, Nepal
| | | | - Szymon Płotka
- Sano Center for Computational Medicine, Poland; Quantitative Healthcare Analysis Group, Informatics Institute, University of Amsterdam, Amsterdam, The Netherlands
| | | | - Arkadiusz Sitek
- Sano Center for Computational Medicine, Poland; Center for Advanced Medical Computing and Simulation, Massachusetts General Hospital, Harvard Medical School, Boston, MA, United States of America
| | - George Attilakos
- Fetal Medicine Unit, Elizabeth Garrett Anderson Wing, University College London Hospital, UK; EGA Institute for Women's Health, Faculty of Population Health Sciences, University College London, UK
| | - Ruwan Wimalasundera
- Fetal Medicine Unit, Elizabeth Garrett Anderson Wing, University College London Hospital, UK; EGA Institute for Women's Health, Faculty of Population Health Sciences, University College London, UK
| | - Anna L David
- Fetal Medicine Unit, Elizabeth Garrett Anderson Wing, University College London Hospital, UK; EGA Institute for Women's Health, Faculty of Population Health Sciences, University College London, UK; Department of Development and Regeneration, University Hospital Leuven, Belgium
| | - Dario Paladini
- Department of Fetal and Perinatal Medicine, Istituto "Giannina Gaslini", Italy
| | - Jan Deprest
- EGA Institute for Women's Health, Faculty of Population Health Sciences, University College London, UK; Department of Development and Regeneration, University Hospital Leuven, Belgium
| | - Elena De Momi
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Italy
| | - Leonardo S Mattos
- Department of Advanced Robotics, Istituto Italiano di Tecnologia, Italy
| | - Sara Moccia
- The BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Italy
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, UK
| |
Collapse
|
2
|
Hernansanz A, Parra J, Sayols N, Eixarch E, Gratacós E, Casals A. Robot assisted Fetoscopic Laser Coagulation: Improvements in navigation, re-location and coagulation. Artif Intell Med 2024; 147:102725. [PMID: 38184348 DOI: 10.1016/j.artmed.2023.102725] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Revised: 11/24/2023] [Accepted: 11/24/2023] [Indexed: 01/08/2024]
Abstract
Fetoscopic Laser Coagulation (FLC) for Twin to Twin Transfusion Syndrome is a challenging intervention due to the working conditions: low quality images acquired from a 3 mm fetoscope inside a turbid liquid environment, local view of the placental surface, unstable surgical field and delicate tissue layers. FLC is based on locating, coagulating and reviewing anastomoses over the placenta's surface. The procedure demands the surgeons to generate a mental map of the placenta with the distribution of the anastomoses, maintaining, at the same time, precision in coagulation and protecting the placenta and amniotic sac from potential damages. This paper describes a teleoperated platform with a cognitive-based control that provides assistance to improve patient safety and surgery performance during fetoscope navigation, target re-location and coagulation processes. A comparative study between manual and teleoperated operation, executed in dry laboratory conditions, analyzes basic fetoscopic skills: fetoscope navigation and laser coagulation. Two exercises are proposed: first, fetoscope guidance and precise coagulation. Second, a resolved placenta (all anastomoses are indicated) to evaluate navigation, re-location and coagulation. The results are analyzed in terms of economy of movement, execution time, coagulation accuracy, amount of coagulated placental surface and risk of placenta puncture. In addition, new metrics, based on navigation and coagulation maps evaluate robotic performance. The results validate the developed platform, showing noticeable improvements in all the metrics.
Collapse
Affiliation(s)
- Albert Hernansanz
- Research Centre for Biomedical Engineering, Technical University of Catalonia, CREB-UPC, 08034 Barcelona, Spain; Simulation, Imaging and Modelling for Biomedical Systems (SIMBIOsys-UPF), Barcelona, Spain.
| | - Johanna Parra
- BCNatal Fetal Medicine Research Center (Hospital Clinic and Hospital Sant Joan de Deu), 08950 Esplugues de Llobregat, Spain
| | - Narcís Sayols
- Research Centre for Biomedical Engineering, Technical University of Catalonia, CREB-UPC, 08034 Barcelona, Spain; Simulation, Imaging and Modelling for Biomedical Systems (SIMBIOsys-UPF), Barcelona, Spain
| | - Elisenda Eixarch
- BCNatal Fetal Medicine Research Center (Hospital Clinic and Hospital Sant Joan de Deu), 08950 Esplugues de Llobregat, Spain; Institut d'Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain; Centre for Biomedical Research on Rare Diseases (CIBER-ER), Barcelona, Spain
| | - Eduard Gratacós
- BCNatal Fetal Medicine Research Center (Hospital Clinic and Hospital Sant Joan de Deu), 08950 Esplugues de Llobregat, Spain; Institut d'Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain; Centre for Biomedical Research on Rare Diseases (CIBER-ER), Barcelona, Spain
| | - Alícia Casals
- Research Centre for Biomedical Engineering, Technical University of Catalonia, CREB-UPC, 08034 Barcelona, Spain
| |
Collapse
|
3
|
Bano S, Vasconcelos F, David AL, Deprest J, Stoyanov D. Placental vessel-guided hybrid framework for fetoscopic mosaicking. COMPUTER METHODS IN BIOMECHANICS AND BIOMEDICAL ENGINEERING: IMAGING & VISUALIZATION 2022. [DOI: 10.1080/21681163.2022.2154278] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Affiliation(s)
- Sophia Bano
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | - Francisco Vasconcelos
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | - Anna L. David
- Fetal Medicine Unit, University College London Hospital, London, UK
| | - Jan Deprest
- Department of Development and Regeneration, University Hospital Leuven, Leuven, Belgium
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| |
Collapse
|
4
|
van der Schot AM, Sikkel E, August Spaanderman ME, Vandenbussche FP. Computer-assisted fetal laser surgery in the treatment of twin-to-twin transfusion syndrome recent trends and prospects. Prenat Diagn 2022; 42:1225-1234. [PMID: 35983630 PMCID: PMC9541851 DOI: 10.1002/pd.6225] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2022] [Revised: 06/06/2022] [Accepted: 08/02/2022] [Indexed: 11/06/2022]
Abstract
Fetal laser surgery has emerged as the preferred treatment of twin-to-twin transfusion syndrome (TTTS). However, the limited field of view of the fetoscope and the complexity of the procedure make the treatment challenging. Therefore, preoperative planning and intraoperative guidance solutions have been proposed to cope with these challenges. This review uncovers the literature on computer-assisted software solutions focused on TTTS. These solutions are classified by the pre- or intraoperative phase of the procedure and further categorized by discussed hardware and software approaches. In addition, it evaluates the current maturity of technologies by the technology readiness level and enumerates the necessary aspects to bring these new technologies to the clinical practice. This article is protected by copyright. All rights reserved.
Collapse
Affiliation(s)
| | - Esther Sikkel
- Department Obstetrics & Gynecology, Radboudumc/Amalia Children's hospital, Nijmegen, the Netherlands
| | - Marc Erich August Spaanderman
- Department Obstetrics & Gynecology, Radboudumc/Amalia Children's hospital, Nijmegen, the Netherlands.,Department Obstetrics & Gynecology, Maastricht UMC+, Maastricht, the Netherlands
| | | |
Collapse
|
5
|
Casella A, Moccia S, Paladini D, Frontoni E, De Momi E, Mattos LS. A shape-constraint adversarial framework with instance-normalized spatio-temporal features for inter-fetal membrane segmentation. Med Image Anal 2021; 70:102008. [PMID: 33647785 DOI: 10.1016/j.media.2021.102008] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2020] [Revised: 12/17/2020] [Accepted: 02/16/2021] [Indexed: 12/01/2022]
Abstract
BACKGROUND AND OBJECTIVES During Twin-to-Twin Transfusion Syndrome (TTTS), abnormal vascular anastomoses in the monochorionic placenta can produce uneven blood flow between the fetuses. In the current practice, this syndrome is surgically treated by closing the abnormal connections using laser ablation. Surgeons commonly use the inter-fetal membrane as a reference. Limited field of view, low fetoscopic image quality and high inter-subject variability make the membrane identification a challenging task. However, currently available tools are not optimal for automatic membrane segmentation in fetoscopic videos, due to membrane texture homogeneity and high illumination variability. METHODS To tackle these challenges, we present a new deep-learning framework for inter-fetal membrane segmentation on in-vivo fetoscopic videos. The framework enhances existing architectures by (i) encoding a novel (instance-normalized) dense block, invariant to illumination changes, that extracts spatio-temporal features to enforce pixel connectivity in time, and (ii) relying on an adversarial training, which constrains macro appearance. RESULTS We performed a comprehensive validation using 20 different videos (2000 frames) from 20 different surgeries, achieving a mean Dice Similarity Coefficient of 0.8780±0.1383. CONCLUSIONS The proposed framework has great potential to positively impact the actual surgical practice for TTTS treatment, allowing the implementation of surgical guidance systems that can enhance context awareness and potentially lower the duration of the surgeries.
Collapse
Affiliation(s)
- Alessandro Casella
- Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy; Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy.
| | - Sara Moccia
- The BioRobotics Institute, Scuola Superiore Sant'Anna, Pisa, Italy; Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Pisa, Italy
| | - Dario Paladini
- Department of Fetal and Perinatal Medicine, Istituto "Giannina Gaslini", Genoa, Italy
| | - Emanuele Frontoni
- Department of Information Engineering, Universitá Politecnica delle Marche, Ancona, Italy
| | - Elena De Momi
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| | - Leonard S Mattos
- Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy
| |
Collapse
|
6
|
Bano S, Vasconcelos F, Tella-Amo M, Dwyer G, Gruijthuijsen C, Vander Poorten E, Vercauteren T, Ourselin S, Deprest J, Stoyanov D. Deep learning-based fetoscopic mosaicking for field-of-view expansion. Int J Comput Assist Radiol Surg 2020; 15:1807-1816. [PMID: 32808148 PMCID: PMC7603466 DOI: 10.1007/s11548-020-02242-8] [Citation(s) in RCA: 19] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2020] [Accepted: 07/30/2020] [Indexed: 11/26/2022]
Abstract
PURPOSE Fetoscopic laser photocoagulation is a minimally invasive surgical procedure used to treat twin-to-twin transfusion syndrome (TTTS), which involves localization and ablation of abnormal vascular connections on the placenta to regulate the blood flow in both fetuses. This procedure is particularly challenging due to the limited field of view, poor visibility, occasional bleeding, and poor image quality. Fetoscopic mosaicking can help in creating an image with the expanded field of view which could facilitate the clinicians during the TTTS procedure. METHODS We propose a deep learning-based mosaicking framework for diverse fetoscopic videos captured from different settings such as simulation, phantoms, ex vivo, and in vivo environments. The proposed mosaicking framework extends an existing deep image homography model to handle video data by introducing the controlled data generation and consistent homography estimation modules. Training is performed on a small subset of fetoscopic images which are independent of the testing videos. RESULTS We perform both quantitative and qualitative evaluations on 5 diverse fetoscopic videos (2400 frames) that captured different environments. To demonstrate the robustness of the proposed framework, a comparison is performed with the existing feature-based and deep image homography methods. CONCLUSION The proposed mosaicking framework outperformed existing methods and generated meaningful mosaic, while reducing the accumulated drift, even in the presence of visual challenges such as specular highlights, reflection, texture paucity, and low video resolution.
Collapse
Affiliation(s)
- Sophia Bano
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | - Francisco Vasconcelos
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | - Marcel Tella-Amo
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | - George Dwyer
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | | | | | - Tom Vercauteren
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK
| | - Sebastien Ourselin
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK
| | - Jan Deprest
- Department of Development and Regeneration, University Hospital Leuven, Leuven, Belgium
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| |
Collapse
|
7
|
Bano S, Vasconcelos F, Vander Poorten E, Vercauteren T, Ourselin S, Deprest J, Stoyanov D. FetNet: a recurrent convolutional network for occlusion identification in fetoscopic videos. Int J Comput Assist Radiol Surg 2020; 15:791-801. [PMID: 32350787 PMCID: PMC7261278 DOI: 10.1007/s11548-020-02169-0] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2019] [Accepted: 04/10/2020] [Indexed: 12/18/2022]
Abstract
PURPOSE Fetoscopic laser photocoagulation is a minimally invasive surgery for the treatment of twin-to-twin transfusion syndrome (TTTS). By using a lens/fibre-optic scope, inserted into the amniotic cavity, the abnormal placental vascular anastomoses are identified and ablated to regulate blood flow to both fetuses. Limited field-of-view, occlusions due to fetus presence and low visibility make it difficult to identify all vascular anastomoses. Automatic computer-assisted techniques may provide better understanding of the anatomical structure during surgery for risk-free laser photocoagulation and may facilitate in improving mosaics from fetoscopic videos. METHODS We propose FetNet, a combined convolutional neural network (CNN) and long short-term memory (LSTM) recurrent neural network architecture for the spatio-temporal identification of fetoscopic events. We adapt an existing CNN architecture for spatial feature extraction and integrated it with the LSTM network for end-to-end spatio-temporal inference. We introduce differential learning rates during the model training to effectively utilising the pre-trained CNN weights. This may support computer-assisted interventions (CAI) during fetoscopic laser photocoagulation. RESULTS We perform quantitative evaluation of our method using 7 in vivo fetoscopic videos captured from different human TTTS cases. The total duration of these videos was 5551 s (138,780 frames). To test the robustness of the proposed approach, we perform 7-fold cross-validation where each video is treated as a hold-out or test set and training is performed using the remaining videos. CONCLUSION FetNet achieved superior performance compared to the existing CNN-based methods and provided improved inference because of the spatio-temporal information modelling. Online testing of FetNet, using a Tesla V100-DGXS-32GB GPU, achieved a frame rate of 114 fps. These results show that our method could potentially provide a real-time solution for CAI and automating occlusion and photocoagulation identification during fetoscopic procedures.
Collapse
Affiliation(s)
- Sophia Bano
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | - Francisco Vasconcelos
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| | | | - Tom Vercauteren
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK
| | - Sebastien Ourselin
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK
| | - Jan Deprest
- Department of Development and Regeneration, University Hospital Leuven, Leuven, Belgium
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) and Department of Computer Science, University College London, London, UK
| |
Collapse
|