1
|
Jaspers TJM, Boers TGW, Kusters CHJ, Jong MR, Jukema JB, de Groof AJ, Bergman JJ, de With PHN, van der Sommen F. Robustness evaluation of deep neural networks for endoscopic image analysis: Insights and strategies. Med Image Anal 2024; 94:103157. [PMID: 38574544 DOI: 10.1016/j.media.2024.103157] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2023] [Revised: 03/19/2024] [Accepted: 03/21/2024] [Indexed: 04/06/2024]
Abstract
Computer-aided detection and diagnosis systems (CADe/CADx) in endoscopy are commonly trained using high-quality imagery, which is not representative for the heterogeneous input typically encountered in clinical practice. In endoscopy, the image quality heavily relies on both the skills and experience of the endoscopist and the specifications of the system used for screening. Factors such as poor illumination, motion blur, and specific post-processing settings can significantly alter the quality and general appearance of these images. This so-called domain gap between the data used for developing the system and the data it encounters after deployment, and the impact it has on the performance of deep neural networks (DNNs) supportive endoscopic CAD systems remains largely unexplored. As many of such systems, for e.g. polyp detection, are already being rolled out in clinical practice, this poses severe patient risks in particularly community hospitals, where both the imaging equipment and experience are subject to considerable variation. Therefore, this study aims to evaluate the impact of this domain gap on the clinical performance of CADe/CADx for various endoscopic applications. For this, we leverage two publicly available data sets (KVASIR-SEG and GIANA) and two in-house data sets. We investigate the performance of commonly-used DNN architectures under synthetic, clinically calibrated image degradations and on a prospectively collected dataset including 342 endoscopic images of lower subjective quality. Additionally, we assess the influence of DNN architecture and complexity, data augmentation, and pretraining techniques for improved robustness. The results reveal a considerable decline in performance of 11.6% (±1.5) as compared to the reference, within the clinically calibrated boundaries of image degradations. Nevertheless, employing more advanced DNN architectures and self-supervised in-domain pre-training effectively mitigate this drop to 7.7% (±2.03). Additionally, these enhancements yield the highest performance on the manually collected test set including images with lower subjective quality. By comprehensively assessing the robustness of popular DNN architectures and training strategies across multiple datasets, this study provides valuable insights into their performance and limitations for endoscopic applications. The findings highlight the importance of including robustness evaluation when developing DNNs for endoscopy applications and propose strategies to mitigate performance loss.
Collapse
Affiliation(s)
- Tim J M Jaspers
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands.
| | - Tim G W Boers
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Carolus H J Kusters
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Martijn R Jong
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Jelmer B Jukema
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Albert J de Groof
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Jacques J Bergman
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Peter H N de With
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Fons van der Sommen
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| |
Collapse
|
2
|
Jukema JB, Kusters CHJ, Jong MR, Fockens KN, Boers T, van der Putten JA, Pouw RE, Duits LC, Weusten BAM, Herrero LA, Houben MHMG, Nagengast WB, Westerhof J, Alkhalaf A, Mallant-Hent R, Scholten P, Ragunath K, Seewald S, Elbe P, Silva FB, Barret M, Fernández-Sordo JO, Villarejo GM, Pech O, Beyna T, Montazeri NSM, van der Sommen F, de With PH, de Groof AJ, Bergman JJ. COMPUTER-AIDED DIAGNOSIS IMPROVES CHARACTERIZATION OF BARRETT'S NEOPLASIA BY GENERAL ENDOSCOPISTS. Gastrointest Endosc 2024:S0016-5107(24)00233-5. [PMID: 38636819 DOI: 10.1016/j.gie.2024.04.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Revised: 01/25/2024] [Accepted: 04/08/2024] [Indexed: 04/20/2024]
Abstract
BACKGROUND & AIMS Characterization of visible abnormalities in Barrett esophagus (BE) patients can be challenging, especially for unexperienced endoscopists. This results in suboptimal diagnostic accuracy and poor inter-observer agreement. Computer-aided diagnosis (CADx) systems may assist endoscopists. We aimed to develop, validate and benchmark a CADx system for BE neoplasia. METHODS The CADx system received pretraining with ImageNet with consecutive domain-specific pretraining with GastroNet which includes 5 million endoscopic images. It was subsequently trained and internally validated using 1,758 narrow-band imaging (NBI) images of early BE neoplasia (352 patients) and 1,838 NBI images of non-dysplastic BE (173 patients) from 8 international centers. CADx was tested prospectively on corresponding image and video test sets with 30 cases (20 patients) of BE neoplasia and 60 cases (31 patients) of non-dysplastic BE. The test set was benchmarked by 44 general endoscopists in two phases (phase 1: no CADx assistance; phase 2: with CADx assistance). Ten international BE experts provided additional benchmark performance. RESULTS Stand-alone sensitivity and specificity of the CADx system were 100% and 98% for images and 93% and 96% for videos, respectively. CADx outperformed general endoscopists without CADx assistance in terms of sensitivity (p=0.04). Sensitivity and specificity of general endoscopist increased from 84% to 96% and 90 to 98% with CAD assistance (p<0.001), respectively. CADx assistance increased endoscopists' confidence in characterization (p<0.001). CADx performance was similar to Barrett experts. CONCLUSION CADx assistance significantly increased characterization performance of BE neoplasia by general endoscopists to the level of expert endoscopists. The use of this CADx system may thereby improve daily Barrett surveillance.
Collapse
Affiliation(s)
- Jelmer B Jukema
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology Endocrinology Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Carolus H J Kusters
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Martijn R Jong
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology Endocrinology Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Kiki N Fockens
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology Endocrinology Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Tim Boers
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Joost A van der Putten
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Roos E Pouw
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology Endocrinology Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Lucas C Duits
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology Endocrinology Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - BasL A M Weusten
- Department of Gastroenterology and Hepatology, UMC Utrecht, University of Utrecht, Utrecht, the Netherlands; Department of Gastroenterology and Hepatology, Sint Antonius hospital, Nieuwegein, the Netherlands
| | - Lorenza Alvarez Herrero
- Department of Gastroenterology and Hepatology, Sint Antonius hospital, Nieuwegein, the Netherlands
| | - Martin H M G Houben
- Department of Gastroenterology and Hepatology, HagaZiekenhuis Den Haag, Den Haag, the Netherlands
| | - Wouter B Nagengast
- Department of Gastroenterology and Hepatology, UMC Groningen, University of Groningen, Groningen, the Netherlands
| | - Jessie Westerhof
- Department of Gastroenterology and Hepatology, UMC Groningen, University of Groningen, Groningen, the Netherlands
| | - Alaa Alkhalaf
- Department of Gastroenterology and Hepatology, Isala Hospital Zwolle, Zwolle, the Netherlands
| | - Rosalie Mallant-Hent
- Department of Gastroenterology and Hepatology, Flevoziekenhuis Almere, Almere, the Netherlands
| | - Pieter Scholten
- Department of Gastroenterology and Hepatology, Onze Lieve Vrouwe Gasthuis, Amsterdam, the Netherlands
| | - Krish Ragunath
- Department of Gastroenterology and Hepatology, Royal Perth Hospital, Curtin University, Perth, Australia
| | - Stefan Seewald
- Department of Gastroenterology and Hepatology, Hirslanden Klinik, Zurich, Switzerland
| | - Peter Elbe
- Department of Digestive Diseases, Karolinska University Hospital, Stockholm, Sweden; Division of Surgery, Department of Clinical Science, Intervention and Technology, CLINTEC, Karolinska Institutet, Stockholm, Sweden
| | - Francisco Baldaque Silva
- Department of Digestive Diseases, Karolinska University Hospital, Stockholm, Sweden; Center for Advanced Endoscopy Carlos Moreira da Silva, Gastroenterology Department, Pedro Hispano Hospital, ULSM Matosinhos, Portugal
| | - Maximilien Barret
- Department of Gastroenterology and Hepatology, Cochin hospital Paris, Paris, France
| | - Jacobo Ortiz Fernández-Sordo
- Department of Gastroenterology and Hepatology, Nottingham University Hospitals NHS Trust, Nottingham, United Kingdom
| | - Guiomar Moral Villarejo
- Department of Gastroenterology and Hepatology, Nottingham University Hospitals NHS Trust, Nottingham, United Kingdom
| | - Oliver Pech
- Department of Gastroenterology and Hepatology, St. John of God Hospital, Regensburg, Germany
| | - Torsten Beyna
- Department of Gastroenterology and Hepatology, Evangalisches Krankenhaus Düsseldorf, Düsseldorf, Germany
| | - Nahid S M Montazeri
- Biostatistics Unit, Department of Gastroenterology and Hepatology, Amsterdam University Medical Center, location Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands
| | - Fons van der Sommen
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Peter H de With
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - A Jeroen de Groof
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology Endocrinology Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Jacques J Bergman
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology Endocrinology Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands.
| |
Collapse
|
3
|
Fockens KN, Jukema JB, Jong MR, Boers T, van der Putten JA, Kusters CHJ, Pouw RE, Duits LC, Sommen FVD, de With PH, de Groof AJ, Bergman JJ. The use of a real-time computer-aided detection system for visible lesions in the Barrett's esophagus during live endoscopic procedures, a pilot study. Gastrointest Endosc 2024:S0016-5107(24)00228-1. [PMID: 38604297 DOI: 10.1016/j.gie.2024.04.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Revised: 03/07/2024] [Accepted: 04/01/2024] [Indexed: 04/13/2024]
Abstract
BACKGROUND AND AIMS In this pilot study we evaluated performance of a recently developed computer-aided detection (CADe) system for Barrett's neoplasia during live endoscopic procedures. METHODS 15 patients with and 15 without a visible lesion were included in this study. A CAD assisted workflow was employed that included: a slow pullback video recording of the entire Barrett's segment with live CADe assistance, followed by CADe assisted level-based video recordings every 2cm of the Barrett's segment. Outcomes were per patient and per level diagnostic accuracy of the CAD assisted workflow, where the primary outcome was per patient in-vivo CADe sensitivity. RESULTS In the per patient analyses, the CADe system detected all visible lesions (sensitivity 100%). Per patient CADe specificity was 53%. Per-level sensitivity and specificity of the CADe assisted workflow were 100% and 73%, respectively. CONCLUSION In this pilot study, the CADe system detected all potentially neoplastic lesions in Barrett's esophagus comparable to an expert endoscopist. Continued refinement of the system may improve specificity. External validation in larger multicenter studies is planned.
Collapse
Affiliation(s)
- Kiki N Fockens
- Department of gastroenterology and hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Jelmer B Jukema
- Department of gastroenterology and hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Martijn R Jong
- Department of gastroenterology and hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Tim Boers
- Department of electrical engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Joost A van der Putten
- Department of electrical engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Carolus H J Kusters
- Department of electrical engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Roos E Pouw
- Department of gastroenterology and hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Lucas C Duits
- Department of gastroenterology and hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Fons van der Sommen
- Department of electrical engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Peter H de With
- Department of electrical engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - A Jeroen de Groof
- Department of gastroenterology and hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - Jacques J Bergman
- Department of gastroenterology and hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| |
Collapse
|
4
|
Fockens KN, Jong MR, Jukema JB, Boers TGW, Kusters CHJ, van der Putten JA, Pouw RE, Duits LC, Montazeri NSM, van Munster SN, Weusten BLAM, Alvarez Herrero L, Houben MHMG, Nagengast WB, Westerhof J, Alkhalaf A, Mallant-Hent RC, Scholten P, Ragunath K, Seewald S, Elbe P, Baldaque-Silva F, Barret M, Ortiz Fernández-Sordo J, Villarejo GM, Pech O, Beyna T, van der Sommen F, de With PH, de Groof AJ, Bergman JJ. A deep learning system for detection of early Barrett's neoplasia: a model development and validation study. Lancet Digit Health 2023; 5:e905-e916. [PMID: 38000874 DOI: 10.1016/s2589-7500(23)00199-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 08/22/2023] [Accepted: 09/18/2023] [Indexed: 11/26/2023]
Abstract
BACKGROUND Computer-aided detection (CADe) systems could assist endoscopists in detecting early neoplasia in Barrett's oesophagus, which could be difficult to detect in endoscopic images. The aim of this study was to develop, test, and benchmark a CADe system for early neoplasia in Barrett's oesophagus. METHODS The CADe system was first pretrained with ImageNet followed by domain-specific pretraining with GastroNet. We trained the CADe system on a dataset of 14 046 images (2506 patients) of confirmed Barrett's oesophagus neoplasia and non-dysplastic Barrett's oesophagus from 15 centres. Neoplasia was delineated by 14 Barrett's oesophagus experts for all datasets. We tested the performance of the CADe system on two independent test sets. The all-comers test set comprised 327 (73 patients) non-dysplastic Barrett's oesophagus images, 82 (46 patients) neoplastic images, 180 (66 of the same patients) non-dysplastic Barrett's oesophagus videos, and 71 (45 of the same patients) neoplastic videos. The benchmarking test set comprised 100 (50 patients) neoplastic images, 300 (125 patients) non-dysplastic images, 47 (47 of the same patients) neoplastic videos, and 141 (82 of the same patients) non-dysplastic videos, and was enriched with subtle neoplasia cases. The benchmarking test set was evaluated by 112 endoscopists from six countries (first without CADe and, after 6 weeks, with CADe) and by 28 external international Barrett's oesophagus experts. The primary outcome was the sensitivity of Barrett's neoplasia detection by general endoscopists without CADe assistance versus with CADe assistance on the benchmarking test set. We compared sensitivity using a mixed-effects logistic regression model with conditional odds ratios (ORs; likelihood profile 95% CIs). FINDINGS Sensitivity for neoplasia detection among endoscopists increased from 74% to 88% with CADe assistance (OR 2·04; 95% CI 1·73-2·42; p<0·0001 for images and from 67% to 79% [2·35; 1·90-2·94; p<0·0001] for video) without compromising specificity (from 89% to 90% [1·07; 0·96-1·19; p=0·20] for images and from 96% to 94% [0·94; 0·79-1·11; ] for video; p=0·46). In the all-comers test set, CADe detected neoplastic lesions in 95% (88-98) of images and 97% (90-99) of videos. In the benchmarking test set, the CADe system was superior to endoscopists in detecting neoplasia (90% vs 74% [OR 3·75; 95% CI 1·93-8·05; p=0·0002] for images and 91% vs 67% [11·68; 3·85-47·53; p<0·0001] for video) and non-inferior to Barrett's oesophagus experts (90% vs 87% [OR 1·74; 95% CI 0·83-3·65] for images and 91% vs 86% [2·94; 0·99-11·40] for video). INTERPRETATION CADe outperformed endoscopists in detecting Barrett's oesophagus neoplasia and, when used as an assistive tool, it improved their detection rate. CADe detected virtually all neoplasia in a test set of consecutive cases. FUNDING Olympus.
Collapse
Affiliation(s)
- K N Fockens
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - M R Jong
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - J B Jukema
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - T G W Boers
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - C H J Kusters
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - J A van der Putten
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - R E Pouw
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - L C Duits
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - N S M Montazeri
- Biostatistics Unit, Department of Gastroenterology and Hepatology, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - S N van Munster
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands; Department of Gastroenterology and Hepatology, St Antonius Hospital, Nieuwegein, Netherlands
| | - B L A M Weusten
- Department of Gastroenterology and Hepatology, UMC Utrecht, University of Utrecht, Utrecht, Netherlands; Department of Gastroenterology and Hepatology, St Antonius Hospital, Nieuwegein, Netherlands
| | - L Alvarez Herrero
- Department of Gastroenterology and Hepatology, St Antonius Hospital, Nieuwegein, Netherlands
| | - M H M G Houben
- Department of Gastroenterology and Hepatology, HagaZiekenhuis Den Haag, Den Haag, Netherlands
| | - W B Nagengast
- Department of Gastroenterology and Hepatology, UMC Groningen, University of Groningen, Groningen, Netherlands
| | - J Westerhof
- Department of Gastroenterology and Hepatology, UMC Groningen, University of Groningen, Groningen, Netherlands
| | - A Alkhalaf
- Department of Gastroenterology and Hepatology, Isala Hospital Zwolle, Zwolle, Netherlands
| | - R C Mallant-Hent
- Department of Gastroenterology and Hepatology, Flevoziekenhuis Almere, Almere, Netherlands
| | - P Scholten
- Department of Gastroenterology and Hepatology, Onze Lieve Vrouwe Gasthuis, Amsterdam, Netherlands
| | - K Ragunath
- Department of Gastroenterology and Hepatology, Royal Perth Hospital, Curtin University, Perth, WA, Australia
| | - S Seewald
- Department of Gastroenterology and Hepatology, Hirslanden Klinik, Zurich, Switzerland
| | - P Elbe
- Department of Digestive Diseases, Karolinska University Hospital, Stockholm, Sweden; Division of Surgery, Department of Clinical Science, Intervention and Technology, Karolinska Institutet, Stockholm, Sweden
| | - F Baldaque-Silva
- Department of Digestive Diseases, Karolinska University Hospital, Stockholm, Sweden; Center for Advanced Endoscopy Carlos Moreira da Silva, Gastroenterology Department, Pedro Hispano Hospital, Matosinhos, Portugal
| | - M Barret
- Department of Gastroenterology and Hepatology, Cochin Hospital Paris, Paris, France
| | - J Ortiz Fernández-Sordo
- Department of Gastroenterology and Hepatology, Nottingham University Hospitals NHS Trust, Nottingham, UK
| | - G Moral Villarejo
- Department of Gastroenterology and Hepatology, Nottingham University Hospitals NHS Trust, Nottingham, UK
| | - O Pech
- Department of Gastroenterology and Hepatology, St John of God Hospital, Regensburg, Germany
| | - T Beyna
- Department of Gastroenterology and Hepatology, Evangalisches Krankenhaus Düsseldorf, Düsseldorf, Germany
| | - F van der Sommen
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - P H de With
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - A J de Groof
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - J J Bergman
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands.
| |
Collapse
|
5
|
Fockens KN, Jukema JB, Boers T, Jong MR, van der Putten JA, Pouw RE, Weusten BLAM, Alvarez Herrero L, Houben MHMG, Nagengast WB, Westerhof J, Alkhalaf A, Mallant R, Ragunath K, Seewald S, Elbe P, Barret M, Ortiz Fernández-Sordo J, Pech O, Beyna T, van der Sommen F, de With PH, de Groof AJ, Bergman JJ. Towards a robust and compact deep learning system for primary detection of early Barrett's neoplasia: Initial image-based results of training on a multi-center retrospectively collected data set. United European Gastroenterol J 2023; 11:324-336. [PMID: 37095718 PMCID: PMC10165317 DOI: 10.1002/ueg2.12363] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/23/2022] [Accepted: 01/09/2023] [Indexed: 04/26/2023] Open
Abstract
INTRODUCTION Endoscopic detection of early neoplasia in Barrett's esophagus is difficult. Computer Aided Detection (CADe) systems may assist in neoplasia detection. The aim of this study was to report the first steps in the development of a CADe system for Barrett's neoplasia and to evaluate its performance when compared with endoscopists. METHODS This CADe system was developed by a consortium, consisting of the Amsterdam University Medical Center, Eindhoven University of Technology, and 15 international hospitals. After pretraining, the system was trained and validated using 1.713 neoplastic (564 patients) and 2.707 non-dysplastic Barrett's esophagus (NDBE; 665 patients) images. Neoplastic lesions were delineated by 14 experts. The performance of the CADe system was tested on three independent test sets. Test set 1 (50 neoplastic and 150 NDBE images) contained subtle neoplastic lesions representing challenging cases and was benchmarked by 52 general endoscopists. Test set 2 (50 neoplastic and 50 NDBE images) contained a heterogeneous case-mix of neoplastic lesions, representing distribution in clinical practice. Test set 3 (50 neoplastic and 150 NDBE images) contained prospectively collected imagery. The main outcome was correct classification of the images in terms of sensitivity. RESULTS The sensitivity of the CADe system on test set 1 was 84%. For general endoscopists, sensitivity was 63%, corresponding to a neoplasia miss-rate of one-third of neoplastic lesions and a potential relative increase in neoplasia detection of 33% for CADe-assisted detection. The sensitivity of the CADe system on test sets 2 and 3 was 100% and 88%, respectively. The specificity of the CADe system varied for the three test sets between 64% and 66%. CONCLUSION This study describes the first steps towards the establishment of an unprecedented data infrastructure for using machine learning to improve the endoscopic detection of Barrett's neoplasia. The CADe system detected neoplasia reliably and outperformed a large group of endoscopists in terms of sensitivity.
Collapse
Affiliation(s)
- Kiki N Fockens
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, University of Amsterdam, Amsterdam, the Netherlands
| | - Jelmer B Jukema
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, University of Amsterdam, Amsterdam, the Netherlands
| | - Tim Boers
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Martijn R Jong
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, University of Amsterdam, Amsterdam, the Netherlands
| | - Joost A van der Putten
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Roos E Pouw
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, University of Amsterdam, Amsterdam, the Netherlands
| | - Bas L A M Weusten
- Department of Gastroenterology and Hepatology, UMC Utrecht, University of Utrecht, Utrecht, the Netherlands
- Department of Gastroenterology and Hepatology, Sint Antonius Hospital, Nieuwegein, the Netherlands
| | - Lorenza Alvarez Herrero
- Department of Gastroenterology and Hepatology, Sint Antonius Hospital, Nieuwegein, the Netherlands
| | - Martin H M G Houben
- Department of Gastroenterology and Hepatology, Haga Teaching Hospital, Den Haag, the Netherlands
| | - Wouter B Nagengast
- Department of Gastroenterology and Hepatology, University of Groningen, Groningen, the Netherlands
| | - Jessie Westerhof
- Department of Gastroenterology and Hepatology, University of Groningen, Groningen, the Netherlands
| | - Alaa Alkhalaf
- Department of Gastroenterology and Hepatology, Isala Hospital Zwolle, Zwolle, the Netherlands
| | - Rosalie Mallant
- Department of Gastroenterology and Hepatology, Flevoziekenhuis Almere, Almere, the Netherlands
| | - Krish Ragunath
- Department of Gastroenterology and Hepatology, Royal Perth Hospital, Perth, Australia
| | - Stefan Seewald
- Department of Gastroenterology and Hepatology, Hirslanden Klinik, Zurich, Switzerland
| | - Peter Elbe
- Department of Digestive Diseasess, Karolinska University Hospital, Stockholm, Sweden
- Division of Surgery, Department of Clinical Science, Intervention and Technology, CLINTEC, Karolinska Institutet, Stockholm, Sweden
| | - Maximilien Barret
- Department of Gastroenterology and Hepatology, Cochin Hospital Paris, Paris, France
| | | | - Oliver Pech
- Department of Gastroenterology and Hepatology, Krankenhaus Barmherzige Brüder Regensburg, Regensburg, Germany
| | - Torsten Beyna
- Department of Gastroenterology and Hepatology, Evangalische Klinik Düsseldorf, Düsseldorf, Germany
| | - Fons van der Sommen
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Peter H de With
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - A Jeroen de Groof
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, University of Amsterdam, Amsterdam, the Netherlands
| | - Jacques J Bergman
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, University of Amsterdam, Amsterdam, the Netherlands
| |
Collapse
|