1
|
Jaspers TJM, Boers TGW, Kusters CHJ, Jong MR, Jukema JB, de Groof AJ, Bergman JJ, de With PHN, van der Sommen F. Robustness evaluation of deep neural networks for endoscopic image analysis: Insights and strategies. Med Image Anal 2024; 94:103157. [PMID: 38574544 DOI: 10.1016/j.media.2024.103157] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2023] [Revised: 03/19/2024] [Accepted: 03/21/2024] [Indexed: 04/06/2024]
Abstract
Computer-aided detection and diagnosis systems (CADe/CADx) in endoscopy are commonly trained using high-quality imagery, which is not representative for the heterogeneous input typically encountered in clinical practice. In endoscopy, the image quality heavily relies on both the skills and experience of the endoscopist and the specifications of the system used for screening. Factors such as poor illumination, motion blur, and specific post-processing settings can significantly alter the quality and general appearance of these images. This so-called domain gap between the data used for developing the system and the data it encounters after deployment, and the impact it has on the performance of deep neural networks (DNNs) supportive endoscopic CAD systems remains largely unexplored. As many of such systems, for e.g. polyp detection, are already being rolled out in clinical practice, this poses severe patient risks in particularly community hospitals, where both the imaging equipment and experience are subject to considerable variation. Therefore, this study aims to evaluate the impact of this domain gap on the clinical performance of CADe/CADx for various endoscopic applications. For this, we leverage two publicly available data sets (KVASIR-SEG and GIANA) and two in-house data sets. We investigate the performance of commonly-used DNN architectures under synthetic, clinically calibrated image degradations and on a prospectively collected dataset including 342 endoscopic images of lower subjective quality. Additionally, we assess the influence of DNN architecture and complexity, data augmentation, and pretraining techniques for improved robustness. The results reveal a considerable decline in performance of 11.6% (±1.5) as compared to the reference, within the clinically calibrated boundaries of image degradations. Nevertheless, employing more advanced DNN architectures and self-supervised in-domain pre-training effectively mitigate this drop to 7.7% (±2.03). Additionally, these enhancements yield the highest performance on the manually collected test set including images with lower subjective quality. By comprehensively assessing the robustness of popular DNN architectures and training strategies across multiple datasets, this study provides valuable insights into their performance and limitations for endoscopic applications. The findings highlight the importance of including robustness evaluation when developing DNNs for endoscopy applications and propose strategies to mitigate performance loss.
Collapse
Affiliation(s)
- Tim J M Jaspers
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands.
| | - Tim G W Boers
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Carolus H J Kusters
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Martijn R Jong
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Jelmer B Jukema
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Albert J de Groof
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Jacques J Bergman
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Peter H N de With
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Fons van der Sommen
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| |
Collapse
|
2
|
Fockens KN, Jong MR, Jukema JB, Boers TGW, Kusters CHJ, van der Putten JA, Pouw RE, Duits LC, Montazeri NSM, van Munster SN, Weusten BLAM, Alvarez Herrero L, Houben MHMG, Nagengast WB, Westerhof J, Alkhalaf A, Mallant-Hent RC, Scholten P, Ragunath K, Seewald S, Elbe P, Baldaque-Silva F, Barret M, Ortiz Fernández-Sordo J, Villarejo GM, Pech O, Beyna T, van der Sommen F, de With PH, de Groof AJ, Bergman JJ. A deep learning system for detection of early Barrett's neoplasia: a model development and validation study. Lancet Digit Health 2023; 5:e905-e916. [PMID: 38000874 DOI: 10.1016/s2589-7500(23)00199-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 08/22/2023] [Accepted: 09/18/2023] [Indexed: 11/26/2023]
Abstract
BACKGROUND Computer-aided detection (CADe) systems could assist endoscopists in detecting early neoplasia in Barrett's oesophagus, which could be difficult to detect in endoscopic images. The aim of this study was to develop, test, and benchmark a CADe system for early neoplasia in Barrett's oesophagus. METHODS The CADe system was first pretrained with ImageNet followed by domain-specific pretraining with GastroNet. We trained the CADe system on a dataset of 14 046 images (2506 patients) of confirmed Barrett's oesophagus neoplasia and non-dysplastic Barrett's oesophagus from 15 centres. Neoplasia was delineated by 14 Barrett's oesophagus experts for all datasets. We tested the performance of the CADe system on two independent test sets. The all-comers test set comprised 327 (73 patients) non-dysplastic Barrett's oesophagus images, 82 (46 patients) neoplastic images, 180 (66 of the same patients) non-dysplastic Barrett's oesophagus videos, and 71 (45 of the same patients) neoplastic videos. The benchmarking test set comprised 100 (50 patients) neoplastic images, 300 (125 patients) non-dysplastic images, 47 (47 of the same patients) neoplastic videos, and 141 (82 of the same patients) non-dysplastic videos, and was enriched with subtle neoplasia cases. The benchmarking test set was evaluated by 112 endoscopists from six countries (first without CADe and, after 6 weeks, with CADe) and by 28 external international Barrett's oesophagus experts. The primary outcome was the sensitivity of Barrett's neoplasia detection by general endoscopists without CADe assistance versus with CADe assistance on the benchmarking test set. We compared sensitivity using a mixed-effects logistic regression model with conditional odds ratios (ORs; likelihood profile 95% CIs). FINDINGS Sensitivity for neoplasia detection among endoscopists increased from 74% to 88% with CADe assistance (OR 2·04; 95% CI 1·73-2·42; p<0·0001 for images and from 67% to 79% [2·35; 1·90-2·94; p<0·0001] for video) without compromising specificity (from 89% to 90% [1·07; 0·96-1·19; p=0·20] for images and from 96% to 94% [0·94; 0·79-1·11; ] for video; p=0·46). In the all-comers test set, CADe detected neoplastic lesions in 95% (88-98) of images and 97% (90-99) of videos. In the benchmarking test set, the CADe system was superior to endoscopists in detecting neoplasia (90% vs 74% [OR 3·75; 95% CI 1·93-8·05; p=0·0002] for images and 91% vs 67% [11·68; 3·85-47·53; p<0·0001] for video) and non-inferior to Barrett's oesophagus experts (90% vs 87% [OR 1·74; 95% CI 0·83-3·65] for images and 91% vs 86% [2·94; 0·99-11·40] for video). INTERPRETATION CADe outperformed endoscopists in detecting Barrett's oesophagus neoplasia and, when used as an assistive tool, it improved their detection rate. CADe detected virtually all neoplasia in a test set of consecutive cases. FUNDING Olympus.
Collapse
Affiliation(s)
- K N Fockens
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - M R Jong
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - J B Jukema
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - T G W Boers
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - C H J Kusters
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - J A van der Putten
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - R E Pouw
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - L C Duits
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - N S M Montazeri
- Biostatistics Unit, Department of Gastroenterology and Hepatology, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - S N van Munster
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands; Department of Gastroenterology and Hepatology, St Antonius Hospital, Nieuwegein, Netherlands
| | - B L A M Weusten
- Department of Gastroenterology and Hepatology, UMC Utrecht, University of Utrecht, Utrecht, Netherlands; Department of Gastroenterology and Hepatology, St Antonius Hospital, Nieuwegein, Netherlands
| | - L Alvarez Herrero
- Department of Gastroenterology and Hepatology, St Antonius Hospital, Nieuwegein, Netherlands
| | - M H M G Houben
- Department of Gastroenterology and Hepatology, HagaZiekenhuis Den Haag, Den Haag, Netherlands
| | - W B Nagengast
- Department of Gastroenterology and Hepatology, UMC Groningen, University of Groningen, Groningen, Netherlands
| | - J Westerhof
- Department of Gastroenterology and Hepatology, UMC Groningen, University of Groningen, Groningen, Netherlands
| | - A Alkhalaf
- Department of Gastroenterology and Hepatology, Isala Hospital Zwolle, Zwolle, Netherlands
| | - R C Mallant-Hent
- Department of Gastroenterology and Hepatology, Flevoziekenhuis Almere, Almere, Netherlands
| | - P Scholten
- Department of Gastroenterology and Hepatology, Onze Lieve Vrouwe Gasthuis, Amsterdam, Netherlands
| | - K Ragunath
- Department of Gastroenterology and Hepatology, Royal Perth Hospital, Curtin University, Perth, WA, Australia
| | - S Seewald
- Department of Gastroenterology and Hepatology, Hirslanden Klinik, Zurich, Switzerland
| | - P Elbe
- Department of Digestive Diseases, Karolinska University Hospital, Stockholm, Sweden; Division of Surgery, Department of Clinical Science, Intervention and Technology, Karolinska Institutet, Stockholm, Sweden
| | - F Baldaque-Silva
- Department of Digestive Diseases, Karolinska University Hospital, Stockholm, Sweden; Center for Advanced Endoscopy Carlos Moreira da Silva, Gastroenterology Department, Pedro Hispano Hospital, Matosinhos, Portugal
| | - M Barret
- Department of Gastroenterology and Hepatology, Cochin Hospital Paris, Paris, France
| | - J Ortiz Fernández-Sordo
- Department of Gastroenterology and Hepatology, Nottingham University Hospitals NHS Trust, Nottingham, UK
| | - G Moral Villarejo
- Department of Gastroenterology and Hepatology, Nottingham University Hospitals NHS Trust, Nottingham, UK
| | - O Pech
- Department of Gastroenterology and Hepatology, St John of God Hospital, Regensburg, Germany
| | - T Beyna
- Department of Gastroenterology and Hepatology, Evangalisches Krankenhaus Düsseldorf, Düsseldorf, Germany
| | - F van der Sommen
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - P H de With
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, Netherlands
| | - A J de Groof
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
| | - J J Bergman
- Department of Gastroenterology and Hepatology, Amsterdam Gastroenterology, Endocrinology and Metabolism, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands.
| |
Collapse
|
3
|
van Eyk HJ, Paiman EHM, Bizino MB, IJzermans SL, Kleiburg F, Boers TGW, Rappel EJ, Burakiewicz J, Kan HE, Smit JWA, Lamb HJ, Jazet IM, Rensen PCN. Liraglutide decreases energy expenditure and does not affect the fat fraction of supraclavicular brown adipose tissue in patients with type 2 diabetes. Nutr Metab Cardiovasc Dis 2020; 30:616-624. [PMID: 32127340 DOI: 10.1016/j.numecd.2019.12.005] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/07/2019] [Revised: 11/22/2019] [Accepted: 12/04/2019] [Indexed: 01/01/2023]
Abstract
BACKGROUND AND AIMS Several studies have shown that glucagon-like peptide-1 (GLP-1) analogues can affect resting energy expenditure, and preclinical studies suggest that they may activate brown adipose tissue (BAT). The aim of the present study was to investigate the effect of treatment with liraglutide on energy metabolism and BAT fat fraction in patients with type 2 diabetes. METHODS AND RESULTS In a 26-week double-blind, placebo-controlled trial, 50 patients with type 2 diabetes were randomized to treatment with liraglutide (1.8 mg/day) or placebo added to standard care. At baseline and after treatment for 4, 12 and 26 weeks, we assessed resting energy expenditure (REE) by indirect calorimetry. Furthermore, at baseline and after 26 weeks, we determined the fat fraction in the supraclavicular BAT depot using chemical-shift water-fat MRI at 3T. Liraglutide reduced REE after 4 weeks, which persisted after 12 weeks and tended to be present after 26 weeks (week 26 vs baseline: liraglutide -52 ± 128 kcal/day; P = 0.071, placebo +44 ± 144 kcal/day; P = 0.153, between group P = 0.057). Treatment with liraglutide for 26 weeks did not decrease the fat fraction in supraclavicular BAT (-0.4 ± 1.7%; P = 0.447) compared to placebo (-0.4 ± 1.4%; P = 0.420; between group P = 0.911). CONCLUSION Treatment with liraglutide decreases REE in the first 12 weeks and tends to decrease this after 26 weeks without affecting the fat fraction in the supraclavicular BAT depot. These findings suggest reduction in energy intake rather than an increase in REE to contribute to the liraglutide-induced weight loss. TRIAL REGISTRY NUMBER NCT01761318.
Collapse
Affiliation(s)
- Huub J van Eyk
- Dept. Medicine, Div. Endocrinology, Leiden University Medical Center (LUMC), Leiden, the Netherlands; Einthoven Laboratory for Experimental Vascular Medicine, LUMC, Leiden, the Netherlands.
| | | | - Maurice B Bizino
- Dept. Medicine, Div. Endocrinology, Leiden University Medical Center (LUMC), Leiden, the Netherlands; Dept. Radiology, LUMC, Leiden, the Netherlands
| | - Suzanne L IJzermans
- Dept. Medicine, Div. Endocrinology, Leiden University Medical Center (LUMC), Leiden, the Netherlands; Einthoven Laboratory for Experimental Vascular Medicine, LUMC, Leiden, the Netherlands
| | - Fleur Kleiburg
- Dept. Medicine, Div. Endocrinology, Leiden University Medical Center (LUMC), Leiden, the Netherlands; Einthoven Laboratory for Experimental Vascular Medicine, LUMC, Leiden, the Netherlands
| | | | | | | | | | - Johannes W A Smit
- Dept. Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
| | | | - Ingrid M Jazet
- Dept. Medicine, Div. Endocrinology, Leiden University Medical Center (LUMC), Leiden, the Netherlands; Einthoven Laboratory for Experimental Vascular Medicine, LUMC, Leiden, the Netherlands
| | - Patrick C N Rensen
- Dept. Medicine, Div. Endocrinology, Leiden University Medical Center (LUMC), Leiden, the Netherlands; Einthoven Laboratory for Experimental Vascular Medicine, LUMC, Leiden, the Netherlands
| |
Collapse
|
4
|
Boers TGW, Hu Y, Gibson E, Barratt DC, Bonmati E, Krdzalic J, van der Heijden F, Hermans JJ, Huisman HJ. Interactive 3D U-net for the segmentation of the pancreas in computed tomography scans. Phys Med Biol 2020; 65:065002. [PMID: 31978921 DOI: 10.1088/1361-6560/ab6f99] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023]
Abstract
The increasing incidence of pancreatic cancer will make it the second deadliest cancer in 2030. Imaging based early diagnosis and image guided treatment are emerging potential solutions. Artificial intelligence (AI) can help provide and improve widespread diagnostic expertise and accurate interventional image interpretation. Accurate segmentation of the pancreas is essential to create annotated data sets to train AI, and for computer assisted interventional guidance. Automated deep learning segmentation performance in pancreas computed tomography (CT) imaging is low due to poor grey value contrast and complex anatomy. A good solution seemed a recent interactive deep learning segmentation framework for brain CT that helped strongly improve initial automated segmentation with minimal user input. This method yielded no satisfactory results for pancreas CT, possibly due to a sub-optimal neural network architecture. We hypothesize that a state-of-the-art U-net neural network architecture is better because it can produce a better initial segmentation and is likely to be extended to work in a similar interactive approach. We implemented the existing interactive method, iFCN, and developed an interactive version of U-net method we call iUnet. The iUnet is fully trained to produce the best possible initial segmentation. In interactive mode it is additionally trained on a partial set of layers on user generated scribbles. We compare initial segmentation performance of iFCN and iUnet on a 100CT dataset using dice similarity coefficient analysis. Secondly, we assessed the performance gain in interactive use with three observers on segmentation quality and time. Average automated baseline performance was 78% (iUnet) versus 72% (FCN). Manual and semi-automatic segmentation performance was: 87% in 15 min. for manual, and 86% in 8 min. for iUNet. We conclude that iUnet provides a better baseline than iFCN and can reach expert manual performance significantly faster than manual segmentation in case of pancreas CT. Our novel iUnet architecture is modality and organ agnostic and can be a potential novel solution for semi-automatic medical imaging segmentation in general.
Collapse
Affiliation(s)
- T G W Boers
- Faculty of Science and Technology, University of Twente, Enschede, The Netherlands
| | | | | | | | | | | | | | | | | |
Collapse
|