1
|
Williams H, Thompson HM, Lee C, Rangnekar A, Gomez JT, Widmar M, Wei IH, Pappou EP, Nash GM, Weiser MR, Paty PB, Smith JJ, Veeraraghavan H, Garcia-Aguilar J. Assessing Endoscopic Response in Locally Advanced Rectal Cancer Treated with Total Neoadjuvant Therapy: Development and Validation of a Highly Accurate Convolutional Neural Network. Ann Surg Oncol 2024:10.1245/s10434-024-15311-y. [PMID: 38700799 DOI: 10.1245/s10434-024-15311-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2024] [Accepted: 04/01/2024] [Indexed: 05/31/2024]
Abstract
BACKGROUND Rectal tumors display varying degrees of response to total neoadjuvant therapy (TNT). We evaluated the performance of a convolutional neural network (CNN) in interpreting endoscopic images of either a non-complete response to TNT or local regrowth during watch-and-wait surveillance. METHODS Endoscopic images from stage II/III rectal cancers treated with TNT from 2012 to 2020 at a single institution were retrospectively reviewed. Images were labelled as Tumor or No Tumor based on endoscopy timing (before, during, or after treatment) and the tumor's endoluminal response. A CNN was trained using ResNet-50 architecture. The area under the curve (AUC) was analyzed during training and for two test sets. The main test set included images of tumors treated with TNT. The other contained images of local regrowth. The model's performance was compared to sixteen surgeons and surgical trainees who evaluated 119 images for evidence of tumor. Fleiss' kappa was calculated by respondent experience level. RESULTS A total of 2717 images from 288 patients were included; 1407 (51.8%) contained tumor. The AUC was 0.99, 0.98, and 0.92 for training, main test, and local regrowth test sets. The model performed on par with surgeons of all experience levels for the main test set. Interobserver agreement was good ( k = 0.71-0.81). All groups outperformed the model in identifying tumor from images of local regrowth. Interobserver agreement was fair to moderate ( k = 0.24-0.52). CONCLUSIONS A highly accurate CNN matched the performance of colorectal surgeons in identifying a noncomplete response to TNT. However, the model demonstrated suboptimal accuracy when analyzing images of local regrowth.
Collapse
Affiliation(s)
- Hannah Williams
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Hannah M Thompson
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Christina Lee
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Aneesh Rangnekar
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, USA
| | - Jorge T Gomez
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, USA
| | - Maria Widmar
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Iris H Wei
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Emmanouil P Pappou
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Garrett M Nash
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Martin R Weiser
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Philip B Paty
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - J Joshua Smith
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Harini Veeraraghavan
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, USA
| | - Julio Garcia-Aguilar
- Department of Surgery, Colorectal Service, Memorial Sloan Kettering Cancer Center, New York, NY, USA.
| |
Collapse
|
2
|
Gong EJ, Bang CS, Lee JJ. Computer-aided diagnosis in real-time endoscopy for all stages of gastric carcinogenesis: Development and validation study. United European Gastroenterol J 2024; 12:487-495. [PMID: 38400815 DOI: 10.1002/ueg2.12551] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Accepted: 01/14/2024] [Indexed: 02/26/2024] Open
Abstract
OBJECTIVE Using endoscopic images, we have previously developed computer-aided diagnosis models to predict the histopathology of gastric neoplasms. However, no model that categorizes every stage of gastric carcinogenesis has been published. In this study, a deep-learning-based diagnosis model was developed and validated to automatically classify all stages of gastric carcinogenesis, including atrophy and intestinal metaplasia, in endoscopy images. DESIGN A total of 18,701 endoscopic images were collected retrospectively and randomly divided into train, validation, and internal-test datasets in an 8:1:1 ratio. The primary outcome was lesion-classification accuracy in six categories: normal/atrophy/intestinal metaplasia/dysplasia/early /advanced gastric cancer. External-validation of performance in the established model used 1427 novel images from other institutions that were not used in training, validation, or internal-tests. RESULTS The internal-test lesion-classification accuracy was 91.2% (95% confidence interval: 89.9%-92.5%). For performance validation, the established model achieved an accuracy of 82.3% (80.3%-84.3%). The external-test per-class receiver operating characteristic in the diagnosis of atrophy and intestinal metaplasia was 93.4 ± 0% and 91.3 ± 0%, respectively. CONCLUSIONS The established model demonstrated high performance in the diagnosis of preneoplastic lesions (atrophy and intestinal metaplasia) as well as gastric neoplasms.
Collapse
Affiliation(s)
- Eun Jeong Gong
- Department of Internal Medicine, Hallym University College of Medicine, Chuncheon, Korea
- Institute for Liver and Digestive Diseases, Hallym University, Chuncheon, Korea
- Institute of New Frontier Research, Hallym University College of Medicine, Chuncheon, Korea
| | - Chang Seok Bang
- Department of Internal Medicine, Hallym University College of Medicine, Chuncheon, Korea
- Institute for Liver and Digestive Diseases, Hallym University, Chuncheon, Korea
- Institute of New Frontier Research, Hallym University College of Medicine, Chuncheon, Korea
- Division of Big Data and Artificial Intelligence, Chuncheon Sacred Heart Hospital, Chuncheon, Korea
| | - Jae Jun Lee
- Institute of New Frontier Research, Hallym University College of Medicine, Chuncheon, Korea
- Division of Big Data and Artificial Intelligence, Chuncheon Sacred Heart Hospital, Chuncheon, Korea
- Department of Anesthesiology and Pain Medicine, Hallym University College of Medicine, Chuncheon, Korea
| |
Collapse
|
3
|
Rey JF. As how artificial intelligence is revolutionizing endoscopy. Clin Endosc 2024; 57:302-308. [PMID: 38454543 PMCID: PMC11133999 DOI: 10.5946/ce.2023.230] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Revised: 10/11/2023] [Accepted: 10/15/2023] [Indexed: 03/09/2024] Open
Abstract
With incessant advances in information technology and its implications in all domains of our lives, artificial intelligence (AI) has emerged as a requirement for improved machine performance. This brings forth the query of how this can benefit endoscopists and improve both diagnostic and therapeutic endoscopy in each part of the gastrointestinal tract. Additionally, it also raises the question of the recent benefits and clinical usefulness of this new technology in daily endoscopic practice. There are two main categories of AI systems: computer-assisted detection (CADe) for lesion detection and computer-assisted diagnosis (CADx) for optical biopsy and lesion characterization. Quality assurance is the next step in the complete monitoring of high-quality colonoscopies. In all cases, computer-aided endoscopy is used, as the overall results rely on the physician. Video capsule endoscopy is a unique example in which a computer operates a device, stores multiple images, and performs an accurate diagnosis. While there are many expectations, we need to standardize and assess various software packages. It is important for healthcare providers to support this new development and make its use an obligation in daily clinical practice. In summary, AI represents a breakthrough in digestive endoscopy. Screening for gastric and colonic cancer detection should be improved, particularly outside expert centers. Prospective and multicenter trials are mandatory before introducing new software into clinical practice.
Collapse
Affiliation(s)
- Jean-Francois Rey
- Institut Arnaut Tzanck Gastrointestinal Unt, Saint Laurent du Var, France
| |
Collapse
|
4
|
Li JW, Wang LM, Ichimasa K, Lin KW, Ngu JCY, Ang TL. Use of artificial intelligence in the management of T1 colorectal cancer: a new tool in the arsenal or is deep learning out of its depth? Clin Endosc 2024; 57:24-35. [PMID: 37743068 PMCID: PMC10834280 DOI: 10.5946/ce.2023.036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Accepted: 05/11/2023] [Indexed: 09/26/2023] Open
Abstract
The field of artificial intelligence is rapidly evolving, and there has been an interest in its use to predict the risk of lymph node metastasis in T1 colorectal cancer. Accurately predicting lymph node invasion may result in fewer patients undergoing unnecessary surgeries; conversely, inadequate assessments will result in suboptimal oncological outcomes. This narrative review aims to summarize the current literature on deep learning for predicting the probability of lymph node metastasis in T1 colorectal cancer, highlighting areas of potential application and barriers that may limit its generalizability and clinical utility.
Collapse
Affiliation(s)
- James Weiquan Li
- Department of Gastroenterology and Hepatology, Changi General Hospital, Singapore Health Services, Singapore
- Academic Medicine Center, Duke-NUS Medical School, Singapore
| | - Lai Mun Wang
- Department of Laboratory Medicine, Changi General Hospital, Singapore Health Services, Singapore
| | - Katsuro Ichimasa
- Digestive Disease Center, Showa University Northern Yokohama Hospital, Yokohama, Japan
- Yong Loo Lin School of Medicine, National University of Singapore, Singapore
| | - Kenneth Weicong Lin
- Department of Gastroenterology and Hepatology, Changi General Hospital, Singapore Health Services, Singapore
- Academic Medicine Center, Duke-NUS Medical School, Singapore
| | - James Chi-Yong Ngu
- Department of General Surgery, Changi General Hospital, Singapore Health Services, Singapore
| | - Tiing Leong Ang
- Department of Gastroenterology and Hepatology, Changi General Hospital, Singapore Health Services, Singapore
- Academic Medicine Center, Duke-NUS Medical School, Singapore
| |
Collapse
|
5
|
Zhu S, Gao J, Liu L, Yin M, Lin J, Xu C, Xu C, Zhu J. Public Imaging Datasets of Gastrointestinal Endoscopy for Artificial Intelligence: a Review. J Digit Imaging 2023; 36:2578-2601. [PMID: 37735308 PMCID: PMC10584770 DOI: 10.1007/s10278-023-00844-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Revised: 05/03/2023] [Accepted: 05/03/2023] [Indexed: 09/23/2023] Open
Abstract
With the advances in endoscopic technologies and artificial intelligence, a large number of endoscopic imaging datasets have been made public to researchers around the world. This study aims to review and introduce these datasets. An extensive literature search was conducted to identify appropriate datasets in PubMed, and other targeted searches were conducted in GitHub, Kaggle, and Simula to identify datasets directly. We provided a brief introduction to each dataset and evaluated the characteristics of the datasets included. Moreover, two national datasets in progress were discussed. A total of 40 datasets of endoscopic images were included, of which 34 were accessible for use. Basic and detailed information on each dataset was reported. Of all the datasets, 16 focus on polyps, and 6 focus on small bowel lesions. Most datasets (n = 16) were constructed by colonoscopy only, followed by normal gastrointestinal endoscopy and capsule endoscopy (n = 9). This review may facilitate the usage of public dataset resources in endoscopic research.
Collapse
Affiliation(s)
- Shiqi Zhu
- Department of Gastroenterology, The First Affiliated Hospital of Soochow University, 188 Shizi Street, Suzhou , Jiangsu, 215000, China
- Suzhou Clinical Center of Digestive Diseases, Suzhou, 215000, China
| | - Jingwen Gao
- Department of Gastroenterology, The First Affiliated Hospital of Soochow University, 188 Shizi Street, Suzhou , Jiangsu, 215000, China
- Suzhou Clinical Center of Digestive Diseases, Suzhou, 215000, China
| | - Lu Liu
- Department of Gastroenterology, The First Affiliated Hospital of Soochow University, 188 Shizi Street, Suzhou , Jiangsu, 215000, China
- Suzhou Clinical Center of Digestive Diseases, Suzhou, 215000, China
| | - Minyue Yin
- Department of Gastroenterology, The First Affiliated Hospital of Soochow University, 188 Shizi Street, Suzhou , Jiangsu, 215000, China
- Suzhou Clinical Center of Digestive Diseases, Suzhou, 215000, China
| | - Jiaxi Lin
- Department of Gastroenterology, The First Affiliated Hospital of Soochow University, 188 Shizi Street, Suzhou , Jiangsu, 215000, China
- Suzhou Clinical Center of Digestive Diseases, Suzhou, 215000, China
| | - Chang Xu
- Department of Gastroenterology, The First Affiliated Hospital of Soochow University, 188 Shizi Street, Suzhou , Jiangsu, 215000, China
- Suzhou Clinical Center of Digestive Diseases, Suzhou, 215000, China
| | - Chunfang Xu
- Department of Gastroenterology, The First Affiliated Hospital of Soochow University, 188 Shizi Street, Suzhou , Jiangsu, 215000, China.
- Suzhou Clinical Center of Digestive Diseases, Suzhou, 215000, China.
| | - Jinzhou Zhu
- Department of Gastroenterology, The First Affiliated Hospital of Soochow University, 188 Shizi Street, Suzhou , Jiangsu, 215000, China.
- Suzhou Clinical Center of Digestive Diseases, Suzhou, 215000, China.
| |
Collapse
|
6
|
Zammarchi I, Santacroce G, Iacucci M. Next-Generation Endoscopy in Inflammatory Bowel Disease. Diagnostics (Basel) 2023; 13:2547. [PMID: 37568910 PMCID: PMC10417286 DOI: 10.3390/diagnostics13152547] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Revised: 07/28/2023] [Accepted: 07/29/2023] [Indexed: 08/13/2023] Open
Abstract
Endoscopic healing is recognized as a primary treatment goal in Inflammatory Bowel Disease (IBD). However, endoscopic remission may not reflect histological remission, which is crucial to achieving favorable long-term outcomes. The development of new advanced techniques has revolutionized the field of IBD assessment and management. These tools can accurately assess vascular and mucosal features, drawing endoscopy closer to histology. Moreover, they can enhance the detection and characterization of IBD-related dysplasia. Given the persistent challenge of interobserver variability, a more standardized approach to endoscopy is warranted, and the integration of artificial intelligence (AI) holds promise for addressing this limitation. Additionally, although molecular endoscopy is still in its infancy, it is a promising tool to forecast response to therapy. This review provides an overview of advanced endoscopic techniques, including dye-based and dye-less chromoendoscopy, and in vivo histological examinations with probe-based confocal laser endomicroscopy and endocytoscopy. The remarkable contribution of these tools to IBD management, especially when integrated with AI, is discussed. Specific attention is given to their role in improving disease assessment, detection, and characterization of IBD-associated lesions, and predicting disease-related outcomes.
Collapse
Affiliation(s)
| | | | - Marietta Iacucci
- APC Microbiome Ireland, College of Medicine and Health, University College Cork, T12 R229 Cork, Ireland; (I.Z.); (G.S.)
| |
Collapse
|
7
|
Gong EJ, Bang CS, Lee JJ, Yang YJ, Baik GH. Impact of the Volume and Distribution of Training Datasets in the Development of Deep-Learning Models for the Diagnosis of Colorectal Polyps in Endoscopy Images. J Pers Med 2022; 12:jpm12091361. [PMID: 36143146 PMCID: PMC9505038 DOI: 10.3390/jpm12091361] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2022] [Revised: 08/13/2022] [Accepted: 08/19/2022] [Indexed: 11/16/2022] Open
Abstract
Background: Establishment of an artificial intelligence model in gastrointestinal endoscopy has no standardized dataset. The optimal volume or class distribution of training datasets has not been evaluated. An artificial intelligence model was previously created by the authors to classify endoscopic images of colorectal polyps into four categories, including advanced colorectal cancer, early cancers/high-grade dysplasia, tubular adenoma, and nonneoplasm. The aim of this study was to evaluate the impact of the volume and distribution of training dataset classes in the development of deep-learning models for colorectal polyp histopathology prediction from endoscopic images. Methods: The same 3828 endoscopic images that were used to create earlier models were used. An additional 6838 images were used to find the optimal volume and class distribution for a deep-learning model. Various amounts of data volume and class distributions were tried to establish deep-learning models. The training of deep-learning models uniformly used no-code platform Neuro-T. Accuracy was the primary outcome on four-class prediction. Results: The highest internal-test classification accuracy in the original dataset, doubled dataset, and tripled dataset was commonly shown by doubling the proportion of data for fewer categories (2:2:1:1 for advanced colorectal cancer: early cancers/high-grade dysplasia: tubular adenoma: non-neoplasm). Doubling the proportion of data for fewer categories in the original dataset showed the highest accuracy (86.4%, 95% confidence interval: 85.0–97.8%) compared to that of the doubled or tripled dataset. The total required number of images in this performance was only 2418 images. Gradient-weighted class activation mapping confirmed that the part that the deep-learning model pays attention to coincides with the part that the endoscopist pays attention to. Conclusion: As a result of a data-volume-dependent performance plateau in the classification model of colonoscopy, a dataset that has been doubled or tripled is not always beneficial to training. Deep-learning models would be more accurate if the proportion of fewer category lesions was increased.
Collapse
Affiliation(s)
- Eun Jeong Gong
- Department of Internal Medicine, Hallym University College of Medicine, Chuncheon 24253, Korea
- Institute of New Frontier Research, Hallym University College of Medicine, Chuncheon 24253, Korea
| | - Chang Seok Bang
- Department of Internal Medicine, Hallym University College of Medicine, Chuncheon 24253, Korea
- Institute of New Frontier Research, Hallym University College of Medicine, Chuncheon 24253, Korea
- Correspondence: ; Tel.: +82-33-240-5821; Fax: +82-33-241-8064
| | - Jae Jun Lee
- Institute of New Frontier Research, Hallym University College of Medicine, Chuncheon 24253, Korea
- Department of Anesthesiology and Pain Medicine, Hallym University College of Medicine, Chuncheon 24253, Korea
| | - Young Joo Yang
- Department of Internal Medicine, Hallym University College of Medicine, Chuncheon 24253, Korea
| | - Gwang Ho Baik
- Department of Internal Medicine, Hallym University College of Medicine, Chuncheon 24253, Korea
| |
Collapse
|