1
|
Khalid M, Deivasigamani S, V S, Rajendran S. An efficient colorectal cancer detection network using atrous convolution with coordinate attention transformer and histopathological images. Sci Rep 2024; 14:19109. [PMID: 39154091 PMCID: PMC11330491 DOI: 10.1038/s41598-024-70117-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2024] [Accepted: 08/13/2024] [Indexed: 08/19/2024] Open
Abstract
The second most common type of malignant tumor worldwide is colorectal cancer. Histopathology image analysis offers crucial data for the clinical diagnosis of colorectal cancer. Currently, deep learning techniques are applied to enhance cancer classification and tumor localization in histopathological image analysis. Moreover, traditional deep learning techniques might loss integrated information in the image while evaluating thousands of patches recovered from whole slide images (WSIs). This research proposes a novel colorectal cancer detection network (CCDNet) that combines coordinate attention transformer with atrous convolution. CCDNet first denoises the input histopathological image using a Wiener based Midpoint weighted non-local means filter (WMW-NLM) for guaranteeing precise diagnoses and maintain image features. Also, a novel atrous convolution with coordinate attention transformer (AConvCAT) is introduced, which successfully combines the advantages of two networks to classify colorectal tissue at various scales by capturing local and global information. Further, coordinate attention model is integrated with a Cross-shaped window (CrSWin) transformer for capturing tiny changes in colorectal tissue from multiple angles. The proposed CCDNet achieved accuracy rates of 98.61% and 98.96%, on the colorectal histological image and NCT-CRC-HE-100 K datasets correspondingly. The comparison analysis demonstrates that the suggested framework performed better than the most advanced methods already in use. In hospitals, clinicians can use the proposed CCDNet to verify the diagnosis.
Collapse
Affiliation(s)
- Majdi Khalid
- Department of Computer Science and Artificial Intelligence, College of Computing, Umm Al-Qura University, 21955, Makkah, Saudi Arabia
| | - Sugitha Deivasigamani
- Department of Computer Science and Engineering, University College of Engineering, A Constituent College of Anna University, Thirukkuvalai, Chennai, India
| | - Sathiya V
- Department of Computer Science and Engineering, Panimalar Engineering College, Chennai, 600123, India
| | - Surendran Rajendran
- Department of Computer Science and Engineering, Saveetha School of Engineering, Saveetha Institute of Medical and Technical Sciences, Chennai, 602105, Tamil Nadu, India.
| |
Collapse
|
2
|
Stirling CE, Neeteson NJ, Walker REA, Boyd SK. Deep learning-based automated detection and segmentation of bone and traumatic bone marrow lesions from MRI following an acute ACL tear. Comput Biol Med 2024; 178:108791. [PMID: 38905892 DOI: 10.1016/j.compbiomed.2024.108791] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Revised: 06/17/2024] [Accepted: 06/18/2024] [Indexed: 06/23/2024]
Abstract
INTRODUCTION Traumatic bone marrow lesions (BML) are frequently identified on knee MRI scans in patients following an acute full-thickness, complete ACL tear. BMLs coincide with regions of elevated localized bone loss, and studies suggest these may act as a precursor to the development of post-traumatic osteoarthritis. This study addresses the labour-intensive manual assessment of BMLs by using a 3D U-Net for automated identification and segmentation from MRI scans. METHODS A multi-task learning approach was used to segment both bone and BML from T2 fat-suppressed (FS) fast spin echo (FSE) MRI sequences for BML assessment. Training and testing utilized datasets from individuals with complete ACL tears, employing a five-fold cross-validation approach and pre-processing involved image intensity normalization and data augmentation. A post-processing algorithm was developed to improve segmentation and remove outliers. Training and testing datasets were acquired from different studies with similar imaging protocol to assess the model's performance robustness across different populations and acquisition conditions. RESULTS The 3D U-Net model exhibited effectiveness in semantic segmentation, while post-processing enhanced segmentation accuracy and precision through morphological operations. The trained model with post-processing achieved a Dice similarity coefficient (DSC) of 0.75 ± 0.08 (mean ± std) and a precision of 0.87 ± 0.07 for BML segmentation on testing data. Additionally, the trained model with post-processing achieved a DSC of 0.93 ± 0.02 and a precision of 0.92 ± 0.02 for bone segmentation on testing data. This demonstrates the approach's high accuracy for capturing true positives and effectively minimizing false positives in the identification and segmentation of bone structures. CONCLUSION Automated segmentation methods are a valuable tool for clinicians and researchers, streamlining the assessment of BMLs and allowing for longitudinal assessments. This study presents a model with promising clinical efficacy and provides a quantitative approach for bone-related pathology research and diagnostics.
Collapse
Affiliation(s)
- Callie E Stirling
- Department of Biomedical Engineering, Schulich School of Engineering, University of Calgary, Calgary, Canada; McCaig Institute for Bone and Joint Health, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Nathan J Neeteson
- Department of Biomedical Engineering, Schulich School of Engineering, University of Calgary, Calgary, Canada; McCaig Institute for Bone and Joint Health, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Richard E A Walker
- McCaig Institute for Bone and Joint Health, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada; Department of Radiology, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada
| | - Steven K Boyd
- Department of Biomedical Engineering, Schulich School of Engineering, University of Calgary, Calgary, Canada; McCaig Institute for Bone and Joint Health, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada; Department of Radiology, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada.
| |
Collapse
|
3
|
Hamada K, Murakami R, Ueda A, Kashima Y, Miyagawa C, Taki M, Yamanoi K, Yamaguchi K, Hamanishi J, Minamiguchi S, Matsumura N, Mandai M. A Deep Learning-Based Assessment Pipeline for Intraepithelial and Stromal Tumor-Infiltrating Lymphocytes in High-Grade Serous Ovarian Carcinoma. THE AMERICAN JOURNAL OF PATHOLOGY 2024; 194:1272-1284. [PMID: 38537936 DOI: 10.1016/j.ajpath.2024.02.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/26/2023] [Revised: 01/22/2024] [Accepted: 02/21/2024] [Indexed: 04/07/2024]
Abstract
Tumor-infiltrating lymphocytes (TILs) are associated with improved survival in patients with epithelial ovarian cancer. However, TIL evaluation has not been used in routine clinical practice because of reproducibility issues. The current study developed two convolutional neural network models to detect TILs and to determine their spatial location in whole slide images, and established a spatial assessment pipeline to objectively quantify intraepithelial and stromal TILs in patients with high-grade serous ovarian carcinoma. The predictions of the established models showed a significant positive correlation with the number of CD8+ T cells and immune gene expressions. Patients with a higher density of intraepithelial TILs had a significantly prolonged overall survival and progression-free survival in multiple cohorts. On the basis of the density of intraepithelial and stromal TILs, patients were classified into three immunophenotypes: immune inflamed, excluded, and desert. The immune-desert subgroup showed the worst prognosis. Gene expression analysis showed that the immune-desert subgroup had lower immune cytolytic activity and T-cell-inflamed gene-expression profile scores, whereas the immune-excluded subgroup had higher expression of interferon-γ and programmed death 1 receptor signaling pathway. The established evaluation method provided detailed and comprehensive quantification of intraepithelial and stromal TILs throughout hematoxylin and eosin-stained slides. It has potential for clinical application for personalized treatment of patients with ovarian cancer.
Collapse
Affiliation(s)
- Kohei Hamada
- Department of Gynecology and Obstetrics, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Ryusuke Murakami
- Department of Gynecology and Obstetrics, Kyoto University Graduate School of Medicine, Kyoto, Japan.
| | - Akihiko Ueda
- Department of Gynecology and Obstetrics, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Yoko Kashima
- Department of Obstetrics and Gynecology, Kindai University Faculty of Medicine, Osaka, Japan
| | - Chiho Miyagawa
- Department of Obstetrics and Gynecology, Kindai University Faculty of Medicine, Osaka, Japan
| | - Mana Taki
- Department of Gynecology and Obstetrics, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Koji Yamanoi
- Department of Gynecology and Obstetrics, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Ken Yamaguchi
- Department of Gynecology and Obstetrics, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Junzo Hamanishi
- Department of Gynecology and Obstetrics, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Sachiko Minamiguchi
- Department of Diagnostic Pathology, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Noriomi Matsumura
- Department of Obstetrics and Gynecology, Kindai University Faculty of Medicine, Osaka, Japan
| | - Masaki Mandai
- Department of Gynecology and Obstetrics, Kyoto University Graduate School of Medicine, Kyoto, Japan
| |
Collapse
|
4
|
Schäfer R, Nicke T, Höfener H, Lange A, Merhof D, Feuerhake F, Schulz V, Lotz J, Kiessling F. Overcoming data scarcity in biomedical imaging with a foundational multi-task model. NATURE COMPUTATIONAL SCIENCE 2024; 4:495-509. [PMID: 39030386 PMCID: PMC11288886 DOI: 10.1038/s43588-024-00662-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Accepted: 06/17/2024] [Indexed: 07/21/2024]
Abstract
Foundational models, pretrained on a large scale, have demonstrated substantial success across non-medical domains. However, training these models typically requires large, comprehensive datasets, which contrasts with the smaller and more specialized datasets common in biomedical imaging. Here we propose a multi-task learning strategy that decouples the number of training tasks from memory requirements. We trained a universal biomedical pretrained model (UMedPT) on a multi-task database including tomographic, microscopic and X-ray images, with various labeling strategies such as classification, segmentation and object detection. The UMedPT foundational model outperformed ImageNet pretraining and previous state-of-the-art models. For classification tasks related to the pretraining database, it maintained its performance with only 1% of the original training data and without fine-tuning. For out-of-domain tasks it required only 50% of the original training data. In an external independent validation, imaging features extracted using UMedPT proved to set a new standard for cross-center transferability.
Collapse
Affiliation(s)
- Raphael Schäfer
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Till Nicke
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Henning Höfener
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Annkristin Lange
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Dorit Merhof
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
- Institute of Image Analysis and Computer Vision, Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany
| | - Friedrich Feuerhake
- Institute for Pathology, Hannover Medical School, Hanover, Germany
- Institute for Neuropathology, Medical Center, University of Freiburg, Freiburg, Germany
| | - Volkmar Schulz
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
- Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany
| | - Johannes Lotz
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
| | - Fabian Kiessling
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
- Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany.
| |
Collapse
|
5
|
Furlanello C, Bussola N, Merzi N, Pievani Trapletti G, Cadei M, Del Sordo R, Sidoni A, Ricci C, Lanzarotto F, Parigi TL, Villanacci V. The development of artificial intelligence in the histological diagnosis of Inflammatory Bowel Disease (IBD-AI). Dig Liver Dis 2024:S1590-8658(24)00791-6. [PMID: 38853093 DOI: 10.1016/j.dld.2024.05.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/01/2023] [Revised: 05/12/2024] [Accepted: 05/28/2024] [Indexed: 06/11/2024]
Abstract
BACKGROUND Inflammatory bowel disease (IBD) includes Crohn's Disease (CD) and Ulcerative Colitis (UC). Correct diagnosis requires the identification of precise morphological features such basal plasmacytosis. However, histopathological interpretation can be challenging, and it is subject to high variability. AIM The IBD-Artificial Intelligence (AI) project aims at the development of an AI-based evaluation system to support the diagnosis of IBD, semi-automatically quantifying basal plasmacytosis. METHODS A deep learning model was trained to detect and quantify plasma cells on a public dataset of 4981 annotated images. The model was then tested on an external validation cohort of 356 intestinal biopsies of CD, UC and healthy controls. AI diagnostic performance was calculated compared to human gold standard. RESULTS The system correctly found that CD and UC samples had a greater prevalence of basal plasma cells with mean number of PCs within ROIs of 38.22 (95 % CI: 31.73, 49.04) for CD, 55.16 (46.57, 65.93) for UC, and 17.25 (CI: 12.17, 27.05) for controls. Overall, OR=4.968 (CI: 1.835, 14.638) was found for IBD compared to normal mucosa (CD: +59 %; UC: +129 %). Additionally, as expected, UC samples were found to have more plasma cells in colon than CD cases. CONCLUSION Our model accurately replicated human assessment of basal plasmacytosis, underscoring the value of AI models as a potential aid IBD diagnosis.
Collapse
Affiliation(s)
| | | | | | | | - Moris Cadei
- Institute of Pathology, ASST Spedali Civili and University of Brescia, Brescia, Italy
| | - Rachele Del Sordo
- Department of Medicine and Surgery, Section of Anatomic Pathology and Histology, Medical School, University of Perugia, Perugia, Italy
| | - Angelo Sidoni
- Department of Medicine and Surgery, Section of Anatomic Pathology and Histology, Medical School, University of Perugia, Perugia, Italy
| | - Chiara Ricci
- Gastroenterology Unit, Clinical and Experimental Sciences Department, Spedali Civili Hospital, University of Brescia, Brescia, Italy
| | - Francesco Lanzarotto
- Gastroenterology Unit, Clinical and Experimental Sciences Department, Spedali Civili Hospital, University of Brescia, Brescia, Italy
| | - Tommaso Lorenzo Parigi
- Division of Immunology, Transplantation and Infectious Disease, University Vita-Salute San Raffaele, Milan, Italy
| | - Vincenzo Villanacci
- Institute of Pathology, ASST Spedali Civili and University of Brescia, Brescia, Italy.
| |
Collapse
|
6
|
Bui DC, Song B, Kim K, Kwak JT. DAX-Net: A dual-branch dual-task adaptive cross-weight feature fusion network for robust multi-class cancer classification in pathology images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 248:108112. [PMID: 38479146 DOI: 10.1016/j.cmpb.2024.108112] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/22/2023] [Revised: 02/15/2024] [Accepted: 03/01/2024] [Indexed: 04/02/2024]
Abstract
BACKGROUND AND OBJECTIVE Multi-class cancer classification has been extensively studied in digital and computational pathology due to its importance in clinical decision-making. Numerous computational tools have been proposed for various types of cancer classification. Many of them are built based on convolutional neural networks. Recently, Transformer-style networks have shown to be effective for cancer classification. Herein, we present a hybrid design that leverages both convolutional neural networks and transformer architecture to obtain superior performance in cancer classification. METHODS We propose a dual-branch dual-task adaptive cross-weight feature fusion network, called DAX-Net, which exploits heterogeneous feature representations from the convolutional neural network and Transformer network, adaptively combines them to boost their representation power, and conducts cancer classification as categorical classification and ordinal classification. For an efficient and effective optimization of the proposed model, we introduce two loss functions that are tailored to the two classification tasks. RESULTS To evaluate the proposed method, we employed colorectal and prostate cancer datasets, of which each contains both in-domain and out-of-domain test sets. For colorectal cancer, the proposed method obtained an accuracy of 88.4%, a quadratic kappa score of 0.945, and an F1 score of 0.831 for the in-domain test set, and 84.4%, 0.910, and 0.768 for the out-of-domain test set. For prostate cancer, it achieved an accuracy of 71.6%, a kappa score of 0.635, and an F1 score of 0.655 for the in-domain test set, 79.2% accuracy, 0.721 kappa score, and 0.686 F1 score for the first out-of-domain test set, and 58.1% accuracy, 0.564 kappa score, and 0.493 F1 score for the second out-of-domain test set. It is worth noting that the performance of the proposed method outperformed other competitors by significant margins, in particular, with respect to the out-of-domain test sets. CONCLUSIONS The experimental results demonstrate that the proposed method is not only accurate but also robust to varying conditions of the test sets in comparison to several, related methods. These results suggest that the proposed method can facilitate automated cancer classification in various clinical settings.
Collapse
Affiliation(s)
- Doanh C Bui
- School of Electrical Engineering, Korea University, Seoul, 02841, Republic of Korea
| | - Boram Song
- Department of Pathology, Kangbuk Samsung Hospital, Sungkyunkwan University School of Medicine, Seoul, 03181, Republic of Korea
| | - Kyungeun Kim
- Department of Pathology, Kangbuk Samsung Hospital, Sungkyunkwan University School of Medicine, Seoul, 03181, Republic of Korea
| | - Jin Tae Kwak
- School of Electrical Engineering, Korea University, Seoul, 02841, Republic of Korea.
| |
Collapse
|
7
|
Hörst F, Rempe M, Heine L, Seibold C, Keyl J, Baldini G, Ugurel S, Siveke J, Grünwald B, Egger J, Kleesiek J. CellViT: Vision Transformers for precise cell segmentation and classification. Med Image Anal 2024; 94:103143. [PMID: 38507894 DOI: 10.1016/j.media.2024.103143] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Revised: 02/14/2024] [Accepted: 03/12/2024] [Indexed: 03/22/2024]
Abstract
Nuclei detection and segmentation in hematoxylin and eosin-stained (H&E) tissue images are important clinical tasks and crucial for a wide range of applications. However, it is a challenging task due to nuclei variances in staining and size, overlapping boundaries, and nuclei clustering. While convolutional neural networks have been extensively used for this task, we explore the potential of Transformer-based networks in combination with large scale pre-training in this domain. Therefore, we introduce a new method for automated instance segmentation of cell nuclei in digitized tissue samples using a deep learning architecture based on Vision Transformer called CellViT. CellViT is trained and evaluated on the PanNuke dataset, which is one of the most challenging nuclei instance segmentation datasets, consisting of nearly 200,000 annotated nuclei into 5 clinically important classes in 19 tissue types. We demonstrate the superiority of large-scale in-domain and out-of-domain pre-trained Vision Transformers by leveraging the recently published Segment Anything Model and a ViT-encoder pre-trained on 104 million histological image patches - achieving state-of-the-art nuclei detection and instance segmentation performance on the PanNuke dataset with a mean panoptic quality of 0.50 and an F1-detection score of 0.83. The code is publicly available at https://github.com/TIO-IKIM/CellViT.
Collapse
Affiliation(s)
- Fabian Hörst
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), 45131 Essen, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), 45147 Essen, Germany.
| | - Moritz Rempe
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), 45131 Essen, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), 45147 Essen, Germany
| | - Lukas Heine
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), 45131 Essen, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), 45147 Essen, Germany
| | - Constantin Seibold
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), 45131 Essen, Germany; Clinic for Nuclear Medicine, University Hospital Essen (AöR), 45147 Essen, Germany
| | - Julius Keyl
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), 45131 Essen, Germany; Institute of Pathology, University Hospital Essen (AöR), 45147 Essen, Germany
| | - Giulia Baldini
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), 45131 Essen, Germany; Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen (AöR), 45147 Essen, Germany
| | - Selma Ugurel
- Department of Dermatology, University Hospital Essen (AöR), 45147 Essen, Germany; German Cancer Consortium (DKTK, Partner site Essen), 69120 Heidelberg, Germany
| | - Jens Siveke
- West German Cancer Center, partner site Essen, a partnership between German Cancer Research Center (DKFZ) and University Hospital Essen, University Hospital Essen (AöR), 45147 Essen, Germany; Bridge Institute of Experimental Tumor Therapy (BIT) and Division of Solid Tumor Translational Oncology (DKTK), West German Cancer Center Essen, University Hospital Essen (AöR), University of Duisburg-Essen, 45147 Essen, Germany
| | - Barbara Grünwald
- Department of Urology, West German Cancer Center, 45147 University Hospital Essen (AöR), Germany; Princess Margaret Cancer Centre, M5G 2M9 Toronto, Ontario, Canada
| | - Jan Egger
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), 45131 Essen, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), 45147 Essen, Germany
| | - Jens Kleesiek
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), 45131 Essen, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), 45147 Essen, Germany; German Cancer Consortium (DKTK, Partner site Essen), 69120 Heidelberg, Germany; Department of Physics, TU Dortmund University, 44227 Dortmund, Germany
| |
Collapse
|
8
|
Wang P, Kline TL, Missert AD, Cook CJ, Callstrom MR, Chan A, Hartman RP, Kelm ZS, Korfiatis P. A Classification-Based Adaptive Segmentation Pipeline: Feasibility Study Using Polycystic Liver Disease and Metastases from Colorectal Cancer CT Images. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024:10.1007/s10278-024-01072-3. [PMID: 38587766 DOI: 10.1007/s10278-024-01072-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/10/2023] [Revised: 02/29/2024] [Accepted: 03/01/2024] [Indexed: 04/09/2024]
Abstract
Automated segmentation tools often encounter accuracy and adaptability issues when applied to images of different pathology. The purpose of this study is to explore the feasibility of building a workflow to efficiently route images to specifically trained segmentation models. By implementing a deep learning classifier to automatically classify the images and route them to appropriate segmentation models, we hope that our workflow can segment the images with different pathology accurately. The data we used in this study are 350 CT images from patients affected by polycystic liver disease and 350 CT images from patients presenting with liver metastases from colorectal cancer. All images had the liver manually segmented by trained imaging analysts. Our proposed adaptive segmentation workflow achieved a statistically significant improvement for the task of total liver segmentation compared to the generic single-segmentation model (non-parametric Wilcoxon signed rank test, n = 100, p-value << 0.001). This approach is applicable in a wide range of scenarios and should prove useful in clinical implementations of segmentation pipelines.
Collapse
Affiliation(s)
- Peilong Wang
- Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | | | | | - Cole J Cook
- Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | | | - Alex Chan
- Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | | | - Zachary S Kelm
- Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | | |
Collapse
|
9
|
Vu VG, Hoang AD, Phan TP, Nguyen ND, Nguyen TT, Nguyen DN, Dao NP, Doan TPL, Nguyen TTH, Trinh TH, Pham TLQ, Le TTT, Thi Hanh P, Pham VT, Tran VC, Vu DL, Tran VL, Nguyen TTT, Pham CP, Pham GL, Luong SB, Pham TD, Nguyen DP, Truong TKA, Nguyen QM, Tran TT, Dang TB, Ta VC, Tran QL, Le DT, Vinh LS. BM-BronchoLC - A rich bronchoscopy dataset for anatomical landmarks and lung cancer lesion recognition. Sci Data 2024; 11:321. [PMID: 38548727 PMCID: PMC10978879 DOI: 10.1038/s41597-024-03145-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2023] [Accepted: 03/14/2024] [Indexed: 04/01/2024] Open
Abstract
Flexible bronchoscopy has revolutionized respiratory disease diagnosis. It offers direct visualization and detection of airway abnormalities, including lung cancer lesions. Accurate identification of airway lesions during flexible bronchoscopy plays an important role in the lung cancer diagnosis. The application of artificial intelligence (AI) aims to support physicians in recognizing anatomical landmarks and lung cancer lesions within bronchoscopic imagery. This work described the development of BM-BronchoLC, a rich bronchoscopy dataset encompassing 106 lung cancer and 102 non-lung cancer patients. The dataset incorporates detailed localization and categorical annotations for both anatomical landmarks and lesions, meticulously conducted by senior doctors at Bach Mai Hospital, Vietnam. To assess the dataset's quality, we evaluate two prevalent AI backbone models, namely UNet++ and ESFPNet, on the image segmentation and classification tasks with single-task and multi-task learning paradigms. We present BM-BronchoLC as a reference dataset in developing AI models to assist diagnostic accuracy for anatomical landmarks and lung cancer lesions in bronchoscopy data.
Collapse
Affiliation(s)
- Van Giap Vu
- Bach Mai hospital, Hanoi, 10000, Vietnam
- Hanoi Medical University, Hanoi, 10000, Vietnam
| | - Anh Duc Hoang
- Bach Mai hospital, Hanoi, 10000, Vietnam
- Hanoi Medical University, Hanoi, 10000, Vietnam
| | - Thu Phuong Phan
- Bach Mai hospital, Hanoi, 10000, Vietnam
- Hanoi Medical University, Hanoi, 10000, Vietnam
| | - Ngoc Du Nguyen
- Bach Mai hospital, Hanoi, 10000, Vietnam
- Hanoi Medical University, Hanoi, 10000, Vietnam
| | - Thanh Thuy Nguyen
- Bach Mai hospital, Hanoi, 10000, Vietnam
- Hanoi Medical University, Hanoi, 10000, Vietnam
| | - Duc Nghia Nguyen
- Bach Mai hospital, Hanoi, 10000, Vietnam
- Hanoi Medical University, Hanoi, 10000, Vietnam
| | - Ngoc Phu Dao
- Bach Mai hospital, Hanoi, 10000, Vietnam
- Hanoi Medical University, Hanoi, 10000, Vietnam
| | | | | | | | | | | | | | | | | | - Dang Luu Vu
- Bach Mai hospital, Hanoi, 10000, Vietnam
- Hanoi Medical University, Hanoi, 10000, Vietnam
| | | | | | | | - Gia Linh Pham
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Son Ba Luong
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Trung-Dung Pham
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Duy-Phuc Nguyen
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Thi Kieu Anh Truong
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Quang Minh Nguyen
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Truong-Thuy Tran
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Tran Binh Dang
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Viet-Cuong Ta
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Quoc Long Tran
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Duc-Trong Le
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam
| | - Le Sy Vinh
- University of Engineering and Technology, Vietnam National University, Hanoi, 10000, Vietnam.
| |
Collapse
|
10
|
Chen RJ, Ding T, Lu MY, Williamson DFK, Jaume G, Song AH, Chen B, Zhang A, Shao D, Shaban M, Williams M, Oldenburg L, Weishaupt LL, Wang JJ, Vaidya A, Le LP, Gerber G, Sahai S, Williams W, Mahmood F. Towards a general-purpose foundation model for computational pathology. Nat Med 2024; 30:850-862. [PMID: 38504018 DOI: 10.1038/s41591-024-02857-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Accepted: 02/05/2024] [Indexed: 03/21/2024]
Abstract
Quantitative evaluation of tissue images is crucial for computational pathology (CPath) tasks, requiring the objective characterization of histopathological entities from whole-slide images (WSIs). The high resolution of WSIs and the variability of morphological features present significant challenges, complicating the large-scale annotation of data for high-performance applications. To address this challenge, current efforts have proposed the use of pretrained image encoders through transfer learning from natural image datasets or self-supervised learning on publicly available histopathology datasets, but have not been extensively developed and evaluated across diverse tissue types at scale. We introduce UNI, a general-purpose self-supervised model for pathology, pretrained using more than 100 million images from over 100,000 diagnostic H&E-stained WSIs (>77 TB of data) across 20 major tissue types. The model was evaluated on 34 representative CPath tasks of varying diagnostic difficulty. In addition to outperforming previous state-of-the-art models, we demonstrate new modeling capabilities in CPath such as resolution-agnostic tissue classification, slide classification using few-shot class prototypes, and disease subtyping generalization in classifying up to 108 cancer types in the OncoTree classification system. UNI advances unsupervised representation learning at scale in CPath in terms of both pretraining data and downstream evaluation, enabling data-efficient artificial intelligence models that can generalize and transfer to a wide range of diagnostically challenging tasks and clinical workflows in anatomic pathology.
Collapse
Affiliation(s)
- Richard J Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Tong Ding
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Ming Y Lu
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT), Cambridge, MA, USA
| | - Drew F K Williamson
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
| | - Guillaume Jaume
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Andrew H Song
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Bowen Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Andrew Zhang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Daniel Shao
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Muhammad Shaban
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Mane Williams
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Lukas Oldenburg
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Luca L Weishaupt
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Judy J Wang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Anurag Vaidya
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Long Phi Le
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Georg Gerber
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Sharifa Sahai
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Systems Biology, Harvard University, Cambridge, MA, USA
| | - Walt Williams
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA.
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA.
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
11
|
Fan L, Gong X, Zheng C, Li J. Data pyramid structure for optimizing EUS-based GISTs diagnosis in multi-center analysis with missing label. Comput Biol Med 2024; 169:107897. [PMID: 38171262 DOI: 10.1016/j.compbiomed.2023.107897] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2023] [Revised: 12/04/2023] [Accepted: 12/23/2023] [Indexed: 01/05/2024]
Abstract
This study introduces the Data Pyramid Structure (DPS) to address data sparsity and missing labels in medical image analysis. The DPS optimizes multi-task learning and enables sustainable expansion of multi-center data analysis. Specifically, It facilitates attribute prediction and malignant tumor diagnosis tasks by implementing a segmentation and aggregation strategy on data with absent attribute labels. To leverage multi-center data, we propose the Unified Ensemble Learning Framework (UELF) and the Unified Federated Learning Framework (UFLF), which incorporate strategies for data transfer and incremental learning in scenarios with missing labels. The proposed method was evaluated on a challenging EUS patient dataset from five centers, achieving promising diagnostic performance. The average accuracy was 0.984 with an AUC of 0.927 for multi-center analysis, surpassing state-of-the-art approaches. The interpretability of the predictions further highlights the potential clinical relevance of our method.
Collapse
Affiliation(s)
- Lin Fan
- School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, Sichuan 611756, China; Manufacturing Industry Chains Collaboration and Information Support Technology Key Laboratory of Sichuan Province, China; Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education, China; National Engineering Laboratory of Integrated Transportation Big Data Application Technology, China
| | - Xun Gong
- School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, Sichuan 611756, China; Manufacturing Industry Chains Collaboration and Information Support Technology Key Laboratory of Sichuan Province, China; Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education, China; National Engineering Laboratory of Integrated Transportation Big Data Application Technology, China.
| | - Cenyang Zheng
- School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, Sichuan 611756, China; Manufacturing Industry Chains Collaboration and Information Support Technology Key Laboratory of Sichuan Province, China; Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education, China; National Engineering Laboratory of Integrated Transportation Big Data Application Technology, China
| | - Jiao Li
- Department of Gastroenterology, The Third People's Hospital of Chendu, Affiliated Hospital of Southwest Jiaotong University, Chengdu 610031, China
| |
Collapse
|
12
|
Kataria T, Rajamani S, Ayubi AB, Bronner M, Jedrzkiewicz J, Knudsen BS, Elhabian SY. Automating Ground Truth Annotations for Gland Segmentation Through Immunohistochemistry. Mod Pathol 2023; 36:100331. [PMID: 37716506 DOI: 10.1016/j.modpat.2023.100331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 08/14/2023] [Accepted: 09/08/2023] [Indexed: 09/18/2023]
Abstract
Microscopic evaluation of glands in the colon is of utmost importance in the diagnosis of inflammatory bowel disease and cancer. When properly trained, deep learning pipelines can provide a systematic, reproducible, and quantitative assessment of disease-related changes in glandular tissue architecture. The training and testing of deep learning models require large amounts of manual annotations, which are difficult, time-consuming, and expensive to obtain. Here, we propose a method for automated generation of ground truth in digital hematoxylin and eosin (H&E)-stained slides using immunohistochemistry (IHC) labels. The image processing pipeline generates annotations of glands in H&E histopathology images from colon biopsy specimens by transfer of gland masks from KRT8/18, CDX2, or EPCAM IHC. The IHC gland outlines are transferred to coregistered H&E images for training of deep learning models. We compared the performance of the deep learning models to that of manual annotations using an internal held-out set of biopsy specimens as well as 2 public data sets. Our results show that EPCAM IHC provides gland outlines that closely match manual gland annotations (Dice = 0.89) and are resilient to damage by inflammation. In addition, we propose a simple data sampling technique that allows models trained on data from several sources to be adapted to a new data source using just a few newly annotated samples. The best performing models achieved average Dice scores of 0.902 and 0.89 on Gland Segmentation and Colorectal Adenocarcinoma Gland colon cancer public data sets, respectively, when trained with only 10% of annotated cases from either public cohort. Altogether, the performances of our models indicate that automated annotations using cell type-specific IHC markers can safely replace manual annotations. Automated IHC labels from single-institution cohorts can be combined with small numbers of hand-annotated cases from multi-institutional cohorts to train models that generalize well to diverse data sources.
Collapse
Affiliation(s)
- Tushar Kataria
- Kahlert School of Computing, University of Utah, Salt Lake City, Utah; Kahlert School of Computing, Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, Utah
| | - Saradha Rajamani
- Kahlert School of Computing, University of Utah, Salt Lake City, Utah; Kahlert School of Computing, Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, Utah
| | - Abdul Bari Ayubi
- Department of Pathology, University of Utah, Salt Lake City, Utah
| | - Mary Bronner
- Department of Pathology, University of Utah, Salt Lake City, Utah; Department of Pathology, ARUP Laboratories, Salt Lake City, Utah
| | - Jolanta Jedrzkiewicz
- Department of Pathology, University of Utah, Salt Lake City, Utah; Department of Pathology, ARUP Laboratories, Salt Lake City, Utah
| | - Beatrice S Knudsen
- Kahlert School of Computing, Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, Utah; Department of Pathology, University of Utah, Salt Lake City, Utah.
| | - Shireen Y Elhabian
- Kahlert School of Computing, University of Utah, Salt Lake City, Utah; Kahlert School of Computing, Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, Utah.
| |
Collapse
|
13
|
Zhao J, He YJ, Zhou SH, Qin J, Xie YN. CNSeg: A dataset for cervical nuclear segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 241:107732. [PMID: 37544166 DOI: 10.1016/j.cmpb.2023.107732] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/20/2022] [Revised: 05/31/2023] [Accepted: 07/23/2023] [Indexed: 08/08/2023]
Abstract
BACKGROUND AND OBJECTIVE Nuclear segmentation in cervical cell images is a crucial technique for automatic cytopathology diagnosis. Experimental evaluation of nuclear segmentation methods with datasets is helpful in promoting the advancement of nuclear segmentation techniques. However, public datasets are not enough for a reasonable and comprehensive evaluation because of insufficient quantity, single data source, and low segmentation difficulty. METHODS Therefore, we provide the largest dataset for cervical nuclear segmentation (CNSeg). It contains 124,000 annotated nuclei collected from 1,530 patients under different conditions. The image styles in this dataset cover most practical application scenarios, including microbial infection, cytopathic heterogeneity, overlapping nuclei, etc. To evaluate the performance of segmentation methods from different aspects, we divided the CNSeg dataset into three subsets, namely the patch segmentation dataset (PatchSeg) with nuclei images collected under complex conditions, the cluster segmentation dataset (ClusterSeg) with cluster nuclei, and the domain segmentation dataset (DomainSeg) with data from different domains. Furthermore, we propose a post-processing method that processes overlapping nuclei single ones. RESULTS AND CONCLUSION Experiments show that our dataset can comprehensively evaluate cervical nuclear segmentation methods from different aspects. We provide guidelines for other researchers to use the dataset. https://github.com/jingzhaohlj/AL-Net.
Collapse
Affiliation(s)
- Jing Zhao
- Northeast Forestry University, Mechanical and Electrical Engineering, Harbin 150006, China
| | - Yong-Jun He
- Harbin Institute of Technology, School of Computer Science, Harbin 150001, China.
| | - Shu-Hang Zhou
- Wenzhou Business College, Information Engineering, Wenzhou 325035, China
| | - Jian Qin
- Harbin University of Science and Technology, School of Computer Science and Technology, No. 52 Xuefu Road, 150080 Harbin, China
| | - Yi-Ning Xie
- Northeast Forestry University, Mechanical and Electrical Engineering, Harbin 150006, China
| |
Collapse
|
14
|
Graham S, Minhas F, Bilal M, Ali M, Tsang YW, Eastwood M, Wahab N, Jahanifar M, Hero E, Dodd K, Sahota H, Wu S, Lu W, Azam A, Benes K, Nimir M, Hewitt K, Bhalerao A, Robinson A, Eldaly H, Raza SEA, Gopalakrishnan K, Snead D, Rajpoot N. Screening of normal endoscopic large bowel biopsies with interpretable graph learning: a retrospective study. Gut 2023; 72:1709-1721. [PMID: 37173125 PMCID: PMC10423541 DOI: 10.1136/gutjnl-2023-329512] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/16/2023] [Accepted: 04/15/2023] [Indexed: 05/15/2023]
Abstract
OBJECTIVE To develop an interpretable artificial intelligence algorithm to rule out normal large bowel endoscopic biopsies, saving pathologist resources and helping with early diagnosis. DESIGN A graph neural network was developed incorporating pathologist domain knowledge to classify 6591 whole-slides images (WSIs) of endoscopic large bowel biopsies from 3291 patients (approximately 54% female, 46% male) as normal or abnormal (non-neoplastic and neoplastic) using clinically driven interpretable features. One UK National Health Service (NHS) site was used for model training and internal validation. External validation was conducted on data from two other NHS sites and one Portuguese site. RESULTS Model training and internal validation were performed on 5054 WSIs of 2080 patients resulting in an area under the curve-receiver operating characteristic (AUC-ROC) of 0.98 (SD=0.004) and AUC-precision-recall (PR) of 0.98 (SD=0.003). The performance of the model, named Interpretable Gland-Graphs using a Neural Aggregator (IGUANA), was consistent in testing over 1537 WSIs of 1211 patients from three independent external datasets with mean AUC-ROC=0.97 (SD=0.007) and AUC-PR=0.97 (SD=0.005). At a high sensitivity threshold of 99%, the proposed model can reduce the number of normal slides to be reviewed by a pathologist by approximately 55%. IGUANA also provides an explainable output highlighting potential abnormalities in a WSI in the form of a heatmap as well as numerical values associating the model prediction with various histological features. CONCLUSION The model achieved consistently high accuracy showing its potential in optimising increasingly scarce pathologist resources. Explainable predictions can guide pathologists in their diagnostic decision-making and help boost their confidence in the algorithm, paving the way for its future clinical adoption.
Collapse
Affiliation(s)
- Simon Graham
- Department of Computer Science, University of Warwick, Coventry, UK
- Histofy Ltd, Birmingham, UK
| | - Fayyaz Minhas
- Department of Computer Science, University of Warwick, Coventry, UK
| | - Mohsin Bilal
- Department of Computer Science, University of Warwick, Coventry, UK
| | - Mahmoud Ali
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - Yee Wah Tsang
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - Mark Eastwood
- Department of Computer Science, University of Warwick, Coventry, UK
| | - Noorul Wahab
- Department of Computer Science, University of Warwick, Coventry, UK
| | | | - Emily Hero
- Department of Pathology, University Hospitals of Leicester NHS Trust, Leicester, UK
| | - Katherine Dodd
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - Harvir Sahota
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - Shaobin Wu
- Department of Pathology, East Suffolk and North Essex NHS Foundation Trust, Colchester, UK
| | - Wenqi Lu
- Department of Computer Science, University of Warwick, Coventry, UK
| | - Ayesha Azam
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - Ksenija Benes
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
- Department of Pathology, Royal Wolverhampton Hospitals NHS Trust, Wolverhampton, UK
| | - Mohammed Nimir
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - Katherine Hewitt
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - Abhir Bhalerao
- Department of Computer Science, University of Warwick, Coventry, UK
| | - Andrew Robinson
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - Hesham Eldaly
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | | | - Kishore Gopalakrishnan
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - David Snead
- Histofy Ltd, Birmingham, UK
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
- Division of Biomedical Sciences, University of Warwick Warwick Medical School, Coventry, UK
| | - Nasir Rajpoot
- Department of Computer Science, University of Warwick, Coventry, UK
- Histofy Ltd, Birmingham, UK
- Department of Pathology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| |
Collapse
|
15
|
Cooper M, Ji Z, Krishnan RG. Machine learning in computational histopathology: Challenges and opportunities. Genes Chromosomes Cancer 2023; 62:540-556. [PMID: 37314068 DOI: 10.1002/gcc.23177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 05/18/2023] [Accepted: 05/20/2023] [Indexed: 06/15/2023] Open
Abstract
Digital histopathological images, high-resolution images of stained tissue samples, are a vital tool for clinicians to diagnose and stage cancers. The visual analysis of patient state based on these images are an important part of oncology workflow. Although pathology workflows have historically been conducted in laboratories under a microscope, the increasing digitization of histopathological images has led to their analysis on computers in the clinic. The last decade has seen the emergence of machine learning, and deep learning in particular, a powerful set of tools for the analysis of histopathological images. Machine learning models trained on large datasets of digitized histopathology slides have resulted in automated models for prediction and stratification of patient risk. In this review, we provide context for the rise of such models in computational histopathology, highlight the clinical tasks they have found success in automating, discuss the various machine learning techniques that have been applied to this domain, and underscore open problems and opportunities.
Collapse
Affiliation(s)
- Michael Cooper
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- University Health Network, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
| | - Zongliang Ji
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
| | - Rahul G Krishnan
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
- Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
16
|
Asif A, Rajpoot K, Graham S, Snead D, Minhas F, Rajpoot N. Unleashing the potential of AI for pathology: challenges and recommendations. J Pathol 2023; 260:564-577. [PMID: 37550878 PMCID: PMC10952719 DOI: 10.1002/path.6168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2023] [Revised: 06/21/2023] [Accepted: 06/22/2023] [Indexed: 08/09/2023]
Abstract
Computational pathology is currently witnessing a surge in the development of AI techniques, offering promise for achieving breakthroughs and significantly impacting the practices of pathology and oncology. These AI methods bring with them the potential to revolutionize diagnostic pipelines as well as treatment planning and overall patient care. Numerous peer-reviewed studies reporting remarkable performance across diverse tasks serve as a testimony to the potential of AI in the field. However, widespread adoption of these methods in clinical and pre-clinical settings still remains a challenge. In this review article, we present a detailed analysis of the major obstacles encountered during the development of effective models and their deployment in practice. We aim to provide readers with an overview of the latest developments, assist them with insights into identifying some specific challenges that may require resolution, and suggest recommendations and potential future research directions. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.
Collapse
Affiliation(s)
- Amina Asif
- Tissue Image Analytics Centre, Department of Computer ScienceUniversity of WarwickCoventryUK
| | - Kashif Rajpoot
- School of Computer ScienceUniversity of BirminghamBirminghamUK
| | - Simon Graham
- Histofy Ltd, Birmingham Business ParkBirminghamUK
| | - David Snead
- Histofy Ltd, Birmingham Business ParkBirminghamUK
- Department of PathologyUniversity Hospitals Coventry & Warwickshire NHS TrustCoventryUK
| | - Fayyaz Minhas
- Tissue Image Analytics Centre, Department of Computer ScienceUniversity of WarwickCoventryUK
- Cancer Research CentreUniversity of WarwickCoventryUK
| | - Nasir Rajpoot
- Tissue Image Analytics Centre, Department of Computer ScienceUniversity of WarwickCoventryUK
- Histofy Ltd, Birmingham Business ParkBirminghamUK
- Cancer Research CentreUniversity of WarwickCoventryUK
- The Alan Turing InstituteLondonUK
| |
Collapse
|
17
|
Ester O, Hörst F, Seibold C, Keyl J, Ting S, Vasileiadis N, Schmitz J, Ivanyi P, Grünwald V, Bräsen JH, Egger J, Kleesiek J. Valuing vicinity: Memory attention framework for context-based semantic segmentation in histopathology. Comput Med Imaging Graph 2023; 107:102238. [PMID: 37207396 DOI: 10.1016/j.compmedimag.2023.102238] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Revised: 04/11/2023] [Accepted: 04/25/2023] [Indexed: 05/21/2023]
Abstract
The segmentation of histopathological whole slide images into tumourous and non-tumourous types of tissue is a challenging task that requires the consideration of both local and global spatial contexts to classify tumourous regions precisely. The identification of subtypes of tumour tissue complicates the issue as the sharpness of separation decreases and the pathologist's reasoning is even more guided by spatial context. However, the identification of detailed tissue types is crucial for providing personalized cancer therapies. Due to the high resolution of whole slide images, existing semantic segmentation methods, restricted to isolated image sections, are incapable of processing context information beyond. To take a step towards better context comprehension, we propose a patch neighbour attention mechanism to query the neighbouring tissue context from a patch embedding memory bank and infuse context embeddings into bottleneck hidden feature maps. Our memory attention framework (MAF) mimics a pathologist's annotation procedure - zooming out and considering surrounding tissue context. The framework can be integrated into any encoder-decoder segmentation method. We evaluate the MAF on two public breast cancer and liver cancer data sets and an internal kidney cancer data set using famous segmentation models (U-Net, DeeplabV3) and demonstrate the superiority over other context-integrating algorithms - achieving a substantial improvement of up to 17% on Dice score. The code is publicly available at https://github.com/tio-ikim/valuing-vicinity.
Collapse
Affiliation(s)
- Oliver Ester
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Essen, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), Essen, Germany
| | - Fabian Hörst
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Essen, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), Essen, Germany.
| | - Constantin Seibold
- Institute of Anthropomatics and Robotics, Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
| | - Julius Keyl
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Essen, Germany; Institute of Pathology, University Hospital Essen (AöR), University of Duisburg-Essen, Essen, Germany
| | - Saskia Ting
- Institute of Pathology, University Hospital Essen (AöR), University of Duisburg-Essen, Essen, Germany; Institute of Pathology Nordhessen, Kassel, Germany
| | - Nikolaos Vasileiadis
- Nephropathology Unit, Institute for Pathology, Hannover Medical School, Hannover, Germany
| | - Jessica Schmitz
- Nephropathology Unit, Institute for Pathology, Hannover Medical School, Hannover, Germany
| | - Philipp Ivanyi
- Department of Hematology, Hemostasis, Oncology and Stem Cell Transplantation, Hannover Medical School, Hannover, Germany
| | - Viktor Grünwald
- Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), Essen, Germany; Clinic for Medical Oncology, Clinic for Urology, West German Cancer Center, University Hospital Essen (AöR), Essen, Germany
| | - Jan Hinrich Bräsen
- Nephropathology Unit, Institute for Pathology, Hannover Medical School, Hannover, Germany
| | - Jan Egger
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Essen, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), Essen, Germany
| | - Jens Kleesiek
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Essen, Germany; Cancer Research Center Cologne Essen (CCCE), West German Cancer Center Essen, University Hospital Essen (AöR), Essen, Germany; German Cancer Consortium (DKTK), Partner Site Essen, Germany
| |
Collapse
|
18
|
Zhao Y, Wang X, Che T, Bao G, Li S. Multi-task deep learning for medical image computing and analysis: A review. Comput Biol Med 2023; 153:106496. [PMID: 36634599 DOI: 10.1016/j.compbiomed.2022.106496] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Revised: 12/06/2022] [Accepted: 12/27/2022] [Indexed: 12/29/2022]
Abstract
The renaissance of deep learning has provided promising solutions to various tasks. While conventional deep learning models are constructed for a single specific task, multi-task deep learning (MTDL) that is capable to simultaneously accomplish at least two tasks has attracted research attention. MTDL is a joint learning paradigm that harnesses the inherent correlation of multiple related tasks to achieve reciprocal benefits in improving performance, enhancing generalizability, and reducing the overall computational cost. This review focuses on the advanced applications of MTDL for medical image computing and analysis. We first summarize four popular MTDL network architectures (i.e., cascaded, parallel, interacted, and hybrid). Then, we review the representative MTDL-based networks for eight application areas, including the brain, eye, chest, cardiac, abdomen, musculoskeletal, pathology, and other human body regions. While MTDL-based medical image processing has been flourishing and demonstrating outstanding performance in many tasks, in the meanwhile, there are performance gaps in some tasks, and accordingly we perceive the open challenges and the perspective trends. For instance, in the 2018 Ischemic Stroke Lesion Segmentation challenge, the reported top dice score of 0.51 and top recall of 0.55 achieved by the cascaded MTDL model indicate further research efforts in high demand to escalate the performance of current models.
Collapse
Affiliation(s)
- Yan Zhao
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100083, China
| | - Xiuying Wang
- School of Computer Science, The University of Sydney, Sydney, NSW, 2008, Australia.
| | - Tongtong Che
- Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, Beijing, 100083, China
| | - Guoqing Bao
- School of Computer Science, The University of Sydney, Sydney, NSW, 2008, Australia
| | - Shuyu Li
- State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, 100875, China.
| |
Collapse
|