1
|
Hosseini MS, Bejnordi BE, Trinh VQH, Chan L, Hasan D, Li X, Yang S, Kim T, Zhang H, Wu T, Chinniah K, Maghsoudlou S, Zhang R, Zhu J, Khaki S, Buin A, Chaji F, Salehi A, Nguyen BN, Samaras D, Plataniotis KN. Computational pathology: A survey review and the way forward. J Pathol Inform 2024; 15:100357. [PMID: 38420608 PMCID: PMC10900832 DOI: 10.1016/j.jpi.2023.100357] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 12/21/2023] [Accepted: 12/23/2023] [Indexed: 03/02/2024] Open
Abstract
Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.
Collapse
Affiliation(s)
- Mahdi S Hosseini
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | | | - Vincent Quoc-Huy Trinh
- Institute for Research in Immunology and Cancer of the University of Montreal, Montreal, QC H3T 1J4, Canada
| | - Lyndon Chan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Danial Hasan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Xingwen Li
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Stephen Yang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Taehyo Kim
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Haochen Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Theodore Wu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Kajanan Chinniah
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Sina Maghsoudlou
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ryan Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Jiadai Zhu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Samir Khaki
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Andrei Buin
- Huron Digitial Pathology, St. Jacobs, ON N0B 2N0, Canada
| | - Fatemeh Chaji
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ala Salehi
- Department of Electrical and Computer Engineering, University of New Brunswick, Fredericton, NB E3B 5A3, Canada
| | - Bich Ngoc Nguyen
- University of Montreal Hospital Center, Montreal, QC H2X 0C2, Canada
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, NY 11794, United States
| | - Konstantinos N Plataniotis
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| |
Collapse
|
2
|
Thandiackal K, Piccinelli L, Gupta R, Pati P, Goksel O. Multi-Scale Feature Alignment for Continual Learning of Unlabeled Domains. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2599-2609. [PMID: 38381642 DOI: 10.1109/tmi.2024.3368365] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/23/2024]
Abstract
Methods for unsupervised domain adaptation (UDA) help to improve the performance of deep neural networks on unseen domains without any labeled data. Especially in medical disciplines such as histopathology, this is crucial since large datasets with detailed annotations are scarce. While the majority of existing UDA methods focus on the adaptation from a labeled source to a single unlabeled target domain, many real-world applications with a long life cycle involve more than one target domain. Thus, the ability to sequentially adapt to multiple target domains becomes essential. In settings where the data from previously seen domains cannot be stored, e.g., due to data protection regulations, the above becomes a challenging continual learning problem. To this end, we propose to use generative feature-driven image replay in conjunction with a dual-purpose discriminator that not only enables the generation of images with realistic features for replay, but also promotes feature alignment during domain adaptation. We evaluate our approach extensively on a sequence of three histopathological datasets for tissue-type classification, achieving state-of-the-art results. We present detailed ablation experiments studying our proposed method components and demonstrate a possible use-case of our continual UDA method for an unsupervised patch-based segmentation task given high-resolution tissue images. Our code is available at: https://github.com/histocartography/multi-scale-feature-alignment.
Collapse
|
3
|
Vray G, Tomar D, Bozorgtabar B, Thiran JP. Distill-SODA: Distilling Self-Supervised Vision Transformer for Source-Free Open-Set Domain Adaptation in Computational Pathology. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2021-2032. [PMID: 38236667 DOI: 10.1109/tmi.2024.3355645] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/03/2024]
Abstract
Developing computational pathology models is essential for reducing manual tissue typing from whole slide images, transferring knowledge from the source domain to an unlabeled, shifted target domain, and identifying unseen categories. We propose a practical setting by addressing the above-mentioned challenges in one fell swoop, i.e., source-free open-set domain adaptation. Our methodology focuses on adapting a pre-trained source model to an unlabeled target dataset and encompasses both closed-set and open-set classes. Beyond addressing the semantic shift of unknown classes, our framework also deals with a covariate shift, which manifests as variations in color appearance between source and target tissue samples. Our method hinges on distilling knowledge from a self-supervised vision transformer (ViT), drawing guidance from either robustly pre-trained transformer models or histopathology datasets, including those from the target domain. In pursuit of this, we introduce a novel style-based adversarial data augmentation, serving as hard positives for self-training a ViT, resulting in highly contextualized embeddings. Following this, we cluster semantically akin target images, with the source model offering weak pseudo-labels, albeit with uncertain confidence. To enhance this process, we present the closed-set affinity score (CSAS), aiming to correct the confidence levels of these pseudo-labels and to calculate weighted class prototypes within the contextualized embedding space. Our approach establishes itself as state-of-the-art across three public histopathological datasets for colorectal cancer assessment. Notably, our self-training method seamlessly integrates with open-set detection methods, resulting in enhanced performance in both closed-set and open-set recognition tasks.
Collapse
|
4
|
Sharkas M, Attallah O. Color-CADx: a deep learning approach for colorectal cancer classification through triple convolutional neural networks and discrete cosine transform. Sci Rep 2024; 14:6914. [PMID: 38519513 PMCID: PMC10959971 DOI: 10.1038/s41598-024-56820-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2022] [Accepted: 03/11/2024] [Indexed: 03/25/2024] Open
Abstract
Colorectal cancer (CRC) exhibits a significant death rate that consistently impacts human lives worldwide. Histopathological examination is the standard method for CRC diagnosis. However, it is complicated, time-consuming, and subjective. Computer-aided diagnostic (CAD) systems using digital pathology can help pathologists diagnose CRC faster and more accurately than manual histopathology examinations. Deep learning algorithms especially convolutional neural networks (CNNs) are advocated for diagnosis of CRC. Nevertheless, most previous CAD systems obtained features from one CNN, these features are of huge dimension. Also, they relied on spatial information only to achieve classification. In this paper, a CAD system is proposed called "Color-CADx" for CRC recognition. Different CNNs namely ResNet50, DenseNet201, and AlexNet are used for end-to-end classification at different training-testing ratios. Moreover, features are extracted from these CNNs and reduced using discrete cosine transform (DCT). DCT is also utilized to acquire spectral representation. Afterward, it is used to further select a reduced set of deep features. Furthermore, DCT coefficients obtained in the previous step are concatenated and the analysis of variance (ANOVA) feature selection approach is applied to choose significant features. Finally, machine learning classifiers are employed for CRC classification. Two publicly available datasets were investigated which are the NCT-CRC-HE-100 K dataset and the Kather_texture_2016_image_tiles dataset. The highest achieved accuracy reached 99.3% for the NCT-CRC-HE-100 K dataset and 96.8% for the Kather_texture_2016_image_tiles dataset. DCT and ANOVA have successfully lowered feature dimensionality thus reducing complexity. Color-CADx has demonstrated efficacy in terms of accuracy, as its performance surpasses that of the most recent advancements.
Collapse
Affiliation(s)
- Maha Sharkas
- Electronics and Communications Engineering Department, College of Engineering and Technology, Arab Academy for Science, Technology, and Maritime Transport, Alexandria, Egypt
| | - Omneya Attallah
- Electronics and Communications Engineering Department, College of Engineering and Technology, Arab Academy for Science, Technology, and Maritime Transport, Alexandria, Egypt.
- Wearables, Biosensing, and Biosignal Processing Laboratory, Arab Academy for Science, Technology and Maritime Transport, Alexandria, 21937, Egypt.
| |
Collapse
|
5
|
Yan S, Li Y, Pan L, Jiang H, Gong L, Jin F. The application of artificial intelligence for Rapid On-Site Evaluation during flexible bronchoscopy. Front Oncol 2024; 14:1360831. [PMID: 38529376 PMCID: PMC10961380 DOI: 10.3389/fonc.2024.1360831] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/24/2023] [Accepted: 02/23/2024] [Indexed: 03/27/2024] Open
Abstract
Background Rapid On-Site Evaluation (ROSE) during flexible bronchoscopy (FB) can improve the adequacy of biopsy specimens and diagnostic yield of lung cancer. However, the lack of cytopathologists has restricted the wide use of ROSE. Objective To develop a ROSE artificial intelligence (AI) system using deep learning techniques to differentiate malignant from benign lesions based on ROSE cytological images, and evaluate the clinical performance of the ROSE AI system. Method 6357 ROSE cytological images from 721 patients who underwent transbronchial biopsy were collected from January to July 2023 at the Tangdu Hospital, Air Force Medical University. A ROSE AI system, composed of a deep convolutional neural network (DCNN), was developed to identify whether there were malignant cells in the ROSE cytological images. Internal testing, external testing, and human-machine competition were used to evaluate the performance of the system. Results The ROSE AI system identified images containing lung malignant cells with the accuracy of 92.97% and 90.26% on the internal testing dataset and external testing dataset respectively, and its performance was comparable to that of the experienced cytopathologist. The ROSE AI system also showed promising performance in diagnosing lung cancer based on ROSE cytological images, with accuracy of 89.61% and 87.59%, and sensitivity of 90.57% and 94.90% on the internal testing dataset and external testing dataset respectively. More specifically, the agreement between the ROSE AI system and the experienced cytopathologist in diagnosing common types of lung cancer, including squamous cell carcinoma, adenocarcinoma, and small cell lung cancer, demonstrated almost perfect consistency in both the internal testing dataset (κ = 0.930 ) and the external testing dataset (κ = 0.932 ). Conclusions The ROSE AI system demonstrated feasibility and robustness in identifying specimen adequacy, showing potential enhancement in the diagnostic yield of FB. Nevertheless, additional enhancements, incorporating a more diverse range of training data and leveraging advanced AI models with increased capabilities, along with rigorous validation through extensive multi-center randomized control assays, are crucial to guarantee the seamless and effective integration of this technology into clinical practice.
Collapse
Affiliation(s)
- Shuang Yan
- Department of Pulmonary and Critical Care Medicine, Tangdu Hospital, Air Force Medical University, Xi’an, China
| | | | - Lei Pan
- Department of Pulmonary and Critical Care Medicine, Tangdu Hospital, Air Force Medical University, Xi’an, China
| | - Hua Jiang
- Department of Pulmonary and Critical Care Medicine, Tangdu Hospital, Air Force Medical University, Xi’an, China
| | - Li Gong
- Department of Pathology, Tangdu Hospital, Air Force Medical University, Xi’an, China
| | - Faguang Jin
- Department of Pulmonary and Critical Care Medicine, Tangdu Hospital, Air Force Medical University, Xi’an, China
| |
Collapse
|
6
|
Kumari S, Singh P. Deep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives. Comput Biol Med 2024; 170:107912. [PMID: 38219643 DOI: 10.1016/j.compbiomed.2023.107912] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 11/02/2023] [Accepted: 12/24/2023] [Indexed: 01/16/2024]
Abstract
Deep learning has demonstrated remarkable performance across various tasks in medical imaging. However, these approaches primarily focus on supervised learning, assuming that the training and testing data are drawn from the same distribution. Unfortunately, this assumption may not always hold true in practice. To address these issues, unsupervised domain adaptation (UDA) techniques have been developed to transfer knowledge from a labeled domain to a related but unlabeled domain. In recent years, significant advancements have been made in UDA, resulting in a wide range of methodologies, including feature alignment, image translation, self-supervision, and disentangled representation methods, among others. In this paper, we provide a comprehensive literature review of recent deep UDA approaches in medical imaging from a technical perspective. Specifically, we categorize current UDA research in medical imaging into six groups and further divide them into finer subcategories based on the different tasks they perform. We also discuss the respective datasets used in the studies to assess the divergence between the different domains. Finally, we discuss emerging areas and provide insights and discussions on future research directions to conclude this survey.
Collapse
Affiliation(s)
- Suruchi Kumari
- Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, India.
| | - Pravendra Singh
- Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, India.
| |
Collapse
|
7
|
Ye J, Kalra S, Miri MS. Cluster-based histopathology phenotype representation learning by self-supervised multi-class-token hierarchical ViT. Sci Rep 2024; 14:3202. [PMID: 38331955 PMCID: PMC10853503 DOI: 10.1038/s41598-024-53361-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2023] [Accepted: 01/31/2024] [Indexed: 02/10/2024] Open
Abstract
Developing a clinical AI model necessitates a significant amount of highly curated and carefully annotated dataset by multiple medical experts, which results in increased development time and costs. Self-supervised learning (SSL) is a method that enables AI models to leverage unlabelled data to acquire domain-specific background knowledge that can enhance their performance on various downstream tasks. In this work, we introduce CypherViT, a cluster-based histo-pathology phenotype representation learning by self-supervised multi-class-token hierarchical Vision Transformer (ViT). CypherViT is a novel backbone that can be integrated into a SSL pipeline, accommodating both coarse and fine-grained feature learning for histopathological images via a hierarchical feature agglomerative attention module with multiple classification (cls) tokens in ViT. Our qualitative analysis showcases that our approach successfully learns semantically meaningful regions of interest that align with morphological phenotypes. To validate the model, we utilize the DINO self-supervised learning (SSL) framework to train CypherViT on a substantial dataset of unlabeled breast cancer histopathological images. This trained model proves to be a generalizable and robust feature extractor for colorectal cancer images. Notably, our model demonstrates promising performance in patch-level tissue phenotyping tasks across four public datasets. The results from our quantitative experiments highlight significant advantages over existing state-of-the-art SSL models and traditional transfer learning methods, such as those relying on ImageNet pre-training.
Collapse
Affiliation(s)
- Jiarong Ye
- Roche Diagnostics Solutions, Santa Clara, CA, USA
| | - Shivam Kalra
- Roche Diagnostics Solutions, Santa Clara, CA, USA.
| | | |
Collapse
|
8
|
Fisher TB, Saini G, Rekha TS, Krishnamurthy J, Bhattarai S, Callagy G, Webber M, Janssen EAM, Kong J, Aneja R. Digital image analysis and machine learning-assisted prediction of neoadjuvant chemotherapy response in triple-negative breast cancer. Breast Cancer Res 2024; 26:12. [PMID: 38238771 PMCID: PMC10797728 DOI: 10.1186/s13058-023-01752-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2023] [Accepted: 12/11/2023] [Indexed: 01/22/2024] Open
Abstract
BACKGROUND Pathological complete response (pCR) is associated with favorable prognosis in patients with triple-negative breast cancer (TNBC). However, only 30-40% of TNBC patients treated with neoadjuvant chemotherapy (NAC) show pCR, while the remaining 60-70% show residual disease (RD). The role of the tumor microenvironment in NAC response in patients with TNBC remains unclear. In this study, we developed a machine learning-based two-step pipeline to distinguish between various histological components in hematoxylin and eosin (H&E)-stained whole slide images (WSIs) of TNBC tissue biopsies and to identify histological features that can predict NAC response. METHODS H&E-stained WSIs of treatment-naïve biopsies from 85 patients (51 with pCR and 34 with RD) of the model development cohort and 79 patients (41 with pCR and 38 with RD) of the validation cohort were separated through a stratified eightfold cross-validation strategy for the first step and leave-one-out cross-validation strategy for the second step. A tile-level histology label prediction pipeline and four machine-learning classifiers were used to analyze 468,043 tiles of WSIs. The best-trained classifier used 55 texture features from each tile to produce a probability profile during testing. The predicted histology classes were used to generate a histology classification map of the spatial distributions of different tissue regions. A patient-level NAC response prediction pipeline was trained with features derived from paired histology classification maps. The top graph-based features capturing the relevant spatial information across the different histological classes were provided to the radial basis function kernel support vector machine (rbfSVM) classifier for NAC treatment response prediction. RESULTS The tile-level prediction pipeline achieved 86.72% accuracy for histology class classification, while the patient-level pipeline achieved 83.53% NAC response (pCR vs. RD) prediction accuracy of the model development cohort. The model was validated with an independent cohort with tile histology validation accuracy of 83.59% and NAC prediction accuracy of 81.01%. The histological class pairs with the strongest NAC response predictive ability were tumor and tumor tumor-infiltrating lymphocytes for pCR and microvessel density and polyploid giant cancer cells for RD. CONCLUSION Our machine learning pipeline can robustly identify clinically relevant histological classes that predict NAC response in TNBC patients and may help guide patient selection for NAC treatment.
Collapse
Affiliation(s)
- Timothy B Fisher
- Department of Biology, Georgia State University, Atlanta, GA, 30302, USA
| | - Geetanjali Saini
- School of Health Professions, University of Alabama at Birmingham, Birmingham, AL, 35294, USA
| | - T S Rekha
- JSSAHER (JSS Academy of Higher Education and Research) Medical College, Mysuru, Karnataka, India
| | - Jayashree Krishnamurthy
- JSSAHER (JSS Academy of Higher Education and Research) Medical College, Mysuru, Karnataka, India
| | - Shristi Bhattarai
- School of Health Professions, University of Alabama at Birmingham, Birmingham, AL, 35294, USA
| | - Grace Callagy
- Discipline of Pathology, University of Galway, Galway, Ireland
| | - Mark Webber
- Discipline of Pathology, University of Galway, Galway, Ireland
| | - Emiel A M Janssen
- Department of Pathology, Stavanger University Hospital, Stavanger, Norway
- Department of Chemistry, Bioscience and Environmental Engineering, University of Stavanger, Stavanger, Norway
| | - Jun Kong
- Department of Mathematics and Statistics, Georgia State University, Atlanta, GA, 30303, USA.
| | - Ritu Aneja
- Department of Biology, Georgia State University, Atlanta, GA, 30302, USA.
- School of Health Professions, University of Alabama at Birmingham, Birmingham, AL, 35294, USA.
| |
Collapse
|
9
|
Doğan RS, Yılmaz B. Histopathology image classification: highlighting the gap between manual analysis and AI automation. Front Oncol 2024; 13:1325271. [PMID: 38298445 PMCID: PMC10827850 DOI: 10.3389/fonc.2023.1325271] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Accepted: 12/19/2023] [Indexed: 02/02/2024] Open
Abstract
The field of histopathological image analysis has evolved significantly with the advent of digital pathology, leading to the development of automated models capable of classifying tissues and structures within diverse pathological images. Artificial intelligence algorithms, such as convolutional neural networks, have shown remarkable capabilities in pathology image analysis tasks, including tumor identification, metastasis detection, and patient prognosis assessment. However, traditional manual analysis methods have generally shown low accuracy in diagnosing colorectal cancer using histopathological images. This study investigates the use of AI in image classification and image analytics using histopathological images using the histogram of oriented gradients method. The study develops an AI-based architecture for image classification using histopathological images, aiming to achieve high performance with less complexity through specific parameters and layers. In this study, we investigate the complicated state of histopathological image classification, explicitly focusing on categorizing nine distinct tissue types. Our research used open-source multi-centered image datasets that included records of 100.000 non-overlapping images from 86 patients for training and 7180 non-overlapping images from 50 patients for testing. The study compares two distinct approaches, training artificial intelligence-based algorithms and manual machine learning models, to automate tissue classification. This research comprises two primary classification tasks: binary classification, distinguishing between normal and tumor tissues, and multi-classification, encompassing nine tissue types, including adipose, background, debris, stroma, lymphocytes, mucus, smooth muscle, normal colon mucosa, and tumor. Our findings show that artificial intelligence-based systems can achieve 0.91 and 0.97 accuracy in binary and multi-class classifications. In comparison, the histogram of directed gradient features and the Random Forest classifier achieved accuracy rates of 0.75 and 0.44 in binary and multi-class classifications, respectively. Our artificial intelligence-based methods are generalizable, allowing them to be integrated into histopathology diagnostics procedures and improve diagnostic accuracy and efficiency. The CNN model outperforms existing machine learning techniques, demonstrating its potential to improve the precision and effectiveness of histopathology image analysis. This research emphasizes the importance of maintaining data consistency and applying normalization methods during the data preparation stage for analysis. It particularly highlights the potential of artificial intelligence to assess histopathological images.
Collapse
Affiliation(s)
- Refika Sultan Doğan
- Department of Bioengineering, Abdullah Gül University, Kayseri, Türkiye
- Biomedical Instrumentation and Signal Analysis Laboratory, Abdullah Gül University, Kayseri, Türkiye
| | - Bülent Yılmaz
- Biomedical Instrumentation and Signal Analysis Laboratory, Abdullah Gül University, Kayseri, Türkiye
- Department of Electrical and Computer Engineering, Abdullah Gul University, Kayseri, Türkiye
- Department of Electrical Engineering, Gulf University for Science and Technology, Mishref, Kuwait
| |
Collapse
|
10
|
Kim S, Rakib Hasan K, Ando Y, Ko S, Lee D, Park NJY, Cho J. Improving Tumor-Infiltrating Lymphocytes Score Prediction in Breast Cancer with Self-Supervised Learning. Life (Basel) 2024; 14:90. [PMID: 38255705 PMCID: PMC11154396 DOI: 10.3390/life14010090] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Revised: 01/02/2024] [Accepted: 01/03/2024] [Indexed: 01/24/2024] Open
Abstract
Tumor microenvironment (TME) plays a pivotal role in immuno-oncology, which investigates the intricate interactions between tumors and the human immune system. Specifically, tumor-infiltrating lymphocytes (TILs) are crucial biomarkers for evaluating the prognosis of breast cancer patients and have the potential to refine immunotherapy precision and accurately identify tumor cells in specific cancer types. In this study, we conducted tissue segmentation and lymphocyte detection tasks to predict TIL scores by employing self-supervised learning (SSL) model-based approaches capable of addressing limited labeling data issues. Our experiments showed a 1.9% improvement in tissue segmentation and a 2% improvement in lymphocyte detection over the ImageNet pre-training model. Using these SSL-based models, we achieved a TIL score of 0.718 with a 4.4% improvement. In particular, when trained with only 10% of the entire dataset, the SwAV pre-trained model exhibited a superior performance over other models. Our work highlights improved tissue segmentation and lymphocyte detection using the SSL model with less labeled data for TIL score prediction.
Collapse
Affiliation(s)
- Sijin Kim
- Department of Biomedical Science, Kyungpook National University, Daegu 41566, Republic of Korea; (S.K.); (K.R.H.); (Y.A.); (S.K.); (D.L.)
| | - Kazi Rakib Hasan
- Department of Biomedical Science, Kyungpook National University, Daegu 41566, Republic of Korea; (S.K.); (K.R.H.); (Y.A.); (S.K.); (D.L.)
| | - Yu Ando
- Department of Biomedical Science, Kyungpook National University, Daegu 41566, Republic of Korea; (S.K.); (K.R.H.); (Y.A.); (S.K.); (D.L.)
| | - Seokhwan Ko
- Department of Biomedical Science, Kyungpook National University, Daegu 41566, Republic of Korea; (S.K.); (K.R.H.); (Y.A.); (S.K.); (D.L.)
| | - Donghyeon Lee
- Department of Biomedical Science, Kyungpook National University, Daegu 41566, Republic of Korea; (S.K.); (K.R.H.); (Y.A.); (S.K.); (D.L.)
| | - Nora Jee-Young Park
- Department of Pathology, School of Medicine, Kyungpook National University, Daegu 41944, Republic of Korea;
- Department of Pathology, Kyungpook National University Chilgok Hospital, Daegu 41404, Republic of Korea
| | - Junghwan Cho
- Clinical Omics Institute, Kyungpook National University, Daegu 41405, Republic of Korea
| |
Collapse
|
11
|
Sauter D, Lodde G, Nensa F, Schadendorf D, Livingstone E, Kukuk M. A Systematic Comparison of Task Adaptation Techniques for Digital Histopathology. Bioengineering (Basel) 2023; 11:19. [PMID: 38247897 PMCID: PMC10813343 DOI: 10.3390/bioengineering11010019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Revised: 12/20/2023] [Accepted: 12/21/2023] [Indexed: 01/23/2024] Open
Abstract
Due to an insufficient amount of image annotation, artificial intelligence in computational histopathology usually relies on fine-tuning pre-trained neural networks. While vanilla fine-tuning has shown to be effective, research on computer vision has recently proposed improved algorithms, promising better accuracy. While initial studies have demonstrated the benefits of these algorithms for medical AI, in particular for radiology, there is no empirical evidence for improved accuracy in histopathology. Therefore, based on the ConvNeXt architecture, our study performs a systematic comparison of nine task adaptation techniques, namely, DELTA, L2-SP, MARS-PGM, Bi-Tuning, BSS, MultiTune, SpotTune, Co-Tuning, and vanilla fine-tuning, on five histopathological classification tasks using eight datasets. The results are based on external testing and statistical validation and reveal a multifaceted picture: some techniques are better suited for histopathology than others, but depending on the classification task, a significant relative improvement in accuracy was observed for five advanced task adaptation techniques over the control method, i.e., vanilla fine-tuning (e.g., Co-Tuning: P(≫) = 0.942, d = 2.623). Furthermore, we studied the classification accuracy for three of the nine methods with respect to the training set size (e.g., Co-Tuning: P(≫) = 0.951, γ = 0.748). Overall, our results show that the performance of advanced task adaptation techniques in histopathology is affected by influencing factors such as the specific classification task or the size of the training dataset.
Collapse
Affiliation(s)
- Daniel Sauter
- Department of Computer Science, Fachhochschule Dortmund, 44227 Dortmund, Germany;
| | - Georg Lodde
- Department of Dermatology, University Hospital Essen, 45147 Essen, Germany; (G.L.); (D.S.); (E.L.)
| | - Felix Nensa
- Institute for AI in Medicine (IKIM), University Hospital Essen, 45131 Essen, Germany;
- Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, 45147 Essen, Germany
| | - Dirk Schadendorf
- Department of Dermatology, University Hospital Essen, 45147 Essen, Germany; (G.L.); (D.S.); (E.L.)
| | - Elisabeth Livingstone
- Department of Dermatology, University Hospital Essen, 45147 Essen, Germany; (G.L.); (D.S.); (E.L.)
| | - Markus Kukuk
- Department of Computer Science, Fachhochschule Dortmund, 44227 Dortmund, Germany;
| |
Collapse
|
12
|
Jabbour S, Fouhey D, Shepard S, Valley TS, Kazerooni EA, Banovic N, Wiens J, Sjoding MW. Measuring the Impact of AI in the Diagnosis of Hospitalized Patients: A Randomized Clinical Vignette Survey Study. JAMA 2023; 330:2275-2284. [PMID: 38112814 PMCID: PMC10731487 DOI: 10.1001/jama.2023.22295] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/27/2023] [Accepted: 10/11/2023] [Indexed: 12/21/2023]
Abstract
Importance Artificial intelligence (AI) could support clinicians when diagnosing hospitalized patients; however, systematic bias in AI models could worsen clinician diagnostic accuracy. Recent regulatory guidance has called for AI models to include explanations to mitigate errors made by models, but the effectiveness of this strategy has not been established. Objectives To evaluate the impact of systematically biased AI on clinician diagnostic accuracy and to determine if image-based AI model explanations can mitigate model errors. Design, Setting, and Participants Randomized clinical vignette survey study administered between April 2022 and January 2023 across 13 US states involving hospitalist physicians, nurse practitioners, and physician assistants. Interventions Clinicians were shown 9 clinical vignettes of patients hospitalized with acute respiratory failure, including their presenting symptoms, physical examination, laboratory results, and chest radiographs. Clinicians were then asked to determine the likelihood of pneumonia, heart failure, or chronic obstructive pulmonary disease as the underlying cause(s) of each patient's acute respiratory failure. To establish baseline diagnostic accuracy, clinicians were shown 2 vignettes without AI model input. Clinicians were then randomized to see 6 vignettes with AI model input with or without AI model explanations. Among these 6 vignettes, 3 vignettes included standard-model predictions, and 3 vignettes included systematically biased model predictions. Main Outcomes and Measures Clinician diagnostic accuracy for pneumonia, heart failure, and chronic obstructive pulmonary disease. Results Median participant age was 34 years (IQR, 31-39) and 241 (57.7%) were female. Four hundred fifty-seven clinicians were randomized and completed at least 1 vignette, with 231 randomized to AI model predictions without explanations, and 226 randomized to AI model predictions with explanations. Clinicians' baseline diagnostic accuracy was 73.0% (95% CI, 68.3% to 77.8%) for the 3 diagnoses. When shown a standard AI model without explanations, clinician accuracy increased over baseline by 2.9 percentage points (95% CI, 0.5 to 5.2) and by 4.4 percentage points (95% CI, 2.0 to 6.9) when clinicians were also shown AI model explanations. Systematically biased AI model predictions decreased clinician accuracy by 11.3 percentage points (95% CI, 7.2 to 15.5) compared with baseline and providing biased AI model predictions with explanations decreased clinician accuracy by 9.1 percentage points (95% CI, 4.9 to 13.2) compared with baseline, representing a nonsignificant improvement of 2.3 percentage points (95% CI, -2.7 to 7.2) compared with the systematically biased AI model. Conclusions and Relevance Although standard AI models improve diagnostic accuracy, systematically biased AI models reduced diagnostic accuracy, and commonly used image-based AI model explanations did not mitigate this harmful effect. Trial Registration ClinicalTrials.gov Identifier: NCT06098950.
Collapse
Affiliation(s)
- Sarah Jabbour
- Computer Science and Engineering, University of Michigan, Ann Arbor
| | - David Fouhey
- Computer Science and Engineering, University of Michigan, Ann Arbor
- Now with Computer Science Courant Institute, New York University, New York
- Now with Electrical and Computer Engineering Tandon School of Engineering, New York University, New York
| | | | - Thomas S. Valley
- Pulmonary and Critical Care Medicine, Department of Internal Medicine, University of Michigan Medical School, Ann Arbor
| | - Ella A. Kazerooni
- Department of Radiology, University of Michigan Medical School, Ann Arbor
| | - Nikola Banovic
- Computer Science and Engineering, University of Michigan, Ann Arbor
| | - Jenna Wiens
- Computer Science and Engineering, University of Michigan, Ann Arbor
| | - Michael W. Sjoding
- Pulmonary and Critical Care Medicine, Department of Internal Medicine, University of Michigan Medical School, Ann Arbor
| |
Collapse
|
13
|
Qi L, Liang JY, Li ZW, Xi SY, Lai YN, Gao F, Zhang XR, Wang DS, Hu MT, Cao Y, Xu LJ, Chan RC, Xing BC, Wang X, Li YH. Deep learning-derived spatial organization features on histology images predicts prognosis in colorectal liver metastasis patients after hepatectomy. iScience 2023; 26:107702. [PMID: 37701575 PMCID: PMC10494211 DOI: 10.1016/j.isci.2023.107702] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2023] [Revised: 07/10/2023] [Accepted: 08/21/2023] [Indexed: 09/14/2023] Open
Abstract
Histopathological images of colorectal liver metastases (CRLM) contain rich morphometric information that may predict patients' outcomes. However, to our knowledge, no study has reported any practical deep learning framework based on the histology images of CRLM, and their direct association with prognosis remains largely unknown. In this study, we developed a deep learning-based framework for fully automated tissue classification and quantification of clinically relevant spatial organization features (SOFs) in H&E-stained images of CRLM. The SOFs based risk-scoring system demonstrated a strong and robust prognostic value that is independent of the current clinical risk score (CRS) system in independent clinical cohorts. Our framework enables fully automated tissue classification of H&E images of CRLM, which could significantly reduce assessment subjectivity and the workload of pathologists. The risk-scoring system provides a time- and cost-efficient tool to assist clinical decision-making for patients with CRLM, which could potentially be implemented in clinical practice.
Collapse
Affiliation(s)
- Lin Qi
- Department of Surgery, The Chinese University of Hong Kong, Hong Kong SAR, China
- Department of Biomedical Sciences, City University of Hong Kong, Hong Kong SAR, China
- Shenzhen Research Institute, City University of Hong Kong, Shenzhen, China
| | - Jie-ying Liang
- Guangdong Provincial Key Laboratory of Malignant Tumor Epigenetics and Gene Regulation, Department of Medical Oncology, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, China
- State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-Sen University Cancer Center, Guangzhou, China
- Department of Medical Oncology, Sun Yat-Sen University Cancer Center, Guangzhou, China
| | - Zhong-wu Li
- Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Department of Pathology, Peking University Cancer Hospital & Institute, Beijing, China
| | - Shao-yan Xi
- State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-Sen University Cancer Center, Guangzhou, China
- Department of Pathology, Sun Yat-Sen University Cancer Center, Guangzhou, China
| | - Yu-ni Lai
- Department of Biomedical Sciences, City University of Hong Kong, Hong Kong SAR, China
| | - Feng Gao
- Guangdong Provincial Key Laboratory of Colorectal and Pelvic Floor Disease, The Sixth Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Xian-rui Zhang
- Department of Surgery, The Chinese University of Hong Kong, Hong Kong SAR, China
- Department of Biomedical Sciences, City University of Hong Kong, Hong Kong SAR, China
- Shenzhen Research Institute, City University of Hong Kong, Shenzhen, China
| | - De-shen Wang
- State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-Sen University Cancer Center, Guangzhou, China
- Department of Medical Oncology, Sun Yat-Sen University Cancer Center, Guangzhou, China
| | - Ming-tao Hu
- State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-Sen University Cancer Center, Guangzhou, China
- Department of Medical Oncology, Sun Yat-Sen University Cancer Center, Guangzhou, China
| | - Yi Cao
- Department of Surgery, The Chinese University of Hong Kong, Hong Kong SAR, China
- Department of Biomedical Sciences, City University of Hong Kong, Hong Kong SAR, China
- Shenzhen Research Institute, City University of Hong Kong, Shenzhen, China
| | - Li-jian Xu
- Centre for Perceptual and Interactive Intelligence, The Chinese University of Hong Kong, Hong Kong SAR, China
| | - Ronald C.K. Chan
- Department of Pathology, The Chinese University of Hong Kong, Hong Kong SAR, China
| | - Bao-cai Xing
- Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Hepatopancreatobiliary Surgery Department I, Peking University Cancer Hospital & Institute, Beijing, China
| | - Xin Wang
- Department of Surgery, The Chinese University of Hong Kong, Hong Kong SAR, China
- Department of Biomedical Sciences, City University of Hong Kong, Hong Kong SAR, China
- Shenzhen Research Institute, City University of Hong Kong, Shenzhen, China
| | - Yu-hong Li
- State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-Sen University Cancer Center, Guangzhou, China
- Department of Medical Oncology, Sun Yat-Sen University Cancer Center, Guangzhou, China
| |
Collapse
|
14
|
Neary-Zajiczek L, Beresna L, Razavi B, Pawar V, Shaw M, Stoyanov D. Minimum resolution requirements of digital pathology images for accurate classification. Med Image Anal 2023; 89:102891. [PMID: 37536022 DOI: 10.1016/j.media.2023.102891] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Revised: 05/22/2023] [Accepted: 07/06/2023] [Indexed: 08/05/2023]
Abstract
Digitization of pathology has been proposed as an essential mitigation strategy for the severe staffing crisis facing most pathology departments. Despite its benefits, several barriers have prevented widespread adoption of digital workflows, including cost and pathologist reluctance due to subjective image quality concerns. In this work, we quantitatively determine the minimum image quality requirements for binary classification of histopathology images of breast tissue in terms of spatial and sampling resolution. We train an ensemble of deep learning classifier models on publicly available datasets to obtain a baseline accuracy and computationally degrade these images according to our derived theoretical model to identify the minimum resolution necessary for acceptable diagnostic accuracy. Our results show that images can be degraded significantly below the resolution of most commercial whole-slide imaging systems while maintaining reasonable accuracy, demonstrating that macroscopic features are sufficient for binary classification of stained breast tissue. A rapid low-cost imaging system capable of identifying healthy tissue not requiring human assessment could serve as a triage system for reducing caseloads and alleviating the significant strain on the current workforce.
Collapse
Affiliation(s)
- Lydia Neary-Zajiczek
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, Charles Bell House, 43-45 Foley Street, Fitzrovia, London, W1W 7TS, United Kingdom; Department of Computer Science, University College London, Gower Street, London, WC1E 6BT, United Kingdom.
| | - Linas Beresna
- Department of Computer Science, University College London, Gower Street, London, WC1E 6BT, United Kingdom
| | - Benjamin Razavi
- University College London Medical School, 74 Huntley Street, London, WC1E 6BT, United Kingdom
| | - Vijay Pawar
- Department of Computer Science, University College London, Gower Street, London, WC1E 6BT, United Kingdom
| | - Michael Shaw
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, Charles Bell House, 43-45 Foley Street, Fitzrovia, London, W1W 7TS, United Kingdom; Department of Computer Science, University College London, Gower Street, London, WC1E 6BT, United Kingdom; National Physical Laboratory, Hampton Road, Teddington, TW11 0LW, United Kingdom
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, Charles Bell House, 43-45 Foley Street, Fitzrovia, London, W1W 7TS, United Kingdom; Department of Computer Science, University College London, Gower Street, London, WC1E 6BT, United Kingdom
| |
Collapse
|
15
|
Lin T, Yu Z, Xu Z, Hu H, Xu Y, Chen CW. SGCL: Spatial guided contrastive learning on whole-slide pathological images. Med Image Anal 2023; 89:102845. [PMID: 37597317 DOI: 10.1016/j.media.2023.102845] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Revised: 05/09/2023] [Accepted: 05/15/2023] [Indexed: 08/21/2023]
Abstract
Self-supervised representation learning (SSL) has achieved remarkable success in its application to natural images while falling behind in performance when applied to whole-slide pathological images (WSIs). This is because the inherent characteristics of WSIs in terms of gigapixel resolution and multiple objects in training patches are fundamentally different from natural images. Directly transferring the state-of-the-art (SOTA) SSL methods designed for natural images to WSIs will inevitably compromise their performance. We present a novel scheme SGCL: Spatial Guided Contrastive Learning, to fully explore the inherent properties of WSIs, leveraging the spatial proximity and multi-object priors for stable self-supervision. Beyond the self-invariance of instance discrimination, we expand and propagate the spatial proximity for the intra-invariance from the same WSI and inter-invariance from different WSIs, as well as propose the spatial-guided multi-cropping for inner-invariance within patches. To adaptively explore such spatial information without supervision, we propose a new loss function and conduct a theoretical analysis to validate it. This novel scheme of SGCL is able to achieve additional improvements over the SOTA pre-training methods on diverse downstream tasks across multiple datasets. Extensive ablation studies have been carried out and visualizations of these results have been presented to aid understanding of the proposed SGCL scheme. As open science, all codes and pre-trained models are available at https://github.com/HHHedo/SGCL.
Collapse
Affiliation(s)
- Tiancheng Lin
- Shanghai Key Lab of Digital Media Processing and Transmission, Shanghai Jiao Tong University, China; MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China
| | - Zhimiao Yu
- Shanghai Key Lab of Digital Media Processing and Transmission, Shanghai Jiao Tong University, China; MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China
| | - Zengchao Xu
- Department of Mathematics and Lab for Educational Big Data and Policymaking, Shanghai Normal University, China
| | - Hongyu Hu
- Shanghai Key Lab of Digital Media Processing and Transmission, Shanghai Jiao Tong University, China; MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China
| | - Yi Xu
- Shanghai Key Lab of Digital Media Processing and Transmission, Shanghai Jiao Tong University, China; MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China.
| | | |
Collapse
|
16
|
Jing Y, Li C, Du T, Jiang T, Sun H, Yang J, Shi L, Gao M, Grzegorzek M, Li X. A comprehensive survey of intestine histopathological image analysis using machine vision approaches. Comput Biol Med 2023; 165:107388. [PMID: 37696178 DOI: 10.1016/j.compbiomed.2023.107388] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Revised: 08/06/2023] [Accepted: 08/25/2023] [Indexed: 09/13/2023]
Abstract
Colorectal Cancer (CRC) is currently one of the most common and deadly cancers. CRC is the third most common malignancy and the fourth leading cause of cancer death worldwide. It ranks as the second most frequent cause of cancer-related deaths in the United States and other developed countries. Histopathological images contain sufficient phenotypic information, they play an indispensable role in the diagnosis and treatment of CRC. In order to improve the objectivity and diagnostic efficiency for image analysis of intestinal histopathology, Computer-aided Diagnosis (CAD) methods based on machine learning (ML) are widely applied in image analysis of intestinal histopathology. In this investigation, we conduct a comprehensive study on recent ML-based methods for image analysis of intestinal histopathology. First, we discuss commonly used datasets from basic research studies with knowledge of intestinal histopathology relevant to medicine. Second, we introduce traditional ML methods commonly used in intestinal histopathology, as well as deep learning (DL) methods. Then, we provide a comprehensive review of the recent developments in ML methods for segmentation, classification, detection, and recognition, among others, for histopathological images of the intestine. Finally, the existing methods have been studied, and the application prospects of these methods in this field are given.
Collapse
Affiliation(s)
- Yujie Jing
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China; Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China
| | - Chen Li
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China; Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China.
| | - Tianming Du
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China; Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China
| | - Tao Jiang
- School of Intelligent Medicine, Chengdu University of Traditional Chinese Medicine, Chengdu, China; International Joint Institute of Robotics and Intelligent Systems, Chengdu University of Information Technology, Chengdu, China
| | - Hongzan Sun
- Shengjing Hospital of China Medical University, Shenyang, China
| | - Jinzhu Yang
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China
| | - Liyu Shi
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China; Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China
| | - Minghe Gao
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, China; Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China
| | - Marcin Grzegorzek
- Institute for Medical Informatics, University of Luebeck, Luebeck, Germany; Department of Knowledge Engineering, University of Economics in Katowice, Katowice, Poland
| | - Xiaoyan Li
- Cancer Hospital of China Medical University, Liaoning Cancer Hospital, Shenyang, China.
| |
Collapse
|
17
|
Shao Z, Dai L, Jonnagaddala J, Chen Y, Wang Y, Fang Z, Zhang Y. Generalizability of Self-Supervised Training Models for Digital Pathology: A Multicountry Comparison in Colorectal Cancer. JCO Clin Cancer Inform 2023; 7:e2200178. [PMID: 37703507 DOI: 10.1200/cci.22.00178] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2022] [Revised: 06/15/2023] [Accepted: 06/27/2023] [Indexed: 09/15/2023] Open
Abstract
PURPOSE In this multicountry study, we aim to explore the effectiveness of self-supervised learning (SSL) in colorectal cancer (CRC)-related predictive tasks using large amount of unlabeled digital pathology imaging data. METHODS We adopted SimSiam to conduct self-supervised pretraining on two large whole-slide image CRC data sets from the United States and Australia. The SSL pretrained encoder is then used in several predictive tasks, including supervised predictive tasks (tissue classification, microsatellite instability v microsatellite stability classification), and weakly supervised predictive tasks (polyp type classification and adenoma grading, and 5-year survival prediction). Performance on the tasks was compared between models using SSL pretraining and those using ImageNet pretraining, and performance for one-country pretraining was compared with two-country pretraining. RESULTS We demonstrate that SSL pretraining outperforms ImageNet pretraining in predictive tasks, that is, SSL pretraining outperforms the ImageNet pretraining by 3.01% of F 1 score on average over supervised predictive tasks and 1.53% of AUC on average over weakly supervised predictive tasks. Furthermore, two-country SSL pretraining has shown more stable performance than single-country pretraining, that is, two-country pretraining outperforms at least one of the single-country pretrainings by 1.93% of F 1 on average over supervised predictive tasks and 1.36% of AUC on average over weakly-supervised predictive tasks. CONCLUSION We find that using unlabeled image data for SSL pretraining in CRC related tasks is more effective than using ImageNet pretraining. Furthermore, SSL pretraining using data from multiple countries achieve more stable performance and better generalization than single-country pretraining.
Collapse
Affiliation(s)
- Zhuchen Shao
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
| | - Liuxi Dai
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
| | | | - Yang Chen
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
| | - Yifeng Wang
- School of Science, Harbin Institute of Technology, Shenzhen, China
| | - Zijie Fang
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
| | - Yongbing Zhang
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
| |
Collapse
|
18
|
Liu Z, Lv Q, Yang Z, Li Y, Lee CH, Shen L. Recent progress in transformer-based medical image analysis. Comput Biol Med 2023; 164:107268. [PMID: 37494821 DOI: 10.1016/j.compbiomed.2023.107268] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 05/30/2023] [Accepted: 07/16/2023] [Indexed: 07/28/2023]
Abstract
The transformer is primarily used in the field of natural language processing. Recently, it has been adopted and shows promise in the computer vision (CV) field. Medical image analysis (MIA), as a critical branch of CV, also greatly benefits from this state-of-the-art technique. In this review, we first recap the core component of the transformer, the attention mechanism, and the detailed structures of the transformer. After that, we depict the recent progress of the transformer in the field of MIA. We organize the applications in a sequence of different tasks, including classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis. The mainstream classification and segmentation tasks are further divided into eleven medical image modalities. A large number of experiments studied in this review illustrate that the transformer-based method outperforms existing methods through comparisons with multiple evaluation metrics. Finally, we discuss the open challenges and future opportunities in this field. This task-modality review with the latest contents, detailed information, and comprehensive comparison may greatly benefit the broad MIA community.
Collapse
Affiliation(s)
- Zhaoshan Liu
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore.
| | - Qiujie Lv
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore; School of Intelligent Systems Engineering, Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, 518107, China.
| | - Ziduo Yang
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore; School of Intelligent Systems Engineering, Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, 518107, China.
| | - Yifan Li
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore.
| | - Chau Hung Lee
- Department of Radiology, Tan Tock Seng Hospital, 11 Jalan Tan Tock Seng, Singapore, 308433, Singapore.
| | - Lei Shen
- Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117575, Singapore.
| |
Collapse
|
19
|
Wang Y, Ali MA, Vallon-Christersson J, Humphreys K, Hartman J, Rantalainen M. Transcriptional intra-tumour heterogeneity predicted by deep learning in routine breast histopathology slides provides independent prognostic information. Eur J Cancer 2023; 191:112953. [PMID: 37494846 DOI: 10.1016/j.ejca.2023.112953] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2023] [Revised: 06/05/2023] [Accepted: 06/17/2023] [Indexed: 07/28/2023]
Abstract
BACKGROUND Intra-tumour heterogeneity (ITH) causes diagnostic challenges and increases the risk for disease recurrence. Quantification of ITH is challenging and has not been demonstrated in large studies. It has previously been shown that deep learning can enable spatially resolved prediction of molecular phenotypes from digital histopathology whole slide images (WSIs). Here we propose a novel method (Deep-ITH) to predict and measure ITH, and we evaluate its prognostic performance in breast cancer. METHODS Deep convolutional neural networks were used to spatially predict gene-expression (PAM50 set) from WSIs. For each predicted transcript, 12 measures of heterogeneity were extracted in the training data set (N = 931). A prognostic score to dichotomise patients into Deep-ITH low- and high-risk groups was established using an elastic-net regularised Cox proportional hazards model (recurrence-free survival). Prognostic performance was evaluated in two independent data sets: SöS-BC-1 (N = 1358) and SCAN-B-Lund (N = 1262). RESULTS We observed an increase in risk of recurrence in the high-risk group with hazard ratio (HR) 2.11 (95%CI:1.22-3.60; p = 0.007) using nested cross-validation. Subgroup analyses confirmed the prognostic performance in oestrogen receptor (ER)-positive, human epidermal growth factor receptor 2 (HER2)-negative, grade 3, and large tumour subgroups. The prognostic value was confirmed in the independent SöS-BC-1 cohort (HR=1.84; 95%CI:1.03-3.3; p = 3.99 ×10-2). In the other external cohort, significant HR was observed in the subgroup of histological grade 2 patients, as well as in the subgroup of patients with small tumours (<20 mm). CONCLUSION We developed a novel method for an automated, scalable, and cost-efficient measure of ITH from WSIs that provides independent prognostic value for breast cancer. SIGNIFICANCE Transcriptional ITH predicted by deep learning models enables prediction of patient survival from routine histopathology WSIs in breast cancer.
Collapse
Affiliation(s)
- Yinxi Wang
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Maya Alsheh Ali
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | | | - Keith Humphreys
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Johan Hartman
- Department of Oncology-Pathology, Karolinska Institutet, Stockholm, Sweden; Department of Clinical Pathology and Cancer Diagnostics, Karolinska University Hospital, Stockholm, Sweden; MedTechLabs, BioClinicum, Karolinska University Hospital, Solna, Sweden
| | - Mattias Rantalainen
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden; MedTechLabs, BioClinicum, Karolinska University Hospital, Solna, Sweden.
| |
Collapse
|
20
|
Fisher TB, Saini G, Ts R, Krishnamurthy J, Bhattarai S, Callagy G, Webber M, Janssen EAM, Kong J, Aneja R. Digital image analysis and machine learning-assisted prediction of neoadjuvant chemotherapy response in triple-negative breast cancer. RESEARCH SQUARE 2023:rs.3.rs-3243195. [PMID: 37645881 PMCID: PMC10462230 DOI: 10.21203/rs.3.rs-3243195/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/31/2023]
Abstract
Background Pathological complete response (pCR) is associated with favorable prognosis in patients with triple-negative breast cancer (TNBC). However, only 30-40% of TNBC patients treated with neoadjuvant chemotherapy (NAC) show pCR, while the remaining 60-70% show residual disease (RD). The role of the tumor microenvironment (TME) in NAC response in patients with TNBC remains unclear. In this study, we developed a machine learning-based two-step pipeline to distinguish between various histological components in hematoxylin and eosin (H&E)-stained whole slide images (WSIs) of TNBC tissue biopsies and to identify histological features that can predict NAC response. Methods H&E-stained WSIs of treatment-naïve biopsies from 85 patients (51 with pCR and 34 with RD) were separated through a stratified 8-fold cross validation strategy for the first step and leave one out cross validation strategy for the second step. A tile-level histology label prediction pipeline and four machine learning classifiers were used to analyze 468,043 tiles of WSIs. The best-trained classifier used 55 texture features from each tile to produce a probability profile during testing. The predicted histology classes were used to generate a histology classification map of the spatial distributions of different tissue regions. A patient-level NAC response prediction pipeline was trained with features derived from paired histology classification maps. The top graph-based features capturing the relevant spatial information across the different histological classes were provided to the radial basis function kernel support vector machine (rbfSVM) classifier for NAC treatment response prediction. Results The tile-level prediction pipeline achieved 86.72% accuracy for histology class classification, while the patient-level pipeline achieved 83.53% NAC response (pCR vs. RD) prediction accuracy. The histological class pairs with the strongest NAC response predictive ability were tumor and tumor tumor-infiltrating lymphocytes for pCR and microvessel density and polyploid giant cancer cells for RD. Conclusion Our machine learning pipeline can robustly identify clinically relevant histological classes that predict NAC response in TNBC patients and may help guide patient selection for NAC treatment.
Collapse
Affiliation(s)
| | | | - Rekha Ts
- JSSAHER (JSS Academy of Higher Education and Research) Medical College
| | | | | | | | | | | | | | | |
Collapse
|
21
|
Baidar Bakht A, Javed S, Gilani SQ, Karki H, Muneeb M, Werghi N. DeepBLS: Deep Feature-Based Broad Learning System for Tissue Phenotyping in Colorectal Cancer WSIs. J Digit Imaging 2023; 36:1653-1662. [PMID: 37059892 PMCID: PMC10406762 DOI: 10.1007/s10278-023-00797-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Revised: 02/09/2023] [Accepted: 02/10/2023] [Indexed: 04/16/2023] Open
Abstract
Tissue phenotyping is a fundamental step in computational pathology for the analysis of tumor micro-environment in whole slide images (WSIs). Automatic tissue phenotyping in whole slide images (WSIs) of colorectal cancer (CRC) assists pathologists in better cancer grading and prognostication. In this paper, we propose a novel algorithm for the identification of distinct tissue components in colon cancer histology images by blending a comprehensive learning system with deep features extraction in the current work. Firstly, we extracted the features from the pre-trained VGG19 network which are then transformed into mapped features space for nodes enhancement generation. Utilizing both mapped features and enhancement nodes, the proposed algorithm classifies seven distinct tissue components including stroma, tumor, complex stroma, necrotic, normal benign, lymphocytes, and smooth muscle. To validate our proposed model, the experiments are performed on two publicly available colorectal cancer histology datasets. We showcase that our approach achieves a remarkable performance boost surpassing existing state-of-the-art methods by (1.3% AvTP, 2% F1) and (7% AvTP, 6% F1) on CRCD-1, and CRCD-2, respectively.
Collapse
Affiliation(s)
- Ahsan Baidar Bakht
- Electrical and Computer Engineering Department, Khalifa University, 12778 Abu Dhabi, United Arab Emirates
| | - Sajid Javed
- Electrical and Computer Engineering Department, Khalifa University, 12778 Abu Dhabi, United Arab Emirates
| | - Syed Qasim Gilani
- Department of Electrical Engineering and Computer Science, Florida Atlantic University, Boca Raton, 33431 USA
| | - Hamad Karki
- Mechanical Engineering Department, Khalifa University, 12778 Abu Dhabi, United Arab Emirates
| | - Muhammad Muneeb
- Electrical and Computer Engineering Department, Khalifa University, 12778 Abu Dhabi, United Arab Emirates
| | - Naoufel Werghi
- Electrical and Computer Engineering Department, Khalifa University, 12778 Abu Dhabi, United Arab Emirates
| |
Collapse
|
22
|
Wu Y, Li Y, Xiong X, Liu X, Lin B, Xu B. Recent advances of pathomics in colorectal cancer diagnosis and prognosis. Front Oncol 2023; 13:1094869. [PMID: 37538112 PMCID: PMC10396402 DOI: 10.3389/fonc.2023.1094869] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Accepted: 06/13/2023] [Indexed: 08/05/2023] Open
Abstract
Colorectal cancer (CRC) is one of the most common malignancies, with the third highest incidence and the second highest mortality in the world. To improve the therapeutic outcome, the risk stratification and prognosis predictions would help guide clinical treatment decisions. Achieving these goals have been facilitated by the fast development of artificial intelligence (AI) -based algorithms using radiological and pathological data, in combination with genomic information. Among them, features extracted from pathological images, termed pathomics, are able to reflect sub-visual characteristics linking to better stratification and prediction of therapeutic responses. In this paper, we review recent advances in pathological image-based algorithms in CRC, focusing on diagnosis of benign and malignant lesions, micro-satellite instability, as well as prediction of neoadjuvant chemoradiotherapy and the prognosis of CRC patients.
Collapse
Affiliation(s)
- Yihan Wu
- School of Medicine, Chongqing University, Chongqing, China
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| | - Yi Li
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
- Bioengineering College, Chongqing University, Chongqing, China
| | - Xiaomin Xiong
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
- Bioengineering College, Chongqing University, Chongqing, China
| | - Xiaohua Liu
- Bioengineering College, Chongqing University, Chongqing, China
| | - Bo Lin
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| | - Bo Xu
- School of Medicine, Chongqing University, Chongqing, China
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| |
Collapse
|
23
|
Chattopadhyay S, Singh PK, Ijaz MF, Kim S, Sarkar R. SnapEnsemFS: a snapshot ensembling-based deep feature selection model for colorectal cancer histological analysis. Sci Rep 2023; 13:9937. [PMID: 37336964 DOI: 10.1038/s41598-023-36921-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Accepted: 06/12/2023] [Indexed: 06/21/2023] Open
Abstract
Colorectal cancer is the third most common type of cancer diagnosed annually, and the second leading cause of death due to cancer. Early diagnosis of this ailment is vital for preventing the tumours to spread and plan treatment to possibly eradicate the disease. However, population-wide screening is stunted by the requirement of medical professionals to analyse histological slides manually. Thus, an automated computer-aided detection (CAD) framework based on deep learning is proposed in this research that uses histological slide images for predictions. Ensemble learning is a popular strategy for fusing the salient properties of several models to make the final predictions. However, such frameworks are computationally costly since it requires the training of multiple base learners. Instead, in this study, we adopt a snapshot ensemble method, wherein, instead of the traditional method of fusing decision scores from the snapshots of a Convolutional Neural Network (CNN) model, we extract deep features from the penultimate layer of the CNN model. Since the deep features are extracted from the same CNN model but for different learning environments, there may be redundancy in the feature set. To alleviate this, the features are fed into Particle Swarm Optimization, a popular meta-heuristic, for dimensionality reduction of the feature space and better classification. Upon evaluation on a publicly available colorectal cancer histology dataset using a five-fold cross-validation scheme, the proposed method obtains a highest accuracy of 97.60% and F1-Score of 97.61%, outperforming existing state-of-the-art methods on the same dataset. Further, qualitative investigation of class activation maps provide visual explainability to medical practitioners, as well as justifies the use of the CAD framework in screening of colorectal histology. Our source codes are publicly accessible at: https://github.com/soumitri2001/SnapEnsemFS .
Collapse
Affiliation(s)
- Soumitri Chattopadhyay
- Department of Information Technology, Jadavpur University, Jadavpur University Second Campus, Plot No. 8, Salt Lake Bypass, LB Block, Sector III, Salt Lake City, Kolkata, 700106, West Bengal, India
| | - Pawan Kumar Singh
- Department of Information Technology, Jadavpur University, Jadavpur University Second Campus, Plot No. 8, Salt Lake Bypass, LB Block, Sector III, Salt Lake City, Kolkata, 700106, West Bengal, India
| | - Muhammad Fazal Ijaz
- Department of Mechanical Engineering, Faculty of Engineering and Information Technology, The University of Melbourne, Grattam Street, Parkville, VIC, 3010, Australia.
| | - SeongKi Kim
- National Centre of Excellence in Software, Sangmyung University, Seoul, 03016, Korea.
| | - Ram Sarkar
- Department of Computer Science & Engineering, Jadavpur University, Kolkata, 700032, India
| |
Collapse
|
24
|
Shao Y, Fan X, Yang X, Li S, Huang L, Zhou X, Zhang S, Zheng M, Sun J. Impact of Cuproptosis-related markers on clinical status, tumor immune microenvironment and immunotherapy in colorectal cancer: A multi-omic analysis. Comput Struct Biotechnol J 2023; 21:3383-3403. [PMID: 37389187 PMCID: PMC10300104 DOI: 10.1016/j.csbj.2023.06.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Revised: 06/07/2023] [Accepted: 06/10/2023] [Indexed: 07/01/2023] Open
Abstract
Background Cuproptosis, a novel identified cell death form induced by copper, is characterized by aggregation of lipoylated mitochondrial enzymes and the destabilization of Fe-S cluster proteins. However, the function and potential clinical value of cuproptosis and cuproptosis-related biomarkers in colorectal cancer (CRC) remain largely unknown. Methods A comprehensive multi-omics (transcriptomics, genomics, and single-cell transcriptome) analysis was performed for identifying the influence of 16 cuproptosis-related markers on clinical status, molecular functions and tumor microenvironment (TME) in CRC. A novel cuproptosis-related scoring system (CuproScore) based on cuproptosis-related markers was also constructed to predict the prognosis of CRC individuals, TME and the response to immunotherapy. In addition, our transcriptome cohort of 15 paired CRC tissue, tissue-array, and various assays in 4 kinds of CRC cell lines in vitro were applied for verification. Results Cuproptosis-related markers were closely associated with both clinical prognosis and molecular functions. And the cuproptosis-related molecular phenotypes and scoring system (CuproScore) could distinguish and predict the prognosis of CRC patients, TME, and the response to immunotherapy in both public and our transcriptome cohorts. Besides, the expression, function and clinical significance of these markers were also checked and analyzed in CRC cell lines and CRC tissues in our own cohorts. Conclusions In conclusion, we indicated that cuproptosis and CPRMs played a significant role in CRC progression and in modeling the TME. Inducing cuproptosis may be a useful tool for tumor therapy in the future.
Collapse
Affiliation(s)
- Yanfei Shao
- Department of General Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Xiaodong Fan
- Department of General Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Xiao Yang
- Department of General Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Shuchun Li
- Department of General Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Ling Huang
- Department of General Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Xueliang Zhou
- Department of General Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Sen Zhang
- Department of General Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Minhua Zheng
- Department of General Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Jing Sun
- Department of General Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
- Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| |
Collapse
|
25
|
Altini N, Marvulli TM, Zito FA, Caputo M, Tommasi S, Azzariti A, Brunetti A, Prencipe B, Mattioli E, De Summa S, Bevilacqua V. The role of unpaired image-to-image translation for stain color normalization in colorectal cancer histology classification. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 234:107511. [PMID: 37011426 DOI: 10.1016/j.cmpb.2023.107511] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/30/2022] [Revised: 03/14/2023] [Accepted: 03/25/2023] [Indexed: 06/19/2023]
Abstract
BACKGROUND Histological assessment of colorectal cancer (CRC) tissue is a crucial and demanding task for pathologists. Unfortunately, manual annotation by trained specialists is a burdensome operation, which suffers from problems like intra- and inter-pathologist variability. Computational models are revolutionizing the Digital Pathology field, offering reliable and fast approaches for challenges like tissue segmentation and classification. With this respect, an important obstacle to overcome consists in stain color variations among different laboratories, which can decrease the performance of classifiers. In this work, we investigated the role of Unpaired Image-to-Image Translation (UI2IT) models for stain color normalization in CRC histology and compared to classical normalization techniques for Hematoxylin-Eosin (H&E) images. METHODS Five Deep Learning normalization models based on Generative Adversarial Networks (GANs) belonging to the UI2IT paradigm have been thoroughly compared to realize a robust stain color normalization pipeline. To avoid the need for training a style transfer GAN between each pair of data domains, in this paper we introduce the concept of training by exploiting a meta-domain, which contains data coming from a wide variety of laboratories. The proposed framework enables a huge saving in terms of training time, by allowing to train a single image normalization model for a target laboratory. To prove the applicability of the proposed workflow in the clinical practice, we conceived a novel perceptive quality measure, which we defined as Pathologist Perceptive Quality (PPQ). The second stage involved the classification of tissue types in CRC histology, where deep features extracted from Convolutional Neural Networks have been exploited to realize a Computer-Aided Diagnosis system based on a Support Vector Machine (SVM). To prove the reliability of the system on new data, an external validation set composed of N = 15,857 tiles has been collected at IRCCS Istituto Tumori "Giovanni Paolo II". RESULTS The exploitation of a meta-domain consented to train normalization models that allowed achieving better classification results than normalization models explicitly trained on the source domain. PPQ metric has been found correlated to quality of distributions (Fréchet Inception Distance - FID) and to similarity of the transformed image to the original one (Learned Perceptual Image Patch Similarity - LPIPS), thus showing that GAN quality measures introduced in natural image processing tasks can be linked to pathologist evaluation of H&E images. Furthermore, FID has been found correlated to accuracies of the downstream classifiers. The SVM trained on DenseNet201 features allowed to obtain the highest classification results in all configurations. The normalization method based on the fast variant of CUT (Contrastive Unpaired Translation), FastCUT, trained with the meta-domain paradigm, allowed to achieve the best classification result for the downstream task and, correspondingly, showed the highest FID on the classification dataset. CONCLUSIONS Stain color normalization is a difficult but fundamental problem in the histopathological setting. Several measures should be considered for properly assessing normalization methods, so that they can be introduced in the clinical practice. UI2IT frameworks offer a powerful and effective way to perform the normalization process, providing realistic images with proper colorization, unlike traditional normalization methods that introduce color artifacts. By adopting the proposed meta-domain framework, the training time can be reduced, and the accuracy of downstream classifiers can be increased.
Collapse
Affiliation(s)
- Nicola Altini
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona, 4, Bari 70126, Italy.
| | - Tommaso Maria Marvulli
- Laboratory of Experimental Pharmacology, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Francesco Alfredo Zito
- Pathology Department, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Mariapia Caputo
- Molecular Diagnostics and Pharmacogenetics Unit, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Stefania Tommasi
- Molecular Diagnostics and Pharmacogenetics Unit, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Amalia Azzariti
- Laboratory of Experimental Pharmacology, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Antonio Brunetti
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona, 4, Bari 70126, Italy; Apulian Bioengineering srl, Via delle Violette, 14, Modugno 70026, Italy
| | - Berardino Prencipe
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona, 4, Bari 70126, Italy
| | - Eliseo Mattioli
- Pathology Department, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Simona De Summa
- Molecular Diagnostics and Pharmacogenetics Unit, IRCCS Istituto Tumori "Giovanni Paolo II", Via O. Flacco, 65, Bari 70124, Italy
| | - Vitoantonio Bevilacqua
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona, 4, Bari 70126, Italy; Apulian Bioengineering srl, Via delle Violette, 14, Modugno 70026, Italy
| |
Collapse
|
26
|
Khazaee Fadafen M, Rezaee K. Ensemble-based multi-tissue classification approach of colorectal cancer histology images using a novel hybrid deep learning framework. Sci Rep 2023; 13:8823. [PMID: 37258631 DOI: 10.1038/s41598-023-35431-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2022] [Accepted: 05/17/2023] [Indexed: 06/02/2023] Open
Abstract
Colorectal cancer (CRC) is the second leading cause of cancer death in the world, so digital pathology is essential for assessing prognosis. Due to the increasing resolution and quantity of whole slide images (WSIs), as well as the lack of annotated information, previous methodologies cannot be generalized as effective decision-making systems. Since deep learning (DL) methods can handle large-scale applications, they can provide a viable alternative to histopathology image (HI) analysis. DL architectures, however, may not be sufficient to classify CRC tissues based on anatomical histopathology data. A dilated ResNet (dResNet) structure and attention module are used to generate deep feature maps in order to classify multiple tissues in HIs. In addition, neighborhood component analysis (NCA) overcomes the constraint of computational complexity. Data is fed into a deep support vector machine (SVM) based on an ensemble learning algorithm called DeepSVM after the features have been selected. CRC-5000 and NCT-CRC-HE-100 K datasets were analyzed to validate and test the hybrid procedure. We demonstrate that the hybrid model achieves 98.75% and 99.76% accuracy on CRC datasets. The results showed that only pathologists' labels could successfully classify unseen WSIs. Furthermore, the hybrid deep learning method outperforms state-of-the-art approaches in terms of computational efficiency and time. Using the proposed mechanism for tissue analysis, it will be possible to correctly predict CRC based on accurate pathology image classification.
Collapse
Affiliation(s)
- Masoud Khazaee Fadafen
- Department of Electrical Engineering, Technical and Vocational University (TVU), Tehran, Iran
| | - Khosro Rezaee
- Department of Biomedical Engineering, Meybod University, Meybod, Iran.
| |
Collapse
|
27
|
Bokhorst JM, Nagtegaal ID, Fraggetta F, Vatrano S, Mesker W, Vieth M, van der Laak J, Ciompi F. Deep learning for multi-class semantic segmentation enables colorectal cancer detection and classification in digital pathology images. Sci Rep 2023; 13:8398. [PMID: 37225743 DOI: 10.1038/s41598-023-35491-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Accepted: 05/18/2023] [Indexed: 05/26/2023] Open
Abstract
In colorectal cancer (CRC), artificial intelligence (AI) can alleviate the laborious task of characterization and reporting on resected biopsies, including polyps, the numbers of which are increasing as a result of CRC population screening programs ongoing in many countries all around the globe. Here, we present an approach to address two major challenges in the automated assessment of CRC histopathology whole-slide images. We present an AI-based method to segment multiple ([Formula: see text]) tissue compartments in the H &E-stained whole-slide image, which provides a different, more perceptible picture of tissue morphology and composition. We test and compare a panel of state-of-the-art loss functions available for segmentation models, and provide indications about their use in histopathology image segmentation, based on the analysis of (a) a multi-centric cohort of CRC cases from five medical centers in the Netherlands and Germany, and (b) two publicly available datasets on segmentation in CRC. We used the best performing AI model as the basis for a computer-aided diagnosis system that classifies colon biopsies into four main categories that are relevant pathologically. We report the performance of this system on an independent cohort of more than 1000 patients. The results show that with a good segmentation network as a base, a tool can be developed which can support pathologists in the risk stratification of colorectal cancer patients, among other possible uses. We have made the segmentation model available for research use on https://grand-challenge.org/algorithms/colon-tissue-segmentation/ .
Collapse
Affiliation(s)
- John-Melle Bokhorst
- Department of pathology, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - Iris D Nagtegaal
- Department of pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Filippo Fraggetta
- Pathology Unit Gravina Hospital, Gravina Hospital, Caltagirone, Italy
| | - Simona Vatrano
- Pathology Unit Gravina Hospital, Gravina Hospital, Caltagirone, Italy
| | - Wilma Mesker
- Leids Universitair Medisch Centrum, Leiden, The Netherlands
| | - Michael Vieth
- Klinikum Bayreuth, Friedrich-Alexander-University Erlangen-Nuremberg, Bayreuth, Germany
| | - Jeroen van der Laak
- Department of pathology, Radboud University Medical Center, Nijmegen, The Netherlands
- Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | - Francesco Ciompi
- Department of pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
28
|
Yong MP, Hum YC, Lai KW, Lee YL, Goh CH, Yap WS, Tee YK. Histopathological Gastric Cancer Detection on GasHisSDB Dataset Using Deep Ensemble Learning. Diagnostics (Basel) 2023; 13:diagnostics13101793. [PMID: 37238277 DOI: 10.3390/diagnostics13101793] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Revised: 05/08/2023] [Accepted: 05/14/2023] [Indexed: 05/28/2023] Open
Abstract
Gastric cancer is a leading cause of cancer-related deaths worldwide, underscoring the need for early detection to improve patient survival rates. The current clinical gold standard for detection is histopathological image analysis, but this process is manual, laborious, and time-consuming. As a result, there has been growing interest in developing computer-aided diagnosis to assist pathologists. Deep learning has shown promise in this regard, but each model can only extract a limited number of image features for classification. To overcome this limitation and improve classification performance, this study proposes ensemble models that combine the decisions of several deep learning models. To evaluate the effectiveness of the proposed models, we tested their performance on the publicly available gastric cancer dataset, Gastric Histopathology Sub-size Image Database. Our experimental results showed that the top 5 ensemble model achieved state-of-the-art detection accuracy in all sub-databases, with the highest detection accuracy of 99.20% in the 160 × 160 pixels sub-database. These results demonstrated that ensemble models could extract important features from smaller patch sizes and achieve promising performance. Overall, our proposed work could assist pathologists in detecting gastric cancer through histopathological image analysis and contribute to early gastric cancer detection to improve patient survival rates.
Collapse
Affiliation(s)
- Ming Ping Yong
- Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Malaysia
| | - Yan Chai Hum
- Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Malaysia
| | - Khin Wee Lai
- Department of Biomedical Engineering, Faculty of Engineering, Universiti Malaya, Kuala Lumpur 50603, Malaysia
| | - Ying Loong Lee
- Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Malaysia
| | - Choon-Hian Goh
- Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Malaysia
| | - Wun-She Yap
- Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Malaysia
| | - Yee Kai Tee
- Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Malaysia
| |
Collapse
|
29
|
Qiao Z, Li L, Zhao X, Liu L, Zhang Q, Hechmi S, Atri M, Li X. An enhanced Runge Kutta boosted machine learning framework for medical diagnosis. Comput Biol Med 2023; 160:106949. [PMID: 37159961 DOI: 10.1016/j.compbiomed.2023.106949] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/24/2022] [Revised: 03/27/2023] [Accepted: 04/15/2023] [Indexed: 05/11/2023]
Abstract
With the development and maturity of machine learning methods, medical diagnosis aided with machine learning methods has become a popular method to assist doctors in diagnosing and treating patients. However, machine learning methods are greatly affected by their hyperparameters, for instance, the kernel parameter in kernel extreme learning machine (KELM) and the learning rate in residual neural networks (ResNet). If the hyperparameters are appropriately set, the performance of the classifier can be significantly improved. To boost the performance of the machine learning methods, this paper proposes to improve the Runge Kutta optimizer (RUN) to adaptively adjust the hyperparameters of the machine learning methods for medical diagnosis purposes. Although RUN has a solid mathematical theoretical foundation, there are still some performance defects when dealing with complex optimization problems. To remedy these defects, this paper proposes a new enhanced RUN method with a grey wolf mechanism and an orthogonal learning mechanism called GORUN. The superior performance of the GORUN was validated against other well-established optimizers on IEEE CEC 2017 benchmark functions. Then, the proposed GORUN is employed to optimize the machine learning models, including the KELM and ResNet, to construct robust models for medical diagnosis. The performance of the proposed machine learning framework was validated on several medical data sets, and the experimental results have demonstrated its superiority.
Collapse
Affiliation(s)
- Zenglin Qiao
- School of Science, Beijing University of Posts and Telecommunications, Beijing, 100876, China.
| | - Lynn Li
- China Telecom Stocks Co.,Ltd., Hangzhou Branch, Hangzhou, 310000, China.
| | - Xinchao Zhao
- School of Science, Beijing University of Posts and Telecommunications, Beijing, 100876, China.
| | - Lei Liu
- College of Computer Science, Sichuan University, Chengdu, Sichuan, 610065, China.
| | - Qian Zhang
- School of Data Science and Artificial Intelligence, Wenzhou University of Technology, Wenzhou, Zhejiang, 325035, China.
| | - Shili Hechmi
- Dept. Computer Sciences, Tabuk University, Tabuk, Saudi Arabia.
| | - Mohamed Atri
- College of Computer Science, King Khalid University, Abha, Saudi Arabia.
| | - Xiaohua Li
- Library, Wenzhou University, Wenzhou, Zhejiang, 325035, China.
| |
Collapse
|
30
|
Mokoatle M, Marivate V, Mapiye D, Bornman R, Hayes VM. A review and comparative study of cancer detection using machine learning: SBERT and SimCSE application. BMC Bioinformatics 2023; 24:112. [PMID: 36959534 PMCID: PMC10037872 DOI: 10.1186/s12859-023-05235-x] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Accepted: 03/17/2023] [Indexed: 03/25/2023] Open
Abstract
BACKGROUND Using visual, biological, and electronic health records data as the sole input source, pretrained convolutional neural networks and conventional machine learning methods have been heavily employed for the identification of various malignancies. Initially, a series of preprocessing steps and image segmentation steps are performed to extract region of interest features from noisy features. Then, the extracted features are applied to several machine learning and deep learning methods for the detection of cancer. METHODS In this work, a review of all the methods that have been applied to develop machine learning algorithms that detect cancer is provided. With more than 100 types of cancer, this study only examines research on the four most common and prevalent cancers worldwide: lung, breast, prostate, and colorectal cancer. Next, by using state-of-the-art sentence transformers namely: SBERT (2019) and the unsupervised SimCSE (2021), this study proposes a new methodology for detecting cancer. This method requires raw DNA sequences of matched tumor/normal pair as the only input. The learnt DNA representations retrieved from SBERT and SimCSE will then be sent to machine learning algorithms (XGBoost, Random Forest, LightGBM, and CNNs) for classification. As far as we are aware, SBERT and SimCSE transformers have not been applied to represent DNA sequences in cancer detection settings. RESULTS The XGBoost model, which had the highest overall accuracy of 73 ± 0.13 % using SBERT embeddings and 75 ± 0.12 % using SimCSE embeddings, was the best performing classifier. In light of these findings, it can be concluded that incorporating sentence representations from SimCSE's sentence transformer only marginally improved the performance of machine learning models.
Collapse
Affiliation(s)
- Mpho Mokoatle
- Department of Computer Science, University of Pretoria, Pretoria, South Africa.
| | - Vukosi Marivate
- Department of Computer Science, University of Pretoria, Pretoria, South Africa
| | | | - Riana Bornman
- School of Health Systems and Public Health, University of Pretoria, Pretoria, South Africa
| | - Vanessa M Hayes
- School of Medical Sciences, The University of Sydney, Sydney, Australia
- School of Health Systems and Public Health, University of Pretoria, Pretoria, South Africa
| |
Collapse
|
31
|
Eminaga O, Abbas M, Shen J, Laurie M, Brooks JD, Liao JC, Rubin DL. PlexusNet: A neural network architectural concept for medical image classification. Comput Biol Med 2023; 154:106594. [PMID: 36753979 DOI: 10.1016/j.compbiomed.2023.106594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2022] [Revised: 01/12/2023] [Accepted: 01/22/2023] [Indexed: 01/27/2023]
Abstract
State-of-the-art (SOTA) convolutional neural network models have been widely adapted in medical imaging and applied to address different clinical problems. However, the complexity and scale of such models may not be justified in medical imaging and subject to the available resource budget. Further increasing the number of representative feature maps for the classification task decreases the model explainability. The current data normalization practice is fixed prior to model development and discounting the specification of the data domain. Acknowledging these issues, the current work proposed a new scalable model family called PlexusNet; the block architecture and model scaling by the network's depth, width, and branch regulate PlexusNet's architecture. The efficient computation costs outlined the dimensions of PlexusNet scaling and design. PlexusNet includes a new learnable data normalization algorithm for better data generalization. We applied a simple yet effective neural architecture search to design PlexusNet tailored to five clinical classification problems that achieve a performance noninferior to the SOTA models ResNet-18 and EfficientNet B0/1. It also does so with lower parameter capacity and representative feature maps in ten-fold ranges than the smallest SOTA models with comparable performance. The visualization of representative features revealed distinguishable clusters associated with categories based on latent features generated by PlexusNet. The package and source code are at https://github.com/oeminaga/PlexusNet.git.
Collapse
Affiliation(s)
- Okyaz Eminaga
- Center for Artificial Intelligence in Medicine & Imaging and Department of Urology, Stanford School of Medicine, Stanford, CA, 94305, USA; Department of Urology, Stanford School of Medicine, Stanford, CA, 94305, USA
| | - Mahmoud Abbas
- Department of Pathology, University of Muenster, Muenster, Germany.
| | - Jeanne Shen
- Department of Pathology, Stanford School of Medicine, Stanford, CA, 94305, USA.
| | - Mark Laurie
- Department of Computer Science, Stanford University, Stanford, CA, 94305, USA.
| | - James D Brooks
- Department of Urology, Stanford School of Medicine, Stanford, CA, 94305, USA.
| | - Joseph C Liao
- Department of Urology, Stanford School of Medicine, Stanford, CA, 94305, USA.
| | - Daniel L Rubin
- Department of Biomedical Data Science, Stanford School of Medicine, Stanford, CA, 94305, USA.
| |
Collapse
|
32
|
EBHI: A new Enteroscope Biopsy Histopathological H&E Image Dataset for image classification evaluation. Phys Med 2023; 107:102534. [PMID: 36804696 DOI: 10.1016/j.ejmp.2023.102534] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/07/2022] [Revised: 08/30/2022] [Accepted: 01/25/2023] [Indexed: 02/19/2023] Open
Abstract
BACKGROUND AND PURPOSE Colorectal cancer has become the third most common cancer worldwide, accounting for approximately 10% of cancer patients. Early detection of the disease is important for the treatment of colorectal cancer patients. Histopathological examination is the gold standard for screening colorectal cancer. However, the current lack of histopathological image datasets of colorectal cancer, especially enteroscope biopsies, hinders the accurate evaluation of computer-aided diagnosis techniques. Therefore, a multi-category colorectal cancer dataset is needed to test various medical image classification methods to find high classification accuracy and strong robustness. METHODS A new publicly available Enteroscope Biopsy Histopathological H&E Image Dataset (EBHI) is published in this paper. To demonstrate the effectiveness of the EBHI dataset, we have utilized several machine learning, convolutional neural networks and novel transformer-based classifiers for experimentation and evaluation, using an image with a magnification of 200×. RESULTS Experimental results show that the deep learning method performs well on the EBHI dataset. Classical machine learning methods achieve maximum accuracy of 76.02% and deep learning method achieves a maximum accuracy of 95.37%. CONCLUSION To the best of our knowledge, EBHI is the first publicly available colorectal histopathology enteroscope biopsy dataset with four magnifications and five types of images of tumor differentiation stages, totaling 5532 images. We believe that EBHI could attract researchers to explore new classification algorithms for the automated diagnosis of colorectal cancer, which could help physicians and patients in clinical settings.
Collapse
|
33
|
Ryou H, Sirinukunwattana K, Aberdeen A, Grindstaff G, Stolz BJ, Byrne H, Harrington HA, Sousos N, Godfrey AL, Harrison CN, Psaila B, Mead AJ, Rees G, Turner GDH, Rittscher J, Royston D. Continuous Indexing of Fibrosis (CIF): improving the assessment and classification of MPN patients. Leukemia 2023; 37:348-358. [PMID: 36470992 PMCID: PMC9898027 DOI: 10.1038/s41375-022-01773-0] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Revised: 11/17/2022] [Accepted: 11/21/2022] [Indexed: 12/09/2022]
Abstract
The grading of fibrosis in myeloproliferative neoplasms (MPN) is an important component of disease classification, prognostication and monitoring. However, current fibrosis grading systems are only semi-quantitative and fail to fully capture sample heterogeneity. To improve the quantitation of reticulin fibrosis, we developed a machine learning approach using bone marrow trephine (BMT) samples (n = 107) from patients diagnosed with MPN or a reactive marrow. The resulting Continuous Indexing of Fibrosis (CIF) enhances the detection and monitoring of fibrosis within BMTs, and aids MPN subtyping. When combined with megakaryocyte feature analysis, CIF discriminates between the frequently challenging differential diagnosis of essential thrombocythemia (ET) and pre-fibrotic myelofibrosis with high predictive accuracy [area under the curve = 0.94]. CIF also shows promise in the identification of MPN patients at risk of disease progression; analysis of samples from 35 patients diagnosed with ET and enrolled in the Primary Thrombocythemia-1 trial identified features predictive of post-ET myelofibrosis (area under the curve = 0.77). In addition to these clinical applications, automated analysis of fibrosis has clear potential to further refine disease classification boundaries and inform future studies of the micro-environmental factors driving disease initiation and progression in MPN and other stem cell disorders.
Collapse
Affiliation(s)
- Hosuk Ryou
- Nuffield Division of Clinical Laboratory Sciences, Radcliffe Department of Medicine, University of Oxford, Oxford, UK
| | - Korsuk Sirinukunwattana
- Institute of Biomedical Engineering (IBME), Department of Engineering Science, University of Oxford, Oxford, UK
- Big Data Institute/Li Ka Shing Centre for Health Information and Discovery, University of Oxford, Oxford, UK
- Ground Truth Labs, Oxford, UK
- Oxford NIHR Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | | | - Gillian Grindstaff
- Department of Mathematics, University of California, Los Angeles, CA, USA
| | - Bernadette J Stolz
- Mathematical Institute, University of Oxford, Oxford, UK
- Laboratory for Topology and Neuroscience, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Helen Byrne
- Mathematical Institute, University of Oxford, Oxford, UK
- Ludwig Institute for Cancer Research, University of Oxford, Oxford, UK
| | - Heather A Harrington
- Mathematical Institute, University of Oxford, Oxford, UK
- Wellcome Centre for Human Genetics, University of Oxford, Oxford, UK
| | - Nikolaos Sousos
- Oxford NIHR Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- Department of Haematology, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- MRC Weatherall Institute of Molecular Medicine, Radcliffe Department of Medicine, University of Oxford, Oxford, UK
| | - Anna L Godfrey
- Haematopathology & Oncology Diagnostics Service, Cambridge University Hospitals NHS Foundation Trust, Cambridge, UK
| | - Claire N Harrison
- Department of Haematology, Guy's and St Thomas' NHS Foundation Trust, London, UK
| | - Bethan Psaila
- Oxford NIHR Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- Department of Haematology, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- MRC Weatherall Institute of Molecular Medicine, Radcliffe Department of Medicine, University of Oxford, Oxford, UK
| | - Adam J Mead
- Oxford NIHR Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- Department of Haematology, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- MRC Weatherall Institute of Molecular Medicine, Radcliffe Department of Medicine, University of Oxford, Oxford, UK
| | - Gabrielle Rees
- Department of Pathology, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Gareth D H Turner
- Department of Pathology, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Jens Rittscher
- Institute of Biomedical Engineering (IBME), Department of Engineering Science, University of Oxford, Oxford, UK
- Big Data Institute/Li Ka Shing Centre for Health Information and Discovery, University of Oxford, Oxford, UK
- Ground Truth Labs, Oxford, UK
- Oxford NIHR Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
- Ludwig Institute for Cancer Research, University of Oxford, Oxford, UK
| | - Daniel Royston
- Nuffield Division of Clinical Laboratory Sciences, Radcliffe Department of Medicine, University of Oxford, Oxford, UK.
- Department of Pathology, Oxford University Hospitals NHS Foundation Trust, Oxford, UK.
| |
Collapse
|
34
|
Zhang H, He Y, Wu X, Huang P, Qin W, Wang F, Ye J, Huang X, Liao Y, Chen H, Guo L, Shi X, Luo L. PathNarratives: Data annotation for pathological human-AI collaborative diagnosis. Front Med (Lausanne) 2023; 9:1070072. [PMID: 36777158 PMCID: PMC9908590 DOI: 10.3389/fmed.2022.1070072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2022] [Accepted: 12/22/2022] [Indexed: 01/27/2023] Open
Abstract
Pathology is the gold standard of clinical diagnosis. Artificial intelligence (AI) in pathology becomes a new trend, but it is still not widely used due to the lack of necessary explanations for pathologists to understand the rationale. Clinic-compliant explanations besides the diagnostic decision of pathological images are essential for AI model training to provide diagnostic suggestions assisting pathologists practice. In this study, we propose a new annotation form, PathNarratives, that includes a hierarchical decision-to-reason data structure, a narrative annotation process, and a multimodal interactive annotation tool. Following PathNarratives, we recruited 8 pathologist annotators to build a colorectal pathological dataset, CR-PathNarratives, containing 174 whole-slide images (WSIs). We further experiment on the dataset with classification and captioning tasks to explore the clinical scenarios of human-AI-collaborative pathological diagnosis. The classification tasks show that fine-grain prediction enhances the overall classification accuracy from 79.56 to 85.26%. In Human-AI collaboration experience, the trust and confidence scores from 8 pathologists raised from 3.88 to 4.63 with providing more details. Results show that the classification and captioning tasks achieve better results with reason labels, provide explainable clues for doctors to understand and make the final decision and thus can support a better experience of human-AI collaboration in pathological diagnosis. In the future, we plan to optimize the tools for the annotation process, and expand the datasets with more WSIs and covering more pathological domains.
Collapse
Affiliation(s)
- Heyu Zhang
- College of Engineering, Peking University, Beijing, China
| | - Yan He
- Department of Pathology, Longgang Central Hospital of Shenzhen, Shenzhen, China
| | - Xiaomin Wu
- College of Engineering, Peking University, Beijing, China
| | - Peixiang Huang
- College of Engineering, Peking University, Beijing, China
| | - Wenkang Qin
- College of Engineering, Peking University, Beijing, China
| | - Fan Wang
- College of Engineering, Peking University, Beijing, China
| | - Juxiang Ye
- Department of Pathology, School of Basic Medical Science, Peking University Health Science Center, Peking University Third Hospital, Beijing, China
| | - Xirui Huang
- Department of Pathology, Longgang Central Hospital of Shenzhen, Shenzhen, China
| | - Yanfang Liao
- Department of Pathology, Longgang Central Hospital of Shenzhen, Shenzhen, China
| | - Hang Chen
- College of Engineering, Peking University, Beijing, China
| | - Limei Guo
- Department of Pathology, School of Basic Medical Science, Peking University Health Science Center, Peking University Third Hospital, Beijing, China,*Correspondence: Limei Guo,
| | - Xueying Shi
- Department of Pathology, School of Basic Medical Science, Peking University Health Science Center, Peking University Third Hospital, Beijing, China,Xueying Shi,
| | - Lin Luo
- College of Engineering, Peking University, Beijing, China,Lin Luo,
| |
Collapse
|
35
|
Wang Z, Lu H, Wu Y, Ren S, Diaty DM, Fu Y, Zou Y, Zhang L, Wang Z, Wang F, Li S, Huo X, Yu W, Xu J, Ye Z. Predicting recurrence in osteosarcoma via a quantitative histological image classifier derived from tumour nuclear morphological features. CAAI TRANSACTIONS ON INTELLIGENCE TECHNOLOGY 2023. [DOI: 10.1049/cit2.12175] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023] Open
Affiliation(s)
- Zhan Wang
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| | - Haoda Lu
- Institute for AI in Medicine School of Artificial Intelligence, Nanjing University of Information Science & Technology Nanjing China
| | - Yan Wu
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| | - Shihong Ren
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Diarra mohamed Diaty
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| | - Yanbiao Fu
- Department of Pathology The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Yi Zou
- Department of Pathology The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Lingling Zhang
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| | - Zenan Wang
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Fangqian Wang
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
| | - Shu Li
- Department of Hematology Shanghai General Hospital Shanghai Jiao Tong University School of Medicine Shanghai China
| | - Xinmi Huo
- Bioinformatics Institute (BII) Agency for Science, Technology and Research (A*STAR) Singapore Singapore
| | - Weimiao Yu
- Bioinformatics Institute (BII) Agency for Science, Technology and Research (A*STAR) Singapore Singapore
| | - Jun Xu
- Institute for AI in Medicine School of Artificial Intelligence, Nanjing University of Information Science & Technology Nanjing China
| | - Zhaoming Ye
- Department of Orthopedic Surgery The Second Affiliated Hospital Zhejiang University School of Medicine Hangzhou China
- Orthopedics Research Institute of Zhejiang University Hangzhou China
- Key Laboratory of Motor System Disease Research and Precision Therapy of Zhejiang Province Hangzhou China
- Clinical Research Center of Motor System Disease of Zhejiang Province Hangzhou China
| |
Collapse
|
36
|
Kumar A, Vishwakarma A, Bajaj V. CRCCN-Net: Automated framework for classification of colorectal tissue using histopathological images. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104172] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/14/2022]
|
37
|
Graham S, Vu QD, Jahanifar M, Raza SEA, Minhas F, Snead D, Rajpoot N. One model is all you need: Multi-task learning enables simultaneous histology image segmentation and classification. Med Image Anal 2023; 83:102685. [PMID: 36410209 DOI: 10.1016/j.media.2022.102685] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Revised: 10/20/2022] [Accepted: 11/03/2022] [Indexed: 11/13/2022]
Abstract
The recent surge in performance for image analysis of digitised pathology slides can largely be attributed to the advances in deep learning. Deep models can be used to initially localise various structures in the tissue and hence facilitate the extraction of interpretable features for biomarker discovery. However, these models are typically trained for a single task and therefore scale poorly as we wish to adapt the model for an increasing number of different tasks. Also, supervised deep learning models are very data hungry and therefore rely on large amounts of training data to perform well. In this paper, we present a multi-task learning approach for segmentation and classification of nuclei, glands, lumina and different tissue regions that leverages data from multiple independent data sources. While ensuring that our tasks are aligned by the same tissue type and resolution, we enable meaningful simultaneous prediction with a single network. As a result of feature sharing, we also show that the learned representation can be used to improve the performance of additional tasks via transfer learning, including nuclear classification and signet ring cell detection. As part of this work, we train our developed Cerberus model on a huge amount of data, consisting of over 600 thousand objects for segmentation and 440 thousand patches for classification. We use our approach to process 599 colorectal whole-slide images from TCGA, where we localise 377 million, 900 thousand and 2.1 million nuclei, glands and lumina respectively. We make this resource available to remove a major barrier in the development of explainable models for computational pathology.
Collapse
Affiliation(s)
- Simon Graham
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom; Histofy Ltd, United Kingdom.
| | - Quoc Dang Vu
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
| | - Mostafa Jahanifar
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
| | - Shan E Ahmed Raza
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
| | - Fayyaz Minhas
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom
| | - David Snead
- Histofy Ltd, United Kingdom; Department of Pathology, University Hospitals Coventry & Warwickshire, United Kingdom
| | - Nasir Rajpoot
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, United Kingdom; Histofy Ltd, United Kingdom; Department of Pathology, University Hospitals Coventry & Warwickshire, United Kingdom
| |
Collapse
|
38
|
Ben Hamida A, Devanne M, Weber J, Truntzer C, Derangère V, Ghiringhelli F, Forestier G, Wemmert C. Weakly Supervised Learning using Attention gates for colon cancer histopathological image segmentation. Artif Intell Med 2022; 133:102407. [PMID: 36328667 DOI: 10.1016/j.artmed.2022.102407] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2022] [Revised: 09/07/2022] [Accepted: 09/15/2022] [Indexed: 02/08/2023]
Abstract
Recently, Artificial Intelligence namely Deep Learning methods have revolutionized a wide range of domains and applications. Besides, Digital Pathology has so far played a major role in the diagnosis and the prognosis of tumors. However, the characteristics of the Whole Slide Images namely the gigapixel size, high resolution and the shortage of richly labeled samples have hindered the efficiency of classical Machine Learning methods. That goes without saying that traditional methods are poor in generalization to different tasks and data contents. Regarding the success of Deep learning when dealing with Large Scale applications, we have resorted to the use of such models for histopathological image segmentation tasks. First, we review and compare the classical UNet and Att-UNet models for colon cancer WSI segmentation in a sparsely annotated data scenario. Then, we introduce novel enhanced models of the Att-UNet where different schemes are proposed for the skip connections and spatial attention gates positions in the network. In fact, spatial attention gates assist the training process and enable the model to avoid irrelevant feature learning. Alternating the presence of such modules namely in our Alter-AttUNet model adds robustness and ensures better image segmentation results. In order to cope with the lack of richly annotated data in our AiCOLO colon cancer dataset, we suggest the use of a multi-step training strategy that also deals with the WSI sparse annotations and unbalanced class issues. All proposed methods outperform state-of-the-art approaches but Alter-AttUNet generates the best compromise between accurate results and light network. The model achieves 95.88% accuracy with our sparse AiCOLO colon cancer datasets. Finally, to evaluate and validate our proposed architectures we resort to publicly available WSI data: the NCT-CRC-HE-100K, the CRC-5000 and the Warwick colon cancer histopathological dataset. Respective accuracies of 99.65%, 99.73% and 79.03% were reached. A comparison with state-of-art approaches is established to view and compare the key solutions for histopathological image segmentation.
Collapse
Affiliation(s)
| | - M Devanne
- IRIMAS, University of Haute-Alsace, France
| | - J Weber
- IRIMAS, University of Haute-Alsace, France
| | - C Truntzer
- Platform of Transform in Biological Oncology, Dijon, France
| | - V Derangère
- Platform of Transform in Biological Oncology, Dijon, France
| | - F Ghiringhelli
- Platform of Transform in Biological Oncology, Dijon, France
| | | | - C Wemmert
- ICube, University of Strasbourg, France
| |
Collapse
|
39
|
Zhang Z, Li X, Sun H. Development of machine learning models integrating PET/CT radiomic and immunohistochemical pathomic features for treatment strategy choice of cervical cancer with negative pelvic lymph node by mediating COX-2 expression. Front Physiol 2022; 13:994304. [PMID: 36311222 PMCID: PMC9614332 DOI: 10.3389/fphys.2022.994304] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2022] [Accepted: 10/03/2022] [Indexed: 11/18/2022] Open
Abstract
Objectives: We aimed to establish machine learning models based on texture analysis predicting pelvic lymph node metastasis (PLNM) and expression of cyclooxygenase-2 (COX-2) in cervical cancer with PET/CT negative pelvic lymph node (PLN). Methods: Eight hundred and thirty-seven texture features were extracted from PET/CT images of 148 early-stage cervical cancer patients with negative PLN. The machine learning models were established by logistic regression from selected features and evaluated by the area under the curve (AUC). The correlation of selected PET/CT texture features predicting PLNM or COX-2 expression and the corresponding immunohistochemical (IHC) texture features was analyzed by the Spearman test. Results: Fourteen texture features were reserved to calculate the Rad-score for PLNM and COX-2. The PLNM model predicting PLNM showed good prediction accuracy in the training and testing dataset (AUC = 0.817, p < 0.001; AUC = 0.786, p < 0.001, respectively). The COX-2 model also behaved well for predicting COX-2 expression levels in the training and testing dataset (AUC = 0.814, p < 0.001; AUC = 0.748, p = 0.001). The wavelet-LHH-GLCM ClusterShade of the PET image selected to predict PLNM was slightly correlated with the corresponding feature of the IHC image (r = −0.165, p < 0.05). There was a weak correlation of wavelet-LLL-GLRLM LongRunEmphasis of the PET image selected to predict COX-2 correlated with the corresponding feature of the IHC image (r = 0.238, p < 0.05). The correlation between PET image selected to predict COX-2 and the corresponding feature of the IHC image based on wavelet-LLL-GLRLM LongRunEmphasis is considered weak positive (r = 0.238, p=<0.05). Conclusion: This study underlined the significant application of the machine learning models based on PET/CT texture analysis for predicting PLNM and COX-2 expression, which could be a novel tool to assist the clinical management of cervical cancer with negative PLN on PET/CT images.
Collapse
|
40
|
Schwarz Schuler JP, Also SR, Puig D, Rashwan H, Abdel-Nasser M. An Enhanced Scheme for Reducing the Complexity of Pointwise Convolutions in CNNs for Image Classification Based on Interleaved Grouped Filters without Divisibility Constraints. ENTROPY (BASEL, SWITZERLAND) 2022; 24:1264. [PMID: 36141151 PMCID: PMC9497893 DOI: 10.3390/e24091264] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/24/2022] [Revised: 09/01/2022] [Accepted: 09/05/2022] [Indexed: 06/16/2023]
Abstract
In image classification with Deep Convolutional Neural Networks (DCNNs), the number of parameters in pointwise convolutions rapidly grows due to the multiplication of the number of filters by the number of input channels that come from the previous layer. Existing studies demonstrated that a subnetwork can replace pointwise convolutional layers with significantly fewer parameters and fewer floating-point computations, while maintaining the learning capacity. In this paper, we propose an improved scheme for reducing the complexity of pointwise convolutions in DCNNs for image classification based on interleaved grouped filters without divisibility constraints. The proposed scheme utilizes grouped pointwise convolutions, in which each group processes a fraction of the input channels. It requires a number of channels per group as a hyperparameter Ch. The subnetwork of the proposed scheme contains two consecutive convolutional layers K and L, connected by an interleaving layer in the middle, and summed at the end. The number of groups of filters and filters per group for layers K and L is determined by exact divisions of the original number of input channels and filters by Ch. If the divisions were not exact, the original layer could not be substituted. In this paper, we refine the previous algorithm so that input channels are replicated and groups can have different numbers of filters to cope with non exact divisibility situations. Thus, the proposed scheme further reduces the number of floating-point computations (11%) and trainable parameters (10%) achieved by the previous method. We tested our optimization on an EfficientNet-B0 as a baseline architecture and made classification tests on the CIFAR-10, Colorectal Cancer Histology, and Malaria datasets. For each dataset, our optimization achieves a saving of 76%, 89%, and 91% of the number of trainable parameters of EfficientNet-B0, while keeping its test classification accuracy.
Collapse
Affiliation(s)
- Joao Paulo Schwarz Schuler
- Departament d’Enginyeria Informatica i Matemátiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain
| | - Santiago Romani Also
- Departament d’Enginyeria Informatica i Matemátiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain
| | - Domenec Puig
- Departament d’Enginyeria Informatica i Matemátiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain
| | - Hatem Rashwan
- Departament d’Enginyeria Informatica i Matemátiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain
| | - Mohamed Abdel-Nasser
- Departament d’Enginyeria Informatica i Matemátiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain
- Electronics and Communication Engineering Section, Electrical Engineering Department, Aswan University, Aswan 81528, Egypt
| |
Collapse
|
41
|
Lu T, Jorns JM, Ye DH, Patton M, Fisher R, Emmrich A, Schmidt TG, Yen T, Yu B. Automated assessment of breast margins in deep ultraviolet fluorescence images using texture analysis. BIOMEDICAL OPTICS EXPRESS 2022; 13:5015-5034. [PMID: 36187258 PMCID: PMC9484420 DOI: 10.1364/boe.464547] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/23/2022] [Revised: 07/07/2022] [Accepted: 07/07/2022] [Indexed: 06/10/2023]
Abstract
Microscopy with ultraviolet surface excitation (MUSE) is increasingly studied for intraoperative assessment of tumor margins during breast-conserving surgery to reduce the re-excision rate. Here we report a two-step classification approach using texture analysis of MUSE images to automate the margin detection. A study dataset consisting of MUSE images from 66 human breast tissues was constructed for model training and validation. Features extracted using six texture analysis methods were investigated for tissue characterization, and a support vector machine was trained for binary classification of image patches within a full image based on selected feature subsets. A weighted majority voting strategy classified a sample as tumor or normal. Using the eight most predictive features ranked by the maximum relevance minimum redundancy and Laplacian scores methods has achieved a sample classification accuracy of 92.4% and 93.0%, respectively. Local binary pattern alone has achieved an accuracy of 90.3%.
Collapse
Affiliation(s)
- Tongtong Lu
- Department of Biomedical Engineering, Marquette University and Medical College of Wisconsin, Milwaukee, WI,
USA
| | - Julie M. Jorns
- Department of Pathology,
Medical College of Wisconsin, Milwaukee,
WI, USA
| | - Dong Hye Ye
- Department of Electrical and Computer
Engineering, Marquette University,
Milwaukee, WI, USA
| | - Mollie Patton
- Department of Pathology,
Medical College of Wisconsin, Milwaukee,
WI, USA
| | - Renee Fisher
- Department of Biomedical Engineering, Marquette University and Medical College of Wisconsin, Milwaukee, WI,
USA
- Currently with Ashfield, part of
UDG Healthcare, Dublin, Ireland
| | - Amanda Emmrich
- Department of Surgery, Medical
College of Wisconsin, Milwaukee, WI, USA
- Currently with DaVita Clinical
Research, Minneapolis, MN 55404, USA
| | - Taly Gilat Schmidt
- Department of Biomedical Engineering, Marquette University and Medical College of Wisconsin, Milwaukee, WI,
USA
| | - Tina Yen
- Department of Surgery, Medical
College of Wisconsin, Milwaukee, WI, USA
| | - Bing Yu
- Department of Biomedical Engineering, Marquette University and Medical College of Wisconsin, Milwaukee, WI,
USA
| |
Collapse
|
42
|
Nanni L, Brahnam S, Paci M, Ghidoni S. Comparison of Different Convolutional Neural Network Activation Functions and Methods for Building Ensembles for Small to Midsize Medical Data Sets. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22166129. [PMID: 36015898 PMCID: PMC9415767 DOI: 10.3390/s22166129] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 08/09/2022] [Accepted: 08/12/2022] [Indexed: 05/08/2023]
Abstract
CNNs and other deep learners are now state-of-the-art in medical imaging research. However, the small sample size of many medical data sets dampens performance and results in overfitting. In some medical areas, it is simply too labor-intensive and expensive to amass images numbering in the hundreds of thousands. Building Deep CNN ensembles of pre-trained CNNs is one powerful method for overcoming this problem. Ensembles combine the outputs of multiple classifiers to improve performance. This method relies on the introduction of diversity, which can be introduced on many levels in the classification workflow. A recent ensembling method that has shown promise is to vary the activation functions in a set of CNNs or within different layers of a single CNN. This study aims to examine the performance of both methods using a large set of twenty activations functions, six of which are presented here for the first time: 2D Mexican ReLU, TanELU, MeLU + GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU. The proposed method was tested on fifteen medical data sets representing various classification tasks. The best performing ensemble combined two well-known CNNs (VGG16 and ResNet50) whose standard ReLU activation layers were randomly replaced with another. Results demonstrate the superiority in performance of this approach.
Collapse
Affiliation(s)
- Loris Nanni
- Department of Information Engineering, University of Padua, Via Gradenigo 6, 35131 Padova, Italy
| | - Sheryl Brahnam
- Department of Information Technology and Cybersecurity, Missouri State University, 901 S. National Street, Springfield, MO 65804, USA
- Correspondence:
| | - Michelangelo Paci
- BioMediTech, Faculty of Medicine and Health Technology, Tampere University, Arvo Ylpön katu 34, D 219, FI-33520 Tampere, Finland
| | - Stefano Ghidoni
- Department of Information Engineering, University of Padua, Via Gradenigo 6, 35131 Padova, Italy
| |
Collapse
|
43
|
A convolution neural network with multi-level convolutional and attention learning for classification of cancer grades and tissue structures in colon histopathological images. Comput Biol Med 2022; 147:105680. [DOI: 10.1016/j.compbiomed.2022.105680] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2022] [Revised: 05/06/2022] [Accepted: 05/30/2022] [Indexed: 12/15/2022]
|
44
|
Jakab A, Patai ÁV, Micsik T. Digital image analysis provides robust tissue microenvironment-based prognosticators in stage I-IV colorectal cancer patients. Hum Pathol 2022; 128:141-151. [PMID: 35820451 DOI: 10.1016/j.humpath.2022.07.003] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/30/2022] [Revised: 06/03/2022] [Accepted: 07/02/2022] [Indexed: 11/26/2022]
Abstract
AIMS In colorectal cancer (CRC) patients, a promising marker is tumor-stroma ratio (TSR). Quantification issues highlight the importance of precise assessment that might be solved by artificial intelligence (AI)-based digital image analysis systems. Some alternatives have been offered so far, although these platforms are either proprietary developments or require additional programming skills. Our aim was to validate a user-friendly, commercially available software running in everyday computational environment to improve TSR assessment and also to compare the prognostic value of assessing TSR in three distinct regions of interests (ROIs), like hotspot, invasive front and whole tumor. Furthermore, we compared the prognostic power of TSR with newly suggested carcinoma percentage (CP) and carcinoma-stroma percentage (CSP). METHODS AND RESULTS Slides of 185 stage I-IV CRC patients with clinical follow up data were scanned and evaluated by a senior pathologist. A machine learning-based digital pathology software was trained to recognize tumoral and stromal compartments. The aforementioned parameters were evaluated in the hotspot, invasive front and whole tumor area, both visually and by machine learning. Patients were classified based on TSR, CP and CSP values. On multivariate analysis, TSR-hotspot was found to be an independent prognostic factor of overall survival (hazard ratio for TSR-hotspotsoftware: 2.005 (95% confidence interval (CI): 1.146-3.507), p=0.011, for TSR-hostpotvisual: 1.781 (CI: 1.060-2.992) p=0.029). Also, TSR was an independent predictor for distant metastasis and local relapse in most settings. Generally, software performance was comparable to visual evaluation and delivered reliable prognostication in more settings also with CP and CSP values. CONCLUSIONS This study presents that software assisted evaluation is a robust prognosticator. Our approach used a less sophisticated and thus easily accessible software without the aid of convolutional neural network; however, it was still effective enough to deliver reliable prognostic information.
Collapse
Affiliation(s)
- Anna Jakab
- Department of Pathology and Experimental Cancer Research, Semmelweis University, Budapest, Hungary, H-1085 Budapest, Üllői őt 26; Interdisciplinary Gastroenterology Working Group, Semmelweis University, Budapest, Hungary, H-1082, Üllői út 78.
| | - Árpád V Patai
- Interdisciplinary Gastroenterology Working Group, Semmelweis University, Budapest, Hungary, H-1082, Üllői út 78; Department of Surgery, Transplantation and Gastroenterology, Semmelweis University, Budapest, H-1082, Üllői út 78
| | - Tamás Micsik
- Department of Pathology and Experimental Cancer Research, Semmelweis University, Budapest, Hungary, H-1085 Budapest, Üllői őt 26; Interdisciplinary Gastroenterology Working Group, Semmelweis University, Budapest, Hungary, H-1082, Üllői út 78; Saint George Teaching Hospital of Fejér County, Székesfehérvár, Hungary, HU-8000, Seregélyesi út 3
| |
Collapse
|
45
|
Akcakır O, Celebi LK, Kamil M, Aly ASI. Automated wide-field malaria parasite infection detection using Fourier ptychography on stain-free thin-smears. BIOMEDICAL OPTICS EXPRESS 2022; 13:3904-3921. [PMID: 35991917 PMCID: PMC9352279 DOI: 10.1364/boe.448099] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Revised: 01/28/2022] [Accepted: 01/28/2022] [Indexed: 06/15/2023]
Abstract
Diagnosis of malaria in endemic areas is hampered by the lack of a rapid, stain-free and sensitive method to directly identify parasites in peripheral blood. Herein, we report the use of Fourier ptychography to generate wide-field high-resolution quantitative phase images of erythrocytes infected with malaria parasites, from a whole blood sample. We are able to image thousands of erythrocytes (red blood cells) in a single field of view and make a determination of infection status of the quantitative phase image of each segmented cell based on machine learning (random forest) and deep learning (VGG16) models. Our random forest model makes use of morphology and texture based features of the quantitative phase images. In order to label the quantitative images of the cells as either infected or uninfected before training the models, we make use of a Plasmodium berghei strain expressing GFP (green fluorescent protein) in all life cycle stages. By overlaying the fluorescence image with the quantitative phase image we could identify the infected subpopulation of erythrocytes for labelling purposes. Our machine learning model (random forest) achieved 91% specificity and 72% sensitivity while our deep learning model (VGG16) achieved 98% specificity and 57% sensitivity. These results highlight the potential for quantitative phase imaging coupled with artificial intelligence to develop an easy to use platform for the rapid and sensitive diagnosis of malaria.
Collapse
Affiliation(s)
- Osman Akcakır
- Beykoz Institute of Life Sciences and Biotechnology (BILSAB), Bezmialem Vakif University, 34820 Istanbul, Turkey
| | - Lutfi Kadir Celebi
- Beykoz Institute of Life Sciences and Biotechnology (BILSAB), Bezmialem Vakif University, 34820 Istanbul, Turkey
- Istanbul Technical University (ITU), Electronics and Communication Engineering Department, Biomedical Engineering Program, 34467 Istanbul, Turkey
| | - Mohd Kamil
- Beykoz Institute of Life Sciences and Biotechnology (BILSAB), Bezmialem Vakif University, 34820 Istanbul, Turkey
| | - Ahmed S. I. Aly
- Beykoz Institute of Life Sciences and Biotechnology (BILSAB), Bezmialem Vakif University, 34820 Istanbul, Turkey
| |
Collapse
|
46
|
Ray I, Raipuria G, Singhal N. Rethinking ImageNet Pre-training for Computational Histopathology. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:3059-3062. [PMID: 36086630 DOI: 10.1109/embc48229.2022.9871687] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Transfer learning from ImageNet pretrained weights is widely used when training Deep Learning models on a Histopathology dataset. However, the visual features of the two domains are different. Rather than ImageNet pretrained weights, pre-training on a Histopathology dataset may provide better initialization. To prove this hypothesis, we train two commonly used Deep Learning model architectures - ResNet and DenseNet on a complex Histopathology classification dataset, and compare transfer learning performance with ImageNet pretrained weights. Based on the fine-tuning on three histopathology datasets including two different stains (H&E and IHC), we show that the domain specific pretrained weights are better suited for transfer learning. This is reflected by higher performance, lower training time as well as better feature reuse. Clinical Relevance - The paper establishes merit of using Histopathology domain specific pretrained weights rather than ImageNet pretrained weights.
Collapse
|
47
|
Wang X, Yang S, Zhang J, Wang M, Zhang J, Yang W, Huang J, Han X. Transformer-based unsupervised contrastive learning for histopathological image classification. Med Image Anal 2022; 81:102559. [DOI: 10.1016/j.media.2022.102559] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2022] [Revised: 06/24/2022] [Accepted: 07/25/2022] [Indexed: 10/16/2022]
|
48
|
HunCRC: annotated pathological slides to enhance deep learning applications in colorectal cancer screening. Sci Data 2022; 9:370. [PMID: 35764660 PMCID: PMC9240013 DOI: 10.1038/s41597-022-01450-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2021] [Accepted: 06/06/2022] [Indexed: 11/28/2022] Open
Abstract
Histopathology is the gold standard method for staging and grading human tumors and provides critical information for the oncoteam’s decision making. Highly-trained pathologists are needed for careful microscopic analysis of the slides produced from tissue taken from biopsy. This is a time-consuming process. A reliable decision support system would assist healthcare systems that often suffer from a shortage of pathologists. Recent advances in digital pathology allow for high-resolution digitalization of pathological slides. Digital slide scanners combined with modern computer vision models, such as convolutional neural networks, can help pathologists in their everyday work, resulting in shortened diagnosis times. In this study, 200 digital whole-slide images are published which were collected via hematoxylin-eosin stained colorectal biopsy. Alongside the whole-slide images, detailed region level annotations are also provided for ten relevant pathological classes. The 200 digital slides, after pre-processing, resulted in 101,389 patches. A single patch is a 512 × 512 pixel image, covering 248 × 248 μm2 tissue area. Versions at higher resolution are available as well. Hopefully, HunCRC, this widely accessible dataset will aid future colorectal cancer computer-aided diagnosis and research. Measurement(s) | H&E slide staining • ex vivo light microscopy with immunohistochemistry and digital image analysis • Image Annotation Statement • Screening Colonoscopy | Technology Type(s) | Hematoxylin and Eosin Staining Method • bright-field microscopy • Observation • Biopsy of Colon | Factor Type(s) | screening status of colon cancer or normal tissue | Sample Characteristic - Organism | Homo sapiens | Sample Characteristic - Location | Central Hungary |
Collapse
|
49
|
Artificial Intelligence-Based Tissue Phenotyping in Colorectal Cancer Histopathology Using Visual and Semantic Features Aggregation. MATHEMATICS 2022. [DOI: 10.3390/math10111909] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
Abstract
Tissue phenotyping of the tumor microenvironment has a decisive role in digital profiling of intra-tumor heterogeneity, epigenetics, and progression of cancer. Most of the existing methods for tissue phenotyping often rely on time-consuming and error-prone manual procedures. Recently, with the advent of advanced technologies, these procedures have been automated using artificial intelligence techniques. In this paper, a novel deep histology heterogeneous feature aggregation network (HHFA-Net) is proposed based on visual and semantic information fusion for the detection of tissue phenotypes in colorectal cancer (CRC). We adopted and tested various data augmentation techniques to avoid computationally expensive stain normalization procedures and handle limited and imbalanced data problems. Three publicly available datasets are used in the experiments: CRC tissue phenotyping (CRC-TP), CRC histology (CRCH), and colon cancer histology (CCH). The proposed HHFA-Net achieves higher accuracies than the state-of-the-art methods for tissue phenotyping in CRC histopathology images.
Collapse
|
50
|
Abbet C, Studer L, Fischer A, Dawson H, Zlobec I, Bozorgtabar B, Thiran JP. Self-Rule to Multi-Adapt: Generalized Multi-source Feature Learning Using Unsupervised Domain Adaptation for Colorectal Cancer Tissue Detection. Med Image Anal 2022; 79:102473. [DOI: 10.1016/j.media.2022.102473] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2021] [Revised: 03/07/2022] [Accepted: 05/03/2022] [Indexed: 10/18/2022]
|