1
|
Tizhoosh H, Pantanowitz L. On image search in histopathology. J Pathol Inform 2024; 15:100375. [PMID: 38645985 PMCID: PMC11033156 DOI: 10.1016/j.jpi.2024.100375] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Revised: 03/21/2024] [Accepted: 03/29/2024] [Indexed: 04/23/2024] Open
Abstract
Pathology images of histopathology can be acquired from camera-mounted microscopes or whole-slide scanners. Utilizing similarity calculations to match patients based on these images holds significant potential in research and clinical contexts. Recent advancements in search technologies allow for implicit quantification of tissue morphology across diverse primary sites, facilitating comparisons, and enabling inferences about diagnosis, and potentially prognosis, and predictions for new patients when compared against a curated database of diagnosed and treated cases. In this article, we comprehensively review the latest developments in image search technologies for histopathology, offering a concise overview tailored for computational pathology researchers seeking effective, fast, and efficient image search methods in their work.
Collapse
Affiliation(s)
- H.R. Tizhoosh
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| | - Liron Pantanowitz
- Department of Pathology, School of Medicine, University of Pittsburgh, PA, USA
| |
Collapse
|
2
|
Hosseini MS, Bejnordi BE, Trinh VQH, Chan L, Hasan D, Li X, Yang S, Kim T, Zhang H, Wu T, Chinniah K, Maghsoudlou S, Zhang R, Zhu J, Khaki S, Buin A, Chaji F, Salehi A, Nguyen BN, Samaras D, Plataniotis KN. Computational pathology: A survey review and the way forward. J Pathol Inform 2024; 15:100357. [PMID: 38420608 PMCID: PMC10900832 DOI: 10.1016/j.jpi.2023.100357] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 12/21/2023] [Accepted: 12/23/2023] [Indexed: 03/02/2024] Open
Abstract
Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.
Collapse
Affiliation(s)
- Mahdi S Hosseini
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | | | - Vincent Quoc-Huy Trinh
- Institute for Research in Immunology and Cancer of the University of Montreal, Montreal, QC H3T 1J4, Canada
| | - Lyndon Chan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Danial Hasan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Xingwen Li
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Stephen Yang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Taehyo Kim
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Haochen Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Theodore Wu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Kajanan Chinniah
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Sina Maghsoudlou
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ryan Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Jiadai Zhu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Samir Khaki
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Andrei Buin
- Huron Digitial Pathology, St. Jacobs, ON N0B 2N0, Canada
| | - Fatemeh Chaji
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ala Salehi
- Department of Electrical and Computer Engineering, University of New Brunswick, Fredericton, NB E3B 5A3, Canada
| | - Bich Ngoc Nguyen
- University of Montreal Hospital Center, Montreal, QC H2X 0C2, Canada
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, NY 11794, United States
| | - Konstantinos N Plataniotis
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| |
Collapse
|
3
|
Ji J, Liu Y, Bao Y, Men Y, Hui Z. Network analysis of histopathological image features and genomics data improving prognosis performance in clear cell renal cell carcinoma. Urol Oncol 2024; 42:249.e1-249.e11. [PMID: 38653593 DOI: 10.1016/j.urolonc.2024.03.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Revised: 02/25/2024] [Accepted: 03/18/2024] [Indexed: 04/25/2024]
Abstract
INTRODUCTION Clear cell renal cell carcinoma is the most common type of kidney cancer, but the prediction of prognosis remains a challenge. METHODS We collected whole-slide histopathological images, corresponding clinical and genetic information from the The Cancer Imaging Archive and The Cancer Genome Atlas databases and randomly divided patients into training (n = 197) and validation (n = 84) cohorts. After feature extraction by CellProfiler, we used 2 different machine learning techniques (Least Absolute Shrinkage and Selector Operation-regularized Cox and Support Vector Machine-Recursive Feature Elimination) and weighted gene co-expression network analysis to select prognosis-related image features and genes, respectively. These features and genes were integrated into a joint model using random forest and used to create a nomogram that combines other predictive indicators. RESULTS A total of 4 overlapped features were identified, represented by the computed histopathological risk score in the random forest model, and showed predictive value for overall survival (test set: 1-year area under the curves (AUC) = 0.726, 3-year AUC = 0.727, and 5-year AUC = 0.764). The histopathological-genetic risk score (HGRS) integrating the genetic information computed performed better than the model that used image features only (test set: 1-year AUC = 0.682, 3-year AUC = 0.734, and 5-year AUC = 0.78). The nomogram (gender, stage, and HGRS) achieved the highest net benefit according to decision curve analysis compared to HGRS or clinical model. CONCLUSION This study developed a histopathological-genetic-related nomogram by combining histopathological features and clinical predictors, providing a more comprehensive prognostic assessment for clear cell renal cell carcinoma patients.
Collapse
Affiliation(s)
- Jianrui Ji
- Department of Radiation Oncology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Yunsong Liu
- Department of Radiation Oncology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Yongxing Bao
- Department of Radiation Oncology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Yu Men
- Department of Radiation Oncology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China; Department of VIP Medical Services, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Zhouguang Hui
- Department of Radiation Oncology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China; Department of VIP Medical Services, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China.
| |
Collapse
|
4
|
Elazab N, Gab Allah W, Elmogy M. Computer-aided diagnosis system for grading brain tumor using histopathology images based on color and texture features. BMC Med Imaging 2024; 24:177. [PMID: 39030508 PMCID: PMC11264763 DOI: 10.1186/s12880-024-01355-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2023] [Accepted: 07/03/2024] [Indexed: 07/21/2024] Open
Abstract
BACKGROUND Cancer pathology shows disease development and associated molecular features. It provides extensive phenotypic information that is cancer-predictive and has potential implications for planning treatment. Based on the exceptional performance of computational approaches in the field of digital pathogenic, the use of rich phenotypic information in digital pathology images has enabled us to identify low-level gliomas (LGG) from high-grade gliomas (HGG). Because the differences between the textures are so slight, utilizing just one feature or a small number of features produces poor categorization results. METHODS In this work, multiple feature extraction methods that can extract distinct features from the texture of histopathology image data are used to compare the classification outcomes. The successful feature extraction algorithms GLCM, LBP, multi-LBGLCM, GLRLM, color moment features, and RSHD have been chosen in this paper. LBP and GLCM algorithms are combined to create LBGLCM. The LBGLCM feature extraction approach is extended in this study to multiple scales using an image pyramid, which is defined by sampling the image both in space and scale. The preprocessing stage is first used to enhance the contrast of the images and remove noise and illumination effects. The feature extraction stage is then carried out to extract several important features (texture and color) from histopathology images. Third, the feature fusion and reduction step is put into practice to decrease the number of features that are processed, reducing the computation time of the suggested system. The classification stage is created at the end to categorize various brain cancer grades. We performed our analysis on the 821 whole-slide pathology images from glioma patients in the Cancer Genome Atlas (TCGA) dataset. Two types of brain cancer are included in the dataset: GBM and LGG (grades II and III). 506 GBM images and 315 LGG images are included in our analysis, guaranteeing representation of various tumor grades and histopathological features. RESULTS The fusion of textural and color characteristics was validated in the glioma patients using the 10-fold cross-validation technique with an accuracy equals to 95.8%, sensitivity equals to 96.4%, DSC equals to 96.7%, and specificity equals to 97.1%. The combination of the color and texture characteristics produced significantly better accuracy, which supported their synergistic significance in the predictive model. The result indicates that the textural characteristics can be an objective, accurate, and comprehensive glioma prediction when paired with conventional imagery. CONCLUSION The results outperform current approaches for identifying LGG from HGG and provide competitive performance in classifying four categories of glioma in the literature. The proposed model can help stratify patients in clinical studies, choose patients for targeted therapy, and customize specific treatment schedules.
Collapse
Affiliation(s)
- Naira Elazab
- Information Technology Department, Faculty of Computers and Information, Mansoura University, 35516, Mansoura, Egypt
| | - Wael Gab Allah
- Information Technology Department, Faculty of Computers and Information, Mansoura University, 35516, Mansoura, Egypt
| | - Mohammed Elmogy
- Information Technology Department, Faculty of Computers and Information, Mansoura University, 35516, Mansoura, Egypt.
| |
Collapse
|
5
|
Alsaafin A, Nejat P, Shafique A, Khan J, Alfasly S, Alabtah G, Tizhoosh HR. SPLICE: Streamlining Digital Pathology Image Processing. THE AMERICAN JOURNAL OF PATHOLOGY 2024:S0002-9440(24)00238-4. [PMID: 39032601 DOI: 10.1016/j.ajpath.2024.06.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/19/2024] [Revised: 06/06/2024] [Accepted: 06/18/2024] [Indexed: 07/23/2024]
Abstract
Digital pathology and the integration of artificial intelligence (AI) models have revolutionized histopathology, opening new opportunities. With the increasing availability of whole-slide images (WSIs), demand is growing for efficient retrieval, processing, and analysis of relevant images from vast biomedical archives. However, processing WSIs presents challenges due to their large size and content complexity. Full computer digestion of WSIs is impractical, and processing all patches individually is prohibitively expensive. In this article, we propose an unsupervised patching algorithm, Sequential Patching Lattice for Image Classification and Enquiry (SPLICE). This novel approach condenses a histopathology WSI into a compact set of representative patches, forming a collage of WSI while minimizing redundancy. SPLICE prioritizes patch quality and uniqueness by sequentially analyzing a WSI and selecting nonredundant representative features. We evaluated SPLICE for search and match applications, showing its improved accuracy, reduced computation time, and storage requirements compared with existing state-of-the-art methods. As an unsupervised method, SPLICE effectively reduces storage requirements for representing tissue images by 50%. This reduction enables numerous algorithms in computational pathology to operate much more efficiently, paving the way for accelerated adoption of digital pathology.
Collapse
Affiliation(s)
- Areej Alsaafin
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Peyman Nejat
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Abubakr Shafique
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Jibran Khan
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Saghir Alfasly
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Ghazal Alabtah
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota
| | - Hamid R Tizhoosh
- KIMIA Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota.
| |
Collapse
|
6
|
Wang Z, Ma J, Gao Q, Bain C, Imoto S, Liò P, Cai H, Chen H, Song J. Dual-stream multi-dependency graph neural network enables precise cancer survival analysis. Med Image Anal 2024; 97:103252. [PMID: 38963973 DOI: 10.1016/j.media.2024.103252] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Revised: 05/24/2024] [Accepted: 06/21/2024] [Indexed: 07/06/2024]
Abstract
Histopathology image-based survival prediction aims to provide a precise assessment of cancer prognosis and can inform personalized treatment decision-making in order to improve patient outcomes. However, existing methods cannot automatically model the complex correlations between numerous morphologically diverse patches in each whole slide image (WSI), thereby preventing them from achieving a more profound understanding and inference of the patient status. To address this, here we propose a novel deep learning framework, termed dual-stream multi-dependency graph neural network (DM-GNN), to enable precise cancer patient survival analysis. Specifically, DM-GNN is structured with the feature updating and global analysis branches to better model each WSI as two graphs based on morphological affinity and global co-activating dependencies. As these two dependencies depict each WSI from distinct but complementary perspectives, the two designed branches of DM-GNN can jointly achieve the multi-view modeling of complex correlations between the patches. Moreover, DM-GNN is also capable of boosting the utilization of dependency information during graph construction by introducing the affinity-guided attention recalibration module as the readout function. This novel module offers increased robustness against feature perturbation, thereby ensuring more reliable and stable predictions. Extensive benchmarking experiments on five TCGA datasets demonstrate that DM-GNN outperforms other state-of-the-art methods and offers interpretable prediction insights based on the morphological depiction of high-attention patches. Overall, DM-GNN represents a powerful and auxiliary tool for personalized cancer prognosis from histopathology images and has great potential to assist clinicians in making personalized treatment decisions and improving patient outcomes.
Collapse
Affiliation(s)
- Zhikang Wang
- Xiangya Hospital, Central South University, Changsha, China; Biomedicine Discovery Institute and Department of Biochemistry and Molecular Biology, Monash University, Melbourne, Australia; Wenzhou Medical University-Monash Biomedicine Discovery Institute (BDI) Alliance in Clinical and Experimental Biomedicine, Wenzhou, China
| | - Jiani Ma
- School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, China
| | - Qian Gao
- Xiangya Hospital, Central South University, Changsha, China
| | - Chris Bain
- Faculty of Information Technology, Monash University, Melbourne, Australia
| | - Seiya Imoto
- Human Genome Center, Institute of Medical Science, The University of Tokyo, Tokyo, Japan
| | - Pietro Liò
- Department of Computer Science and Technology, The University of Cambridge, Cambridge, United Kingdom
| | - Hongmin Cai
- School of Computer Science and Engineering, South China University of Technology, Guangzhou, China
| | - Hao Chen
- Department of Computer Science and Engineering and Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology, Hong Kong, China
| | - Jiangning Song
- Biomedicine Discovery Institute and Department of Biochemistry and Molecular Biology, Monash University, Melbourne, Australia; Wenzhou Medical University-Monash Biomedicine Discovery Institute (BDI) Alliance in Clinical and Experimental Biomedicine, Wenzhou, China.
| |
Collapse
|
7
|
Baheti B, Innani S, Nasrallah M, Bakas S. Prognostic stratification of glioblastoma patients by unsupervised clustering of morphology patterns on whole slide images furthering our disease understanding. Front Neurosci 2024; 18:1304191. [PMID: 38831756 PMCID: PMC11146603 DOI: 10.3389/fnins.2024.1304191] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2023] [Accepted: 04/25/2024] [Indexed: 06/05/2024] Open
Abstract
Introduction Glioblastoma (GBM) is a highly aggressive malignant tumor of the central nervous system that displays varying molecular and morphological profiles, leading to challenging prognostic assessments. Stratifying GBM patients according to overall survival (OS) from H&E-stained whole slide images (WSI) using advanced computational methods is challenging, but with direct clinical implications. Methods This work is focusing on GBM (IDH-wildtype, CNS WHO Gr.4) cases, identified from the TCGA-GBM and TCGA-LGG collections after considering the 2021 WHO classification criteria. The proposed approach starts with patch extraction in each WSI, followed by comprehensive patch-level curation to discard artifactual content, i.e., glass reflections, pen markings, dust on the slide, and tissue tearing. Each patch is then computationally described as a feature vector defined by a pre-trained VGG16 convolutional neural network. Principal component analysis provides a feature representation of reduced dimensionality, further facilitating identification of distinct groups of morphology patterns, via unsupervised k-means clustering. Results The optimal number of clusters, according to cluster reproducibility and separability, is automatically determined based on the rand index and silhouette coefficient, respectively. Our proposed approach achieved prognostic stratification accuracy of 83.33% on a multi-institutional independent unseen hold-out test set with sensitivity and specificity of 83.33%. Discussion We hypothesize that the quantification of these clusters of morphology patterns, reflect the tumor's spatial heterogeneity and yield prognostic relevant information to distinguish between short and long survivors using a decision tree classifier. The interpretability analysis of the obtained results can contribute to furthering and quantifying our understanding of GBM and potentially improving our diagnostic and prognostic predictions.
Collapse
Affiliation(s)
- Bhakti Baheti
- Division of Computational Pathology, Department of Pathology and Laboratory Medicine, Indiana University School of Medicine, Indianapolis, IN, United States
- Center for Artificial Intelligence and Data Science for Integrated Diagnostics (AI2D) and Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, United States
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
| | - Shubham Innani
- Division of Computational Pathology, Department of Pathology and Laboratory Medicine, Indiana University School of Medicine, Indianapolis, IN, United States
- Center for Artificial Intelligence and Data Science for Integrated Diagnostics (AI2D) and Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, United States
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
| | - MacLean Nasrallah
- Center for Artificial Intelligence and Data Science for Integrated Diagnostics (AI2D) and Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, United States
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
| | - Spyridon Bakas
- Division of Computational Pathology, Department of Pathology and Laboratory Medicine, Indiana University School of Medicine, Indianapolis, IN, United States
- Center for Artificial Intelligence and Data Science for Integrated Diagnostics (AI2D) and Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, United States
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
- Department of Computer Science, Luddy School of Informatics, Computing, and Engineering, Indiana University, Indianapolis, IN, United States
| |
Collapse
|
8
|
Li J, Liu H, Liu W, Zong P, Huang K, Li Z, Li H, Xiong T, Tian G, Li C, Yang J. Predicting gastric cancer tumor mutational burden from histopathological images using multimodal deep learning. Brief Funct Genomics 2024; 23:228-238. [PMID: 37525540 DOI: 10.1093/bfgp/elad032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Revised: 06/13/2023] [Accepted: 07/13/2023] [Indexed: 08/02/2023] Open
Abstract
Tumor mutational burden (TMB) is a significant predictive biomarker for selecting patients that may benefit from immune checkpoint inhibitor therapy. Whole exome sequencing is a common method for measuring TMB; however, its clinical application is limited by the high cost and time-consuming wet-laboratory experiments and bioinformatics analysis. To address this challenge, we downloaded multimodal data of 326 gastric cancer patients from The Cancer Genome Atlas, including histopathological images, clinical data and various molecular data. Using these data, we conducted a comprehensive analysis to investigate the relationship between TMB, clinical factors, gene expression and image features extracted from hematoxylin and eosin images. We further explored the feasibility of predicting TMB levels, i.e. high and low TMB, by utilizing a residual network (Resnet)-based deep learning algorithm for histopathological image analysis. Moreover, we developed a multimodal fusion deep learning model that combines histopathological images with omics data to predict TMB levels. We evaluated the performance of our models against various state-of-the-art methods using different TMB thresholds and obtained promising results. Specifically, our histopathological image analysis model achieved an area under curve (AUC) of 0.749. Notably, the multimodal fusion model significantly outperformed the model that relied only on histopathological images, with the highest AUC of 0.971. Our findings suggest that histopathological images could be used with reasonable accuracy to predict TMB levels in gastric cancer patients, while multimodal deep learning could achieve even higher levels of accuracy. This study sheds new light on predicting TMB in gastric cancer patients.
Collapse
Affiliation(s)
- Jing Li
- School of Mathematics and Statistics, Hainan Normal University, Haikou 571158, China
- Geneis Beijing Co., Ltd., Beijing 100102, China
| | - Haiyan Liu
- College of Information Engineering, Changsha Medical University, Changsha 410219, Hunan, China
| | - Wei Liu
- Department of Internal Medicine, Beijing Sanhuan Cancer Hospital, Beijing 100023, China
| | - Peijun Zong
- Department of Pathology, Yidu Central Hospital of Weifang, Shandong 262500, China
| | - Kaimei Huang
- Department of Mathematics, Zhejiang Normal University, Jinhua 321004, China
| | - Zibo Li
- Hunan Key Laboratory of the Research and Development of Novel Pharmaceutical Preparations, Changsha Medical University, Changsha 410219, China
| | - Haigang Li
- Hunan Key Laboratory of the Research and Development of Novel Pharmaceutical Preparations, Changsha Medical University, Changsha 410219, China
| | - Ting Xiong
- Department of Pharmacy, Changsha Medical University, Changsha 410219, Hunan, China
| | - Geng Tian
- Geneis Beijing Co., Ltd., Beijing 100102, China
| | - Chun Li
- School of Mathematics and Statistics, Hainan Normal University, Haikou 571158, China
| | | |
Collapse
|
9
|
Ghaderi S, Mohammadi S, Ghaderi K, Kiasat F, Mohammadi M. Marker-controlled watershed algorithm and fuzzy C-means clustering machine learning: automated segmentation of glioblastoma from MRI images in a case series. Ann Med Surg (Lond) 2024; 86:1460-1475. [PMID: 38463066 PMCID: PMC10923355 DOI: 10.1097/ms9.0000000000001756] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Accepted: 01/16/2024] [Indexed: 03/12/2024] Open
Abstract
Introduction and importance Automated segmentation of glioblastoma multiforme (GBM) from MRI images is crucial for accurate diagnosis and treatment planning. This paper presents a new and innovative approach for automating the segmentation of GBM from MRI images using the marker-controlled watershed segmentation (MCWS) algorithm. Case presentation and methods The technique involves several image processing techniques, including adaptive thresholding, morphological filtering, gradient magnitude calculation, and regional maxima identification. The MCWS algorithm efficiently segments images based on local intensity structures using the watershed transform, and fuzzy c-means (FCM) clustering improves segmentation accuracy. The presented approach achieved improved segmentation accuracy in detecting and segmenting GBM tumours from axial T2-weighted (T2-w) MRI images, as demonstrated by the mean characteristics performance metrics for GBM segmentation (sensitivity: 0.9905, specificity: 0.9483, accuracy: 0.9508, precision: 0.5481, F_measure: 0.7052, and jaccard: 0.9340). Clinical discussion The results of this study underline the importance of reliable and accurate image segmentation for effective diagnosis and treatment planning of GBM tumours. Conclusion The MCWS technique provides an effective and efficient approach for the segmentation of challenging medical images.
Collapse
Affiliation(s)
- Sadegh Ghaderi
- Department of Neuroscience and Addiction Studies, School of Advanced Technologies in Medicine, Tehran University of Medical Sciences, Tehran
| | - Sana Mohammadi
- Department of Medical Sciences, School of Medicine, Iran University of Medical Sciences, Tehran
| | - Kayvan Ghaderi
- Department of Information Technology and Computer Engineering, Faculty of Engineering, University of Kurdistan, Sanandaj
| | - Fereshteh Kiasat
- Department of Information Technology and Computer Engineering, Faculty of Engineering, University of Kurdistan, Sanandaj
| | - Mahdi Mohammadi
- Department of Medical Physics and Biomedical Engineering, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran
| |
Collapse
|
10
|
Elazab N, Gab-Allah WA, Elmogy M. A multi-class brain tumor grading system based on histopathological images using a hybrid YOLO and RESNET networks. Sci Rep 2024; 14:4584. [PMID: 38403597 PMCID: PMC10894864 DOI: 10.1038/s41598-024-54864-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2023] [Accepted: 02/17/2024] [Indexed: 02/27/2024] Open
Abstract
Gliomas are primary brain tumors caused by glial cells. These cancers' classification and grading are crucial for prognosis and treatment planning. Deep learning (DL) can potentially improve the digital pathology investigation of brain tumors. In this paper, we developed a technique for visualizing a predictive tumor grading model on histopathology pictures to help guide doctors by emphasizing characteristics and heterogeneity in forecasts. The proposed technique is a hybrid model based on YOLOv5 and ResNet50. The function of YOLOv5 is to localize and classify the tumor in large histopathological whole slide images (WSIs). The suggested technique incorporates ResNet into the feature extraction of the YOLOv5 framework, and the detection results show that our hybrid network is effective for identifying brain tumors from histopathological images. Next, we estimate the glioma grades using the extreme gradient boosting classifier. The high-dimensional characteristics and nonlinear interactions present in histopathology images are well-handled by this classifier. DL techniques have been used in previous computer-aided diagnosis systems for brain tumor diagnosis. However, by combining the YOLOv5 and ResNet50 architectures into a hybrid model specifically designed for accurate tumor localization and predictive grading within histopathological WSIs, our study presents a new approach that advances the field. By utilizing the advantages of both models, this creative integration goes beyond traditional techniques to produce improved tumor localization accuracy and thorough feature extraction. Additionally, our method ensures stable training dynamics and strong model performance by integrating ResNet50 into the YOLOv5 framework, addressing concerns about gradient explosion. The proposed technique is tested using the cancer genome atlas dataset. During the experiments, our model outperforms the other standard ways on the same dataset. Our results indicate that the proposed hybrid model substantially impacts tumor subtype discrimination between low-grade glioma (LGG) II and LGG III. With 97.2% of accuracy, 97.8% of precision, 98.6% of sensitivity, and the Dice similarity coefficient of 97%, the proposed model performs well in classifying four grades. These results outperform current approaches for identifying LGG from high-grade glioma and provide competitive performance in classifying four categories of glioma in the literature.
Collapse
Affiliation(s)
- Naira Elazab
- Information Technology Department, Faculty of Computers and Information, Mansoura University, Mansoura, 35516, Egypt
| | - Wael A Gab-Allah
- Information Technology Department, Faculty of Computers and Information, Mansoura University, Mansoura, 35516, Egypt
| | - Mohammed Elmogy
- Information Technology Department, Faculty of Computers and Information, Mansoura University, Mansoura, 35516, Egypt.
| |
Collapse
|
11
|
Tang Z, Liu L, Shen Y, Chen Z, Ma G, Dong J, Sun X, Zhang X, Li C, Zheng Q, Yang L. Explainable survival analysis with uncertainty using convolution-involved vision transformer. Comput Med Imaging Graph 2023; 110:102302. [PMID: 37839216 DOI: 10.1016/j.compmedimag.2023.102302] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2023] [Revised: 09/20/2023] [Accepted: 09/20/2023] [Indexed: 10/17/2023]
Abstract
Image-based precision medicine research is able to help doctors make better decisions on treatments. Among all kinds of medical images, a special form is called Whole Slide Image (WSI), which is used for diagnosing patients with cancer, aiming to enable more accurate survival prediction with its high resolution. However, One unique challenge of the WSI-based prediction models is processing the gigabyte-size or even terabyte-size WSIs, which would make most models computationally infeasible. Although existing models mostly use a pre-selected subset of key patches or patch clusters as input, they might discard some important morphology information, making the prediction inferior. Another challenge is improving the prediction models' explainability, which is crucial to help doctors understand the predictions given by the models and make faithful decisions with high confidence. To address the above two challenges, in this work, we propose a novel explainable survival prediction model based on Vision Transformer. Specifically, we adopt dual-channel convolutional layers to utilize the complete WSIs for more accurate predictions. We also introduce the aleatoric uncertainty into our model to understand its limitation and avoid overconfidence in using the prediction results. Additionally, we present a post-hoc explainable method to identify the most salient patches and distinct morphology features as supporting evidence for predictions. Evaluations of two large cancer datasets show that our proposed model is able to make survival predictions more effectively and has better explainability for cancer diagnosis.
Collapse
Affiliation(s)
- Zhihao Tang
- Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications, Beijing, China.
| | - Li Liu
- Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China.
| | - Yifan Shen
- Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications, Beijing, China.
| | - Zongyi Chen
- Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications, Beijing, China.
| | - Guixiang Ma
- Department of Computer Science, University of Illinois at Chicago, Chicago, USA.
| | - Jiyan Dong
- Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China.
| | - Xujie Sun
- Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China.
| | - Xi Zhang
- Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications, Beijing, China.
| | - Chaozhuo Li
- Microsoft Research Lab - Asia, Beijing, China.
| | - Qingfeng Zheng
- Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China.
| | - Lin Yang
- Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China.
| |
Collapse
|
12
|
Sun C, Luo T, Liu Z, Ge J, Shao L, Liu X, Li B, Zhang S, Qiu Q, Wei W, Wang S, Bian XW, Tian J. Tumor Mutation Burden-Related Histopathologic Features for Predicting Overall Survival in Gliomas Using Graph Deep Learning. THE AMERICAN JOURNAL OF PATHOLOGY 2023; 193:2111-2121. [PMID: 37741452 DOI: 10.1016/j.ajpath.2023.08.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Revised: 08/08/2023] [Accepted: 08/25/2023] [Indexed: 09/25/2023]
Abstract
Tumor mutation burden (TMB) is a potential biomarker for evaluating the prognosis and response to immune checkpoint inhibitors, but its costly and time-consuming method of measurement limits its widespread application. This study aimed to identify the TMB-related histopathologic features from hematoxylin and eosin slides and explore their prognostic value in gliomas. TMB-related features were detected using a graph convolutional neural network from whole-slide images of patients from The Cancer Genome Atlas data set (619 patients), and the correlation between features and TMB was evaluated in an external validation set (237 patients). TMB-related features were used for predicting overall survival (OS) of patients to investigate whether these features have potential for prognostic prediction. Moreover, biological pathways underlying the prognostic value of the features were further explored. Histopathologic features derived from whole-slide images were significantly associated with patient TMB (P = 0.007 in the external validation set). TMB-related features showed excellent performance for OS prediction, and patients with lower-grade gliomas could be further stratified into different risk groups according to the features (P = 0.00013; hazard ratio, 4.004). Pathways involved in the cell cycle and execution of immune response were enriched in patients with higher OS risk. The TMB-related features could be used to estimate TMB and aid in prognostic risk stratification of patients with glioma with dysregulated biological pathways.
Collapse
Affiliation(s)
- Caixia Sun
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, China; Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing; Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Tao Luo
- Institute of Pathology and Southwest Cancer Center, Southwest Hospital, Third Military Medical University (Army Medical University) and Key Laboratory of Tumor Immunopathology, Ministry of Education of China, Chongqing
| | - Zhenyu Liu
- Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing
| | - Jia Ge
- Institute of Pathology and Southwest Cancer Center, Southwest Hospital, Third Military Medical University (Army Medical University) and Key Laboratory of Tumor Immunopathology, Ministry of Education of China, Chongqing
| | - Lizhi Shao
- Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Xiangyu Liu
- Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China; Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi'an, China
| | - Bao Li
- Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Song Zhang
- Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Qi Qiu
- Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Wei Wei
- Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Shuo Wang
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, China; Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing; Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Xiu-Wu Bian
- Institute of Pathology and Southwest Cancer Center, Southwest Hospital, Third Military Medical University (Army Medical University) and Key Laboratory of Tumor Immunopathology, Ministry of Education of China, Chongqing.
| | - Jie Tian
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, China; Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing; Chinese Academy of Sciences Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, China; Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi'an, China.
| |
Collapse
|
13
|
Wang Z, Gao Q, Yi X, Zhang X, Zhang Y, Zhang D, Liò P, Bain C, Bassed R, Li S, Guo Y, Imoto S, Yao J, Daly RJ, Song J. Surformer: An interpretable pattern-perceptive survival transformer for cancer survival prediction from histopathology whole slide images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 241:107733. [PMID: 37572513 DOI: 10.1016/j.cmpb.2023.107733] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/21/2023] [Revised: 06/12/2023] [Accepted: 07/23/2023] [Indexed: 08/14/2023]
Abstract
BACKGROUND AND OBJECTIVE High-resolution histopathology whole slide images (WSIs) contain abundant valuable information for cancer prognosis. However, most computational pathology methods for survival prediction have weak interpretability and cannot explain the decision-making processes reasonably. To address this issue, we propose a highly interpretable neural network termed pattern-perceptive survival transformer (Surformer) for cancer survival prediction from WSIs. METHODS Notably, Surformer can quantify specific histological patterns through bag-level labels without any patch/cell-level auxiliary information. Specifically, the proposed ratio-reserved cross-attention module (RRCA) generates global and local features with the learnable prototypes (pglobal, plocals) as detectors and quantifies the patches correlative to each plocal in the form of ratio factors (rfs). Afterward, multi-head self&cross-attention modules proceed with the computation for feature enhancement against noise. Eventually, the designed disentangling loss function guides multiple local features to focus on distinct patterns, thereby assisting rfs from RRCA in achieving more explicit histological feature quantification. RESULTS Extensive experiments on five TCGA datasets illustrate that Surformer outperforms existing state-of-the-art methods. In addition, we highlight its interpretation by visualizing rfs distribution across high-risk and low-risk cohorts and retrieving and analyzing critical histological patterns contributing to the survival prediction. CONCLUSIONS Surformer is expected to be exploited as a useful tool for performing histopathology image data-driven analysis and gaining new insights for interpreting the associations between such images and patient survival states.
Collapse
Affiliation(s)
- Zhikang Wang
- Xiangya Hospital, Central South University, Changsha, Hunan, PR China; Biomedicine Discovery Institute and Department of Biochemistry and Molecular Biology, Monash University, Melbourne, Australia
| | - Qian Gao
- Xiangya Hospital, Central South University, Changsha, Hunan, PR China
| | - Xiaoping Yi
- Xiangya Hospital, Central South University, Changsha, Hunan, PR China
| | - Xinyu Zhang
- Faculty of Information Technology, Monash University, Melbourne, Australia
| | - Yiwen Zhang
- Climate, Air Quality Research Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Daokun Zhang
- Faculty of Information Technology, Monash University, Melbourne, Australia
| | - Pietro Liò
- Department of Computer Science and Technology, The University of Cambridge, Cambridge, United Kingdom
| | - Chris Bain
- Faculty of Information Technology, Monash University, Melbourne, Australia
| | - Richard Bassed
- Victorian Institute of Forensic Medicine, Melbourne, Australia
| | - Shanshan Li
- Climate, Air Quality Research Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Yuming Guo
- Climate, Air Quality Research Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Seiya Imoto
- Human Genome Center, The Institute of Medical Science, The University of Tokyo, Japan
| | - Jianhua Yao
- Tencent AI Lab, Tencent, Shenzhen, PR China.
| | - Roger J Daly
- Biomedicine Discovery Institute and Department of Biochemistry and Molecular Biology, Monash University, Melbourne, Australia.
| | - Jiangning Song
- Biomedicine Discovery Institute and Department of Biochemistry and Molecular Biology, Monash University, Melbourne, Australia.
| |
Collapse
|
14
|
Shanmugam K, Rajaguru H. Exploration and Enhancement of Classifiers in the Detection of Lung Cancer from Histopathological Images. Diagnostics (Basel) 2023; 13:3289. [PMID: 37892110 PMCID: PMC10606104 DOI: 10.3390/diagnostics13203289] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Revised: 10/20/2023] [Accepted: 10/21/2023] [Indexed: 10/29/2023] Open
Abstract
Lung cancer is a prevalent malignancy that impacts individuals of all genders and is often diagnosed late due to delayed symptoms. To catch it early, researchers are developing algorithms to study lung cancer images. The primary objective of this work is to propose a novel approach for the detection of lung cancer using histopathological images. In this work, the histopathological images underwent preprocessing, followed by segmentation using a modified approach of KFCM-based segmentation and the segmented image intensity values were dimensionally reduced using Particle Swarm Optimization (PSO) and Grey Wolf Optimization (GWO). Algorithms such as KL Divergence and Invasive Weed Optimization (IWO) are used for feature selection. Seven different classifiers such as SVM, KNN, Random Forest, Decision Tree, Softmax Discriminant, Multilayer Perceptron, and BLDC were used to analyze and classify the images as benign or malignant. Results were compared using standard metrics, and kappa analysis assessed classifier agreement. The Decision Tree Classifier with GWO feature extraction achieved good accuracy of 85.01% without feature selection and hyperparameter tuning approaches. Furthermore, we present a methodology to enhance the accuracy of the classifiers by employing hyperparameter tuning algorithms based on Adam and RAdam. By combining features from GWO and IWO, and using the RAdam algorithm, the Decision Tree classifier achieves the commendable accuracy of 91.57%.
Collapse
Affiliation(s)
| | - Harikumar Rajaguru
- Department of Electronics and Communication Engineering, Bannari Amman Institute of Technology, Sathyamangalam 638401, India;
| |
Collapse
|
15
|
Anjum S, Ahmed I, Asif M, Aljuaid H, Alturise F, Ghadi YY, Elhabob R. Lung Cancer Classification in Histopathology Images Using Multiresolution Efficient Nets. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2023; 2023:7282944. [PMID: 37876944 PMCID: PMC10593544 DOI: 10.1155/2023/7282944] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Revised: 11/07/2022] [Accepted: 11/29/2022] [Indexed: 10/26/2023]
Abstract
Histopathological images are very effective for investigating the status of various biological structures and diagnosing diseases like cancer. In addition, digital histopathology increases diagnosis precision and provides better image quality and more detail for the pathologist with multiple viewing options and team annotations. As a result of the benefits above, faster treatment is available, increasing therapy success rates and patient recovery and survival chances. However, the present manual examination of these images is tedious and time-consuming for pathologists. Therefore, reliable automated techniques are needed to effectively classify normal and malignant cancer images. This paper applied a deep learning approach, namely, EfficientNet and its variants from B0 to B7. We used different image resolutions for each model, from 224 × 224 pixels to 600 × 600 pixels. We also applied transfer learning and parameter tuning techniques to improve the results and overcome the overfitting problem. We collected the dataset from the Lung and Colon Cancer Histopathological Image LC25000 image dataset. The dataset acquisition consists of 25,000 histopathology images of five classes (lung adenocarcinoma, lung squamous cell carcinoma, benign lung tissue, colon adenocarcinoma, and colon benign tissue). Then, we performed preprocessing on the dataset to remove the noisy images and bring them into a standard format. The model's performance was evaluated in terms of classification accuracy and loss. We have achieved good accuracy results for all variants; however, the results of EfficientNetB2 stand excellent, with an accuracy of 97% for 260 × 260 pixels resolution images.
Collapse
Affiliation(s)
- Sunila Anjum
- Center of Excellence in Information Technology, Institute of Management Sciences, Hayatabad, Peshawar 25000, Pakistan
| | - Imran Ahmed
- School of Computing and Information Science, Anglia Ruskin University, Cambridge, UK
| | - Muhammad Asif
- Department of Computer Science, National Textile University, Faisalabad, Pakistan
| | - Hanan Aljuaid
- Computer Sciences Department, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University (PNU), P.O. Box 84428, Riyadh 11671, Saudi Arabia
| | - Fahad Alturise
- Department of Computer, College of Science and Arts in Ar Rass, Qassim University, Ar Rass, Qassim, Saudi Arabia
| | - Yazeed Yasin Ghadi
- Department of Software Engineering/Computer Science, Al Ain University, Al Ain, UAE
| | - Rashad Elhabob
- College of Computer Science and Information Technology, Karary University, Omdurman, Sudan
| |
Collapse
|
16
|
Mercaldo F, Brunese L, Martinelli F, Santone A, Cesarelli M. Explainable Convolutional Neural Networks for Brain Cancer Detection and Localisation. SENSORS (BASEL, SWITZERLAND) 2023; 23:7614. [PMID: 37688069 PMCID: PMC10490676 DOI: 10.3390/s23177614] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Revised: 08/07/2023] [Accepted: 08/31/2023] [Indexed: 09/10/2023]
Abstract
Brain cancer is widely recognised as one of the most aggressive types of tumors. In fact, approximately 70% of patients diagnosed with this malignant cancer do not survive. In this paper, we propose a method aimed to detect and localise brain cancer, starting from the analysis of magnetic resonance images. The proposed method exploits deep learning, in particular convolutional neural networks and class activation mapping, in order to provide explainability by highlighting the areas of the medical image related to brain cancer (from the model point of view). We evaluate the proposed method with 3000 magnetic resonances using a free available dataset. The results we obtained are encouraging. We reach an accuracy ranging from 97.83% to 99.67% in brain cancer detection by exploiting four different models: VGG16, ResNet50, Alex_Net, and MobileNet, thus showing the effectiveness of the proposed method.
Collapse
Affiliation(s)
- Francesco Mercaldo
- Department of Medicine and Health Sciences “Vincenzo Tiberio”, University of Molise, 86100 Campobasso, Italy; (L.B.); (A.S.)
- Institute for Informatics and Telematics, National Research Council of Italy, 56121 Pisa, Italy
| | - Luca Brunese
- Department of Medicine and Health Sciences “Vincenzo Tiberio”, University of Molise, 86100 Campobasso, Italy; (L.B.); (A.S.)
| | - Fabio Martinelli
- Institute for Informatics and Telematics, National Research Council of Italy, 56121 Pisa, Italy
| | - Antonella Santone
- Department of Medicine and Health Sciences “Vincenzo Tiberio”, University of Molise, 86100 Campobasso, Italy; (L.B.); (A.S.)
| | - Mario Cesarelli
- Department of Engineering, University of Sannio, 82100 Benevento, Italy;
| |
Collapse
|
17
|
Li S, Zhao Y, Zhang J, Yu T, Zhang J, Gao Y. High-Order Correlation-Guided Slide-Level Histology Retrieval With Self-Supervised Hashing. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:11008-11023. [PMID: 37097802 DOI: 10.1109/tpami.2023.3269810] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Histopathological Whole Slide Images (WSIs) play a crucial role in cancer diagnosis. It is of significant importance for pathologists to search for images sharing similar content with the query WSI, especially in the case-based diagnosis. While slide-level retrieval could be more intuitive and practical in clinical applications, most methods are designed for patch-level retrieval. A few recently unsupervised slide-level methods only focus on integrating patch features directly, without perceiving slide-level information, and thus severely limits the performance of WSI retrieval. To tackle the issue, we propose a High-Order Correlation-Guided Self-Supervised Hashing-Encoding Retrieval (HSHR) method. Specifically, we train an attention-based hash encoder with slide-level representation in a self-supervised manner, enabling it to generate more representative slide-level hash codes of cluster centers and assign weights for each. These optimized and weighted codes are leveraged to establish a similarity-based hypergraph, in which a hypergraph-guided retrieval module is adopted to explore high-order correlations in the multi-pairwise manifold to conduct WSI retrieval. Extensive experiments on multiple TCGA datasets with over 24,000 WSIs spanning 30 cancer subtypes demonstrate that HSHR achieves state-of-the-art performance compared with other unsupervised histology WSI retrieval methods.
Collapse
|
18
|
Deng C, Li D, Feng M, Han D, Huang Q. The value of deep neural networks in the pathological classification of thyroid tumors. Diagn Pathol 2023; 18:95. [PMID: 37598149 PMCID: PMC10439627 DOI: 10.1186/s13000-023-01380-2] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Accepted: 08/08/2023] [Indexed: 08/21/2023] Open
Abstract
BACKGROUND To explore the distinguishing diagnostic value and clinical application potential of deep neural networks (DNN) for pathological images of thyroid tumors. METHODS A total of 799 pathological thyroid images of 559 patients with thyroid tumors were retrospectively analyzed. The pathological types included papillary thyroid carcinoma (PTC), medullary thyroid carcinoma (MTC), follicular thyroid carcinoma (FTC), adenomatous goiter, adenoma, and normal thyroid gland. The dataset was divided into a training set and a test set. Resnet50, Resnext50, EfficientNet, and Densenet121 were trained using the training set data and tested with the test set data to determine the diagnostic efficiency of different pathology types and to further analyze the causes of misdiagnosis. RESULTS The recall, precision, negative predictive value (NPV), accuracy, specificity, and F1 scores of the four models ranged from 33.33% to 100.00%. The area under curve (AUC) ranged from 0.822 to 0.994, and the Kappa coefficient ranged from 0.7508 to 0.7713. However, the performance of diagnosing FTC, adenoma, and adenomatous goiter was slightly inferior to other types of pathological tissues. CONCLUSION The DNN model achieved satisfactory results in the task of classifying thyroid tumors by learning thyroid pathology images. These results indicate the potential of the DNN model for the efficient diagnosis of thyroid tumor histopathology.
Collapse
Affiliation(s)
- Chengwen Deng
- Department of Nuclear Medicine, Chaohu Hospital of Anhui Medical University, Heifei, 238000, China
| | - Dan Li
- Department of Nuclear Medicine, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, 510289, China
| | - Ming Feng
- Tongji University, Shanghai, 200082, China
| | - Dongyan Han
- Shanghai Tenth People's Hospital Tongji University, Shanghai, 200072, China
| | - Qingqing Huang
- Shanghai Key Laboratory of Molecular Imaging, Shanghai University of Medicine and Health Sciences, Shanghai, 201318, China.
| |
Collapse
|
19
|
Jose L, Liu S, Russo C, Cong C, Song Y, Rodriguez M, Di Ieva A. Artificial Intelligence-Assisted Classification of Gliomas Using Whole Slide Images. Arch Pathol Lab Med 2023; 147:916-924. [PMID: 36445697 DOI: 10.5858/arpa.2021-0518-oa] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/01/2022] [Indexed: 07/28/2023]
Abstract
CONTEXT.— Glioma is the most common primary brain tumor in adults. The diagnosis and grading of different pathological subtypes of glioma is essential in treatment planning and prognosis. OBJECTIVE.— To propose a deep learning-based approach for the automated classification of glioma histopathology images. Two classification methods, the ensemble method based on 2 binary classifiers and the multiclass method using a single multiclass classifier, were implemented to classify glioma images into astrocytoma, oligodendroglioma, and glioblastoma, according to the 5th edition of the World Health Organization classification of central nervous system tumors, published in 2021. DESIGN.— We tested 2 different deep neural network architectures (VGG19 and ResNet50) and extensively validated the proposed approach based on The Cancer Genome Atlas data set (n = 700). We also studied the effects of stain normalization and data augmentation on the glioma classification task. RESULTS.— With the binary classifiers, our model could distinguish astrocytoma and oligodendroglioma (combined) from glioblastoma with an accuracy of 0.917 (area under the curve [AUC] = 0.976) and astrocytoma from oligodendroglioma (accuracy = 0.821, AUC = 0.865). The multiclass method (accuracy = 0.861, AUC = 0.961) outperformed the ensemble method (accuracy = 0.847, AUC = 0.933) with the best performance displayed by the ResNet50 architecture. CONCLUSIONS.— With the high performance of our model (>80%), the proposed method can assist pathologists and physicians to support examination and differential diagnosis of glioma histopathology images, with the aim to expedite personalized medical care.
Collapse
Affiliation(s)
- Laya Jose
- From the Computational NeuroSurgery Lab (Jose, Liu, Russo, Di Ieva), Macquarie University, Sydney, Australia
| | - Sidong Liu
- From the Computational NeuroSurgery Lab (Jose, Liu, Russo, Di Ieva), Macquarie University, Sydney, Australia
- Australian Institute of Health Innovation, Centre for Health Informatics (Liu), Macquarie University, Sydney, Australia
| | - Carlo Russo
- From the Computational NeuroSurgery Lab (Jose, Liu, Russo, Di Ieva), Macquarie University, Sydney, Australia
| | - Cong Cong
- The School of Computer Science and Engineering, University of New South Wales, Sydney, Australia (Cong, Song)
| | - Yang Song
- The School of Computer Science and Engineering, University of New South Wales, Sydney, Australia (Cong, Song)
| | - Michael Rodriguez
- Macquarie Medical School (Rodriguez), Macquarie University, Sydney, Australia
| | - Antonio Di Ieva
- From the Computational NeuroSurgery Lab (Jose, Liu, Russo, Di Ieva), Macquarie University, Sydney, Australia
| |
Collapse
|
20
|
Xu H, Wu A, Ren H, Yu C, Liu G, Liu L. Classification of colorectal cancer consensus molecular subtypes using attention-based multi-instance learning network on whole-slide images. Acta Histochem 2023; 125:152057. [PMID: 37300984 DOI: 10.1016/j.acthis.2023.152057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2023] [Accepted: 06/01/2023] [Indexed: 06/12/2023]
Abstract
Colorectal cancer (CRC) is the third most common and second most lethal cancer globally. It is highly heterogeneous with different clinical-pathological characteristics, prognostic status, and therapy responses. Thus, the precise diagnosis of CRC subtypes is of great significance for improving the prognosis and survival of CRC patients. Nowadays, the most commonly used molecular-level CRC classification system is the Consensus Molecular Subtypes (CMSs). In this study, we applied a weakly supervised deep learning method, named attention-based multi-instance learning (MIL), on formalin-fixed paraffin-embedded (FFPE) whole-slide images (WSIs) to distinguish CMS1 subtype from CMS2, CMS3, and CMS4 subtypes, as well as distinguish CMS4 from CMS1, CMS2, and CMS3 subtypes. The advantage of MIL is training a bag of the tiled instance with bag-level labels only. Our experiment was performed on 1218 WSIs obtained from The Cancer Genome Atlas (TCGA). We constructed three convolutional neural network-based structures for model training and evaluated the ability of the max-pooling operator and mean-pooling operator on aggregating bag-level scores. The results showed that the 3-layer model achieved the best performance in both comparison groups. When compared CMS1 with CMS234, max-pooling reached the ACC of 83.86 % and the mean-pooling operator reached the AUC of 0.731. While comparing CMS4 with CMS123, mean-pooling reached the ACC of 74.26 % and max-pooling reached the AUC of 0.609. Our results implied that WSIs could be utilized to classify CMSs, and manual pixel-level annotation is not a necessity for computational pathology imaging analysis.
Collapse
Affiliation(s)
- Huilin Xu
- Institutes of Biomedical Sciences and Intelligent Medicine Institute, Fudan University, Shanghai 200032, China
| | - Aoshen Wu
- Institutes of Biomedical Sciences and Intelligent Medicine Institute, Fudan University, Shanghai 200032, China
| | - He Ren
- Faculty of Medical Instrumentation, Shanghai University of Medicine and Health Sciences, Shanghai 201318, China
| | - Chenghang Yu
- National Institute of Parasitic Diseases, Chinese Center for Disease Control and Prevention (Chinese Center for Tropical Diseases Research), Key Laboratory of Parasite and Vector Biology, National Health Commission of the People's Republic of China, WHO Collaborating Center for Tropical Diseases, Shanghai 200025, China
| | - Gang Liu
- Institutes of Biomedical Sciences and Intelligent Medicine Institute, Fudan University, Shanghai 200032, China.
| | - Lei Liu
- Institutes of Biomedical Sciences and Intelligent Medicine Institute, Fudan University, Shanghai 200032, China.
| |
Collapse
|
21
|
Hossain MS, Shahriar GM, Syeed MMM, Uddin MF, Hasan M, Shivam S, Advani S. Region of interest (ROI) selection using vision transformer for automatic analysis using whole slide images. Sci Rep 2023; 13:11314. [PMID: 37443188 PMCID: PMC10344922 DOI: 10.1038/s41598-023-38109-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Accepted: 07/03/2023] [Indexed: 07/15/2023] Open
Abstract
Selecting regions of interest (ROI) is a common step in medical image analysis across all imaging modalities. An ROI is a subset of an image appropriate for the intended analysis and identified manually by experts. In modern pathology, the analysis involves processing multidimensional and high resolution whole slide image (WSI) tiles automatically with an overwhelming quantity of structural and functional information. Despite recent improvements in computing capacity, analyzing such a plethora of data is challenging but vital to accurate analysis. Automatic ROI detection can significantly reduce the number of pixels to be processed, speed the analysis, improve accuracy and reduce dependency on pathologists. In this paper, we present an ROI detection method for WSI and demonstrated it for human epidermal growth factor receptor 2 (HER2) grading for breast cancer patients. Existing HER2 grading relies on manual ROI selection, which is tedious, time-consuming and suffers from inter-observer and intra-observer variability. This study found that the HER2 grade changes with ROI selection. We proposed an ROI detection method using Vision Transformer and investigated the role of image magnification for ROI detection. This method yielded an accuracy of 99% using 20 × WSI and 97% using 10 × WSI for the ROI detection. In the demonstration, the proposed method increased the diagnostic agreement to 99.3% with the clinical scores and reduced the time to 15 seconds for automated HER2 grading.
Collapse
Affiliation(s)
- Md Shakhawat Hossain
- Department of Computer Science and Engineering, Independent University Bangladesh, Dhaka, 1229, Bangladesh.
- RIoT Research Center, Independent University Bangladesh, Dhaka, 1229, Bangladesh.
| | | | - M M Mahbubul Syeed
- Department of Computer Science and Engineering, Independent University Bangladesh, Dhaka, 1229, Bangladesh
- RIoT Research Center, Independent University Bangladesh, Dhaka, 1229, Bangladesh
| | - Mohammad Faisal Uddin
- Department of Computer Science and Engineering, Independent University Bangladesh, Dhaka, 1229, Bangladesh
- RIoT Research Center, Independent University Bangladesh, Dhaka, 1229, Bangladesh
| | - Mahady Hasan
- Department of Computer Science and Engineering, Independent University Bangladesh, Dhaka, 1229, Bangladesh
- RIoT Research Center, Independent University Bangladesh, Dhaka, 1229, Bangladesh
| | - Shingla Shivam
- Department of Pathology, SL Raheja Hospital, Mumbai, 400016, India
| | - Suresh Advani
- Department of Pathology, SL Raheja Hospital, Mumbai, 400016, India
| |
Collapse
|
22
|
Shi L, Shen L, Jian J, Xia W, Yang KD, Tian Y, Huang J, Yuan B, Shen L, Liu Z, Zhang J, Zhang R, Wu K, Jing D, Gao X. Contribution of whole slide imaging-based deep learning in the assessment of intraoperative and postoperative sections in neuropathology. Brain Pathol 2023:e13160. [PMID: 37186490 DOI: 10.1111/bpa.13160] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2022] [Accepted: 04/04/2023] [Indexed: 05/17/2023] Open
Abstract
The pathological diagnosis of intracranial germinoma (IG), oligodendroglioma, and low-grade astrocytoma on intraoperative frozen section (IFS) and hematoxylin-eosin (HE)-staining section directly determines patients' treatment options, but it is a difficult task for pathologists. We aimed to investigate whether whole-slide imaging (WSI)-based deep learning can contribute new precision to the diagnosis of IG, oligodendroglioma, and low-grade astrocytoma. Two types of WSIs (500 IFSs and 832 HE-staining sections) were collected from 379 patients at multiple medical centers. Patients at Center 1 were split into the training, testing, and internal validation sets (3:1:1), while the other centers were the external validation sets. First, we subdivided WSIs into small tiles and selected tissue tiles using a tissue tile selection model. Then a tile-level classification model was established, and the majority voting method was used to determine the final diagnoses. Color jitter was applied to the tiles so that the deep learning (DL) models could adapt to the variations in the staining. Last, we investigated the effectiveness of model assistance. The internal validation accuracies of the IFS and HE models were 93.9% and 95.3%, respectively. The external validation accuracies of the IFS and HE models were 82.0% and 76.9%, respectively. Furthermore, the IFS and HE models can predict Ki-67 positive cell areas with R2 of 0.81 and 0.86, respectively. With model assistance, the IFS and HE diagnosis accuracy of pathologists improved from 54.6%-69.7% and 53.5%-83.7% to 87.9%-93.9% and 86.0%-90.7%, respectively. Both the IFS model and the HE model can differentiate the three tumors, predict the expression of Ki-67, and improve the diagnostic accuracy of pathologists. The use of our model can assist clinicians in providing patients with optimal and timely treatment options.
Collapse
Affiliation(s)
- Liting Shi
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
| | - Lin Shen
- Department of Oncology, National Clinical Research Center for Geriatric Disorders, Xiangya Hospital, Central South University, Changsha, China
| | - Junming Jian
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
| | - Wei Xia
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
- Jinan Guoke Medical Engineering and Technology Development Co., Ltd., Jinan, China
- Department of Radiology, Shanxi Province Cancer Hospital, Shanxi Medical University, Taiyuan, China
| | - Ke-Da Yang
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Yifu Tian
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Jianghai Huang
- Department of Pathology, the Second Xiangya Hospital, Central South University, Changsha, China
| | - Bowen Yuan
- Department of Pathology, The Third Xiangya Hospital, Central South University, Changsha, China
| | - Liangfang Shen
- Department of Oncology, National Clinical Research Center for Geriatric Disorders, Xiangya Hospital, Central South University, Changsha, China
| | - Zhengzheng Liu
- Department of Oncology, National Clinical Research Center for Geriatric Disorders, Xiangya Hospital, Central South University, Changsha, China
| | - Jiayi Zhang
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
- Jinan Guoke Medical Engineering and Technology Development Co., Ltd., Jinan, China
- Department of Radiology, Shanxi Province Cancer Hospital, Shanxi Medical University, Taiyuan, China
| | - Rui Zhang
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
| | - Keqing Wu
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
| | - Di Jing
- Department of Oncology, National Clinical Research Center for Geriatric Disorders, Xiangya Hospital, Central South University, Changsha, China
| | - Xin Gao
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
- Jinan Guoke Medical Engineering and Technology Development Co., Ltd., Jinan, China
- Department of Radiology, Shanxi Province Cancer Hospital, Shanxi Medical University, Taiyuan, China
| |
Collapse
|
23
|
Ozer E, Bilecen AE, Ozer NB, Yanikoglu B. Intraoperative cytological diagnosis of brain tumours: A preliminary study using a deep learning model. Cytopathology 2023; 34:113-119. [PMID: 36458464 DOI: 10.1111/cyt.13192] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2022] [Revised: 10/26/2022] [Accepted: 11/23/2022] [Indexed: 12/05/2022]
Abstract
BACKGROUND Intraoperative pathological diagnosis of central nervous system (CNS) tumours is essential to planning patient management in neuro-oncology. Frozen section slides and cytological preparations provide architectural and cellular information that is analysed by pathologists to reach an intraoperative diagnosis. Progress in the fields of artificial intelligence and machine learning means that AI systems have significant potential for the provision of highly accurate real-time diagnosis in cytopathology. OBJECTIVE To investigate the efficiency of machine-learning models in the intraoperative cytological diagnosis of CNS tumours. MATERIALS AND METHODS We trained a deep neural network to classify biopsy material for intraoperative tissue diagnosis of four major brain lesions. Overall, 205 medical images were obtained from squash smear slides of histologically correlated cases, with 18 high-grade and 11 low-grade gliomas, 17 metastatic carcinomas, and 9 non-neoplastic pathological brain tissue samples. The neural network model was trained and evaluated using 5-fold cross-validation. RESULTS The model achieved 95% and 97% diagnostic accuracy in the patch-level classification and patient-level classification tasks, respectively. CONCLUSIONS We conclude that deep learning-based classification of cytological preparations may be a promising complementary method for the rapid and accurate intraoperative diagnosis of CNS tumours.
Collapse
Affiliation(s)
- Erdener Ozer
- Department of Pathology, Dokuz Eylul University School of Medicine, Izmir, Turkey.,Division of Anatomical Pathology, Sidra Medicine and Research Center, Doha, Qatar
| | - Ali Enver Bilecen
- Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Turkey
| | - Nur Basak Ozer
- Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Turkey
| | - Berrin Yanikoglu
- Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Turkey.,Center of Excellence in Data Analytics (VERIM), Sabanci University, Istanbul, Turkey
| |
Collapse
|
24
|
Predicting gene mutation status via artificial intelligence technologies based on multimodal integration (MMI) to advance precision oncology. Semin Cancer Biol 2023; 91:1-15. [PMID: 36801447 DOI: 10.1016/j.semcancer.2023.02.006] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2022] [Revised: 01/30/2023] [Accepted: 02/15/2023] [Indexed: 02/21/2023]
Abstract
Personalized treatment strategies for cancer frequently rely on the detection of genetic alterations which are determined by molecular biology assays. Historically, these processes typically required single-gene sequencing, next-generation sequencing, or visual inspection of histopathology slides by experienced pathologists in a clinical context. In the past decade, advances in artificial intelligence (AI) technologies have demonstrated remarkable potential in assisting physicians with accurate diagnosis of oncology image-recognition tasks. Meanwhile, AI techniques make it possible to integrate multimodal data such as radiology, histology, and genomics, providing critical guidance for the stratification of patients in the context of precision therapy. Given that the mutation detection is unaffordable and time-consuming for a considerable number of patients, predicting gene mutations based on routine clinical radiological scans or whole-slide images of tissue with AI-based methods has become a hot issue in actual clinical practice. In this review, we synthesized the general framework of multimodal integration (MMI) for molecular intelligent diagnostics beyond standard techniques. Then we summarized the emerging applications of AI in the prediction of mutational and molecular profiles of common cancers (lung, brain, breast, and other tumor types) pertaining to radiology and histology imaging. Furthermore, we concluded that there truly exist multiple challenges of AI techniques in the way of its real-world application in the medical field, including data curation, feature fusion, model interpretability, and practice regulations. Despite these challenges, we still prospect the clinical implementation of AI as a highly potential decision-support tool to aid oncologists in future cancer treatment management.
Collapse
|
25
|
Li L, Liang Y, Shao M, Lu S, Liao S, Ouyang D. Self-supervised learning-based Multi-Scale feature Fusion Network for survival analysis from whole slide images. Comput Biol Med 2023; 153:106482. [PMID: 36586231 DOI: 10.1016/j.compbiomed.2022.106482] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2022] [Revised: 12/16/2022] [Accepted: 12/25/2022] [Indexed: 12/29/2022]
Abstract
Understanding prognosis and mortality is critical for evaluating the treatment plan of patients. Advances in digital pathology and deep learning techniques have made it practical to perform survival analysis in whole slide images (WSIs). Current methods are usually based on a multi-stage framework which includes patch sampling, feature extraction and prediction. However, the random patch sampling strategy is highly unstable and prone to sampling non-ROI. Feature extraction typically relies on hand-crafted features or convolutional neural networks (CNNs) pre-trained on ImageNet, while the artificial error or domain gaps may affect the survival prediction performance. Besides, the limited information representation of local sampling patches will create a bottleneck limitation on the effectiveness of prediction. To address the above challenges, we propose a novel patch sampling strategy based on image information entropy and construct a Multi-Scale feature Fusion Network (MSFN) based on self-supervised feature extractor. Specifically, we adopt image information entropy as a criterion to select representative sampling patches, thereby avoiding the noise interference caused by random to blank regions. Meanwhile, we pretrain the feature extractor utilizing self-supervised learning mechanism to improve the efficiency of feature extraction. Furthermore, a global-local feature fusion prediction network based on the attention mechanism is constructed to improve the survival prediction effect of WSIs with comprehensive multi-scale information representation. The proposed method is validated by adequate experiments and achieves competitive results on both of the most popular WSIs survival analysis datasets, TCGA-GBM and TCGA-LUSC. Code and trained models are made available at: https://github.com/Mercuriiio/MSFN.
Collapse
Affiliation(s)
- Le Li
- Faculty of Innovation Engineering, Macau University of Science and Technology, 999078, Macao Special Administrative Region of China.
| | - Yong Liang
- Peng Cheng Laboratory, Shenzhen, 518055, China.
| | - Mingwen Shao
- College of Computer Science and Technology, China University of Petroleum, Qingdao 266580, China.
| | - Shanghui Lu
- Faculty of Innovation Engineering, Macau University of Science and Technology, 999078, Macao Special Administrative Region of China.
| | - Shuilin Liao
- Faculty of Innovation Engineering, Macau University of Science and Technology, 999078, Macao Special Administrative Region of China.
| | - Dong Ouyang
- Faculty of Innovation Engineering, Macau University of Science and Technology, 999078, Macao Special Administrative Region of China.
| |
Collapse
|
26
|
Jiang Y, Sui X, Ding Y, Xiao W, Zheng Y, Zhang Y. A semi-supervised learning approach with consistency regularization for tumor histopathological images analysis. Front Oncol 2023; 12:1044026. [PMID: 36698401 PMCID: PMC9870542 DOI: 10.3389/fonc.2022.1044026] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2022] [Accepted: 12/06/2022] [Indexed: 01/12/2023] Open
Abstract
Introduction Manual inspection of histopathological images is important in clinical cancer diagnosis. Pathologists implement pathological diagnosis and prognostic evaluation through the microscopic examination of histopathological slices. This entire process is time-consuming, laborious, and challenging for pathologists. The modern use of whole-slide imaging, which scans histopathology slides to digital slices, and analysis using computer-aided diagnosis is an essential problem. Methods To solve the problem of difficult labeling of histopathological data, and improve the flexibility of histopathological analysis in clinical applications, we herein propose a semi-supervised learning algorithm coupled with consistency regularization strategy, called"Semi- supervised Histopathology Analysis Network"(Semi-His-Net), for automated normal-versus-tumor and subtype classifications. Specifically, when inputted disturbing versions of the same image, the model should predict similar outputs. Based on this, the model itself can assign artificial labels to unlabeled data for subsequent model training, thereby effectively reducing the labeled data required for training. Results Our Semi-His-Net is able to classify patches from breast cancer histopathological images into normal tissue and three other different tumor subtypes, achieving an accuracy was 90%. The average AUC of cross-classification between tumors reached 0.893. Discussion To overcome the limitations of visual inspection by pathologists for histopathology images, such as long time and low repeatability, we have developed a deep learning-based framework (Semi-His-Net) for automatic classification subdivision of the subtypes contained in the whole pathological images. This learning-based framework has great potential to improve the efficiency and repeatability of histopathological image diagnosis.
Collapse
Affiliation(s)
- Yanyun Jiang
- School of Mathematics and Statistics, Shandong Normal University, Jinan, China
| | - Xiaodan Sui
- School of Mathematics and Statistics, Shandong Normal University, Jinan, China
| | - Yanhui Ding
- School of Mathematics and Statistics, Shandong Normal University, Jinan, China
| | - Wei Xiao
- Shandong Provincial Hospital, Shandong University, Jinan, China
| | - Yuanjie Zheng
- School of Mathematics and Statistics, Shandong Normal University, Jinan, China,*Correspondence: Yuanjie Zheng, ; Yongxin Zhang,
| | - Yongxin Zhang
- School of Mathematics and Statistics, Shandong Normal University, Jinan, China,*Correspondence: Yuanjie Zheng, ; Yongxin Zhang,
| |
Collapse
|
27
|
Jeong J, Kim KD, Nam Y, Cho CE, Go H, Kim N. Stain normalization using score-based diffusion model through stain separation and overlapped moving window patch strategies. Comput Biol Med 2023; 152:106335. [PMID: 36473344 DOI: 10.1016/j.compbiomed.2022.106335] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 11/10/2022] [Accepted: 11/15/2022] [Indexed: 11/30/2022]
Abstract
Hematoxylin and eosin (H&E) staining is the gold standard modality for diagnosis in medicine. However, the dosage ratio of hematoxylin to eosin in H&E staining has not been standardized yet. Additionally, H&E stains fade out at various speeds. Therefore, the staining quality could differ among each image, and stain normalization is a critical preprocessing approach for training deep learning (DL) models, especially in long-term and/or multicenter digital pathology studies. However, conventional methods for stain normalization have some significant drawbacks, such as collapsing in the structure and/or texture of tissue. In addition, conventional methods must require a reference patch or slide. Meanwhile, DL-based methods have a risk of overfitting and/or grid artifacts. We developed a score-based diffusion model of colorization for stain normalization. However, mistransfer, in which the model confuses hematoxylin with eosin, can occur using a score-based diffusion model due to its high diversity nature. To overcome this mistransfer, we propose a stain separation method using sparse non-negative matrix factorization (SNMF), which can decompose pathology slide into Hematoxylin and Eosin to normalize each stain component. Furthermore, inpainting with overlapped moving window patches was used to prevent grid artifacts of whole slide image normalization. Our method can normalize the whole slide pathology images through this stain normalization pipeline with decent performance.
Collapse
Affiliation(s)
- Jiheon Jeong
- Department of Biomedical Engineering, Asan Medical Institute of Convergence Science and Technology, Asan Medical Center, College of Medicine, University of Ulsan, Seoul, Republic of Korea; Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea.
| | - Ki Duk Kim
- Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea.
| | - Yujin Nam
- Department of Biomedical Engineering, Asan Medical Institute of Convergence Science and Technology, Asan Medical Center, College of Medicine, University of Ulsan, Seoul, Republic of Korea; Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
| | - Cristina Eunbee Cho
- Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
| | - Heounjeong Go
- Department of Pathology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Republic of Korea.
| | - Namkug Kim
- Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea; Department of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Republic of Korea.
| |
Collapse
|
28
|
Familiar AM, Mahtabfar A, Fathi Kazerooni A, Kiani M, Vossough A, Viaene A, Storm PB, Resnick AC, Nabavizadeh A. Radio-pathomic approaches in pediatric neuro-oncology: Opportunities and challenges. Neurooncol Adv 2023; 5:vdad119. [PMID: 37841693 PMCID: PMC10576517 DOI: 10.1093/noajnl/vdad119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2023] Open
Abstract
With medical software platforms moving to cloud environments with scalable storage and computing, the translation of predictive artificial intelligence (AI) models to aid in clinical decision-making and facilitate personalized medicine for cancer patients is becoming a reality. Medical imaging, namely radiologic and histologic images, has immense analytical potential in neuro-oncology, and models utilizing integrated radiomic and pathomic data may yield a synergistic effect and provide a new modality for precision medicine. At the same time, the ability to harness multi-modal data is met with challenges in aggregating data across medical departments and institutions, as well as significant complexity in modeling the phenotypic and genotypic heterogeneity of pediatric brain tumors. In this paper, we review recent pathomic and integrated pathomic, radiomic, and genomic studies with clinical applications. We discuss current challenges limiting translational research on pediatric brain tumors and outline technical and analytical solutions. Overall, we propose that to empower the potential residing in radio-pathomics, systemic changes in cross-discipline data management and end-to-end software platforms to handle multi-modal data sets are needed, in addition to embracing modern AI-powered approaches. These changes can improve the performance of predictive models, and ultimately the ability to advance brain cancer treatments and patient outcomes through the development of such models.
Collapse
Affiliation(s)
- Ariana M Familiar
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Aria Mahtabfar
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Thomas Jefferson University Hospital, Philadelphia, PA, USA
| | - Anahita Fathi Kazerooni
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Mahsa Kiani
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Arastoo Vossough
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Angela Viaene
- Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Phillip B Storm
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Adam C Resnick
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Ali Nabavizadeh
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
29
|
Differential diagnosis of thyroid nodule capsules using random forest guided selection of image features. Sci Rep 2022; 12:21636. [PMID: 36517531 PMCID: PMC9751070 DOI: 10.1038/s41598-022-25788-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Accepted: 12/05/2022] [Indexed: 12/15/2022] Open
Abstract
Microscopic evaluation of tissue sections stained with hematoxylin and eosin is the current gold standard for diagnosing thyroid pathology. Digital pathology is gaining momentum providing the pathologist with additional cues to traditional routes when placing a diagnosis, therefore it is extremely important to develop new image analysis methods that can extract image features with diagnostic potential. In this work, we use histogram and texture analysis to extract features from microscopic images acquired on thin thyroid nodule capsules sections and demonstrate how they enable the differential diagnosis of thyroid nodules. Targeted thyroid nodules are benign (i.e., follicular adenoma) and malignant (i.e., papillary thyroid carcinoma and its sub-type arising within a follicular adenoma). Our results show that the considered image features can enable the quantitative characterization of the collagen capsule surrounding thyroid nodules and provide an accurate classification of the latter's type using random forest.
Collapse
|
30
|
Yang M, Xie Z, Wang Z, Yuan Y, Zhang J. Su-MICL: Severity-Guided Multiple Instance Curriculum Learning for Histopathology Image Interpretable Classification. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:3533-3543. [PMID: 35786552 DOI: 10.1109/tmi.2022.3188326] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Histopathology image classification plays a critical role in clinical diagnosis. However, due to the absence of clinical interpretability, most existing image-level classifiers remain impractical. To acquire the essential interpretability, lesion-level diagnosis is also provided, relying on detailed lesion-level annotations. Although the multiple-instance learning (MIL)-based approach can identify lesions by only utilizing image-level annotations, it requires overly strict prior information and has limited accuracy in lesion-level tasks. Here, we present a novel severity-guided multiple instance curriculum learning (Su-MICL) strategy to avoid tedious labeling. The proposed Su-MICL is under a MIL framework with a neglected prior: disease severity to define the learning difficulty of training images. Based on the difficulty degree, a curriculum is developed to train a model utilizing images from easy to hard. The experimental results for two histopathology image datasets demonstrate that Su-MICL achieves comparable performance to the state-of-the-art weakly supervised methods for image-level classification, and its performance for identifying lesions is closest to the supervised learning method. Without tedious lesion labeling, the Su-MICL approach can provide an interpretable diagnosis, as well as an effective insight to aid histopathology image diagnosis.
Collapse
|
31
|
Alzoubi I, Bao G, Zheng Y, Wang X, Graeber MB. Artificial intelligence techniques for neuropathological diagnostics and research. Neuropathology 2022. [PMID: 36443935 DOI: 10.1111/neup.12880] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 10/17/2022] [Accepted: 10/23/2022] [Indexed: 12/03/2022]
Abstract
Artificial intelligence (AI) research began in theoretical neurophysiology, and the resulting classical paper on the McCulloch-Pitts mathematical neuron was written in a psychiatry department almost 80 years ago. However, the application of AI in digital neuropathology is still in its infancy. Rapid progress is now being made, which prompted this article. Human brain diseases represent distinct system states that fall outside the normal spectrum. Many differ not only in functional but also in structural terms, and the morphology of abnormal nervous tissue forms the traditional basis of neuropathological disease classifications. However, only a few countries have the medical specialty of neuropathology, and, given the sheer number of newly developed histological tools that can be applied to the study of brain diseases, a tremendous shortage of qualified hands and eyes at the microscope is obvious. Similarly, in neuroanatomy, human observers no longer have the capacity to process the vast amounts of connectomics data. Therefore, it is reasonable to assume that advances in AI technology and, especially, whole-slide image (WSI) analysis will greatly aid neuropathological practice. In this paper, we discuss machine learning (ML) techniques that are important for understanding WSI analysis, such as traditional ML and deep learning, introduce a recently developed neuropathological AI termed PathoFusion, and present thoughts on some of the challenges that must be overcome before the full potential of AI in digital neuropathology can be realized.
Collapse
Affiliation(s)
- Islam Alzoubi
- School of Computer Science The University of Sydney Sydney New South Wales Australia
| | - Guoqing Bao
- School of Computer Science The University of Sydney Sydney New South Wales Australia
| | - Yuqi Zheng
- Ken Parker Brain Tumour Research Laboratories Brain and Mind Centre, Faculty of Medicine and Health, University of Sydney Camperdown New South Wales Australia
| | - Xiuying Wang
- School of Computer Science The University of Sydney Sydney New South Wales Australia
| | - Manuel B. Graeber
- Ken Parker Brain Tumour Research Laboratories Brain and Mind Centre, Faculty of Medicine and Health, University of Sydney Camperdown New South Wales Australia
| |
Collapse
|
32
|
Ji J, Wan T, Chen D, Wang H, Zheng M, Qin Z. A deep learning method for automatic evaluation of diagnostic information from multi-stained histopathological images. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.109820] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/31/2022]
|
33
|
Multiregional Radiomic Signatures Based on Functional Parametric Maps from DCE-MRI for Preoperative Identification of Estrogen Receptor and Progesterone Receptor Status in Breast Cancer. Diagnostics (Basel) 2022; 12:diagnostics12102558. [PMID: 36292247 PMCID: PMC9601361 DOI: 10.3390/diagnostics12102558] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2022] [Revised: 10/01/2022] [Accepted: 10/07/2022] [Indexed: 11/16/2022] Open
Abstract
Radiomics based on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) has been used for breast estrogen receptor (ER) and progesterone receptor (PR) status evaluation. However, the radiomic features of peritumoral regions were not thoroughly analyzed. This study aimed to establish and validate the multiregional radiomic signatures (RSs) for the preoperative identification of the ER and PR status in breast cancer. A total of 443 patients with breast cancer were divided into training (n = 356) and validation (n = 87) sets. Radiomic features were extracted from intra- and peritumoral regions on six functional parametric maps from DCE-MRI. A two-sample t-test, least absolute shrinkage and selection operator regression, and stepwise were used for feature selections. Three RSs for predicting the ER and PR status were constructed using a logistic regression model based on selected intratumoral, peritumoral, and combined intra- and peritumoral radiomic features. The area under the receiver operator characteristic curve (AUC) was used to assess the discriminative performance of three RSs. The AUCs of intra- and peritumoral RSs for identifying the ER status were 0.828/0.791 and 0.755/0.733 in the training and validation sets, respectively. For predicting the PR status, intra- and peritumoral RSs resulted in AUCs of 0.816/0.749 and 0.806/0.708 in the training and validation sets, respectively. Multiregional RSs achieved the best AUCs among three RSs for evaluating the ER (0.851 and 0.833) and PR (0.848 and 0.763) status. In conclusion, multiregional RSs based on functional parametric maps from DCE-MRI showed promising results for preoperatively evaluating the ER and PR status in breast cancer patients. Further studies using a larger cohort from multiple centers are necessary to confirm the reliability of the established models before clinical application.
Collapse
|
34
|
Popat M, Patel S. Research perspective and review towards brain tumour segmentation and classification using different image modalities. COMPUTER METHODS IN BIOMECHANICS AND BIOMEDICAL ENGINEERING: IMAGING & VISUALIZATION 2022. [DOI: 10.1080/21681163.2022.2124546] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Affiliation(s)
- Mayuri Popat
- U & P.U. Patel Department of Computer Engineering, Chandubhai S Patel Institute of Technology (CSPIT), Charotar University of Science and Technology (CHARUSAT), Gujarat, India
| | - Sanskruti Patel
- Smt. Chandaben Mohanbhai Patel Institute of Computer Applications (CMPICA), Charotar University of Science and Technology (CHARUSAT), Gujarat, India
| |
Collapse
|
35
|
Shastry KA, Vijayakumar V, V MKM, B A M, B N C. Deep Learning Techniques for the Effective Prediction of Alzheimer’s Disease: A Comprehensive Review. Healthcare (Basel) 2022; 10:healthcare10101842. [PMID: 36292289 PMCID: PMC9601959 DOI: 10.3390/healthcare10101842] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2022] [Revised: 09/15/2022] [Accepted: 09/15/2022] [Indexed: 11/23/2022] Open
Abstract
“Alzheimer’s disease” (AD) is a neurodegenerative disorder in which the memory shrinks and neurons die. “Dementia” is described as a gradual decline in mental, psychological, and interpersonal qualities that hinders a person’s ability to function autonomously. AD is the most common degenerative brain disease. Among the first signs of AD are missing recent incidents or conversations. “Deep learning” (DL) is a type of “machine learning” (ML) that allows computers to learn by doing, much like people do. DL techniques can attain cutting-edge precision, beating individuals in certain cases. A large quantity of tagged information with multi-layered “neural network” architectures is used to perform analysis. Because significant advancements in computed tomography have resulted in sizable heterogeneous brain signals, the use of DL for the timely identification as well as automatic classification of AD has piqued attention lately. With these considerations in mind, this paper provides an in-depth examination of the various DL approaches and their implementations for the identification and diagnosis of AD. Diverse research challenges are also explored, as well as current methods in the field.
Collapse
Affiliation(s)
- K Aditya Shastry
- Department of Information Science and Engineering, Nitte Meenakshi Institute of Technology, Bangalore 560064, India
- Correspondence: (K.A.S.); (V.V.)
| | - V Vijayakumar
- School of Computer Science and Engineering, The University of New South Wales, Sydney, NSW 2052, Australia
- School of NUOVOS, Ajeenkya D Y Patil University, Pune 412105, India
- Swiss School of Business and Management, 1213 Geneva, Switzerland
- Correspondence: (K.A.S.); (V.V.)
| | - Manoj Kumar M V
- Department of Information Science and Engineering, Nitte Meenakshi Institute of Technology, Bangalore 560064, India
| | - Manjunatha B A
- Department of Information Science and Engineering, Nitte Meenakshi Institute of Technology, Bangalore 560064, India
| | - Chandrashekhar B N
- Department of Information Science and Engineering, Nitte Meenakshi Institute of Technology, Bangalore 560064, India
| |
Collapse
|
36
|
Wong ANN, He Z, Leung KL, To CCK, Wong CY, Wong SCC, Yoo JS, Chan CKR, Chan AZ, Lacambra MD, Yeung MHY. Current Developments of Artificial Intelligence in Digital Pathology and Its Future Clinical Applications in Gastrointestinal Cancers. Cancers (Basel) 2022; 14:3780. [PMID: 35954443 PMCID: PMC9367360 DOI: 10.3390/cancers14153780] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2022] [Revised: 07/27/2022] [Accepted: 08/01/2022] [Indexed: 02/05/2023] Open
Abstract
The implementation of DP will revolutionize current practice by providing pathologists with additional tools and algorithms to improve workflow. Furthermore, DP will open up opportunities for development of AI-based tools for more precise and reproducible diagnosis through computational pathology. One of the key features of AI is its capability to generate perceptions and recognize patterns beyond the human senses. Thus, the incorporation of AI into DP can reveal additional morphological features and information. At the current rate of AI development and adoption of DP, the interest in computational pathology is expected to rise in tandem. There have already been promising developments related to AI-based solutions in prostate cancer detection; however, in the GI tract, development of more sophisticated algorithms is required to facilitate histological assessment of GI specimens for early and accurate diagnosis. In this review, we aim to provide an overview of the current histological practices in AP laboratories with respect to challenges faced in image preprocessing, present the existing AI-based algorithms, discuss their limitations and present clinical insight with respect to the application of AI in early detection and diagnosis of GI cancer.
Collapse
Affiliation(s)
- Alex Ngai Nick Wong
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China; (A.N.N.W.); (Z.H.); (K.L.L.); (C.Y.W.); (S.C.C.W.); (J.S.Y.)
| | - Zebang He
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China; (A.N.N.W.); (Z.H.); (K.L.L.); (C.Y.W.); (S.C.C.W.); (J.S.Y.)
| | - Ka Long Leung
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China; (A.N.N.W.); (Z.H.); (K.L.L.); (C.Y.W.); (S.C.C.W.); (J.S.Y.)
| | - Curtis Chun Kit To
- Department of Anatomical and Cellular Pathology, The Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, Hong Kong SAR, China; (C.C.K.T.); (C.K.R.C.); (M.D.L.)
| | - Chun Yin Wong
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China; (A.N.N.W.); (Z.H.); (K.L.L.); (C.Y.W.); (S.C.C.W.); (J.S.Y.)
| | - Sze Chuen Cesar Wong
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China; (A.N.N.W.); (Z.H.); (K.L.L.); (C.Y.W.); (S.C.C.W.); (J.S.Y.)
| | - Jung Sun Yoo
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China; (A.N.N.W.); (Z.H.); (K.L.L.); (C.Y.W.); (S.C.C.W.); (J.S.Y.)
| | - Cheong Kin Ronald Chan
- Department of Anatomical and Cellular Pathology, The Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, Hong Kong SAR, China; (C.C.K.T.); (C.K.R.C.); (M.D.L.)
| | - Angela Zaneta Chan
- Department of Anatomical and Cellular Pathology, Prince of Wales Hospital, Shatin, Hong Kong SAR, China;
| | - Maribel D. Lacambra
- Department of Anatomical and Cellular Pathology, The Chinese University of Hong Kong, Prince of Wales Hospital, Shatin, Hong Kong SAR, China; (C.C.K.T.); (C.K.R.C.); (M.D.L.)
| | - Martin Ho Yin Yeung
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China; (A.N.N.W.); (Z.H.); (K.L.L.); (C.Y.W.); (S.C.C.W.); (J.S.Y.)
| |
Collapse
|
37
|
BM-Net: CNN-Based MobileNet-V3 and Bilinear Structure for Breast Cancer Detection in Whole Slide Images. Bioengineering (Basel) 2022; 9:bioengineering9060261. [PMID: 35735504 PMCID: PMC9220285 DOI: 10.3390/bioengineering9060261] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2022] [Revised: 06/15/2022] [Accepted: 06/15/2022] [Indexed: 11/17/2022] Open
Abstract
Breast cancer is one of the most common types of cancer and is the leading cause of cancer-related death. Diagnosis of breast cancer is based on the evaluation of pathology slides. In the era of digital pathology, these slides can be converted into digital whole slide images (WSIs) for further analysis. However, due to their sheer size, digital WSIs diagnoses are time consuming and challenging. In this study, we present a lightweight architecture that consists of a bilinear structure and MobileNet-V3 network, bilinear MobileNet-V3 (BM-Net), to analyze breast cancer WSIs. We utilized the WSI dataset from the ICIAR2018 Grand Challenge on Breast Cancer Histology Images (BACH) competition, which contains four classes: normal, benign, in situ carcinoma, and invasive carcinoma. We adopted data augmentation techniques to increase diversity and utilized focal loss to remove class imbalance. We achieved high performance, with 0.88 accuracy in patch classification and an average 0.71 score, which surpassed state-of-the-art models. Our BM-Net shows great potential in detecting cancer in WSIs and is a promising clinical tool.
Collapse
|
38
|
Sheikh TS, Kim JY, Shim J, Cho M. Unsupervised Learning Based on Multiple Descriptors for WSIs Diagnosis. Diagnostics (Basel) 2022; 12:diagnostics12061480. [PMID: 35741289 PMCID: PMC9222016 DOI: 10.3390/diagnostics12061480] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2022] [Revised: 06/11/2022] [Accepted: 06/14/2022] [Indexed: 11/16/2022] Open
Abstract
An automatic pathological diagnosis is a challenging task because histopathological images with different cellular heterogeneity representations are sometimes limited. To overcome this, we investigated how the holistic and local appearance features with limited information can be fused to enhance the analysis performance. We propose an unsupervised deep learning model for whole-slide image diagnosis, which uses stacked autoencoders simultaneously feeding multiple-image descriptors such as the histogram of oriented gradients and local binary patterns along with the original image to fuse the heterogeneous features. The pre-trained latent vectors are extracted from each autoencoder, and these fused feature representations are utilized for classification. We observed that training with additional descriptors helps the model to overcome the limitations of multiple variants and the intricate cellular structure of histopathology data by various experiments. Our model outperforms existing state-of-the-art approaches by achieving the highest accuracies of 87.2 for ICIAR2018, 94.6 for Dartmouth, and other significant metrics for public benchmark datasets. Our model does not rely on a specific set of pre-trained features based on classifiers to achieve high performance. Unsupervised spaces are learned from the number of independent multiple descriptors and can be used with different variants of classifiers to classify cancer diseases from whole-slide images. Furthermore, we found that the proposed model classifies the types of breast and lung cancer similar to the viewpoint of pathologists by visualization. We also designed our whole-slide image processing toolbox to extract and process the patches from whole-slide images.
Collapse
Affiliation(s)
| | - Jee-Yeon Kim
- Department of Pathology, Pusan National University Yangsan Hospital, School of Medicine, Pusan National University, Yangsan-si 50612, Korea;
| | - Jaesool Shim
- School of Mechanical Engineering, Yeungnam University, Gyeongsan 38541, Korea
- Correspondence: (J.S.); (M.C.)
| | - Migyung Cho
- Department of Computer & Media Engineering, Tongmyong University, Busan 48520, Korea;
- Correspondence: (J.S.); (M.C.)
| |
Collapse
|
39
|
Meirelles AL, Kurc T, Saltz J, Teodoro G. Effective active learning in digital pathology: A case study in tumor infiltrating lymphocytes. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 220:106828. [PMID: 35500506 DOI: 10.1016/j.cmpb.2022.106828] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/05/2021] [Revised: 04/09/2022] [Accepted: 04/19/2022] [Indexed: 06/14/2023]
Abstract
BACKGROUND AND OBJECTIVE Deep learning methods have demonstrated remarkable performance in pathology image analysis, but they require a large amount of annotated training data from expert pathologists. The aim of this study is to minimize the data annotation need in these analyses. METHODS Active learning (AL) is an iterative approach to training deep learning models. It was used in our context with a Tumor Infiltrating Lymphocytes (TIL) classification task to minimize annotation. State-of-the-art AL methods were evaluated with the TIL application and we have proposed and evaluated a more efficient and effective AL acquisition method. The proposed method uses data grouping based on imaging features and model prediction uncertainty to select meaningful training samples (image patches). RESULTS An experimental evaluation with a collection of cancer tissue images shows that: (i) Our approach reduces the number of patches required to attain a given AUC as compared to other approaches, and (ii) our optimization (subpooling) leads to AL execution time improvement of about 2.12×. CONCLUSIONS This strategy enabled TIL based deep learning analyses using smaller annotation demand. We expect this approach may be used to build other analyses in digital pathology with fewer training samples.
Collapse
Affiliation(s)
- André Ls Meirelles
- Department of Computer Science, University of Brasília, Brasília, 70910-900, Brazil
| | - Tahsin Kurc
- Biomedical Informatics Department, Stony Brook University, Stony Brook, 11794-8322, USA
| | - Joel Saltz
- Biomedical Informatics Department, Stony Brook University, Stony Brook, 11794-8322, USA
| | - George Teodoro
- Department of Computer Science, Universidade Federal de Minas Gerais, Belo Horizonte, 31270-901, Brazil.
| |
Collapse
|
40
|
Meirelles ALS, Kurc T, Kong J, Ferreira R, Saltz JH, Teodoro G. Building Efficient CNN Architectures for Histopathology Images Analysis: A Case-Study in Tumor-Infiltrating Lymphocytes Classification. Front Med (Lausanne) 2022; 9:894430. [PMID: 35712087 PMCID: PMC9197439 DOI: 10.3389/fmed.2022.894430] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2022] [Accepted: 05/11/2022] [Indexed: 11/13/2022] Open
Abstract
Background Deep learning methods have demonstrated remarkable performance in pathology image analysis, but they are computationally very demanding. The aim of our study is to reduce their computational cost to enable their use with large tissue image datasets. Methods We propose a method called Network Auto-Reduction (NAR) that simplifies a Convolutional Neural Network (CNN) by reducing the network to minimize the computational cost of doing a prediction. NAR performs a compound scaling in which the width, depth, and resolution dimensions of the network are reduced together to maintain a balance among them in the resulting simplified network. We compare our method with a state-of-the-art solution called ResRep. The evaluation is carried out with popular CNN architectures and a real-world application that identifies distributions of tumor-infiltrating lymphocytes in tissue images. Results The experimental results show that both ResRep and NAR are able to generate simplified, more efficient versions of ResNet50 V2. The simplified versions by ResRep and NAR require 1.32× and 3.26× fewer floating-point operations (FLOPs), respectively, than the original network without a loss in classification power as measured by the Area under the Curve (AUC) metric. When applied to a deeper and more computationally expensive network, Inception V4, NAR is able to generate a version that requires 4× lower than the original version with the same AUC performance. Conclusions NAR is able to achieve substantial reductions in the execution cost of two popular CNN architectures, while resulting in small or no loss in model accuracy. Such cost savings can significantly improve the use of deep learning methods in digital pathology. They can enable studies with larger tissue image datasets and facilitate the use of less expensive and more accessible graphics processing units (GPUs), thus reducing the computing costs of a study.
Collapse
Affiliation(s)
| | - Tahsin Kurc
- Biomedical Informatics Department, Stony Brook University, Stony Brook, NY, United States
| | - Jun Kong
- Department of Mathematics and Statistics and Computer Science, Georgia State University, Atlanta, GA, United States
| | - Renato Ferreira
- Department of Computer Science, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil
| | - Joel H. Saltz
- Biomedical Informatics Department, Stony Brook University, Stony Brook, NY, United States
| | - George Teodoro
- Department of Computer Science, Universidade de Brasília, Brasília, Brazil
- Department of Computer Science, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil
| |
Collapse
|
41
|
Deng C, Han D, Feng M, Lv Z, Li D. Differential diagnostic value of the ResNet50, random forest, and DS ensemble models for papillary thyroid carcinoma and other thyroid nodules. J Int Med Res 2022; 50:3000605221094276. [PMID: 35469474 PMCID: PMC9087260 DOI: 10.1177/03000605221094276] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Objective To explore the differential diagnostic efficiency of the
residual network (ResNet)50, random forest (RF), and DS ensemble models for
papillary thyroid carcinoma (PTC) and other pathological types of thyroid
nodules. Methods This study retrospectively analyzed 559 patients with
thyroid nodules and collected thyroid pathological images and auxiliary
examination results (laboratory and ultrasound results) to construct datasets.
The pathological image dataset was used to train a ResNet50 model, the text
dataset was used to train a random forest (RF) model, and a DS ensemble model
was constructed from the results of the two models. The differential diagnostic
values of the three models for PTC and other types of thyroid nodules were then
compared. Results The DS ensemble model had the highest sensitivity,
specificity, accuracy, and area under the receiver operating characteristic
curve (85.87%, 97.18%, 93.77%, and 0.982, respectively). Conclusions Compared with Resnet50 and the RF models trained only on
imaging data or text information, respectively, the DS ensemble model showed
better diagnostic value for PTC.
Collapse
Affiliation(s)
- Chengwen Deng
- Shanghai Tenth People's Hospital Tongji University, Shanghai, China
| | - Dongyan Han
- Shanghai Tenth People's Hospital Tongji University, Shanghai, China
| | | | - Zhongwei Lv
- Shanghai Tenth People's Hospital Tongji University, Shanghai, China
| | - Dan Li
- Shanghai Tenth People's Hospital Tongji University, Shanghai, China
| |
Collapse
|
42
|
Automated bone marrow cytology using deep learning to generate a histogram of cell types. COMMUNICATIONS MEDICINE 2022; 2:45. [PMID: 35603269 PMCID: PMC9053230 DOI: 10.1038/s43856-022-00107-6] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2021] [Accepted: 03/23/2022] [Indexed: 02/07/2023] Open
Abstract
Background Bone marrow cytology is required to make a hematological diagnosis, influencing critical clinical decision points in hematology. However, bone marrow cytology is tedious, limited to experienced reference centers and associated with inter-observer variability. This may lead to a delayed or incorrect diagnosis, leaving an unmet need for innovative supporting technologies. Methods We develop an end-to-end deep learning-based system for automated bone marrow cytology. Starting with a bone marrow aspirate digital whole slide image, our system rapidly and automatically detects suitable regions for cytology, and subsequently identifies and classifies all bone marrow cells in each region. This collective cytomorphological information is captured in a representation called Histogram of Cell Types (HCT) quantifying bone marrow cell class probability distribution and acting as a cytological patient fingerprint. Results Our system achieves high accuracy in region detection (0.97 accuracy and 0.99 ROC AUC), and cell detection and cell classification (0.75 mean average precision, 0.78 average F1-score, Log-average miss rate of 0.31). Conclusions HCT has potential to eventually support more efficient and accurate diagnosis in hematology, supporting AI-enabled computational pathology. Identifying and counting cells in bone marrow samples, known as cytology, is critical for the diagnosis of blood disorders. This is a complex and labor-intensive process, with some variation in how hematopathologists interpret these samples. Here, we develop an artificial intelligence system for automated bone marrow cytology, which automatically detects and identifies all types of cells found in the bone marrow. This information is summarized in a chart that we call the Histogram of Cell Types (HCT), a new way to represent complex information generated in bone marrow cytology. Our system achieves high accuracy and precision in classifying the different types of bone marrow cells as a HCT. This tool may eventually help clinicians to make more efficient and accurate diagnoses. Moosavi Tayebi et al. develop a deep learning-based computational pathology tool for automated bone marrow cytology from whole slide images. Their approach generates a histogram of cell types present within the bone marrow aspirate to aid in diagnostic haematopathology.
Collapse
|
43
|
Hsu WW, Guo JM, Pei L, Chiang LA, Li YF, Hsiao JC, Colen R, Liu P. A weakly supervised deep learning-based method for glioma subtype classification using WSI and mpMRIs. Sci Rep 2022; 12:6111. [PMID: 35414643 PMCID: PMC9005548 DOI: 10.1038/s41598-022-09985-1] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2022] [Accepted: 03/30/2022] [Indexed: 11/09/2022] Open
Abstract
Accurate glioma subtype classification is critical for the treatment management of patients with brain tumors. Developing an automatically computer-aided algorithm for glioma subtype classification is challenging due to many factors. One of the difficulties is the label constraint. Specifically, each case is simply labeled the glioma subtype without precise annotations of lesion regions information. In this paper, we propose a novel hybrid fully convolutional neural network (CNN)-based method for glioma subtype classification using both whole slide imaging (WSI) and multiparametric magnetic resonance imagings (mpMRIs). It is comprised of two methods: a WSI-based method and a mpMRIs-based method. For the WSI-based method, we categorize the glioma subtype using a 2D CNN on WSIs. To overcome the label constraint issue, we extract the truly representative patches for the glioma subtype classification in a weakly supervised fashion. For the mpMRIs-based method, we develop a 3D CNN-based method by analyzing the mpMRIs. The mpMRIs-based method consists of brain tumor segmentation and classification. Finally, to enhance the robustness of the predictions, we fuse the WSI-based and mpMRIs-based results guided by a confidence index. The experimental results on the validation dataset in the competition of CPM-RadPath 2020 show the comprehensive judgments from both two modalities can achieve better performance than the ones by solely using WSI or mpMRIs. Furthermore, our result using the proposed method ranks the third place in the CPM-RadPath 2020 in the testing phase. The proposed method demonstrates a competitive performance, which is creditable to the success of weakly supervised approach and the strategy of label agreement from multi-modality data.
Collapse
Affiliation(s)
- Wei-Wen Hsu
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, ROC
| | - Jing-Ming Guo
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, ROC
| | - Linmin Pei
- Imaging and Visualization Group, ABCS, Frederick National Laboratory for Cancer Research, Frederick, MD, 21702, USA.
| | - Ling-An Chiang
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, ROC
| | - Yao-Feng Li
- Department of Pathology, Tri-Service General Hospital and National Defense Medical Center, Taipei, 11490, Taiwan, ROC
| | - Jui-Chien Hsiao
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan, ROC
| | - Rivka Colen
- Department of Radiology, University of Pittsburgh, Pittsburgh, PA, 15232, USA.,Hillman Cancer Center, University of Pittsburgh Medical Center, Pittsburgh, PA, 15260, USA
| | - Peizhong Liu
- College of Engineering, Huaqiao University, Quanzhou, China
| |
Collapse
|
44
|
A comprehensive review of computer-aided whole-slide image analysis: from datasets to feature extraction, segmentation, classification and detection approaches. Artif Intell Rev 2022. [DOI: 10.1007/s10462-021-10121-0] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
|
45
|
Artificial intelligence as a tool for diagnosis in digital pathology whole slide images: A systematic review. J Pathol Inform 2022; 13:100138. [PMID: 36268059 PMCID: PMC9577128 DOI: 10.1016/j.jpi.2022.100138] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Revised: 09/04/2022] [Accepted: 09/04/2022] [Indexed: 12/22/2022] Open
Abstract
Digital pathology had a recent growth, stimulated by the implementation of digital whole slide images (WSIs) in clinical practice, and the pathology field faces shortage of pathologists in the last few years. This scenario created fronts of research applying artificial intelligence (AI) to help pathologists. One of them is the automated diagnosis, helping in the clinical decision support, increasing efficiency and quality of diagnosis. However, the complexity nature of the WSIs requires special treatments to create a reliable AI model for diagnosis. Therefore, we systematically reviewed the literature to analyze and discuss all the methods and results in AI in digital pathology performed in WSIs on H&E stain, investigating the capacity of AI as a diagnostic support tool for the pathologist in the routine real-world scenario. This review analyzes 26 studies, reporting in detail all the best methods to apply AI as a diagnostic tool, as well as the main limitations, and suggests new ideas to improve the AI field in digital pathology as a whole. We hope that this study could lead to a better use of AI as a diagnostic tool in pathology, helping future researchers in the development of new studies and projects.
Collapse
|
46
|
Arunachalam P, Venkatakrishnan P, Janakiraman N, Sangeetha S. Detection of Structure Characteristics and Its Discontinuity Based Field Programmable Gate Array Processor in Cancer Cell by Wavelet Transform. JOURNAL OF MEDICAL IMAGING AND HEALTH INFORMATICS 2021. [DOI: 10.1166/jmihi.2021.3902] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Digital clinical histopathology is one of the crucial techniques for precise cancer cell diagnosing in modern medicine. The Synovial Sarcoma (SS) cancer cell patterns seem to be a spindle shaped cell (SSC) structure and it is very difficult to identify the exact oval shaped cell
structure through pathologist’s eye perception. Meanwhile, there is necessitating for monitoring and securing the successful and effective image data processing in the the huge network data which is also a complex one. A field programmable Gate Array (FPGA) was regarded as a necessary
one for this. In this work, based on FPGA a Cancer Cell classification is made for the regulation and execution. Hence, mathematically the SSC regularity structures and its discontinuities are measured by the holder exponent (HE) function. In this research work, HE values have been
determined by Wavelet Transform Modulus Maxima (WTMM) and Wavelet Leader (WL) methods with basis function of Haar wavelet based on FPGA Processor. The quantitative parameters such as Mean of Asymptotic Discontinuity (MAD), Mean of Removable Discontinuity (MRD) and Number of Discontinuity Points
(NDPs) have been considered to determine the prediction of discontinuity detection between WTMM and WL methods. With the help of receiver operating characteristics (ROC) curve, the significant difference of discontinuity detection performance between both the methods has been analyzed. From
the experimental results, it is clear that the WL method is more practically feasible and it gives satisfactory performance, in terms of sensitivity and specificity percentage values, which are 80.56% and 59.46%, respectively in the blue color components of the SNR 20 dB noise image.
Collapse
Affiliation(s)
- P. Arunachalam
- Electronics and Communication Engineering Department, KLN College of Engineering-Madurai, Affiliated to Centre for Research Anna University-Chennai 630612, Tamilnadu, India
| | - P. Venkatakrishnan
- Electronics and Communication Engineering Department, CMR Technical Campus, Telangana 501401, India
| | - N. Janakiraman
- Electronics and Communication Engineering Department, KLN College of Engineering-Madurai, Affiliated to Centre for Research Anna University-Chennai 630612, Tamilnadu, India
| | - S. Sangeetha
- Electrical and Electronics Engineering Department, CMR College of Engineering & Technology, Telangana 501401, India
| |
Collapse
|
47
|
Bouvier C, Souedet N, Levy J, Jan C, You Z, Herard AS, Mergoil G, Rodriguez BH, Clouchoux C, Delzescaux T. Reduced and stable feature sets selection with random forest for neurons segmentation in histological images of macaque brain. Sci Rep 2021; 11:22973. [PMID: 34836996 PMCID: PMC8626511 DOI: 10.1038/s41598-021-02344-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2020] [Accepted: 10/27/2021] [Indexed: 01/01/2023] Open
Abstract
In preclinical research, histology images are produced using powerful optical microscopes to digitize entire sections at cell scale. Quantification of stained tissue relies on machine learning driven segmentation. However, such methods require multiple additional information, or features, which are increasing the quantity of data to process. As a result, the quantity of features to deal with represents a drawback to process large series or massive histological images rapidly in a robust manner. Existing feature selection methods can reduce the amount of required information but the selected subsets lack reproducibility. We propose a novel methodology operating on high performance computing (HPC) infrastructures and aiming at finding small and stable sets of features for fast and robust segmentation of high-resolution histological images. This selection has two steps: (1) selection at features families scale (an intermediate pool of features, between spaces and individual features) and (2) feature selection performed on pre-selected features families. We show that the selected sets of features are stables for two different neuron staining. In order to test different configurations, one of these dataset is a mono-subject dataset and the other is a multi-subjects dataset to test different configurations. Furthermore, the feature selection results in a significant reduction of computation time and memory cost. This methodology will allow exhaustive histological studies at a high-resolution scale on HPC infrastructures for both preclinical and clinical research.
Collapse
Affiliation(s)
- C Bouvier
- CEA, CNRS, MIRCen, Laboratoire Des Maladies Neurodégénératives, Université Paris-Saclay, Fontenay-aux-Roses, France
- Witsee, Paris, France
| | - N Souedet
- CEA, CNRS, MIRCen, Laboratoire Des Maladies Neurodégénératives, Université Paris-Saclay, Fontenay-aux-Roses, France
| | - J Levy
- Service de Médecine Physique Et de Réadaptation - APHP Hôpital Raymond Poincaré, Garches, France
- UMR 1179, Handicap Neuromusculaire - INSERM-UVSQ, Montigny le Bretonneux, France
| | - C Jan
- CEA, CNRS, MIRCen, Laboratoire Des Maladies Neurodégénératives, Université Paris-Saclay, Fontenay-aux-Roses, France
| | - Z You
- CEA, CNRS, MIRCen, Laboratoire Des Maladies Neurodégénératives, Université Paris-Saclay, Fontenay-aux-Roses, France
- Shaanxi Key Laboratory for Network Computing and Security Technology, School of Computer Science and Engineering, Xi'an University of Technology, Xi'an, China
| | - A-S Herard
- CEA, CNRS, MIRCen, Laboratoire Des Maladies Neurodégénératives, Université Paris-Saclay, Fontenay-aux-Roses, France
| | | | | | - C Clouchoux
- CEA, CNRS, MIRCen, Laboratoire Des Maladies Neurodégénératives, Université Paris-Saclay, Fontenay-aux-Roses, France
- Witsee, Paris, France
| | - T Delzescaux
- CEA, CNRS, MIRCen, Laboratoire Des Maladies Neurodégénératives, Université Paris-Saclay, Fontenay-aux-Roses, France.
| |
Collapse
|
48
|
Williams S, Layard Horsfall H, Funnell JP, Hanrahan JG, Khan DZ, Muirhead W, Stoyanov D, Marcus HJ. Artificial Intelligence in Brain Tumour Surgery-An Emerging Paradigm. Cancers (Basel) 2021; 13:cancers13195010. [PMID: 34638495 PMCID: PMC8508169 DOI: 10.3390/cancers13195010] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2021] [Revised: 10/02/2021] [Accepted: 10/03/2021] [Indexed: 01/01/2023] Open
Abstract
Artificial intelligence (AI) platforms have the potential to cause a paradigm shift in brain tumour surgery. Brain tumour surgery augmented with AI can result in safer and more effective treatment. In this review article, we explore the current and future role of AI in patients undergoing brain tumour surgery, including aiding diagnosis, optimising the surgical plan, providing support during the operation, and better predicting the prognosis. Finally, we discuss barriers to the successful clinical implementation, the ethical concerns, and we provide our perspective on how the field could be advanced.
Collapse
Affiliation(s)
- Simon Williams
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
- Correspondence:
| | - Hugo Layard Horsfall
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - Jonathan P. Funnell
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - John G. Hanrahan
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - Danyal Z. Khan
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - William Muirhead
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - Danail Stoyanov
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| | - Hani J. Marcus
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London WC1N 3BG, UK; (H.L.H.); (J.P.F.); (J.G.H.); (D.Z.K.); (W.M.); (H.J.M.)
- Wellcome/Engineering and Physical Sciences Research Council (EPSRC) Centre for Interventional and Surgical Sciences (WEISS), London W1W 7TY, UK;
| |
Collapse
|
49
|
Senousy Z, Abdelsamea MM, Gaber MM, Abdar M, Acharya UR, Khosravi A, Nahavandi S. MCUa: Multi-level Context and Uncertainty aware Dynamic Deep Ensemble for Breast Cancer Histology Image Classification. IEEE Trans Biomed Eng 2021; 69:818-829. [PMID: 34460359 DOI: 10.1109/tbme.2021.3107446] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Breast histology image classification is a crucial step in the early diagnosis of breast cancer. In breast pathological diagnosis, Convolutional Neural Networks (CNNs) have demonstrated great success using digitized histology slides. However, tissue classification is still challenging due to the high visual variability of the large-sized digitized samples and the lack of contextual information. In this paper, we propose a novel CNN, called Multi-level Context and Uncertainty aware (MCUa) dynamic deep learning ensemble model. MCUa model consists of several multi-level context-aware models to learn the spatial dependency between image patches in a layer-wise fashion. It exploits the high sensitivity to the multi-level contextual information using an uncertainty quantification component to accomplish a novel dynamic ensemble model. MCUa model has achieved a high accuracy of 98.11% on a breast cancer histology image dataset. Experimental results show the superior effectiveness of the proposed solution compared to the state-of-the-art histology classification models.
Collapse
|
50
|
Jiang J, Tekin B, Guo R, Liu H, Huang Y, Wang C. Digital Pathology-based Study of Cell- and Tissue-level Morphologic Features in Serous Borderline Ovarian Tumor and High-grade Serous Ovarian Cancer. J Pathol Inform 2021; 12:24. [PMID: 34447604 PMCID: PMC8356706 DOI: 10.4103/jpi.jpi_76_20] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2020] [Revised: 12/28/2020] [Accepted: 02/11/2021] [Indexed: 12/22/2022] Open
Abstract
Background: Serous borderline ovarian tumor (SBOT) and high-grade serous ovarian cancer (HGSOC) are two distinct subtypes of epithelial ovarian tumors, with markedly different biologic background, behavior, prognosis, and treatment. However, the histologic diagnosis of serous ovarian tumors can be subjectively variable and labor-intensive as multiple tumor slides/blocks need to be thoroughly examined to search for these features. Materials and Methods: We developed a novel informatics system to facilitate objective and scalable diagnosis screening for SBOT and HGSOC. The system was built upon Groovy scripts and QuPath to enable interactive annotation and data exchange. Results: The system was used to successfully detect cellular boundaries and extract an expanded set of cellular features representing cell- and tissue-level characteristics. The performance of cell-level classification for both tumor and stroma cells achieved >90% accuracy. The performance of differentiating HGSOC versus SBOT achieved 91%–95% accuracy for 6485 imaging patches which have sufficient tumor and stroma cells (minimum of ten each) and 97% accuracy for classifying patients when aggregating the results to whole-slide image based on consensus. Conclusions: Cellular features digitally extracted from pathological images can be used for cell classification and SBOT v. HGSOC differentiation. Introducing digital pathology into ovarian cancer research could be beneficial to discover potential clinical implications. A larger cohort is required to further evaluate the system.
Collapse
Affiliation(s)
- Jun Jiang
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| | - Burak Tekin
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA
| | - Ruifeng Guo
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA
| | - Hongfang Liu
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| | - Yajue Huang
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA
| | - Chen Wang
- Department of Health Science Research, Mayo Clinic, Rochester, MN, USA
| |
Collapse
|