1
|
Makhlouf Y, Singh VK, Craig S, McArdle A, French D, Loughrey MB, Oliver N, Acevedo JB, O’Reilly P, James JA, Maxwell P, Salto-Tellez M. True-T - Improving T-cell response quantification with holistic artificial intelligence based prediction in immunohistochemistry images. Comput Struct Biotechnol J 2024; 23:174-185. [PMID: 38146436 PMCID: PMC10749253 DOI: 10.1016/j.csbj.2023.11.048] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Revised: 11/24/2023] [Accepted: 11/24/2023] [Indexed: 12/27/2023] Open
Abstract
The immune response associated with oncogenesis and potential oncological ther- apeutic interventions has dominated the field of cancer research over the last decade. T-cell lymphocytes in the tumor microenvironment are a crucial aspect of cancer's adaptive immunity, and the quantification of T-cells in specific can- cer types has been suggested as a potential diagnostic aid. However, this is cur- rently not part of routine diagnostics. To address this challenge, we present a new method called True-T, which employs artificial intelligence-based techniques to quantify T-cells in colorectal cancer (CRC) using immunohistochemistry (IHC) images. True-T analyses the chromogenic tissue hybridization signal of three widely recognized T-cell markers (CD3, CD4, and CD8). Our method employs a pipeline consisting of three stages: T-cell segmentation, density estimation from the segmented mask, and prediction of individual five-year survival rates. In the first stage, we utilize the U-Net method, where a pre-trained ResNet-34 is em- ployed as an encoder to extract clinically relevant T-cell features. The segmenta- tion model is trained and evaluated individually, demonstrating its generalization in detecting the CD3, CD4, and CD8 biomarkers in IHC images. In the second stage, the density of T-cells is estimated using the predicted mask, which serves as a crucial indicator for patient survival statistics in the third stage. This ap- proach was developed and tested in 1041 patients from four reference diagnostic institutions, ensuring broad applicability. The clinical effectiveness of True-T is demonstrated in stages II-IV CRC by offering valuable prognostic information that surpasses previous quantitative gold standards, opening possibilities for po- tential clinical applications. Finally, to evaluate the robustness and broader ap- plicability of our approach without additional training, we assessed the universal accuracy of the CD3 component of the True-T algorithm across 13 distinct solid tumors.
Collapse
Affiliation(s)
- Yasmine Makhlouf
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Vivek Kumar Singh
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Stephanie Craig
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Aoife McArdle
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Dominique French
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Maurice B. Loughrey
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
- Cellular Pathology, Belfast Health and Social Care Trust, Belfast City Hospital, Lisburn Road, Belfast BT9 7AB, UK
| | - Nicola Oliver
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Juvenal Baena Acevedo
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | | | - Jacqueline A. James
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
- Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast BT9 7AE, UK
| | - Perry Maxwell
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Manuel Salto-Tellez
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
- Sonrai Analytics, Belfast BT9 7AE, UK
- Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast BT9 7AE, UK
- Integrated Pathology Unit, Institute of Cancer Research and Royal Marsden Hospital, London SW7 3RP, UK
| |
Collapse
|
2
|
Wei B, Li L, Feng Y, Liu S, Fu P, Tian L. Exploring prognostic biomarkers in pathological images of colorectal cancer patients via deep learning. J Pathol Clin Res 2024; 10:e70003. [PMID: 39343999 PMCID: PMC11439587 DOI: 10.1002/2056-4538.70003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2024] [Revised: 07/17/2024] [Accepted: 08/30/2024] [Indexed: 10/01/2024]
Abstract
Hematoxylin and eosin (H&E) whole slide images provide valuable information for predicting prognostic outcomes in colorectal cancer (CRC) patients. However, extracting prognostic indicators from pathological images is challenging due to the subtle complexities of phenotypic information. We trained a weakly supervised deep learning model on data from 640 CRC patients in the prostate, lung, colorectal, and ovarian (PLCO) cancer screening trial dataset and validated it using data from 522 CRC patients in the cancer genome atlas (TCGA) dataset. We created the colorectal cancer risk score (CRCRS) to assess patient prognosis, visualized the pathological phenotype of the risk score using Grad-CAM, and employed multiomics data from the TCGA CRC cohort to investigate the potential biological mechanisms underlying the risk score. The overall survival analysis revealed that the CRCRS served as an independent prognostic indicator for both the PLCO cohort (p < 0.001) and the TCGA cohort (p < 0.001), with its predictive efficacy remaining unaffected by the clinical staging system. Additionally, satisfactory chemotherapeutic benefits were observed in stage II/III CRC patients with high CRCRS but not in those with low CRCRS. A pathomics nomogram constructed by integrating the CRCRS with the tumor-node-metastasis (TNM) staging system enhanced prognostic prediction accuracy compared with using the TNM staging system alone. Noteworthy features of the risk score were identified, such as immature tumor mesenchyme, disorganized gland structures, small clusters of cancer cells associated with unfavorable prognosis, and infiltrating inflammatory cells associated with favorable prognosis. The TCGA multiomics data revealed potential correlations between the CRCRS and the activation of energy production and metabolic pathways, the tumor immune microenvironment, and genetic mutations in APC, SMAD2, EEF1AKMT4, EPG5, and TANC1. In summary, our deep learning algorithm identified the CRCRS as a prognostic indicator in CRC, providing a significant approach for prognostic risk stratification and tailoring precise treatment strategies for individual patients.
Collapse
Affiliation(s)
- Binshen Wei
- Department of Pathology, The First Affiliated Hospital of Harbin Medical University, Harbin, PR China
| | - Linqing Li
- Department of Pathology, The First Affiliated Hospital of Harbin Medical University, Harbin, PR China
| | - Yenan Feng
- Department of Pathology, The First Affiliated Hospital of Harbin Medical University, Harbin, PR China
| | - Sihan Liu
- Department of Pathology, The First Affiliated Hospital of Harbin Medical University, Harbin, PR China
| | - Peng Fu
- Department of Nuclear Medicine, The First Affiliated Hospital of Harbin Medical University, Harbin, PR China
| | - Lin Tian
- Department of Pathology, The First Affiliated Hospital of Harbin Medical University, Harbin, PR China
| |
Collapse
|
3
|
Lipkova J, Kather JN. The age of foundation models. Nat Rev Clin Oncol 2024; 21:769-770. [PMID: 39237731 DOI: 10.1038/s41571-024-00941-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/07/2024]
Affiliation(s)
- Jana Lipkova
- Department of Pathology and Laboratory Medicine, University of California Irvine, Irvine, CA, USA
- Department of Biomedical Engineering, University of California Irvine, Irvine, CA, USA
- Chao Family Comprehensive Cancer Center, UC Health, Orange, CA, USA
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden, Dresden, Germany.
- Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumor Diseases, University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
4
|
Zheng K, Duan J, Wang R, Chen H, He H, Zheng X, Zhao Z, Jing B, Zhang Y, Liu S, Xie D, Lin Y, Sun Y, Zhang N, Cai M. Deep learning model with pathological knowledge for detection of colorectal neuroendocrine tumor. Cell Rep Med 2024; 5:101785. [PMID: 39413732 DOI: 10.1016/j.xcrm.2024.101785] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2024] [Revised: 08/19/2024] [Accepted: 09/19/2024] [Indexed: 10/18/2024]
Abstract
Colorectal neuroendocrine tumors (NETs) differ significantly from colorectal carcinoma (CRC) in terms of treatment strategy and prognosis, necessitating a cost-effective approach for accurate discrimination. Here, we propose an approach for distinguishing between colorectal NET and CRC based on pathological images by utilizing pathological prior information to facilitate the generation of robust slide-level features. By calculating the similarity between morphological descriptions and patches, our approach selects only 2% of the diagnostically relevant patches for both training and inference, achieving an area under the receiver operating characteristic curve (AUROC) of 0.9974 on the internal dataset, and AUROCs of 0.9724 and 0.9513 on two external datasets. Our model effectively identifies NETs from CRCs, reducing unnecessary immunohistochemical tests and enhancing the precise treatment for patients with colorectal tumors. Our approach also enables researchers to investigate methods with high accuracy and low computational complexity, thereby advancing the application of artificial intelligence in clinical settings.
Collapse
Affiliation(s)
- Ke Zheng
- Department of Pathology, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, Guangzhou 510060, China
| | - Jinling Duan
- Department of Pathology, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, Guangzhou 510060, China
| | - Ruixuan Wang
- School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China
| | - Haohua Chen
- Artificial Intelligence Laboratory, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, Guangzhou 510060, China
| | - Haiyang He
- Department of Pathology, The First Affiliated Hospital, Sun Yat-sen University, Guangzhou 510080, China
| | - Xueyi Zheng
- Department of Pathology, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, Guangzhou 510060, China
| | - Zihan Zhao
- Department of Pathology, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, Guangzhou 510060, China
| | - Bingzhong Jing
- Artificial Intelligence Laboratory, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, Guangzhou 510060, China
| | - Yuqian Zhang
- Electrical Engineering & Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Shasha Liu
- Department of Pathology, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300000, China
| | - Dan Xie
- Department of Pathology, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, Guangzhou 510060, China
| | - Yuan Lin
- Department of Pathology, The First Affiliated Hospital, Sun Yat-sen University, Guangzhou 510080, China.
| | - Yan Sun
- Department of Pathology, Tianjin Medical University Cancer Institute and Hospital, Tianjin 300000, China.
| | - Ning Zhang
- Department of Gastroenterology and Hepatology, The First Affiliated Hospital, Sun Yat-Sen University, Guangzhou 510060, China.
| | - Muyan Cai
- Department of Pathology, State Key Laboratory of Oncology in South China, Guangdong Provincial Clinical Research Center for Cancer, Sun Yat-sen University Cancer Center, Guangzhou 510060, China.
| |
Collapse
|
5
|
Cai C, Shi Q, Li J, Jiao Y, Xu A, Zhou Y, Wang X, Peng C, Zhang X, Cui X, Chen J, Xu J, Sun Q. Pathologist-level diagnosis of ulcerative colitis inflammatory activity level using an automated histological grading method. Int J Med Inform 2024; 192:105648. [PMID: 39396418 DOI: 10.1016/j.ijmedinf.2024.105648] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2024] [Revised: 09/18/2024] [Accepted: 10/06/2024] [Indexed: 10/15/2024]
Abstract
BACKGROUND AND AIMS Inflammatory bowel disease (IBD) is a global disease that is evolving with increasing incidence. However, there are few works on computationally assisted diagnosis of IBD based on pathological images. Therefore, based on the UK and Chinese IBD diagnostic guidelines, our study established an artificial intelligence-assisted diagnostic system for histologic grading of inflammatory activity in ulcerative colitis (UC). METHODS We proposed an efficient deep-learning (DL) method for grading inflammatory activity in whole-slide images (WSIs) of UC pathology. Our model was constructed using 603 UC WSIs from Nanjing Drum Tower Hospital for model train set and internal test set. We collected 212 UC WSIs from Zhujiang Hospital as an external test set. Initially, the pre-trained ResNet50 model on the ImageNet dataset was employed to extract image patch features from UC patients. Subsequently, a multi-instance learning (MIL) approach with embedded self-attention was utilized to aggregate tissue image patch features, representing the entire WSI. Finally, the model was trained based on the aggregated features and WSI annotations provided by senior gastrointestinal pathologists to predict the level of inflammatory activity in UC WSIs. RESULTS In the task of distinguishing the presence or absence of inflammatory activity, the Area Under Curve (AUC) value in the internal test set is 0.863 (95% confidence interval [CI] 0.829, 0.898), with a sensitivity of 0.913 (95% [CI] 0.866, 0.961), and specificity of 0.816 (95% [CI] 0.771, 0.861). The AUC in the external test set is 0.947 (95% confidence interval [CI] 0.939, 0.955), with a sensitivity of 0.889 (905% [CI] 0.837, 0.940), and specificity of 0.858 (95% [CI] 0.777, 0.939). For distinguishing different levels of inflammatory activity in UC, the average Macro-AUC in the internal test set and the external test set are 0.827 (95% [CI] 0.803, 0.850) and 0.908 (95% [CI] 0.882, 0.935). the average Micro-AUC in the internal test set and the external test set are 0.816 (95% [CI] 0.792, 0.840) and 0.898 (95% [CI] 0.869, 0.926). CONCLUSIONS Comparative analysis with diagnoses made by pathologists at different expertise levels revealed that the algorithm reached a proficiency comparable to the pathologist with 5 years of experience. Furthermore, our algorithm performed superior to other MIL algorithms.
Collapse
Affiliation(s)
- Chengfei Cai
- School of Automation, Nanjing University of Information Science and Technology, Nanjing 21004, Jiangsu Province, China; Jiangsu Key Laboratory of Intelligent Medical Image Computing, School of Future Technology, Nanjing University of Information Science and Technology, Nanjing 21004, Jiangsu Province, China; College of Information Engineering, Taizhou University, Taizhou 225300, Jiangsu Province, China
| | - Qianyun Shi
- Department of Pathology, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing 210008, Jiangsu Province, China
| | - Jun Li
- Jiangsu Key Laboratory of Intelligent Medical Image Computing, School of Future Technology, Nanjing University of Information Science and Technology, Nanjing 21004, Jiangsu Province, China
| | - Yiping Jiao
- Jiangsu Key Laboratory of Intelligent Medical Image Computing, School of Future Technology, Nanjing University of Information Science and Technology, Nanjing 21004, Jiangsu Province, China
| | - Andi Xu
- Department of Pathology, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing 210008, Jiangsu Province, China
| | - Yangshu Zhou
- Department of Pathology, Zhujiang Hospital of Southern Medical University, Guangzhou 510280, Guangdong Province, China
| | - Xiangxue Wang
- Jiangsu Key Laboratory of Intelligent Medical Image Computing, School of Future Technology, Nanjing University of Information Science and Technology, Nanjing 21004, Jiangsu Province, China
| | - Chunyan Peng
- Department of Gastroenterology, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing 210008, Jiangsu Province, China
| | - Xiaoqi Zhang
- Department of Gastroenterology, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing 210008, Jiangsu Province, China
| | - Xiaobin Cui
- Department of Pathology, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing 210008, Jiangsu Province, China
| | - Jun Chen
- Department of Pathology, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing 210008, Jiangsu Province, China
| | - Jun Xu
- Jiangsu Key Laboratory of Intelligent Medical Image Computing, School of Future Technology, Nanjing University of Information Science and Technology, Nanjing 21004, Jiangsu Province, China.
| | - Qi Sun
- Department of Pathology, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing 210008, Jiangsu Province, China; Center for Digestive Medicine, Nanjing Drum Tower Hospital, The Affiliated Hospital of Nanjing University Medical School, Nanjing 210008, Jiangsu Province, China.
| |
Collapse
|
6
|
Loeffler CML, El Nahhas OSM, Muti HS, Carrero ZI, Seibel T, van Treeck M, Cifci D, Gustav M, Bretz K, Gaisa NT, Lehmann KV, Leary A, Selenica P, Reis-Filho JS, Ortiz-Bruechle N, Kather JN. Prediction of homologous recombination deficiency from routine histology with attention-based multiple instance learning in nine different tumor types. BMC Biol 2024; 22:225. [PMID: 39379982 PMCID: PMC11462727 DOI: 10.1186/s12915-024-02022-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2024] [Accepted: 09/26/2024] [Indexed: 10/10/2024] Open
Abstract
BACKGROUND Homologous recombination deficiency (HRD) is recognized as a pan-cancer predictive biomarker that potentially indicates who could benefit from treatment with PARP inhibitors (PARPi). Despite its clinical significance, HRD testing is highly complex. Here, we investigated in a proof-of-concept study whether Deep Learning (DL) can predict HRD status solely based on routine hematoxylin & eosin (H&E) histology images across nine different cancer types. METHODS We developed a deep learning pipeline with attention-weighted multiple instance learning (attMIL) to predict HRD status from histology images. As part of our approach, we calculated a genomic scar HRD score by combining loss of heterozygosity (LOH), telomeric allelic imbalance (TAI), and large-scale state transitions (LST) from whole genome sequencing (WGS) data of n = 5209 patients across two independent cohorts. The model's effectiveness was evaluated using the area under the receiver operating characteristic curve (AUROC), focusing on its accuracy in predicting genomic HRD against a clinically recognized cutoff value. RESULTS Our study demonstrated the predictability of genomic HRD status in endometrial, pancreatic, and lung cancers reaching cross-validated AUROCs of 0.79, 0.58, and 0.66, respectively. These predictions generalized well to an external cohort, with AUROCs of 0.93, 0.81, and 0.73. Moreover, a breast cancer-trained image-based HRD classifier yielded an AUROC of 0.78 in the internal validation cohort and was able to predict HRD in endometrial, prostate, and pancreatic cancer with AUROCs of 0.87, 0.84, and 0.67, indicating that a shared HRD-like phenotype occurs across these tumor entities. CONCLUSIONS This study establishes that HRD can be directly predicted from H&E slides using attMIL, demonstrating its applicability across nine different tumor types.
Collapse
Affiliation(s)
- Chiara Maria Lavinia Loeffler
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine I, Faculty of Medicine Carl Gustav Carus, University Hospitaland, Technische Universität Dresden , Dresden, Germany
| | - Omar S M El Nahhas
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Hannah Sophie Muti
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department for Visceral, Thoracic and Vascular Surgery, University Hospital Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Zunamys I Carrero
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Tobias Seibel
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Marko van Treeck
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Didem Cifci
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Marco Gustav
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Kevin Bretz
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Nadine T Gaisa
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
- Joint Research Center Computational Biomedicine, University Hospital RWTH Aachen, Aachen, Germany
| | - Kjong-Van Lehmann
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
- Joint Research Center Computational Biomedicine, University Hospital RWTH Aachen, Aachen, Germany
- Center for Integrated Oncology, Aachen Bonn Cologne Duesseldorf (CIO ABCD), Duesseldorf, Germany
- Cancer Research Center Cologne-Essen, University Hospital Cologne, Cologne, Germany
| | - Alexandra Leary
- Gynecological Cancer Unit, Department of Medicine, Institut Gustave Roussy, Villejuif, France
| | - Pier Selenica
- Experimental Pathology, Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Jorge S Reis-Filho
- Experimental Pathology, Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Nadina Ortiz-Bruechle
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
- Center for Integrated Oncology, Aachen Bonn Cologne Duesseldorf (CIO ABCD), Duesseldorf, Germany
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
- Department of Medicine I, Faculty of Medicine Carl Gustav Carus, University Hospitaland, Technische Universität Dresden , Dresden, Germany.
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
7
|
Lu MY, Chen B, Williamson DFK, Chen RJ, Zhao M, Chow AK, Ikemura K, Kim A, Pouli D, Patel A, Soliman A, Chen C, Ding T, Wang JJ, Gerber G, Liang I, Le LP, Parwani AV, Weishaupt LL, Mahmood F. A multimodal generative AI copilot for human pathology. Nature 2024; 634:466-473. [PMID: 38866050 PMCID: PMC11464372 DOI: 10.1038/s41586-024-07618-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2023] [Accepted: 05/28/2024] [Indexed: 06/14/2024]
Abstract
Computational pathology1,2 has witnessed considerable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders3,4. However, despite the explosive growth of generative artificial intelligence (AI), there have been few studies on building general-purpose multimodal AI assistants and copilots5 tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology. We built PathChat by adapting a foundational vision encoder for pathology, combining it with a pretrained large language model and fine-tuning the whole system on over 456,000 diverse visual-language instructions consisting of 999,202 question and answer turns. We compare PathChat with several multimodal vision-language AI assistants and GPT-4V, which powers the commercially available multimodal general-purpose AI assistant ChatGPT-4 (ref. 6). PathChat achieved state-of-the-art performance on multiple-choice diagnostic questions from cases with diverse tissue origins and disease models. Furthermore, using open-ended questions and human expert evaluation, we found that overall PathChat produced more accurate and pathologist-preferable responses to diverse queries related to pathology. As an interactive vision-language AI copilot that can flexibly handle both visual and natural language inputs, PathChat may potentially find impactful applications in pathology education, research and human-in-the-loop clinical decision-making.
Collapse
Affiliation(s)
- Ming Y Lu
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT), Cambridge, MA, USA
| | - Bowen Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Drew F K Williamson
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
| | - Richard J Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
| | - Melissa Zhao
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Aaron K Chow
- Department of Pathology, Wexner Medical Center, Ohio State University, Columbus, OH, USA
| | - Kenji Ikemura
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Ahrong Kim
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Pusan National University, Busan, South Korea
| | - Dimitra Pouli
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Ankush Patel
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA
| | - Amr Soliman
- Department of Pathology, Wexner Medical Center, Ohio State University, Columbus, OH, USA
| | - Chengkuan Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Tong Ding
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Judy J Wang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Georg Gerber
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Ivy Liang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Long Phi Le
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Anil V Parwani
- Department of Pathology, Wexner Medical Center, Ohio State University, Columbus, OH, USA
| | - Luca L Weishaupt
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA.
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
8
|
Wang X, Zhao J, Marostica E, Yuan W, Jin J, Zhang J, Li R, Tang H, Wang K, Li Y, Wang F, Peng Y, Zhu J, Zhang J, Jackson CR, Zhang J, Dillon D, Lin NU, Sholl L, Denize T, Meredith D, Ligon KL, Signoretti S, Ogino S, Golden JA, Nasrallah MP, Han X, Yang S, Yu KH. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature 2024; 634:970-978. [PMID: 39232164 DOI: 10.1038/s41586-024-07894-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2023] [Accepted: 08/01/2024] [Indexed: 09/06/2024]
Abstract
Histopathology image evaluation is indispensable for cancer diagnoses and subtype classification. Standard artificial intelligence methods for histopathology image analyses have focused on optimizing specialized models for each diagnostic task1,2. Although such methods have achieved some success, they often have limited generalizability to images generated by different digitization protocols or samples collected from different populations3. Here, to address this challenge, we devised the Clinical Histopathology Imaging Evaluation Foundation (CHIEF) model, a general-purpose weakly supervised machine learning framework to extract pathology imaging features for systematic cancer evaluation. CHIEF leverages two complementary pretraining methods to extract diverse pathology representations: unsupervised pretraining for tile-level feature identification and weakly supervised pretraining for whole-slide pattern recognition. We developed CHIEF using 60,530 whole-slide images spanning 19 anatomical sites. Through pretraining on 44 terabytes of high-resolution pathology imaging datasets, CHIEF extracted microscopic representations useful for cancer cell detection, tumour origin identification, molecular profile characterization and prognostic prediction. We successfully validated CHIEF using 19,491 whole-slide images from 32 independent slide sets collected from 24 hospitals and cohorts internationally. Overall, CHIEF outperformed the state-of-the-art deep learning methods by up to 36.1%, showing its ability to address domain shifts observed in samples from diverse populations and processed by different slide preparation methods. CHIEF provides a generalizable foundation for efficient digital pathology evaluation for patients with cancer.
Collapse
Affiliation(s)
- Xiyue Wang
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA
| | - Junhan Zhao
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Eliana Marostica
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Division of Health Sciences and Technology, Harvard-Massachusetts Institute of Technology, Boston, MA, USA
| | - Wei Yuan
- College of Biomedical Engineering, Sichuan University, Chengdu, China
| | - Jietian Jin
- Department of Pathology, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Jiayu Zhang
- College of Biomedical Engineering, Sichuan University, Chengdu, China
| | - Ruijiang Li
- Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA
| | - Hongping Tang
- Department of Pathology, Shenzhen Maternity & Child Healthcare Hospital, Shenzhen, China
| | - Kanran Wang
- Department of Radiation Oncology, Chongqing University Cancer Hospital, Chongqing, China
| | - Yu Li
- Department of Pathology, Chongqing University Cancer Hospital, Chongqing, China
| | - Fang Wang
- Department of Pathology, The Affiliated Yantai Yuhuangding Hospital of Qingdao University, Yantai, China
| | - Yulong Peng
- Department of Pathology, The First Affiliated Hospital of Jinan University, Guangzhou, China
| | - Junyou Zhu
- Department of Burn, The First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
| | - Jing Zhang
- College of Biomedical Engineering, Sichuan University, Chengdu, China
| | - Christopher R Jackson
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Department of Pathology and Laboratory Medicine, Pennsylvania State University, Hummelstown, PA, USA
- Department of Pathology, Massachusetts General Hospital, Boston, MA, USA
| | | | - Deborah Dillon
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA
| | - Nancy U Lin
- Department of Medical Oncology, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Lynette Sholl
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA
- Department of Pathology, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Thomas Denize
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA
- Department of Pathology, Dana-Farber Cancer Institute, Boston, MA, USA
| | - David Meredith
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA
| | - Keith L Ligon
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA
- Department of Pathology, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Sabina Signoretti
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA
- Department of Pathology, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Shuji Ogino
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Broad Institute of MIT and Harvard, Cambridge, MA, USA
| | - Jeffrey A Golden
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA
- Department of Pathology, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - MacLean P Nasrallah
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA
| | | | - Sen Yang
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
- Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, USA.
| | - Kun-Hsing Yu
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
- Department of Pathology, Brigham and Women's Hospital, Boston, MA, USA.
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
9
|
Vorontsov E, Bozkurt A, Casson A, Shaikovski G, Zelechowski M, Severson K, Zimmermann E, Hall J, Tenenholtz N, Fusi N, Yang E, Mathieu P, van Eck A, Lee D, Viret J, Robert E, Wang YK, Kunz JD, Lee MCH, Bernhard JH, Godrich RA, Oakley G, Millar E, Hanna M, Wen H, Retamero JA, Moye WA, Yousfi R, Kanan C, Klimstra DS, Rothrock B, Liu S, Fuchs TJ. A foundation model for clinical-grade computational pathology and rare cancers detection. Nat Med 2024; 30:2924-2935. [PMID: 39039250 PMCID: PMC11485232 DOI: 10.1038/s41591-024-03141-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2024] [Accepted: 06/19/2024] [Indexed: 07/24/2024]
Abstract
The analysis of histopathology images with artificial intelligence aims to enable clinical decision support systems and precision medicine. The success of such applications depends on the ability to model the diverse patterns observed in pathology images. To this end, we present Virchow, the largest foundation model for computational pathology to date. In addition to the evaluation of biomarker prediction and cell identification, we demonstrate that a large foundation model enables pan-cancer detection, achieving 0.95 specimen-level area under the (receiver operating characteristic) curve across nine common and seven rare cancers. Furthermore, we show that with less training data, the pan-cancer detector built on Virchow can achieve similar performance to tissue-specific clinical-grade models in production and outperform them on some rare variants of cancer. Virchow's performance gains highlight the value of a foundation model and open possibilities for many high-impact applications with limited amounts of labeled training data.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | | | | | - Ellen Yang
- Memorial Sloan Kettering Cancer Center, New York, NY, US
| | | | | | | | | | | | | | | | | | | | | | | | - Ewan Millar
- NSW Health Pathology, St George Hospital, Sydney, New South Wales, Australia
| | - Matthew Hanna
- Memorial Sloan Kettering Cancer Center, New York, NY, US
| | - Hannah Wen
- Memorial Sloan Kettering Cancer Center, New York, NY, US
| | | | | | | | | | | | | | | | | |
Collapse
|
10
|
Prezja F, Annala L, Kiiskinen S, Lahtinen S, Ojala T, Ruusuvuori P, Kuopio T. Improving performance in colorectal cancer histology decomposition using deep and ensemble machine learning. Heliyon 2024; 10:e37561. [PMID: 39309850 PMCID: PMC11415691 DOI: 10.1016/j.heliyon.2024.e37561] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2024] [Accepted: 09/05/2024] [Indexed: 09/25/2024] Open
Abstract
In routine colorectal cancer management, histologic samples stained with hematoxylin and eosin are commonly used. Nonetheless, their potential for defining objective biomarkers for patient stratification and treatment selection is still being explored. The current gold standard relies on expensive and time-consuming genetic tests. However, recent research highlights the potential of convolutional neural networks (CNNs) to facilitate the extraction of clinically relevant biomarkers from these readily available images. These CNN-based biomarkers can predict patient outcomes comparably to golden standards, with the added advantages of speed, automation, and minimal cost. The predictive potential of CNN-based biomarkers fundamentally relies on the ability of CNNs to accurately classify diverse tissue types from whole slide microscope images. Consequently, enhancing the accuracy of tissue class decomposition is critical to amplifying the prognostic potential of imaging-based biomarkers. This study introduces a hybrid deep transfer learning and ensemble machine learning model that improves upon previous approaches, including a transformer and neural architecture search baseline for this task. We employed a pairing of the EfficientNetV2 architecture with a random forest classification head. Our model achieved 96.74% accuracy (95% CI: 96.3%-97.1%) on the external test set and 99.89% on the internal test set. Recognizing the potential of these models in the task, we have made them publicly available.
Collapse
Affiliation(s)
- Fabi Prezja
- University of Jyväskylä, Faculty of Information Technology, Jyväskylä, 40014, Finland
| | - Leevi Annala
- University of Helsinki, Faculty of Science, Department of Computer Science, Helsinki, Finland
- University of Helsinki, Faculty of Agriculture and Forestry, Department of Food and Nutrition, Helsinki, Finland
| | - Sampsa Kiiskinen
- University of Jyväskylä, Faculty of Information Technology, Jyväskylä, 40014, Finland
| | - Suvi Lahtinen
- University of Jyväskylä, Faculty of Information Technology, Jyväskylä, 40014, Finland
- University of Jyväskylä, Faculty of Mathematics and Science, Department of Biological and Environmental Science, Jyväskylä, 40014, Finland
| | - Timo Ojala
- University of Jyväskylä, Faculty of Information Technology, Jyväskylä, 40014, Finland
| | - Pekka Ruusuvuori
- University of Turku, Institute of Biomedicine, Cancer Research Unit, Turku, 20014, Finland
- Turku University Hospital, FICAN West Cancer Centre, Turku, 20521, Finland
| | - Teijo Kuopio
- University of Jyväskylä, Department of Biological and Environmental Science, Jyväskylä, 40014, Finland
- Hospital Nova of Central Finland, Department of Pathology, Jyväskylä, 40620, Finland
| |
Collapse
|
11
|
Brooks JA, Kallenbach M, Radu IP, Berzigotti A, Dietrich CF, Kather JN, Luedde T, Seraphin TP. Artificial Intelligence for Contrast-Enhanced Ultrasound of the Liver: A Systematic Review. Digestion 2024:1-18. [PMID: 39312896 DOI: 10.1159/000541540] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/04/2024] [Accepted: 09/18/2024] [Indexed: 09/25/2024]
Abstract
INTRODUCTION The research field of artificial intelligence (AI) in medicine and especially in gastroenterology is rapidly progressing with the first AI tools entering routine clinical practice, for example, in colorectal cancer screening. Contrast-enhanced ultrasound (CEUS) is a highly reliable, low-risk, and low-cost diagnostic modality for the examination of the liver. However, doctors need many years of training and experience to master this technique and, despite all efforts to standardize CEUS, it is often believed to contain significant interrater variability. As has been shown for endoscopy, AI holds promise to support examiners at all training levels in their decision-making and efficiency. METHODS In this systematic review, we analyzed and compared original research studies applying AI methods to CEUS examinations of the liver published between January 2010 and February 2024. We performed a structured literature search on PubMed, Web of Science, and IEEE. Two independent reviewers screened the articles and subsequently extracted relevant methodological features, e.g., cohort size, validation process, machine learning algorithm used, and indicative performance measures from the included articles. RESULTS We included 41 studies with most applying AI methods for classification tasks related to focal liver lesions. These included distinguishing benign versus malignant or classifying the entity itself, while a few studies tried to classify tumor grading, microvascular invasion status, or response to transcatheter arterial chemoembolization directly from CEUS. Some articles tried to segment or detect focal liver lesions, while others aimed to predict survival and recurrence after ablation. The majority (25/41) of studies used hand-picked and/or annotated images as data input to their models. We observed mostly good to high reported model performances with accuracies ranging between 58.6% and 98.9%, while noticing a general lack of external validation. CONCLUSION Even though multiple proof-of-concept studies for the application of AI methods to CEUS examinations of the liver exist and report high performance, more prospective, externally validated, and multicenter research is needed to bring such algorithms from desk to bedside.
Collapse
Affiliation(s)
- James A Brooks
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Dusseldorf, Medical Faculty at Heinrich-Heine-University, Dusseldorf, Germany
| | - Michael Kallenbach
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Dusseldorf, Medical Faculty at Heinrich-Heine-University, Dusseldorf, Germany
| | - Iuliana-Pompilia Radu
- Department for Visceral Surgery and Medicine, Inselspital, University of Bern, Bern, Switzerland
| | - Annalisa Berzigotti
- Department for Visceral Surgery and Medicine, Inselspital, University of Bern, Bern, Switzerland
| | - Christoph F Dietrich
- Department Allgemeine Innere Medizin (DAIM), Kliniken Hirslanden Beau Site, Salem and Permanence, Bern, Switzerland
| | - Jakob N Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Tom Luedde
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Dusseldorf, Medical Faculty at Heinrich-Heine-University, Dusseldorf, Germany
| | - Tobias P Seraphin
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Dusseldorf, Medical Faculty at Heinrich-Heine-University, Dusseldorf, Germany
| |
Collapse
|
12
|
Nowak M, Jabbar F, Rodewald AK, Gneo L, Tomasevic T, Harkin A, Iveson T, Saunders M, Kerr R, Oein K, Maka N, Hay J, Edwards J, Tomlinson I, Sansom O, Kelly C, Pezzella F, Kerr D, Easton A, Domingo E, Koelzer VH, Church DN. Single-cell AI-based detection and prognostic and predictive value of DNA mismatch repair deficiency in colorectal cancer. Cell Rep Med 2024; 5:101727. [PMID: 39293403 DOI: 10.1016/j.xcrm.2024.101727] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Revised: 05/16/2024] [Accepted: 08/15/2024] [Indexed: 09/20/2024]
Abstract
Testing for DNA mismatch repair deficiency (MMRd) is recommended for all colorectal cancers (CRCs). Automating this would enable precision medicine, particularly if providing information on etiology not captured by deep learning (DL) methods. We present AIMMeR, an AI-based method for determination of mismatch repair (MMR) protein expression at a single-cell level in routine pathology samples. AIMMeR shows an area under the receiver-operator curve (AUROC) of 0.98, and specificity of ≥75% at 98% sensitivity against pathologist ground truth in stage II/III in two trial cohorts, with positive predictive value of ≥98% for the commonest pattern of somatic MMRd. Lower agreement with microsatellite instability (MSI) testing (AUROC 0.86) reflects discordance between MMR and MSI PCR rather than AIMMeR misclassification. Analysis of the SCOT trial confirms MMRd prognostic value in oxaliplatin-treated patients; while MMRd does not predict differential benefit of chemotherapy duration, it correlates with difference in relapse by regimen (PInteraction = 0.04). AIMMeR may help reduce pathologist workload and streamline diagnostics in CRC.
Collapse
Affiliation(s)
- Marta Nowak
- Department of Pathology and Molecular Pathology, Zurich, Zurich, Switzerland
| | - Faiz Jabbar
- Cancer Genomics and Immunology Group, The Wellcome Centre for Human Genetics, University of Oxford, Roosevelt Drive, Oxford OX3 7BN, UK
| | - Ann-Katrin Rodewald
- Department of Pathology and Molecular Pathology, Zurich, Zurich, Switzerland
| | - Luciana Gneo
- Cancer Genomics and Immunology Group, The Wellcome Centre for Human Genetics, University of Oxford, Roosevelt Drive, Oxford OX3 7BN, UK
| | - Tijana Tomasevic
- Cancer Genomics and Immunology Group, The Wellcome Centre for Human Genetics, University of Oxford, Roosevelt Drive, Oxford OX3 7BN, UK
| | - Andrea Harkin
- CRUK Glasgow Clinical Trials Unit, University of Glasgow, Glasgow, UK
| | - Tim Iveson
- University of Southampton, Southampton, UK
| | | | - Rachel Kerr
- Department of Oncology, University of Oxford, Oxford, UK
| | - Karin Oein
- Glasgow Tissue Research Facility, University of Glasgow, Queen Elizabeth University Hospital, Glasgow, UK
| | - Noori Maka
- Glasgow Tissue Research Facility, University of Glasgow, Queen Elizabeth University Hospital, Glasgow, UK
| | - Jennifer Hay
- Glasgow Tissue Research Facility, University of Glasgow, Queen Elizabeth University Hospital, Glasgow, UK
| | - Joanne Edwards
- School of Cancer Sciences, University of Glasgow, Glasgow, UK
| | - Ian Tomlinson
- Department of Oncology, University of Oxford, Oxford, UK
| | - Owen Sansom
- CRUK Beatson Institute of Cancer Research, Garscube Estate, Glasgow, UK
| | - Caroline Kelly
- CRUK Glasgow Clinical Trials Unit, University of Glasgow, Glasgow, UK
| | | | - David Kerr
- Nuffield Department of Clinical and Laboratory Sciences, University of Oxford, Oxford, UK
| | | | - Enric Domingo
- Department of Oncology, University of Oxford, Oxford, UK
| | - Viktor H Koelzer
- Department of Pathology and Molecular Pathology, Zurich, Zurich, Switzerland; Department of Oncology, University of Oxford, Oxford, UK; Nuffield Department of Medicine, University of Oxford, Oxford, UK; Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
| | - David N Church
- Cancer Genomics and Immunology Group, The Wellcome Centre for Human Genetics, University of Oxford, Roosevelt Drive, Oxford OX3 7BN, UK; Oxford NIHR Comprehensive Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK.
| |
Collapse
|
13
|
Hyeon D, Kim Y, Hwang Y, Bae JM, Kang GH, Kim K. Deep learning-based histological predictions of chromosomal instability in colorectal cancer. Am J Cancer Res 2024; 14:4495-4505. [PMID: 39417190 PMCID: PMC11477831 DOI: 10.62347/jynd6488] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2024] [Accepted: 08/23/2024] [Indexed: 10/19/2024] Open
Abstract
Colorectal cancer (CRC) is a lethal malignancy and a leading cause of cancer-related mortality worldwide. Chromosomal instability (CIN) is a key driver of genomic instability in CRC and is characterized by aneuploidy and somatic copy-number alterations. This study aimed to predict CIN in CRC using histological data from whole slide images (WSIs). CRC samples from TCGA were analyzed, with tumor regions segmented into tiles and nuclei for feature extraction using convolutional neural network (CNN) and morphologic analysis. Binary classification models were developed to distinguish high and low aneuploidy scores (AS) based on slide-level features. The analysis included 313 patients with 315 WSIs, resulting in over 350,000 tumor tiles and nearly 2.7 million tumor cell nuclei. The ResNet18-SSL model, pre-trained on histopathological images, demonstrated superior accuracy in tile-based AS prediction, while DenseNet121 excelled in nucleus-based prediction. Combining CNN-based and morphological features enhanced the classification accuracy of nucleus-based predictions. Additionally, significant correlations were observed between morphological features and copy-number signatures. Unsupervised clustering of nuclear features revealed that distinct groups are significantly correlated with CIN and TP53 mutations. This study underscores the potential of histological features from WSIs to predict CIN in CRC samples. Nuclear feature analysis, combined with deep-learning techniques, offers a robust method for CIN prediction, highlighting the importance of further research into the relationships between histological and molecular phenotypes.
Collapse
Affiliation(s)
- Dongwoo Hyeon
- Institute of Biomedical Research, Seoul National University HospitalSeoul, South Korea
| | - Younghoon Kim
- Department of Pathology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of KoreaSeoul, South Korea
| | - Yaeeun Hwang
- Department of Veterinary Medicine, Seoul National UniversitySeoul, South Korea
| | - Jeong Mo Bae
- Department of Pathology, Seoul National University HospitalSeoul, South Korea
| | - Gyeong Hoon Kang
- Department of Pathology, College of Medicine, Seoul National UniversitySeoul, South Korea
| | - Kwangsoo Kim
- Department of Transdisciplinary Medicine, Institute of Convergence Medicine with Innovative Technology, Seoul National University HospitalSeoul, South Korea
- Department of Medicine, Seoul National UniversitySeoul, South Korea
| |
Collapse
|
14
|
Wang S, Pan J, Zhang X, Li Y, Liu W, Lin R, Wang X, Kang D, Li Z, Huang F, Chen L, Chen J. Towards next-generation diagnostic pathology: AI-empowered label-free multiphoton microscopy. LIGHT, SCIENCE & APPLICATIONS 2024; 13:254. [PMID: 39277586 PMCID: PMC11401902 DOI: 10.1038/s41377-024-01597-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/26/2024] [Revised: 08/04/2024] [Accepted: 08/21/2024] [Indexed: 09/17/2024]
Abstract
Diagnostic pathology, historically dependent on visual scrutiny by experts, is essential for disease detection. Advances in digital pathology and developments in computer vision technology have led to the application of artificial intelligence (AI) in this field. Despite these advancements, the variability in pathologists' subjective interpretations of diagnostic criteria can lead to inconsistent outcomes. To meet the need for precision in cancer therapies, there is an increasing demand for accurate pathological diagnoses. Consequently, traditional diagnostic pathology is evolving towards "next-generation diagnostic pathology", prioritizing on the development of a multi-dimensional, intelligent diagnostic approach. Using nonlinear optical effects arising from the interaction of light with biological tissues, multiphoton microscopy (MPM) enables high-resolution label-free imaging of multiple intrinsic components across various human pathological tissues. AI-empowered MPM further improves the accuracy and efficiency of diagnosis, holding promise for providing auxiliary pathology diagnostic methods based on multiphoton diagnostic criteria. In this review, we systematically outline the applications of MPM in pathological diagnosis across various human diseases, and summarize common multiphoton diagnostic features. Moreover, we examine the significant role of AI in enhancing multiphoton pathological diagnosis, including aspects such as image preprocessing, refined differential diagnosis, and the prognostication of outcomes. We also discuss the challenges and perspectives faced by the integration of MPM and AI, encompassing equipment, datasets, analytical models, and integration into the existing clinical pathways. Finally, the review explores the synergy between AI and label-free MPM to forge novel diagnostic frameworks, aiming to accelerate the adoption and implementation of intelligent multiphoton pathology systems in clinical settings.
Collapse
Affiliation(s)
- Shu Wang
- School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou, 350108, China
- Key Laboratory of OptoElectronic Science and Technology for Medicine of Ministry of Education, Fujian Provincial Key Laboratory of Photonics Technology, Fujian Normal University, Fuzhou, 350007, China
| | - Junlin Pan
- School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou, 350108, China
| | - Xiao Zhang
- College of Computer and Data Science, Fuzhou University, Fuzhou, 350108, China
| | - Yueying Li
- School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou, 350108, China
| | - Wenxi Liu
- College of Computer and Data Science, Fuzhou University, Fuzhou, 350108, China
| | - Ruolan Lin
- Department of Radiology, Fujian Medical University Union Hospital, Fuzhou, 350001, China
| | - Xingfu Wang
- Department of Pathology, The First Affiliated Hospital of Fujian Medical University, Fuzhou, 350005, China
| | - Deyong Kang
- Department of Pathology, Fujian Medical University Union Hospital, Fuzhou, 350001, China
| | - Zhijun Li
- Key Laboratory of OptoElectronic Science and Technology for Medicine of Ministry of Education, Fujian Provincial Key Laboratory of Photonics Technology, Fujian Normal University, Fuzhou, 350007, China
| | - Feng Huang
- School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou, 350108, China.
| | - Liangyi Chen
- New Cornerstone Laboratory, State Key Laboratory of Membrane Biology, Beijing Key Laboratory of Cardiometabolic Molecular Medicine, Institute of Molecular Medicine, National Biomedical Imaging Center, School of Future Technology, Peking University, Beijing, 100091, China.
| | - Jianxin Chen
- Key Laboratory of OptoElectronic Science and Technology for Medicine of Ministry of Education, Fujian Provincial Key Laboratory of Photonics Technology, Fujian Normal University, Fuzhou, 350007, China.
| |
Collapse
|
15
|
Hezi H, Shats D, Gurevich D, Maruvka YE, Freiman M. Exploring the interplay between colorectal cancer subtypes genomic variants and cellular morphology: A deep-learning approach. PLoS One 2024; 19:e0309380. [PMID: 39255280 PMCID: PMC11386451 DOI: 10.1371/journal.pone.0309380] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2024] [Accepted: 08/10/2024] [Indexed: 09/12/2024] Open
Abstract
Molecular subtypes of colorectal cancer (CRC) significantly influence treatment decisions. While convolutional neural networks (CNNs) have recently been introduced for automated CRC subtype identification using H&E stained histopathological images, the correlation between CRC subtype genomic variants and their corresponding cellular morphology expressed by their imaging phenotypes is yet to be fully explored. The goal of this study was to determine such correlations by incorporating genomic variants in CNN models for CRC subtype classification from H&E images. We utilized the publicly available TCGA-CRC-DX dataset, which comprises whole slide images from 360 CRC-diagnosed patients (260 for training and 100 for testing). This dataset also provides information on CRC subtype classifications and genomic variations. We trained CNN models for CRC subtype classification that account for potential correlation between genomic variations within CRC subtypes and their corresponding cellular morphology patterns. We assessed the interplay between CRC subtypes' genomic variations and cellular morphology patterns by evaluating the CRC subtype classification accuracy of the different models in a stratified 5-fold cross-validation experimental setup using the area under the ROC curve (AUROC) and average precision (AP) as the performance metrics. The CNN models that account for potential correlation between genomic variations within CRC subtypes and their cellular morphology pattern achieved superior accuracy compared to the baseline CNN classification model that does not account for genomic variations when using either single-nucleotide-polymorphism (SNP) molecular features (AUROC: 0.824±0.02 vs. 0.761±0.04, p<0.05, AP: 0.652±0.06 vs. 0.58±0.08) or CpG-Island methylation phenotype (CIMP) molecular features (AUROC: 0.834±0.01 vs. 0.787±0.03, p<0.05, AP: 0.687±0.02 vs. 0.64±0.05). Combining the CNN models account for variations in CIMP and SNP further improved classification accuracy (AUROC: 0.847±0.01 vs. 0.787±0.03, p = 0.01, AP: 0.68±0.02 vs. 0.64±0.05). The improved accuracy of CNN models for CRC subtype classification that account for potential correlation between genomic variations within CRC subtypes and their corresponding cellular morphology as expressed by H&E imaging phenotypes may elucidate the biological cues impacting cancer histopathological imaging phenotypes. Moreover, considering CRC subtypes genomic variations has the potential to improve the accuracy of deep-learning models in discerning cancer subtype from histopathological imaging data.
Collapse
Affiliation(s)
- Hadar Hezi
- Faculty of Biomedical Engineering, Technion - Israel Institute of Technology, Haifa, Israel
| | - Daniel Shats
- Faculty of Computer Science, Technion - Israel Institute of Technology, Haifa, Israel
| | - Daniel Gurevich
- Faculty of Biotechnology and Food Engineering, Technion - Israel Institute of Technology, Haifa, Israel
- Lokey Center for Life Science and Engineering, Technion - Israel Institute of Technology, Haifa, Israel
| | - Yosef E Maruvka
- Faculty of Biotechnology and Food Engineering, Technion - Israel Institute of Technology, Haifa, Israel
- Lokey Center for Life Science and Engineering, Technion - Israel Institute of Technology, Haifa, Israel
| | - Moti Freiman
- Faculty of Biomedical Engineering, Technion - Israel Institute of Technology, Haifa, Israel
| |
Collapse
|
16
|
Reitsam NG, Grosser B, Steiner DF, Grozdanov V, Wulczyn E, L'Imperio V, Plass M, Müller H, Zatloukal K, Muti HS, Kather JN, Märkl B. Converging deep learning and human-observed tumor-adipocyte interaction as a biomarker in colorectal cancer. COMMUNICATIONS MEDICINE 2024; 4:163. [PMID: 39147895 PMCID: PMC11327259 DOI: 10.1038/s43856-024-00589-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2023] [Accepted: 08/05/2024] [Indexed: 08/17/2024] Open
Abstract
BACKGROUND Tumor-Adipose-Feature (TAF) as well as SARIFA (Stroma AReactive Invasion Front Areas) are two histologic features/biomarkers linking tumor-associated adipocytes to poor outcomes in colorectal cancer (CRC) patients. Whereas TAF was identified by deep learning (DL) algorithms, SARIFA was established as a human-observed histopathologic biomarker. METHODS To study the overlap between TAF and SARIFA, we performed a systematic pathological review of TAF based on all published image tiles. Additionally, we analyzed the presence/absence of TAF in SARIFA-negative CRC cases to elucidate the biologic and prognostic role of a direct tumor-adipocyte contact. TCGA-CRC gene expression data is investigated to assess the association of FABP4 (fatty-acid binding protein 4) and CD36 (fatty-acid translocase) with both TAF and CRC prognosis. RESULTS By investigating the TAF/SARIFA overlap, we show that many TAF patches correspond to the recently described SARIFA-phenomenon. Even though there is a pronounced morphological and biological overlap, there are differences in the concepts. The presence of TAF in SARIFA-negative CRCs is not associated with poor outcomes in this cohort, potentially highlighting the importance of a direct tumor-adipocyte interaction. Upregulation of FABP4 and CD36 gene expression seem both linked to a poor prognosis in CRC. CONCLUSIONS By proving the substantial overlap between human-observed SARIFA and DL-based TAF as morphologic biomarkers, we demonstrate that linking DL-based image features to independently developed histopathologic biomarkers is a promising tool in the identification of clinically and biologically meaningful biomarkers. Adipocyte-tumor-cell interactions seem to be crucial in CRC, which should be considered as biomarkers for further investigations.
Collapse
Affiliation(s)
- Nic G Reitsam
- Pathology, Medical Faculty, University of Augsburg, Augsburg, Germany.
- Bavarian Cancer Research Center (BZKF), Augsburg, Germany.
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
| | - Bianca Grosser
- Pathology, Medical Faculty, University of Augsburg, Augsburg, Germany
- Bavarian Cancer Research Center (BZKF), Augsburg, Germany
| | | | | | - Ellery Wulczyn
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Vincenzo L'Imperio
- Department of Medicine and Surgery, Pathology, University of Milano-Bicocca, IRCCS (Scientific Institute for Research, Hospitalization and Healthcare) Fondazione San Gerardo dei Tintori, Monza, Italy
| | - Markus Plass
- Medical University of Graz, Diagnostic and Research Institute of Pathology, Graz, Austria
| | - Heimo Müller
- Medical University of Graz, Diagnostic and Research Institute of Pathology, Graz, Austria
| | - Kurt Zatloukal
- Medical University of Graz, Diagnostic and Research Institute of Pathology, Graz, Austria
| | - Hannah S Muti
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital Carl Gustav Carus Dresden, Dresden, Germany
| | - Jakob N Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
- Department of Medicine I, University Hospital Dresden, Dresden, Germany
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Bruno Märkl
- Pathology, Medical Faculty, University of Augsburg, Augsburg, Germany
- Bavarian Cancer Research Center (BZKF), Augsburg, Germany
| |
Collapse
|
17
|
Hölscher DL, Bülow RD. Decoding pathology: the role of computational pathology in research and diagnostics. Pflugers Arch 2024:10.1007/s00424-024-03002-2. [PMID: 39095655 DOI: 10.1007/s00424-024-03002-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2024] [Revised: 07/25/2024] [Accepted: 07/25/2024] [Indexed: 08/04/2024]
Abstract
Traditional histopathology, characterized by manual quantifications and assessments, faces challenges such as low-throughput and inter-observer variability that hinder the introduction of precision medicine in pathology diagnostics and research. The advent of digital pathology allowed the introduction of computational pathology, a discipline that leverages computational methods, especially based on deep learning (DL) techniques, to analyze histopathology specimens. A growing body of research shows impressive performances of DL-based models in pathology for a multitude of tasks, such as mutation prediction, large-scale pathomics analyses, or prognosis prediction. New approaches integrate multimodal data sources and increasingly rely on multi-purpose foundation models. This review provides an introductory overview of advancements in computational pathology and discusses their implications for the future of histopathology in research and diagnostics.
Collapse
Affiliation(s)
- David L Hölscher
- Department for Nephrology and Clinical Immunology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074, Aachen, Germany
- Institute for Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074, Aachen, Germany
| | - Roman D Bülow
- Institute for Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074, Aachen, Germany.
| |
Collapse
|
18
|
Calderaro J, Žigutytė L, Truhn D, Jaffe A, Kather JN. Artificial intelligence in liver cancer - new tools for research and patient management. Nat Rev Gastroenterol Hepatol 2024; 21:585-599. [PMID: 38627537 DOI: 10.1038/s41575-024-00919-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 03/11/2024] [Indexed: 07/31/2024]
Abstract
Liver cancer has high incidence and mortality globally. Artificial intelligence (AI) has advanced rapidly, influencing cancer care. AI systems are already approved for clinical use in some tumour types (for example, colorectal cancer screening). Crucially, research demonstrates that AI can analyse histopathology, radiology and natural language in liver cancer, and can replace manual tasks and access hidden information in routinely available clinical data. However, for liver cancer, few of these applications have translated into large-scale clinical trials or clinically approved products. Here, we advocate for the incorporation of AI in all stages of liver cancer management. We present a taxonomy of AI approaches in liver cancer, highlighting areas with academic and commercial potential, and outline a policy for AI-based liver cancer management, including interdisciplinary training of researchers, clinicians and patients. The potential of AI in liver cancer is immense, but effort is required to ensure that AI can fulfil expectations.
Collapse
Affiliation(s)
- Julien Calderaro
- Département de Pathologie, Assistance Publique Hôpitaux de Paris, Groupe Hospitalier Henri Mondor, Créteil, France
- Institut Mondor de Recherche Biomédicale, MINT-HEP Mondor Integrative Hepatology, Université Paris Est Créteil, Créteil, France
| | - Laura Žigutytė
- Else Kroener Fresenius Center for Digital Health (EKFZ), Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Ariel Jaffe
- Mayo Clinic, Rochester, MN, USA
- Department of Internal Medicine, Section of Digestive Diseases, Yale School of Medicine, New Haven, CT, USA
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health (EKFZ), Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany.
- Department of Medicine I, University Hospital Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumour Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
19
|
Singh VK, Makhlouf Y, Sarker MMK, Craig S, Baena J, Greene C, Mason L, James JA, Salto-Tellez M, O'Reilly P, Maxwell P. KRASFormer: a fully vision transformer-based framework for predicting KRASgene mutations in histopathological images of colorectal cancer. Biomed Phys Eng Express 2024; 10:055012. [PMID: 38925106 DOI: 10.1088/2057-1976/ad5bed] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2024] [Accepted: 06/26/2024] [Indexed: 06/28/2024]
Abstract
Detecting the Kirsten Rat Sarcoma Virus (KRAS) gene mutation is significant for colorectal cancer (CRC) patients. TheKRASgene encodes a protein involved in the epidermal growth factor receptor (EGFR) signaling pathway, and mutations in this gene can negatively impact the use of monoclonal antibodies in anti-EGFR therapy and affect treatment decisions. Currently, commonly used methods like next-generation sequencing (NGS) identifyKRASmutations but are expensive, time-consuming, and may not be suitable for every cancer patient sample. To address these challenges, we have developedKRASFormer, a novel framework that predictsKRASgene mutations from Haematoxylin and Eosin (H & E) stained WSIs that are widely available for most CRC patients.KRASFormerconsists of two stages: the first stage filters out non-tumor regions and selects only tumour cells using a quality screening mechanism, and the second stage predicts theKRASgene either wildtype' or mutant' using a Vision Transformer-based XCiT method. The XCiT employs cross-covariance attention to capture clinically meaningful long-range representations of textural patterns in tumour tissue andKRASmutant cells. We evaluated the performance of the first stage using an independent CRC-5000 dataset, and the second stage included both The Cancer Genome Atlas colon and rectal cancer (TCGA-CRC-DX) and in-house cohorts. The results of our experiments showed that the XCiT outperformed existing state-of-the-art methods, achieving AUCs for ROC curves of 0.691 and 0.653 on TCGA-CRC-DX and in-house datasets, respectively. Our findings emphasize three key consequences: the potential of using H & E-stained tissue slide images for predictingKRASgene mutations as a cost-effective and time-efficient means for guiding treatment choice with CRC patients; the increase in performance metrics of a Transformer-based model; and the value of the collaboration between pathologists and data scientists in deriving a morphologically meaningful model.
Collapse
Affiliation(s)
- Vivek Kumar Singh
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
- Centre for Biomarkers and Biotherapeutics, Barts Cancer Institute, Queen Mary University of London, London, EC1M 6BQ, United Kingdom
| | - Yasmine Makhlouf
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | | | - Stephanie Craig
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | - Juvenal Baena
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | - Christine Greene
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | - Lee Mason
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | - Jacqueline A James
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
- Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, BT9 7AE, United Kingdom
| | - Manuel Salto-Tellez
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
- Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, BT9 7AE, United Kingdom
- Sonrai Analytics, Belfast, BT9 7AE, United Kingdom
- Cellular Pathology, Belfast Health and Social Care Trust, Belfast City Hospital, Lisburn Road, Belfast BT9 7AB, United Kingdom
| | | | - Perry Maxwell
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| |
Collapse
|
20
|
White BS, Woo XY, Koc S, Sheridan T, Neuhauser SB, Wang S, Evrard YA, Chen L, Foroughi pour A, Landua JD, Mashl RJ, Davies SR, Fang B, Rosa MG, Evans KW, Bailey MH, Chen Y, Xiao M, Rubinstein JC, Sanderson BJ, Lloyd MW, Domanskyi S, Dobrolecki LE, Fujita M, Fujimoto J, Xiao G, Fields RC, Mudd JL, Xu X, Hollingshead MG, Jiwani S, Acevedo S, Davis-Dusenbery BN, Robinson PN, Moscow JA, Doroshow JH, Mitsiades N, Kaochar S, Pan CX, Carvajal-Carmona LG, Welm AL, Welm BE, Govindan R, Li S, Davies MA, Roth JA, Meric-Bernstam F, Xie Y, Herlyn M, Ding L, Lewis MT, Bult CJ, Dean DA, Chuang JH. A Pan-Cancer Patient-Derived Xenograft Histology Image Repository with Genomic and Pathologic Annotations Enables Deep Learning Analysis. Cancer Res 2024; 84:2060-2072. [PMID: 39082680 PMCID: PMC11217732 DOI: 10.1158/0008-5472.can-23-1349] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2023] [Revised: 10/13/2023] [Accepted: 03/27/2024] [Indexed: 08/04/2024]
Abstract
Patient-derived xenografts (PDX) model human intra- and intertumoral heterogeneity in the context of the intact tissue of immunocompromised mice. Histologic imaging via hematoxylin and eosin (H&E) staining is routinely performed on PDX samples, which could be harnessed for computational analysis. Prior studies of large clinical H&E image repositories have shown that deep learning analysis can identify intercellular and morphologic signals correlated with disease phenotype and therapeutic response. In this study, we developed an extensive, pan-cancer repository of >1,000 PDX and paired parental tumor H&E images. These images, curated from the PDX Development and Trial Centers Research Network Consortium, had a range of associated genomic and transcriptomic data, clinical metadata, pathologic assessments of cell composition, and, in several cases, detailed pathologic annotations of neoplastic, stromal, and necrotic regions. The amenability of these images to deep learning was highlighted through three applications: (i) development of a classifier for neoplastic, stromal, and necrotic regions; (ii) development of a predictor of xenograft-transplant lymphoproliferative disorder; and (iii) application of a published predictor of microsatellite instability. Together, this PDX Development and Trial Centers Research Network image repository provides a valuable resource for controlled digital pathology analysis, both for the evaluation of technical issues and for the development of computational image-based methods that make clinical predictions based on PDX treatment studies. Significance: A pan-cancer repository of >1,000 patient-derived xenograft hematoxylin and eosin-stained images will facilitate cancer biology investigations through histopathologic analysis and contributes important model system data that expand existing human histology repositories.
Collapse
Affiliation(s)
- Brian S. White
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | - Xing Yi Woo
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
- Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore, Singapore.
| | - Soner Koc
- Velsera, Charlestown, Massachusetts.
| | - Todd Sheridan
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | | | - Shidan Wang
- University of Texas Southwestern Medical Center, Dallas, Texas.
| | - Yvonne A. Evrard
- Leidos Biomedical Research Inc., Frederick National Laboratory for Cancer Research, Frederick, Maryland.
| | - Li Chen
- Leidos Biomedical Research Inc., Frederick National Laboratory for Cancer Research, Frederick, Maryland.
| | - Ali Foroughi pour
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | | | - R. Jay Mashl
- Washington University School of Medicine, St. Louis, Missouri.
| | | | - Bingliang Fang
- University of Texas MD Anderson Cancer Center, Houston, Texas.
| | | | - Kurt W. Evans
- University of Texas MD Anderson Cancer Center, Houston, Texas.
| | - Matthew H. Bailey
- Simmons Center for Cancer Research, Brigham Young University, Provo, Utah.
| | - Yeqing Chen
- The Wistar Institute, Philadelphia, Pennsylvania.
| | - Min Xiao
- The Wistar Institute, Philadelphia, Pennsylvania.
| | | | | | | | - Sergii Domanskyi
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | | | - Maihi Fujita
- Huntsman Cancer Institute, University of Utah, Salt Lake City, Utah.
| | - Junya Fujimoto
- University of Texas MD Anderson Cancer Center, Houston, Texas.
| | - Guanghua Xiao
- University of Texas Southwestern Medical Center, Dallas, Texas.
| | - Ryan C. Fields
- Washington University School of Medicine, St. Louis, Missouri.
| | | | - Xiaowei Xu
- The Wistar Institute, Philadelphia, Pennsylvania.
| | | | - Shahanawaz Jiwani
- Leidos Biomedical Research Inc., Frederick National Laboratory for Cancer Research, Frederick, Maryland.
| | | | | | | | - Peter N. Robinson
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | | | | | | | | | | | | | - Alana L. Welm
- Huntsman Cancer Institute, University of Utah, Salt Lake City, Utah.
| | - Bryan E. Welm
- Huntsman Cancer Institute, University of Utah, Salt Lake City, Utah.
| | | | - Shunqiang Li
- Washington University School of Medicine, St. Louis, Missouri.
| | | | - Jack A. Roth
- University of Texas MD Anderson Cancer Center, Houston, Texas.
| | | | - Yang Xie
- University of Texas Southwestern Medical Center, Dallas, Texas.
| | | | - Li Ding
- Washington University School of Medicine, St. Louis, Missouri.
| | | | | | | | - Jeffrey H. Chuang
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| |
Collapse
|
21
|
Volinsky-Fremond S, Horeweg N, Andani S, Barkey Wolf J, Lafarge MW, de Kroon CD, Ørtoft G, Høgdall E, Dijkstra J, Jobsen JJ, Lutgens LCHW, Powell ME, Mileshkin LR, Mackay H, Leary A, Katsaros D, Nijman HW, de Boer SM, Nout RA, de Bruyn M, Church D, Smit VTHBM, Creutzberg CL, Koelzer VH, Bosse T. Prediction of recurrence risk in endometrial cancer with multimodal deep learning. Nat Med 2024; 30:1962-1973. [PMID: 38789645 PMCID: PMC11271412 DOI: 10.1038/s41591-024-02993-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 04/11/2024] [Indexed: 05/26/2024]
Abstract
Predicting distant recurrence of endometrial cancer (EC) is crucial for personalized adjuvant treatment. The current gold standard of combined pathological and molecular profiling is costly, hampering implementation. Here we developed HECTOR (histopathology-based endometrial cancer tailored outcome risk), a multimodal deep learning prognostic model using hematoxylin and eosin-stained, whole-slide images and tumor stage as input, on 2,072 patients from eight EC cohorts including the PORTEC-1/-2/-3 randomized trials. HECTOR demonstrated C-indices in internal (n = 353) and two external (n = 160 and n = 151) test sets of 0.789, 0.828 and 0.815, respectively, outperforming the current gold standard, and identified patients with markedly different outcomes (10-year distant recurrence-free probabilities of 97.0%, 77.7% and 58.1% for HECTOR low-, intermediate- and high-risk groups, respectively, by Kaplan-Meier analysis). HECTOR also predicted adjuvant chemotherapy benefit better than current methods. Morphological and genomic feature extraction identified correlates of HECTOR risk groups, some with therapeutic potential. HECTOR improves on the current gold standard and may help delivery of personalized treatment in EC.
Collapse
Affiliation(s)
| | - Nanda Horeweg
- Department of Radiation Oncology, Leiden University Medical Center, Leiden, The Netherlands
| | - Sonali Andani
- Department of Computer Science, ETH Zurich, Zurich, Switzerland
- Department of Pathology and Molecular Pathology, University Hospital, University of Zurich, Zurich, Switzerland
- Swiss Institute of Bioinformatics, Lausanne, Switzerland
| | - Jurriaan Barkey Wolf
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands
| | - Maxime W Lafarge
- Department of Pathology and Molecular Pathology, University Hospital, University of Zurich, Zurich, Switzerland
| | - Cor D de Kroon
- Department of Gynecology and Obstetrics, Leiden University Medical Center, Leiden, The Netherlands
| | - Gitte Ørtoft
- Department of Gynecology, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark
| | - Estrid Høgdall
- Department of Pathology, Herlev University Hospital, Herlev, Denmark
| | - Jouke Dijkstra
- Department of Radiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Jan J Jobsen
- Department of Radiation Oncology, Medisch Spectrum Twente, Enschede, The Netherlands
| | | | - Melanie E Powell
- Department of Clinical Oncology, Barts Health NHS Trust, London, UK
| | - Linda R Mileshkin
- Department of Medical Oncology, Peter MacCallum Cancer Center, Melbourne, Victoria, Australia
| | - Helen Mackay
- Department of Medical Oncology and Hematology, Odette Cancer Center Sunnybrook Health Sciences Center, Toronto, Ontario, Canada
| | - Alexandra Leary
- Department Medical Oncology, Gustave Roussy Institute, Villejuif, France
| | - Dionyssios Katsaros
- Department of Surgical Sciences, Gynecologic Oncology, Città della Salute and S Anna Hospital, University of Turin, Turin, Italy
| | - Hans W Nijman
- Department of Obstetrics and Gynecology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands
| | - Stephanie M de Boer
- Department of Radiation Oncology, Leiden University Medical Center, Leiden, The Netherlands
| | - Remi A Nout
- Department of Radiotherapy, Erasmus MC Cancer Institute, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Marco de Bruyn
- Department of Obstetrics and Gynecology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands
| | - David Church
- Wellcome Centre for Human Genetics, University of Oxford, Oxford, UK
- Oxford NIHR Comprehensive Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Vincent T H B M Smit
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands
| | - Carien L Creutzberg
- Department of Radiation Oncology, Leiden University Medical Center, Leiden, The Netherlands
| | - Viktor H Koelzer
- Department of Pathology and Molecular Pathology, University Hospital, University of Zurich, Zurich, Switzerland
- Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
| | - Tjalling Bosse
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands.
| |
Collapse
|
22
|
Yan C, Sun J, Guan Y, Feng J, Liu H, Liu J. PhiHER2: phenotype-informed weakly supervised model for HER2 status prediction from pathological images. Bioinformatics 2024; 40:i79-i90. [PMID: 38940163 PMCID: PMC11211833 DOI: 10.1093/bioinformatics/btae236] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/29/2024] Open
Abstract
MOTIVATION Human epidermal growth factor receptor 2 (HER2) status identification enables physicians to assess the prognosis risk and determine the treatment schedule for patients. In clinical practice, pathological slides serve as the gold standard, offering morphological information on cellular structure and tumoral regions. Computational analysis of pathological images has the potential to discover morphological patterns associated with HER2 molecular targets and achieve precise status prediction. However, pathological images are typically equipped with high-resolution attributes, and HER2 expression in breast cancer (BC) images often manifests the intratumoral heterogeneity. RESULTS We present a phenotype-informed weakly supervised multiple instance learning architecture (PhiHER2) for the prediction of the HER2 status from pathological images of BC. Specifically, a hierarchical prototype clustering module is designed to identify representative phenotypes across whole slide images. These phenotype embeddings are then integrated into a cross-attention module, enhancing feature interaction and aggregation on instances. This yields a phenotype-based feature space that leverages the intratumoral morphological heterogeneity for HER2 status prediction. Extensive results demonstrate that PhiHER2 captures a better WSI-level representation by the typical phenotype guidance and significantly outperforms existing methods on real-world datasets. Additionally, interpretability analyses of both phenotypes and WSIs provide explicit insights into the heterogeneity of morphological patterns associated with molecular HER2 status. AVAILABILITY AND IMPLEMENTATION Our model is available at https://github.com/lyotvincent/PhiHER2.
Collapse
Affiliation(s)
- Chaoyang Yan
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| | - Jialiang Sun
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| | - Yiming Guan
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| | - Jiuxin Feng
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| | - Hong Liu
- The Second Surgical Department of Breast Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute & Hospital, Tianjin 300060, China
| | - Jian Liu
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| |
Collapse
|
23
|
Reitsam NG, Enke JS, Vu Trung K, Märkl B, Kather JN. Artificial Intelligence in Colorectal Cancer: From Patient Screening over Tailoring Treatment Decisions to Identification of Novel Biomarkers. Digestion 2024; 105:331-344. [PMID: 38865982 PMCID: PMC11457979 DOI: 10.1159/000539678] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Accepted: 06/04/2024] [Indexed: 06/14/2024]
Abstract
BACKGROUND Artificial intelligence (AI) is increasingly entering and transforming not only medical research but also clinical practice. In the last 10 years, new AI methods have enabled computers to perform visual tasks, reaching high performance and thereby potentially supporting and even outperforming human experts. This is in particular relevant for colorectal cancer (CRC), which is the 3rd most common cancer type in general, as along the CRC patient journey many complex visual tasks need to be performed: from endoscopy over imaging to histopathology; the screening, diagnosis, and treatment of CRC involve visual image analysis tasks. SUMMARY In all these clinical areas, AI models have shown promising results by supporting physicians, improving accuracy, and providing new biological insights and biomarkers. By predicting prognostic and predictive biomarkers from routine images/slides, AI models could lead to an improved patient stratification for precision oncology approaches in the near future. Moreover, it is conceivable that AI models, in particular together with innovative techniques such as single-cell or spatial profiling, could help identify novel clinically as well as biologically meaningful biomarkers that could pave the way to new therapeutic approaches. KEY MESSAGES Here, we give a comprehensive overview of AI in colorectal cancer, describing and discussing these developments as well as the next steps which need to be taken to incorporate AI methods more broadly into the clinical care of CRC.
Collapse
Affiliation(s)
- Nic Gabriel Reitsam
- Pathology, Faculty of Medicine, University of Augsburg, Augsburg, Germany,
- Bavarian Cancer Research Center (BZKF), Augsburg, Germany,
| | - Johanna Sophie Enke
- Nuclear Medicine, Faculty of Medicine, University of Augsburg, Augsburg, Germany
| | - Kien Vu Trung
- Division of Gastroenterology, Medical Department II, University of Leipzig Medical Center, Leipzig, Germany
| | - Bruno Märkl
- Pathology, Faculty of Medicine, University of Augsburg, Augsburg, Germany
- Bavarian Cancer Research Center (BZKF), Augsburg, Germany
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Department of Medicine I, University Hospital Dresden, Dresden, Germany
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| |
Collapse
|
24
|
Perez-Lopez R, Ghaffari Laleh N, Mahmood F, Kather JN. A guide to artificial intelligence for cancer researchers. Nat Rev Cancer 2024; 24:427-441. [PMID: 38755439 DOI: 10.1038/s41568-024-00694-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/09/2024] [Indexed: 05/18/2024]
Abstract
Artificial intelligence (AI) has been commoditized. It has evolved from a specialty resource to a readily accessible tool for cancer researchers. AI-based tools can boost research productivity in daily workflows, but can also extract hidden information from existing data, thereby enabling new scientific discoveries. Building a basic literacy in these tools is useful for every cancer researcher. Researchers with a traditional biological science focus can use AI-based tools through off-the-shelf software, whereas those who are more computationally inclined can develop their own AI-based software pipelines. In this article, we provide a practical guide for non-computational cancer researchers to understand how AI-based tools can benefit them. We convey general principles of AI for applications in image analysis, natural language processing and drug discovery. In addition, we give examples of how non-computational researchers can get started on the journey to productively use AI in their own work.
Collapse
Affiliation(s)
- Raquel Perez-Lopez
- Radiomics Group, Vall d'Hebron Institute of Oncology, Vall d'Hebron Barcelona Hospital Campus, Barcelona, Spain
| | - Narmin Ghaffari Laleh
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
- Department of Medicine I, University Hospital Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumour Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
25
|
Niehues JM, Müller-Franzes G, Schirris Y, Wagner SJ, Jendrusch M, Kloor M, Pearson AT, Muti HS, Hewitt KJ, Veldhuizen GP, Zigutyte L, Truhn D, Kather JN. Using histopathology latent diffusion models as privacy-preserving dataset augmenters improves downstream classification performance. Comput Biol Med 2024; 175:108410. [PMID: 38678938 DOI: 10.1016/j.compbiomed.2024.108410] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 03/23/2024] [Accepted: 04/02/2024] [Indexed: 05/01/2024]
Abstract
Latent diffusion models (LDMs) have emerged as a state-of-the-art image generation method, outperforming previous Generative Adversarial Networks (GANs) in terms of training stability and image quality. In computational pathology, generative models are valuable for data sharing and data augmentation. However, the impact of LDM-generated images on histopathology tasks compared to traditional GANs has not been systematically studied. We trained three LDMs and a styleGAN2 model on histology tiles from nine colorectal cancer (CRC) tissue classes. The LDMs include 1) a fine-tuned version of stable diffusion v1.4, 2) a Kullback-Leibler (KL)-autoencoder (KLF8-DM), and 3) a vector quantized (VQ)-autoencoder deploying LDM (VQF8-DM). We assessed image quality through expert ratings, dimensional reduction methods, distribution similarity measures, and their impact on training a multiclass tissue classifier. Additionally, we investigated image memorization in the KLF8-DM and styleGAN2 models. All models provided a high image quality, with the KLF8-DM achieving the best Frechet Inception Distance (FID) and expert rating scores for complex tissue classes. For simpler classes, the VQF8-DM and styleGAN2 models performed better. Image memorization was negligible for both styleGAN2 and KLF8-DM models. Classifiers trained on a mix of KLF8-DM generated and real images achieved a 4% improvement in overall classification accuracy, highlighting the usefulness of these images for dataset augmentation. Our systematic study of generative methods showed that KLF8-DM produces the highest quality images with negligible image memorization. The higher classifier performance in the generatively augmented dataset suggests that this augmentation technique can be employed to enhance histopathology classifiers for various tasks.
Collapse
Affiliation(s)
- Jan M Niehues
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Gustav Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Yoni Schirris
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Netherlands Cancer Institute, 1066 CX, Amsterdam, the Netherlands; University of Amsterdam, 1012 WP, Amsterdam, the Netherlands
| | - Sophia Janine Wagner
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany; School of Computation, Information and Technology, Technical University of Munich, Munich, Germany
| | - Michael Jendrusch
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany
| | - Matthias Kloor
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany
| | | | - Hannah Sophie Muti
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Katherine J Hewitt
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Gregory P Veldhuizen
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Laura Zigutyte
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom; Department of Medicine I, University Hospital Dresden, Dresden, Germany; Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
26
|
Gustav M, Reitsam NG, Carrero ZI, Loeffler CML, van Treeck M, Yuan T, West NP, Quirke P, Brinker TJ, Brenner H, Favre L, Märkl B, Stenzinger A, Brobeil A, Hoffmeister M, Calderaro J, Pujals A, Kather JN. Deep learning for dual detection of microsatellite instability and POLE mutations in colorectal cancer histopathology. NPJ Precis Oncol 2024; 8:115. [PMID: 38783059 PMCID: PMC11116442 DOI: 10.1038/s41698-024-00592-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Accepted: 04/14/2024] [Indexed: 05/25/2024] Open
Abstract
In the spectrum of colorectal tumors, microsatellite-stable (MSS) tumors with DNA polymerase ε (POLE) mutations exhibit a hypermutated profile, holding the potential to respond to immunotherapy similarly to their microsatellite-instable (MSI) counterparts. Yet, due to their rarity and the associated testing costs, systematic screening for these mutations is not commonly pursued. Notably, the histopathological phenotype resulting from POLE mutations is theorized to resemble that of MSI. This resemblance not only could facilitate their detection by a transformer-based Deep Learning (DL) system trained on MSI pathology slides, but also indicates the possibility for MSS patients with POLE mutations to access enhanced treatment options, which might otherwise be overlooked. To harness this potential, we trained a Deep Learning classifier on a large dataset with the ground truth for microsatellite status and subsequently validated its capabilities for MSI and POLE detection across three external cohorts. Our model accurately identified MSI status in both the internal and external resection cohorts using pathology images alone. Notably, with a classification threshold of 0.5, over 75% of POLE driver mutant patients in the external resection cohorts were flagged as "positive" by a DL system trained on MSI status. In a clinical setting, deploying this DL model as a preliminary screening tool could facilitate the efficient identification of clinically relevant MSI and POLE mutations in colorectal tumors, in one go.
Collapse
Affiliation(s)
- Marco Gustav
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | | | - Zunamys I Carrero
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Chiara M L Loeffler
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine I, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Marko van Treeck
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Tanwei Yuan
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Nicholas P West
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
| | - Philip Quirke
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
| | - Titus J Brinker
- Digital Biomarkers for Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), Heidelberg, Germany
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Loëtitia Favre
- Université Paris Est Créteil, INSERM, IMRB, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- INSERM, U955, Team Oncogenèse des lymphomes et tumeurs de la Neurofibromatose 1, Créteil, France
| | - Bruno Märkl
- Pathology, Faculty of Medicine, University of Augsburg, Augsburg, Germany
| | | | - Alexander Brobeil
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany
- Tissue Bank of the National Center for Tumor Diseases (NCT) Heidelberg, Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Julien Calderaro
- Université Paris Est Créteil, INSERM, IMRB, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- INSERM, U955, Team Oncogenèse des lymphomes et tumeurs de la Neurofibromatose 1, Créteil, France
| | - Anaïs Pujals
- Université Paris Est Créteil, INSERM, IMRB, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- INSERM, U955, Team Oncogenèse des lymphomes et tumeurs de la Neurofibromatose 1, Créteil, France
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
- Department of Medicine I, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany.
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
27
|
Gao R, Yuan X, Ma Y, Wei T, Johnston L, Shao Y, Lv W, Zhu T, Zhang Y, Zheng J, Chen G, Sun J, Wang YG, Yu Z. Harnessing TME depicted by histological images to improve cancer prognosis through a deep learning system. Cell Rep Med 2024; 5:101536. [PMID: 38697103 PMCID: PMC11149411 DOI: 10.1016/j.xcrm.2024.101536] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Revised: 02/26/2024] [Accepted: 04/08/2024] [Indexed: 05/04/2024]
Abstract
Spatial transcriptomics (ST) provides insights into the tumor microenvironment (TME), which is closely associated with cancer prognosis, but ST has limited clinical availability. In this study, we provide a powerful deep learning system to augment TME information based on histological images for patients without ST data, thereby empowering precise cancer prognosis. The system provides two connections to bridge existing gaps. The first is the integrated graph and image deep learning (IGI-DL) model, which predicts ST expression based on histological images with a 0.171 increase in mean correlation across three cancer types compared with five existing methods. The second connection is the cancer prognosis prediction model, based on TME depicted by spatial gene expression. Our survival model, using graphs with predicted ST features, achieves superior accuracy with a concordance index of 0.747 and 0.725 for The Cancer Genome Atlas breast cancer and colorectal cancer cohorts, outperforming other survival models. For the external Molecular and Cellular Oncology colorectal cancer cohort, our survival model maintains a stable advantage.
Collapse
Affiliation(s)
- Ruitian Gao
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Xin Yuan
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Yanran Ma
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Ting Wei
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Luke Johnston
- School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Yanfei Shao
- Department of General Surgery, Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Wenwen Lv
- Clinical Research Institute, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Tengteng Zhu
- Clinical Research Institute, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Yue Zhang
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Junke Zheng
- Key Laboratory of Cell Differentiation and Apoptosis of the Chinese Ministry of Education, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Guoqiang Chen
- State Key Laboratory of Systems Medicine for Cancer, Ren-Ji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Jing Sun
- Department of General Surgery, Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China.
| | - Yu Guang Wang
- School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai 200240, China; Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai 200240, China; Zhangjiang Institute of Advanced Research, Shanghai Jiao Tong University, Shanghai 201203, China.
| | - Zhangsheng Yu
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China; School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai 200240, China; Clinical Research Institute, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China.
| |
Collapse
|
28
|
Tian F, Liu D, Wei N, Fu Q, Sun L, Liu W, Sui X, Tian K, Nemeth G, Feng J, Xu J, Xiao L, Han J, Fu J, Shi Y, Yang Y, Liu J, Hu C, Feng B, Sun Y, Wang Y, Yu G, Kong D, Wang M, Li W, Chen K, Li X. Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning. Nat Med 2024; 30:1309-1319. [PMID: 38627559 PMCID: PMC11108774 DOI: 10.1038/s41591-024-02915-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2023] [Accepted: 03/07/2024] [Indexed: 04/26/2024]
Abstract
Cancer of unknown primary (CUP) site poses diagnostic challenges due to its elusive nature. Many cases of CUP manifest as pleural and peritoneal serous effusions. Leveraging cytological images from 57,220 cases at four tertiary hospitals, we developed a deep-learning method for tumor origin differentiation using cytological histology (TORCH) that can identify malignancy and predict tumor origin in both hydrothorax and ascites. We examined its performance on three internal (n = 12,799) and two external (n = 14,538) testing sets. In both internal and external testing sets, TORCH achieved area under the receiver operating curve values ranging from 0.953 to 0.991 for cancer diagnosis and 0.953 to 0.979 for tumor origin localization. TORCH accurately predicted primary tumor origins, with a top-1 accuracy of 82.6% and top-3 accuracy of 98.9%. Compared with results derived from pathologists, TORCH showed better prediction efficacy (1.677 versus 1.265, P < 0.001), enhancing junior pathologists' diagnostic scores significantly (1.326 versus 1.101, P < 0.001). Patients with CUP whose initial treatment protocol was concordant with TORCH-predicted origins had better overall survival than those who were administrated discordant treatment (27 versus 17 months, P = 0.006). Our study underscores the potential of TORCH as a valuable ancillary tool in clinical practice, although further validation in randomized trials is warranted.
Collapse
Affiliation(s)
- Fei Tian
- Department of Abdominal Cancer, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Dong Liu
- Department of Radiology, The First Affiliated Hospital of Suzhou University, Suzhou, China
| | - Na Wei
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Qianqian Fu
- Department of Pathology, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Lin Sun
- Department of Pathology, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Wei Liu
- Department of Pathology, The First Affiliated Hospital of Suzhou University, Suzhou, China
| | - Xiaolong Sui
- Department of Pathology, Yantai Yuhuangding Hospital of Qingdao University, Yantai, China
| | - Kathryn Tian
- Harvard Dunster House, Harvard University, Cambridge, MA, USA
| | | | - Jingyu Feng
- School of Biological Science and Medical Engineering, Beihang University, Beijing, China
| | - Jingjing Xu
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Lin Xiao
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Junya Han
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Jingjie Fu
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Yinhua Shi
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Yichen Yang
- Tianjin Cancer Institute, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Jia Liu
- Department of Abdominal Cancer, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Chunhong Hu
- Department of Radiology, The First Affiliated Hospital of Suzhou University, Suzhou, China
| | - Bin Feng
- Department of Pathology, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Yan Sun
- Department of Pathology, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Yunjun Wang
- Department of Pathology, Yantai Yuhuangding Hospital of Qingdao University, Yantai, China
| | - Guohua Yu
- Department of Pathology, Yantai Yuhuangding Hospital of Qingdao University, Yantai, China
| | - Dalu Kong
- Department of Abdominal Cancer, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Meiyun Wang
- Department of Radiology, Henan Provincial People's Hospital, The People's Hospital of Zhengzhou University, Zhengzhou, China
| | - Wencai Li
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China.
| | - Kexin Chen
- Department of Epidemiology and Biostatistics, Tianjin's Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China.
| | - Xiangchun Li
- Tianjin Cancer Institute, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China.
| |
Collapse
|
29
|
Yuan L, Zhou H, Xiao X, Zhang X, Chen F, Liu L, Liu J, Bao S, Tao K. Development and external validation of a transfer learning-based system for the pathological diagnosis of colorectal cancer: a large emulated prospective study. Front Oncol 2024; 14:1365364. [PMID: 38725622 PMCID: PMC11079287 DOI: 10.3389/fonc.2024.1365364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Accepted: 04/11/2024] [Indexed: 05/12/2024] Open
Abstract
Background The progress in Colorectal cancer (CRC) screening and management has resulted in an unprecedented caseload for histopathological diagnosis. While artificial intelligence (AI) presents a potential solution, the predominant emphasis on slide-level aggregation performance without thorough verification of cancer in each location, impedes both explainability and transparency. Effectively addressing these challenges is crucial to ensuring the reliability and efficacy of AI in histology applications. Method In this study, we created an innovative AI algorithm using transfer learning from a polyp segmentation model in endoscopy. The algorithm precisely localized CRC targets within 0.25 mm² grids from whole slide imaging (WSI). We assessed the CRC detection capabilities at this fine granularity and examined the influence of AI on the diagnostic behavior of pathologists. The evaluation utilized an extensive dataset comprising 858 consecutive patient cases with 1418 WSIs obtained from an external center. Results Our results underscore a notable sensitivity of 90.25% and specificity of 96.60% at the grid level, accompanied by a commendable area under the curve (AUC) of 0.962. This translates to an impressive 99.39% sensitivity at the slide level, coupled with a negative likelihood ratio of <0.01, signifying the dependability of the AI system to preclude diagnostic considerations. The positive likelihood ratio of 26.54, surpassing 10 at the grid level, underscores the imperative for meticulous scrutiny of any AI-generated highlights. Consequently, all four participating pathologists demonstrated statistically significant diagnostic improvements with AI assistance. Conclusion Our transfer learning approach has successfully yielded an algorithm that can be validated for CRC histological localizations in whole slide imaging. The outcome advocates for the integration of the AI system into histopathological diagnosis, serving either as a diagnostic exclusion application or a computer-aided detection (CADe) tool. This integration has the potential to alleviate the workload of pathologists and ultimately benefit patients.
Collapse
Affiliation(s)
- Liuhong Yuan
- Department of Pathology, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | - Henghua Zhou
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | | | - Xiuqin Zhang
- Department of Pathology, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | - Feier Chen
- Department of Pathology, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | - Lin Liu
- Institute of Natural Sciences, MOE-LSC, School of Mathematical Sciences, CMA-Shanghai, SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai, China
| | | | - Shisan Bao
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | - Kun Tao
- Department of Pathology, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| |
Collapse
|
30
|
Zamanitajeddin N, Jahanifar M, Bilal M, Eastwood M, Rajpoot N. Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer. Med Image Anal 2024; 93:103071. [PMID: 38199068 DOI: 10.1016/j.media.2023.103071] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Revised: 11/14/2023] [Accepted: 12/19/2023] [Indexed: 01/12/2024]
Abstract
Colorectal cancer (CRC) is a primary global health concern, and identifying the molecular pathways, genetic subtypes, and mutations associated with CRC is crucial for precision medicine. However, traditional measurement techniques such as gene sequencing are costly and time-consuming, while most deep learning methods proposed for this task lack interpretability. This study offers a new approach to enhance the state-of-the-art deep learning methods for molecular pathways and key mutation prediction by incorporating cell network information. We build cell graphs with nuclei as nodes and nuclei connections as edges of the network and leverage Social Network Analysis (SNA) measures to extract abstract, perceivable, and interpretable features that explicitly describe the cell network characteristics in an image. Our approach does not rely on precise nuclei segmentation or feature extraction, is computationally efficient, and is easily scalable. In this study, we utilize the TCGA-CRC-DX dataset, comprising 499 patients and 502 diagnostic slides from primary colorectal tumours, sourced from 36 distinct medical centres in the United States. By incorporating the SNA features alongside deep features in two multiple instance learning frameworks, we demonstrate improved performance for chromosomal instability (CIN), hypermutated tumour (HM), TP53 gene, BRAF gene, and Microsatellite instability (MSI) status prediction tasks (2.4%-4% and 7-8.8% improvement in AUROC and AUPRC on average). Additionally, our method achieves outstanding performance on MSI prediction in an external PAIP dataset (99% AUROC and 98% AUPRC), demonstrating its generalizability. Our findings highlight the discrimination power of SNA features and how they can be beneficial to deep learning models' performance and provide insights into the correlation of cell network profiles with molecular pathways and key mutations.
Collapse
Affiliation(s)
- Neda Zamanitajeddin
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK.
| | - Mostafa Jahanifar
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Mohsin Bilal
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Mark Eastwood
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Nasir Rajpoot
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK; Histofy Ltd., Birmingham, UK.
| |
Collapse
|
31
|
Unger M, Kather JN. Deep learning in cancer genomics and histopathology. Genome Med 2024; 16:44. [PMID: 38539231 PMCID: PMC10976780 DOI: 10.1186/s13073-024-01315-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2023] [Accepted: 03/13/2024] [Indexed: 07/08/2024] Open
Abstract
Histopathology and genomic profiling are cornerstones of precision oncology and are routinely obtained for patients with cancer. Traditionally, histopathology slides are manually reviewed by highly trained pathologists. Genomic data, on the other hand, is evaluated by engineered computational pipelines. In both applications, the advent of modern artificial intelligence methods, specifically machine learning (ML) and deep learning (DL), have opened up a fundamentally new way of extracting actionable insights from raw data, which could augment and potentially replace some aspects of traditional evaluation workflows. In this review, we summarize current and emerging applications of DL in histopathology and genomics, including basic diagnostic as well as advanced prognostic tasks. Based on a growing body of evidence, we suggest that DL could be the groundwork for a new kind of workflow in oncology and cancer research. However, we also point out that DL models can have biases and other flaws that users in healthcare and research need to know about, and we propose ways to address them.
Collapse
Affiliation(s)
- Michaela Unger
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany.
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany.
- Department of Medicine I, University Hospital Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
32
|
Howard FM, Hieromnimon HM, Ramesh S, Dolezal J, Kochanny S, Zhang Q, Feiger B, Peterson J, Fan C, Perou CM, Vickery J, Sullivan M, Cole K, Khramtsova G, Pearson AT. Generative Adversarial Networks Accurately Reconstruct Pan-Cancer Histology from Pathologic, Genomic, and Radiographic Latent Features. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.22.586306. [PMID: 38585926 PMCID: PMC10996476 DOI: 10.1101/2024.03.22.586306] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/09/2024]
Abstract
Artificial intelligence models have been increasingly used in the analysis of tumor histology to perform tasks ranging from routine classification to identification of novel molecular features. These approaches distill cancer histologic images into high-level features which are used in predictions, but understanding the biologic meaning of such features remains challenging. We present and validate a custom generative adversarial network - HistoXGAN - capable of reconstructing representative histology using feature vectors produced by common feature extractors. We evaluate HistoXGAN across 29 cancer subtypes and demonstrate that reconstructed images retain information regarding tumor grade, histologic subtype, and gene expression patterns. We leverage HistoXGAN to illustrate the underlying histologic features for deep learning models for actionable mutations, identify model reliance on histologic batch effect in predictions, and demonstrate accurate reconstruction of tumor histology from radiographic imaging for a 'virtual biopsy'.
Collapse
|
33
|
Liu B, Polack M, Coudray N, Quiros AC, Sakellaropoulos T, Crobach ASLP, van Krieken JHJM, Yuan K, Tollenaar RAEM, Mesker WE, Tsirigos A. Self-Supervised Learning Reveals Clinically Relevant Histomorphological Patterns for Therapeutic Strategies in Colon Cancer. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.26.582106. [PMID: 38496571 PMCID: PMC10942268 DOI: 10.1101/2024.02.26.582106] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/19/2024]
Abstract
Self-supervised learning (SSL) automates the extraction and interpretation of histopathology features on unannotated hematoxylin-and-eosin-stained whole-slide images (WSIs). We trained an SSL Barlow Twins-encoder on 435 TCGA colon adenocarcinoma WSIs to extract features from small image patches. Leiden community detection then grouped tiles into histomorphological phenotype clusters (HPCs). HPC reproducibility and predictive ability for overall survival was confirmed in an independent clinical trial cohort (N=1213 WSIs). This unbiased atlas resulted in 47 HPCs displaying unique and sharing clinically significant histomorphological traits, highlighting tissue type, quantity, and architecture, especially in the context of tumor stroma. Through in-depth analysis of these HPCs, including immune landscape and gene set enrichment analysis, and association to clinical outcomes, we shed light on the factors influencing survival and responses to treatments like standard adjuvant chemotherapy and experimental therapies. Further exploration of HPCs may unveil new insights and aid decision-making and personalized treatments for colon cancer patients.
Collapse
|
34
|
Chen RJ, Ding T, Lu MY, Williamson DFK, Jaume G, Song AH, Chen B, Zhang A, Shao D, Shaban M, Williams M, Oldenburg L, Weishaupt LL, Wang JJ, Vaidya A, Le LP, Gerber G, Sahai S, Williams W, Mahmood F. Towards a general-purpose foundation model for computational pathology. Nat Med 2024; 30:850-862. [PMID: 38504018 PMCID: PMC11403354 DOI: 10.1038/s41591-024-02857-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Accepted: 02/05/2024] [Indexed: 03/21/2024]
Abstract
Quantitative evaluation of tissue images is crucial for computational pathology (CPath) tasks, requiring the objective characterization of histopathological entities from whole-slide images (WSIs). The high resolution of WSIs and the variability of morphological features present significant challenges, complicating the large-scale annotation of data for high-performance applications. To address this challenge, current efforts have proposed the use of pretrained image encoders through transfer learning from natural image datasets or self-supervised learning on publicly available histopathology datasets, but have not been extensively developed and evaluated across diverse tissue types at scale. We introduce UNI, a general-purpose self-supervised model for pathology, pretrained using more than 100 million images from over 100,000 diagnostic H&E-stained WSIs (>77 TB of data) across 20 major tissue types. The model was evaluated on 34 representative CPath tasks of varying diagnostic difficulty. In addition to outperforming previous state-of-the-art models, we demonstrate new modeling capabilities in CPath such as resolution-agnostic tissue classification, slide classification using few-shot class prototypes, and disease subtyping generalization in classifying up to 108 cancer types in the OncoTree classification system. UNI advances unsupervised representation learning at scale in CPath in terms of both pretraining data and downstream evaluation, enabling data-efficient artificial intelligence models that can generalize and transfer to a wide range of diagnostically challenging tasks and clinical workflows in anatomic pathology.
Collapse
Affiliation(s)
- Richard J Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Tong Ding
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Ming Y Lu
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT), Cambridge, MA, USA
| | - Drew F K Williamson
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
| | - Guillaume Jaume
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Andrew H Song
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Bowen Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Andrew Zhang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Daniel Shao
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Muhammad Shaban
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Mane Williams
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Lukas Oldenburg
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Luca L Weishaupt
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Judy J Wang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Anurag Vaidya
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Long Phi Le
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Georg Gerber
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Sharifa Sahai
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Systems Biology, Harvard University, Cambridge, MA, USA
| | - Walt Williams
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA.
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA.
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
35
|
Uchikov P, Khalid U, Kraev K, Hristov B, Kraeva M, Tenchev T, Chakarov D, Sandeva M, Dragusheva S, Taneva D, Batashki A. Artificial Intelligence in the Diagnosis of Colorectal Cancer: A Literature Review. Diagnostics (Basel) 2024; 14:528. [PMID: 38472999 DOI: 10.3390/diagnostics14050528] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2024] [Revised: 02/26/2024] [Accepted: 02/27/2024] [Indexed: 03/14/2024] Open
Abstract
BACKGROUND The aim of this review is to explore the role of artificial intelligence in the diagnosis of colorectal cancer, how it impacts CRC morbidity and mortality, and why its role in clinical medicine is limited. METHODS A targeted, non-systematic review of the published literature relating to colorectal cancer diagnosis was performed with PubMed databases that were scouted to help provide a more defined understanding of the recent advances regarding artificial intelligence and their impact on colorectal-related morbidity and mortality. Articles were included if deemed relevant and including information associated with the keywords. RESULTS The advancements in artificial intelligence have been significant in facilitating an earlier diagnosis of CRC. In this review, we focused on evaluating genomic biomarkers, the integration of instruments with artificial intelligence, MR and hyperspectral imaging, and the architecture of neural networks. We found that these neural networks seem practical and yield positive results in initial testing. Furthermore, we explored the use of deep-learning-based majority voting methods, such as bag of words and PAHLI, in improving diagnostic accuracy in colorectal cancer detection. Alongside this, the autonomous and expansive learning ability of artificial intelligence, coupled with its ability to extract increasingly complex features from images or videos without human reliance, highlight its impact in the diagnostic sector. Despite this, as most of the research involves a small sample of patients, a diversification of patient data is needed to enhance cohort stratification for a more sensitive and specific neural model. We also examined the successful application of artificial intelligence in predicting microsatellite instability, showcasing its potential in stratifying patients for targeted therapies. CONCLUSIONS Since its commencement in colorectal cancer, artificial intelligence has revealed a multitude of functionalities and augmentations in the diagnostic sector of CRC. Given its early implementation, its clinical application remains a fair way away, but with steady research dedicated to improving neural architecture and expanding its applicational range, there is hope that these advanced neural software could directly impact the early diagnosis of CRC. The true promise of artificial intelligence, extending beyond the medical sector, lies in its potential to significantly influence the future landscape of CRC's morbidity and mortality.
Collapse
Affiliation(s)
- Petar Uchikov
- Department of Special Surgery, Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Usman Khalid
- Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Krasimir Kraev
- Department of Propaedeutics of Internal Diseases "Prof. Dr. Anton Mitov", Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Bozhidar Hristov
- Section "Gastroenterology", Second Department of Internal Diseases, Medical Faculty, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Maria Kraeva
- Department of Otorhinolaryngology, Medical Faculty, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Tihomir Tenchev
- Department of Special Surgery, Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Dzhevdet Chakarov
- Department of Propaedeutics of Surgical Diseases, Section of General Surgery, Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Milena Sandeva
- Department of Midwifery, Faculty of Public Health, Medical University of Plovdiv, 4000 Plovdiv, Bulgaria
| | - Snezhanka Dragusheva
- Department of Nursing Care, Faculty of Public Health, Medical University of Plovdiv, 4000 Plovdiv, Bulgaria
| | - Daniela Taneva
- Department of Nursing Care, Faculty of Public Health, Medical University of Plovdiv, 4000 Plovdiv, Bulgaria
| | - Atanas Batashki
- Department of Special Surgery, Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| |
Collapse
|
36
|
Ligero M, Serna G, El Nahhas OS, Sansano I, Mauchanski S, Viaplana C, Calderaro J, Toledo RA, Dienstmann R, Vanguri RS, Sauter JL, Sanchez-Vega F, Shah SP, Ramón y Cajal S, Garralda E, Nuciforo P, Perez-Lopez R, Kather JN. Weakly Supervised Deep Learning Predicts Immunotherapy Response in Solid Tumors Based on PD-L1 Expression. CANCER RESEARCH COMMUNICATIONS 2024; 4:92-102. [PMID: 38126740 PMCID: PMC10782919 DOI: 10.1158/2767-9764.crc-23-0287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/06/2023] [Revised: 10/11/2023] [Accepted: 12/12/2023] [Indexed: 12/23/2023]
Abstract
Programmed death-ligand 1 (PD-L1) IHC is the most commonly used biomarker for immunotherapy response. However, quantification of PD-L1 status in pathology slides is challenging. Neither manual quantification nor a computer-based mimicking of manual readouts is perfectly reproducible, and the predictive performance of both approaches regarding immunotherapy response is limited. In this study, we developed a deep learning (DL) method to predict PD-L1 status directly from raw IHC image data, without explicit intermediary steps such as cell detection or pigment quantification. We trained the weakly supervised model on PD-L1-stained slides from the non-small cell lung cancer (NSCLC)-Memorial Sloan Kettering (MSK) cohort (N = 233) and validated it on the pan-cancer-Vall d'Hebron Institute of Oncology (VHIO) cohort (N = 108). We also investigated the performance of the model to predict response to immune checkpoint inhibitors (ICI) in terms of progression-free survival. In the pan-cancer-VHIO cohort, the performance was compared with tumor proportion score (TPS) and combined positive score (CPS). The DL model showed good performance in predicting PD-L1 expression (TPS ≥ 1%) in both NSCLC-MSK and pan-cancer-VHIO cohort (AUC 0.88 ± 0.06 and 0.80 ± 0.03, respectively). The predicted PD-L1 status showed an improved association with response to ICIs [HR: 1.5 (95% confidence interval: 1-2.3), P = 0.049] compared with TPS [HR: 1.4 (0.96-2.2), P = 0.082] and CPS [HR: 1.2 (0.79-1.9), P = 0.386]. Notably, our explainability analysis showed that the model does not just look at the amount of brown pigment in the IHC slides, but also considers morphologic factors such as lymphocyte conglomerates. Overall, end-to-end weakly supervised DL shows potential for improving patient stratification for cancer immunotherapy by analyzing PD-L1 IHC, holistically integrating morphology and PD-L1 staining intensity. SIGNIFICANCE The weakly supervised DL model to predict PD-L1 status from raw IHC data, integrating tumor staining intensity and morphology, enables enhanced patient stratification in cancer immunotherapy compared with traditional pathologist assessment.
Collapse
Affiliation(s)
- Marta Ligero
- Radiomics Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Garazi Serna
- Molecular Oncology Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Omar S.M. El Nahhas
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Irene Sansano
- Pathology Department, Vall d'Hebron University Hospital (VHUH), Barcelona, Spain
| | - Siarhei Mauchanski
- Molecular Oncology Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Cristina Viaplana
- Oncology Data Science (ODysSey) Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Julien Calderaro
- Assistance Publique-Hôpitaux de Paris, Département de Pathologie, CHU Henri Mondor, Créteil, France
- Université Paris-Est Créteil, Faculté de Médecine, Créteil, France
| | - Rodrigo A. Toledo
- Biomakers and Clonal Dynamics Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Rodrigo Dienstmann
- Oncology Data Science (ODysSey) Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Rami S. Vanguri
- Department of Pathology and Laboratory Medicine, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
| | - Jennifer L. Sauter
- Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York
| | | | - Sohrab P. Shah
- Computational Oncology, Memorial Sloan Kettering Cancer Center, New York, New York
| | | | - Elena Garralda
- Department of Medical Oncology, Vall d'Hebron University Hospital and Institute of Oncology (VHIO), Barcelona, Spain
| | - Paolo Nuciforo
- Molecular Oncology Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Raquel Perez-Lopez
- Radiomics Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
- Department of Medicine I, University Hospital Dresden, Dresden, Germany
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| |
Collapse
|
37
|
Jia Y, Liu J, Chen L, Zhao T, Wang Y. THItoGene: a deep learning method for predicting spatial transcriptomics from histological images. Brief Bioinform 2023; 25:bbad464. [PMID: 38145948 PMCID: PMC10749789 DOI: 10.1093/bib/bbad464] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 11/10/2023] [Accepted: 11/18/2023] [Indexed: 12/27/2023] Open
Abstract
Spatial transcriptomics unveils the complex dynamics of cell regulation and transcriptomes, but it is typically cost-prohibitive. Predicting spatial gene expression from histological images via artificial intelligence offers a more affordable option, yet existing methods fall short in extracting deep-level information from pathological images. In this paper, we present THItoGene, a hybrid neural network that utilizes dynamic convolutional and capsule networks to adaptively sense potential molecular signals in histological images for exploring the relationship between high-resolution pathology image phenotypes and regulation of gene expression. A comprehensive benchmark evaluation using datasets from human breast cancer and cutaneous squamous cell carcinoma has demonstrated the superior performance of THItoGene in spatial gene expression prediction. Moreover, THItoGene has demonstrated its capacity to decipher both the spatial context and enrichment signals within specific tissue regions. THItoGene can be freely accessed at https://github.com/yrjia1015/THItoGene.
Collapse
Affiliation(s)
- Yuran Jia
- Institute for Bioinformatics, School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150040, China
| | - Junliang Liu
- Institute for Bioinformatics, School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150040, China
| | - Li Chen
- School of Life Sciences, Westlake University, Hangzhou, Zhejiang 310024, China
| | - Tianyi Zhao
- School of Medicine and Health, Harbin Institute of Technology, Harbin, 150040, China
| | - Yadong Wang
- School of Medicine and Health, Harbin Institute of Technology, Harbin, 150040, China
| |
Collapse
|
38
|
Jang HJ, Go JH, Kim Y, Lee SH. Deep Learning for the Pathologic Diagnosis of Hepatocellular Carcinoma, Cholangiocarcinoma, and Metastatic Colorectal Cancer. Cancers (Basel) 2023; 15:5389. [PMID: 38001649 PMCID: PMC10670046 DOI: 10.3390/cancers15225389] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Revised: 11/01/2023] [Accepted: 11/09/2023] [Indexed: 11/26/2023] Open
Abstract
Diagnosing primary liver cancers, particularly hepatocellular carcinoma (HCC) and cholangiocarcinoma (CC), is a challenging and labor-intensive process, even for experts, and secondary liver cancers further complicate the diagnosis. Artificial intelligence (AI) offers promising solutions to these diagnostic challenges by facilitating the histopathological classification of tumors using digital whole slide images (WSIs). This study aimed to develop a deep learning model for distinguishing HCC, CC, and metastatic colorectal cancer (mCRC) using histopathological images and to discuss its clinical implications. The WSIs from HCC, CC, and mCRC were used to train the classifiers. For normal/tumor classification, the areas under the curve (AUCs) were 0.989, 0.988, and 0.991 for HCC, CC, and mCRC, respectively. Using proper tumor tissues, the HCC/other cancer type classifier was trained to effectively distinguish HCC from CC and mCRC, with a concatenated AUC of 0.998. Subsequently, the CC/mCRC classifier differentiated CC from mCRC with a concatenated AUC of 0.995. However, testing on an external dataset revealed that the HCC/other cancer type classifier underperformed with an AUC of 0.745. After combining the original training datasets with external datasets and retraining, the classification drastically improved, all achieving AUCs of 1.000. Although these results are promising and offer crucial insights into liver cancer, further research is required for model refinement and validation.
Collapse
Affiliation(s)
- Hyun-Jong Jang
- Department of Physiology, CMC Institute for Basic Medical Science, College of Medicine, The Catholic University of Korea, Seoul 06591, Republic of Korea;
| | - Jai-Hyang Go
- Department of Pathology, Dankook University College of Medicine, Cheonan 31116, Republic of Korea;
| | - Younghoon Kim
- Department of Hospital Pathology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul 06591, Republic of Korea;
| | - Sung Hak Lee
- Department of Hospital Pathology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul 06591, Republic of Korea;
| |
Collapse
|