1
|
Makhlouf Y, Singh VK, Craig S, McArdle A, French D, Loughrey MB, Oliver N, Acevedo JB, O’Reilly P, James JA, Maxwell P, Salto-Tellez M. True-T - Improving T-cell response quantification with holistic artificial intelligence based prediction in immunohistochemistry images. Comput Struct Biotechnol J 2024; 23:174-185. [PMID: 38146436 PMCID: PMC10749253 DOI: 10.1016/j.csbj.2023.11.048] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Revised: 11/24/2023] [Accepted: 11/24/2023] [Indexed: 12/27/2023] Open
Abstract
The immune response associated with oncogenesis and potential oncological ther- apeutic interventions has dominated the field of cancer research over the last decade. T-cell lymphocytes in the tumor microenvironment are a crucial aspect of cancer's adaptive immunity, and the quantification of T-cells in specific can- cer types has been suggested as a potential diagnostic aid. However, this is cur- rently not part of routine diagnostics. To address this challenge, we present a new method called True-T, which employs artificial intelligence-based techniques to quantify T-cells in colorectal cancer (CRC) using immunohistochemistry (IHC) images. True-T analyses the chromogenic tissue hybridization signal of three widely recognized T-cell markers (CD3, CD4, and CD8). Our method employs a pipeline consisting of three stages: T-cell segmentation, density estimation from the segmented mask, and prediction of individual five-year survival rates. In the first stage, we utilize the U-Net method, where a pre-trained ResNet-34 is em- ployed as an encoder to extract clinically relevant T-cell features. The segmenta- tion model is trained and evaluated individually, demonstrating its generalization in detecting the CD3, CD4, and CD8 biomarkers in IHC images. In the second stage, the density of T-cells is estimated using the predicted mask, which serves as a crucial indicator for patient survival statistics in the third stage. This ap- proach was developed and tested in 1041 patients from four reference diagnostic institutions, ensuring broad applicability. The clinical effectiveness of True-T is demonstrated in stages II-IV CRC by offering valuable prognostic information that surpasses previous quantitative gold standards, opening possibilities for po- tential clinical applications. Finally, to evaluate the robustness and broader ap- plicability of our approach without additional training, we assessed the universal accuracy of the CD3 component of the True-T algorithm across 13 distinct solid tumors.
Collapse
Affiliation(s)
- Yasmine Makhlouf
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Vivek Kumar Singh
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Stephanie Craig
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Aoife McArdle
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Dominique French
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Maurice B. Loughrey
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
- Cellular Pathology, Belfast Health and Social Care Trust, Belfast City Hospital, Lisburn Road, Belfast BT9 7AB, UK
| | - Nicola Oliver
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Juvenal Baena Acevedo
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | | | - Jacqueline A. James
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
- Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast BT9 7AE, UK
| | - Perry Maxwell
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
| | - Manuel Salto-Tellez
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston, Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
- Sonrai Analytics, Belfast BT9 7AE, UK
- Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast BT9 7AE, UK
- Integrated Pathology Unit, Institute of Cancer Research and Royal Marsden Hospital, London SW7 3RP, UK
| |
Collapse
|
2
|
Reitsam NG, Grosser B, Steiner DF, Grozdanov V, Wulczyn E, L'Imperio V, Plass M, Müller H, Zatloukal K, Muti HS, Kather JN, Märkl B. Converging deep learning and human-observed tumor-adipocyte interaction as a biomarker in colorectal cancer. COMMUNICATIONS MEDICINE 2024; 4:163. [PMID: 39147895 PMCID: PMC11327259 DOI: 10.1038/s43856-024-00589-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2023] [Accepted: 08/05/2024] [Indexed: 08/17/2024] Open
Abstract
BACKGROUND Tumor-Adipose-Feature (TAF) as well as SARIFA (Stroma AReactive Invasion Front Areas) are two histologic features/biomarkers linking tumor-associated adipocytes to poor outcomes in colorectal cancer (CRC) patients. Whereas TAF was identified by deep learning (DL) algorithms, SARIFA was established as a human-observed histopathologic biomarker. METHODS To study the overlap between TAF and SARIFA, we performed a systematic pathological review of TAF based on all published image tiles. Additionally, we analyzed the presence/absence of TAF in SARIFA-negative CRC cases to elucidate the biologic and prognostic role of a direct tumor-adipocyte contact. TCGA-CRC gene expression data is investigated to assess the association of FABP4 (fatty-acid binding protein 4) and CD36 (fatty-acid translocase) with both TAF and CRC prognosis. RESULTS By investigating the TAF/SARIFA overlap, we show that many TAF patches correspond to the recently described SARIFA-phenomenon. Even though there is a pronounced morphological and biological overlap, there are differences in the concepts. The presence of TAF in SARIFA-negative CRCs is not associated with poor outcomes in this cohort, potentially highlighting the importance of a direct tumor-adipocyte interaction. Upregulation of FABP4 and CD36 gene expression seem both linked to a poor prognosis in CRC. CONCLUSIONS By proving the substantial overlap between human-observed SARIFA and DL-based TAF as morphologic biomarkers, we demonstrate that linking DL-based image features to independently developed histopathologic biomarkers is a promising tool in the identification of clinically and biologically meaningful biomarkers. Adipocyte-tumor-cell interactions seem to be crucial in CRC, which should be considered as biomarkers for further investigations.
Collapse
Affiliation(s)
- Nic G Reitsam
- Pathology, Medical Faculty, University of Augsburg, Augsburg, Germany.
- Bavarian Cancer Research Center (BZKF), Augsburg, Germany.
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
| | - Bianca Grosser
- Pathology, Medical Faculty, University of Augsburg, Augsburg, Germany
- Bavarian Cancer Research Center (BZKF), Augsburg, Germany
| | | | | | - Ellery Wulczyn
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Vincenzo L'Imperio
- Department of Medicine and Surgery, Pathology, University of Milano-Bicocca, IRCCS (Scientific Institute for Research, Hospitalization and Healthcare) Fondazione San Gerardo dei Tintori, Monza, Italy
| | - Markus Plass
- Medical University of Graz, Diagnostic and Research Institute of Pathology, Graz, Austria
| | - Heimo Müller
- Medical University of Graz, Diagnostic and Research Institute of Pathology, Graz, Austria
| | - Kurt Zatloukal
- Medical University of Graz, Diagnostic and Research Institute of Pathology, Graz, Austria
| | - Hannah S Muti
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital Carl Gustav Carus Dresden, Dresden, Germany
| | - Jakob N Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
- Department of Medicine I, University Hospital Dresden, Dresden, Germany
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Bruno Märkl
- Pathology, Medical Faculty, University of Augsburg, Augsburg, Germany
- Bavarian Cancer Research Center (BZKF), Augsburg, Germany
| |
Collapse
|
3
|
Hölscher DL, Bülow RD. Decoding pathology: the role of computational pathology in research and diagnostics. Pflugers Arch 2024:10.1007/s00424-024-03002-2. [PMID: 39095655 DOI: 10.1007/s00424-024-03002-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2024] [Revised: 07/25/2024] [Accepted: 07/25/2024] [Indexed: 08/04/2024]
Abstract
Traditional histopathology, characterized by manual quantifications and assessments, faces challenges such as low-throughput and inter-observer variability that hinder the introduction of precision medicine in pathology diagnostics and research. The advent of digital pathology allowed the introduction of computational pathology, a discipline that leverages computational methods, especially based on deep learning (DL) techniques, to analyze histopathology specimens. A growing body of research shows impressive performances of DL-based models in pathology for a multitude of tasks, such as mutation prediction, large-scale pathomics analyses, or prognosis prediction. New approaches integrate multimodal data sources and increasingly rely on multi-purpose foundation models. This review provides an introductory overview of advancements in computational pathology and discusses their implications for the future of histopathology in research and diagnostics.
Collapse
Affiliation(s)
- David L Hölscher
- Department for Nephrology and Clinical Immunology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074, Aachen, Germany
- Institute for Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074, Aachen, Germany
| | - Roman D Bülow
- Institute for Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074, Aachen, Germany.
| |
Collapse
|
4
|
Calderaro J, Žigutytė L, Truhn D, Jaffe A, Kather JN. Artificial intelligence in liver cancer - new tools for research and patient management. Nat Rev Gastroenterol Hepatol 2024; 21:585-599. [PMID: 38627537 DOI: 10.1038/s41575-024-00919-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 03/11/2024] [Indexed: 07/31/2024]
Abstract
Liver cancer has high incidence and mortality globally. Artificial intelligence (AI) has advanced rapidly, influencing cancer care. AI systems are already approved for clinical use in some tumour types (for example, colorectal cancer screening). Crucially, research demonstrates that AI can analyse histopathology, radiology and natural language in liver cancer, and can replace manual tasks and access hidden information in routinely available clinical data. However, for liver cancer, few of these applications have translated into large-scale clinical trials or clinically approved products. Here, we advocate for the incorporation of AI in all stages of liver cancer management. We present a taxonomy of AI approaches in liver cancer, highlighting areas with academic and commercial potential, and outline a policy for AI-based liver cancer management, including interdisciplinary training of researchers, clinicians and patients. The potential of AI in liver cancer is immense, but effort is required to ensure that AI can fulfil expectations.
Collapse
Affiliation(s)
- Julien Calderaro
- Département de Pathologie, Assistance Publique Hôpitaux de Paris, Groupe Hospitalier Henri Mondor, Créteil, France
- Institut Mondor de Recherche Biomédicale, MINT-HEP Mondor Integrative Hepatology, Université Paris Est Créteil, Créteil, France
| | - Laura Žigutytė
- Else Kroener Fresenius Center for Digital Health (EKFZ), Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Ariel Jaffe
- Mayo Clinic, Rochester, MN, USA
- Department of Internal Medicine, Section of Digestive Diseases, Yale School of Medicine, New Haven, CT, USA
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health (EKFZ), Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany.
- Department of Medicine I, University Hospital Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumour Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
5
|
Vorontsov E, Bozkurt A, Casson A, Shaikovski G, Zelechowski M, Severson K, Zimmermann E, Hall J, Tenenholtz N, Fusi N, Yang E, Mathieu P, van Eck A, Lee D, Viret J, Robert E, Wang YK, Kunz JD, Lee MCH, Bernhard JH, Godrich RA, Oakley G, Millar E, Hanna M, Wen H, Retamero JA, Moye WA, Yousfi R, Kanan C, Klimstra DS, Rothrock B, Liu S, Fuchs TJ. A foundation model for clinical-grade computational pathology and rare cancers detection. Nat Med 2024:10.1038/s41591-024-03141-0. [PMID: 39039250 DOI: 10.1038/s41591-024-03141-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2024] [Accepted: 06/19/2024] [Indexed: 07/24/2024]
Abstract
The analysis of histopathology images with artificial intelligence aims to enable clinical decision support systems and precision medicine. The success of such applications depends on the ability to model the diverse patterns observed in pathology images. To this end, we present Virchow, the largest foundation model for computational pathology to date. In addition to the evaluation of biomarker prediction and cell identification, we demonstrate that a large foundation model enables pan-cancer detection, achieving 0.95 specimen-level area under the (receiver operating characteristic) curve across nine common and seven rare cancers. Furthermore, we show that with less training data, the pan-cancer detector built on Virchow can achieve similar performance to tissue-specific clinical-grade models in production and outperform them on some rare variants of cancer. Virchow's performance gains highlight the value of a foundation model and open possibilities for many high-impact applications with limited amounts of labeled training data.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | | | | | - Ellen Yang
- Memorial Sloan Kettering Cancer Center, New York, NY, US
| | | | | | | | | | | | | | | | | | | | | | | | - Ewan Millar
- NSW Health Pathology, St George Hospital, Sydney, New South Wales, Australia
| | - Matthew Hanna
- Memorial Sloan Kettering Cancer Center, New York, NY, US
| | - Hannah Wen
- Memorial Sloan Kettering Cancer Center, New York, NY, US
| | | | | | | | | | | | | | | | | |
Collapse
|
6
|
Singh VK, Makhlouf Y, Sarker MMK, Craig S, Baena J, Greene C, Mason L, James JA, Salto-Tellez M, O'Reilly P, Maxwell P. KRASFormer: a fully vision transformer-based framework for predicting KRASgene mutations in histopathological images of colorectal cancer. Biomed Phys Eng Express 2024; 10:055012. [PMID: 38925106 DOI: 10.1088/2057-1976/ad5bed] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2024] [Accepted: 06/26/2024] [Indexed: 06/28/2024]
Abstract
Detecting the Kirsten Rat Sarcoma Virus (KRAS) gene mutation is significant for colorectal cancer (CRC) patients. TheKRASgene encodes a protein involved in the epidermal growth factor receptor (EGFR) signaling pathway, and mutations in this gene can negatively impact the use of monoclonal antibodies in anti-EGFR therapy and affect treatment decisions. Currently, commonly used methods like next-generation sequencing (NGS) identifyKRASmutations but are expensive, time-consuming, and may not be suitable for every cancer patient sample. To address these challenges, we have developedKRASFormer, a novel framework that predictsKRASgene mutations from Haematoxylin and Eosin (H & E) stained WSIs that are widely available for most CRC patients.KRASFormerconsists of two stages: the first stage filters out non-tumor regions and selects only tumour cells using a quality screening mechanism, and the second stage predicts theKRASgene either wildtype' or mutant' using a Vision Transformer-based XCiT method. The XCiT employs cross-covariance attention to capture clinically meaningful long-range representations of textural patterns in tumour tissue andKRASmutant cells. We evaluated the performance of the first stage using an independent CRC-5000 dataset, and the second stage included both The Cancer Genome Atlas colon and rectal cancer (TCGA-CRC-DX) and in-house cohorts. The results of our experiments showed that the XCiT outperformed existing state-of-the-art methods, achieving AUCs for ROC curves of 0.691 and 0.653 on TCGA-CRC-DX and in-house datasets, respectively. Our findings emphasize three key consequences: the potential of using H & E-stained tissue slide images for predictingKRASgene mutations as a cost-effective and time-efficient means for guiding treatment choice with CRC patients; the increase in performance metrics of a Transformer-based model; and the value of the collaboration between pathologists and data scientists in deriving a morphologically meaningful model.
Collapse
Affiliation(s)
- Vivek Kumar Singh
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
- Centre for Biomarkers and Biotherapeutics, Barts Cancer Institute, Queen Mary University of London, London, EC1M 6BQ, United Kingdom
| | - Yasmine Makhlouf
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | | | - Stephanie Craig
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | - Juvenal Baena
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | - Christine Greene
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | - Lee Mason
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| | - Jacqueline A James
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
- Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, BT9 7AE, United Kingdom
| | - Manuel Salto-Tellez
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
- Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, BT9 7AE, United Kingdom
- Sonrai Analytics, Belfast, BT9 7AE, United Kingdom
- Cellular Pathology, Belfast Health and Social Care Trust, Belfast City Hospital, Lisburn Road, Belfast BT9 7AB, United Kingdom
| | | | - Perry Maxwell
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, BT9 7AE, United Kingdom
| |
Collapse
|
7
|
White BS, Woo XY, Koc S, Sheridan T, Neuhauser SB, Wang S, Evrard YA, Chen L, Foroughi pour A, Landua JD, Mashl RJ, Davies SR, Fang B, Rosa MG, Evans KW, Bailey MH, Chen Y, Xiao M, Rubinstein JC, Sanderson BJ, Lloyd MW, Domanskyi S, Dobrolecki LE, Fujita M, Fujimoto J, Xiao G, Fields RC, Mudd JL, Xu X, Hollingshead MG, Jiwani S, Acevedo S, Davis-Dusenbery BN, Robinson PN, Moscow JA, Doroshow JH, Mitsiades N, Kaochar S, Pan CX, Carvajal-Carmona LG, Welm AL, Welm BE, Govindan R, Li S, Davies MA, Roth JA, Meric-Bernstam F, Xie Y, Herlyn M, Ding L, Lewis MT, Bult CJ, Dean DA, Chuang JH. A Pan-Cancer Patient-Derived Xenograft Histology Image Repository with Genomic and Pathologic Annotations Enables Deep Learning Analysis. Cancer Res 2024; 84:2060-2072. [PMID: 39082680 PMCID: PMC11217732 DOI: 10.1158/0008-5472.can-23-1349] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2023] [Revised: 10/13/2023] [Accepted: 03/27/2024] [Indexed: 08/04/2024]
Abstract
Patient-derived xenografts (PDX) model human intra- and intertumoral heterogeneity in the context of the intact tissue of immunocompromised mice. Histologic imaging via hematoxylin and eosin (H&E) staining is routinely performed on PDX samples, which could be harnessed for computational analysis. Prior studies of large clinical H&E image repositories have shown that deep learning analysis can identify intercellular and morphologic signals correlated with disease phenotype and therapeutic response. In this study, we developed an extensive, pan-cancer repository of >1,000 PDX and paired parental tumor H&E images. These images, curated from the PDX Development and Trial Centers Research Network Consortium, had a range of associated genomic and transcriptomic data, clinical metadata, pathologic assessments of cell composition, and, in several cases, detailed pathologic annotations of neoplastic, stromal, and necrotic regions. The amenability of these images to deep learning was highlighted through three applications: (i) development of a classifier for neoplastic, stromal, and necrotic regions; (ii) development of a predictor of xenograft-transplant lymphoproliferative disorder; and (iii) application of a published predictor of microsatellite instability. Together, this PDX Development and Trial Centers Research Network image repository provides a valuable resource for controlled digital pathology analysis, both for the evaluation of technical issues and for the development of computational image-based methods that make clinical predictions based on PDX treatment studies. Significance: A pan-cancer repository of >1,000 patient-derived xenograft hematoxylin and eosin-stained images will facilitate cancer biology investigations through histopathologic analysis and contributes important model system data that expand existing human histology repositories.
Collapse
Affiliation(s)
- Brian S. White
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | - Xing Yi Woo
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
- Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore, Singapore.
| | - Soner Koc
- Velsera, Charlestown, Massachusetts.
| | - Todd Sheridan
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | | | - Shidan Wang
- University of Texas Southwestern Medical Center, Dallas, Texas.
| | - Yvonne A. Evrard
- Leidos Biomedical Research Inc., Frederick National Laboratory for Cancer Research, Frederick, Maryland.
| | - Li Chen
- Leidos Biomedical Research Inc., Frederick National Laboratory for Cancer Research, Frederick, Maryland.
| | - Ali Foroughi pour
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | | | - R. Jay Mashl
- Washington University School of Medicine, St. Louis, Missouri.
| | | | - Bingliang Fang
- University of Texas MD Anderson Cancer Center, Houston, Texas.
| | | | - Kurt W. Evans
- University of Texas MD Anderson Cancer Center, Houston, Texas.
| | - Matthew H. Bailey
- Simmons Center for Cancer Research, Brigham Young University, Provo, Utah.
| | - Yeqing Chen
- The Wistar Institute, Philadelphia, Pennsylvania.
| | - Min Xiao
- The Wistar Institute, Philadelphia, Pennsylvania.
| | | | | | | | - Sergii Domanskyi
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | | | - Maihi Fujita
- Huntsman Cancer Institute, University of Utah, Salt Lake City, Utah.
| | - Junya Fujimoto
- University of Texas MD Anderson Cancer Center, Houston, Texas.
| | - Guanghua Xiao
- University of Texas Southwestern Medical Center, Dallas, Texas.
| | - Ryan C. Fields
- Washington University School of Medicine, St. Louis, Missouri.
| | | | - Xiaowei Xu
- The Wistar Institute, Philadelphia, Pennsylvania.
| | | | - Shahanawaz Jiwani
- Leidos Biomedical Research Inc., Frederick National Laboratory for Cancer Research, Frederick, Maryland.
| | | | | | | | - Peter N. Robinson
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| | | | | | | | | | | | | | - Alana L. Welm
- Huntsman Cancer Institute, University of Utah, Salt Lake City, Utah.
| | - Bryan E. Welm
- Huntsman Cancer Institute, University of Utah, Salt Lake City, Utah.
| | | | - Shunqiang Li
- Washington University School of Medicine, St. Louis, Missouri.
| | | | - Jack A. Roth
- University of Texas MD Anderson Cancer Center, Houston, Texas.
| | | | - Yang Xie
- University of Texas Southwestern Medical Center, Dallas, Texas.
| | | | - Li Ding
- Washington University School of Medicine, St. Louis, Missouri.
| | | | | | | | - Jeffrey H. Chuang
- The Jackson Laboratory for Genomic Medicine, Farmington, Connecticut.
| |
Collapse
|
8
|
Volinsky-Fremond S, Horeweg N, Andani S, Barkey Wolf J, Lafarge MW, de Kroon CD, Ørtoft G, Høgdall E, Dijkstra J, Jobsen JJ, Lutgens LCHW, Powell ME, Mileshkin LR, Mackay H, Leary A, Katsaros D, Nijman HW, de Boer SM, Nout RA, de Bruyn M, Church D, Smit VTHBM, Creutzberg CL, Koelzer VH, Bosse T. Prediction of recurrence risk in endometrial cancer with multimodal deep learning. Nat Med 2024; 30:1962-1973. [PMID: 38789645 PMCID: PMC11271412 DOI: 10.1038/s41591-024-02993-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 04/11/2024] [Indexed: 05/26/2024]
Abstract
Predicting distant recurrence of endometrial cancer (EC) is crucial for personalized adjuvant treatment. The current gold standard of combined pathological and molecular profiling is costly, hampering implementation. Here we developed HECTOR (histopathology-based endometrial cancer tailored outcome risk), a multimodal deep learning prognostic model using hematoxylin and eosin-stained, whole-slide images and tumor stage as input, on 2,072 patients from eight EC cohorts including the PORTEC-1/-2/-3 randomized trials. HECTOR demonstrated C-indices in internal (n = 353) and two external (n = 160 and n = 151) test sets of 0.789, 0.828 and 0.815, respectively, outperforming the current gold standard, and identified patients with markedly different outcomes (10-year distant recurrence-free probabilities of 97.0%, 77.7% and 58.1% for HECTOR low-, intermediate- and high-risk groups, respectively, by Kaplan-Meier analysis). HECTOR also predicted adjuvant chemotherapy benefit better than current methods. Morphological and genomic feature extraction identified correlates of HECTOR risk groups, some with therapeutic potential. HECTOR improves on the current gold standard and may help delivery of personalized treatment in EC.
Collapse
Affiliation(s)
| | - Nanda Horeweg
- Department of Radiation Oncology, Leiden University Medical Center, Leiden, The Netherlands
| | - Sonali Andani
- Department of Computer Science, ETH Zurich, Zurich, Switzerland
- Department of Pathology and Molecular Pathology, University Hospital, University of Zurich, Zurich, Switzerland
- Swiss Institute of Bioinformatics, Lausanne, Switzerland
| | - Jurriaan Barkey Wolf
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands
| | - Maxime W Lafarge
- Department of Pathology and Molecular Pathology, University Hospital, University of Zurich, Zurich, Switzerland
| | - Cor D de Kroon
- Department of Gynecology and Obstetrics, Leiden University Medical Center, Leiden, The Netherlands
| | - Gitte Ørtoft
- Department of Gynecology, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark
| | - Estrid Høgdall
- Department of Pathology, Herlev University Hospital, Herlev, Denmark
| | - Jouke Dijkstra
- Department of Radiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Jan J Jobsen
- Department of Radiation Oncology, Medisch Spectrum Twente, Enschede, The Netherlands
| | | | - Melanie E Powell
- Department of Clinical Oncology, Barts Health NHS Trust, London, UK
| | - Linda R Mileshkin
- Department of Medical Oncology, Peter MacCallum Cancer Center, Melbourne, Victoria, Australia
| | - Helen Mackay
- Department of Medical Oncology and Hematology, Odette Cancer Center Sunnybrook Health Sciences Center, Toronto, Ontario, Canada
| | - Alexandra Leary
- Department Medical Oncology, Gustave Roussy Institute, Villejuif, France
| | - Dionyssios Katsaros
- Department of Surgical Sciences, Gynecologic Oncology, Città della Salute and S Anna Hospital, University of Turin, Turin, Italy
| | - Hans W Nijman
- Department of Obstetrics and Gynecology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands
| | - Stephanie M de Boer
- Department of Radiation Oncology, Leiden University Medical Center, Leiden, The Netherlands
| | - Remi A Nout
- Department of Radiotherapy, Erasmus MC Cancer Institute, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Marco de Bruyn
- Department of Obstetrics and Gynecology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands
| | - David Church
- Wellcome Centre for Human Genetics, University of Oxford, Oxford, UK
- Oxford NIHR Comprehensive Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Vincent T H B M Smit
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands
| | - Carien L Creutzberg
- Department of Radiation Oncology, Leiden University Medical Center, Leiden, The Netherlands
| | - Viktor H Koelzer
- Department of Pathology and Molecular Pathology, University Hospital, University of Zurich, Zurich, Switzerland
- Institute of Medical Genetics and Pathology, University Hospital Basel, Basel, Switzerland
| | - Tjalling Bosse
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands.
| |
Collapse
|
9
|
Yan C, Sun J, Guan Y, Feng J, Liu H, Liu J. PhiHER2: phenotype-informed weakly supervised model for HER2 status prediction from pathological images. Bioinformatics 2024; 40:i79-i90. [PMID: 38940163 PMCID: PMC11211833 DOI: 10.1093/bioinformatics/btae236] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/29/2024] Open
Abstract
MOTIVATION Human epidermal growth factor receptor 2 (HER2) status identification enables physicians to assess the prognosis risk and determine the treatment schedule for patients. In clinical practice, pathological slides serve as the gold standard, offering morphological information on cellular structure and tumoral regions. Computational analysis of pathological images has the potential to discover morphological patterns associated with HER2 molecular targets and achieve precise status prediction. However, pathological images are typically equipped with high-resolution attributes, and HER2 expression in breast cancer (BC) images often manifests the intratumoral heterogeneity. RESULTS We present a phenotype-informed weakly supervised multiple instance learning architecture (PhiHER2) for the prediction of the HER2 status from pathological images of BC. Specifically, a hierarchical prototype clustering module is designed to identify representative phenotypes across whole slide images. These phenotype embeddings are then integrated into a cross-attention module, enhancing feature interaction and aggregation on instances. This yields a phenotype-based feature space that leverages the intratumoral morphological heterogeneity for HER2 status prediction. Extensive results demonstrate that PhiHER2 captures a better WSI-level representation by the typical phenotype guidance and significantly outperforms existing methods on real-world datasets. Additionally, interpretability analyses of both phenotypes and WSIs provide explicit insights into the heterogeneity of morphological patterns associated with molecular HER2 status. AVAILABILITY AND IMPLEMENTATION Our model is available at https://github.com/lyotvincent/PhiHER2.
Collapse
Affiliation(s)
- Chaoyang Yan
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| | - Jialiang Sun
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| | - Yiming Guan
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| | - Jiuxin Feng
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| | - Hong Liu
- The Second Surgical Department of Breast Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute & Hospital, Tianjin 300060, China
| | - Jian Liu
- College of Computer Science, Nankai University, Tianjin 300071, China
- Centre for Bioinformatics and Intelligent Medicine, Nankai University, Tianjin 300071, China
| |
Collapse
|
10
|
Perez-Lopez R, Ghaffari Laleh N, Mahmood F, Kather JN. A guide to artificial intelligence for cancer researchers. Nat Rev Cancer 2024; 24:427-441. [PMID: 38755439 DOI: 10.1038/s41568-024-00694-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/09/2024] [Indexed: 05/18/2024]
Abstract
Artificial intelligence (AI) has been commoditized. It has evolved from a specialty resource to a readily accessible tool for cancer researchers. AI-based tools can boost research productivity in daily workflows, but can also extract hidden information from existing data, thereby enabling new scientific discoveries. Building a basic literacy in these tools is useful for every cancer researcher. Researchers with a traditional biological science focus can use AI-based tools through off-the-shelf software, whereas those who are more computationally inclined can develop their own AI-based software pipelines. In this article, we provide a practical guide for non-computational cancer researchers to understand how AI-based tools can benefit them. We convey general principles of AI for applications in image analysis, natural language processing and drug discovery. In addition, we give examples of how non-computational researchers can get started on the journey to productively use AI in their own work.
Collapse
Affiliation(s)
- Raquel Perez-Lopez
- Radiomics Group, Vall d'Hebron Institute of Oncology, Vall d'Hebron Barcelona Hospital Campus, Barcelona, Spain
| | - Narmin Ghaffari Laleh
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
- Department of Medicine I, University Hospital Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumour Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
11
|
Niehues JM, Müller-Franzes G, Schirris Y, Wagner SJ, Jendrusch M, Kloor M, Pearson AT, Muti HS, Hewitt KJ, Veldhuizen GP, Zigutyte L, Truhn D, Kather JN. Using histopathology latent diffusion models as privacy-preserving dataset augmenters improves downstream classification performance. Comput Biol Med 2024; 175:108410. [PMID: 38678938 DOI: 10.1016/j.compbiomed.2024.108410] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 03/23/2024] [Accepted: 04/02/2024] [Indexed: 05/01/2024]
Abstract
Latent diffusion models (LDMs) have emerged as a state-of-the-art image generation method, outperforming previous Generative Adversarial Networks (GANs) in terms of training stability and image quality. In computational pathology, generative models are valuable for data sharing and data augmentation. However, the impact of LDM-generated images on histopathology tasks compared to traditional GANs has not been systematically studied. We trained three LDMs and a styleGAN2 model on histology tiles from nine colorectal cancer (CRC) tissue classes. The LDMs include 1) a fine-tuned version of stable diffusion v1.4, 2) a Kullback-Leibler (KL)-autoencoder (KLF8-DM), and 3) a vector quantized (VQ)-autoencoder deploying LDM (VQF8-DM). We assessed image quality through expert ratings, dimensional reduction methods, distribution similarity measures, and their impact on training a multiclass tissue classifier. Additionally, we investigated image memorization in the KLF8-DM and styleGAN2 models. All models provided a high image quality, with the KLF8-DM achieving the best Frechet Inception Distance (FID) and expert rating scores for complex tissue classes. For simpler classes, the VQF8-DM and styleGAN2 models performed better. Image memorization was negligible for both styleGAN2 and KLF8-DM models. Classifiers trained on a mix of KLF8-DM generated and real images achieved a 4% improvement in overall classification accuracy, highlighting the usefulness of these images for dataset augmentation. Our systematic study of generative methods showed that KLF8-DM produces the highest quality images with negligible image memorization. The higher classifier performance in the generatively augmented dataset suggests that this augmentation technique can be employed to enhance histopathology classifiers for various tasks.
Collapse
Affiliation(s)
- Jan M Niehues
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Gustav Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Yoni Schirris
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Netherlands Cancer Institute, 1066 CX, Amsterdam, the Netherlands; University of Amsterdam, 1012 WP, Amsterdam, the Netherlands
| | - Sophia Janine Wagner
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany; School of Computation, Information and Technology, Technical University of Munich, Munich, Germany
| | - Michael Jendrusch
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany
| | - Matthias Kloor
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany
| | | | - Hannah Sophie Muti
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Katherine J Hewitt
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Gregory P Veldhuizen
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Laura Zigutyte
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom; Department of Medicine I, University Hospital Dresden, Dresden, Germany; Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
12
|
Gustav M, Reitsam NG, Carrero ZI, Loeffler CML, van Treeck M, Yuan T, West NP, Quirke P, Brinker TJ, Brenner H, Favre L, Märkl B, Stenzinger A, Brobeil A, Hoffmeister M, Calderaro J, Pujals A, Kather JN. Deep learning for dual detection of microsatellite instability and POLE mutations in colorectal cancer histopathology. NPJ Precis Oncol 2024; 8:115. [PMID: 38783059 PMCID: PMC11116442 DOI: 10.1038/s41698-024-00592-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Accepted: 04/14/2024] [Indexed: 05/25/2024] Open
Abstract
In the spectrum of colorectal tumors, microsatellite-stable (MSS) tumors with DNA polymerase ε (POLE) mutations exhibit a hypermutated profile, holding the potential to respond to immunotherapy similarly to their microsatellite-instable (MSI) counterparts. Yet, due to their rarity and the associated testing costs, systematic screening for these mutations is not commonly pursued. Notably, the histopathological phenotype resulting from POLE mutations is theorized to resemble that of MSI. This resemblance not only could facilitate their detection by a transformer-based Deep Learning (DL) system trained on MSI pathology slides, but also indicates the possibility for MSS patients with POLE mutations to access enhanced treatment options, which might otherwise be overlooked. To harness this potential, we trained a Deep Learning classifier on a large dataset with the ground truth for microsatellite status and subsequently validated its capabilities for MSI and POLE detection across three external cohorts. Our model accurately identified MSI status in both the internal and external resection cohorts using pathology images alone. Notably, with a classification threshold of 0.5, over 75% of POLE driver mutant patients in the external resection cohorts were flagged as "positive" by a DL system trained on MSI status. In a clinical setting, deploying this DL model as a preliminary screening tool could facilitate the efficient identification of clinically relevant MSI and POLE mutations in colorectal tumors, in one go.
Collapse
Affiliation(s)
- Marco Gustav
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | | | - Zunamys I Carrero
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Chiara M L Loeffler
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine I, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Marko van Treeck
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Tanwei Yuan
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Nicholas P West
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
| | - Philip Quirke
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
| | - Titus J Brinker
- Digital Biomarkers for Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), Heidelberg, Germany
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Loëtitia Favre
- Université Paris Est Créteil, INSERM, IMRB, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- INSERM, U955, Team Oncogenèse des lymphomes et tumeurs de la Neurofibromatose 1, Créteil, France
| | - Bruno Märkl
- Pathology, Faculty of Medicine, University of Augsburg, Augsburg, Germany
| | | | - Alexander Brobeil
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany
- Tissue Bank of the National Center for Tumor Diseases (NCT) Heidelberg, Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Julien Calderaro
- Université Paris Est Créteil, INSERM, IMRB, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- INSERM, U955, Team Oncogenèse des lymphomes et tumeurs de la Neurofibromatose 1, Créteil, France
| | - Anaïs Pujals
- Université Paris Est Créteil, INSERM, IMRB, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- INSERM, U955, Team Oncogenèse des lymphomes et tumeurs de la Neurofibromatose 1, Créteil, France
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
- Department of Medicine I, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany.
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
13
|
Gao R, Yuan X, Ma Y, Wei T, Johnston L, Shao Y, Lv W, Zhu T, Zhang Y, Zheng J, Chen G, Sun J, Wang YG, Yu Z. Harnessing TME depicted by histological images to improve cancer prognosis through a deep learning system. Cell Rep Med 2024; 5:101536. [PMID: 38697103 PMCID: PMC11149411 DOI: 10.1016/j.xcrm.2024.101536] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Revised: 02/26/2024] [Accepted: 04/08/2024] [Indexed: 05/04/2024]
Abstract
Spatial transcriptomics (ST) provides insights into the tumor microenvironment (TME), which is closely associated with cancer prognosis, but ST has limited clinical availability. In this study, we provide a powerful deep learning system to augment TME information based on histological images for patients without ST data, thereby empowering precise cancer prognosis. The system provides two connections to bridge existing gaps. The first is the integrated graph and image deep learning (IGI-DL) model, which predicts ST expression based on histological images with a 0.171 increase in mean correlation across three cancer types compared with five existing methods. The second connection is the cancer prognosis prediction model, based on TME depicted by spatial gene expression. Our survival model, using graphs with predicted ST features, achieves superior accuracy with a concordance index of 0.747 and 0.725 for The Cancer Genome Atlas breast cancer and colorectal cancer cohorts, outperforming other survival models. For the external Molecular and Cellular Oncology colorectal cancer cohort, our survival model maintains a stable advantage.
Collapse
Affiliation(s)
- Ruitian Gao
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Xin Yuan
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Yanran Ma
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Ting Wei
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Luke Johnston
- School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Yanfei Shao
- Department of General Surgery, Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Wenwen Lv
- Clinical Research Institute, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Tengteng Zhu
- Clinical Research Institute, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Yue Zhang
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Junke Zheng
- Key Laboratory of Cell Differentiation and Apoptosis of the Chinese Ministry of Education, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Guoqiang Chen
- State Key Laboratory of Systems Medicine for Cancer, Ren-Ji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Jing Sun
- Department of General Surgery, Shanghai Minimally Invasive Surgery Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China.
| | - Yu Guang Wang
- School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai 200240, China; Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai 200240, China; Zhangjiang Institute of Advanced Research, Shanghai Jiao Tong University, Shanghai 201203, China.
| | - Zhangsheng Yu
- Department of Bioinformatics and Biostatistics, School of Life Sciences and Biotechnology, Shanghai Jiao Tong University, Shanghai 200240, China; SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai 200240, China; School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai 200240, China; Clinical Research Institute, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China.
| |
Collapse
|
14
|
Tian F, Liu D, Wei N, Fu Q, Sun L, Liu W, Sui X, Tian K, Nemeth G, Feng J, Xu J, Xiao L, Han J, Fu J, Shi Y, Yang Y, Liu J, Hu C, Feng B, Sun Y, Wang Y, Yu G, Kong D, Wang M, Li W, Chen K, Li X. Prediction of tumor origin in cancers of unknown primary origin with cytology-based deep learning. Nat Med 2024; 30:1309-1319. [PMID: 38627559 PMCID: PMC11108774 DOI: 10.1038/s41591-024-02915-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2023] [Accepted: 03/07/2024] [Indexed: 04/26/2024]
Abstract
Cancer of unknown primary (CUP) site poses diagnostic challenges due to its elusive nature. Many cases of CUP manifest as pleural and peritoneal serous effusions. Leveraging cytological images from 57,220 cases at four tertiary hospitals, we developed a deep-learning method for tumor origin differentiation using cytological histology (TORCH) that can identify malignancy and predict tumor origin in both hydrothorax and ascites. We examined its performance on three internal (n = 12,799) and two external (n = 14,538) testing sets. In both internal and external testing sets, TORCH achieved area under the receiver operating curve values ranging from 0.953 to 0.991 for cancer diagnosis and 0.953 to 0.979 for tumor origin localization. TORCH accurately predicted primary tumor origins, with a top-1 accuracy of 82.6% and top-3 accuracy of 98.9%. Compared with results derived from pathologists, TORCH showed better prediction efficacy (1.677 versus 1.265, P < 0.001), enhancing junior pathologists' diagnostic scores significantly (1.326 versus 1.101, P < 0.001). Patients with CUP whose initial treatment protocol was concordant with TORCH-predicted origins had better overall survival than those who were administrated discordant treatment (27 versus 17 months, P = 0.006). Our study underscores the potential of TORCH as a valuable ancillary tool in clinical practice, although further validation in randomized trials is warranted.
Collapse
Affiliation(s)
- Fei Tian
- Department of Abdominal Cancer, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Dong Liu
- Department of Radiology, The First Affiliated Hospital of Suzhou University, Suzhou, China
| | - Na Wei
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Qianqian Fu
- Department of Pathology, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Lin Sun
- Department of Pathology, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Wei Liu
- Department of Pathology, The First Affiliated Hospital of Suzhou University, Suzhou, China
| | - Xiaolong Sui
- Department of Pathology, Yantai Yuhuangding Hospital of Qingdao University, Yantai, China
| | - Kathryn Tian
- Harvard Dunster House, Harvard University, Cambridge, MA, USA
| | | | - Jingyu Feng
- School of Biological Science and Medical Engineering, Beihang University, Beijing, China
| | - Jingjing Xu
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Lin Xiao
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Junya Han
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Jingjie Fu
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Yinhua Shi
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Yichen Yang
- Tianjin Cancer Institute, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Jia Liu
- Department of Abdominal Cancer, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Chunhong Hu
- Department of Radiology, The First Affiliated Hospital of Suzhou University, Suzhou, China
| | - Bin Feng
- Department of Pathology, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Yan Sun
- Department of Pathology, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Yunjun Wang
- Department of Pathology, Yantai Yuhuangding Hospital of Qingdao University, Yantai, China
| | - Guohua Yu
- Department of Pathology, Yantai Yuhuangding Hospital of Qingdao University, Yantai, China
| | - Dalu Kong
- Department of Abdominal Cancer, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China
| | - Meiyun Wang
- Department of Radiology, Henan Provincial People's Hospital, The People's Hospital of Zhengzhou University, Zhengzhou, China
| | - Wencai Li
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China.
| | - Kexin Chen
- Department of Epidemiology and Biostatistics, Tianjin's Clinical Research Center for Cancer, Key Laboratory of Molecular Cancer Epidemiology of Tianjin, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China.
| | - Xiangchun Li
- Tianjin Cancer Institute, Tianjin's Clinical Research Center for Cancer, Tianjin Key Laboratory of Digestive Cancer, National Clinical Research Center for Cancer, Tianjin Medical University Cancer Institute and Hospital, Tianjin Medical University, Tianjin, China.
| |
Collapse
|
15
|
Yuan L, Zhou H, Xiao X, Zhang X, Chen F, Liu L, Liu J, Bao S, Tao K. Development and external validation of a transfer learning-based system for the pathological diagnosis of colorectal cancer: a large emulated prospective study. Front Oncol 2024; 14:1365364. [PMID: 38725622 PMCID: PMC11079287 DOI: 10.3389/fonc.2024.1365364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Accepted: 04/11/2024] [Indexed: 05/12/2024] Open
Abstract
Background The progress in Colorectal cancer (CRC) screening and management has resulted in an unprecedented caseload for histopathological diagnosis. While artificial intelligence (AI) presents a potential solution, the predominant emphasis on slide-level aggregation performance without thorough verification of cancer in each location, impedes both explainability and transparency. Effectively addressing these challenges is crucial to ensuring the reliability and efficacy of AI in histology applications. Method In this study, we created an innovative AI algorithm using transfer learning from a polyp segmentation model in endoscopy. The algorithm precisely localized CRC targets within 0.25 mm² grids from whole slide imaging (WSI). We assessed the CRC detection capabilities at this fine granularity and examined the influence of AI on the diagnostic behavior of pathologists. The evaluation utilized an extensive dataset comprising 858 consecutive patient cases with 1418 WSIs obtained from an external center. Results Our results underscore a notable sensitivity of 90.25% and specificity of 96.60% at the grid level, accompanied by a commendable area under the curve (AUC) of 0.962. This translates to an impressive 99.39% sensitivity at the slide level, coupled with a negative likelihood ratio of <0.01, signifying the dependability of the AI system to preclude diagnostic considerations. The positive likelihood ratio of 26.54, surpassing 10 at the grid level, underscores the imperative for meticulous scrutiny of any AI-generated highlights. Consequently, all four participating pathologists demonstrated statistically significant diagnostic improvements with AI assistance. Conclusion Our transfer learning approach has successfully yielded an algorithm that can be validated for CRC histological localizations in whole slide imaging. The outcome advocates for the integration of the AI system into histopathological diagnosis, serving either as a diagnostic exclusion application or a computer-aided detection (CADe) tool. This integration has the potential to alleviate the workload of pathologists and ultimately benefit patients.
Collapse
Affiliation(s)
- Liuhong Yuan
- Department of Pathology, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | - Henghua Zhou
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | | | - Xiuqin Zhang
- Department of Pathology, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | - Feier Chen
- Department of Pathology, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | - Lin Liu
- Institute of Natural Sciences, MOE-LSC, School of Mathematical Sciences, CMA-Shanghai, SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University, Shanghai, China
| | | | - Shisan Bao
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| | - Kun Tao
- Department of Pathology, Tongji Hospital, School of Medicine, Tongji University, Shanghai, China
- Department of Pathology, Tongren Hospital, School of Medicine Shanghai Jiaotong University, Shanghai, China
| |
Collapse
|
16
|
Zamanitajeddin N, Jahanifar M, Bilal M, Eastwood M, Rajpoot N. Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer. Med Image Anal 2024; 93:103071. [PMID: 38199068 DOI: 10.1016/j.media.2023.103071] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Revised: 11/14/2023] [Accepted: 12/19/2023] [Indexed: 01/12/2024]
Abstract
Colorectal cancer (CRC) is a primary global health concern, and identifying the molecular pathways, genetic subtypes, and mutations associated with CRC is crucial for precision medicine. However, traditional measurement techniques such as gene sequencing are costly and time-consuming, while most deep learning methods proposed for this task lack interpretability. This study offers a new approach to enhance the state-of-the-art deep learning methods for molecular pathways and key mutation prediction by incorporating cell network information. We build cell graphs with nuclei as nodes and nuclei connections as edges of the network and leverage Social Network Analysis (SNA) measures to extract abstract, perceivable, and interpretable features that explicitly describe the cell network characteristics in an image. Our approach does not rely on precise nuclei segmentation or feature extraction, is computationally efficient, and is easily scalable. In this study, we utilize the TCGA-CRC-DX dataset, comprising 499 patients and 502 diagnostic slides from primary colorectal tumours, sourced from 36 distinct medical centres in the United States. By incorporating the SNA features alongside deep features in two multiple instance learning frameworks, we demonstrate improved performance for chromosomal instability (CIN), hypermutated tumour (HM), TP53 gene, BRAF gene, and Microsatellite instability (MSI) status prediction tasks (2.4%-4% and 7-8.8% improvement in AUROC and AUPRC on average). Additionally, our method achieves outstanding performance on MSI prediction in an external PAIP dataset (99% AUROC and 98% AUPRC), demonstrating its generalizability. Our findings highlight the discrimination power of SNA features and how they can be beneficial to deep learning models' performance and provide insights into the correlation of cell network profiles with molecular pathways and key mutations.
Collapse
Affiliation(s)
- Neda Zamanitajeddin
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK.
| | - Mostafa Jahanifar
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Mohsin Bilal
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Mark Eastwood
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Nasir Rajpoot
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK; Histofy Ltd., Birmingham, UK.
| |
Collapse
|
17
|
Unger M, Kather JN. Deep learning in cancer genomics and histopathology. Genome Med 2024; 16:44. [PMID: 38539231 PMCID: PMC10976780 DOI: 10.1186/s13073-024-01315-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2023] [Accepted: 03/13/2024] [Indexed: 07/08/2024] Open
Abstract
Histopathology and genomic profiling are cornerstones of precision oncology and are routinely obtained for patients with cancer. Traditionally, histopathology slides are manually reviewed by highly trained pathologists. Genomic data, on the other hand, is evaluated by engineered computational pipelines. In both applications, the advent of modern artificial intelligence methods, specifically machine learning (ML) and deep learning (DL), have opened up a fundamentally new way of extracting actionable insights from raw data, which could augment and potentially replace some aspects of traditional evaluation workflows. In this review, we summarize current and emerging applications of DL in histopathology and genomics, including basic diagnostic as well as advanced prognostic tasks. Based on a growing body of evidence, we suggest that DL could be the groundwork for a new kind of workflow in oncology and cancer research. However, we also point out that DL models can have biases and other flaws that users in healthcare and research need to know about, and we propose ways to address them.
Collapse
Affiliation(s)
- Michaela Unger
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany.
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany.
- Department of Medicine I, University Hospital Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
18
|
Howard FM, Hieromnimon HM, Ramesh S, Dolezal J, Kochanny S, Zhang Q, Feiger B, Peterson J, Fan C, Perou CM, Vickery J, Sullivan M, Cole K, Khramtsova G, Pearson AT. Generative Adversarial Networks Accurately Reconstruct Pan-Cancer Histology from Pathologic, Genomic, and Radiographic Latent Features. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.22.586306. [PMID: 38585926 PMCID: PMC10996476 DOI: 10.1101/2024.03.22.586306] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/09/2024]
Abstract
Artificial intelligence models have been increasingly used in the analysis of tumor histology to perform tasks ranging from routine classification to identification of novel molecular features. These approaches distill cancer histologic images into high-level features which are used in predictions, but understanding the biologic meaning of such features remains challenging. We present and validate a custom generative adversarial network - HistoXGAN - capable of reconstructing representative histology using feature vectors produced by common feature extractors. We evaluate HistoXGAN across 29 cancer subtypes and demonstrate that reconstructed images retain information regarding tumor grade, histologic subtype, and gene expression patterns. We leverage HistoXGAN to illustrate the underlying histologic features for deep learning models for actionable mutations, identify model reliance on histologic batch effect in predictions, and demonstrate accurate reconstruction of tumor histology from radiographic imaging for a 'virtual biopsy'.
Collapse
|
19
|
Liu B, Polack M, Coudray N, Quiros AC, Sakellaropoulos T, Crobach ASLP, van Krieken JHJM, Yuan K, Tollenaar RAEM, Mesker WE, Tsirigos A. Self-Supervised Learning Reveals Clinically Relevant Histomorphological Patterns for Therapeutic Strategies in Colon Cancer. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.26.582106. [PMID: 38496571 PMCID: PMC10942268 DOI: 10.1101/2024.02.26.582106] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/19/2024]
Abstract
Self-supervised learning (SSL) automates the extraction and interpretation of histopathology features on unannotated hematoxylin-and-eosin-stained whole-slide images (WSIs). We trained an SSL Barlow Twins-encoder on 435 TCGA colon adenocarcinoma WSIs to extract features from small image patches. Leiden community detection then grouped tiles into histomorphological phenotype clusters (HPCs). HPC reproducibility and predictive ability for overall survival was confirmed in an independent clinical trial cohort (N=1213 WSIs). This unbiased atlas resulted in 47 HPCs displaying unique and sharing clinically significant histomorphological traits, highlighting tissue type, quantity, and architecture, especially in the context of tumor stroma. Through in-depth analysis of these HPCs, including immune landscape and gene set enrichment analysis, and association to clinical outcomes, we shed light on the factors influencing survival and responses to treatments like standard adjuvant chemotherapy and experimental therapies. Further exploration of HPCs may unveil new insights and aid decision-making and personalized treatments for colon cancer patients.
Collapse
|
20
|
Chen RJ, Ding T, Lu MY, Williamson DFK, Jaume G, Song AH, Chen B, Zhang A, Shao D, Shaban M, Williams M, Oldenburg L, Weishaupt LL, Wang JJ, Vaidya A, Le LP, Gerber G, Sahai S, Williams W, Mahmood F. Towards a general-purpose foundation model for computational pathology. Nat Med 2024; 30:850-862. [PMID: 38504018 DOI: 10.1038/s41591-024-02857-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Accepted: 02/05/2024] [Indexed: 03/21/2024]
Abstract
Quantitative evaluation of tissue images is crucial for computational pathology (CPath) tasks, requiring the objective characterization of histopathological entities from whole-slide images (WSIs). The high resolution of WSIs and the variability of morphological features present significant challenges, complicating the large-scale annotation of data for high-performance applications. To address this challenge, current efforts have proposed the use of pretrained image encoders through transfer learning from natural image datasets or self-supervised learning on publicly available histopathology datasets, but have not been extensively developed and evaluated across diverse tissue types at scale. We introduce UNI, a general-purpose self-supervised model for pathology, pretrained using more than 100 million images from over 100,000 diagnostic H&E-stained WSIs (>77 TB of data) across 20 major tissue types. The model was evaluated on 34 representative CPath tasks of varying diagnostic difficulty. In addition to outperforming previous state-of-the-art models, we demonstrate new modeling capabilities in CPath such as resolution-agnostic tissue classification, slide classification using few-shot class prototypes, and disease subtyping generalization in classifying up to 108 cancer types in the OncoTree classification system. UNI advances unsupervised representation learning at scale in CPath in terms of both pretraining data and downstream evaluation, enabling data-efficient artificial intelligence models that can generalize and transfer to a wide range of diagnostically challenging tasks and clinical workflows in anatomic pathology.
Collapse
Affiliation(s)
- Richard J Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Tong Ding
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Ming Y Lu
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT), Cambridge, MA, USA
| | - Drew F K Williamson
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
| | - Guillaume Jaume
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Andrew H Song
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Bowen Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Andrew Zhang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Daniel Shao
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Muhammad Shaban
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Mane Williams
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Lukas Oldenburg
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Luca L Weishaupt
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Judy J Wang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Anurag Vaidya
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Long Phi Le
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Georg Gerber
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Sharifa Sahai
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Systems Biology, Harvard University, Cambridge, MA, USA
| | - Walt Williams
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA.
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA.
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
21
|
Uchikov P, Khalid U, Kraev K, Hristov B, Kraeva M, Tenchev T, Chakarov D, Sandeva M, Dragusheva S, Taneva D, Batashki A. Artificial Intelligence in the Diagnosis of Colorectal Cancer: A Literature Review. Diagnostics (Basel) 2024; 14:528. [PMID: 38472999 DOI: 10.3390/diagnostics14050528] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2024] [Revised: 02/26/2024] [Accepted: 02/27/2024] [Indexed: 03/14/2024] Open
Abstract
BACKGROUND The aim of this review is to explore the role of artificial intelligence in the diagnosis of colorectal cancer, how it impacts CRC morbidity and mortality, and why its role in clinical medicine is limited. METHODS A targeted, non-systematic review of the published literature relating to colorectal cancer diagnosis was performed with PubMed databases that were scouted to help provide a more defined understanding of the recent advances regarding artificial intelligence and their impact on colorectal-related morbidity and mortality. Articles were included if deemed relevant and including information associated with the keywords. RESULTS The advancements in artificial intelligence have been significant in facilitating an earlier diagnosis of CRC. In this review, we focused on evaluating genomic biomarkers, the integration of instruments with artificial intelligence, MR and hyperspectral imaging, and the architecture of neural networks. We found that these neural networks seem practical and yield positive results in initial testing. Furthermore, we explored the use of deep-learning-based majority voting methods, such as bag of words and PAHLI, in improving diagnostic accuracy in colorectal cancer detection. Alongside this, the autonomous and expansive learning ability of artificial intelligence, coupled with its ability to extract increasingly complex features from images or videos without human reliance, highlight its impact in the diagnostic sector. Despite this, as most of the research involves a small sample of patients, a diversification of patient data is needed to enhance cohort stratification for a more sensitive and specific neural model. We also examined the successful application of artificial intelligence in predicting microsatellite instability, showcasing its potential in stratifying patients for targeted therapies. CONCLUSIONS Since its commencement in colorectal cancer, artificial intelligence has revealed a multitude of functionalities and augmentations in the diagnostic sector of CRC. Given its early implementation, its clinical application remains a fair way away, but with steady research dedicated to improving neural architecture and expanding its applicational range, there is hope that these advanced neural software could directly impact the early diagnosis of CRC. The true promise of artificial intelligence, extending beyond the medical sector, lies in its potential to significantly influence the future landscape of CRC's morbidity and mortality.
Collapse
Affiliation(s)
- Petar Uchikov
- Department of Special Surgery, Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Usman Khalid
- Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Krasimir Kraev
- Department of Propaedeutics of Internal Diseases "Prof. Dr. Anton Mitov", Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Bozhidar Hristov
- Section "Gastroenterology", Second Department of Internal Diseases, Medical Faculty, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Maria Kraeva
- Department of Otorhinolaryngology, Medical Faculty, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Tihomir Tenchev
- Department of Special Surgery, Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Dzhevdet Chakarov
- Department of Propaedeutics of Surgical Diseases, Section of General Surgery, Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| | - Milena Sandeva
- Department of Midwifery, Faculty of Public Health, Medical University of Plovdiv, 4000 Plovdiv, Bulgaria
| | - Snezhanka Dragusheva
- Department of Nursing Care, Faculty of Public Health, Medical University of Plovdiv, 4000 Plovdiv, Bulgaria
| | - Daniela Taneva
- Department of Nursing Care, Faculty of Public Health, Medical University of Plovdiv, 4000 Plovdiv, Bulgaria
| | - Atanas Batashki
- Department of Special Surgery, Faculty of Medicine, Medical University of Plovdiv, 4002 Plovdiv, Bulgaria
| |
Collapse
|
22
|
Ligero M, Serna G, El Nahhas OS, Sansano I, Mauchanski S, Viaplana C, Calderaro J, Toledo RA, Dienstmann R, Vanguri RS, Sauter JL, Sanchez-Vega F, Shah SP, Ramón y Cajal S, Garralda E, Nuciforo P, Perez-Lopez R, Kather JN. Weakly Supervised Deep Learning Predicts Immunotherapy Response in Solid Tumors Based on PD-L1 Expression. CANCER RESEARCH COMMUNICATIONS 2024; 4:92-102. [PMID: 38126740 PMCID: PMC10782919 DOI: 10.1158/2767-9764.crc-23-0287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/06/2023] [Revised: 10/11/2023] [Accepted: 12/12/2023] [Indexed: 12/23/2023]
Abstract
Programmed death-ligand 1 (PD-L1) IHC is the most commonly used biomarker for immunotherapy response. However, quantification of PD-L1 status in pathology slides is challenging. Neither manual quantification nor a computer-based mimicking of manual readouts is perfectly reproducible, and the predictive performance of both approaches regarding immunotherapy response is limited. In this study, we developed a deep learning (DL) method to predict PD-L1 status directly from raw IHC image data, without explicit intermediary steps such as cell detection or pigment quantification. We trained the weakly supervised model on PD-L1-stained slides from the non-small cell lung cancer (NSCLC)-Memorial Sloan Kettering (MSK) cohort (N = 233) and validated it on the pan-cancer-Vall d'Hebron Institute of Oncology (VHIO) cohort (N = 108). We also investigated the performance of the model to predict response to immune checkpoint inhibitors (ICI) in terms of progression-free survival. In the pan-cancer-VHIO cohort, the performance was compared with tumor proportion score (TPS) and combined positive score (CPS). The DL model showed good performance in predicting PD-L1 expression (TPS ≥ 1%) in both NSCLC-MSK and pan-cancer-VHIO cohort (AUC 0.88 ± 0.06 and 0.80 ± 0.03, respectively). The predicted PD-L1 status showed an improved association with response to ICIs [HR: 1.5 (95% confidence interval: 1-2.3), P = 0.049] compared with TPS [HR: 1.4 (0.96-2.2), P = 0.082] and CPS [HR: 1.2 (0.79-1.9), P = 0.386]. Notably, our explainability analysis showed that the model does not just look at the amount of brown pigment in the IHC slides, but also considers morphologic factors such as lymphocyte conglomerates. Overall, end-to-end weakly supervised DL shows potential for improving patient stratification for cancer immunotherapy by analyzing PD-L1 IHC, holistically integrating morphology and PD-L1 staining intensity. SIGNIFICANCE The weakly supervised DL model to predict PD-L1 status from raw IHC data, integrating tumor staining intensity and morphology, enables enhanced patient stratification in cancer immunotherapy compared with traditional pathologist assessment.
Collapse
Affiliation(s)
- Marta Ligero
- Radiomics Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Garazi Serna
- Molecular Oncology Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Omar S.M. El Nahhas
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Irene Sansano
- Pathology Department, Vall d'Hebron University Hospital (VHUH), Barcelona, Spain
| | - Siarhei Mauchanski
- Molecular Oncology Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Cristina Viaplana
- Oncology Data Science (ODysSey) Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Julien Calderaro
- Assistance Publique-Hôpitaux de Paris, Département de Pathologie, CHU Henri Mondor, Créteil, France
- Université Paris-Est Créteil, Faculté de Médecine, Créteil, France
| | - Rodrigo A. Toledo
- Biomakers and Clonal Dynamics Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Rodrigo Dienstmann
- Oncology Data Science (ODysSey) Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Rami S. Vanguri
- Department of Pathology and Laboratory Medicine, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
| | - Jennifer L. Sauter
- Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York
| | | | - Sohrab P. Shah
- Computational Oncology, Memorial Sloan Kettering Cancer Center, New York, New York
| | | | - Elena Garralda
- Department of Medical Oncology, Vall d'Hebron University Hospital and Institute of Oncology (VHIO), Barcelona, Spain
| | - Paolo Nuciforo
- Molecular Oncology Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Raquel Perez-Lopez
- Radiomics Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
- Department of Medicine I, University Hospital Dresden, Dresden, Germany
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| |
Collapse
|
23
|
Jia Y, Liu J, Chen L, Zhao T, Wang Y. THItoGene: a deep learning method for predicting spatial transcriptomics from histological images. Brief Bioinform 2023; 25:bbad464. [PMID: 38145948 PMCID: PMC10749789 DOI: 10.1093/bib/bbad464] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 11/10/2023] [Accepted: 11/18/2023] [Indexed: 12/27/2023] Open
Abstract
Spatial transcriptomics unveils the complex dynamics of cell regulation and transcriptomes, but it is typically cost-prohibitive. Predicting spatial gene expression from histological images via artificial intelligence offers a more affordable option, yet existing methods fall short in extracting deep-level information from pathological images. In this paper, we present THItoGene, a hybrid neural network that utilizes dynamic convolutional and capsule networks to adaptively sense potential molecular signals in histological images for exploring the relationship between high-resolution pathology image phenotypes and regulation of gene expression. A comprehensive benchmark evaluation using datasets from human breast cancer and cutaneous squamous cell carcinoma has demonstrated the superior performance of THItoGene in spatial gene expression prediction. Moreover, THItoGene has demonstrated its capacity to decipher both the spatial context and enrichment signals within specific tissue regions. THItoGene can be freely accessed at https://github.com/yrjia1015/THItoGene.
Collapse
Affiliation(s)
- Yuran Jia
- Institute for Bioinformatics, School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150040, China
| | - Junliang Liu
- Institute for Bioinformatics, School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150040, China
| | - Li Chen
- School of Life Sciences, Westlake University, Hangzhou, Zhejiang 310024, China
| | - Tianyi Zhao
- School of Medicine and Health, Harbin Institute of Technology, Harbin, 150040, China
| | - Yadong Wang
- School of Medicine and Health, Harbin Institute of Technology, Harbin, 150040, China
| |
Collapse
|
24
|
Jang HJ, Go JH, Kim Y, Lee SH. Deep Learning for the Pathologic Diagnosis of Hepatocellular Carcinoma, Cholangiocarcinoma, and Metastatic Colorectal Cancer. Cancers (Basel) 2023; 15:5389. [PMID: 38001649 PMCID: PMC10670046 DOI: 10.3390/cancers15225389] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Revised: 11/01/2023] [Accepted: 11/09/2023] [Indexed: 11/26/2023] Open
Abstract
Diagnosing primary liver cancers, particularly hepatocellular carcinoma (HCC) and cholangiocarcinoma (CC), is a challenging and labor-intensive process, even for experts, and secondary liver cancers further complicate the diagnosis. Artificial intelligence (AI) offers promising solutions to these diagnostic challenges by facilitating the histopathological classification of tumors using digital whole slide images (WSIs). This study aimed to develop a deep learning model for distinguishing HCC, CC, and metastatic colorectal cancer (mCRC) using histopathological images and to discuss its clinical implications. The WSIs from HCC, CC, and mCRC were used to train the classifiers. For normal/tumor classification, the areas under the curve (AUCs) were 0.989, 0.988, and 0.991 for HCC, CC, and mCRC, respectively. Using proper tumor tissues, the HCC/other cancer type classifier was trained to effectively distinguish HCC from CC and mCRC, with a concatenated AUC of 0.998. Subsequently, the CC/mCRC classifier differentiated CC from mCRC with a concatenated AUC of 0.995. However, testing on an external dataset revealed that the HCC/other cancer type classifier underperformed with an AUC of 0.745. After combining the original training datasets with external datasets and retraining, the classification drastically improved, all achieving AUCs of 1.000. Although these results are promising and offer crucial insights into liver cancer, further research is required for model refinement and validation.
Collapse
Affiliation(s)
- Hyun-Jong Jang
- Department of Physiology, CMC Institute for Basic Medical Science, College of Medicine, The Catholic University of Korea, Seoul 06591, Republic of Korea;
| | - Jai-Hyang Go
- Department of Pathology, Dankook University College of Medicine, Cheonan 31116, Republic of Korea;
| | - Younghoon Kim
- Department of Hospital Pathology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul 06591, Republic of Korea;
| | - Sung Hak Lee
- Department of Hospital Pathology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul 06591, Republic of Korea;
| |
Collapse
|