1
|
Hosseini MS, Bejnordi BE, Trinh VQH, Chan L, Hasan D, Li X, Yang S, Kim T, Zhang H, Wu T, Chinniah K, Maghsoudlou S, Zhang R, Zhu J, Khaki S, Buin A, Chaji F, Salehi A, Nguyen BN, Samaras D, Plataniotis KN. Computational pathology: A survey review and the way forward. J Pathol Inform 2024; 15:100357. [PMID: 38420608 PMCID: PMC10900832 DOI: 10.1016/j.jpi.2023.100357] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 12/21/2023] [Accepted: 12/23/2023] [Indexed: 03/02/2024] Open
Abstract
Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.
Collapse
Affiliation(s)
- Mahdi S Hosseini
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | | | - Vincent Quoc-Huy Trinh
- Institute for Research in Immunology and Cancer of the University of Montreal, Montreal, QC H3T 1J4, Canada
| | - Lyndon Chan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Danial Hasan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Xingwen Li
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Stephen Yang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Taehyo Kim
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Haochen Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Theodore Wu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Kajanan Chinniah
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Sina Maghsoudlou
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ryan Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Jiadai Zhu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Samir Khaki
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Andrei Buin
- Huron Digitial Pathology, St. Jacobs, ON N0B 2N0, Canada
| | - Fatemeh Chaji
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ala Salehi
- Department of Electrical and Computer Engineering, University of New Brunswick, Fredericton, NB E3B 5A3, Canada
| | - Bich Ngoc Nguyen
- University of Montreal Hospital Center, Montreal, QC H2X 0C2, Canada
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, NY 11794, United States
| | - Konstantinos N Plataniotis
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| |
Collapse
|
2
|
Gonzalez R, Saha A, Campbell CJ, Nejat P, Lokker C, Norgan AP. Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities. J Pathol Inform 2024; 15:100347. [PMID: 38162950 PMCID: PMC10755052 DOI: 10.1016/j.jpi.2023.100347] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2023] [Revised: 10/06/2023] [Accepted: 11/01/2023] [Indexed: 01/03/2024] Open
Abstract
This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.
Collapse
Affiliation(s)
- Ricardo Gonzalez
- DeGroote School of Business, McMaster University, Hamilton, Ontario, Canada
- Division of Computational Pathology and Artificial Intelligence, Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, United States
| | - Ashirbani Saha
- Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada
- Escarpment Cancer Research Institute, McMaster University and Hamilton Health Sciences, Hamilton, Ontario, Canada
| | - Clinton J.V. Campbell
- William Osler Health System, Brampton, Ontario, Canada
- Department of Pathology and Molecular Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada
| | - Peyman Nejat
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, United States
| | - Cynthia Lokker
- Health Information Research Unit, Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
| | - Andrew P. Norgan
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, United States
| |
Collapse
|
3
|
Zhang W, Wang W, Xu Y, Wu K, Shi J, Li M, Feng Z, Liu Y, Zheng Y, Wu H. Prediction of epidermal growth factor receptor mutation subtypes in non-small cell lung cancer from hematoxylin and eosin-stained slides using deep learning. J Transl Med 2024:102094. [PMID: 38871058 DOI: 10.1016/j.labinv.2024.102094] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2023] [Revised: 04/28/2024] [Accepted: 06/04/2024] [Indexed: 06/15/2024] Open
Abstract
Accurate assessment of epidermal growth factor receptor (EGFR) mutation status and subtype are critical for the treatment of non-small cell lung cancer (NSCLC) patients. Conventional molecular testing methods for detecting EGFR mutations have limitations. In this study, an artificial intelligence-powered deep learning framework was developed for weakly supervised prediction of EGFR mutations in NSCLC from hematoxylin and eosin (H&E)-stained histopathology whole-slide images (WSIs). The study cohort was partitioned into training and validation subsets. Foreground regions containing tumor tissue were extracted from WSIs. A convolutional neural network (CNN) employing a contrastive learning paradigm was implemented to extract patch-level morphological features. These features were aggregated using a vision-transformer-based model to predict EGFR mutation status and classify patient cases. The established prediction model was validated on unseen datasets. In internal validation with a cohort from (USTC)(n=172), the model achieved patient-level areas under the receiver operating characteristic (ROC) curve (AUCs) of 0.927 and 0.907, sensitivities of 81.6% and 93.0%, and specificities of 83.3% and 92.3%, for surgical resection and biopsy specimens in EGFR mutation subtype prediction, respectively. External validation with cohorts from the Second Affiliated Hospital of Anhui Medical University (AMU) and the First Affiliated Hospital of Wannan Medical College (WMC) (n=193) yielded patient-level AUCs of 0.849 and 0.871, sensitivities of 75.7% and 72.1%, and specificities of 90.5% and 90.3% for surgical and biopsy specimens, respectively. Further validation with The Cancer Genome Atlas (TCGA) dataset (n=81) showed an AUC of 0.861, sensitivity of 84.6%, and specificity of 90.5%. Deep learning solutions demonstrate potential advantages for automated, non-invasive, fast, cost-effective, and accurate inference of EGFR alterations from histomorphology. Integration of such artificial intelligence frameworks into routine digital pathology workflows could augment existing molecular testing pipelines.
Collapse
Affiliation(s)
- Wanqiu Zhang
- Department of Pathology, the First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui 230036, China; Intelligent Pathology Institute, the First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui 230036, China
| | - Wei Wang
- Department of Pathology, the First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui 230036, China; Intelligent Pathology Institute, the First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui 230036, China
| | - Yao Xu
- Department of pathology, Wannan Medical College First Affiliated Hospital, Yijishan Hospital, Wuhu 241001, China
| | - Kun Wu
- School of Engineering Medicine, Beijing Advanced Innovation Center on Biomedical Engineering, Beihang University, Beijing 100191, China
| | - Jun Shi
- School of Software, Hefei University of Technology, Hefei 230601, China
| | - Ming Li
- Department of Pathology, the Second Affiliated Hospital of Anhui Medical University, Hefei, Anhui 230601, China
| | - Zhengzhong Feng
- Department of Pathology, the Second Affiliated Hospital of Anhui Medical University, Hefei, Anhui 230601, China.
| | - Yinhua Liu
- Department of pathology, Wannan Medical College First Affiliated Hospital, Yijishan Hospital, Wuhu 241001, China.
| | - Yushan Zheng
- School of Engineering Medicine, Beijing Advanced Innovation Center on Biomedical Engineering, Beihang University, Beijing 100191, China.
| | - Haibo Wu
- Department of Pathology, the First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui 230036, China; Intelligent Pathology Institute, the First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui 230036, China.
| |
Collapse
|
4
|
Katayama A, Aoki Y, Watanabe Y, Horiguchi J, Rakha EA, Oyama T. Current status and prospects of artificial intelligence in breast cancer pathology: convolutional neural networks to prospective Vision Transformers. Int J Clin Oncol 2024:10.1007/s10147-024-02513-3. [PMID: 38619651 DOI: 10.1007/s10147-024-02513-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2024] [Accepted: 03/12/2024] [Indexed: 04/16/2024]
Abstract
Breast cancer is the most prevalent cancer among women, and its diagnosis requires the accurate identification and classification of histological features for effective patient management. Artificial intelligence, particularly through deep learning, represents the next frontier in cancer diagnosis and management. Notably, the use of convolutional neural networks and emerging Vision Transformers (ViT) has been reported to automate pathologists' tasks, including tumor detection and classification, in addition to improving the efficiency of pathology services. Deep learning applications have also been extended to the prediction of protein expression, molecular subtype, mutation status, therapeutic efficacy, and outcome prediction directly from hematoxylin and eosin-stained slides, bypassing the need for immunohistochemistry or genetic testing. This review explores the current status and prospects of deep learning in breast cancer diagnosis with a focus on whole-slide image analysis. Artificial intelligence applications are increasingly applied to many tasks in breast pathology ranging from disease diagnosis to outcome prediction, thus serving as valuable tools for assisting pathologists and supporting breast cancer management.
Collapse
Affiliation(s)
- Ayaka Katayama
- Diagnostic Pathology, Gunma University Graduate School of Medicine, 3-39-22 Showamachi, Maebashi, Gunma, 371-8511, Japan.
| | - Yuki Aoki
- Center for Mathematics and Data Science, Gunma University, Maebashi, Japan
| | - Yukako Watanabe
- Clinical Training Center, Gunma University Hospital, Maebashi, Japan
| | - Jun Horiguchi
- Department of Breast Surgery, International University of Health and Welfare, Narita, Japan
| | - Emad A Rakha
- Department of Histopathology School of Medicine, University of Nottingham, University Park, Nottingham, UK
- Department of Pathology, Hamad Medical Corporation, Doha, Qatar
| | - Tetsunari Oyama
- Diagnostic Pathology, Gunma University Graduate School of Medicine, 3-39-22 Showamachi, Maebashi, Gunma, 371-8511, Japan
| |
Collapse
|
5
|
Godson L, Alemi N, Nsengimana J, Cook GP, Clarke EL, Treanor D, Bishop DT, Newton-Bishop J, Gooya A, Magee D. Immune subtyping of melanoma whole slide images using multiple instance learning. Med Image Anal 2024; 93:103097. [PMID: 38325154 DOI: 10.1016/j.media.2024.103097] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Revised: 01/15/2024] [Accepted: 01/25/2024] [Indexed: 02/09/2024]
Abstract
Determining early-stage prognostic markers and stratifying patients for effective treatment are two key challenges for improving outcomes for melanoma patients. Previous studies have used tumour transcriptome data to stratify patients into immune subgroups, which were associated with differential melanoma specific survival and potential predictive biomarkers. However, acquiring transcriptome data is a time-consuming and costly process. Moreover, it is not routinely used in the current clinical workflow. Here, we attempt to overcome this by developing deep learning models to classify gigapixel haematoxylin and eosin (H&E) stained pathology slides, which are well established in clinical workflows, into these immune subgroups. We systematically assess six different multiple instance learning (MIL) frameworks, using five different image resolutions and three different feature extraction methods. We show that pathology-specific self-supervised models using 10x resolution patches generate superior representations for the classification of immune subtypes. In addition, in a primary melanoma dataset, we achieve a mean area under the receiver operating characteristic curve (AUC) of 0.80 for classifying histopathology images into 'high' or 'low immune' subgroups and a mean AUC of 0.82 in an independent TCGA melanoma dataset. Furthermore, we show that these models are able to stratify patients into 'high' and 'low immune' subgroups with significantly different melanoma specific survival outcomes (log rank test, P< 0.005). We anticipate that MIL methods will allow us to find new biomarkers of high importance, act as a tool for clinicians to infer the immune landscape of tumours and stratify patients, without needing to carry out additional expensive genetic tests.
Collapse
Affiliation(s)
- Lucy Godson
- School of Computing, University of Leeds, Woodhouse, Leeds, LS2 9JT, United Kingdom.
| | - Navid Alemi
- School of Computing, University of Leeds, Woodhouse, Leeds, LS2 9JT, United Kingdom
| | - Jérémie Nsengimana
- Population Health Sciences Institute, Newcastle University, Newcastle upon Tyne, NE1 7RU, United Kingdom
| | - Graham P Cook
- Leeds Institute of Medical Research, University of Leeds School of Medicine, St. James's University Hospital, Leeds, United Kingdom
| | - Emily L Clarke
- Department of Histopathology, Leeds Teaching Hospitals NHS Trust, Leeds, United Kingdom; Division of Pathology and Data Analytics, Leeds Institute of Cancer and Pathology, University of Leeds, Beckett Street, Leeds, LS9 7TF, United Kingdom
| | - Darren Treanor
- Department of Histopathology, Leeds Teaching Hospitals NHS Trust, Leeds, United Kingdom; Division of Pathology and Data Analytics, Leeds Institute of Cancer and Pathology, University of Leeds, Beckett Street, Leeds, LS9 7TF, United Kingdom; Department of Clinical Pathology and Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden; Centre for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden
| | - D Timothy Bishop
- Leeds Institute of Medical Research, University of Leeds School of Medicine, St. James's University Hospital, Leeds, United Kingdom
| | - Julia Newton-Bishop
- Division of Pathology and Data Analytics, Leeds Institute of Cancer and Pathology, University of Leeds, Beckett Street, Leeds, LS9 7TF, United Kingdom
| | - Ali Gooya
- School of Computing, University of Glasgow, Glasgow, G12 8QQ, United Kingdom
| | - Derek Magee
- School of Computing, University of Leeds, Woodhouse, Leeds, LS2 9JT, United Kingdom
| |
Collapse
|
6
|
Arslan S, Schmidt J, Bass C, Mehrotra D, Geraldes A, Singhal S, Hense J, Li X, Raharja-Liu P, Maiques O, Kather JN, Pandya P. A systematic pan-cancer study on deep learning-based prediction of multi-omic biomarkers from routine pathology images. COMMUNICATIONS MEDICINE 2024; 4:48. [PMID: 38491101 PMCID: PMC10942985 DOI: 10.1038/s43856-024-00471-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Accepted: 02/29/2024] [Indexed: 03/18/2024] Open
Abstract
BACKGROUND The objective of this comprehensive pan-cancer study is to evaluate the potential of deep learning (DL) for molecular profiling of multi-omic biomarkers directly from hematoxylin and eosin (H&E)-stained whole slide images. METHODS A total of 12,093 DL models predicting 4031 multi-omic biomarkers across 32 cancer types were trained and validated. The study included a broad range of genetic, transcriptomic, and proteomic biomarkers, as well as established prognostic markers, molecular subtypes, and clinical outcomes. RESULTS Here we show that 50% of the models achieve an area under the curve (AUC) of 0.644 or higher. The observed AUC for 25% of the models is at least 0.719 and exceeds 0.834 for the top 5%. Molecular profiling with image-based histomorphological features is generally considered feasible for most of the investigated biomarkers and across different cancer types. The performance appears to be independent of tumor purity, sample size, and class ratio (prevalence), suggesting a degree of inherent predictability in histomorphology. CONCLUSIONS The results demonstrate that DL holds promise to predict a wide range of biomarkers across the omics spectrum using only H&E-stained histological slides of solid tumors. This paves the way for accelerating diagnosis and developing more precise treatments for cancer patients.
Collapse
Affiliation(s)
| | | | | | - Debapriya Mehrotra
- Panakeia Technologies, London, UK
- Department of Pathology, Barking, Havering and Redbridge University NHS Trust, Romford, UK
| | | | - Shikha Singhal
- Panakeia Technologies, London, UK
- Department of Pathology, The Royal Wolverhampton NHS Trust, Wolverhampton, UK
| | | | - Xiusi Li
- Panakeia Technologies, London, UK
| | | | - Oscar Maiques
- Cytoskeleton and Cancer Metastasis Group, Breast Cancer Now Toby Robins Breast Cancer Research Centre, The Institute of Cancer Research, London, UK
- Cancer Biomarkers & Biotherapeutics, Barts Cancer Institute, Queen Mary University of London, John Vane Science Building, London, UK
| | - Jakob Nikolas Kather
- Medical Oncology, National Center for Tumor Diseases, University Hospital Heidelberg, Heidelberg, Germany
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | | |
Collapse
|
7
|
Weber A, Enderle-Ammour K, Kurowski K, Metzger MC, Poxleitner P, Werner M, Rothweiler R, Beck J, Straehle J, Schmelzeisen R, Steybe D, Bronsert P. AI-Based Detection of Oral Squamous Cell Carcinoma with Raman Histology. Cancers (Basel) 2024; 16:689. [PMID: 38398080 PMCID: PMC10886627 DOI: 10.3390/cancers16040689] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2024] [Revised: 02/02/2024] [Accepted: 02/02/2024] [Indexed: 02/25/2024] Open
Abstract
Stimulated Raman Histology (SRH) employs the stimulated Raman scattering (SRS) of photons at biomolecules in tissue samples to generate histological images. Subsequent pathological analysis allows for an intraoperative evaluation without the need for sectioning and staining. The objective of this study was to investigate a deep learning-based classification of oral squamous cell carcinoma (OSCC) and the sub-classification of non-malignant tissue types, as well as to compare the performances of the classifier between SRS and SRH images. Raman shifts were measured at wavenumbers k1 = 2845 cm-1 and k2 = 2930 cm-1. SRS images were transformed into SRH images resembling traditional H&E-stained frozen sections. The annotation of 6 tissue types was performed on images obtained from 80 tissue samples from eight OSCC patients. A VGG19-based convolutional neural network was then trained on 64 SRS images (and corresponding SRH images) and tested on 16. A balanced accuracy of 0.90 (0.87 for SRH images) and F1-scores of 0.91 (0.91 for SRH) for stroma, 0.98 (0.96 for SRH) for adipose tissue, 0.90 (0.87 for SRH) for squamous epithelium, 0.92 (0.76 for SRH) for muscle, 0.87 (0.90 for SRH) for glandular tissue, and 0.88 (0.87 for SRH) for tumor were achieved. The results of this study demonstrate the suitability of deep learning for the intraoperative identification of tissue types directly on SRS and SRH images.
Collapse
Affiliation(s)
- Andreas Weber
- Institute for Surgical Pathology, Medical Center, University of Freiburg, 79106 Freiburg, Germany
- Faculty of Biology, University of Freiburg, 79104 Freiburg, Germany
| | - Kathrin Enderle-Ammour
- Institute for Surgical Pathology, Medical Center, University of Freiburg, 79106 Freiburg, Germany
| | - Konrad Kurowski
- Institute for Surgical Pathology, Medical Center, University of Freiburg, 79106 Freiburg, Germany
- Tumorbank Comprehensive Cancer Center Freiburg, Medical Center, University of Freiburg, 79106 Freiburg, Germany
- Core Facility for Histopathology and Digital Pathology, Medical Center, University of Freiburg, 79106 Freiburg, Germany
| | - Marc C. Metzger
- Department of Oral and Maxillofacial Surgery, Medical Center, University of Freiburg, 79106 Freiburg, Germany
| | - Philipp Poxleitner
- Department of Oral and Maxillofacial Surgery, Medical Center, University of Freiburg, 79106 Freiburg, Germany
- Center for Advanced Surgical Tissue Analysis (CAST), University of Freiburg, 79106 Freiburg, Germany
- Department of Oral and Maxillofacial Surgery and Facial Plastic Surgery, University Hospital, LMU Munich, 80337 Munich, Germany
| | - Martin Werner
- Institute for Surgical Pathology, Medical Center, University of Freiburg, 79106 Freiburg, Germany
- Tumorbank Comprehensive Cancer Center Freiburg, Medical Center, University of Freiburg, 79106 Freiburg, Germany
| | - René Rothweiler
- Department of Oral and Maxillofacial Surgery, Medical Center, University of Freiburg, 79106 Freiburg, Germany
| | - Jürgen Beck
- Center for Advanced Surgical Tissue Analysis (CAST), University of Freiburg, 79106 Freiburg, Germany
- Department of Neurosurgery, Medical Center, University of Freiburg, 79106 Freiburg, Germany
| | - Jakob Straehle
- Center for Advanced Surgical Tissue Analysis (CAST), University of Freiburg, 79106 Freiburg, Germany
- Department of Neurosurgery, Medical Center, University of Freiburg, 79106 Freiburg, Germany
| | - Rainer Schmelzeisen
- Department of Oral and Maxillofacial Surgery, Medical Center, University of Freiburg, 79106 Freiburg, Germany
- Center for Advanced Surgical Tissue Analysis (CAST), University of Freiburg, 79106 Freiburg, Germany
| | - David Steybe
- Department of Oral and Maxillofacial Surgery, Medical Center, University of Freiburg, 79106 Freiburg, Germany
- Center for Advanced Surgical Tissue Analysis (CAST), University of Freiburg, 79106 Freiburg, Germany
- Department of Oral and Maxillofacial Surgery and Facial Plastic Surgery, University Hospital, LMU Munich, 80337 Munich, Germany
| | - Peter Bronsert
- Institute for Surgical Pathology, Medical Center, University of Freiburg, 79106 Freiburg, Germany
- Tumorbank Comprehensive Cancer Center Freiburg, Medical Center, University of Freiburg, 79106 Freiburg, Germany
- Core Facility for Histopathology and Digital Pathology, Medical Center, University of Freiburg, 79106 Freiburg, Germany
| |
Collapse
|
8
|
Drachneris J, Morkunas M, Fabijonavicius M, Cekauskas A, Jankevicius F, Laurinavicius A. Prediction of Non-Muscle Invasive Papillary Urothelial Carcinoma Relapse from Hematoxylin-Eosin Images Using Deep Multiple Instance Learning in Patients Treated with Bacille Calmette-Guérin Immunotherapy. Biomedicines 2024; 12:360. [PMID: 38397962 PMCID: PMC10886666 DOI: 10.3390/biomedicines12020360] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2023] [Revised: 01/29/2024] [Accepted: 02/01/2024] [Indexed: 02/25/2024] Open
Abstract
The limited reproducibility of the grading of non-muscle invasive papillary urothelial carcinoma (NMIPUC) necessitates the search for more robust image-based predictive factors. In a cohort of 157 NMIPUC patients treated with Bacille Calmette-Guérin (BCG) immunotherapy, we explored the multiple instance learning (MIL)-based classification approach for the prediction of 2-year and 5-year relapse-free survival and the multiple instance survival learning (MISL) framework for survival regression. We used features extracted from image patches sampled from whole slide images of hematoxylin-eosin-stained transurethral resection (TUR) NPMIPUC specimens and tested several patch sampling and feature extraction network variations to optimize the model performance. We selected the model showing the best patient survival stratification for further testing in the context of clinical and pathological variables. MISL with the multiresolution patch sampling technique achieved the best patient risk stratification (concordance index = 0.574, p = 0.010), followed by a 2-year MIL classification. The best-selected model revealed an independent prognostic value in the context of other clinical and pathologic variables (tumor stage, grade, and presence of tumor on the repeated TUR) with statistically significant patient risk stratification. Our findings suggest that MISL-based predictions can improve NMIPUC patient risk stratification, while validation studies are needed to test the generalizability of our models.
Collapse
Affiliation(s)
- Julius Drachneris
- Department of Pathology and Forensic Medicine, Institute of Biomedical Sciences, Faculty of Medicine, Vilnius University, 03101 Vilnius, Lithuania;
- National Center of Pathology, Affiliate of Vilnius University Hospital Santaros Klinikos, 08406 Vilnius, Lithuania
| | - Mindaugas Morkunas
- Clinic of Gastroenterology, Nephrourology and Surgery, Institute of Clinical Medicine, Faculty of Medicine, Vilnius University, 08406 Vilnius, Lithuania; (M.M.); (A.C.); (F.J.)
| | - Mantas Fabijonavicius
- Center of Urology, Vilnius University Hospital Santaros Klinikos, 08406 Vilnius, Lithuania;
| | - Albertas Cekauskas
- Clinic of Gastroenterology, Nephrourology and Surgery, Institute of Clinical Medicine, Faculty of Medicine, Vilnius University, 08406 Vilnius, Lithuania; (M.M.); (A.C.); (F.J.)
- Center of Urology, Vilnius University Hospital Santaros Klinikos, 08406 Vilnius, Lithuania;
| | - Feliksas Jankevicius
- Clinic of Gastroenterology, Nephrourology and Surgery, Institute of Clinical Medicine, Faculty of Medicine, Vilnius University, 08406 Vilnius, Lithuania; (M.M.); (A.C.); (F.J.)
- Center of Urology, Vilnius University Hospital Santaros Klinikos, 08406 Vilnius, Lithuania;
| | - Arvydas Laurinavicius
- Department of Pathology and Forensic Medicine, Institute of Biomedical Sciences, Faculty of Medicine, Vilnius University, 03101 Vilnius, Lithuania;
- National Center of Pathology, Affiliate of Vilnius University Hospital Santaros Klinikos, 08406 Vilnius, Lithuania
| |
Collapse
|
9
|
Amgad M, Hodge JM, Elsebaie MAT, Bodelon C, Puvanesarajah S, Gutman DA, Siziopikou KP, Goldstein JA, Gaudet MM, Teras LR, Cooper LAD. A population-level digital histologic biomarker for enhanced prognosis of invasive breast cancer. Nat Med 2024; 30:85-97. [PMID: 38012314 DOI: 10.1038/s41591-023-02643-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2023] [Accepted: 10/13/2023] [Indexed: 11/29/2023]
Abstract
Breast cancer is a heterogeneous disease with variable survival outcomes. Pathologists grade the microscopic appearance of breast tissue using the Nottingham criteria, which are qualitative and do not account for noncancerous elements within the tumor microenvironment. Here we present the Histomic Prognostic Signature (HiPS), a comprehensive, interpretable scoring of the survival risk incurred by breast tumor microenvironment morphology. HiPS uses deep learning to accurately map cellular and tissue structures to measure epithelial, stromal, immune, and spatial interaction features. It was developed using a population-level cohort from the Cancer Prevention Study-II and validated using data from three independent cohorts, including the Prostate, Lung, Colorectal, and Ovarian Cancer trial, Cancer Prevention Study-3, and The Cancer Genome Atlas. HiPS consistently outperformed pathologists in predicting survival outcomes, independent of tumor-node-metastasis stage and pertinent variables. This was largely driven by stromal and immune features. In conclusion, HiPS is a robustly validated biomarker to support pathologists and improve patient prognosis.
Collapse
Affiliation(s)
- Mohamed Amgad
- Department of Pathology, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - James M Hodge
- Department of Population Science, American Cancer Society, Atlanta, GA, USA
| | - Maha A T Elsebaie
- Department of Medicine, John H. Stroger, Jr. Hospital of Cook County, Chicago, IL, USA
| | - Clara Bodelon
- Department of Population Science, American Cancer Society, Atlanta, GA, USA
| | | | - David A Gutman
- Department of Pathology, Emory University School of Medicine, Atlanta, GA, USA
| | - Kalliopi P Siziopikou
- Department of Pathology, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Jeffery A Goldstein
- Department of Pathology, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Mia M Gaudet
- Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, MD, USA
| | - Lauren R Teras
- Department of Population Science, American Cancer Society, Atlanta, GA, USA
| | - Lee A D Cooper
- Department of Pathology, Northwestern University Feinberg School of Medicine, Chicago, IL, USA.
| |
Collapse
|
10
|
Ng CW, Wong KK. Deep learning-enabled breast cancer endocrine response determination from H&E staining based on ESR1 signaling activity. Sci Rep 2023; 13:21454. [PMID: 38052873 PMCID: PMC10698147 DOI: 10.1038/s41598-023-48830-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Accepted: 11/30/2023] [Indexed: 12/07/2023] Open
Abstract
Estrogen receptor (ER) positivity by immunohistochemistry has long been a main selection criterium for breast cancer patients to be treated with endocrine therapy. However, ER positivity might not directly correlate with activated ER signaling activity, which is a better predictor for endocrine therapy responsiveness. In this study, we investigated if a deep learning method using whole-slide H&E-stained images could predict ER signaling activity. First, ER signaling activity score was determined using RNAseq data available from each of the 1082 breast cancer samples in the TCGA Pan-Cancer dataset based on the Hallmark Estrogen Response Early gene set from the Molecular Signature Database (MSigDB). Then the processed H&E-stained images and ER signaling activity scores from a training cohort were fed into ResNet101 with three additional fully connected layers to generate a predicted ER activity score. The trained models were subsequently applied to an independent testing cohort. The result demonstrated that ER + /HER2- breast cancer patients with a higher predicted ER activity score had longer progression-free survival (p = 0.0368) than those with lower predicted ER activity score. In conclusion, a convolutional deep neural network can predict prognosis and endocrine therapy response in breast cancer patients based on whole-slide H&E-stained images. The trained models were found to robustly predict the prognosis of ER + /HER2- patients. This information is valuable for patient management, as it does not require RNA-seq or microarray data analyses. Thus, these models can reduce the cost of the diagnosis workflow if such information is required.
Collapse
Affiliation(s)
- Chun Wai Ng
- Department of Gynecologic Oncology and Reproductive Medicine, Unit 1362, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd, Houston, TX, 77030, USA
| | - Kwong-Kwok Wong
- Department of Gynecologic Oncology and Reproductive Medicine, Unit 1362, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd, Houston, TX, 77030, USA.
| |
Collapse
|
11
|
Hatta S, Ichiuji Y, Mabu S, Kugler M, Hontani H, Okoshi T, Fuse H, Kawada T, Kido S, Imamura Y, Naiki H, Inai K. Improved artificial intelligence discrimination of minor histological populations by supplementing with color-adjusted images. Sci Rep 2023; 13:19068. [PMID: 37925580 PMCID: PMC10625567 DOI: 10.1038/s41598-023-46472-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2023] [Accepted: 11/01/2023] [Indexed: 11/06/2023] Open
Abstract
Despite the dedicated research of artificial intelligence (AI) for pathological images, the construction of AI applicable to histopathological tissue subtypes, is limited by insufficient dataset collection owing to disease infrequency. Here, we present a solution involving the addition of supplemental tissue array (TA) images that are adjusted to the tonality of the main data using a cycle-consistent generative adversarial network (CycleGAN) to the training data for rare tissue types. F1 scores of rare tissue types that constitute < 1.2% of the training data were significantly increased by improving recall values after adding color-adjusted TA images constituting < 0.65% of total training patches. The detector also enabled the equivalent discrimination of clinical images from two distinct hospitals and the capability was more increased following color-correction of test data before AI identification (F1 score from 45.2 ± 27.1 to 77.1 ± 10.3, p < 0.01). These methods also classified intraoperative frozen sections, while excessive supplementation paradoxically decreased F1 scores. These results identify strategies for building an AI that preserves the imbalance between training data with large differences in actual disease frequencies, which is important for constructing AI for practical histopathological classification.
Collapse
Affiliation(s)
- Satomi Hatta
- Division of Molecular Pathology, Department of Pathological Sciences, University of Fukui, 23-3 Matsuoka-Shimoaizuki, Eiheiji, Fukui, 910-1193, Japan
- Division of Diagnostic/Surgical Pathology, University of Fukui Hospital, Eiheiji, Japan
| | - Yoshihito Ichiuji
- Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Yamaguchi, Japan
| | - Shingo Mabu
- Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Yamaguchi, Japan
| | - Mauricio Kugler
- Department of Computer Science, Nagoya Institute of Technology, Nagoya, Japan
| | - Hidekata Hontani
- Department of Computer Science, Nagoya Institute of Technology, Nagoya, Japan
| | - Tadakazu Okoshi
- Department of Pathology, Fukui Red Cross Hospital, Fukui, Japan
| | - Haruki Fuse
- Department of Clinical Inspection, Maizuru Kyosai Hospital, Maizuru, Japan
| | - Takako Kawada
- Department of Clinical Inspection, Maizuru Kyosai Hospital, Maizuru, Japan
| | - Shoji Kido
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, Suita, Japan
| | - Yoshiaki Imamura
- Division of Diagnostic/Surgical Pathology, University of Fukui Hospital, Eiheiji, Japan
| | - Hironobu Naiki
- Division of Molecular Pathology, Department of Pathological Sciences, University of Fukui, 23-3 Matsuoka-Shimoaizuki, Eiheiji, Fukui, 910-1193, Japan
| | - Kunihiro Inai
- Division of Molecular Pathology, Department of Pathological Sciences, University of Fukui, 23-3 Matsuoka-Shimoaizuki, Eiheiji, Fukui, 910-1193, Japan.
| |
Collapse
|
12
|
Yan S, Li J, Wu W. Artificial intelligence in breast cancer: application and future perspectives. J Cancer Res Clin Oncol 2023; 149:16179-16190. [PMID: 37656245 DOI: 10.1007/s00432-023-05337-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Accepted: 08/24/2023] [Indexed: 09/02/2023]
Abstract
Breast cancer is one of the most common cancers and is one of the leading causes of cancer-related deaths in women worldwide. Early diagnosis and treatment are the key for a favorable prognosis. The application of artificial intelligence technology in the medical field is increasingly extensive, including image analysis, automated diagnosis, intelligent pharmaceutical system, personalized treatment and so on. AI-based breast cancer imaging, pathology and adjuvant therapy technology cannot only reduce the workload of clinicians, but also continuously improve the accuracy and sensitivity of breast cancer diagnosis and treatment. This paper reviews the application of AI in breast cancer, as well as looks ahead and poses challenges to the future development of AI for breast cancer detection and therapeutic, so as to provide ideas for future research.
Collapse
Affiliation(s)
- Shuixin Yan
- The Affiliated Lihuili Hospital of Ningbo University, Ningbo, 315000, Zhejiang, China
| | - Jiadi Li
- The Affiliated Lihuili Hospital of Ningbo University, Ningbo, 315000, Zhejiang, China
| | - Weizhu Wu
- The Affiliated Lihuili Hospital of Ningbo University, Ningbo, 315000, Zhejiang, China.
| |
Collapse
|
13
|
Perry C, Greenberg O, Haberman S, Herskovitz N, Gazy I, Avinoam A, Paz-Yaacov N, Hershkovitz D, Avivi I. Image-Based Deep Learning Detection of High-Grade B-Cell Lymphomas Directly from Hematoxylin and Eosin Images. Cancers (Basel) 2023; 15:5205. [PMID: 37958379 PMCID: PMC10650414 DOI: 10.3390/cancers15215205] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Revised: 10/04/2023] [Accepted: 10/17/2023] [Indexed: 11/15/2023] Open
Abstract
Deep learning applications are emerging as promising new tools that can support the diagnosis and classification of different cancer types. While such solutions hold great potential for hematological malignancies, there have been limited studies describing the use of such applications in this field. The rapid diagnosis of double/triple-hit lymphomas (DHLs/THLs) involving MYC, BCL2 and/or BCL6 rearrangements is obligatory for optimal patient care. Here, we present a novel deep learning tool for diagnosing DHLs/THLs directly from scanned images of biopsy slides. A total of 57 biopsies, including 32 in a training set (including five DH lymphoma cases) and 25 in a validation set (including 10 DH/TH cases), were included. The DHL-classifier demonstrated a sensitivity of 100%, a specificity of 87% and an AUC of 0.95, with only two false positive cases, compared to FISH. The DHL-classifier showed a 92% predictive value as a screening tool for performing conventional FISH analysis, over-performing currently used criteria. The work presented here provides the proof of concept for the potential use of an AI tool for the identification of DH/TH events. However, more extensive follow-up studies are required to assess the robustness of this tool and achieve high performances in a diverse population.
Collapse
Affiliation(s)
- Chava Perry
- Hematology Division, Tel Aviv Sourasky Medical Center, Tel Aviv 6423906, Israel
- Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Orli Greenberg
- Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
- Pathology Department, Tel Aviv Sourasky Medical Center, Tel Aviv 6492601, Israel
| | - Shira Haberman
- Hematology Division, Tel Aviv Sourasky Medical Center, Tel Aviv 6423906, Israel
- Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Neta Herskovitz
- Hematology Division, Tel Aviv Sourasky Medical Center, Tel Aviv 6423906, Israel
| | - Inbal Gazy
- Imagene AI Ltd., Tel Aviv 6721409, Israel (N.P.-Y.)
| | | | | | - Dov Hershkovitz
- Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
- Pathology Department, Tel Aviv Sourasky Medical Center, Tel Aviv 6492601, Israel
| | - Irit Avivi
- Hematology Division, Tel Aviv Sourasky Medical Center, Tel Aviv 6423906, Israel
- Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
| |
Collapse
|
14
|
Yang Y, Sun K, Gao Y, Wang K, Yu G. Preparing Data for Artificial Intelligence in Pathology with Clinical-Grade Performance. Diagnostics (Basel) 2023; 13:3115. [PMID: 37835858 PMCID: PMC10572440 DOI: 10.3390/diagnostics13193115] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Revised: 09/27/2023] [Accepted: 09/28/2023] [Indexed: 10/15/2023] Open
Abstract
The pathology is decisive for disease diagnosis but relies heavily on experienced pathologists. In recent years, there has been growing interest in the use of artificial intelligence in pathology (AIP) to enhance diagnostic accuracy and efficiency. However, the impressive performance of deep learning-based AIP in laboratory settings often proves challenging to replicate in clinical practice. As the data preparation is important for AIP, the paper has reviewed AIP-related studies in the PubMed database published from January 2017 to February 2022, and 118 studies were included. An in-depth analysis of data preparation methods is conducted, encompassing the acquisition of pathological tissue slides, data cleaning, screening, and subsequent digitization. Expert review, image annotation, dataset division for model training and validation are also discussed. Furthermore, we delve into the reasons behind the challenges in reproducing the high performance of AIP in clinical settings and present effective strategies to enhance AIP's clinical performance. The robustness of AIP depends on a randomized collection of representative disease slides, incorporating rigorous quality control and screening, correction of digital discrepancies, reasonable annotation, and sufficient data volume. Digital pathology is fundamental in clinical-grade AIP, and the techniques of data standardization and weakly supervised learning methods based on whole slide image (WSI) are effective ways to overcome obstacles of performance reproduction. The key to performance reproducibility lies in having representative data, an adequate amount of labeling, and ensuring consistency across multiple centers. Digital pathology for clinical diagnosis, data standardization and the technique of WSI-based weakly supervised learning will hopefully build clinical-grade AIP.
Collapse
Affiliation(s)
- Yuanqing Yang
- Department of Biomedical Engineering, School of Basic Medical Sciences, Central South University, Changsha 410013, China; (Y.Y.); (K.S.)
- Department of Biomedical Engineering, School of Medical, Tsinghua University, Beijing 100084, China
| | - Kai Sun
- Department of Biomedical Engineering, School of Basic Medical Sciences, Central South University, Changsha 410013, China; (Y.Y.); (K.S.)
- Furong Laboratory, Changsha 410013, China
| | - Yanhua Gao
- Department of Ultrasound, Shaanxi Provincial People’s Hospital, Xi’an 710068, China;
| | - Kuansong Wang
- Department of Pathology, School of Basic Medical Sciences, Central South University, Changsha 410013, China;
- Department of Pathology, Xiangya Hospital, Central South University, Changsha 410013, China
| | - Gang Yu
- Department of Biomedical Engineering, School of Basic Medical Sciences, Central South University, Changsha 410013, China; (Y.Y.); (K.S.)
| |
Collapse
|
15
|
Mandair D, Reis-Filho JS, Ashworth A. Biological insights and novel biomarker discovery through deep learning approaches in breast cancer histopathology. NPJ Breast Cancer 2023; 9:21. [PMID: 37024522 PMCID: PMC10079681 DOI: 10.1038/s41523-023-00518-1] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Accepted: 02/27/2023] [Indexed: 04/08/2023] Open
Abstract
Breast cancer remains a highly prevalent disease with considerable inter- and intra-tumoral heterogeneity complicating prognostication and treatment decisions. The utilization and depth of genomic, transcriptomic and proteomic data for cancer has exploded over recent times and the addition of spatial context to this information, by understanding the correlating morphologic and spatial patterns of cells in tissue samples, has created an exciting frontier of research, histo-genomics. At the same time, deep learning (DL), a class of machine learning algorithms employing artificial neural networks, has rapidly progressed in the last decade with a confluence of technical developments - including the advent of modern graphic processing units (GPU), allowing efficient implementation of increasingly complex architectures at scale; advances in the theoretical and practical design of network architectures; and access to larger datasets for training - all leading to sweeping advances in image classification and object detection. In this review, we examine recent developments in the application of DL in breast cancer histology with particular emphasis of those producing biologic insights or novel biomarkers, spanning the extraction of genomic information to the use of stroma to predict cancer recurrence, with the aim of suggesting avenues for further advancing this exciting field.
Collapse
Affiliation(s)
- Divneet Mandair
- UCSF Helen Diller Family Comprehensive Cancer Center, San Francisco, CA, 94158, USA
| | | | - Alan Ashworth
- UCSF Helen Diller Family Comprehensive Cancer Center, San Francisco, CA, 94158, USA.
| |
Collapse
|
16
|
Wu S, Yue M, Zhang J, Li X, Li Z, Zhang H, Wang X, Han X, Cai L, Shang J, Jia Z, Wang X, Li J, Liu Y. The Role of Artificial Intelligence in Accurate Interpretation of HER2 Immunohistochemical Scores 0 and 1+ in Breast Cancer. Mod Pathol 2023; 36:100054. [PMID: 36788100 DOI: 10.1016/j.modpat.2022.100054] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2022] [Revised: 10/28/2022] [Accepted: 11/20/2022] [Indexed: 01/11/2023]
Abstract
The new human epidermal growth factor receptor (HER)2-targeting antibody-drug conjugate offers the opportunity to treat patients with HER2-low breast cancer. Distinguishing HER2 immunohistochemical (IHC) scores of 0 and 1+ is not only critical but also challenging owing to HER2 heterogeneity and variability of observers. In this study, we aimed to increase the interpretation accuracy and consistency of HER2 IHC 0 and 1+ evaluation through assistance from an artificial intelligence (AI) algorithm. In addition, we examined the value of our AI algorithm in evaluating HER2 IHC scores in tumors with heterogeneity. AI-assisted interpretation consisted of AI algorithms and an augmenting reality module with a microscope. Fifteen pathologists (5 junior, 5 midlevel, and 5 senior) participated in this multi-institutional 2-round ring study that included 246 infiltrating duct carcinoma cases that were not otherwise specified. In round 1, pathologists analyzed 246 HER2 IHC slides by microscope without AI assistance. After a 2-week washout period, the pathologists read the same slides with AI algorithm assistance and rendered the definitive results by adjusting to the AI algorithm. The accuracy of interpretation accuracy with AI assistance (0.93 vs 0.80), thereby the evaluation precision of HER2 0 and the recall of HER2 1+. In addition, the AI algorithm improved the total consistency (intraclass correlation coefficient = 0.542-0.812), especially in HER2 1+ cases. In cases with heterogeneity, accuracy improved significantly (0.68 to 0.89) and to a similar level as in cases without heterogeneity (accuracy, 0.97). Both accuracy and consistency improved more for junior pathologists than those for the midlevel and senior pathologists. To the best of our knowledge, this is the first study to show that the accuracy and consistency of HER2 IHC 0 and 1+ evaluation and the accuracy of HER2 IHC evaluation in breast cancers with heterogeneity can be significantly improved using AI-assisted interpretation.
Collapse
Affiliation(s)
- Si Wu
- Department of Pathology, the Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei, China
| | - Meng Yue
- Department of Pathology, the Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei, China
| | - Jun Zhang
- Tencent AI Lab, Nanshan District, Tencent Binhai Building, Shenzhen, Guangdong, China
| | - Xiaoxian Li
- Department of Pathology and Laboratory Medicine, The Emory University School of Medicine, Atlanta, Georgia
| | - Zaibo Li
- Department of Pathology, The Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Huina Zhang
- Department of Pathology, University of Rochester Medical Center, Rochester, New York
| | - Xinran Wang
- Department of Pathology, the Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei, China
| | - Xiao Han
- Tencent AI Lab, Nanshan District, Tencent Binhai Building, Shenzhen, Guangdong, China
| | - Lijing Cai
- Department of Pathology, the Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei, China
| | - Jiuyan Shang
- Department of Pathology, the Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei, China
| | - Zhanli Jia
- Department of Pathology, the Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei, China
| | - Xiaoxiao Wang
- Department of Pathology, the Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei, China
| | - Jinze Li
- Department of Pathology, the Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei, China
| | - Yueping Liu
- Department of Pathology, the Fourth Hospital of Hebei Medical University, Shijiazhuang, Hebei, China.
| |
Collapse
|
17
|
Wen Z, Wang S, Yang DM, Xie Y, Chen M, Bishop J, Xiao G. Deep learning in digital pathology for personalized treatment plans of cancer patients. Semin Diagn Pathol 2023; 40:109-119. [PMID: 36890029 DOI: 10.1053/j.semdp.2023.02.003] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/12/2023] [Accepted: 02/22/2023] [Indexed: 02/27/2023]
Abstract
Over the past decade, many new cancer treatments have been developed and made available to patients. However, in most cases, these treatments only benefit a specific subgroup of patients, making the selection of treatment for a specific patient an essential but challenging task for oncologists. Although some biomarkers were found to associate with treatment response, manual assessment is time-consuming and subjective. With the rapid developments and expanded implementation of artificial intelligence (AI) in digital pathology, many biomarkers can be quantified automatically from histopathology images. This approach allows for a more efficient and objective assessment of biomarkers, aiding oncologists in formulating personalized treatment plans for cancer patients. This review presents an overview and summary of the recent studies on biomarker quantification and treatment response prediction using hematoxylin-eosin (H&E) stained pathology images. These studies have shown that an AI-based digital pathology approach can be practical and will become increasingly important in improving the selection of cancer treatments for patients.
Collapse
Affiliation(s)
- Zhuoyu Wen
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Shidan Wang
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Donghan M Yang
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Yang Xie
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA; Simmons Comprehensive Cancer Center, UT Southwestern Medical Center, Dallas, TX, USA; Department of Pathology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Mingyi Chen
- Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
| | - Justin Bishop
- Department of Bioinformatics, UT Southwestern Medical Center, Dallas, TX, USA
| | - Guanghua Xiao
- Quantitative Biomedical Research Center, Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, USA; Simmons Comprehensive Cancer Center, UT Southwestern Medical Center, Dallas, TX, USA; Department of Pathology, University of Texas Southwestern Medical Center, Dallas, TX, USA.
| |
Collapse
|
18
|
Nasser M, Yusof UK. Deep Learning Based Methods for Breast Cancer Diagnosis: A Systematic Review and Future Direction. Diagnostics (Basel) 2023; 13:diagnostics13010161. [PMID: 36611453 PMCID: PMC9818155 DOI: 10.3390/diagnostics13010161] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2022] [Revised: 12/19/2022] [Accepted: 12/19/2022] [Indexed: 01/06/2023] Open
Abstract
Breast cancer is one of the precarious conditions that affect women, and a substantive cure has not yet been discovered for it. With the advent of Artificial intelligence (AI), recently, deep learning techniques have been used effectively in breast cancer detection, facilitating early diagnosis and therefore increasing the chances of patients' survival. Compared to classical machine learning techniques, deep learning requires less human intervention for similar feature extraction. This study presents a systematic literature review on the deep learning-based methods for breast cancer detection that can guide practitioners and researchers in understanding the challenges and new trends in the field. Particularly, different deep learning-based methods for breast cancer detection are investigated, focusing on the genomics and histopathological imaging data. The study specifically adopts the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), which offer a detailed analysis and synthesis of the published articles. Several studies were searched and gathered, and after the eligibility screening and quality evaluation, 98 articles were identified. The results of the review indicated that the Convolutional Neural Network (CNN) is the most accurate and extensively used model for breast cancer detection, and the accuracy metrics are the most popular method used for performance evaluation. Moreover, datasets utilized for breast cancer detection and the evaluation metrics are also studied. Finally, the challenges and future research direction in breast cancer detection based on deep learning models are also investigated to help researchers and practitioners acquire in-depth knowledge of and insight into the area.
Collapse
|
19
|
Zhao Y, Zhang J, Hu D, Qu H, Tian Y, Cui X. Application of Deep Learning in Histopathology Images of Breast Cancer: A Review. MICROMACHINES 2022; 13:2197. [PMID: 36557496 PMCID: PMC9781697 DOI: 10.3390/mi13122197] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/01/2022] [Revised: 12/04/2022] [Accepted: 12/09/2022] [Indexed: 06/17/2023]
Abstract
With the development of artificial intelligence technology and computer hardware functions, deep learning algorithms have become a powerful auxiliary tool for medical image analysis. This study was an attempt to use statistical methods to analyze studies related to the detection, segmentation, and classification of breast cancer in pathological images. After an analysis of 107 articles on the application of deep learning to pathological images of breast cancer, this study is divided into three directions based on the types of results they report: detection, segmentation, and classification. We introduced and analyzed models that performed well in these three directions and summarized the related work from recent years. Based on the results obtained, the significant ability of deep learning in the application of breast cancer pathological images can be recognized. Furthermore, in the classification and detection of pathological images of breast cancer, the accuracy of deep learning algorithms has surpassed that of pathologists in certain circumstances. Our study provides a comprehensive review of the development of breast cancer pathological imaging-related research and provides reliable recommendations for the structure of deep learning network models in different application scenarios.
Collapse
Affiliation(s)
- Yue Zhao
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Shenyang 110169, China
- Key Laboratory of Data Analytics and Optimization for Smart Industry, Northeastern University, Shenyang 110169, China
| | - Jie Zhang
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China
| | - Dayu Hu
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China
| | - Hui Qu
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China
| | - Ye Tian
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China
| | - Xiaoyu Cui
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang 110169, China
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Shenyang 110169, China
- Key Laboratory of Data Analytics and Optimization for Smart Industry, Northeastern University, Shenyang 110169, China
| |
Collapse
|
20
|
Couture HD. Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review. J Pers Med 2022; 12:2022. [PMID: 36556243 PMCID: PMC9784641 DOI: 10.3390/jpm12122022] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Revised: 10/26/2022] [Accepted: 12/05/2022] [Indexed: 12/12/2022] Open
Abstract
Molecular and genomic properties are critical in selecting cancer treatments to target individual tumors, particularly for immunotherapy. However, the methods to assess such properties are expensive, time-consuming, and often not routinely performed. Applying machine learning to H&E images can provide a more cost-effective screening method. Dozens of studies over the last few years have demonstrated that a variety of molecular biomarkers can be predicted from H&E alone using the advancements of deep learning: molecular alterations, genomic subtypes, protein biomarkers, and even the presence of viruses. This article reviews the diverse applications across cancer types and the methodology to train and validate these models on whole slide images. From bottom-up to pathologist-driven to hybrid approaches, the leading trends include a variety of weakly supervised deep learning-based approaches, as well as mechanisms for training strongly supervised models in select situations. While results of these algorithms look promising, some challenges still persist, including small training sets, rigorous validation, and model explainability. Biomarker prediction models may yield a screening method to determine when to run molecular tests or an alternative when molecular tests are not possible. They also create new opportunities in quantifying intratumoral heterogeneity and predicting patient outcomes.
Collapse
|
21
|
Deep learning-based image analysis predicts PD-L1 status from H&E-stained histopathology images in breast cancer. Nat Commun 2022; 13:6753. [PMID: 36347854 PMCID: PMC9643479 DOI: 10.1038/s41467-022-34275-9] [Citation(s) in RCA: 29] [Impact Index Per Article: 14.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2022] [Accepted: 10/18/2022] [Indexed: 11/10/2022] Open
Abstract
Programmed death ligand-1 (PD-L1) has been recently adopted for breast cancer as a predictive biomarker for immunotherapies. The cost, time, and variability of PD-L1 quantification by immunohistochemistry (IHC) are a challenge. In contrast, hematoxylin and eosin (H&E) is a robust staining used routinely for cancer diagnosis. Here, we show that PD-L1 expression can be predicted from H&E-stained images by employing state-of-the-art deep learning techniques. With the help of two expert pathologists and a designed annotation software, we construct a dataset to assess the feasibility of PD-L1 prediction from H&E in breast cancer. In a cohort of 3,376 patients, our system predicts the PD-L1 status in a high area under the curve (AUC) of 0.91 - 0.93. Our system is validated on two external datasets, including an independent clinical trial cohort, showing consistent prediction performance. Furthermore, the proposed system predicts which cases are prone to pathologists miss-interpretation, showing it can serve as a decision support and quality assurance system in clinical practice.
Collapse
|
22
|
Vuong TTL, Song B, Kwak JT, Kim K. Prediction of Epstein-Barr Virus Status in Gastric Cancer Biopsy Specimens Using a Deep Learning Algorithm. JAMA Netw Open 2022; 5:e2236408. [PMID: 36205993 PMCID: PMC9547324 DOI: 10.1001/jamanetworkopen.2022.36408] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
IMPORTANCE Epstein-Barr virus (EBV)-associated gastric cancer (EBV-GC) is 1 of 4 molecular subtypes of GC and is confirmed by an expensive molecular test, EBV-encoded small RNA in situ hybridization. EBV-GC has 2 histologic characteristics, lymphoid stroma and lace-like tumor pattern, but projecting EBV-GC at biopsy is difficult even for experienced pathologists. OBJECTIVE To develop and validate a deep learning algorithm to predict EBV status from pathology images of GC biopsy. DESIGN, SETTING, AND PARTICIPANTS This diagnostic study developed a deep learning classifier to predict EBV-GC using image patches of tissue microarray (TMA) and whole slide images (WSIs) of GC and applied it to GC biopsy specimens from GCs diagnosed at Kangbuk Samsung Hospital between 2011 and 2020. For a quantitative evaluation and EBV-GC prediction on biopsy specimens, the area of each class and the fraction in total tissue or tumor area were calculated. Data were analyzed from March 5, 2021, to February 10, 2022. MAIN OUTCOMES AND MEASURES Evaluation metrics of predictive model performance were assessed on accuracy, recall, precision, F1 score, area under the receiver operating characteristic curve (AUC), and κ coefficient. RESULTS This study included 137 184 image patches from 16 TMAs (708 tissue cores), 24 WSIs, and 286 biopsy images of GC. The classifier was able to classify EBV-GC image patches from TMAs and WSIs with 94.70% accuracy, 0.936 recall, 0.938 precision, 0.937 F1 score, and 0.909 κ coefficient. The classifier was used for predicting and measuring the area and fraction of EBV-GC on biopsy tissue specimens. A 10% cutoff value for the predicted fraction of EBV-GC to tissue (EBV-GC/tissue area) produced the best prediction results in EBV-GC biopsy specimens and showed the highest AUC value (0.8723; 95% CI, 0.7560-0.9501). That cutoff also obtained high sensitivity (0.895) and moderate specificity (0.745) compared with experienced pathologist sensitivity (0.842) and specificity (0.854) when using the presence of lymphoid stroma and a lace-like pattern as diagnostic criteria. On prediction maps, EBV-GCs with lace-like pattern and lymphoid stroma showed the same prediction results as EBV-GC, but cases lacking these histologic features revealed heterogeneous prediction results of EBV-GC and non-EBV-GC areas. CONCLUSIONS AND RELEVANCE This study showed the feasibility of EBV-GC prediction using a deep learning algorithm, even in biopsy samples. Use of such an image-based classifier before a confirmatory molecular test will reduce costs and tissue waste.
Collapse
Affiliation(s)
- Trinh Thi Le Vuong
- School of Electrical Engineering, Korea University, Seoul, Republic of Korea
| | - Boram Song
- Department of Pathology, Kangbuk Samsung Hospital, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| | - Jin T. Kwak
- School of Electrical Engineering, Korea University, Seoul, Republic of Korea
| | - Kyungeun Kim
- Department of Pathology, Kangbuk Samsung Hospital, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea
| |
Collapse
|
23
|
Lee SH, Jang HJ. Deep learning-based prediction of molecular cancer biomarkers from tissue slides: A new tool for precision oncology. Clin Mol Hepatol 2022; 28:754-772. [PMID: 35443570 PMCID: PMC9597228 DOI: 10.3350/cmh.2021.0394] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Accepted: 04/17/2022] [Indexed: 02/06/2023] Open
Abstract
Molecular tests are necessary to stratify cancer patients for targeted therapy. However, high cost and technical barriers limit the application of these tests, hindering optimal treatment. Recently, deep learning (DL) has been applied to predict molecular test results from digitized images of tissue slides. Furthermore, treatment response and prognosis can be predicted from tissue slides using DL. In this review, we summarized DL-based studies regarding the prediction of genetic mutation, microsatellite instability, tumor mutational burden, molecular subtypes, gene expression, treatment response, and prognosis directly from hematoxylin- and eosin-stained tissue slides. Although performance needs to be improved, these studies clearly demonstrated the feasibility of DL-based prediction of key molecular features in cancer tissues. With the accumulation of data and technical advances, the performance of the DL system could be improved in the near future. Therefore, we expect that DL could provide cost- and time-effective alternative tools for patient stratification in the era of precision oncology.
Collapse
Affiliation(s)
- Sung Hak Lee
- Department of Hospital Pathology, Seoul St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Korea
| | - Hyun-Jong Jang
- Catholic Big Data Integration Center, Department of Physiology, College of Medicine, The Catholic University of Korea, Seoul, Korea,Corresponding author : Hyun-Jong Jang Department of Physiology, College of Medicine, The Catholic University of Korea, 222 Banpo-daero, Seocho-gu, Seoul 06591, Korea Tel: +82-2-2258-7274, Fax: +82-2-532-9575, E-mail:
| |
Collapse
|
24
|
Fremond S, Koelzer VH, Horeweg N, Bosse T. The evolving role of morphology in endometrial cancer diagnostics: From histopathology and molecular testing towards integrative data analysis by deep learning. Front Oncol 2022; 12:928977. [PMID: 36059702 PMCID: PMC9433878 DOI: 10.3389/fonc.2022.928977] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2022] [Accepted: 07/15/2022] [Indexed: 11/13/2022] Open
Abstract
Endometrial cancer (EC) diagnostics is evolving into a system in which molecular aspects are increasingly important. The traditional histological subtype-driven classification has shifted to a molecular-based classification that stratifies EC into DNA polymerase epsilon mutated (POLEmut), mismatch repair deficient (MMRd), and p53 abnormal (p53abn), and the remaining EC as no specific molecular profile (NSMP). The molecular EC classification has been implemented in the World Health Organization 2020 classification and the 2021 European treatment guidelines, as it serves as a better basis for patient management. As a result, the integration of the molecular class with histopathological variables has become a critical focus of recent EC research. Pathologists have observed and described several morphological characteristics in association with specific genomic alterations, but these appear insufficient to accurately classify patients according to molecular subgroups. This requires pathologists to rely on molecular ancillary tests in routine workup. In this new era, it has become increasingly challenging to assign clinically relevant weights to histological and molecular features on an individual patient basis. Deep learning (DL) technology opens new options for the integrative analysis of multi-modal image and molecular datasets with clinical outcomes. Proof-of-concept studies in other cancers showed promising accuracy in predicting molecular alterations from H&E-stained tumor slide images. This suggests that some morphological characteristics that are associated with molecular alterations could be identified in EC, too, expanding the current understanding of the molecular-driven EC classification. Here in this review, we report the morphological characteristics of the molecular EC classification currently identified in the literature. Given the new challenges in EC diagnostics, this review discusses, therefore, the potential supportive role that DL could have, by providing an outlook on all relevant studies using DL on histopathology images in various cancer types with a focus on EC. Finally, we touch upon how DL might shape the management of future EC patients.
Collapse
Affiliation(s)
- Sarah Fremond
- Department of Pathology, Leiden University Medical Center (LUMC), Leiden, Netherlands
| | - Viktor Hendrik Koelzer
- Department of Pathology and Molecular Pathology, University Hospital and University of Zürich, Zürich, Switzerland
| | - Nanda Horeweg
- Department of Radiotherapy, Leiden University Medical Center, Leiden, Netherlands
| | - Tjalling Bosse
- Department of Pathology, Leiden University Medical Center (LUMC), Leiden, Netherlands
- *Correspondence: Tjalling Bosse,
| |
Collapse
|
25
|
Lu W, Toss M, Dawood M, Rakha E, Rajpoot N, Minhas F. SlideGraph +: Whole slide image level graphs to predict HER2 status in breast cancer. Med Image Anal 2022; 80:102486. [DOI: 10.1016/j.media.2022.102486] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2021] [Revised: 04/15/2022] [Accepted: 05/20/2022] [Indexed: 01/18/2023]
|
26
|
Liu H, Xu WD, Shang ZH, Wang XD, Zhou HY, Ma KW, Zhou H, Qi JL, Jiang JR, Tan LL, Zeng HM, Cai HJ, Wang KS, Qian YL. Breast Cancer Molecular Subtype Prediction on Pathological Images with Discriminative Patch Selection and Multi-Instance Learning. Front Oncol 2022; 12:858453. [PMID: 35494021 PMCID: PMC9046851 DOI: 10.3389/fonc.2022.858453] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2022] [Accepted: 03/14/2022] [Indexed: 11/13/2022] Open
Abstract
Molecular subtypes of breast cancer are important references to personalized clinical treatment. For cost and labor savings, only one of the patient’s paraffin blocks is usually selected for subsequent immunohistochemistry (IHC) to obtain molecular subtypes. Inevitable block sampling error is risky due to the tumor heterogeneity and could result in a delay in treatment. Molecular subtype prediction from conventional H&E pathological whole slide images (WSI) using the AI method is useful and critical to assist pathologists to pre-screen proper paraffin block for IHC. It is a challenging task since only WSI-level labels of molecular subtypes from IHC can be obtained without detailed local region information. Gigapixel WSIs are divided into a huge amount of patches to be computationally feasible for deep learning, while with coarse slide-level labels, patch-based methods may suffer from abundant noise patches, such as folds, overstained regions, or non-tumor tissues. A weakly supervised learning framework based on discriminative patch selection and multi-instance learning was proposed for breast cancer molecular subtype prediction from H&E WSIs. Firstly, co-teaching strategy using two networks was adopted to learn molecular subtype representations and filter out some noise patches. Then, a balanced sampling strategy was used to handle the imbalance in subtypes in the dataset. In addition, a noise patch filtering algorithm that used local outlier factor based on cluster centers was proposed to further select discriminative patches. Finally, a loss function integrating local patch with global slide constraint information was used to fine-tune MIL framework on obtained discriminative patches and further improve the prediction performance of molecular subtyping. The experimental results confirmed the effectiveness of the proposed AI method and our models outperformed even senior pathologists, which has the potential to assist pathologists to pre-screen paraffin blocks for IHC in clinic.
Collapse
Affiliation(s)
- Hong Liu
- Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
- *Correspondence: Hong Liu, ; Kuan-Song Wang,
| | - Wen-Dong Xu
- Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Zi-Hao Shang
- Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Xiang-Dong Wang
- Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
| | - Hai-Yan Zhou
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Ke-Wen Ma
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Huan Zhou
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Jia-Lin Qi
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Jia-Rui Jiang
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Li-Lan Tan
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Hui-Min Zeng
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Hui-Juan Cai
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
| | - Kuan-Song Wang
- Department of Pathology, Xiangya Hospital, Central South University, Changsha, China
- School of Basic Medical Science, Central South University, Changsha, China
- *Correspondence: Hong Liu, ; Kuan-Song Wang,
| | - Yue-Liang Qian
- Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
27
|
Nassif AB, Talib MA, Nasir Q, Afadar Y, Elgendy O. Breast cancer detection using artificial intelligence techniques: A systematic literature review. Artif Intell Med 2022; 127:102276. [DOI: 10.1016/j.artmed.2022.102276] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2021] [Revised: 10/18/2021] [Accepted: 03/04/2022] [Indexed: 02/07/2023]
|
28
|
Garberis I, Andre F, Lacroix-Triki M. L’intelligence artificielle pourrait-elle intervenir dans l’aide au diagnostic des cancers du sein ? – L’exemple de HER2. Bull Cancer 2022; 108:11S35-11S45. [PMID: 34969514 DOI: 10.1016/s0007-4551(21)00635-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Abstract
HER2 is an important prognostic and predictive biomarker in breast cancer. Its detection makes it possible to define which patients will benefit from a targeted treatment. While assessment of HER2 status by immunohistochemistry in positive vs negative categories is well implemented and reproducible, the introduction of a new "HER2-low" category could raise some concerns about its scoring and reproducibility. We herein described the current HER2 testing methods and the application of innovative machine learning techniques to improve these determinations, as well as the main challenges and opportunities related to the implementation of digital pathology in the up-and-coming AI era.
Collapse
Affiliation(s)
- Ingrid Garberis
- Inserm UMR 981, Gustave Roussy Cancer Campus, Villejuif, France; Université Paris-Saclay, 94270 Le Kremlin-Bicêtre, France.
| | - Fabrice Andre
- Inserm UMR 981, Gustave Roussy Cancer Campus, Villejuif, France; Université Paris-Saclay, 94270 Le Kremlin-Bicêtre, France; Département d'oncologie médicale, Gustave-Roussy, Villejuif, France
| | - Magali Lacroix-Triki
- Inserm UMR 981, Gustave Roussy Cancer Campus, Villejuif, France; Département d'anatomie et cytologie pathologiques, Gustave-Roussy, Villejuif, France
| |
Collapse
|
29
|
Bychkov D, Joensuu H, Nordling S, Tiulpin A, Kücükel H, Lundin M, Sihto H, Isola J, Lehtimäki T, Kellokumpu-Lehtinen PL, Smitten K, Lundin J, Linder N. Outcome and biomarker supervised deep learning for survival prediction in two multicenter breast cancer series. J Pathol Inform 2022; 13:9. [PMID: 35136676 PMCID: PMC8794033 DOI: 10.4103/jpi.jpi_29_21] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2021] [Revised: 06/10/2021] [Accepted: 06/20/2021] [Indexed: 11/06/2022] Open
Abstract
Background: Prediction of clinical outcomes for individual cancer patients is an important step in the disease diagnosis and subsequently guides the treatment and patient counseling. In this work, we develop and evaluate a joint outcome and biomarker supervised (estrogen receptor expression and ERBB2 expression and gene amplification) multitask deep learning model for prediction of outcome in breast cancer patients in two nation-wide multicenter studies in Finland (the FinProg and FinHer studies). Our approach combines deep learning with expert knowledge to provide more accurate, robust, and integrated prediction of breast cancer outcomes. Materials and Methods: Using deep learning, we trained convolutional neural networks (CNNs) with digitized tissue microarray (TMA) samples of primary hematoxylin-eosin-stained breast cancer specimens from 693 patients in the FinProg series as input and breast cancer-specific survival as the endpoint. The trained algorithms were tested on 354 TMA patient samples in the same series. An independent set of whole-slide (WS) tumor samples from 674 patients in another multicenter study (FinHer) was used to validate and verify the generalization of the outcome prediction based on CNN models by Cox survival regression and concordance index (c-index). Visual cancer tissue characterization, i.e., number of mitoses, tubules, nuclear pleomorphism, tumor-infiltrating lymphocytes, and necrosis was performed on TMA samples in the FinProg test set by a pathologist and combined with deep learning-based outcome prediction in a multitask algorithm. Results: The multitask algorithm achieved a hazard ratio (HR) of 2.0 (95% confidence interval [CI] 1.30–3.00), P < 0.001, c-index of 0.59 on the 354 test set of FinProg patients, and an HR of 1.7 (95% CI 1.2–2.6), P = 0.003, c-index 0.57 on the WS tumor samples from 674 patients in the independent FinHer series. The multitask CNN remained a statistically independent predictor of survival in both test sets when adjusted for histological grade, tumor size, and axillary lymph node status in a multivariate Cox analyses. An improved accuracy (c-index 0.66) was achieved when deep learning was combined with the tissue characteristics assessed visually by a pathologist. Conclusions: A multitask deep learning algorithm supervised by both patient outcome and biomarker status learned features in basic tissue morphology predictive of survival in a nationwide, multicenter series of patients with breast cancer. The algorithms generalized to another independent multicenter patient series and whole-slide breast cancer samples and provide prognostic information complementary to that of a comprehensive series of established prognostic factors.
Collapse
|
30
|
Direct identification of ALK and ROS1 fusions in non-small cell lung cancer from hematoxylin and eosin-stained slides using deep learning algorithms. Mod Pathol 2022; 35:1882-1887. [PMID: 36057739 PMCID: PMC9708557 DOI: 10.1038/s41379-022-01141-4] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2022] [Revised: 07/20/2022] [Accepted: 07/20/2022] [Indexed: 12/24/2022]
Abstract
Anaplastic lymphoma kinase (ALK) and ROS oncogene 1 (ROS1) gene fusions are well-established key players in non-small cell lung cancer (NSCLC). Although their frequency is relatively low, their detection is important for patient care and guides therapeutic decisions. The accepted methods used for their detection are immunohistochemistry (IHC) and fluorescence in situ hybridization (FISH) assay, as well as DNA and RNA-based sequencing methodologies. These assays are expensive, time-consuming, and require technical expertise and specialized equipment as well as biological specimens that are not always available. Here we present an alternative detection method using a computer vision deep learning approach. An advanced convolutional neural network (CNN) was used to generate classifier models to detect ALK and ROS1-fusions directly from scanned hematoxylin and eosin (H&E) whole slide images prepared from NSCLC tumors of patients. A two-step training approach was applied, with an initial unsupervised training step performed on a pan-cancer sample cohort followed by a semi-supervised fine-tuning step, which supported the development of a classifier with performances equal to those accepted for diagnostic tests. Validation of the ALK/ROS1 classifier on a cohort of 72 lung cancer cases who underwent ALK and ROS1-fusion testing at the pathology department at Sheba Medical Center displayed sensitivities of 100% for both genes (six ALK-positive and two ROS1-positive cases) and specificities of 100% and 98.6% respectively for ALK and ROS1, with only one false-positive result for ROS1-alteration. These results demonstrate the potential advantages that machine learning solutions may have in the molecular pathology domain, by allowing fast, standardized, accurate, and robust biomarker detection overcoming many limitations encountered when using current techniques. The integration of such novel solutions into the routine pathology workflow can support and improve the current clinical pipeline.
Collapse
|
31
|
Validation of HER2 Status in Whole Genome Sequencing Data of Breast Cancers with the Ploidy-Corrected Copy Number Approach. Mol Diagn Ther 2021; 26:105-116. [PMID: 34932189 PMCID: PMC8766398 DOI: 10.1007/s40291-021-00571-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/02/2021] [Indexed: 11/02/2022]
Abstract
BACKGROUND AND OBJECTIVE Human epidermal growth factor receptor 2 (HER2) protein overexpression is one of the most significant biomarkers for breast cancer diagnostics, treatment prediction, and prognostics. The high accessibility of HER2 inhibitors in routine clinical practice directly translates into the diagnostic need for precise and robust marker identification. Even though multigene next-generation sequencing methodologies have slowly taken over the field of single-biomarker molecular tests, the copy number alterations such as amplification of the HER2-coding ERBB2 gene are hard to validate on next-generation sequencing platforms as they are characterized by chromosomal structural heterogeneity, polysomy, and genomic context of ploidy. In our study, we tested the approach of using whole genome sequencing instead of next-generation sequencing panels to determine HER2 status in the clinical set-up. METHODS We used a large dataset of 876 patients with breast cancer whole genomes with curated clinical data and an additional set of 551 patients' external genomic data. We used the decision-tree-based algorithm for optimization of the diagnostic tool for HER2 status assessment by whole genome sequencing. RESULTS The most efficient approach to assess HER2 status in whole genome sequencing data was the ploidy-corrected copy number, utilizing ERBB2 copy number and mean tumor ploidy. The classifier achieved sensitivity of 91.18% and specificity of 98.69% on the internal validation dataset and 89.86% and 96.06% on the external data, which is similar to other next-generation sequencing methods, currently tested in the clinic. CONCLUSIONS We provide evidence that the HER2 status may be reliably determined by whole genome sequencing and is applicable across different laboratory protocols and pipelines. We suggest using the ploidy-corrected copy number for diagnostic purposes.
Collapse
|
32
|
Yousif M, van Diest PJ, Laurinavicius A, Rimm D, van der Laak J, Madabhushi A, Schnitt S, Pantanowitz L. Artificial intelligence applied to breast pathology. Virchows Arch 2021; 480:191-209. [PMID: 34791536 DOI: 10.1007/s00428-021-03213-3] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2021] [Revised: 09/12/2021] [Accepted: 09/27/2021] [Indexed: 12/12/2022]
Abstract
The convergence of digital pathology and computer vision is increasingly enabling computers to perform tasks performed by humans. As a result, artificial intelligence (AI) is having an astoundingly positive effect on the field of pathology, including breast pathology. Research using machine learning and the development of algorithms that learn patterns from labeled digital data based on "deep learning" neural networks and feature-engineered approaches to analyze histology images have recently provided promising results. Thus far, image analysis and more complex AI-based tools have demonstrated excellent success performing tasks such as the quantification of breast biomarkers and Ki67, mitosis detection, lymph node metastasis recognition, tissue segmentation for diagnosing breast carcinoma, prognostication, computational assessment of tumor-infiltrating lymphocytes, and prediction of molecular expression as well as treatment response and benefit of therapy from routine H&E images. This review critically examines the literature regarding these applications of AI in the area of breast pathology.
Collapse
Affiliation(s)
- Mustafa Yousif
- Department of Pathology, University of Michigan, Ann Arbor, MI, USA.
- Department of Pathology, Vanderbilt University Medical Center, Nashville, TN, USA.
| | - Paul J van Diest
- Department of Pathology, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Arvydas Laurinavicius
- Department of Pathology, Pharmacology and Forensic Medicine, Faculty of Medicine, Vilnius University, and National Center of Pathology, Affiliate of Vilnius University Hospital Santaros Klinikos, Vilnius, Lithuania
| | - David Rimm
- Department of Pathology, Yale University School of Medicine, New Haven, CT, USA
| | - Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Anant Madabhushi
- Department of Biomedical Engineering, Case Western Reserve University, and Louis Stokes Cleveland Veterans Administration Medical Center, Cleveland, OH, USA
- Louis Stokes Cleveland Veterans Administration Medical Center, Cleveland, OH, USA
| | - Stuart Schnitt
- Department of Pathology, Brigham and Women's Hospital and Harvard Medical School, Breast Oncology Program, Dana-Farber/Brigham and Women's Cancer Center, Boston, MA, USA
| | | |
Collapse
|
33
|
Applying Self-Supervised Learning to Medicine: Review of the State of the Art and Medical Implementations. INFORMATICS 2021. [DOI: 10.3390/informatics8030059] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Machine learning has become an increasingly ubiquitous technology, as big data continues to inform and influence everyday life and decision-making. Currently, in medicine and healthcare, as well as in most other industries, the two most prevalent machine learning paradigms are supervised learning and transfer learning. Both practices rely on large-scale, manually annotated datasets to train increasingly complex models. However, the requirement of data to be manually labeled leaves an excess of unused, unlabeled data available in both public and private data repositories. Self-supervised learning (SSL) is a growing area of machine learning that can take advantage of unlabeled data. Contrary to other machine learning paradigms, SSL algorithms create artificial supervisory signals from unlabeled data and pretrain algorithms on these signals. The aim of this review is two-fold: firstly, we provide a formal definition of SSL, divide SSL algorithms into their four unique subsets, and review the state of the art published in each of those subsets between the years of 2014 and 2020. Second, this work surveys recent SSL algorithms published in healthcare, in order to provide medical experts with a clearer picture of how they can integrate SSL into their research, with the objective of leveraging unlabeled data.
Collapse
|
34
|
Murchan P, Ó’Brien C, O’Connell S, McNevin CS, Baird AM, Sheils O, Ó Broin P, Finn SP. Deep Learning of Histopathological Features for the Prediction of Tumour Molecular Genetics. Diagnostics (Basel) 2021; 11:1406. [PMID: 34441338 PMCID: PMC8393642 DOI: 10.3390/diagnostics11081406] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2021] [Revised: 07/24/2021] [Accepted: 07/27/2021] [Indexed: 12/12/2022] Open
Abstract
Advanced diagnostics are enabling cancer treatments to become increasingly tailored to the individual through developments in immunotherapies and targeted therapies. However, long turnaround times and high costs of molecular testing hinder the widespread implementation of targeted cancer treatments. Meanwhile, gold-standard histopathological assessment carried out by a trained pathologist is widely regarded as routine and mandatory in most cancers. Recently, methods have been developed to mine hidden information from histopathological slides using deep learning applied to scanned and digitized slides; deep learning comprises a collection of computational methods which learn patterns in data in order to make predictions. Such methods have been reported to be successful in a variety of cancers for predicting the presence of biomarkers such as driver mutations, tumour mutational burden, and microsatellite instability. This information could prove valuable to pathologists and oncologists in clinical decision making for cancer treatment and triage for in-depth sequencing. In addition to identifying molecular features, deep learning has been applied to predict prognosis and treatment response in certain cancers. Despite reported successes, many challenges remain before the clinical implementation of such diagnostic strategies in the clinical setting is possible. This review aims to outline recent developments in the field of deep learning for predicting molecular genetics from histopathological slides, as well as to highlight limitations and pitfalls of working with histopathology slides in deep learning.
Collapse
Affiliation(s)
- Pierre Murchan
- Department of Histopathology and Morbid Anatomy, Trinity Translational Medicine Institute, Trinity College Dublin, D08 W9RT Dublin, Ireland; (P.M.); (C.Ó.); (C.S.M.)
| | - Cathal Ó’Brien
- Department of Histopathology and Morbid Anatomy, Trinity Translational Medicine Institute, Trinity College Dublin, D08 W9RT Dublin, Ireland; (P.M.); (C.Ó.); (C.S.M.)
- Department of Histopathology, St James’s Hospital, P.O. Box 580, James’s Street, D08 X4RX Dublin, Ireland
| | - Shane O’Connell
- School of Mathematics, Statistics, and Applied Mathematics, National University of Ireland Galway, H91 TK33 Galway, Ireland; (S.O.); (P.Ó.B.)
| | - Ciara S. McNevin
- Department of Histopathology and Morbid Anatomy, Trinity Translational Medicine Institute, Trinity College Dublin, D08 W9RT Dublin, Ireland; (P.M.); (C.Ó.); (C.S.M.)
- Department of Medical Oncology, St James’s Hospital, D08 NHY1 Dublin, Ireland
| | - Anne-Marie Baird
- School of Medicine, Trinity Translational Medicine Institute, Trinity College Dublin, D02 A440 Dublin, Ireland; (A.-M.B.); (O.S.)
| | - Orla Sheils
- School of Medicine, Trinity Translational Medicine Institute, Trinity College Dublin, D02 A440 Dublin, Ireland; (A.-M.B.); (O.S.)
| | - Pilib Ó Broin
- School of Mathematics, Statistics, and Applied Mathematics, National University of Ireland Galway, H91 TK33 Galway, Ireland; (S.O.); (P.Ó.B.)
| | - Stephen P. Finn
- Department of Histopathology and Morbid Anatomy, Trinity Translational Medicine Institute, Trinity College Dublin, D08 W9RT Dublin, Ireland; (P.M.); (C.Ó.); (C.S.M.)
- Department of Histopathology, St James’s Hospital, P.O. Box 580, James’s Street, D08 X4RX Dublin, Ireland
| |
Collapse
|
35
|
Gamble P, Jaroensri R, Wang H, Tan F, Moran M, Brown T, Flament-Auvigne I, Rakha EA, Toss M, Dabbs DJ, Regitnig P, Olson N, Wren JH, Robinson C, Corrado GS, Peng LH, Liu Y, Mermel CH, Steiner DF, Chen PHC. Determining breast cancer biomarker status and associated morphological features using deep learning. COMMUNICATIONS MEDICINE 2021; 1:14. [PMID: 35602213 PMCID: PMC9037318 DOI: 10.1038/s43856-021-00013-3] [Citation(s) in RCA: 36] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2021] [Accepted: 06/18/2021] [Indexed: 11/09/2022] Open
Abstract
Abstract
Background
Breast cancer management depends on biomarkers including estrogen receptor, progesterone receptor, and human epidermal growth factor receptor 2 (ER/PR/HER2). Though existing scoring systems are widely used and well-validated, they can involve costly preparation and variable interpretation. Additionally, discordances between histology and expected biomarker findings can prompt repeat testing to address biological, interpretative, or technical reasons for unexpected results.
Methods
We developed three independent deep learning systems (DLS) to directly predict ER/PR/HER2 status for both focal tissue regions (patches) and slides using hematoxylin-and-eosin-stained (H&E) images as input. Models were trained and evaluated using pathologist annotated slides from three data sources. Areas under the receiver operator characteristic curve (AUCs) were calculated for test sets at both a patch-level (>135 million patches, 181 slides) and slide-level (n = 3274 slides, 1249 cases, 37 sites). Interpretability analyses were performed using Testing with Concept Activation Vectors (TCAV), saliency analysis, and pathologist review of clustered patches.
Results
The patch-level AUCs are 0.939 (95%CI 0.936–0.941), 0.938 (0.936–0.940), and 0.808 (0.802–0.813) for ER/PR/HER2, respectively. At the slide level, AUCs are 0.86 (95%CI 0.84–0.87), 0.75 (0.73–0.77), and 0.60 (0.56–0.64) for ER/PR/HER2, respectively. Interpretability analyses show known biomarker-histomorphology associations including associations of low-grade and lobular histology with ER/PR positivity, and increased inflammatory infiltrates with triple-negative staining.
Conclusions
This study presents rapid breast cancer biomarker estimation from routine H&E slides and builds on prior advances by prioritizing interpretability of computationally learned features in the context of existing pathological knowledge.
Collapse
|
36
|
van der Laak J, Litjens G, Ciompi F. Deep learning in histopathology: the path to the clinic. Nat Med 2021; 27:775-784. [PMID: 33990804 DOI: 10.1038/s41591-021-01343-4] [Citation(s) in RCA: 266] [Impact Index Per Article: 88.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2020] [Accepted: 03/31/2021] [Indexed: 02/08/2023]
Abstract
Machine learning techniques have great potential to improve medical diagnostics, offering ways to improve accuracy, reproducibility and speed, and to ease workloads for clinicians. In the field of histopathology, deep learning algorithms have been developed that perform similarly to trained pathologists for tasks such as tumor detection and grading. However, despite these promising results, very few algorithms have reached clinical implementation, challenging the balance between hope and hype for these new techniques. This Review provides an overview of the current state of the field, as well as describing the challenges that still need to be addressed before artificial intelligence in histopathology can achieve clinical value.
Collapse
Affiliation(s)
- Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands. .,Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden.
| | - Geert Litjens
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Francesco Ciompi
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| |
Collapse
|
37
|
Machine-Learning-Based Evaluation of Intratumoral Heterogeneity and Tumor-Stroma Interface for Clinical Guidance. THE AMERICAN JOURNAL OF PATHOLOGY 2021; 191:1724-1731. [PMID: 33895120 DOI: 10.1016/j.ajpath.2021.04.008] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/16/2021] [Accepted: 04/15/2021] [Indexed: 12/21/2022]
Abstract
Assessment of intratumoral heterogeneity and tumor-host interaction within the tumor microenvironment is becoming increasingly important for innovative cancer therapy decisions because of the unique information it can generate about the state of the disease. However, its assessment and quantification are limited by ambiguous definitions of the tumor-host interface and by human cognitive capacity in current pathology practice. Advances in machine learning and artificial intelligence have opened the field of digital pathology to novel tissue image analytics and feature extraction for generation of high-capacity computational disease management models. A particular benefit is expected from machine-learning applications that can perform extraction and quantification of subvisual features of both intratumoral heterogeneity and tumor microenvironment aspects. These methods generate information about cancer cell subpopulation heterogeneity, potential tumor-host interactions, and tissue microarchitecture, derived from morphologically resolved content using both explicit and implicit features. Several studies have achieved promising diagnostic, prognostic, and predictive artificial intelligence models that often outperform current clinical and pathology criteria. However, further effort is needed for clinical adoption of such methods through development of standardizable high-capacity workflows and proper validation studies.
Collapse
|
38
|
Artificial Intelligence & Tissue Biomarkers: Advantages, Risks and Perspectives for Pathology. Cells 2021; 10:cells10040787. [PMID: 33918173 PMCID: PMC8066881 DOI: 10.3390/cells10040787] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2021] [Revised: 03/30/2021] [Accepted: 03/31/2021] [Indexed: 12/18/2022] Open
Abstract
Tissue Biomarkers are information written in the tissue and used in Pathology to recognize specific subsets of patients with diagnostic, prognostic or predictive purposes, thus representing the key elements of Personalized Medicine. The advent of Artificial Intelligence (AI) promises to further reinforce the role of Pathology in the scenario of Personalized Medicine: AI-based devices are expected to standardize the evaluation of tissue biomarkers and also to discover novel information, which would otherwise be ignored by human review, and use them to make specific predictions. In this review we will present how AI has been used to support Tissue Biomarkers evaluation in the specific field of Pathology, give an insight to the intriguing field of AI-based biomarkers and discuss possible advantages, risk and perspectives for Pathology.
Collapse
|
39
|
Stenzinger A, Alber M, Allgäuer M, Jurmeister P, Bockmayr M, Budczies J, Lennerz J, Eschrich J, Kazdal D, Schirmacher P, Wagner AH, Tacke F, Capper D, Müller KR, Klauschen F. Artificial intelligence and pathology: From principles to practice and future applications in histomorphology and molecular profiling. Semin Cancer Biol 2021; 84:129-143. [PMID: 33631297 DOI: 10.1016/j.semcancer.2021.02.011] [Citation(s) in RCA: 33] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2020] [Revised: 01/29/2021] [Accepted: 02/16/2021] [Indexed: 02/07/2023]
Abstract
The complexity of diagnostic (surgical) pathology has increased substantially over the last decades with respect to histomorphological and molecular profiling. Pathology has steadily expanded its role in tumor diagnostics and beyond from disease entity identification via prognosis estimation to precision therapy prediction. It is therefore not surprising that pathology is among the disciplines in medicine with high expectations in the application of artificial intelligence (AI) or machine learning approaches given their capabilities to analyze complex data in a quantitative and standardized manner to further enhance scope and precision of diagnostics. While an obvious application is the analysis of histological images, recent applications for the analysis of molecular profiling data from different sources and clinical data support the notion that AI will enhance both histopathology and molecular pathology in the future. At the same time, current literature should not be misunderstood in a way that pathologists will likely be replaced by AI applications in the foreseeable future. Although AI will transform pathology in the coming years, recent studies reporting AI algorithms to diagnose cancer or predict certain molecular properties deal with relatively simple diagnostic problems that fall short of the diagnostic complexity pathologists face in clinical routine. Here, we review the pertinent literature of AI methods and their applications to pathology, and put the current achievements and what can be expected in the future in the context of the requirements for research and routine diagnostics.
Collapse
Affiliation(s)
- Albrecht Stenzinger
- Institute of Pathology, University Hospital Heidelberg, Im Neuenheimer Feld 224, Heidelberg, 69120, Germany; German Cancer Consortium (DKTK), Partner Site Heidelberg, and German Cancer Research Center (DKFZ), Heidelberg, Germany; German Center for Lung Research (DZL), Partner Site Heidelberg, Heidelberg, Germany.
| | - Maximilian Alber
- Institute of Pathology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany; Aignostics GmbH, Schumannstr. 17, Berlin, 10117, Germany
| | - Michael Allgäuer
- Institute of Pathology, University Hospital Heidelberg, Im Neuenheimer Feld 224, Heidelberg, 69120, Germany
| | - Philipp Jurmeister
- Institute of Pathology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany; German Cancer Consortium (DKTK), Partner Site Berlin, and German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Michael Bockmayr
- Institute of Pathology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany; Department of Pediatric Hematology and Oncology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany; Research Institute, Children's Cancer Center Hamburg, Hamburg, Germany
| | - Jan Budczies
- Institute of Pathology, University Hospital Heidelberg, Im Neuenheimer Feld 224, Heidelberg, 69120, Germany; German Cancer Consortium (DKTK), Partner Site Heidelberg, and German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jochen Lennerz
- Department of Pathology, Center for Integrated Diagnostics, Harvard Medical School, Massachusetts General Hospital, Boston, MA, USA
| | - Johannes Eschrich
- Department of Hepatology & Gastroenterology, Charité University Medical Center, Berlin, Germany
| | - Daniel Kazdal
- Institute of Pathology, University Hospital Heidelberg, Im Neuenheimer Feld 224, Heidelberg, 69120, Germany; German Center for Lung Research (DZL), Partner Site Heidelberg, Heidelberg, Germany
| | - Peter Schirmacher
- Institute of Pathology, University Hospital Heidelberg, Im Neuenheimer Feld 224, Heidelberg, 69120, Germany; German Cancer Consortium (DKTK), Partner Site Heidelberg, and German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alex H Wagner
- The Steve and Cindy Rasmussen Institute for Genomic Medicine, Nationwide Children's Hospital, Columbus, OH, 43205, USA; Department of Pediatrics, The Ohio State University, Columbus, OH, 43210, USA
| | - Frank Tacke
- Department of Hepatology & Gastroenterology, Charité University Medical Center, Berlin, Germany
| | - David Capper
- German Cancer Consortium (DKTK), Partner Site Berlin, and German Cancer Research Center (DKFZ), Heidelberg, Germany; Department of Neuropathology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin, Germany
| | - Klaus-Robert Müller
- Machine Learning Group, Technische Universität Berlin, Berlin, 10587, Germany; Department of Artificial Intelligence, Korea University, Seoul, 136-713, South Korea; Max-Planck-Institute for Informatics, Saarland Informatics Campus E1 4, Saarbrücken, 66123, Germany; Google Research, Brain Team, Berlin, Germany.
| | - Frederick Klauschen
- Institute of Pathology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin and Berlin Institute of Health, Berlin, Germany; German Cancer Consortium (DKTK), Partner Site Berlin, and German Cancer Research Center (DKFZ), Heidelberg, Germany; Institute of Pathology, Ludwig-Maximilians-Universität München, Thalkirchner Strasse 36, München, 80337, Germany.
| |
Collapse
|
40
|
Bychkov D, Linder N, Tiulpin A, Kücükel H, Lundin M, Nordling S, Sihto H, Isola J, Lehtimäki T, Kellokumpu-Lehtinen PL, von Smitten K, Joensuu H, Lundin J. Deep learning identifies morphological features in breast cancer predictive of cancer ERBB2 status and trastuzumab treatment efficacy. Sci Rep 2021; 11:4037. [PMID: 33597560 PMCID: PMC7890057 DOI: 10.1038/s41598-021-83102-6] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2020] [Accepted: 01/29/2021] [Indexed: 02/08/2023] Open
Abstract
The treatment of patients with ERBB2 (HER2)-positive breast cancer with anti-ERBB2 therapy is based on the detection of ERBB2 gene amplification or protein overexpression. Machine learning (ML) algorithms can predict the amplification of ERBB2 based on tumor morphological features, but it is not known whether ML-derived features can predict survival and efficacy of anti-ERBB2 treatment. In this study, we trained a deep learning model with digital images of hematoxylin–eosin (H&E)-stained formalin-fixed primary breast tumor tissue sections, weakly supervised by ERBB2 gene amplification status. The gene amplification was determined by chromogenic in situ hybridization (CISH). The training data comprised digitized tissue microarray (TMA) samples from 1,047 patients. The correlation between the deep learning–predicted ERBB2 status, which we call H&E-ERBB2 score, and distant disease-free survival (DDFS) was investigated on a fully independent test set, which included whole-slide tumor images from 712 patients with trastuzumab treatment status available. The area under the receiver operating characteristic curve (AUC) in predicting gene amplification in the test sets was 0.70 (95% CI, 0.63–0.77) on 354 TMA samples and 0.67 (95% CI, 0.62–0.71) on 712 whole-slide images. Among patients with ERBB2-positive cancer treated with trastuzumab, those with a higher than the median morphology–based H&E-ERBB2 score derived from machine learning had more favorable DDFS than those with a lower score (hazard ratio [HR] 0.37; 95% CI, 0.15–0.93; P = 0.034). A high H&E-ERBB2 score was associated with unfavorable survival in patients with ERBB2-negative cancer as determined by CISH. ERBB2-associated morphology correlated with the efficacy of adjuvant anti-ERBB2 treatment and can contribute to treatment-predictive information in breast cancer.
Collapse
Affiliation(s)
- Dmitrii Bychkov
- Institute for Molecular Medicine Finland (FIMM), Nordic EMBL Partnership for Molecular Medicine, University of Helsinki, Helsinki, Finland. .,iCAN Digital Precision Cancer Medicine Flagship, Helsinki, Finland.
| | - Nina Linder
- Institute for Molecular Medicine Finland (FIMM), Nordic EMBL Partnership for Molecular Medicine, University of Helsinki, Helsinki, Finland.,iCAN Digital Precision Cancer Medicine Flagship, Helsinki, Finland.,Department of Women's and Children's Health, International Maternal and Child Health, Uppsala University, Uppsala, Sweden
| | - Aleksei Tiulpin
- Physics and Technology, Research Unit of Medical Imaging, University of Oulu, Oulu, Finland.,Department of Diagnostic Radiology, Oulu University Hospital, Oulu, Finland.,Ailean Technologies Oy, Oulu, Finland
| | - Hakan Kücükel
- Institute for Molecular Medicine Finland (FIMM), Nordic EMBL Partnership for Molecular Medicine, University of Helsinki, Helsinki, Finland.,iCAN Digital Precision Cancer Medicine Flagship, Helsinki, Finland
| | - Mikael Lundin
- Institute for Molecular Medicine Finland (FIMM), Nordic EMBL Partnership for Molecular Medicine, University of Helsinki, Helsinki, Finland
| | - Stig Nordling
- Department of Pathology, Medicum, University of Helsinki, Helsinki, Finland
| | - Harri Sihto
- Department of Pathology, Medicum, University of Helsinki, Helsinki, Finland
| | - Jorma Isola
- Department of Cancer Biology, BioMediTech, University of Tampere, Tampere, Finland
| | | | | | | | - Heikki Joensuu
- iCAN Digital Precision Cancer Medicine Flagship, Helsinki, Finland.,Department of Oncology, Helsinki University Hospital and University of Helsinki, Helsinki, Finland
| | - Johan Lundin
- Institute for Molecular Medicine Finland (FIMM), Nordic EMBL Partnership for Molecular Medicine, University of Helsinki, Helsinki, Finland.,iCAN Digital Precision Cancer Medicine Flagship, Helsinki, Finland.,Department of Global Public Health, Karolinska Institutet, Stockholm, Sweden
| |
Collapse
|
41
|
Krijgsman D, van Leeuwen MB, van der Ven J, Almeida V, Vlutters R, Halter D, Kuppen PJK, van de Velde CJH, Wimberger-Friedl R. Quantitative Whole Slide Assessment of Tumor-Infiltrating CD8-Positive Lymphocytes in ER-Positive Breast Cancer in Relation to Clinical Outcome. IEEE J Biomed Health Inform 2021; 25:381-392. [DOI: 10.1109/jbhi.2020.3003475] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
42
|
Esteva A, Chou K, Yeung S, Naik N, Madani A, Mottaghi A, Liu Y, Topol E, Dean J, Socher R. Deep learning-enabled medical computer vision. NPJ Digit Med 2021; 4:5. [PMID: 33420381 PMCID: PMC7794558 DOI: 10.1038/s41746-020-00376-2] [Citation(s) in RCA: 227] [Impact Index Per Article: 75.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2020] [Accepted: 12/01/2020] [Indexed: 02/07/2023] Open
Abstract
A decade of unprecedented progress in artificial intelligence (AI) has demonstrated the potential for many fields-including medicine-to benefit from the insights that AI techniques can extract from data. Here we survey recent progress in the development of modern computer vision techniques-powered by deep learning-for medical applications, focusing on medical imaging, medical video, and clinical deployment. We start by briefly summarizing a decade of progress in convolutional neural networks, including the vision tasks they enable, in the context of healthcare. Next, we discuss several example medical imaging applications that stand to benefit-including cardiology, pathology, dermatology, ophthalmology-and propose new avenues for continued work. We then expand into general medical video, highlighting ways in which clinical workflows can integrate computer vision to enhance care. Finally, we discuss the challenges and hurdles required for real-world clinical deployment of these technologies.
Collapse
Affiliation(s)
| | | | | | - Nikhil Naik
- Salesforce AI Research, San Francisco, CA, USA
| | - Ali Madani
- Salesforce AI Research, San Francisco, CA, USA
| | | | - Yun Liu
- Google Research, Mountain View, CA, USA
| | - Eric Topol
- Scripps Research Translational Institute, La Jolla, CA, USA
| | - Jeff Dean
- Google Research, Mountain View, CA, USA
| | | |
Collapse
|