1
|
Hosseini MS, Bejnordi BE, Trinh VQH, Chan L, Hasan D, Li X, Yang S, Kim T, Zhang H, Wu T, Chinniah K, Maghsoudlou S, Zhang R, Zhu J, Khaki S, Buin A, Chaji F, Salehi A, Nguyen BN, Samaras D, Plataniotis KN. Computational pathology: A survey review and the way forward. J Pathol Inform 2024; 15:100357. [PMID: 38420608 PMCID: PMC10900832 DOI: 10.1016/j.jpi.2023.100357] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2023] [Revised: 12/21/2023] [Accepted: 12/23/2023] [Indexed: 03/02/2024] Open
Abstract
Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.
Collapse
Affiliation(s)
- Mahdi S Hosseini
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | | | - Vincent Quoc-Huy Trinh
- Institute for Research in Immunology and Cancer of the University of Montreal, Montreal, QC H3T 1J4, Canada
| | - Lyndon Chan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Danial Hasan
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Xingwen Li
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Stephen Yang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Taehyo Kim
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Haochen Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Theodore Wu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Kajanan Chinniah
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Sina Maghsoudlou
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ryan Zhang
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Jiadai Zhu
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Samir Khaki
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Andrei Buin
- Huron Digitial Pathology, St. Jacobs, ON N0B 2N0, Canada
| | - Fatemeh Chaji
- Department of Computer Science and Software Engineering (CSSE), Concordia Univeristy, Montreal, QC H3H 2R9, Canada
| | - Ala Salehi
- Department of Electrical and Computer Engineering, University of New Brunswick, Fredericton, NB E3B 5A3, Canada
| | - Bich Ngoc Nguyen
- University of Montreal Hospital Center, Montreal, QC H2X 0C2, Canada
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, NY 11794, United States
| | - Konstantinos N Plataniotis
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 3G4, Canada
| |
Collapse
|
2
|
Wiedenmann M, Barch M, Chang PS, Giltnane J, Risom T, Zijlstra A. An immunofluorescence-guided segmentation model in H&E images is enabled by tissue artifact correction by CycleGAN. Mod Pathol 2024:100591. [PMID: 39147031 DOI: 10.1016/j.modpat.2024.100591] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2024] [Accepted: 08/01/2024] [Indexed: 08/17/2024]
Abstract
Despite recent advances, the adoption of computer vision methods into clinical and commercial applications has been hampered by the limited availability of accurate ground truth tissue annotations required to train robust supervised models. Generating such ground truth can be accelerated by annotating tissue molecularly using immunofluorescence staining (IF) and mapping these annotations to a post-IF H&E (terminal H&E). Mapping the annotations between the IF and the terminal H&E increases both the scale and accuracy by which ground truth could be generated. However, discrepancies between terminal H&E and conventional H&E caused by IF tissue processing have limited this implementation. We sought to overcome this challenge and achieve compatibility between these parallel modalities using synthetic image generation, in which a cycle-consistent generative adversarial network (CycleGAN) was applied to transfer the appearance of conventional H&E such that it emulates the terminal H&E. These synthetic emulations allowed us to train a deep learning (DL) model for the segmentation of epithelium in the terminal H&E that could be validated against the IF staining of epithelial-based cytokeratins. The combination of this segmentation model with the CycleGAN stain transfer model enabled performative epithelium segmentation in conventional H&E images. The approach demonstrates that the training of accurate segmentation models for the breadth of conventional H&E data can be executed free of human-expert annotations by leveraging molecular annotation strategies such as IF, so long as the tissue impacts of the molecular annotation protocol are captured by generative models that can be deployed prior to the segmentation process.
Collapse
Affiliation(s)
- Marcel Wiedenmann
- Department of Computer and Information Science, University of Konstanz, Universitätsstraße 10, 78464 Konstanz, Germany.
| | - Mariya Barch
- Department of Research Pathology, Genentech, Inc., 1 DNA Way, South San Francisco, 94080, CA, USA.
| | - Patrick S Chang
- Department of Research Pathology, Genentech, Inc., 1 DNA Way, South San Francisco, 94080, CA, USA.
| | - Jennifer Giltnane
- Department of Research Pathology, Genentech, Inc., 1 DNA Way, South San Francisco, 94080, CA, USA.
| | - Tyler Risom
- Department of Research Pathology, Genentech, Inc., 1 DNA Way, South San Francisco, 94080, CA, USA.
| | - Andries Zijlstra
- Department of Pathology, Microbiology and Immunology, Vanderbilt University Medical Center, Nashville, TN, USA.
| |
Collapse
|
3
|
Feng Y, Ding H, Huang X, Zhang Y, Lu M, Zhang T, Wang H, Chen Y, Mao Q, Xia W, Chen B, Zhang Y, Chen C, Gu T, Xu L, Dong G, Jiang F. Deep learning-based detection and semi-quantitative model for spread through air spaces (STAS) in lung adenocarcinoma. NPJ Precis Oncol 2024; 8:173. [PMID: 39103596 DOI: 10.1038/s41698-024-00664-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2023] [Accepted: 07/25/2024] [Indexed: 08/07/2024] Open
Abstract
Tumor spread through air spaces (STAS) is a distinctive metastatic pattern affecting prognosis in lung adenocarcinoma (LUAD) patients. Several challenges are associated with STAS detection, including misdetection, low interobserver agreement, and lack of quantitative analysis. In this research, a total of 489 digital whole slide images (WSIs) were collected. The deep learning-based STAS detection model, named STASNet, was constructed to calculate semi-quantitative parameters associated with STAS density and distance. STASNet demonstrated an accuracy of 0.93 for STAS detection at the tiles level and had an AUC of 0.72-0.78 for determining the STAS status at the WSI level. Among the semi-quantitative parameters, T10S, combined with the spatial location information, significantly stratified stage I LUAD patients on disease-free survival. Additionally, STASNet was deployed into a real-time pathological diagnostic environment, which boosted the STAS detection rate and led to the identification of three easily misidentified types of occult STAS.
Collapse
Affiliation(s)
- Yipeng Feng
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China
| | - Hanlin Ding
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China
| | - Xing Huang
- Pathological Department of Jiangsu Cancer Hospital, Nanjing, P. R. China
| | - Yijian Zhang
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China
| | - Mengyi Lu
- Department of Biostatistics, School of Public Health, Nanjing Medical University, 101 Longmian Avenue, Nanjing, 211166, Jiangsu, China
| | - Te Zhang
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China
| | - Hui Wang
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China
| | - Yuzhong Chen
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China
| | - Qixing Mao
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
| | - Wenjie Xia
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
| | - Bing Chen
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
| | - Yi Zhang
- Pathological Department of Jiangsu Cancer Hospital, Nanjing, P. R. China
| | - Chen Chen
- School of Control Science and Engineering, Shandong University, Jinan, 250061, China
| | - Tianhao Gu
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China
| | - Lin Xu
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China
- Collaborative Innovation Center for Cancer Personalized Medicine, Nanjing Medical University, Nanjing, China
| | - Gaochao Dong
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China.
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China.
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China.
| | - Feng Jiang
- Department of Thoracic Surgery, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, 21009, Nanjing, China.
- Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Cancer Institute of Jiangsu Province, Nanjing, China.
- The Fourth Clinical College of Nanjing Medical University, Nanjing, China.
| |
Collapse
|
4
|
Miyahira AK, Kamran SC, Jamaspishvili T, Marshall CH, Maxwell KN, Parolia A, Zorko NA, Pienta KJ, Soule HR. Disrupting prostate cancer research: Challenge accepted; report from the 2023 Coffey-Holden Prostate Cancer Academy Meeting. Prostate 2024; 84:993-1015. [PMID: 38682886 DOI: 10.1002/pros.24721] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/12/2024] [Accepted: 04/16/2024] [Indexed: 05/01/2024]
Abstract
INTRODUCTION The 2023 Coffey-Holden Prostate Cancer Academy (CHPCA) Meeting, themed "Disrupting Prostate Cancer Research: Challenge Accepted," was convened at the University of California, Los Angeles, Luskin Conference Center, in Los Angeles, CA, from June 22 to 25, 2023. METHODS The 2023 marked the 10th Annual CHPCA Meeting, a discussion-oriented scientific think-tank conference convened annually by the Prostate Cancer Foundation, which centers on innovative and emerging research topics deemed pivotal for advancing critical unmet needs in prostate cancer research and clinical care. The 2023 CHPCA Meeting was attended by 81 academic investigators and included 40 talks across 8 sessions. RESULTS The central topic areas covered at the meeting included: targeting transcription factor neo-enhancesomes in cancer, AR as a pro-differentiation and oncogenic transcription factor, why few are cured with androgen deprivation therapy and how to change dogma to cure metastatic prostate cancer without castration, reducing prostate cancer morbidity and mortality with genetics, opportunities for radiation to enhance therapeutic benefit in oligometastatic prostate cancer, novel immunotherapeutic approaches, and the new era of artificial intelligence-driven precision medicine. DISCUSSION This article provides an overview of the scientific presentations delivered at the 2023 CHPCA Meeting, such that this knowledge can help in facilitating the advancement of prostate cancer research worldwide.
Collapse
Affiliation(s)
- Andrea K Miyahira
- Science Department, Prostate Cancer Foundation, Santa Monica, California, USA
| | - Sophia C Kamran
- Department of Radiation Oncology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - Tamara Jamaspishvili
- Department of Pathology and Laboratory Medicine, SUNY Upstate Medical University, Syracuse, New York, USA
| | - Catherine H Marshall
- Department of Oncology, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| | - Kara N Maxwell
- Department of Medicine-Hematology/Oncology and Department of Genetics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Medicine Service, Corporal Michael J. Crescenz VA Medical Center, Philadelphia, Pennsylvania, USA
| | - Abhijit Parolia
- Department of Pathology, Rogel Cancer Center, University of Michigan, Ann Arbor, Michigan, USA
| | - Nicholas A Zorko
- Division of Hematology, Oncology and Transplantation, Department of Medicine, University of Minnesota, Minneapolis, Minnesota, USA
- University of Minnesota Masonic Cancer Center, University of Minnesota, Minneapolis, Minnesota, USA
| | - Kenneth J Pienta
- The James Buchanan Brady Urological Institute, The Johns Hopkins School of Medicine, Baltimore, Maryland, USA
| | - Howard R Soule
- Science Department, Prostate Cancer Foundation, Santa Monica, California, USA
| |
Collapse
|
5
|
Asadi-Aghbolaghi M, Darbandsari A, Zhang A, Contreras-Sanz A, Boschman J, Ahmadvand P, Köbel M, Farnell D, Huntsman DG, Churg A, Black PC, Wang G, Gilks CB, Farahani H, Bashashati A. Learning generalizable AI models for multi-center histopathology image classification. NPJ Precis Oncol 2024; 8:151. [PMID: 39030380 PMCID: PMC11271637 DOI: 10.1038/s41698-024-00652-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 07/11/2024] [Indexed: 07/21/2024] Open
Abstract
Investigation of histopathology slides by pathologists is an indispensable component of the routine diagnosis of cancer. Artificial intelligence (AI) has the potential to enhance diagnostic accuracy, improve efficiency, and patient outcomes in clinical pathology. However, variations in tissue preparation, staining protocols, and histopathology slide digitization could result in over-fitting of deep learning models when trained on the data from only one center, thereby underscoring the necessity to generalize deep learning networks for multi-center use. Several techniques, including the use of grayscale images, color normalization techniques, and Adversarial Domain Adaptation (ADA) have been suggested to generalize deep learning algorithms, but there are limitations to their effectiveness and discriminability. Convolutional Neural Networks (CNNs) exhibit higher sensitivity to variations in the amplitude spectrum, whereas humans predominantly rely on phase-related components for object recognition. As such, we propose Adversarial fourIer-based Domain Adaptation (AIDA) which applies the advantages of a Fourier transform in adversarial domain adaptation. We conducted a comprehensive examination of subtype classification tasks in four cancers, incorporating cases from multiple medical centers. Specifically, the datasets included multi-center data for 1113 ovarian cancer cases, 247 pleural cancer cases, 422 bladder cancer cases, and 482 breast cancer cases. Our proposed approach significantly improved performance, achieving superior classification results in the target domain, surpassing the baseline, color augmentation and normalization techniques, and ADA. Furthermore, extensive pathologist reviews suggested that our proposed approach, AIDA, successfully identifies known histotype-specific features. This superior performance highlights AIDA's potential in addressing generalization challenges in deep learning models for multi-center histopathology datasets.
Collapse
Affiliation(s)
| | - Amirali Darbandsari
- Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, BC, Canada
| | - Allen Zhang
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, BC, Canada
- Vancouver General Hospital, Vancouver, BC, Canada
| | | | - Jeffrey Boschman
- School of Biomedical Engineering, University of British Columbia, Vancouver, BC, Canada
| | - Pouya Ahmadvand
- School of Biomedical Engineering, University of British Columbia, Vancouver, BC, Canada
| | - Martin Köbel
- Department of Pathology and Laboratory Medicine, University of Calgary, Calgary, AB, Canada
| | - David Farnell
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, BC, Canada
- Vancouver General Hospital, Vancouver, BC, Canada
| | - David G Huntsman
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, BC, Canada
- BC Cancer Research Institute, Vancouver, BC, Canada
| | - Andrew Churg
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, BC, Canada
- Vancouver General Hospital, Vancouver, BC, Canada
| | - Peter C Black
- Department of Urologic Sciences, University of British Columbia, Vancouver, BC, Canada
| | - Gang Wang
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, BC, Canada
| | - C Blake Gilks
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, BC, Canada
- Vancouver General Hospital, Vancouver, BC, Canada
| | - Hossein Farahani
- School of Biomedical Engineering, University of British Columbia, Vancouver, BC, Canada
| | - Ali Bashashati
- School of Biomedical Engineering, University of British Columbia, Vancouver, BC, Canada.
- Department of Pathology and Laboratory Medicine, University of British Columbia, Vancouver, BC, Canada.
| |
Collapse
|
6
|
Baldeon-Calisto M, Rivera-Velastegui F, Lai-Yuen SK, Riofrío D, Pérez-Pérez N, Benítez D, Flores-Moyano R. DistilIQA: Distilling Vision Transformers for no-reference perceptual CT image quality assessment. Comput Biol Med 2024; 177:108670. [PMID: 38838558 DOI: 10.1016/j.compbiomed.2024.108670] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2023] [Revised: 04/25/2024] [Accepted: 05/26/2024] [Indexed: 06/07/2024]
Abstract
No-reference image quality assessment (IQA) is a critical step in medical image analysis, with the objective of predicting perceptual image quality without the need for a pristine reference image. The application of no-reference IQA to CT scans is valuable in providing an automated and objective approach to assessing scan quality, optimizing radiation dose, and improving overall healthcare efficiency. In this paper, we introduce DistilIQA, a novel distilled Vision Transformer network designed for no-reference CT image quality assessment. DistilIQA integrates convolutional operations and multi-head self-attention mechanisms by incorporating a powerful convolutional stem at the beginning of the traditional ViT network. Additionally, we present a two-step distillation methodology aimed at improving network performance and efficiency. In the initial step, a "teacher ensemble network" is constructed by training five vision Transformer networks using a five-fold division schema. In the second step, a "student network", comprising of a single Vision Transformer, is trained using the original labeled dataset and the predictions generated by the teacher network as new labels. DistilIQA is evaluated in the task of quality score prediction from low-dose chest CT scans obtained from the LDCT and Projection data of the Cancer Imaging Archive, along with low-dose abdominal CT images from the LDCTIQAC2023 Grand Challenge. Our results demonstrate DistilIQA's remarkable performance in both benchmarks, surpassing the capabilities of various CNNs and Transformer architectures. Moreover, our comprehensive experimental analysis demonstrates the effectiveness of incorporating convolutional operations within the ViT architecture and highlights the advantages of our distillation methodology.
Collapse
Affiliation(s)
- Maria Baldeon-Calisto
- Departamento de Ingeniería Industrial and Instituto de Innovación en Productividad y Logística CATENA-USFQ, Universidad San Francisco de Quito USFQ, Quito, 170157, Ecuador; Colegio de Ciencias e Ingenierías "El Politécnico", Universidad San Francisco de Quito USFQ, Quito, 170157, Ecuador.
| | | | - Susana K Lai-Yuen
- Department of Industrial and Management Systems Engineering, University of South Florida, Tampa, 33620, FL, USA.
| | - Daniel Riofrío
- Colegio de Ciencias e Ingenierías "El Politécnico", Universidad San Francisco de Quito USFQ, Quito, 170157, Ecuador.
| | - Noel Pérez-Pérez
- Colegio de Ciencias e Ingenierías "El Politécnico", Universidad San Francisco de Quito USFQ, Quito, 170157, Ecuador.
| | - Diego Benítez
- Colegio de Ciencias e Ingenierías "El Politécnico", Universidad San Francisco de Quito USFQ, Quito, 170157, Ecuador.
| | - Ricardo Flores-Moyano
- Colegio de Ciencias e Ingenierías "El Politécnico", Universidad San Francisco de Quito USFQ, Quito, 170157, Ecuador.
| |
Collapse
|
7
|
Xu H, Usuyama N, Bagga J, Zhang S, Rao R, Naumann T, Wong C, Gero Z, González J, Gu Y, Xu Y, Wei M, Wang W, Ma S, Wei F, Yang J, Li C, Gao J, Rosemon J, Bower T, Lee S, Weerasinghe R, Wright BJ, Robicsek A, Piening B, Bifulco C, Wang S, Poon H. A whole-slide foundation model for digital pathology from real-world data. Nature 2024; 630:181-188. [PMID: 38778098 PMCID: PMC11153137 DOI: 10.1038/s41586-024-07441-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Accepted: 04/19/2024] [Indexed: 05/25/2024]
Abstract
Digital pathology poses unique computational challenges, as a standard gigapixel slide may comprise tens of thousands of image tiles1-3. Prior models have often resorted to subsampling a small portion of tiles for each slide, thus missing the important slide-level context4. Here we present Prov-GigaPath, a whole-slide pathology foundation model pretrained on 1.3 billion 256 × 256 pathology image tiles in 171,189 whole slides from Providence, a large US health network comprising 28 cancer centres. The slides originated from more than 30,000 patients covering 31 major tissue types. To pretrain Prov-GigaPath, we propose GigaPath, a novel vision transformer architecture for pretraining gigapixel pathology slides. To scale GigaPath for slide-level learning with tens of thousands of image tiles, GigaPath adapts the newly developed LongNet5 method to digital pathology. To evaluate Prov-GigaPath, we construct a digital pathology benchmark comprising 9 cancer subtyping tasks and 17 pathomics tasks, using both Providence and TCGA data6. With large-scale pretraining and ultra-large-context modelling, Prov-GigaPath attains state-of-the-art performance on 25 out of 26 tasks, with significant improvement over the second-best method on 18 tasks. We further demonstrate the potential of Prov-GigaPath on vision-language pretraining for pathology7,8 by incorporating the pathology reports. In sum, Prov-GigaPath is an open-weight foundation model that achieves state-of-the-art performance on various digital pathology tasks, demonstrating the importance of real-world data and whole-slide modelling.
Collapse
Affiliation(s)
- Hanwen Xu
- Microsoft Research, Redmond, WA, USA
- Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA
| | | | | | | | | | | | | | | | | | - Yu Gu
- Microsoft Research, Redmond, WA, USA
| | - Yanbo Xu
- Microsoft Research, Redmond, WA, USA
| | - Mu Wei
- Microsoft Research, Redmond, WA, USA
| | | | | | - Furu Wei
- Microsoft Research, Redmond, WA, USA
| | | | | | | | | | | | - Soohee Lee
- Providence Research Network, Renton, WA, USA
| | | | | | | | - Brian Piening
- Providence Genomics, Portland, OR, USA
- Earle A. Chiles Research Institute, Providence Cancer Institute, Portland, OR, USA
| | - Carlo Bifulco
- Providence Genomics, Portland, OR, USA.
- Earle A. Chiles Research Institute, Providence Cancer Institute, Portland, OR, USA.
| | - Sheng Wang
- Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA.
- Department of Surgery, University of Washington, Seattle, WA, USA.
| | | |
Collapse
|
8
|
Aubreville M, Stathonikos N, Donovan TA, Klopfleisch R, Ammeling J, Ganz J, Wilm F, Veta M, Jabari S, Eckstein M, Annuscheit J, Krumnow C, Bozaba E, Çayır S, Gu H, Chen X'A, Jahanifar M, Shephard A, Kondo S, Kasai S, Kotte S, Saipradeep VG, Lafarge MW, Koelzer VH, Wang Z, Zhang Y, Yang S, Wang X, Breininger K, Bertram CA. Domain generalization across tumor types, laboratories, and species - Insights from the 2022 edition of the Mitosis Domain Generalization Challenge. Med Image Anal 2024; 94:103155. [PMID: 38537415 DOI: 10.1016/j.media.2024.103155] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2023] [Revised: 01/19/2024] [Accepted: 03/20/2024] [Indexed: 04/16/2024]
Abstract
Recognition of mitotic figures in histologic tumor specimens is highly relevant to patient outcome assessment. This task is challenging for algorithms and human experts alike, with deterioration of algorithmic performance under shifts in image representations. Considerable covariate shifts occur when assessment is performed on different tumor types, images are acquired using different digitization devices, or specimens are produced in different laboratories. This observation motivated the inception of the 2022 challenge on MItosis Domain Generalization (MIDOG 2022). The challenge provided annotated histologic tumor images from six different domains and evaluated the algorithmic approaches for mitotic figure detection provided by nine challenge participants on ten independent domains. Ground truth for mitotic figure detection was established in two ways: a three-expert majority vote and an independent, immunohistochemistry-assisted set of labels. This work represents an overview of the challenge tasks, the algorithmic strategies employed by the participants, and potential factors contributing to their success. With an F1 score of 0.764 for the top-performing team, we summarize that domain generalization across various tumor domains is possible with today's deep learning-based recognition pipelines. However, we also found that domain characteristics not present in the training set (feline as new species, spindle cell shape as new morphology and a new scanner) led to small but significant decreases in performance. When assessed against the immunohistochemistry-assisted reference standard, all methods resulted in reduced recall scores, with only minor changes in the order of participants in the ranking.
Collapse
Affiliation(s)
| | | | - Taryn A Donovan
- Department of Anatomic Pathology, The Schwarzman Animal Medical Center, NY, USA
| | - Robert Klopfleisch
- Institute of Veterinary Pathology, Freie Universität Berlin, Berlin, Germany
| | | | - Jonathan Ganz
- Technische Hochschule Ingolstadt, Ingolstadt, Germany
| | - Frauke Wilm
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany; Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Mitko Veta
- Computational Pathology Group, Radboud UMC Nijmegen, The Netherlands
| | - Samir Jabari
- Institute of Neuropathology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Markus Eckstein
- Institute of Pathology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nünberg, Erlangen, Germany
| | | | | | - Engin Bozaba
- Artificial Intelligence Research Team, Virasoft Corporation, NY, USA
| | - Sercan Çayır
- Artificial Intelligence Research Team, Virasoft Corporation, NY, USA
| | - Hongyan Gu
- University of California, Los Angeles, USA
| | | | | | | | | | - Satoshi Kasai
- Niigata University of Health and Welfare, Niigata, Japan
| | - Sujatha Kotte
- TCS Research, Tata Consultancy Services Ltd, Hyderabad, India
| | - V G Saipradeep
- TCS Research, Tata Consultancy Services Ltd, Hyderabad, India
| | - Maxime W Lafarge
- Department of Pathology and Molecular Pathology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Viktor H Koelzer
- Department of Pathology and Molecular Pathology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Ziyue Wang
- Harbin Institute of Technology, Shenzhen, China
| | | | - Sen Yang
- College of Biomedical Engineering, Sichuan University, Chengdu, China
| | - Xiyue Wang
- Department of Radiation Oncology, Stanford University School of Medicine, Palo Alto, USA
| | - Katharina Breininger
- Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Christof A Bertram
- Institute of Pathology, University of Veterinary Medicine, Vienna, Austria
| |
Collapse
|
9
|
Jahanifar M, Shephard A, Zamanitajeddin N, Graham S, Raza SEA, Minhas F, Rajpoot N. Mitosis detection, fast and slow: Robust and efficient detection of mitotic figures. Med Image Anal 2024; 94:103132. [PMID: 38442527 DOI: 10.1016/j.media.2024.103132] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2022] [Revised: 02/28/2024] [Accepted: 03/01/2024] [Indexed: 03/07/2024]
Abstract
Counting of mitotic figures is a fundamental step in grading and prognostication of several cancers. However, manual mitosis counting is tedious and time-consuming. In addition, variation in the appearance of mitotic figures causes a high degree of discordance among pathologists. With advances in deep learning models, several automatic mitosis detection algorithms have been proposed but they are sensitive to domain shift often seen in histology images. We propose a robust and efficient two-stage mitosis detection framework, which comprises mitosis candidate segmentation (Detecting Fast) and candidate refinement (Detecting Slow) stages. The proposed candidate segmentation model, termed EUNet, is fast and accurate due to its architectural design. EUNet can precisely segment candidates at a lower resolution to considerably speed up candidate detection. Candidates are then refined using a deeper classifier network, EfficientNet-B7, in the second stage. We make sure both stages are robust against domain shift by incorporating domain generalization methods. We demonstrate state-of-the-art performance and generalizability of the proposed model on the three largest publicly available mitosis datasets, winning the two mitosis domain generalization challenge contests (MIDOG21 and MIDOG22). Finally, we showcase the utility of the proposed algorithm by processing the TCGA breast cancer cohort (1,124 whole-slide images) to generate and release a repository of more than 620K potential mitotic figures (not exhaustively validated).
Collapse
Affiliation(s)
- Mostafa Jahanifar
- Tissue Image Analytic (TIA) Center, Department of Computer Science, University of Warwick, UK.
| | - Adam Shephard
- Tissue Image Analytic (TIA) Center, Department of Computer Science, University of Warwick, UK
| | - Neda Zamanitajeddin
- Tissue Image Analytic (TIA) Center, Department of Computer Science, University of Warwick, UK
| | - Simon Graham
- Tissue Image Analytic (TIA) Center, Department of Computer Science, University of Warwick, UK; Histofy Ltd, Birmingham, UK
| | - Shan E Ahmed Raza
- Tissue Image Analytic (TIA) Center, Department of Computer Science, University of Warwick, UK
| | - Fayyaz Minhas
- Tissue Image Analytic (TIA) Center, Department of Computer Science, University of Warwick, UK
| | - Nasir Rajpoot
- Tissue Image Analytic (TIA) Center, Department of Computer Science, University of Warwick, UK; Histofy Ltd, Birmingham, UK.
| |
Collapse
|
10
|
Jin D, Liang S, Shmatko A, Arnold A, Horst D, Grünewald TGP, Gerstung M, Bai X. Teacher-student collaborated multiple instance learning for pan-cancer PDL1 expression prediction from histopathology slides. Nat Commun 2024; 15:3063. [PMID: 38594278 PMCID: PMC11004138 DOI: 10.1038/s41467-024-46764-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2023] [Accepted: 03/08/2024] [Indexed: 04/11/2024] Open
Abstract
Programmed cell death ligand 1 (PDL1), as an important biomarker, is quantified by immunohistochemistry (IHC) with few established histopathological patterns. Deep learning aids in histopathological assessment, yet heterogeneity and lacking spatially resolved annotations challenge precise analysis. Here, we present a weakly supervised learning approach using bulk RNA sequencing for PDL1 expression prediction from hematoxylin and eosin (H&E) slides. Our method extends the multiple instance learning paradigm with the teacher-student framework, which assigns dynamic pseudo-labels for intra-slide heterogeneity and retrieves unlabeled instances using temporal ensemble model distillation. The approach, evaluated on 12,299 slides across 20 solid tumor types, achieves a weighted average area under the curve of 0.83 on fresh-frozen and 0.74 on formalin-fixed specimens for 9 tumors with PDL1 as an established biomarker. Our method predicts PDL1 expression patterns, validated by IHC on 20 slides, offering insights into histologies relevant to PDL1. This demonstrates the potential of deep learning in identifying diverse histological patterns for molecular changes from H&E images.
Collapse
Affiliation(s)
- Darui Jin
- Image Processing Center, Beihang University, Beijing, 102206, China
- Division of AI in Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Shen Yuan Honors College, Beihang University, Beijing, 100191, China
| | - Shangying Liang
- Image Processing Center, Beihang University, Beijing, 102206, China
| | - Artem Shmatko
- Division of AI in Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alexander Arnold
- Charité - Universitätsmedizin Berlin, Institute of Pathology, 10117, Berlin, Germany
| | - David Horst
- Charité - Universitätsmedizin Berlin, Institute of Pathology, 10117, Berlin, Germany
- German Cancer Consortium (DKTK), partner site Berlin, a partnership between DKFZ and Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Thomas G P Grünewald
- Institute of Pathology, Heidelberg University Hospital, Heidelberg, Germany.
- Division of Translational Pediatric Sarcoma Research, German Cancer Research Center (DKFZ), German Cancer Consortium (DKTK), Heidelberg, Germany.
- Hopp Children's Cancer Center (KiTZ) Heidelberg, Heidelberg, Germany.
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and Heidelberg University Hospital, Heidelberg, Germany.
| | - Moritz Gerstung
- Division of AI in Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany.
| | - Xiangzhi Bai
- Image Processing Center, Beihang University, Beijing, 102206, China.
- State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, 100191, China.
- Advanced Innovation Center for Biomedical Engineering, Beihang University, Beijing, 100083, China.
| |
Collapse
|
11
|
Leon-Ferre RA, Carter JM, Zahrieh D, Sinnwell JP, Salgado R, Suman VJ, Hillman DW, Boughey JC, Kalari KR, Couch FJ, Ingle JN, Balkenhol M, Ciompi F, van der Laak J, Goetz MP. Automated mitotic spindle hotspot counts are highly associated with clinical outcomes in systemically untreated early-stage triple-negative breast cancer. NPJ Breast Cancer 2024; 10:25. [PMID: 38553444 PMCID: PMC10980681 DOI: 10.1038/s41523-024-00629-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Accepted: 03/08/2024] [Indexed: 04/02/2024] Open
Abstract
Operable triple-negative breast cancer (TNBC) has a higher risk of recurrence and death compared to other subtypes. Tumor size and nodal status are the primary clinical factors used to guide systemic treatment, while biomarkers of proliferation have not demonstrated value. Recent studies suggest that subsets of TNBC have a favorable prognosis, even without systemic therapy. We evaluated the association of fully automated mitotic spindle hotspot (AMSH) counts with recurrence-free (RFS) and overall survival (OS) in two separate cohorts of patients with early-stage TNBC who did not receive systemic therapy. AMSH counts were obtained from areas with the highest mitotic density in digitized whole slide images processed with a convolutional neural network trained to detect mitoses. In 140 patients from the Mayo Clinic TNBC cohort, AMSH counts were significantly associated with RFS and OS in a multivariable model controlling for nodal status, tumor size, and tumor-infiltrating lymphocytes (TILs) (p < 0.0001). For every 10-point increase in AMSH counts, there was a 16% increase in the risk of an RFS event (HR 1.16, 95% CI 1.08-1.25), and a 7% increase in the risk of death (HR 1.07, 95% CI 1.00-1.14). We corroborated these findings in a separate cohort of systemically untreated TNBC patients from Radboud UMC in the Netherlands. Our findings suggest that AMSH counts offer valuable prognostic information in patients with early-stage TNBC who did not receive systemic therapy, independent of tumor size, nodal status, and TILs. If further validated, AMSH counts could help inform future systemic therapy de-escalation strategies.
Collapse
Affiliation(s)
| | | | | | | | - Roberto Salgado
- GZA-ZNA-Hospitals, Antwerp, Belgium
- Peter Mac Callum Cancer Centre, Melbourne, Australia
| | | | | | | | | | | | | | | | | | - Jeroen van der Laak
- Radboud University Medical Center, Nijmegen, Netherlands
- Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | | |
Collapse
|
12
|
Lindroth H, Nalaie K, Raghu R, Ayala IN, Busch C, Bhattacharyya A, Moreno Franco P, Diedrich DA, Pickering BW, Herasevich V. Applied Artificial Intelligence in Healthcare: A Review of Computer Vision Technology Application in Hospital Settings. J Imaging 2024; 10:81. [PMID: 38667979 PMCID: PMC11050909 DOI: 10.3390/jimaging10040081] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Revised: 03/08/2024] [Accepted: 03/11/2024] [Indexed: 04/28/2024] Open
Abstract
Computer vision (CV), a type of artificial intelligence (AI) that uses digital videos or a sequence of images to recognize content, has been used extensively across industries in recent years. However, in the healthcare industry, its applications are limited by factors like privacy, safety, and ethical concerns. Despite this, CV has the potential to improve patient monitoring, and system efficiencies, while reducing workload. In contrast to previous reviews, we focus on the end-user applications of CV. First, we briefly review and categorize CV applications in other industries (job enhancement, surveillance and monitoring, automation, and augmented reality). We then review the developments of CV in the hospital setting, outpatient, and community settings. The recent advances in monitoring delirium, pain and sedation, patient deterioration, mechanical ventilation, mobility, patient safety, surgical applications, quantification of workload in the hospital, and monitoring for patient events outside the hospital are highlighted. To identify opportunities for future applications, we also completed journey mapping at different system levels. Lastly, we discuss the privacy, safety, and ethical considerations associated with CV and outline processes in algorithm development and testing that limit CV expansion in healthcare. This comprehensive review highlights CV applications and ideas for its expanded use in healthcare.
Collapse
Affiliation(s)
- Heidi Lindroth
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
- Center for Aging Research, Regenstrief Institute, School of Medicine, Indiana University, Indianapolis, IN 46202, USA
- Center for Health Innovation and Implementation Science, School of Medicine, Indiana University, Indianapolis, IN 46202, USA
| | - Keivan Nalaie
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA; (D.A.D.); (B.W.P.); (V.H.)
| | - Roshini Raghu
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
| | - Ivan N. Ayala
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
| | - Charles Busch
- Division of Nursing Research, Department of Nursing, Mayo Clinic, Rochester, MN 55905, USA; (K.N.); (R.R.); (I.N.A.); (C.B.)
- College of Engineering, University of Wisconsin-Madison, Madison, WI 53705, USA
| | | | - Pablo Moreno Franco
- Department of Transplantation Medicine, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Daniel A. Diedrich
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA; (D.A.D.); (B.W.P.); (V.H.)
| | - Brian W. Pickering
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA; (D.A.D.); (B.W.P.); (V.H.)
| | - Vitaly Herasevich
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN 55905, USA; (D.A.D.); (B.W.P.); (V.H.)
| |
Collapse
|
13
|
Dolezal JM, Kochanny S, Dyer E, Ramesh S, Srisuwananukorn A, Sacco M, Howard FM, Li A, Mohan P, Pearson AT. Slideflow: deep learning for digital histopathology with real-time whole-slide visualization. BMC Bioinformatics 2024; 25:134. [PMID: 38539070 PMCID: PMC10967068 DOI: 10.1186/s12859-024-05758-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2023] [Accepted: 03/20/2024] [Indexed: 05/04/2024] Open
Abstract
Deep learning methods have emerged as powerful tools for analyzing histopathological images, but current methods are often specialized for specific domains and software environments, and few open-source options exist for deploying models in an interactive interface. Experimenting with different deep learning approaches typically requires switching software libraries and reprocessing data, reducing the feasibility and practicality of experimenting with new architectures. We developed a flexible deep learning library for histopathology called Slideflow, a package which supports a broad array of deep learning methods for digital pathology and includes a fast whole-slide interface for deploying trained models. Slideflow includes unique tools for whole-slide image data processing, efficient stain normalization and augmentation, weakly-supervised whole-slide classification, uncertainty quantification, feature generation, feature space analysis, and explainability. Whole-slide image processing is highly optimized, enabling whole-slide tile extraction at 40x magnification in 2.5 s per slide. The framework-agnostic data processing pipeline enables rapid experimentation with new methods built with either Tensorflow or PyTorch, and the graphical user interface supports real-time visualization of slides, predictions, heatmaps, and feature space characteristics on a variety of hardware devices, including ARM-based devices such as the Raspberry Pi.
Collapse
Affiliation(s)
- James M Dolezal
- Section of Hematology/Oncology, Department of Medicine, University of Chicago Medical Center, Chicago, IL, USA.
| | - Sara Kochanny
- Section of Hematology/Oncology, Department of Medicine, University of Chicago Medical Center, Chicago, IL, USA
| | - Emma Dyer
- Section of Hematology/Oncology, Department of Medicine, University of Chicago Medical Center, Chicago, IL, USA
| | - Siddhi Ramesh
- Section of Hematology/Oncology, Department of Medicine, University of Chicago Medical Center, Chicago, IL, USA
| | - Andrew Srisuwananukorn
- Division of Hematology, Department of Internal Medicine, The Ohio State University Comprehensive Cancer Center, Columbus, OH, USA
| | - Matteo Sacco
- Section of Hematology/Oncology, Department of Medicine, University of Chicago Medical Center, Chicago, IL, USA
| | - Frederick M Howard
- Section of Hematology/Oncology, Department of Medicine, University of Chicago Medical Center, Chicago, IL, USA
| | - Anran Li
- Section of Hematology/Oncology, Department of Medicine, University of Chicago Medical Center, Chicago, IL, USA
| | - Prajval Mohan
- Department of Computer Science, University of Chicago, Chicago, IL, USA
| | - Alexander T Pearson
- Section of Hematology/Oncology, Department of Medicine, University of Chicago Medical Center, Chicago, IL, USA.
| |
Collapse
|
14
|
Fernandez-Martín C, Silva-Rodriguez J, Kiraz U, Morales S, Janssen EAM, Naranjo V. Uninformed Teacher-Student for hard-samples distillation in weakly supervised mitosis localization. Comput Med Imaging Graph 2024; 112:102328. [PMID: 38244279 DOI: 10.1016/j.compmedimag.2024.102328] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2023] [Revised: 11/02/2023] [Accepted: 12/12/2023] [Indexed: 01/22/2024]
Abstract
BACKGROUND AND OBJECTIVE Mitotic activity is a crucial biomarker for diagnosing and predicting outcomes for different types of cancers, particularly breast cancer. However, manual mitosis counting is challenging and time-consuming for pathologists, with moderate reproducibility due to biopsy slide size, low mitotic cell density, and pattern heterogeneity. In recent years, deep learning methods based on convolutional neural networks (CNNs) have been proposed to address these limitations. Nonetheless, these methods have been hampered by the available data labels, which usually consist only of the centroids of mitosis, and by the incoming noise from annotated hard negatives. As a result, complex algorithms with multiple stages are often required to refine the labels at the pixel level and reduce the number of false positives. METHODS This article presents a novel weakly supervised approach for mitosis detection that utilizes only image-level labels on histological hematoxylin and eosin (H&E) images, avoiding the need for complex labeling scenarios. Also, an Uninformed Teacher-Student (UTS) pipeline is introduced to detect and distill hard samples by comparing weakly supervised localizations and the annotated centroids, using strong augmentations to enhance uncertainty. Additionally, an automatic proliferation score is proposed that mimicks the pathologist-annotated mitotic activity index (MAI). The proposed approach is evaluated on three publicly available datasets for mitosis detection on breast histology samples, and two datasets for mitotic activity counting in whole-slide images. RESULTS The proposed framework achieves competitive performance with relevant prior literature in all the datasets used for evaluation without explicitly using the mitosis location information during training. This approach challenges previous methods that rely on strong mitosis location information and multiple stages to refine false positives. Furthermore, the proposed pipeline for hard-sample distillation demonstrates promising dataset-specific improvements. Concretely, when the annotation has not been thoroughly refined by multiple pathologists, the UTS model offers improvements of up to ∼4% in mitosis localization, thanks to the detection and distillation of uncertain cases. Concerning the mitosis counting task, the proposed automatic proliferation score shows a moderate positive correlation with the MAI annotated by pathologists at the biopsy level on two external datasets. CONCLUSIONS The proposed Uninformed Teacher-Student pipeline leverages strong augmentations to distill uncertain samples and measure dissimilarities between predicted and annotated mitosis. Results demonstrate the feasibility of the weakly supervised approach and highlight its potential as an objective evaluation tool for tumor proliferation.
Collapse
Affiliation(s)
- Claudio Fernandez-Martín
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-tech, Universitat Politècnica de València, Valencia, Spain.
| | | | - Umay Kiraz
- Department of Chemistry, Bioscience and Environmental Engineering, University of Stavanger, Stavanger, Norway; Department of Pathology, Stavanger University Hospital, Stavanger, Norway
| | - Sandra Morales
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-tech, Universitat Politècnica de València, Valencia, Spain
| | - Emiel A M Janssen
- Department of Chemistry, Bioscience and Environmental Engineering, University of Stavanger, Stavanger, Norway; Department of Pathology, Stavanger University Hospital, Stavanger, Norway
| | - Valery Naranjo
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-tech, Universitat Politècnica de València, Valencia, Spain
| |
Collapse
|
15
|
Franchet C, Schwob R, Bataillon G, Syrykh C, Péricart S, Frenois FX, Penault-Llorca F, Lacroix-Triki M, Arnould L, Lemonnier J, Alliot JM, Filleron T, Brousset P. Bias reduction using combined stain normalization and augmentation for AI-based classification of histological images. Comput Biol Med 2024; 171:108130. [PMID: 38387381 DOI: 10.1016/j.compbiomed.2024.108130] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Revised: 02/07/2024] [Accepted: 02/07/2024] [Indexed: 02/24/2024]
Abstract
Artificial intelligence (AI)-assisted diagnosis is an ongoing revolution in pathology. However, a frequent drawback of AI models is their propension to make decisions based rather on bias in training dataset than on concrete biological features, thus weakening pathologists' trust in these tools. Technically, it is well known that microscopic images are altered by tissue processing and staining procedures, being one of the main sources of bias in machine learning for digital pathology. So as to deal with it, many teams have written about color normalization and augmentation methods. However, only a few of them have monitored their effects on bias reduction and model generalizability. In our study, two methods for stain augmentation (AugmentHE) and fast normalization (HEnorm) have been created and their effect on bias reduction has been monitored. Actually, they have also been compared to previously described strategies. To that end, a multicenter dataset created for breast cancer histological grading has been used. Thanks to it, classification models have been trained in a single center before assessing its performance in other centers images. This setting led to extensively monitor bias reduction while providing accurate insight of both augmentation and normalization methods. AugmentHE provided an 81% increase in color dispersion compared to geometric augmentations only. In addition, every classification model that involved AugmentHE presented a significant increase in the area under receiving operator characteristic curve (AUC) over the widely used RGB shift. More precisely, AugmentHE-based models showed at least 0.14 AUC increase over RGB shift-based models. Regarding normalization, HEnorm appeared to be up to 78x faster than conventional methods. It also provided satisfying results in terms of bias reduction. Altogether, our pipeline composed of AugmentHE and HEnorm improved AUC on biased data by up to 21.7% compared to usual augmentations. Conventional normalization methods coupled with AugmentHE yielded similar results while being much slower. In conclusion, we have validated an open-source tool that can be used in any deep learning-based digital pathology project on H&E whole slide images (WSI) that efficiently reduces stain-induced bias and later on might help increase pathologists' confidence when using AI-based products.
Collapse
Affiliation(s)
- Camille Franchet
- Institut Universitaire du Cancer-Oncopole, Pathology Department, F-31059 Toulouse, France; Université Toulouse III-Paul Sabatier, F-31000 Toulouse, France.
| | - Robin Schwob
- Thales Services Numériques, Augmented data department, F-31670 Labège, France
| | - Guillaume Bataillon
- Institut Universitaire du Cancer-Oncopole, Pathology Department, F-31059 Toulouse, France
| | - Charlotte Syrykh
- Institut Universitaire du Cancer-Oncopole, Pathology Department, F-31059 Toulouse, France
| | - Sarah Péricart
- Institut Universitaire du Cancer-Oncopole, Pathology Department, F-31059 Toulouse, France
| | | | - Frédérique Penault-Llorca
- Centre Jean Perrin, Université Clermont Auvergne, INSERM, U1240 Imagerie Moléculaire et Stratégies Théranostiques, F-63000 Clermont Ferrand, France
| | - Magali Lacroix-Triki
- Département de Pathologie, Gustave-Roussy Cancer Campus, F-94805 Villejuif, France
| | - Laurent Arnould
- Department of Pathology and Tumor Biology, Centre Georges François Leclerc, Dijon, France
| | | | - Jean-Marc Alliot
- Institut de Recherche en Informatique de Toulouse, Toulouse University, Toulouse, France; CHU Toulouse, Toulouse, France
| | - Thomas Filleron
- Institut Universitaire du Cancer-Oncopole, Institut Claudius Regaud, Biostatistics & Health Data Science Unit, F-31059 Toulouse, France
| | - Pierre Brousset
- Institut Universitaire du Cancer-Oncopole, Pathology Department, F-31059 Toulouse, France; Université Toulouse III-Paul Sabatier, F-31000 Toulouse, France; Laboratoire d'Excellence Toulouse Cancer-TOUCAN, UMR1037 CRCT, F-31037 Toulouse, France.
| |
Collapse
|
16
|
Faryna K, van der Laak J, Litjens G. Automatic data augmentation to improve generalization of deep learning in H&E stained histopathology. Comput Biol Med 2024; 170:108018. [PMID: 38281317 DOI: 10.1016/j.compbiomed.2024.108018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Revised: 01/03/2024] [Accepted: 01/22/2024] [Indexed: 01/30/2024]
Abstract
In histopathology practice, scanners, tissue processing, staining, and image acquisition protocols vary from center to center, resulting in subtle variations in images. Vanilla convolutional neural networks are sensitive to such domain shifts. Data augmentation is a popular way to improve domain generalization. Currently, state-of-the-art domain generalization in computational pathology is achieved using a manually curated set of augmentation transforms. However, manual tuning of augmentation parameters is time-consuming and can lead to sub-optimal generalization performance. Meta-learning frameworks can provide efficient ways to find optimal training hyper-parameters, including data augmentation. In this study, we hypothesize that an automated search of augmentation hyper-parameters can provide superior generalization performance and reduce experimental optimization time. We select four state-of-the-art automatic augmentation methods from general computer vision and investigate their capacity to improve domain generalization in histopathology. We analyze their performance on data from 25 centers across two different tasks: tumor metastasis detection in lymph nodes and breast cancer tissue type classification. On tumor metastasis detection, most automatic augmentation methods achieve comparable performance to state-of-the-art manual augmentation. On breast cancer tissue type classification, the leading automatic augmentation method significantly outperforms state-of-the-art manual data augmentation.
Collapse
Affiliation(s)
- Khrystyna Faryna
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA, Nijmegen, The Netherlands.
| | - Jeroen van der Laak
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA, Nijmegen, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, SE-581 83, Linköping, Sweden
| | - Geert Litjens
- Department of Pathology, Radboud Institute for Health Sciences, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA, Nijmegen, The Netherlands
| |
Collapse
|
17
|
Dunn C, Brettle D, Cockroft M, Keating E, Revie C, Treanor D. Quantitative assessment of H&E staining for pathology: development and clinical evaluation of a novel system. Diagn Pathol 2024; 19:42. [PMID: 38395890 PMCID: PMC10885446 DOI: 10.1186/s13000-024-01461-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Accepted: 02/03/2024] [Indexed: 02/25/2024] Open
Abstract
BACKGROUND Staining tissue samples to visualise cellular detail and tissue structure is at the core of pathology diagnosis, but variations in staining can result in significantly different appearances of the tissue sample. While the human visual system is adept at compensating for stain variation, with the growth of digital imaging in pathology, the impact of this variation can be more profound. Despite the ubiquity of haematoxylin and eosin staining in clinical practice worldwide, objective quantification is not yet available. We propose a method for quantitative haematoxylin and eosin stain assessment to facilitate quality assurance of histopathology staining, enabling truly quantitative quality control and improved standardisation. METHODS The stain quantification method comprises conventional microscope slides with a stain-responsive biopolymer film affixed to one side, called stain assessment slides. The stain assessment slides were characterised with haematoxylin and eosin, and implemented in one clinical laboratory to quantify variation levels. RESULTS Stain assessment slide stain uptake increased linearly with duration of haematoxylin and eosin staining (r = 0.99), and demonstrated linearly comparable staining to samples of human liver tissue (r values 0.98-0.99). Laboratory implementation of this technique quantified intra- and inter-instrument variation of staining instruments at one point in time and across a five-day period. CONCLUSION The proposed method has been shown to reliably quantify stain uptake, providing an effective laboratory quality control method for stain variation. This is especially important for whole slide imaging and the future development of artificial intelligence in digital pathology.
Collapse
Affiliation(s)
- Catriona Dunn
- National Pathology Imaging Co-operative, Leeds Teaching Hospitals NHS Trust, Leeds, UK.
- Department of Pathology and Data Analytics, University of Leeds, Leeds, UK.
| | - David Brettle
- National Pathology Imaging Co-operative, Leeds Teaching Hospitals NHS Trust, Leeds, UK
| | - Martin Cockroft
- New Technology Group, Futamura Chemical UK Limited, Wigton, UK
| | | | | | - Darren Treanor
- National Pathology Imaging Co-operative, Leeds Teaching Hospitals NHS Trust, Leeds, UK
- Department of Pathology and Data Analytics, University of Leeds, Leeds, UK
- Department of Histopathology, Leeds Teaching Hospitals NHS Trust, Leeds, UK
- Department of Clinical Pathology and Clinical and Experimental Medicine, Linköping University, Linköping, Sweden
- Centre for Medical Image Science and Visualisation, Linköping University, Linköping, Sweden
| |
Collapse
|
18
|
Liu P, Ji L, Ye F, Fu B. AdvMIL: Adversarial multiple instance learning for the survival analysis on whole-slide images. Med Image Anal 2024; 91:103020. [PMID: 37926034 DOI: 10.1016/j.media.2023.103020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Revised: 04/05/2023] [Accepted: 10/31/2023] [Indexed: 11/07/2023]
Abstract
The survival analysis on histological whole-slide images (WSIs) is one of the most important means to estimate patient prognosis. Although many weakly-supervised deep learning models have been developed for gigapixel WSIs, their potential is generally restricted by classical survival analysis rules and fully-supervised learning requirements. As a result, these models provide patients only with a completely-certain point estimation of time-to-event, and they could only learn from the labeled WSI data currently at a small scale. To tackle these problems, we propose a novel adversarial multiple instance learning (AdvMIL) framework. This framework is based on adversarial time-to-event modeling, and integrates the multiple instance learning (MIL) that is much necessary for WSI representation learning. It is a plug-and-play one, so that most existing MIL-based end-to-end methods can be easily upgraded by applying this framework, gaining the improved abilities of survival distribution estimation and semi-supervised learning. Our extensive experiments show that AdvMIL not only could often bring performance improvement to mainstream WSI survival analysis methods at a relatively low computational cost, but also enables these methods to effectively utilize unlabeled data via semi-supervised learning. Moreover, it is observed that AdvMIL could help improving the robustness of models against patch occlusion and two representative image noises. The proposed AdvMIL framework could promote the research of survival analysis in computational pathology with its novel adversarial MIL paradigm.
Collapse
Affiliation(s)
- Pei Liu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, Chengdu, 611731, Sichuan, China.
| | - Luping Ji
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, Chengdu, 611731, Sichuan, China.
| | - Feng Ye
- Institute of Clinical Pathology, West China Hospital, Sichuan University, Guo Xue Xiang, Chengdu, 610041, Sichuan, China.
| | - Bo Fu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, Chengdu, 611731, Sichuan, China.
| |
Collapse
|
19
|
Altini N, Rossini M, Turkevi-Nagy S, Pesce F, Pontrelli P, Prencipe B, Berloco F, Seshan S, Gibier JB, Pedraza Dorado A, Bueno G, Peruzzi L, Rossi M, Eccher A, Li F, Koumpis A, Beyan O, Barratt J, Vo HQ, Mohan C, Nguyen HV, Cicalese PA, Ernst A, Gesualdo L, Bevilacqua V, Becker JU. Performance and limitations of a supervised deep learning approach for the histopathological Oxford Classification of glomeruli with IgA nephropathy. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 242:107814. [PMID: 37722311 DOI: 10.1016/j.cmpb.2023.107814] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Revised: 09/11/2023] [Accepted: 09/12/2023] [Indexed: 09/20/2023]
Abstract
BACKGROUND AND OBJECTIVE The Oxford Classification for IgA nephropathy is the most successful example of an evidence-based nephropathology classification system. The aim of our study was to replicate the glomerular components of Oxford scoring with an end-to-end deep learning pipeline that involves automatic glomerular segmentation followed by classification for mesangial hypercellularity (M), endocapillary hypercellularity (E), segmental sclerosis (S) and active crescents (C). METHODS A total number of 1056 periodic acid-Schiff (PAS) whole slide images (WSIs), coming from 386 kidney biopsies, were annotated. Several detection models for glomeruli, based on the Mask R-CNN architecture, were trained on 587 WSIs, validated on 161 WSIs, and tested on 127 WSIs. For the development of segmentation models, 20,529 glomeruli were annotated, of which 16,571 as training and 3958 as validation set. The test set of the segmentation module comprised of 2948 glomeruli. For the Oxford classification, 6206 expert-annotated glomeruli from 308 PAS WSIs were labelled for M, E, S, C and split into a training set of 4298 glomeruli from 207 WSIs, and a test set of 1908 glomeruli. We chose the best-performing models to construct an end-to-end pipeline, which we named MESCnn (MESC classification by neural network), for the glomerular Oxford classification of WSIs. RESULTS Instance segmentation yielded excellent results with an AP50 ranging between 78.2-80.1 % (79.4 ± 0.7 %) on the validation and 75.1-77.7 % (76.5 ± 0.9 %) on the test set. The aggregated Jaccard Index was between 73.4-75.9 % (75.0 ± 0.8 %) on the validation and 69.1-73.4 % (72.2 ± 1.4 %) on the test set. At granular glomerular level, Oxford Classification was best replicated for M with EfficientNetV2-L with a mean ROC-AUC of 90.2 % and a mean precision/recall area under the curve (PR-AUC) of 81.8 %, best for E with MobileNetV2 (ROC-AUC 94.7 %) and ResNet50 (PR-AUC 75.8 %), best for S with EfficientNetV2-M (mean ROC-AUC 92.7 %, mean PR-AUC 87.7 %), best for C with EfficientNetV2-L (ROC-AUC 92.3 %) and EfficientNetV2-S (PR-AUC 54.7 %). At biopsy-level, correlation between expert and deep learning labels fulfilled the demands of the Oxford Classification. CONCLUSION We designed an end-to-end pipeline for glomerular Oxford Classification on both a granular glomerular and an entire biopsy level. Both the glomerular segmentation and the classification modules are freely available for further development to the renal medicine community.
Collapse
Affiliation(s)
- Nicola Altini
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona n.4, Bari 70126, Italy
| | - Michele Rossini
- Nephrology Dialysis and Transplantation Unit, Department of Emergency and Organ Transplantation, University of Bari, Bari, Italy
| | - Sándor Turkevi-Nagy
- Department of Pathology, Albert Szent-Györgyi Health Center, University of Szeged, Szeged, Hungary
| | - Francesco Pesce
- Nephrology Dialysis and Transplantation Unit, Department of Emergency and Organ Transplantation, University of Bari, Bari, Italy; Division of Renal Medicine, "Fatebenefratelli Isola Tiberina - Gemelli Isola", Rome, Italy
| | - Paola Pontrelli
- Nephrology Dialysis and Transplantation Unit, Department of Emergency and Organ Transplantation, University of Bari, Bari, Italy
| | - Berardino Prencipe
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona n.4, Bari 70126, Italy
| | - Francesco Berloco
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona n.4, Bari 70126, Italy
| | - Surya Seshan
- Department of Pathology, Weill-Cornell Medical Center/New York Presbyterian Hospital, New York, NY, USA
| | - Jean-Baptiste Gibier
- Department of Pathology, Pathology Institute, Lille University Hospital (CHU), Lille, France
| | | | - Gloria Bueno
- VISILAB Research Group, University of Castilla-La Mancha, Ciudad Real, Spain
| | - Licia Peruzzi
- AOU Città della Salute e della Scienza di Torino, Regina Margherita Children's Hospital, Turin, Italy
| | - Mattia Rossi
- Division of Nephrology, Department of Medicine, University of Verona, Verona, Italy
| | - Albino Eccher
- Section of Pathology, Department of Medical and Surgical Sciences for Children and Adults, University of Modena and Reggio Emilia, University Hospital of Modena, Modena, Italy
| | - Feifei Li
- Faculty of Medicine, University Hospital Cologne, University of Cologne, Institute for Medical Informatics, Cologne, Germany
| | - Adamantios Koumpis
- Faculty of Medicine, University Hospital Cologne, University of Cologne, Institute for Medical Informatics, Cologne, Germany
| | - Oya Beyan
- Faculty of Medicine, University Hospital Cologne, University of Cologne, Institute for Medical Informatics, Cologne, Germany
| | - Jonathan Barratt
- Department of Cardiovascular Sciences, University of Leicester, Leicester, UK
| | - Huy Quoc Vo
- Department of Biomedical Engineering, University of Houston, Houston, USA
| | - Chandra Mohan
- Department of Biomedical Engineering, University of Houston, Houston, USA
| | - Hien Van Nguyen
- Department of Biomedical Engineering, University of Houston, Houston, USA
| | | | - Angela Ernst
- Institute of Medical Statistics and Computational Biology, University of Cologne, Cologne, Germany
| | - Loreto Gesualdo
- Nephrology Dialysis and Transplantation Unit, Department of Emergency and Organ Transplantation, University of Bari, Bari, Italy
| | - Vitoantonio Bevilacqua
- Department of Electrical and Information Engineering (DEI), Polytechnic University of Bari, Via Edoardo Orabona n.4, Bari 70126, Italy; Apulian Bioengineering s.r.l., Via delle Violette n.14, Modugno 70026, Italy.
| | - Jan Ulrich Becker
- Institute of Pathology, University Hospital of Cologne, Cologne, Germany
| |
Collapse
|
20
|
Aswolinskiy W, Munari E, Horlings HM, Mulder L, Bogina G, Sanders J, Liu YH, van den Belt-Dusebout AW, Tessier L, Balkenhol M, Stegeman M, Hoven J, Wesseling J, van der Laak J, Lips EH, Ciompi F. PROACTING: predicting pathological complete response to neoadjuvant chemotherapy in breast cancer from routine diagnostic histopathology biopsies with deep learning. Breast Cancer Res 2023; 25:142. [PMID: 37957667 PMCID: PMC10644597 DOI: 10.1186/s13058-023-01726-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Accepted: 10/02/2023] [Indexed: 11/15/2023] Open
Abstract
BACKGROUND Invasive breast cancer patients are increasingly being treated with neoadjuvant chemotherapy; however, only a fraction of the patients respond to it completely. To prevent overtreatment, there is an urgent need for biomarkers to predict treatment response before administering the therapy. METHODS In this retrospective study, we developed hypothesis-driven interpretable biomarkers based on deep learning, to predict the pathological complete response (pCR, i.e., the absence of tumor cells in the surgical resection specimens) to neoadjuvant chemotherapy solely using digital pathology H&E images of pre-treatment breast biopsies. Our approach consists of two steps: First, we use deep learning to characterize aspects of the tumor micro-environment by detecting mitoses and segmenting tissue into several morphology compartments including tumor, lymphocytes and stroma. Second, we derive computational biomarkers from the segmentation and detection output to encode slide-level relationships of components of the tumor microenvironment, such as tumor and mitoses, stroma, and tumor infiltrating lymphocytes (TILs). RESULTS We developed and evaluated our method on slides from n = 721 patients from three European medical centers with triple-negative and Luminal B breast cancers and performed external independent validation on n = 126 patients from a public dataset. We report the predictive value of the investigated biomarkers for predicting pCR with areas under the receiver operating characteristic curve between 0.66 and 0.88 across the tested cohorts. CONCLUSION The proposed computational biomarkers predict pCR, but will require more evaluation and finetuning for clinical application. Our results further corroborate the potential role of deep learning to automate TILs quantification, and their predictive value in breast cancer neoadjuvant treatment planning, along with automated mitoses quantification. We made our method publicly available to extract segmentation-based biomarkers for research purposes.
Collapse
Affiliation(s)
- Witali Aswolinskiy
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Enrico Munari
- Department of Molecular and Translational Medicine, University of Brescia, Brescia, Italy
| | - Hugo M Horlings
- The Netherlands Cancer Institute (NKI), Amsterdam, The Netherlands
| | - Lennart Mulder
- The Netherlands Cancer Institute (NKI), Amsterdam, The Netherlands
| | - Giuseppe Bogina
- Pathology Unit, IRCCS Sacro Cuore Don Calabria Hospital, Negrar di Valpolicella, Verona, Italy
| | - Joyce Sanders
- The Netherlands Cancer Institute (NKI), Amsterdam, The Netherlands
| | - Yat-Hee Liu
- The Netherlands Cancer Institute (NKI), Amsterdam, The Netherlands
| | | | - Leslie Tessier
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
- Center for Integrated Oncology (Institut du cancer de l'Ouest), Angers, France
| | - Maschenka Balkenhol
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Michelle Stegeman
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Jeffrey Hoven
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Jelle Wesseling
- The Netherlands Cancer Institute (NKI), Amsterdam, The Netherlands
- Leiden University Medical Center, Leiden, The Netherlands
| | - Jeroen van der Laak
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Esther H Lips
- The Netherlands Cancer Institute (NKI), Amsterdam, The Netherlands
| | - Francesco Ciompi
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands.
| |
Collapse
|
21
|
Wang H, Huang G, Zhao Z, Cheng L, Juncker-Jensen A, Nagy ML, Lu X, Zhang X, Chen DZ. CCF-GNN: A Unified Model Aggregating Appearance, Microenvironment, and Topology for Pathology Image Classification. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3179-3193. [PMID: 37027573 DOI: 10.1109/tmi.2023.3249343] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Pathology images contain rich information of cell appearance, microenvironment, and topology features for cancer analysis and diagnosis. Among such features, topology becomes increasingly important in analysis for cancer immunotherapy. By analyzing geometric and hierarchically structured cell distribution topology, oncologists can identify densely-packed and cancer-relevant cell communities (CCs) for making decisions. Compared to commonly-used pixel-level Convolution Neural Network (CNN) features and cell-instance-level Graph Neural Network (GNN) features, CC topology features are at a higher level of granularity and geometry. However, topological features have not been well exploited by recent deep learning (DL) methods for pathology image classification due to lack of effective topological descriptors for cell distribution and gathering patterns. In this paper, inspired by clinical practice, we analyze and classify pathology images by comprehensively learning cell appearance, microenvironment, and topology in a fine-to-coarse manner. To describe and exploit topology, we design Cell Community Forest (CCF), a novel graph that represents the hierarchical formulation process of big-sparse CCs from small-dense CCs. Using CCF as a new geometric topological descriptor of tumor cells in pathology images, we propose CCF-GNN, a GNN model that successively aggregates heterogeneous features (e.g., appearance, microenvironment) from cell-instance-level, cell-community-level, into image-level for pathology image classification. Extensive cross-validation experiments show that our method significantly outperforms alternative methods on H&E-stained and immunofluorescence images for disease grading tasks with multiple cancer types. Our proposed CCF-GNN establishes a new topological data analysis (TDA) based method, which facilitates integrating multi-level heterogeneous features of point clouds (e.g., for cells) into a unified DL framework.
Collapse
|
22
|
Hanna MG, Brogi E. Future Practices of Breast Pathology Using Digital and Computational Pathology. Adv Anat Pathol 2023; 30:421-433. [PMID: 37737690 DOI: 10.1097/pap.0000000000000414] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/23/2023]
Abstract
Pathology clinical practice has evolved by adopting technological advancements initially regarded as potentially disruptive, such as electron microscopy, immunohistochemistry, and genomic sequencing. Breast pathology has a critical role as a medical domain, where the patient's pathology diagnosis has significant implications for prognostication and treatment of diseases. The advent of digital and computational pathology has brought about significant advancements in the field, offering new possibilities for enhancing diagnostic accuracy and improving patient care. Digital slide scanning enables to conversion of glass slides into high-fidelity digital images, supporting the review of cases in a digital workflow. Digitization offers the capability to render specimen diagnoses, digital archival of patient specimens, collaboration, and telepathology. Integration of image analysis and machine learning-based systems layered atop the high-resolution digital images offers novel workflows to assist breast pathologists in their clinical, educational, and research endeavors. Decision support tools may improve the detection and classification of breast lesions and the quantification of immunohistochemical studies. Computational biomarkers may help to contribute to patient management or outcomes. Furthermore, using digital and computational pathology may increase standardization and quality assurance, especially in areas with high interobserver variability. This review explores the current landscape and possible future applications of digital and computational techniques in the field of breast pathology.
Collapse
Affiliation(s)
- Matthew G Hanna
- Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, NY
| | | |
Collapse
|
23
|
Post CS, Cheng J, Pantanowitz L, Westerhoff M. Utility of Machine Learning to Detect Cytomegalovirus in Digital Hematoxylin and Eosin-Stained Slides. J Transl Med 2023; 103:100225. [PMID: 37527779 DOI: 10.1016/j.labinv.2023.100225] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2023] [Revised: 07/01/2023] [Accepted: 07/25/2023] [Indexed: 08/03/2023] Open
Abstract
Rapid and accurate cytomegalovirus (CMV) identification in immunosuppressed or immunocompromised patients presenting with diarrhea is essential for therapeutic management. Due to viral latency, however, the gold standard for CMV diagnosis remains to identify viral cytopathic inclusions on routine hematoxylin and eosin (H&E)-stained tissue sections. Therefore, biopsies may be taken and "rushed" for pathology evaluation. Here, we propose the use of artificial intelligence to detect CMV inclusions on routine H&E-stained whole-slide images to aid pathologists in evaluating these cases. Fifty-eight representative H&E slides from 30 cases with CMV inclusions were identified and scanned. The resulting whole-slide images were manually annotated for CMV inclusions and tiled into 300 × 300 pixel patches. Patches containing annotations were labeled "positive," and these tiles were oversampled with image augmentation to account for class imbalance. The remaining patches were labeled "negative." Data were then divided into training, validation, and holdout sets. Multiple deep learning models were provided with training data, and their performance was analyzed. All tested models showed excellent performance. The highest performance was seen using the EfficientNetV2BO model, which had a test (holdout) accuracy of 99.93%, precision of 100.0%, recall (sensitivity) of 99.85%, and area under the curve of 0.9998. Of 518,941 images in the holdout set, there were only 346 false negatives and 2 false positives. This shows proof of concept for the use of digital tools to assist pathologists in screening "rush" biopsies for CMV infection. Given the high precision, cases screened as "positive" can be quickly confirmed by a pathologist, reducing missed CMV inclusions and improving the confidence of preliminary results. Additionally, this may reduce the need for immunohistochemistry in limited tissue samples, reducing associated costs and turnaround time.
Collapse
Affiliation(s)
- Corey S Post
- Department of Pathology, Michigan Medicine, Ann Arbor, Michigan.
| | - Jerome Cheng
- Department of Pathology, Michigan Medicine, Ann Arbor, Michigan
| | | | | |
Collapse
|
24
|
Al-Thelaya K, Gilal NU, Alzubaidi M, Majeed F, Agus M, Schneider J, Househ M. Applications of discriminative and deep learning feature extraction methods for whole slide image analysis: A survey. J Pathol Inform 2023; 14:100335. [PMID: 37928897 PMCID: PMC10622844 DOI: 10.1016/j.jpi.2023.100335] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Revised: 07/17/2023] [Accepted: 07/19/2023] [Indexed: 11/07/2023] Open
Abstract
Digital pathology technologies, including whole slide imaging (WSI), have significantly improved modern clinical practices by facilitating storing, viewing, processing, and sharing digital scans of tissue glass slides. Researchers have proposed various artificial intelligence (AI) solutions for digital pathology applications, such as automated image analysis, to extract diagnostic information from WSI for improving pathology productivity, accuracy, and reproducibility. Feature extraction methods play a crucial role in transforming raw image data into meaningful representations for analysis, facilitating the characterization of tissue structures, cellular properties, and pathological patterns. These features have diverse applications in several digital pathology applications, such as cancer prognosis and diagnosis. Deep learning-based feature extraction methods have emerged as a promising approach to accurately represent WSI contents and have demonstrated superior performance in histology-related tasks. In this survey, we provide a comprehensive overview of feature extraction methods, including both manual and deep learning-based techniques, for the analysis of WSIs. We review relevant literature, analyze the discriminative and geometric features of WSIs (i.e., features suited to support the diagnostic process and extracted by "engineered" methods as opposed to AI), and explore predictive modeling techniques using AI and deep learning. This survey examines the advances, challenges, and opportunities in this rapidly evolving field, emphasizing the potential for accurate diagnosis, prognosis, and decision-making in digital pathology.
Collapse
Affiliation(s)
- Khaled Al-Thelaya
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Nauman Ullah Gilal
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Mahmood Alzubaidi
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Fahad Majeed
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Marco Agus
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Jens Schneider
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Mowafa Househ
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| |
Collapse
|
25
|
Liu Y, Han D, Parwani AV, Li Z. Applications of Artificial Intelligence in Breast Pathology. Arch Pathol Lab Med 2023; 147:1003-1013. [PMID: 36800539 DOI: 10.5858/arpa.2022-0457-ra] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/15/2022] [Indexed: 02/19/2023]
Abstract
CONTEXT.— Increasing implementation of whole slide imaging together with digital workflow and advances in computing capacity enable the use of artificial intelligence (AI) in pathology, including breast pathology. Breast pathologists often face a significant workload, with diagnosis complexity, tedious repetitive tasks, and semiquantitative evaluation of biomarkers. Recent advances in developing AI algorithms have provided promising approaches to meet the demand in breast pathology. OBJECTIVE.— To provide an updated review of AI in breast pathology. We examined the success and challenges of current and potential AI applications in diagnosing and grading breast carcinomas and other pathologic changes, detecting lymph node metastasis, quantifying breast cancer biomarkers, predicting prognosis and therapy response, and predicting potential molecular changes. DATA SOURCES.— We obtained data and information by searching and reviewing literature on AI in breast pathology from PubMed and based our own experience. CONCLUSIONS.— With the increasing application in breast pathology, AI not only assists in pathology diagnosis to improve accuracy and reduce pathologists' workload, but also provides new information in predicting prognosis and therapy response.
Collapse
Affiliation(s)
- Yueping Liu
- From the Department of Pathology, The Fourth Hospital of Hebei Medical University, Shijiazhuang, China (Liu, Han)
| | - Dandan Han
- From the Department of Pathology, The Fourth Hospital of Hebei Medical University, Shijiazhuang, China (Liu, Han)
| | - Anil V Parwani
- The Department of Pathology, The Ohio State University, Columbus (Parwani, Li)
| | - Zaibo Li
- From the Department of Pathology, The Fourth Hospital of Hebei Medical University, Shijiazhuang, China (Liu, Han)
| |
Collapse
|
26
|
Cooper M, Ji Z, Krishnan RG. Machine learning in computational histopathology: Challenges and opportunities. Genes Chromosomes Cancer 2023; 62:540-556. [PMID: 37314068 DOI: 10.1002/gcc.23177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 05/18/2023] [Accepted: 05/20/2023] [Indexed: 06/15/2023] Open
Abstract
Digital histopathological images, high-resolution images of stained tissue samples, are a vital tool for clinicians to diagnose and stage cancers. The visual analysis of patient state based on these images are an important part of oncology workflow. Although pathology workflows have historically been conducted in laboratories under a microscope, the increasing digitization of histopathological images has led to their analysis on computers in the clinic. The last decade has seen the emergence of machine learning, and deep learning in particular, a powerful set of tools for the analysis of histopathological images. Machine learning models trained on large datasets of digitized histopathology slides have resulted in automated models for prediction and stratification of patient risk. In this review, we provide context for the rise of such models in computational histopathology, highlight the clinical tasks they have found success in automating, discuss the various machine learning techniques that have been applied to this domain, and underscore open problems and opportunities.
Collapse
Affiliation(s)
- Michael Cooper
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- University Health Network, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
| | - Zongliang Ji
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
| | - Rahul G Krishnan
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
- Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
27
|
Yosofvand M, Khan SY, Dhakal R, Nejat A, Moustaid-Moussa N, Rahman RL, Moussa H. Automated Detection and Scoring of Tumor-Infiltrating Lymphocytes in Breast Cancer Histopathology Slides. Cancers (Basel) 2023; 15:3635. [PMID: 37509295 PMCID: PMC10377197 DOI: 10.3390/cancers15143635] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Revised: 07/06/2023] [Accepted: 07/10/2023] [Indexed: 07/30/2023] Open
Abstract
Detection of tumor-infiltrating lymphocytes (TILs) in cancer images has gained significant importance as these lymphocytes can be used as a biomarker in cancer detection and treatment procedures. Our goal was to develop and apply a TILs detection tool that utilizes deep learning models, following two sequential steps. First, based on the guidelines from the International Immuno-Oncology Biomarker Working Group (IIOBWG) on Breast Cancer, we labeled 63 large pathology imaging slides and annotated the TILs in the stroma area to create the dataset required for model development. In the second step, various machine learning models were employed and trained to detect the stroma where U-Net deep learning structure was able to achieve 98% accuracy. After detecting the stroma area, a Mask R-CNN model was employed for the TILs detection task. The R-CNN model detected the TILs in various images and was used as the backbone analysis network for the GUI development of the TILs detection tool. This is the first study to combine two deep learning models for TILs detection at the cellular level in breast tumor histopathology slides. Our novel approach can be applied to scoring TILs in large cancer slides. Statistical analysis showed that the output of the implemented approach had 95% concordance with the scores assigned by the pathologists, with a p-value of 0.045 (n = 63). This demonstrated that the results from the developed software were statistically meaningful and highly accurate. The implemented approach in analyzing whole tumor histology slides and the newly developed TILs detection tool can be used for research purposes in biomedical and pathology applications and it can provide researchers and clinicians with the TIL score for various input images. Future research using additional breast cancer slides from various sources for further training and validation of the developed models is necessary for more inclusive, rigorous, and robust clinical applications.
Collapse
Affiliation(s)
- Mohammad Yosofvand
- Department of Mechanical Engineering, Texas Tech University, Lubbock, TX 79409, USA
| | - Sonia Y Khan
- Breast Center of Excellence, Department of Surgery, Texas Tech University Health Sciences Center, Lubbock, TX 79430, USA
| | - Rabin Dhakal
- Department of Mechanical Engineering, Texas Tech University, Lubbock, TX 79409, USA
| | - Ali Nejat
- Department of Civil, Environmental, & Construction Engineering, Texas Tech University, Lubbock, TX 79409, USA
| | - Naima Moustaid-Moussa
- Department of Nutritional Sciences, Texas Tech University, Lubbock, TX 79409, USA
- Obesity Research Institute, Texas Tech University, Lubbock, TX 79409, USA
| | - Rakhshanda Layeequr Rahman
- Breast Center of Excellence, Department of Surgery, Texas Tech University Health Sciences Center, Lubbock, TX 79430, USA
| | - Hanna Moussa
- Department of Mechanical Engineering, Texas Tech University, Lubbock, TX 79409, USA
- Obesity Research Institute, Texas Tech University, Lubbock, TX 79409, USA
| |
Collapse
|
28
|
Ram S, Tang W, Bell AJ, Pal R, Spencer C, Buschhaus A, Hatt CR, diMagliano MP, Rehemtulla A, Rodríguez JJ, Galban S, Galban CJ. Lung cancer lesion detection in histopathology images using graph-based sparse PCA network. Neoplasia 2023; 42:100911. [PMID: 37269818 DOI: 10.1016/j.neo.2023.100911] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2023] [Accepted: 05/17/2023] [Indexed: 06/05/2023]
Abstract
Early detection of lung cancer is critical for improvement of patient survival. To address the clinical need for efficacious treatments, genetically engineered mouse models (GEMM) have become integral in identifying and evaluating the molecular underpinnings of this complex disease that may be exploited as therapeutic targets. Assessment of GEMM tumor burden on histopathological sections performed by manual inspection is both time consuming and prone to subjective bias. Therefore, an interplay of needs and challenges exists for computer-aided diagnostic tools, for accurate and efficient analysis of these histopathology images. In this paper, we propose a simple machine learning approach called the graph-based sparse principal component analysis (GS-PCA) network, for automated detection of cancerous lesions on histological lung slides stained by hematoxylin and eosin (H&E). Our method comprises four steps: 1) cascaded graph-based sparse PCA, 2) PCA binary hashing, 3) block-wise histograms, and 4) support vector machine (SVM) classification. In our proposed architecture, graph-based sparse PCA is employed to learn the filter banks of the multiple stages of a convolutional network. This is followed by PCA hashing and block histograms for indexing and pooling. The meaningful features extracted from this GS-PCA are then fed to an SVM classifier. We evaluate the performance of the proposed algorithm on H&E slides obtained from an inducible K-rasG12D lung cancer mouse model using precision/recall rates, Fβ-score, Tanimoto coefficient, and area under the curve (AUC) of the receiver operator characteristic (ROC) and show that our algorithm is efficient and provides improved detection accuracy compared to existing algorithms.
Collapse
Affiliation(s)
- Sundaresh Ram
- Departments of Radiology, and Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, USA.
| | - Wenfei Tang
- Department of Computer Science and Engineering, University of Michigan, Ann Arbor, MI 48109, USA
| | - Alexander J Bell
- Departments of Radiology, and Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, USA
| | - Ravi Pal
- Department of Radiology, University of Michigan, Ann Arbor, MI 48109, USA
| | - Cara Spencer
- Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor, MI 48109, USA
| | | | - Charles R Hatt
- Department of Radiology, University of Michigan, Ann Arbor, MI 48109, USA; Imbio LLC, Minneapolis, MN 55405, USA
| | - Marina Pasca diMagliano
- Departments of Surgery, and Cell and Developmental Biology, University of Michigan, Ann Arbor, MI 48109, USA
| | - Alnawaz Rehemtulla
- Departments of Radiology, and Radiation Oncology, University of Michigan, Ann Arbor, MI 48109, USA
| | - Jeffrey J Rodríguez
- Departments of Electrical and Computer Engineering, and Biomedical Engineering, The University of Arizona, Tucson, AZ 85721, USA
| | - Stefanie Galban
- Department of Radiology, University of Michigan, Ann Arbor, MI 48109, USA
| | - Craig J Galban
- Departments of Radiology, and Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, USA
| |
Collapse
|
29
|
Hu W, Li X, Li C, Li R, Jiang T, Sun H, Huang X, Grzegorzek M, Li X. A state-of-the-art survey of artificial neural networks for Whole-slide Image analysis: From popular Convolutional Neural Networks to potential visual transformers. Comput Biol Med 2023; 161:107034. [PMID: 37230019 DOI: 10.1016/j.compbiomed.2023.107034] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Revised: 04/13/2023] [Accepted: 05/10/2023] [Indexed: 05/27/2023]
Abstract
In recent years, with the advancement of computer-aided diagnosis (CAD) technology and whole slide image (WSI), histopathological WSI has gradually played a crucial aspect in the diagnosis and analysis of diseases. To increase the objectivity and accuracy of pathologists' work, artificial neural network (ANN) methods have been generally needed in the segmentation, classification, and detection of histopathological WSI. However, the existing review papers only focus on equipment hardware, development status and trends, and do not summarize the art neural network used for full-slide image analysis in detail. In this paper, WSI analysis methods based on ANN are reviewed. Firstly, the development status of WSI and ANN methods is introduced. Secondly, we summarize the common ANN methods. Next, we discuss publicly available WSI datasets and evaluation metrics. These ANN architectures for WSI processing are divided into classical neural networks and deep neural networks (DNNs) and then analyzed. Finally, the application prospect of the analytical method in this field is discussed. The important potential method is Visual Transformers.
Collapse
Affiliation(s)
- Weiming Hu
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Xintong Li
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Chen Li
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China.
| | - Rui Li
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Tao Jiang
- School of Intelligent Medicine, Chengdu University of Traditional Chinese Medicine, Chengdu, China; International Joint Institute of Robotics and Intelligent Systems, Chengdu University of Information Technology, Chengdu, China
| | - Hongzan Sun
- Shengjing Hospital of China Medical University, Shenyang, China
| | - Xinyu Huang
- Institute for Medical Informatics, University of Luebeck, Luebeck, Germany
| | - Marcin Grzegorzek
- Institute for Medical Informatics, University of Luebeck, Luebeck, Germany; Department of Knowledge Engineering, University of Economics in Katowice, Katowice, Poland
| | - Xiaoyan Li
- Cancer Hospital of China Medical University, Shenyang, China.
| |
Collapse
|
30
|
Sikaroudi M, Hosseini M, Gonzalez R, Rahnamayan S, Tizhoosh HR. Generalization of vision pre-trained models for histopathology. Sci Rep 2023; 13:6065. [PMID: 37055519 PMCID: PMC10102232 DOI: 10.1038/s41598-023-33348-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Accepted: 04/12/2023] [Indexed: 04/15/2023] Open
Abstract
Out-of-distribution (OOD) generalization, especially for medical setups, is a key challenge in modern machine learning which has only recently received much attention. We investigate how different convolutional pre-trained models perform on OOD test data-that is data from domains that have not been seen during training-on histopathology repositories attributed to different trial sites. Different trial site repositories, pre-trained models, and image transformations are examined as specific aspects of pre-trained models. A comparison is also performed among models trained entirely from scratch (i.e., without pre-training) and models already pre-trained. The OOD performance of pre-trained models on natural images, i.e., (1) vanilla pre-trained ImageNet, (2) semi-supervised learning (SSL), and (3) semi-weakly-supervised learning (SWSL) models pre-trained on IG-1B-Targeted are examined in this study. In addition, the performance of a histopathology model (i.e., KimiaNet) trained on the most comprehensive histopathology dataset, i.e., TCGA, has also been studied. Although the performance of SSL and SWSL pre-trained models are conducive to better OOD performance in comparison to the vanilla ImageNet pre-trained model, the histopathology pre-trained model is still the best in overall. In terms of top-1 accuracy, we demonstrate that diversifying the images in the training using reasonable image transformations is effective to avoid learning shortcuts when the distribution shift is significant. In addition, XAI techniques-which aim to achieve high-quality human-understandable explanations of AI decisions-are leveraged for further investigations.
Collapse
Affiliation(s)
| | | | - Ricardo Gonzalez
- Kimia Lab, University of Waterloo, Waterloo, ON, Canada
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA
| | - Shahryar Rahnamayan
- Kimia Lab, University of Waterloo, Waterloo, ON, Canada
- Engineering Department, Brock University, St. Catharines, ON, Canada
| | - H R Tizhoosh
- Kimia Lab, University of Waterloo, Waterloo, ON, Canada.
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA.
- Rhazes Lab, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA.
| |
Collapse
|
31
|
Yan R, He Q, Liu Y, Ye P, Zhu L, Shi S, Gou J, He Y, Guan T, Zhou G. Unpaired virtual histological staining using prior-guided generative adversarial networks. Comput Med Imaging Graph 2023; 105:102185. [PMID: 36764189 DOI: 10.1016/j.compmedimag.2023.102185] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Revised: 01/05/2023] [Accepted: 01/05/2023] [Indexed: 01/24/2023]
Abstract
Fibrosis is an inevitable stage in the development of chronic liver disease and has an irreplaceable role in characterizing the degree of progression of chronic liver disease. Histopathological diagnosis is the gold standard for the interpretation of fibrosis parameters. Conventional hematoxylin-eosin (H&E) staining can only reflect the gross structure of the tissue and the distribution of hepatocytes, while Masson trichrome can highlight specific types of collagen fiber structure, thus providing the necessary structural information for fibrosis scoring. However, the expensive costs of time, economy, and patient specimens as well as the non-uniform preparation and staining process make the conversion of existing H&E staining into virtual Masson trichrome staining a solution for fibrosis evaluation. Existing translation approaches fail to extract fiber features accurately enough, and the decoder of staining is unable to converge due to the inconsistent color of physical staining. In this work, we propose a prior-guided generative adversarial network, based on unpaired data for effective Masson trichrome stained image generation from the corresponding H&E stained image. Conducted on a small training set, our method takes full advantage of prior knowledge to set up better constraints on both the encoder and the decoder. Experiments indicate the superior performance of our method that surpasses the previous approaches. For various liver diseases, our results demonstrate a high correlation between the staging of real and virtual stains (ρ=0.82; 95% CI: 0.73-0.89). In addition, our finetuning strategy is able to standardize the staining color and release the memory and computational burden, which can be employed in clinical assessment.
Collapse
Affiliation(s)
- Renao Yan
- Shenzhen International Graduate School, Tsinghua University, Xili University City, Shenzhen, 518055, Guangdong, China
| | - Qiming He
- Shenzhen International Graduate School, Tsinghua University, Xili University City, Shenzhen, 518055, Guangdong, China
| | - Yiqing Liu
- Shenzhen International Graduate School, Tsinghua University, Xili University City, Shenzhen, 518055, Guangdong, China
| | - Peng Ye
- Shenzhen International Graduate School, Tsinghua University, Xili University City, Shenzhen, 518055, Guangdong, China
| | - Lianghui Zhu
- Shenzhen International Graduate School, Tsinghua University, Xili University City, Shenzhen, 518055, Guangdong, China
| | - Shanshan Shi
- Shenzhen International Graduate School, Tsinghua University, Xili University City, Shenzhen, 518055, Guangdong, China
| | - Jizhou Gou
- The Third People's Hospital of Shenzhen, Buji Buran Road 29, Shenzhen, 518112, Guangdong, China
| | - Yonghong He
- Shenzhen International Graduate School, Tsinghua University, Xili University City, Shenzhen, 518055, Guangdong, China
| | - Tian Guan
- Shenzhen International Graduate School, Tsinghua University, Xili University City, Shenzhen, 518055, Guangdong, China.
| | - Guangde Zhou
- The Third People's Hospital of Shenzhen, Buji Buran Road 29, Shenzhen, 518112, Guangdong, China.
| |
Collapse
|
32
|
Awan R, Raza SEA, Lotz J, Weiss N, Rajpoot N. Deep feature based cross-slide registration. Comput Med Imaging Graph 2023; 104:102162. [PMID: 36584537 DOI: 10.1016/j.compmedimag.2022.102162] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2022] [Revised: 11/15/2022] [Accepted: 12/08/2022] [Indexed: 12/23/2022]
Abstract
Registration of multiple sections in a tissue block is an important pre-requisite task before any cross-slide image analysis. Non-rigid registration methods are capable of finding correspondence by locally transforming a moving image. These methods often rely on an initial guess to roughly align an image pair linearly and globally. This is essential to prevent convergence to a non-optimal minimum. We explore a deep feature based registration (DFBR) method which utilises data-driven descriptors to estimate the global transformation. A multi-stage strategy is adopted for improving the quality of registration. A visualisation tool is developed to view registered pairs of WSIs at different magnifications. With the help of this tool, one can apply a transformation on the fly without the need to generate a transformed moving WSI in a pyramidal form. We compare the performance on our dataset of data-driven descriptors with that of hand-crafted descriptors. Our approach can align the images with only small registration errors. The efficacy of our proposed method is evaluated for a subsequent non-rigid registration step. To this end, the first two steps of the ANHIR winner's framework are replaced with DFBR to register image pairs provided by the challenge. The modified framework produce comparable results to those of the challenge winning team.
Collapse
Affiliation(s)
- Ruqayya Awan
- Department of Computer Science, University of Warwick, CV4 7AL Coventry, UK.
| | - Shan E Ahmed Raza
- Department of Computer Science, University of Warwick, CV4 7AL Coventry, UK.
| | - Johannes Lotz
- Fraunhofer Institute for Digital Medicine MEVIS, Lübeck, Germany.
| | - Nick Weiss
- Fraunhofer Institute for Digital Medicine MEVIS, Lübeck, Germany.
| | - Nasir Rajpoot
- Department of Computer Science, University of Warwick, CV4 7AL Coventry, UK; Department of Pathology, University Hospitals Coventry, Warwickshire, UK; The Alan Turing Institute, London, UK.
| |
Collapse
|
33
|
A generalizable and robust deep learning algorithm for mitosis detection in multicenter breast histopathological images. Med Image Anal 2023; 84:102703. [PMID: 36481608 DOI: 10.1016/j.media.2022.102703] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2022] [Revised: 09/16/2022] [Accepted: 11/21/2022] [Indexed: 11/24/2022]
Abstract
Mitosis counting of biopsies is an important biomarker for breast cancer patients, which supports disease prognostication and treatment planning. Developing a robust mitotic cell detection model is highly challenging due to its complex growth pattern and high similarities with non-mitotic cells. Most mitosis detection algorithms have poor generalizability across image domains and lack reproducibility and validation in multicenter settings. To overcome these issues, we propose a generalizable and robust mitosis detection algorithm (called FMDet), which is independently tested on multicenter breast histopathological images. To capture more refined morphological features of cells, we convert the object detection task as a semantic segmentation problem. The pixel-level annotations for mitotic nuclei are obtained by taking the intersection of the masks generated from a well-trained nuclear segmentation model and the bounding boxes provided by the MIDOG 2021 challenge. In our segmentation framework, a robust feature extractor is developed to capture the appearance variations of mitotic cells, which is constructed by integrating a channel-wise multi-scale attention mechanism into a fully convolutional network structure. Benefiting from the fact that the changes in the low-level spectrum do not affect the high-level semantic perception, we employ a Fourier-based data augmentation method to reduce domain discrepancies by exchanging the low-frequency spectrum between two domains. Our FMDet algorithm has been tested in the MIDOG 2021 challenge and ranked first place. Further, our algorithm is also externally validated on four independent datasets for mitosis detection, which exhibits state-of-the-art performance in comparison with previously published results. These results demonstrate that our algorithm has the potential to be deployed as an assistant decision support tool in clinical practice. Our code has been released at https://github.com/Xiyue-Wang/1st-in-MICCAI-MIDOG-2021-challenge.
Collapse
|
34
|
Nasser M, Yusof UK. Deep Learning Based Methods for Breast Cancer Diagnosis: A Systematic Review and Future Direction. Diagnostics (Basel) 2023; 13:diagnostics13010161. [PMID: 36611453 PMCID: PMC9818155 DOI: 10.3390/diagnostics13010161] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2022] [Revised: 12/19/2022] [Accepted: 12/19/2022] [Indexed: 01/06/2023] Open
Abstract
Breast cancer is one of the precarious conditions that affect women, and a substantive cure has not yet been discovered for it. With the advent of Artificial intelligence (AI), recently, deep learning techniques have been used effectively in breast cancer detection, facilitating early diagnosis and therefore increasing the chances of patients' survival. Compared to classical machine learning techniques, deep learning requires less human intervention for similar feature extraction. This study presents a systematic literature review on the deep learning-based methods for breast cancer detection that can guide practitioners and researchers in understanding the challenges and new trends in the field. Particularly, different deep learning-based methods for breast cancer detection are investigated, focusing on the genomics and histopathological imaging data. The study specifically adopts the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), which offer a detailed analysis and synthesis of the published articles. Several studies were searched and gathered, and after the eligibility screening and quality evaluation, 98 articles were identified. The results of the review indicated that the Convolutional Neural Network (CNN) is the most accurate and extensively used model for breast cancer detection, and the accuracy metrics are the most popular method used for performance evaluation. Moreover, datasets utilized for breast cancer detection and the evaluation metrics are also studied. Finally, the challenges and future research direction in breast cancer detection based on deep learning models are also investigated to help researchers and practitioners acquire in-depth knowledge of and insight into the area.
Collapse
|
35
|
Querzoli G, Veronesi G, Corti B, Nottegar A, Dika E. Basic Elements of Artificial Intelligence Tools in the Diagnosis of Cutaneous Melanoma. Crit Rev Oncog 2023; 28:37-41. [PMID: 37968992 DOI: 10.1615/critrevoncog.2023050220] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2023]
Abstract
Cutaneous melanoma (CM) incidence has dramatically increased in the last years. Early diagnosis is of paramount importance in terms of prognosis. Artificial Intelligence (AI) tools are being proposed for clinicians and pathologists as an adjunct support in the diagnostic process. We described herein an overview of the most important parameters that a potential AI tool should take into consideration in histopathology to evaluate a skin lesion. First of all, recognition of a melanocytic or non-melanocytic nature. Furthermore, melanocytic lesions should be stratified according to at least four parameters: silhouette and asymmetry; identification and spatial distribution of the cells; mitosis count; presence of ulceration. According to the number of parameters the AI tools might stratify the risk of CM and prioritize the pathologist's work.
Collapse
Affiliation(s)
- Giulia Querzoli
- Pathology Unit, IRCCS Azienda Ospedaliero Universitaria di Bologna, Italy
| | - Giulia Veronesi
- Division of Dermatology, IRCCS Azienda Ospedaliero Universitaria di Bologna, Italy; Alma Mater Studiorum, University of Bologna, Italy
| | - Barbara Corti
- Pathology Unit, IRCCS Azienda Ospedaliero Universitaria di Bologna, Italy
| | | | - Emi Dika
- Division of Dermatology, IRCCS Azienda Ospedaliero Universitaria di Bologna, Italy; Alma Mater Studiorum, University of Bologna, Italy
| |
Collapse
|
36
|
Chan RC, To CKC, Cheng KCT, Yoshikazu T, Yan LLA, Tse GM. Artificial intelligence in breast cancer histopathology. Histopathology 2023; 82:198-210. [PMID: 36482271 DOI: 10.1111/his.14820] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 09/22/2022] [Accepted: 09/28/2022] [Indexed: 12/13/2022]
Abstract
This is a review on the use of artificial intelligence for digital breast pathology. A systematic search on PubMed was conducted, identifying 17,324 research papers related to breast cancer pathology. Following a semimanual screening, 664 papers were retrieved and pursued. The papers are grouped into six major tasks performed by pathologists-namely, molecular and hormonal analysis, grading, mitotic figure counting, ki-67 indexing, tumour-infiltrating lymphocyte assessment, and lymph node metastases identification. Under each task, open-source datasets for research to build artificial intelligence (AI) tools are also listed. Many AI tools showed promise and demonstrated feasibility in the automation of routine pathology investigations. We expect continued growth of AI in this field as new algorithms mature.
Collapse
Affiliation(s)
- Ronald Ck Chan
- Department of Anatomical and Cellular Pathology, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong, Hong Kong
| | - Chun Kit Curtis To
- Department of Anatomical and Cellular Pathology, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong, Hong Kong
| | - Ka Chuen Tom Cheng
- Department of Anatomical and Cellular Pathology, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong, Hong Kong
| | - Tada Yoshikazu
- Department of Anatomical and Cellular Pathology, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong, Hong Kong
| | - Lai Ling Amy Yan
- Department of Anatomical and Cellular Pathology, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong, Hong Kong
| | - Gary M Tse
- Department of Anatomical and Cellular Pathology, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong, Hong Kong
| |
Collapse
|
37
|
Deep Learning for Skin Melanocytic Tumors in Whole-Slide Images: A Systematic Review. Cancers (Basel) 2022; 15:cancers15010042. [PMID: 36612037 PMCID: PMC9817526 DOI: 10.3390/cancers15010042] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 12/05/2022] [Accepted: 12/16/2022] [Indexed: 12/24/2022] Open
Abstract
The rise of Artificial Intelligence (AI) has shown promising performance as a support tool in clinical pathology workflows. In addition to the well-known interobserver variability between dermatopathologists, melanomas present a significant challenge in their histological interpretation. This study aims to analyze all previously published studies on whole-slide images of melanocytic tumors that rely on deep learning techniques for automatic image analysis. Embase, Pubmed, Web of Science, and Virtual Health Library were used to search for relevant studies for the systematic review, in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. Articles from 2015 to July 2022 were included, with an emphasis placed on the used artificial intelligence methods. Twenty-eight studies that fulfilled the inclusion criteria were grouped into four groups based on their clinical objectives, including pathologists versus deep learning models (n = 10), diagnostic prediction (n = 7); prognosis (n = 5), and histological features (n = 6). These were then analyzed to draw conclusions on the general parameters and conditions of AI in pathology, as well as the necessary factors for better performance in real scenarios.
Collapse
|
38
|
Villamanca JJ, Hermogino LJ, Ong KD, Paguia B, Abanilla L, Lim A, Angeles LM, Espiritu B, Isais M, Tomas RC, Albano PM. Predicting the Likelihood of Colorectal Cancer with Artificial Intelligence Tools Using Fourier Transform Infrared Signals Obtained from Tumor Samples. APPLIED SPECTROSCOPY 2022; 76:1412-1428. [PMID: 35821580 DOI: 10.1177/00037028221116083] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
The early and accurate detection of colorectal cancer (CRC) significantly affects its prognosis and clinical management. However, current standard diagnostic procedures for CRC often lack sensitivity and specificity since most rely on visual examination. Hence, there is a need to develop more accurate methods for its diagnosis. Support vector machine (SVM) and feedforward neural network (FNN) models were designed using the Fourier transform infrared (FT-IR) spectral data of several colorectal tissues that were unanimously identified as either benign or malignant by different unrelated pathologists. The set of samples in which the pathologists had discordant readings were then analyzed using the AI models described above. Between the SVM and NN models, the NN model was able to outperform the SVM model based on their prediction confidence scores. Using the spectral data of the concordant samples as training set, the FNN was able to predict the histologically diagnosed malignant tissues (n = 118) at 59.9-99.9% confidence (average = 93.5%). Of the 118 samples, 84 (71.18%) were classified with an above average confidence score, 34 (28.81%) classified below the average confidence score, and none was misclassified. Moreover, it was able to correctly identify the histologically confirmed benign samples (n = 83) at 51.5-99.7% confidence (average = 91.64%). Of the 83 samples, 60 (72.29%) were classified with an above average confidence score, 22 (26.51%) classified below the average confidence score, and only 1 sample (1.20%) was misclassified. The study provides additional proof of the ability of attenuated total reflection (ATR) FT-IR enhanced by AI tools to predict the likelihood of CRC without dependence on morphological changes in tissues.
Collapse
Affiliation(s)
- John Jerald Villamanca
- Department of Biological Sciences, College of Science, 564927University of Santo Tomas, Manila, Philippines
| | - Lemuel John Hermogino
- Department of Biological Sciences, College of Science, 564927University of Santo Tomas, Manila, Philippines
| | - Katherine Denise Ong
- Department of Biological Sciences, College of Science, 564927University of Santo Tomas, Manila, Philippines
| | - Brian Paguia
- Department of Biological Sciences, College of Science, 564927University of Santo Tomas, Manila, Philippines
| | - Lorenzo Abanilla
- Department of Pathology, Divine Word Hospital, Tacloban City, Philippines
| | - Antonio Lim
- Department of Pathology, Divine Word Hospital, Tacloban City, Philippines
| | - Lara Mae Angeles
- Department of Pathology, 596481University of Santo Tomas Hospital, Manila, Philippines
| | - Bernadette Espiritu
- Department of Pathology, 603332Bulacan Medical Center, Malolos City, Philippines
| | - Maura Isais
- Department of Pathology, 603332Bulacan Medical Center, Malolos City, Philippines
- The Graduate School, 595547University of Santo Tomas, Manila, Philippines
| | - Rock Christian Tomas
- Department of Electrical Engineering, 54729University of the Philippines Los Baños, Los Baños, Philippines
| | - Pia Marie Albano
- Department of Biological Sciences, College of Science, 564927University of Santo Tomas, Manila, Philippines
- Department of Pathology, Divine Word Hospital, Tacloban City, Philippines
- Research Center for the Natural and Applied Sciences, 564927University of Santo Tomas, Manila, Philippines
| |
Collapse
|
39
|
Bias reduction in representation of histopathology images using deep feature selection. Sci Rep 2022; 12:19994. [PMID: 36411301 PMCID: PMC9678861 DOI: 10.1038/s41598-022-24317-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Accepted: 11/14/2022] [Indexed: 11/22/2022] Open
Abstract
Appearing traces of bias in deep networks is a serious reliability issue which can play a significant role in ethics and generalization related concerns. Recent studies report that the deep features extracted from the histopathology images of The Cancer Genome Atlas (TCGA), the largest publicly available archive, are surprisingly able to accurately classify the whole slide images (WSIs) based on their acquisition site while these features are extracted to primarily discriminate cancer types. This is clear evidence that the utilized Deep Neural Networks (DNNs) unexpectedly detect the specific patterns of the source site, i.e, the hospital of origin, rather than histomorphologic patterns, a biased behavior resulting in degraded trust and generalization. This observation motivated us to propose a method to alleviate the destructive impact of hospital bias through a novel feature selection process. To this effect, we have proposed an evolutionary strategy to select a small set of optimal features to not only accurately represent the histological patterns of tissue samples but also to eliminate the features contributing to internal bias toward the institution. The defined objective function for an optimal subset selection of features is to minimize the accuracy of the model to classify the source institutions which is basically defined as a bias indicator. By the conducted experiments, the selected features extracted by the state-of-the-art network trained on TCGA images (i.e., the KimiaNet), considerably decreased the institutional bias, while improving the quality of features to discriminate the cancer types. In addition, the selected features could significantly improve the results of external validation compared to the entire set of features which has been negatively affected by bias. The proposed scheme is a model-independent approach which can be employed when it is possible to define a bias indicator as a participating objective in a feature selection process; even with unknown bias sources.
Collapse
|
40
|
Mercan C, Balkenhol M, Salgado R, Sherman M, Vielh P, Vreuls W, Polónia A, Horlings HM, Weichert W, Carter JM, Bult P, Christgen M, Denkert C, van de Vijver K, Bokhorst JM, van der Laak J, Ciompi F. Deep learning for fully-automated nuclear pleomorphism scoring in breast cancer. NPJ Breast Cancer 2022; 8:120. [DOI: 10.1038/s41523-022-00488-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2021] [Accepted: 10/21/2022] [Indexed: 11/11/2022] Open
Abstract
AbstractTo guide the choice of treatment, every new breast cancer is assessed for aggressiveness (i.e., graded) by an experienced histopathologist. Typically, this tumor grade consists of three components, one of which is the nuclear pleomorphism score (the extent of abnormalities in the overall appearance of tumor nuclei). The degree of nuclear pleomorphism is subjectively classified from 1 to 3, where a score of 1 most closely resembles epithelial cells of normal breast epithelium and 3 shows the greatest abnormalities. Establishing numerical criteria for grading nuclear pleomorphism is challenging, and inter-observer agreement is poor. Therefore, we studied the use of deep learning to develop fully automated nuclear pleomorphism scoring in breast cancer. The reference standard used for training the algorithm consisted of the collective knowledge of an international panel of 10 pathologists on a curated set of regions of interest covering the entire spectrum of tumor morphology in breast cancer. To fully exploit the information provided by the pathologists, a first-of-its-kind deep regression model was trained to yield a continuous scoring rather than limiting the pleomorphism scoring to the standard three-tiered system. Our approach preserves the continuum of nuclear pleomorphism without necessitating a large data set with explicit annotations of tumor nuclei. Once translated to the traditional system, our approach achieves top pathologist-level performance in multiple experiments on regions of interest and whole-slide images, compared to a panel of 10 and 4 pathologists, respectively.
Collapse
|
41
|
Nielsen PS, Georgsen JB, Vinding MS, Østergaard LR, Steiniche T. Computer-Assisted Annotation of Digital H&E/SOX10 Dual Stains Generates High-Performing Convolutional Neural Network for Calculating Tumor Burden in H&E-Stained Cutaneous Melanoma. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:14327. [PMID: 36361209 PMCID: PMC9654525 DOI: 10.3390/ijerph192114327] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Revised: 10/07/2022] [Accepted: 10/26/2022] [Indexed: 06/16/2023]
Abstract
Deep learning for the analysis of H&E stains requires a large annotated training set. This may form a labor-intensive task involving highly skilled pathologists. We aimed to optimize and evaluate computer-assisted annotation based on digital dual stains of the same tissue section. H&E stains of primary and metastatic melanoma (N = 77) were digitized, re-stained with SOX10, and re-scanned. Because images were aligned, annotations of SOX10 image analysis were directly transferred to H&E stains of the training set. Based on 1,221,367 annotated nuclei, a convolutional neural network for calculating tumor burden (CNNTB) was developed. For primary melanomas, precision of annotation was 100% (95%CI, 99% to 100%) for tumor cells and 99% (95%CI, 98% to 100%) for normal cells. Due to low or missing tumor-cell SOX10 positivity, precision for normal cells was markedly reduced in lymph-node and organ metastases compared with primary melanomas (p < 0.001). Compared with stereological counts within skin lesions, mean difference in tumor burden was 6% (95%CI, -1% to 13%, p = 0.10) for CNNTB and 16% (95%CI, 4% to 28%, p = 0.02) for pathologists. Conclusively, the technique produced a large annotated H&E training set with high quality within a reasonable timeframe for primary melanomas and subcutaneous metastases. For these lesion types, the training set generated a high-performing CNNTB, which was superior to the routine assessments of pathologists.
Collapse
Affiliation(s)
- Patricia Switten Nielsen
- Department of Pathology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 35, DK-8200 Aarhus, Denmark
- Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 82, DK-8200 Aarhus, Denmark
| | - Jeanette Baehr Georgsen
- Department of Pathology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 35, DK-8200 Aarhus, Denmark
- Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 82, DK-8200 Aarhus, Denmark
| | - Mads Sloth Vinding
- Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 82, DK-8200 Aarhus, Denmark
- Center of Functionally Integrative Neuroscience, Aarhus University Hospital, Palle Juul-Jensens Boulevard 99, DK-8200 Aarhus, Denmark
| | - Lasse Riis Østergaard
- Department of Health Science and Technology, Aalborg University, Fredrik Bajers Vej 7E, DK-9220 Aalborg, Denmark
| | - Torben Steiniche
- Department of Pathology, Aarhus University Hospital, Palle Juul-Jensens Boulevard 35, DK-8200 Aarhus, Denmark
- Department of Clinical Medicine, Aarhus University, Palle Juul-Jensens Boulevard 82, DK-8200 Aarhus, Denmark
| |
Collapse
|
42
|
Vasiljević J, Nisar Z, Feuerhake F, Wemmert C, Lampert T. CycleGAN for virtual stain transfer: Is seeing really believing? Artif Intell Med 2022; 133:102420. [PMID: 36328671 DOI: 10.1016/j.artmed.2022.102420] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2021] [Revised: 03/16/2022] [Accepted: 10/02/2022] [Indexed: 01/18/2023]
Abstract
Digital Pathology is an area prone to high variation due to multiple factors which can strongly affect diagnostic quality and visual appearance of the Whole-Slide-Images (WSIs). The state-of-the art methods to deal with such variation tend to address this through style-transfer inspired approaches. Usually, these solutions directly apply successful approaches from the literature, potentially with some task-related modifications. The majority of the obtained results are visually convincing, however, this paper shows that this is not a guarantee that such images can be directly used for either medical diagnosis or reducing domain shift.This article shows that slight modification in a stain transfer architecture, such as a choice of normalisation layer, while resulting in a variety of visually appealing results, surprisingly greatly effects the ability of a stain transfer model to reduce domain shift. By extensive qualitative and quantitative evaluations, we confirm that translations resulting from different stain transfer architectures are distinct from each other and from the real samples. Therefore conclusions made by visual inspection or pretrained model evaluation might be misleading.
Collapse
Affiliation(s)
- Jelica Vasiljević
- ICube, University of Strasbourg, CNRS (UMR 7357), France; University of Belgrade, Belgrade, Serbia; Faculty of Science, University of Kragujevac, Kragujevac, Serbia.
| | - Zeeshan Nisar
- ICube, University of Strasbourg, CNRS (UMR 7357), France
| | - Friedrich Feuerhake
- Institute of Pathology, Hannover Medical School, Germany; University Clinic, Freiburg, Germany
| | - Cédric Wemmert
- ICube, University of Strasbourg, CNRS (UMR 7357), France
| | - Thomas Lampert
- ICube, University of Strasbourg, CNRS (UMR 7357), France
| |
Collapse
|
43
|
Deep learning models for histologic grading of breast cancer and association with disease prognosis. NPJ Breast Cancer 2022; 8:113. [PMID: 36192400 PMCID: PMC9530224 DOI: 10.1038/s41523-022-00478-y] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2022] [Accepted: 09/01/2022] [Indexed: 12/02/2022] Open
Abstract
Histologic grading of breast cancer involves review and scoring of three well-established morphologic features: mitotic count, nuclear pleomorphism, and tubule formation. Taken together, these features form the basis of the Nottingham Grading System which is used to inform breast cancer characterization and prognosis. In this study, we develop deep learning models to perform histologic scoring of all three components using digitized hematoxylin and eosin-stained slides containing invasive breast carcinoma. We first evaluate model performance using pathologist-based reference standards for each component. To complement this typical approach to evaluation, we further evaluate the deep learning models via prognostic analyses. The individual component models perform at or above published benchmarks for algorithm-based grading approaches, achieving high concordance rates with pathologist grading. Further, prognostic performance using deep learning-based grading is on par with that of pathologists performing review of matched slides. By providing scores for each component feature, the deep-learning based approach also provides the potential to identify the grading components contributing most to prognostic value. This may enable optimized prognostic models, opportunities to improve access to consistent grading, and approaches to better understand the links between histologic features and clinical outcomes in breast cancer.
Collapse
|
44
|
Zhou W, Deng Z, Liu Y, Shen H, Deng H, Xiao H. Global Research Trends of Artificial Intelligence on Histopathological Images: A 20-Year Bibliometric Analysis. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:ijerph191811597. [PMID: 36141871 PMCID: PMC9517580 DOI: 10.3390/ijerph191811597] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/03/2022] [Revised: 08/31/2022] [Accepted: 09/01/2022] [Indexed: 06/13/2023]
Abstract
Cancer has become a major threat to global health care. With the development of computer science, artificial intelligence (AI) has been widely applied in histopathological images (HI) analysis. This study analyzed the publications of AI in HI from 2001 to 2021 by bibliometrics, exploring the research status and the potential popular directions in the future. A total of 2844 publications from the Web of Science Core Collection were included in the bibliometric analysis. The country/region, institution, author, journal, keyword, and references were analyzed by using VOSviewer and CiteSpace. The results showed that the number of publications has grown rapidly in the last five years. The USA is the most productive and influential country with 937 publications and 23,010 citations, and most of the authors and institutions with higher numbers of publications and citations are from the USA. Keyword analysis showed that breast cancer, prostate cancer, colorectal cancer, and lung cancer are the tumor types of greatest concern. Co-citation analysis showed that classification and nucleus segmentation are the main research directions of AI-based HI studies. Transfer learning and self-supervised learning in HI is on the rise. This study performed the first bibliometric analysis of AI in HI from multiple indicators, providing insights for researchers to identify key cancer types and understand the research trends of AI application in HI.
Collapse
Affiliation(s)
- Wentong Zhou
- Center for System Biology, Data Sciences, and Reproductive Health, School of Basic Medical Science, Central South University, Changsha 410031, China
| | - Ziheng Deng
- Center for System Biology, Data Sciences, and Reproductive Health, School of Basic Medical Science, Central South University, Changsha 410031, China
| | - Yong Liu
- Center for System Biology, Data Sciences, and Reproductive Health, School of Basic Medical Science, Central South University, Changsha 410031, China
| | - Hui Shen
- Tulane Center of Biomedical Informatics and Genomics, Deming Department of Medicine, School of Medicine, Tulane University School, New Orleans, LA 70112, USA
| | - Hongwen Deng
- Tulane Center of Biomedical Informatics and Genomics, Deming Department of Medicine, School of Medicine, Tulane University School, New Orleans, LA 70112, USA
| | - Hongmei Xiao
- Center for System Biology, Data Sciences, and Reproductive Health, School of Basic Medical Science, Central South University, Changsha 410031, China
| |
Collapse
|
45
|
Noothout JMH, Lessmann N, van Eede MC, van Harten LD, Sogancioglu E, Heslinga FG, Veta M, van Ginneken B, Išgum I. Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation. J Med Imaging (Bellingham) 2022; 9:052407. [PMID: 35692896 PMCID: PMC9142841 DOI: 10.1117/1.jmi.9.5.052407] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2021] [Accepted: 05/12/2022] [Indexed: 06/21/2024] Open
Abstract
Purpose: Ensembles of convolutional neural networks (CNNs) often outperform a single CNN in medical image segmentation tasks, but inference is computationally more expensive and makes ensembles unattractive for some applications. We compared the performance of differently constructed ensembles with the performance of CNNs derived from these ensembles using knowledge distillation, a technique for reducing the footprint of large models such as ensembles. Approach: We investigated two different types of ensembles, namely, diverse ensembles of networks with three different architectures and two different loss-functions, and uniform ensembles of networks with the same architecture but initialized with different random seeds. For each ensemble, additionally, a single student network was trained to mimic the class probabilities predicted by the teacher model, the ensemble. We evaluated the performance of each network, the ensembles, and the corresponding distilled networks across three different publicly available datasets. These included chest computed tomography scans with four annotated organs of interest, brain magnetic resonance imaging (MRI) with six annotated brain structures, and cardiac cine-MRI with three annotated heart structures. Results: Both uniform and diverse ensembles obtained better results than any of the individual networks in the ensemble. Furthermore, applying knowledge distillation resulted in a single network that was smaller and faster without compromising performance compared with the ensemble it learned from. The distilled networks significantly outperformed the same network trained with reference segmentation instead of knowledge distillation. Conclusion: Knowledge distillation can compress segmentation ensembles of uniform or diverse composition into a single CNN while maintaining the performance of the ensemble.
Collapse
Affiliation(s)
- Julia M. H. Noothout
- Amsterdam University Medical Center, University of Amsterdam, Department of Biomedical Engineering and Physics, Amsterdam, The Netherlands
| | - Nikolas Lessmann
- Radboud University Medical Center, Department of Medical Imaging, Nijmegen, The Netherlands
| | - Matthijs C. van Eede
- Amsterdam University Medical Center, University of Amsterdam, Department of Biomedical Engineering and Physics, Amsterdam, The Netherlands
| | - Louis D. van Harten
- Amsterdam University Medical Center, University of Amsterdam, Department of Biomedical Engineering and Physics, Amsterdam, The Netherlands
| | - Ecem Sogancioglu
- Radboud University Medical Center, Department of Medical Imaging, Nijmegen, The Netherlands
| | - Friso G. Heslinga
- Eindhoven University of Technology, Department of Biomedical Engineering, Eindhoven, The Netherlands
| | - Mitko Veta
- Eindhoven University of Technology, Department of Biomedical Engineering, Eindhoven, The Netherlands
| | - Bram van Ginneken
- Radboud University Medical Center, Department of Medical Imaging, Nijmegen, The Netherlands
| | - Ivana Išgum
- Amsterdam University Medical Center, University of Amsterdam, Department of Biomedical Engineering and Physics, Amsterdam, The Netherlands
- Amsterdam University Medical Center, University of Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam, The Netherlands
- Amsterdam University Medical Center, University of Amsterdam, Amsterdam Cardiovascular Sciences, Heart Failure & Arrhythmias, Amsterdam, The Netherlands
- University of Amsterdam, Informatics Institute, Amsterdam, The Netherlands
| |
Collapse
|
46
|
Li Y, Xue Y, Li L, Zhang X, Qian X. Domain Adaptive Box-Supervised Instance Segmentation Network for Mitosis Detection. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:2469-2485. [PMID: 35389862 DOI: 10.1109/tmi.2022.3165518] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
The number of mitotic cells present in histopathological slides is an important predictor of tumor proliferation in the diagnosis of breast cancer. However, the current approaches can hardly perform precise pixel-level prediction for mitosis datasets with only weak labels (i.e., only provide the centroid location of mitotic cells), and take no account of the large domain gap across histopathological slides from different pathology laboratories. In this work, we propose a Domain adaptive Box-supervised Instance segmentation Network (DBIN) to address the above issues. In DBIN, we propose a high-performance Box-supervised Instance-Aware (BIA) head with the core idea of redesigning three box-supervised mask loss terms. Furthermore, we add a Pseudo-Mask-supervised Semantic (PMS) head for enriching characteristics extracted from underlying feature maps. Besides, we align the pixel-level feature distributions between source and target domains by a Cross-Domain Adaptive Module (CDAM), so as to adapt the detector learned from one lab can work well on unlabeled data from another lab. The proposed method achieves state-of-the-art performance across four mainstream datasets. A series of analysis and experiments show that our proposed BIA and PMS head can accomplish mitosis pixel-wise localization under weak supervision, and we can boost the generalization ability of our model by CDAM.
Collapse
|
47
|
Wu Y, Koyuncu CF, Toro P, Corredor G, Feng Q, Buzzy C, Old M, Teknos T, Connelly ST, Jordan RC, Lang Kuhs KA, Lu C, Lewis JS, Madabhushi A. A machine learning model for separating epithelial and stromal regions in oral cavity squamous cell carcinomas using H&E-stained histology images: A multi-center, retrospective study. Oral Oncol 2022; 131:105942. [PMID: 35689952 DOI: 10.1016/j.oraloncology.2022.105942] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2022] [Revised: 04/12/2022] [Accepted: 05/24/2022] [Indexed: 01/30/2023]
Abstract
OBJECTIVE Tissue slides from Oral cavity squamous cell carcinoma (OC-SCC), particularly the epithelial regions, hold morphologic features that are both diagnostic and prognostic. Yet, previously developed approaches for automated epithelium segmentation in OC-SCC have not been independently tested in a multi-center setting. In this study, we aimed to investigate the effectiveness and applicability of a convolutional neural network (CNN) model to perform epithelial segmentation using digitized H&E-stained diagnostic slides from OC-SCC patients in a multi-center setting. METHODS A CNN model was developed to segment the epithelial regions of digitized slides (n = 810), retrospectively collected from five different centers. Deep learning models were trained and validated using well-annotated tissue microarray (TMA) images (n = 212) at various magnifications. The best performing model was locked down and used for independent testing with a total of 478 whole-slide images (WSIs). Manually annotated epithelial regions were used as the reference standard for evaluation. We also compared the model generated results with IHC-stained epithelium (n = 120) as the reference. RESULTS The locked-down CNN model trained on the TMA image training cohorts with 10x magnification achieved the best segmentation performance. The locked-down model performed consistently and yielded Pixel Accuracy, Recall Rate, Precision Rate, and Dice Coefficient that ranged from 95.8% to 96.6%, 79.1% to 93.8%, 85.7% to 89.3%, and 82.3% to 89.0%, respectively for the three independent testing WSI cohorts. CONCLUSION The automated model achieved a consistently accurate performance for automated epithelial region segmentation compared to manual annotations. This model could be integrated into a computer-aided diagnosis or prognosis system.
Collapse
Affiliation(s)
- Yuxin Wu
- Shandong Junteng Medical Technology Co., Ltd, Jinan, China; College of Computer Science, Shaanxi Normal University, Xian, China
| | - Can F Koyuncu
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA; Louis Stokes Cleveland VA Medical Center, Cleveland, OH, USA
| | - Paula Toro
- Department of Pathology, Cleveland Clinic, OH, USA
| | - German Corredor
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA; Louis Stokes Cleveland VA Medical Center, Cleveland, OH, USA
| | - Qianyu Feng
- College of Computer Science, Shaanxi Normal University, Xian, China
| | - Christina Buzzy
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA
| | - Matthew Old
- Department of Otolaryngology, Ohio State University Medical Center, OH, USA
| | - Theodoros Teknos
- Department of Otolaryngology, University Hospitals Cleveland Medical Center, Cleveland, OH, USA
| | - Stephen Thaddeus Connelly
- Department of Oral and Maxillofacial Surgery, San Francisco Veterans Affairs Health Care System, University of California, San Francisco, San Francisco, CA, USA
| | - Richard C Jordan
- Departments of Orofacial Sciences, Pathology and Radiation Oncology, University of California San Francisco, CA, USA
| | - Krystle A Lang Kuhs
- Department of Epidemiology, College of Public Health, University of Kentucky, Lexington, KY, USA; Department of Medicine, Vanderbilt University Medical Cancer, Nashville, TN, USA
| | - Cheng Lu
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA.
| | - James S Lewis
- Department of Pathology, Microbiology, and Immunology, Vanderbilt University Medical Center, Nashville, TN, USA; Department of Otolaryngology - Head and Neck Surgery, Vanderbilt University Medical Center, Nashville, TN, USA.
| | - Anant Madabhushi
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA; Louis Stokes Cleveland VA Medical Center, Cleveland, OH, USA.
| |
Collapse
|
48
|
Cong C, Liu S, Di Ieva A, Pagnucco M, Berkovsky S, Song Y. Colour adaptive generative networks for stain normalisation of histopathology images. Med Image Anal 2022; 82:102580. [DOI: 10.1016/j.media.2022.102580] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2021] [Revised: 05/12/2022] [Accepted: 08/11/2022] [Indexed: 10/31/2022]
|
49
|
Huang H, Zhou G, Liu X, Deng L, Wu C, Zhang D, Liu H. Contrastive learning-based computational histopathology predict differential expression of cancer driver genes. Brief Bioinform 2022; 23:6651307. [PMID: 35901472 DOI: 10.1093/bib/bbac294] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2022] [Revised: 06/16/2022] [Accepted: 06/29/2022] [Indexed: 11/14/2022] Open
Abstract
MOTIVATION Digital pathological analysis is run as the main examination used for cancer diagnosis. Recently, deep learning-driven feature extraction from pathology images is able to detect genetic variations and tumor environment, but few studies focus on differential gene expression in tumor cells. RESULTS In this paper, we propose a self-supervised contrastive learning framework, HistCode, to infer differential gene expression from whole slide images (WSIs). We leveraged contrastive learning on large-scale unannotated WSIs to derive slide-level histopathological features in latent space, and then transfer it to tumor diagnosis and prediction of differentially expressed cancer driver genes. Our experiments showed that our method outperformed other state-of-the-art models in tumor diagnosis tasks, and also effectively predicted differential gene expression. Interestingly, we found the genes with higher fold change can be more precisely predicted. To intuitively illustrate the ability to extract informative features from pathological images, we spatially visualized the WSIs colored by the attention scores of image tiles. We found that the tumor and necrosis areas were highly consistent with the annotations of experienced pathologists. Moreover, the spatial heatmap generated by lymphocyte-specific gene expression patterns was also consistent with the manually labeled WSIs.
Collapse
Affiliation(s)
- Haojie Huang
- School of Computer Science and Engineering, Central South University, 410075, Changsha, China
| | - Gongming Zhou
- School of Computer Science and Engineering, Central South University, 410075, Changsha, China
| | - Xuejun Liu
- School of Computer Science and Technology, Nanjing Tech University, 211816, Nanjing, China
| | - Lei Deng
- School of Computer Science and Engineering, Central South University, 410075, Changsha, China
| | - Chen Wu
- The third affiliated hospital of Soochow University, 213100, Changzhou, China
| | - Dachuan Zhang
- The third affiliated hospital of Soochow University, 213100, Changzhou, China
| | - Hui Liu
- School of Computer Science and Technology, Nanjing Tech University, 211816, Nanjing, China
| |
Collapse
|
50
|
Ray I, Raipuria G, Singhal N. Rethinking ImageNet Pre-training for Computational Histopathology. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:3059-3062. [PMID: 36086630 DOI: 10.1109/embc48229.2022.9871687] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Transfer learning from ImageNet pretrained weights is widely used when training Deep Learning models on a Histopathology dataset. However, the visual features of the two domains are different. Rather than ImageNet pretrained weights, pre-training on a Histopathology dataset may provide better initialization. To prove this hypothesis, we train two commonly used Deep Learning model architectures - ResNet and DenseNet on a complex Histopathology classification dataset, and compare transfer learning performance with ImageNet pretrained weights. Based on the fine-tuning on three histopathology datasets including two different stains (H&E and IHC), we show that the domain specific pretrained weights are better suited for transfer learning. This is reflected by higher performance, lower training time as well as better feature reuse. Clinical Relevance - The paper establishes merit of using Histopathology domain specific pretrained weights rather than ImageNet pretrained weights.
Collapse
|