1
|
Kaczmarzyk JR, O'Callaghan A, Inglis F, Gat S, Kurc T, Gupta R, Bremer E, Bankhead P, Saltz JH. Open and reusable deep learning for pathology with WSInfer and QuPath. NPJ Precis Oncol 2024; 8:9. [PMID: 38200147 PMCID: PMC10781748 DOI: 10.1038/s41698-024-00499-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2023] [Accepted: 12/21/2023] [Indexed: 01/12/2024] Open
Abstract
Digital pathology has seen a proliferation of deep learning models in recent years, but many models are not readily reusable. To address this challenge, we developed WSInfer: an open-source software ecosystem designed to streamline the sharing and reuse of deep learning models for digital pathology. The increased access to trained models can augment research on the diagnostic, prognostic, and predictive capabilities of digital pathology.
Collapse
Affiliation(s)
- Jakub R Kaczmarzyk
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA.
| | - Alan O'Callaghan
- Centre for Genomic & Experimental Medicine, Institute of Genetics and Cancer, The University of Edinburgh, Edinburgh, UK
| | - Fiona Inglis
- Centre for Genomic & Experimental Medicine, Institute of Genetics and Cancer, The University of Edinburgh, Edinburgh, UK
| | - Swarad Gat
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Tahsin Kurc
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Erich Bremer
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Peter Bankhead
- Centre for Genomic & Experimental Medicine, Institute of Genetics and Cancer, The University of Edinburgh, Edinburgh, UK
- Edinburgh Pathology and CRUK Scotland Centre, Institute of Genetics and Cancer, The University of Edinburgh, Edinburgh, UK
| | - Joel H Saltz
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| |
Collapse
|
2
|
Ricker CA, Meli K, Van Allen EM. Historical perspective and future directions: computational science in immuno-oncology. J Immunother Cancer 2024; 12:e008306. [PMID: 38191244 PMCID: PMC10826578 DOI: 10.1136/jitc-2023-008306] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/07/2023] [Indexed: 01/10/2024] Open
Abstract
Immuno-oncology holds promise for transforming patient care having achieved durable clinical response rates across a variety of advanced and metastatic cancers. Despite these achievements, only a minority of patients respond to immunotherapy, underscoring the importance of elucidating molecular mechanisms responsible for response and resistance to inform the development and selection of treatments. Breakthroughs in molecular sequencing technologies have led to the generation of an immense amount of genomic and transcriptomic sequencing data that can be mined to uncover complex tumor-immune interactions using computational tools. In this review, we discuss existing and emerging computational methods that contextualize the composition and functional state of the tumor microenvironment, infer the reactivity and clonal dynamics from reconstructed immune cell receptor repertoires, and predict the antigenic landscape for immune cell recognition. We further describe the advantage of multi-omics analyses for capturing multidimensional relationships and artificial intelligence techniques for integrating omics data with histopathological and radiological images to encapsulate patterns of treatment response and tumor-immune biology. Finally, we discuss key challenges impeding their widespread use and clinical application and conclude with future perspectives. We are hopeful that this review will both serve as a guide for prospective researchers seeking to use existing tools for scientific discoveries and inspire the optimization or development of novel tools to enhance precision, ultimately expediting advancements in immunotherapy that improve patient survival and quality of life.
Collapse
Affiliation(s)
- Cora A Ricker
- Medical Oncology, Dana-Farber Cancer Institute, Boston, Massachusetts, USA
| | - Kevin Meli
- Medical Oncology, Dana-Farber Cancer Institute, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
| | | |
Collapse
|
3
|
Glüge S, Balabanov S, Koelzer VH, Ott T. Evaluation of deep learning training strategies for the classification of bone marrow cell images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 243:107924. [PMID: 37979517 DOI: 10.1016/j.cmpb.2023.107924] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/11/2023] [Revised: 09/28/2023] [Accepted: 11/06/2023] [Indexed: 11/20/2023]
Abstract
BACKGROUND AND OBJECTIVE The classification of bone marrow (BM) cells by light microscopy is an important cornerstone of hematological diagnosis, performed thousands of times a day by highly trained specialists in laboratories worldwide. As the manual evaluation of blood or BM smears is very time-consuming and prone to inter-observer variation, new reliable automated systems are needed. METHODS We aim to improve the automatic classification performance of hematological cell types. Therefore, we evaluate four state-of-the-art Convolutional Neural Network (CNN) architectures on a dataset of 171,374 microscopic cytological single-cell images obtained from BM smears from 945 patients diagnosed with a variety of hematological diseases. We further evaluate the effect of an in-domain vs. out-of-domain pre-training, and assess whether class activation maps provide human-interpretable explanations for the models' predictions. RESULTS The best performing pre-trained model (Regnet_y_32gf) yields a mean precision, recall, and F1 scores of 0.787±0.060, 0.755±0.061, and 0.762±0.050, respectively. This is a 53.5% improvement in precision and 7.3% improvement in recall over previous results with CNNs (ResNeXt-50) that were trained from scratch. The out-of-domain pre-training apparently yields general feature extractors/filters that apply very well to the BM cell classification use case. The class activation maps on cell types with characteristic morphological features were found to be consistent with the explanations of a human domain expert. For example, the Auer rods in the cytoplasm were the predictive cellular feature for correctly classified images of faggot cells. CONCLUSIONS Our study provides data that can help hematology laboratories to choose the optimal training strategy for blood cell classification deep learning models to improve computer-assisted blood and bone marrow cell identification. It also highlights the need for more specific training data, i.e. images of difficult-to-classify classes, including cells labeled with disease information.
Collapse
Affiliation(s)
- Stefan Glüge
- Institute of Computational Life Sciences, Zurich University of Applied Sciences, Schloss 1, 8820 Wädenswil, Switzerland.
| | - Stefan Balabanov
- Department of Medical Oncology and Haematology, University Hospital Zurich and University of Zurich, Rämistrasse 100, 8091 Zurich, Switzerland
| | - Viktor Hendrik Koelzer
- Department of Pathology and Molecular Pathology, University Hospital Zurich and University of Zurich, Schmelzbergstrasse 12, 8091 Zurich, Switzerland
| | - Thomas Ott
- Institute of Computational Life Sciences, Zurich University of Applied Sciences, Schloss 1, 8820 Wädenswil, Switzerland
| |
Collapse
|
4
|
Verghese G, Lennerz JK, Ruta D, Ng W, Thavaraj S, Siziopikou KP, Naidoo T, Rane S, Salgado R, Pinder SE, Grigoriadis A. Computational pathology in cancer diagnosis, prognosis, and prediction - present day and prospects. J Pathol 2023; 260:551-563. [PMID: 37580849 PMCID: PMC10785705 DOI: 10.1002/path.6163] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 06/12/2023] [Accepted: 06/17/2023] [Indexed: 08/16/2023]
Abstract
Computational pathology refers to applying deep learning techniques and algorithms to analyse and interpret histopathology images. Advances in artificial intelligence (AI) have led to an explosion in innovation in computational pathology, ranging from the prospect of automation of routine diagnostic tasks to the discovery of new prognostic and predictive biomarkers from tissue morphology. Despite the promising potential of computational pathology, its integration in clinical settings has been limited by a range of obstacles including operational, technical, regulatory, ethical, financial, and cultural challenges. Here, we focus on the pathologists' perspective of computational pathology: we map its current translational research landscape, evaluate its clinical utility, and address the more common challenges slowing clinical adoption and implementation. We conclude by describing contemporary approaches to drive forward these techniques. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.
Collapse
Affiliation(s)
- Gregory Verghese
- School of Cancer & Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
- The Breast Cancer Now Research Unit, School of Cancer and Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
| | - Jochen K Lennerz
- Center for Integrated Diagnostics, Department of PathologyMassachusetts General Hospital/Harvard Medical SchoolBostonMAUSA
| | - Danny Ruta
- Guy's CancerGuy's and St Thomas’ NHS Foundation TrustLondonUK
| | - Wen Ng
- Department of Cellular PathologyGuy's and St Thomas NHS Foundation TrustLondonUK
| | - Selvam Thavaraj
- Head & Neck PathologyGuy's and St Thomas NHS Foundation TrustLondonUK
- Centre for Clinical, Oral & Translational Science, Faculty of Dentistry, Oral & Craniofacial SciencesKing's College LondonLondonUK
| | - Kalliopi P Siziopikou
- Department of Pathology, Section of Breast PathologyNorthwestern University Feinberg School of MedicineChicagoILUSA
| | - Threnesan Naidoo
- Department of Laboratory Medicine and Pathology, Walter Sisulu University, Mthatha, Eastern CapeSouth Africa and Africa Health Research InstituteDurbanSouth Africa
| | - Swapnil Rane
- Department of PathologyTata Memorial Centre – ACTRECHBNINavi MumbaiIndia
- Computational Pathology, AI & Imaging LaboratoryTata Memorial Centre – ACTREC, HBNINavi MumbaiIndia
| | - Roberto Salgado
- Department of PathologyGZA–ZNA ZiekenhuizenAntwerpBelgium
- Division of ResearchPeter MacCallum Cancer CentreMelbourneVictoriaAustralia
| | - Sarah E Pinder
- School of Cancer & Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
- Department of Cellular PathologyGuy's and St Thomas NHS Foundation TrustLondonUK
| | - Anita Grigoriadis
- School of Cancer & Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
- The Breast Cancer Now Research Unit, School of Cancer and Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
| |
Collapse
|
5
|
Sadafi A, Bordukova M, Makhro A, Navab N, Bogdanova A, Marr C. RedTell: an AI tool for interpretable analysis of red blood cell morphology. Front Physiol 2023; 14:1058720. [PMID: 37304818 PMCID: PMC10250619 DOI: 10.3389/fphys.2023.1058720] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Accepted: 04/13/2023] [Indexed: 06/13/2023] Open
Abstract
Introduction: Hematologists analyze microscopic images of red blood cells to study their morphology and functionality, detect disorders and search for drugs. However, accurate analysis of a large number of red blood cells needs automated computational approaches that rely on annotated datasets, expensive computational resources, and computer science expertise. We introduce RedTell, an AI tool for the interpretable analysis of red blood cell morphology comprising four single-cell modules: segmentation, feature extraction, assistance in data annotation, and classification. Methods: Cell segmentation is performed by a trained Mask R-CNN working robustly on a wide range of datasets requiring no or minimum fine-tuning. Over 130 features that are regularly used in research are extracted for every detected red blood cell. If required, users can train task-specific, highly accurate decision tree-based classifiers to categorize cells, requiring a minimal number of annotations and providing interpretable feature importance. Results: We demonstrate RedTell's applicability and power in three case studies. In the first case study we analyze the difference of the extracted features between the cells coming from patients suffering from different diseases, in the second study we use RedTell to analyze the control samples and use the extracted features to classify cells into echinocytes, discocytes and stomatocytes and finally in the last use case we distinguish sickle cells in sickle cell disease patients. Discussion: We believe that RedTell can accelerate and standardize red blood cell research and help gain new insights into mechanisms, diagnosis, and treatment of red blood cell associated disorders.
Collapse
Affiliation(s)
- Ario Sadafi
- Institute of AI for Health, Helmholtz Zentrum München—German Research Center for Environmental Health, Neuherberg, Germany
- Chair for Computer Aided Medical Procedures & Augmented Reality, Technical University of Munich, Garching, Germany
| | - Maria Bordukova
- Institute of AI for Health, Helmholtz Zentrum München—German Research Center for Environmental Health, Neuherberg, Germany
| | - Asya Makhro
- Red Blood Cell Research Group, Institute of Veterinary Physiology, Vetsuisse Faculty and the Zurich Center for Integrative Human Physiology, University of Zurich, Zurich, Switzerland
| | - Nassir Navab
- Chair for Computer Aided Medical Procedures & Augmented Reality, Technical University of Munich, Garching, Germany
- Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, MD, United States
| | - Anna Bogdanova
- Red Blood Cell Research Group, Institute of Veterinary Physiology, Vetsuisse Faculty and the Zurich Center for Integrative Human Physiology, University of Zurich, Zurich, Switzerland
| | - Carsten Marr
- Institute of AI for Health, Helmholtz Zentrum München—German Research Center for Environmental Health, Neuherberg, Germany
| |
Collapse
|