1
|
Delgado-Coka L, Horowitz M, Torrente-Goncalves M, Roa-Peña L, Leiton CV, Hasan M, Babu S, Fassler D, Oentoro J, Bai JDK, Petricoin EF, Matrisian LM, Blais EM, Marchenko N, Allard FD, Jiang W, Larson B, Hendifar A, Chen C, Abousamra S, Samaras D, Kurc T, Saltz J, Escobar-Hoyos LF, Shroyer KR. Keratin 17 modulates the immune topography of pancreatic cancer. J Transl Med 2024; 22:443. [PMID: 38730319 PMCID: PMC11087249 DOI: 10.1186/s12967-024-05252-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2024] [Accepted: 04/29/2024] [Indexed: 05/12/2024] Open
Abstract
BACKGROUND The immune microenvironment impacts tumor growth, invasion, metastasis, and patient survival and may provide opportunities for therapeutic intervention in pancreatic ductal adenocarcinoma (PDAC). Although never studied as a potential modulator of the immune response in most cancers, Keratin 17 (K17), a biomarker of the most aggressive (basal) molecular subtype of PDAC, is intimately involved in the histogenesis of the immune response in psoriasis, basal cell carcinoma, and cervical squamous cell carcinoma. Thus, we hypothesized that K17 expression could also impact the immune cell response in PDAC, and that uncovering this relationship could provide insight to guide the development of immunotherapeutic opportunities to extend patient survival. METHODS Multiplex immunohistochemistry (mIHC) and automated image analysis based on novel computational imaging technology were used to decipher the abundance and spatial distribution of T cells, macrophages, and tumor cells, relative to K17 expression in 235 PDACs. RESULTS K17 expression had profound effects on the exclusion of intratumoral CD8+ T cells and was also associated with decreased numbers of peritumoral CD8+ T cells, CD16+ macrophages, and CD163+ macrophages (p < 0.0001). The differences in the intratumor and peritumoral CD8+ T cell abundance were not impacted by neoadjuvant therapy, tumor stage, grade, lymph node status, histologic subtype, nor KRAS, p53, SMAD4, or CDKN2A mutations. CONCLUSIONS Thus, K17 expression correlates with major differences in the immune microenvironment that are independent of any tested clinicopathologic or tumor intrinsic variables, suggesting that targeting K17-mediated immune effects on the immune system could restore the innate immunologic response to PDAC and might provide novel opportunities to restore immunotherapeutic approaches for this most deadly form of cancer.
Collapse
Affiliation(s)
- Lyanne Delgado-Coka
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
- Program of Public Health and Department of Preventative Medicine, Renaissance School of Medicine, Stony Brook University, Stony Brook, NY, USA
| | - Michael Horowitz
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
| | - Mariana Torrente-Goncalves
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
| | - Lucia Roa-Peña
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
- Department of Pathology, School of Medicine, Universidad Nacional de Colombia, Bogotá, Colombia
| | - Cindy V Leiton
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
| | - Mahmudul Hasan
- Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
| | - Sruthi Babu
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
| | - Danielle Fassler
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
| | - Jaymie Oentoro
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
| | - Ji-Dong K Bai
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
| | - Emanuel F Petricoin
- Center for Applied Proteomics and Molecular Medicine, George Mason University, Fairfax, VA, USA
- Perthera, McLean, VA, USA
| | - Lynn M Matrisian
- Scientific and Medical Affairs, Pancreatic Cancer Action Network, Manhattan Beach, CA, USA
| | | | - Natalia Marchenko
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA
| | - Felicia D Allard
- Department of Pathology, University of Arkansas for Medical Sciences, Little Rock, AR, USA
| | - Wei Jiang
- Department of Pathology and Genomic Medicine, Sidney Kimmel Cancer Center, Thomas Jefferson University Hospital, Philadelphia, PA, USA
| | - Brent Larson
- Departments of Pathology and Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Andrew Hendifar
- Departments of Pathology and Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Chao Chen
- Department of Biomedical Informatics, Renaissance School of Medicine, Stony Brook University, Stony Brook, NY, USA
| | - Shahira Abousamra
- Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
| | - Tahsin Kurc
- Department of Biomedical Informatics, Renaissance School of Medicine, Stony Brook University, Stony Brook, NY, USA
| | - Joel Saltz
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA.
- Department of Biomedical Informatics, Renaissance School of Medicine, Stony Brook University, Stony Brook, NY, USA.
| | - Luisa F Escobar-Hoyos
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA.
- Department of Therapeutic Radiology, Yale University, New Haven, CT, USA.
- Department of Molecular Biophysics and Biochemistry, Yale University, New Haven, CT, USA.
- Division of Oncology, Department of Medicine, Yale University, New Haven, CT, USA.
| | - Kenneth R Shroyer
- Department of Pathology, Renaissance School of Medicine, Stony Brook University, 101 Nicolls Road, Stony Brook, NY, 11794, USA.
| |
Collapse
|
2
|
Delgado-Coka LA, Horowitz M, Torrente-Goncalves M, Roa-Peña L, Leiton CV, Hasan M, Babu S, Fassler D, Oentoro J, Karen Bai JD, Petricoin EF, Matrisian LM, Blais EM, Marchenko N, Allard FD, Jiang W, Larson B, Hendifar A, Chen C, Abousamra S, Samaras D, Kurc T, Saltz J, Escobar-Hoyos LF, Shroyer K. Keratin 17 modulates the immune topography of pancreatic cancer. Res Sq 2024:rs.3.rs-3886691. [PMID: 38464123 PMCID: PMC10925455 DOI: 10.21203/rs.3.rs-3886691/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
Background The immune microenvironment impacts tumor growth, invasion, metastasis, and patient survival and may provide opportunities for therapeutic intervention in pancreatic ductal adenocarcinoma (PDAC). Although never studied as a potential modulator of the immune response in most cancers, Keratin 17 (K17), a biomarker of the most aggressive (basal) molecular subtype of PDAC, is intimately involved in the histogenesis of the immune response in psoriasis, basal cell carcinoma, and cervical squamous cell carcinoma. Thus, we hypothesized that K17 expression could also impact the immune cell response in PDAC, and that uncovering this relationship could provide insight to guide the development of immunotherapeutic opportunities to extend patient survival. Methods Multiplex immunohistochemistry (mIHC) and automated image analysis based on novel computational imaging technology were used to decipher the abundance and spatial distribution of T cells, macrophages, and tumor cells, relative to K17 expression in 235 PDACs. Results K17 expression had profound effects on the exclusion of intratumoral CD8 + T cells and was also associated with decreased numbers of peritumoral CD8 + T cells, CD16 + macrophages, and CD163 + macrophages (p < 0.0001). The differences in the intratumor and peritumoral CD8 + T cell abundance were not impacted by neoadjuvant therapy, tumor stage, grade, lymph node status, histologic subtype, nor KRAS, p53, SMAD4, or CDKN2A mutations. Conclusions Thus, K17 expression correlates with major differences in the immune microenvironment that are independent of any tested clinicopathologic or tumor intrinsic variables, suggesting that targeting K17-mediated immune effects on the immune system could restore the innate immunologic response to PDAC and might provide novel opportunities to restore immunotherapeutic approaches for this most deadly form of cancer.
Collapse
|
3
|
Kaczmarzyk JR, Gupta R, Kurc TM, Abousamra S, Saltz JH, Koo PK. ChampKit: A framework for rapid evaluation of deep neural networks for patch-based histopathology classification. Comput Methods Programs Biomed 2023; 239:107631. [PMID: 37271050 PMCID: PMC11093625 DOI: 10.1016/j.cmpb.2023.107631] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/19/2023] [Revised: 04/23/2023] [Accepted: 05/28/2023] [Indexed: 06/06/2023]
Abstract
BACKGROUND AND OBJECTIVE Histopathology is the gold standard for diagnosis of many cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for many tasks, including the detection of immune cells and microsatellite instability. However, it remains difficult to identify optimal models and training configurations for different histopathology classification tasks due to the abundance of available architectures and the lack of systematic evaluations. Our objective in this work is to present a software tool that addresses this need and enables robust, systematic evaluation of neural network models for patch classification in histology in a light-weight, easy-to-use package for both algorithm developers and biomedical researchers. METHODS Here we present ChampKit (Comprehensive Histopathology Assessment of Model Predictions toolKit): an extensible, fully reproducible evaluation toolkit that is a one-stop-shop to train and evaluate deep neural networks for patch classification. ChampKit curates a broad range of public datasets. It enables training and evaluation of models supported by timm directly from the command line, without the need for users to write any code. External models are enabled through a straightforward API and minimal coding. As a result, Champkit facilitates the evaluation of existing and new models and deep learning architectures on pathology datasets, making it more accessible to the broader scientific community. To demonstrate the utility of ChampKit, we establish baseline performance for a subset of possible models that could be employed with ChampKit, focusing on several popular deep learning models, namely ResNet18, ResNet50, and R26-ViT, a hybrid vision transformer. In addition, we compare each model trained either from random weight initialization or with transfer learning from ImageNet pretrained models. For ResNet18, we also consider transfer learning from a self-supervised pretrained model. RESULTS The main result of this paper is the ChampKit software. Using ChampKit, we were able to systemically evaluate multiple neural networks across six datasets. We observed mixed results when evaluating the benefits of pretraining versus random intialization, with no clear benefit except in the low data regime, where transfer learning was found to be beneficial. Surprisingly, we found that transfer learning from self-supervised weights rarely improved performance, which is counter to other areas of computer vision. CONCLUSIONS Choosing the right model for a given digital pathology dataset is nontrivial. ChampKit provides a valuable tool to fill this gap by enabling the evaluation of hundreds of existing (or user-defined) deep learning models across a variety of pathology tasks. Source code and data for the tool are freely accessible at https://github.com/SBU-BMI/champkit.
Collapse
Affiliation(s)
- Jakub R Kaczmarzyk
- Department of Biomedical Informatics, Stony Brook Medicine, 101 Nicolls Rd, Stony Brook, 11794, NY, USA; Simons Center for Quantitative Biology, 1 Bungtown Rd, Cold Spring Harbor, 11724, NY, USA.
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook Medicine, 101 Nicolls Rd, Stony Brook, 11794, NY, USA
| | - Tahsin M Kurc
- Department of Biomedical Informatics, Stony Brook Medicine, 101 Nicolls Rd, Stony Brook, 11794, NY, USA
| | - Shahira Abousamra
- Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
| | - Joel H Saltz
- Department of Biomedical Informatics, Stony Brook Medicine, 101 Nicolls Rd, Stony Brook, 11794, NY, USA.
| | - Peter K Koo
- Simons Center for Quantitative Biology, 1 Bungtown Rd, Cold Spring Harbor, 11724, NY, USA.
| |
Collapse
|
4
|
Abousamra S, Gupta R, Kurc T, Samaras D, Saltz J, Chen C. Topology-Guided Multi-Class Cell Context Generation for Digital Pathology. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2023; 2023:3323-3333. [PMID: 38741683 PMCID: PMC11090253 DOI: 10.1109/cvpr52729.2023.00324] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
In digital pathology, the spatial context of cells is important for cell classification, cancer diagnosis and prognosis. To model such complex cell context, however, is challenging. Cells form different mixtures, lineages, clusters and holes. To model such structural patterns in a learnable fashion, we introduce several mathematical tools from spatial statistics and topological data analysis. We incorporate such structural descriptors into a deep generative model as both conditional inputs and a differentiable loss. This way, we are able to generate high quality multi-class cell layouts for the first time. We show that the topology-rich cell layouts can be used for data augmentation and improve the performance of downstream tasks such as cell classification.
Collapse
Affiliation(s)
| | - Rajarsi Gupta
- Stony Brook University, Department of Biomedical Informatics, USA
| | - Tahsin Kurc
- Stony Brook University, Department of Biomedical Informatics, USA
| | | | - Joel Saltz
- Stony Brook University, Department of Biomedical Informatics, USA
| | - Chao Chen
- Stony Brook University, Department of Biomedical Informatics, USA
| |
Collapse
|
5
|
Abousamra S, Gupta R, Hou L, Batiste R, Zhao T, Shankar A, Rao A, Chen C, Samaras D, Kurc T, Saltz J. Deep Learning-Based Mapping of Tumor Infiltrating Lymphocytes in Whole Slide Images of 23 Types of Cancer. Front Oncol 2022; 11:806603. [PMID: 35251953 PMCID: PMC8889499 DOI: 10.3389/fonc.2021.806603] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2021] [Accepted: 12/31/2021] [Indexed: 12/12/2022] Open
Abstract
The role of tumor infiltrating lymphocytes (TILs) as a biomarker to predict disease progression and clinical outcomes has generated tremendous interest in translational cancer research. We present an updated and enhanced deep learning workflow to classify 50x50 um tiled image patches (100x100 pixels at 20x magnification) as TIL positive or negative based on the presence of 2 or more TILs in gigapixel whole slide images (WSIs) from the Cancer Genome Atlas (TCGA). This workflow generates TIL maps to study the abundance and spatial distribution of TILs in 23 different types of cancer. We trained three state-of-the-art, popular convolutional neural network (CNN) architectures (namely VGG16, Inception-V4, and ResNet-34) with a large volume of training data, which combined manual annotations from pathologists (strong annotations) and computer-generated labels from our previously reported first-generation TIL model for 13 cancer types (model-generated annotations). Specifically, this training dataset contains TIL positive and negative patches from cancers in additional organ sites and curated data to help improve algorithmic performance by decreasing known false positives and false negatives. Our new TIL workflow also incorporates automated thresholding to convert model predictions into binary classifications to generate TIL maps. The new TIL models all achieve better performance with improvements of up to 13% in accuracy and 15% in F-score. We report these new TIL models and a curated dataset of TIL maps, referred to as TIL-Maps-23, for 7983 WSIs spanning 23 types of cancer with complex and diverse visual appearances, which will be publicly available along with the code to evaluate performance. Code Available at:https://github.com/ShahiraAbousamra/til_classification.
Collapse
Affiliation(s)
- Shahira Abousamra
- Department of Computer Science, Stony Brook University, Stony Brook, NY, United States
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, United States
| | - Le Hou
- Department of Computer Science, Stony Brook University, Stony Brook, NY, United States
| | - Rebecca Batiste
- Department of Pathology, Stony Brook University, Stony Brook, NY, United States
| | - Tianhao Zhao
- Department of Pathology, Stony Brook University, Stony Brook, NY, United States
| | - Anand Shankar
- Department of Computational Medicine & Bioinformatics, University of Michigan, Ann Arbor, MI, United States
| | - Arvind Rao
- Department of Computational Medicine & Bioinformatics, University of Michigan, Ann Arbor, MI, United States
| | - Chao Chen
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, United States
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, NY, United States
| | - Tahsin Kurc
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, United States
| | - Joel Saltz
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, United States
| |
Collapse
|
6
|
Foran DJ, Durbin EB, Chen W, Sadimin E, Sharma A, Banerjee I, Kurc T, Li N, Stroup AM, Harris G, Gu A, Schymura M, Gupta R, Bremer E, Balsamo J, DiPrima T, Wang F, Abousamra S, Samaras D, Hands I, Ward K, Saltz JH. An Expandable Informatics Framework for Enhancing Central Cancer Registries with Digital Pathology Specimens, Computational Imaging Tools, and Advanced Mining Capabilities. J Pathol Inform 2022; 13:5. [PMID: 35136672 PMCID: PMC8794027 DOI: 10.4103/jpi.jpi_31_21] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2021] [Accepted: 04/30/2021] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Population-based state cancer registries are an authoritative source for cancer statistics in the United States. They routinely collect a variety of data, including patient demographics, primary tumor site, stage at diagnosis, first course of treatment, and survival, on every cancer case that is reported across all U.S. states and territories. The goal of our project is to enrich NCI's Surveillance, Epidemiology, and End Results (SEER) registry data with high-quality population-based biospecimen data in the form of digital pathology, machine-learning-based classifications, and quantitative histopathology imaging feature sets (referred to here as Pathomics features). MATERIALS AND METHODS As part of the project, the underlying informatics infrastructure was designed, tested, and implemented through close collaboration with several participating SEER registries to ensure consistency with registry processes, computational scalability, and ability to support creation of population cohorts that span multiple sites. Utilizing computational imaging algorithms and methods to both generate indices and search for matches makes it possible to reduce inter- and intra-observer inconsistencies and to improve the objectivity with which large image repositories are interrogated. RESULTS Our team has created and continues to expand a well-curated repository of high-quality digitized pathology images corresponding to subjects whose data are routinely collected by the collaborating registries. Our team has systematically deployed and tested key, visual analytic methods to facilitate automated creation of population cohorts for epidemiological studies and tools to support visualization of feature clusters and evaluation of whole-slide images. As part of these efforts, we are developing and optimizing advanced search and matching algorithms to facilitate automated, content-based retrieval of digitized specimens based on their underlying image features and staining characteristics. CONCLUSION To meet the challenges of this project, we established the analytic pipelines, methods, and workflows to support the expansion and management of a growing repository of high-quality digitized pathology and information-rich, population cohorts containing objective imaging and clinical attributes to facilitate studies that seek to discriminate among different subtypes of disease, stratify patient populations, and perform comparisons of tumor characteristics within and across patient cohorts. We have also successfully developed a suite of tools based on a deep-learning method to perform quantitative characterizations of tumor regions, assess infiltrating lymphocyte distributions, and generate objective nuclear feature measurements. As part of these efforts, our team has implemented reliable methods that enable investigators to systematically search through large repositories to automatically retrieve digitized pathology specimens and correlated clinical data based on their computational signatures.
Collapse
Affiliation(s)
- David J. Foran
- Center for Biomedical Informatics, Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, USA
- Department of Pathology and Laboratory Medicine, Rutgers-Robert Wood Johnson Medical School, Piscataway, NJ, USA
| | - Eric B. Durbin
- Kentucky Cancer Registry, Markey Cancer Center, University of Kentucky, Lexington, KY, USA
- Division of Biomedical Informatics, Department of Internal Medicine, College of Medicine, Lexington, KY, USA
| | - Wenjin Chen
- Center for Biomedical Informatics, Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, USA
| | - Evita Sadimin
- Center for Biomedical Informatics, Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, USA
- Department of Pathology and Laboratory Medicine, Rutgers-Robert Wood Johnson Medical School, Piscataway, NJ, USA
| | - Ashish Sharma
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, USA
| | - Imon Banerjee
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, USA
| | - Tahsin Kurc
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Nan Li
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, USA
| | - Antoinette M. Stroup
- New Jersey State Cancer Registry, Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, USA
| | - Gerald Harris
- New Jersey State Cancer Registry, Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, USA
| | - Annie Gu
- Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, GA, USA
| | - Maria Schymura
- New York State Cancer Registry, New York State Department of Health, Albany, NY, USA
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Erich Bremer
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Joseph Balsamo
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Tammy DiPrima
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Feiqiao Wang
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Shahira Abousamra
- Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
| | - Isaac Hands
- Division of Biomedical Informatics, Department of Internal Medicine, College of Medicine, Lexington, KY, USA
| | - Kevin Ward
- Georgia State Cancer Registry, Georgia Department of Public Health, Atlanta, GA, USA
| | - Joel H. Saltz
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| |
Collapse
|
7
|
Abousamra S, Belinsky D, Van Arnam J, Allard F, Yee E, Gupta R, Kurc T, Samaras D, Saltz J, Chen C. Multi-Class Cell Detection Using Spatial Context Representation. Proc IEEE Int Conf Comput Vis 2021; 2021:3985-3994. [PMID: 38783989 PMCID: PMC11114143 DOI: 10.1109/iccv48922.2021.00397] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/25/2024]
Abstract
In digital pathology, both detection and classification of cells are important for automatic diagnostic and prognostic tasks. Classifying cells into subtypes, such as tumor cells, lymphocytes or stromal cells is particularly challenging. Existing methods focus on morphological appearance of individual cells, whereas in practice pathologists often infer cell classes through their spatial context. In this paper, we propose a novel method for both detection and classification that explicitly incorporates spatial contextual information. We use the spatial statistical function to describe local density in both a multi-class and a multi-scale manner. Through representation learning and deep clustering techniques, we learn advanced cell representation with both appearance and spatial context. On various benchmarks, our method achieves better performance than state-of-the-arts, especially on the classification task. We also create a new dataset for multi-class cell detection and classification in breast cancer and we make both our code and data publicly available.
Collapse
Affiliation(s)
| | | | | | | | - Eric Yee
- Stony Brook University, Stony Brook, NY 11794, USA
| | | | - Tahsin Kurc
- Stony Brook University, Stony Brook, NY 11794, USA
| | | | - Joel Saltz
- Stony Brook University, Stony Brook, NY 11794, USA
| | - Chao Chen
- Stony Brook University, Stony Brook, NY 11794, USA
| |
Collapse
|
8
|
Fassler DJ, Abousamra S, Gupta R, Chen C, Zhao M, Paredes D, Batool SA, Knudsen BS, Escobar-Hoyos L, Shroyer KR, Samaras D, Kurc T, Saltz J. Publisher Correction to: Deep learning-based image analysis methods for brightfield-acquired multiplex immunohistochemistry images. Diagn Pathol 2020; 15:116. [PMID: 32972449 PMCID: PMC7513292 DOI: 10.1186/s13000-020-01021-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022] Open
Affiliation(s)
- Danielle J Fassler
- Department of Pathology, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Shahira Abousamra
- Department of Computer Science, Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Chao Chen
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Maozheng Zhao
- Department of Computer Science, Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA
| | - David Paredes
- Department of Computer Science, Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA
| | - Syeda Areeha Batool
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Beatrice S Knudsen
- Department of Pathology, University of Utah, 2000 Circle of Hope, Salt Lake City, UT, 84112, USA
| | - Luisa Escobar-Hoyos
- Department of Pathology, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA.,Department Therapeutic Radiology, Yale University, 15 York Street, New Haven, CT, 06513, USA
| | - Kenneth R Shroyer
- Department of Pathology, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA
| | - Tahsin Kurc
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Joel Saltz
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA.
| |
Collapse
|
9
|
Fassler DJ, Abousamra S, Gupta R, Chen C, Zhao M, Paredes D, Batool SA, Knudsen BS, Escobar-Hoyos L, Shroyer KR, Samaras D, Kurc T, Saltz J. Deep learning-based image analysis methods for brightfield-acquired multiplex immunohistochemistry images. Diagn Pathol 2020; 15:100. [PMID: 32723384 PMCID: PMC7385962 DOI: 10.1186/s13000-020-01003-0] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2019] [Accepted: 07/12/2020] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND Multiplex immunohistochemistry (mIHC) permits the labeling of six or more distinct cell types within a single histologic tissue section. The classification of each cell type requires detection of the unique colored chromogens localized to cells expressing biomarkers of interest. The most comprehensive and reproducible method to evaluate such slides is to employ digital pathology and image analysis pipelines to whole-slide images (WSIs). Our suite of deep learning tools quantitatively evaluates the expression of six biomarkers in mIHC WSIs. These methods address the current lack of readily available methods to evaluate more than four biomarkers and circumvent the need for specialized instrumentation to spectrally separate different colors. The use case application for our methods is a study that investigates tumor immune interactions in pancreatic ductal adenocarcinoma (PDAC) with a customized mIHC panel. METHODS Six different colored chromogens were utilized to label T-cells (CD3, CD4, CD8), B-cells (CD20), macrophages (CD16), and tumor cells (K17) in formalin-fixed paraffin-embedded (FFPE) PDAC tissue sections. We leveraged pathologist annotations to develop complementary deep learning-based methods: (1) ColorAE is a deep autoencoder which segments stained objects based on color; (2) U-Net is a convolutional neural network (CNN) trained to segment cells based on color, texture and shape; and ensemble methods that employ both ColorAE and U-Net, collectively referred to as (3) ColorAE:U-Net. We assessed the performance of our methods using: structural similarity and DICE score to evaluate segmentation results of ColorAE against traditional color deconvolution; F1 score, sensitivity, positive predictive value, and DICE score to evaluate the predictions from ColorAE, U-Net, and ColorAE:U-Net ensemble methods against pathologist-generated ground truth. We then used prediction results for spatial analysis (nearest neighbor). RESULTS We observed that (1) the performance of ColorAE is comparable to traditional color deconvolution for single-stain IHC images (note: traditional color deconvolution cannot be used for mIHC); (2) ColorAE and U-Net are complementary methods that detect 6 different classes of cells with comparable performance; (3) combinations of ColorAE and U-Net into ensemble methods outperform using either ColorAE and U-Net alone; and (4) ColorAE:U-Net ensemble methods can be employed for detailed analysis of the tumor microenvironment (TME). We developed a suite of scalable deep learning methods to analyze 6 distinctly labeled cell populations in mIHC WSIs. We evaluated our methods and found that they reliably detected and classified cells in the PDAC tumor microenvironment. We also present a use case, wherein we apply the ColorAE:U-Net ensemble method across 3 mIHC WSIs and use the predictions to quantify all stained cell populations and perform nearest neighbor spatial analysis. Thus, we provide proof of concept that these methods can be employed to quantitatively describe the spatial distribution immune cells within the tumor microenvironment. These complementary deep learning methods are readily deployable for use in clinical research studies.
Collapse
Affiliation(s)
- Danielle J Fassler
- Department of Pathology, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Shahira Abousamra
- Department of Computer Science, Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Chao Chen
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Maozheng Zhao
- Department of Computer Science, Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA
| | - David Paredes
- Department of Computer Science, Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA
| | - Syeda Areeha Batool
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Beatrice S Knudsen
- Department of Pathology, University of Utah, 2000 Circle of Hope, Salt Lake City, UT, 84112, USA
| | - Luisa Escobar-Hoyos
- Department of Pathology, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
- Department Therapeutic Radiology, Yale University, 15 York Street, New Haven, CT, 06513, USA
| | - Kenneth R Shroyer
- Department of Pathology, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA
| | - Tahsin Kurc
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA
| | - Joel Saltz
- Department of Biomedical Informatics, Stony Brook University Renaissance School of Medicine, 101 Nicolls Rd, Stony Brook, 11794, USA.
| |
Collapse
|
10
|
Le H, Gupta R, Hou L, Abousamra S, Fassler D, Torre-Healy L, Moffitt RA, Kurc T, Samaras D, Batiste R, Zhao T, Rao A, Van Dyke AL, Sharma A, Bremer E, Almeida JS, Saltz J. Utilizing Automated Breast Cancer Detection to Identify Spatial Distributions of Tumor-Infiltrating Lymphocytes in Invasive Breast Cancer. Am J Pathol 2020; 190:1491-1504. [PMID: 32277893 PMCID: PMC7369575 DOI: 10.1016/j.ajpath.2020.03.012] [Citation(s) in RCA: 49] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/25/2019] [Revised: 02/28/2020] [Accepted: 03/19/2020] [Indexed: 11/22/2022]
Abstract
Quantitative assessment of spatial relations between tumor and tumor-infiltrating lymphocytes (TIL) is increasingly important in both basic science and clinical aspects of breast cancer research. We have developed and evaluated convolutional neural network analysis pipelines to generate combined maps of cancer regions and TILs in routine diagnostic breast cancer whole slide tissue images. The combined maps provide insight about the structural patterns and spatial distribution of lymphocytic infiltrates and facilitate improved quantification of TILs. Both tumor and TIL analyses were evaluated by using three convolutional neural network networks (34-layer ResNet, 16-layer VGG, and Inception v4); the results compared favorably with those obtained by using the best published methods. We have produced open-source tools and a public data set consisting of tumor/TIL maps for 1090 invasive breast cancer images from The Cancer Genome Atlas. The maps can be downloaded for further downstream analyses.
Collapse
Affiliation(s)
- Han Le
- Department of Computer Science, Stony Brook University, Stony Brook, New York.
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook Medicine, Stony Brook, New York; Department of Pathology, Stony Brook University Hospital, Stony Brook, New York
| | - Le Hou
- Department of Computer Science, Stony Brook University, Stony Brook, New York
| | - Shahira Abousamra
- Department of Computer Science, Stony Brook University, Stony Brook, New York
| | - Danielle Fassler
- Department of Pathology, Stony Brook University Hospital, Stony Brook, New York
| | - Luke Torre-Healy
- Department of Biomedical Informatics, Stony Brook Medicine, Stony Brook, New York
| | - Richard A Moffitt
- Department of Biomedical Informatics, Stony Brook Medicine, Stony Brook, New York; Department of Pathology, Stony Brook University Hospital, Stony Brook, New York
| | - Tahsin Kurc
- Department of Biomedical Informatics, Stony Brook Medicine, Stony Brook, New York
| | - Dimitris Samaras
- Department of Computer Science, Stony Brook University, Stony Brook, New York
| | - Rebecca Batiste
- Department of Pathology, Stony Brook University Hospital, Stony Brook, New York
| | - Tianhao Zhao
- Department of Pathology, Stony Brook University Hospital, Stony Brook, New York
| | - Arvind Rao
- Department of Computational Medicine & Bioinformatics, University of Michigan Medical School, Ann Arbor, Michigan
| | - Alison L Van Dyke
- Surveillance Research Program, Division of Cancer Control and Population Sciences, National Cancer Institute, National Institutes of Health, Bethesda, Maryland
| | - Ashish Sharma
- Department of Biomedical Informatics, Emory University, Atlanta, Georgia
| | - Erich Bremer
- Department of Biomedical Informatics, Stony Brook Medicine, Stony Brook, New York
| | - Jonas S Almeida
- Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Bethesda, Maryland
| | - Joel Saltz
- Department of Biomedical Informatics, Stony Brook Medicine, Stony Brook, New York
| |
Collapse
|
11
|
Kobayashi S, Le H, Chrastecka L, Gupta R, Hou L, Abousamra S, Fassler D, Shroyer KR, Samaras D, Kurc T, Moffitt RA, Saltz JH. Abstract A27: Deep learning for analysis of tumor-lymphocyte interactions in pancreatic ductal adenocarcinoma. Cancer Res 2019. [DOI: 10.1158/1538-7445.panca19-a27] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Abstract
The interaction of tumor, stroma, and immune cells in pancreatic ductal adenocarcinoma (PDAC) is complex and difficult to quantify in patient samples. Recently, deep learning algorithms have shown successes in identifying tumor and lymphocytes regions on whole-slide images derived from routinely collected histopathologic specimens. The Cancer Genome Atlas (TCGA) in particular has generated whole-slide images as well as paired molecular data, thus allowing for combined spatial and molecular analyses of tumor-lymphocyte interactions. We have previously highlighted this resource by computationally mapping tumor-infiltrating lymphocytes (TILs) on digital images across 13 tumor types. To achieve this, convolutional neural networks were trained on lymphocyte images annotated by expert pathologists and then used to detect spatial TIL patterns. This led to identification of four qualitative TIL pattern categories, which varied depending on tumor type as well as molecular immune subtype, demonstrating the potential of these spatial structures to provide further insights into tumor microenvironments and their relationship to overall survival. We have now extended this deep learning pipeline to include identification of tumor regions in PDAC, allowing study of TIL patterns in the context of their relative spatial localization to tumors. Using the deep learning algorithm to define the tumor region, we applied erosion and dilation operations to further capture the peritumoral region, the outer and inner regions of the tumor, as well as desmoplasia far from the tumor cells. We thus defined lymphocytes by their spatial localization as being internal, tumoral, peritumoral, or outer with these masks. We then used nearest-neighbor and density-based approaches to quantify TIL infiltration patterns with respect to tumor. These features vary significantly across the previously identified TIL patterns and may serve as additional parameters to define the microenvironment conditions in patient samples. Here we demonstrate that features extracted using our pipeline recapitulate canonical histologic properties. Using immune cell abundance estimates from gene expression generated by CIBERSORT, we find that samples with tumor TIL densities above median have more M1 macrophages, while those below median have more M2 macrophages. We also observe that slides with a higher peritumoral TIL density relative to tumoral TIL density have higher Treg fractions. Ongoing work on improving the resolution and cell specificity of our pipeline will allow us to ask more specific questions and permit higher granularity in linking clinical outcomes to spatial immune phenotypes.
Citation Format: Soma Kobayashi, Han Le, Lucie Chrastecka, Rajarsi Gupta, Le Hou, Shahira Abousamra, Danielle Fassler, Kenneth R. Shroyer, Dimitris Samaras, Tahsin Kurc, Richard A. Moffitt, Joel H. Saltz. Deep learning for analysis of tumor-lymphocyte interactions in pancreatic ductal adenocarcinoma [abstract]. In: Proceedings of the AACR Special Conference on Pancreatic Cancer: Advances in Science and Clinical Care; 2019 Sept 6-9; Boston, MA. Philadelphia (PA): AACR; Cancer Res 2019;79(24 Suppl):Abstract nr A27.
Collapse
Affiliation(s)
| | - Han Le
- Stony Brook University, Stony Brook, NY
| | | | | | - Le Hou
- Stony Brook University, Stony Brook, NY
| | | | | | | | | | | | | | | |
Collapse
|