Building tools for machine learning and artificial intelligence in cancer research: best practices and a case study with the PathML toolkit for computational pathology

ChatGPT for digital pathology research

The rapid evolution of generative artificial intelligence (AI) models including OpenAI's ChatGPT signals a promising era for medical research. In this Viewpoint, we explore the integration and challenges of large language models (LLMs) in digital pathology, a rapidly evolving domain demanding intricate contextual understanding. The restricted domain-specific efficiency of LLMs necessitates the advent of tailored AI tools, as illustrated by advancements seen in the last few years including FrugalGPT and BioBERT. Our initiative in digital pathology emphasises the potential of domain-specific AI tools, where a curated literature database coupled with a user-interactive web application facilitates precise, referenced information retrieval. Motivated by the success of this initiative, we discuss how domain-specific approaches substantially minimise the risk of inaccurate responses, enhancing the reliability and accuracy of information extraction. We also highlight the broader implications of such tools, particularly in streamlining access to scientific research and democratising access to computational pathology techniques for scientists with little coding experience. This Viewpoint calls for an enhanced integration of domain-specific text-generation AI tools in academic settings to facilitate continuous learning and adaptation to the dynamically evolving landscape of medical research.

A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an indirect two-step approach consisting of (1) DeepPT, a deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response to targeted and immune therapies from the inferred expression values. We show that DeepPT successfully predicts transcriptomics in all 16 The Cancer Genome Atlas cohorts tested and generalizes well to two independent datasets. ENLIGHT-DeepPT successfully predicts true responders in five independent patient cohorts involving four different treatments spanning six cancer types, with an overall odds ratio of 2.28 and a 39.5% increased response rate among predicted responders versus the baseline rate. Notably, its prediction accuracy, obtained without any training on the treatment data, is comparable to that achieved by directly predicting the response from the images, which requires specific training on the treatment evaluation cohorts.

AnVILWorkflow: A runnable workflow package for Cloud-implemented bioinformatics analysis pipelines

Advancements in sequencing technologies and the development of new data collection methods produce large volumes of biological data. The Genomic Data Science Analysis, Visualization, and Informatics Lab-space (AnVIL) provides a cloud-based platform for democratizing access to large-scale genomics data and analysis tools. However, utilizing the full capabilities of AnVIL can be challenging for researchers without extensive bioinformatics expertise, especially for executing complex workflows. Here we present the AnVILWorkflow R package, which enables the convenient execution of bioinformatics workflows hosted on AnVIL directly from an R environment. AnVILWorkflowsimplifies the setup of the cloud computing environment, input data formatting, workflow submission, and retrieval of results through intuitive functions. We demonstrate the utility of AnVILWorkflowfor three use cases: bulk RNA-seq analysis with Salmon, metagenomics analysis with bioBakery, and digital pathology image processing with PathML. The key features of AnVILWorkflow include user-friendly browsing of available data and workflows, seamless integration of R and non-R tools within a reproducible analysis pipeline, and accessibility to scalable computing resources without direct management overhead. While some limitations exist around workflow customization, AnVILWorkflowlowers the barrier to taking advantage of AnVIL's resources, especially for exploratory analyses or bulk processing with established workflows. This empowers a broader community of researchers to leverage the latest genomics tools and datasets using familiar R syntax. This package is distributed through the Bioconductor project (https://bioconductor.org/packages/AnVILWorkflow), and the source code is available through GitHub (https://github.com/shbrief/AnVILWorkflow).

Genomic and Immunophenotypic Landscape of Acquired Resistance to PD-(L)1 Blockade in Non-Small-Cell Lung Cancer

PURPOSE

Although immune checkpoint inhibitors (ICI) have extended survival in patients with non-small-cell lung cancer (NSCLC), acquired resistance (AR) to ICI frequently develops after an initial benefit. However, the mechanisms of AR to ICI in NSCLC are largely unknown.

METHODS

Comprehensive tumor genomic profiling, machine learning-based assessment of tumor-infiltrating lymphocytes, multiplexed immunofluorescence, and/or HLA-I immunohistochemistry (IHC) were performed on matched pre- and post-ICI tumor biopsies from patients with NSCLC treated with ICI at the Dana-Farber Cancer Institute who developed AR to ICI. Two additional cohorts of patients with intervening chemotherapy or targeted therapies between biopsies were included as controls.

RESULTS

We performed comprehensive genomic profiling and immunophenotypic characterization on samples from 82 patients with NSCLC and matched pre- and post-ICI biopsies and compared findings with a control cohort of patients with non-ICI intervening therapies between biopsies (chemotherapy, N = 32; targeted therapies, N = 89; both, N = 17). Putative resistance mutations were identified in 27.8% of immunotherapy-treated cases and included acquired loss-of-function mutations in STK11, B2M, APC, MTOR, KEAP1, and JAK1/2; these acquired alterations were not observed in the control groups. Immunophenotyping of matched pre- and post-ICI samples demonstrated significant decreases in intratumoral lymphocytes, CD3e+ and CD8a+ T cells, and PD-L1-PD1 engagement, as well as increased distance between tumor cells and CD8+PD-1+ T cells. There was a significant decrease in HLA class I expression in the immunotherapy cohort at the time of AR compared with the chemotherapy (P = .005) and the targeted therapy (P = .01) cohorts.

CONCLUSION

These findings highlight the genomic and immunophenotypic heterogeneity of ICI resistance in NSCLC, which will need to be considered when developing novel therapeutic strategies aimed at overcoming resistance.

Slideflow: deep learning for digital histopathology with real-time whole-slide visualization

Deep learning methods have emerged as powerful tools for analyzing histopathological images, but current methods are often specialized for specific domains and software environments, and few open-source options exist for deploying models in an interactive interface. Experimenting with different deep learning approaches typically requires switching software libraries and reprocessing data, reducing the feasibility and practicality of experimenting with new architectures. We developed a flexible deep learning library for histopathology called Slideflow, a package which supports a broad array of deep learning methods for digital pathology and includes a fast whole-slide interface for deploying trained models. Slideflow includes unique tools for whole-slide image data processing, efficient stain normalization and augmentation, weakly-supervised whole-slide classification, uncertainty quantification, feature generation, feature space analysis, and explainability. Whole-slide image processing is highly optimized, enabling whole-slide tile extraction at 40x magnification in 2.5 s per slide. The framework-agnostic data processing pipeline enables rapid experimentation with new methods built with either Tensorflow or PyTorch, and the graphical user interface supports real-time visualization of slides, predictions, heatmaps, and feature space characteristics on a variety of hardware devices, including ARM-based devices such as the Raspberry Pi.

Distinct mesenchymal cell states mediate prostate cancer progression

In the complex tumor microenvironment (TME), mesenchymal cells are key players, yet their specific roles in prostate cancer (PCa) progression remain to be fully deciphered. This study employs single-cell RNA sequencing to delineate molecular changes in tumor stroma that influence PCa progression and metastasis. Analyzing mesenchymal cells from four genetically engineered mouse models (GEMMs) and correlating these findings with human tumors, we identify eight stromal cell populations with distinct transcriptional identities consistent across both species. Notably, stromal signatures in advanced mouse disease reflect those in human bone metastases, highlighting periostin's role in invasion and differentiation. From these insights, we derive a gene signature that predicts metastatic progression in localized disease beyond traditional Gleason scores. Our results illuminate the critical influence of stromal dynamics on PCa progression, suggesting new prognostic tools and therapeutic targets.

Spatially aware deep learning reveals tumor heterogeneity patterns that encode distinct kidney cancer states

Clear cell renal cell carcinoma (ccRCC) is molecularly heterogeneous, immune infiltrated, and selectively sensitive to immune checkpoint inhibition (ICI). However, the joint tumor-immune states that mediate ICI response remain elusive. We develop spatially aware deep-learning models of tumor and immune features to learn representations of ccRCC tumors using diagnostic whole-slide images (WSIs) in untreated and treated contexts (n = 1,102 patients). We identify patterns of grade heterogeneity in WSIs not achievable through human pathologist analysis, and these graph-based "microheterogeneity" structures associate with PBRM1 loss of function and with patient outcomes. Joint analysis of tumor phenotypes and immune infiltration identifies a subpopulation of highly infiltrated, microheterogeneous tumors responsive to ICI. In paired multiplex immunofluorescence images of ccRCC, microheterogeneity associates with greater PD1 activation in CD8+ lymphocytes and increased tumor-immune interactions. Our work reveals spatially interacting tumor-immune structures underlying ccRCC biology that may also inform selective response to ICI.

Prediction of cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin (H&E)-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an approach for predicting response to multiple targeted and immunotherapies from H&E-slides. In difference from existing approaches that aim to predict treatment response directly from the slides, ENLIGHT-DeepPT is an indirect two-step approach consisting of (1) DeepPT, a new deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response based on the DeepPT inferred expression values. DeepPT successfully predicts transcriptomics in all 16 TCGA cohorts tested and generalizes well to two independent datasets. Our key contribution is showing that ENLIGHT-DeepPT successfully predicts true responders in five independent patients' cohorts involving four different treatments spanning six cancer types with an overall odds ratio of 2.44, increasing the baseline response rate by 43.47% among predicted responders, without the need for any treatment data for training. Furthermore, its prediction accuracy on these datasets is comparable to a supervised approach predicting the response directly from the images, which needs to be trained and tested on the same cohort. ENLIGHT-DeepPT future application could provide clinicians with rapid treatment recommendations to an array of different therapies and importantly, may contribute to advancing precision oncology in developing countries.

SliDL: A toolbox for processing whole-slide images in deep learning

The inspection of stained tissue slides by pathologists is essential for the early detection, diagnosis and monitoring of disease. Recently, deep learning methods for the analysis of whole-slide images (WSIs) have shown excellent performance on these tasks, and have the potential to substantially reduce the workload of pathologists. However, WSIs present a number of unique challenges for analysis, requiring special consideration of image annotations, slide and image artefacts, and evaluation of WSI-trained model performance. Here we introduce SliDL, a Python library for performing pre- and post-processing of WSIs. SliDL makes WSI data handling easy, allowing users to perform essential processing tasks in a few simple lines of code, bridging the gap between standard image analysis and WSI analysis. We introduce each of the main functionalities within SliDL: from annotation and tile extraction to tissue detection and model evaluation. We also provide 'code snippets' to guide the user in running SliDL. SliDL has been designed to interact with PyTorch, one of the most widely used deep learning libraries, allowing seamless integration into deep learning workflows. By providing a framework in which deep learning methods for WSI analysis can be developed and applied, SliDL aims to increase the accessibility of an important application of deep learning.

Federated benchmarking of medical artificial intelligence with MedPerf

Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, heterogeneous data capturing diverse patient populations. Here, to meet this need, we introduce MedPerf, an open platform for benchmarking AI models in the medical domain. MedPerf focuses on enabling federated evaluation of AI models, by securely distributing them to different facilities, such as healthcare organizations. This process of bringing the model to the data empowers each facility to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status and real-world deployment, our roadmap and, importantly, the use of MedPerf with multiple international institutions within cloud-based technology and on-premises scenarios. Finally, we welcome new contributions by researchers and organizations to further strengthen MedPerf as an open benchmarking platform.

Distinct mesenchymal cell states mediate prostate cancer progression

Alterations in tumor stroma influence prostate cancer progression and metastatic potential. However, the molecular underpinnings of this stromal-epithelial crosstalk are largely unknown. Here, we compare mesenchymal cells from four genetically engineered mouse models (GEMMs) of prostate cancer representing different stages of the disease to their wild-type (WT) counterparts by single-cell RNA sequencing (scRNA-seq) and, ultimately, to human tumors with comparable genotypes. We identified 8 transcriptionally and functionally distinct stromal populations responsible for common and GEMM-specific transcriptional programs. We show that stromal responses are conserved in mouse models and human prostate cancers with the same genomic alterations. We noted striking similarities between the transcriptional profiles of the stroma of murine models of advanced disease and those of of human prostate cancer bone metastases. These profiles were then used to build a robust gene signature that can predict metastatic progression in prostate cancer patients with localized disease and is also associated with progression-free survival independent of Gleason score. Taken together, this offers new evidence that stromal microenvironment mediates prostate cancer progression, further identifying tissue-based biomarkers and potential therapeutic targets of aggressive and metastatic disease.

Interoperable slide microscopy viewer and annotation tool for imaging data science and computational pathology

The exchange of large and complex slide microscopy imaging data in biomedical research and pathology practice is impeded by a lack of data standardization and interoperability, which is detrimental to the reproducibility of scientific findings and clinical integration of technological innovations. We introduce Slim, an open-source, web-based slide microscopy viewer that implements the internationally accepted Digital Imaging and Communications in Medicine (DICOM) standard to achieve interoperability with a multitude of existing medical imaging systems. We showcase the capabilities of Slim as the slide microscopy viewer of the NCI Imaging Data Commons and demonstrate how the viewer enables interactive visualization of traditional brightfield microscopy and highly-multiplexed immunofluorescence microscopy images from The Cancer Genome Atlas and Human Tissue Atlas Network, respectively, using standard DICOMweb services. We further show how Slim enables the collection of standardized image annotations for the development or validation of machine learning models and the visual interpretation of model inference results in the form of segmentation masks, spatial heat maps, or image-derived measurements.

Spatially aware deep learning reveals tumor heterogeneity patterns that encode distinct kidney cancer states

Clear cell renal cell carcinoma (ccRCC) is molecularly heterogeneous, immune infiltrated, and selectively sensitive to immune checkpoint inhibition (ICI). Established histopathology paradigms like nuclear grade have baseline prognostic relevance for ccRCC, although whether existing or novel histologic features encode additional heterogeneous biological and clinical states in ccRCC is uncertain. Here, we developed spatially aware deep learning models of tumor- and immune-related features to learn representations of ccRCC tumors using diagnostic whole-slide images (WSI) in untreated and treated contexts (n = 1102 patients). We discovered patterns of nuclear grade heterogeneity in WSI not achievable through human pathologist analysis, and these graph-based "microheterogeneity" structures associated with PBRM1 loss of function, adverse clinical factors, and selective patient response to ICI. Joint computer vision analysis of tumor phenotypes with inferred tumor infiltrating lymphocyte density identified a further subpopulation of highly infiltrated, microheterogeneous tumors responsive to ICI. In paired multiplex immunofluorescence images of ccRCC, microheterogeneity associated with greater PD1 activation in CD8+ lymphocytes and increased tumor-immune interactions. Thus, our work reveals novel spatially interacting tumor-immune structures underlying ccRCC biology that can also inform selective response to ICI.

TIAToolbox as an end-to-end library for advanced tissue image analytics

Background

Computational pathology has seen rapid growth in recent years, driven by advanced deep-learning algorithms. Due to the sheer size and complexity of multi-gigapixel whole-slide images, to the best of our knowledge, there is no open-source software library providing a generic end-to-end API for pathology image analysis using best practices. Most researchers have designed custom pipelines from the bottom up, restricting the development of advanced algorithms to specialist users. To help overcome this bottleneck, we present TIAToolbox, a Python toolbox designed to make computational pathology accessible to computational, biomedical, and clinical researchers.

Methods

By creating modular and configurable components, we enable the implementation of computational pathology algorithms in a way that is easy to use, flexible and extensible. We consider common sub-tasks including reading whole slide image data, patch extraction, stain normalization and augmentation, model inference, and visualization. For each of these steps, we provide a user-friendly application programming interface for commonly used methods and models.

Results

We demonstrate the use of the interface to construct a full computational pathology deep-learning pipeline. We show, with the help of examples, how state-of-the-art deep-learning algorithms can be reimplemented in a streamlined manner using our library with minimal effort.

Conclusions

We provide a usable and adaptable library with efficient, cutting-edge, and unit-tested tools for data loading, pre-processing, model inference, post-processing, and visualization. This enables a range of users to easily build upon recent deep-learning developments in the computational pathology literature.

Computational software is being introduced to pathology, the study of the causes and effects of disease. Recently various computational pathology algorithms have been developed to analyze digital histology images. However, the software code written for these algorithms often combines functionality from several software packages which have specific setup requirements and code styles. This makes it difficult to re-use this code in other projects. We developed a computational software named TIAToolbox to alleviate this problem and hope this will help accelerate the use of computational software in pathology.

Pocock, Graham et al. present TIAToolbox, a Python toolbox for computational pathology. The extendable library can be used for data loading, pre-processing, model inference, post-processing, and visualization.

Artificial intelligence in histopathology: enhancing cancer research and clinical oncology

Artificial intelligence (AI) methods have multiplied our capabilities to extract quantitative information from digital histopathology images. AI is expected to reduce workload for human experts, improve the objectivity and consistency of pathology reports, and have a clinical impact by extracting hidden information from routinely available data. Here, we describe how AI can be used to predict cancer outcome, treatment response, genetic alterations and gene expression from digitized histopathology slides. We summarize the underlying technologies and emerging approaches, noting limitations, including the need for data sharing and standards. Finally, we discuss the broader implications of AI in cancer research and oncology.

Computational pathology in ovarian cancer

Histopathologic evaluations of tissue sections are key to diagnosing and managing ovarian cancer. Pathologists empirically assess and integrate visual information, such as cellular density, nuclear atypia, mitotic figures, architectural growth patterns, and higher-order patterns, to determine the tumor type and grade, which guides oncologists in selecting appropriate treatment options. Latent data embedded in pathology slides can be extracted using computational imaging. Computers can analyze digital slide images to simultaneously quantify thousands of features, some of which are visible with a manual microscope, such as nuclear size and shape, while others, such as entropy, eccentricity, and fractal dimensions, are quantitatively beyond the grasp of the human mind. Applications of artificial intelligence and machine learning tools to interpret digital image data provide new opportunities to explore and quantify the spatial organization of tissues, cells, and subcellular structures. In comparison to genomic, epigenomic, transcriptomic, and proteomic patterns, morphologic and spatial patterns are expected to be more informative as quantitative biomarkers of complex and dynamic tumor biology. As computational pathology is not limited to visual data, nuanced subvisual alterations that occur in the seemingly “normal” pre-cancer microenvironment could facilitate research in early cancer detection and prevention. Currently, efforts to maximize the utility of computational pathology are focused on integrating image data with other -omics platforms that lack spatial information, thereby providing a new way to relate the molecular, spatial, and microenvironmental characteristics of cancer. Despite a dire need for improvements in ovarian cancer prevention, early detection, and treatment, the ovarian cancer field has lagged behind other cancers in the application of computational pathology. The intent of this review is to encourage ovarian cancer research teams to apply existing and/or develop additional tools in computational pathology for ovarian cancer and actively contribute to advancing this important field.

Emerging Promise of Computational Techniques in Anti-Cancer Research: At a Glance

Research on the immune system and cancer has led to the development of new medicines that enable the former to attack cancer cells. Drugs that specifically target and destroy cancer cells are on the horizon; there are also drugs that use specific signals to stop cancer cells multiplying. Machine learning algorithms can significantly support and increase the rate of research on complicated diseases to help find new remedies. One area of medical study that could greatly benefit from machine learning algorithms is the exploration of cancer genomes and the discovery of the best treatment protocols for different subtypes of the disease. However, developing a new drug is time-consuming, complicated, dangerous, and costly. Traditional drug production can take up to 15 years, costing over USD 1 billion. Therefore, computer-aided drug design (CADD) has emerged as a powerful and promising technology to develop quicker, cheaper, and more efficient designs. Many new technologies and methods have been introduced to enhance drug development productivity and analytical methodologies, and they have become a crucial part of many drug discovery programs; many scanning programs, for example, use ligand screening and structural virtual screening techniques from hit detection to optimization. In this review, we examined various types of computational methods focusing on anticancer drugs. Machine-based learning in basic and translational cancer research that could reach new levels of personalized medicine marked by speedy and advanced data analysis is still beyond reach. Ending cancer as we know it means ensuring that every patient has access to safe and effective therapies. Recent developments in computational drug discovery technologies have had a large and remarkable impact on the design of anticancer drugs and have also yielded useful insights into the field of cancer therapy. With an emphasis on anticancer medications, we covered the various components of computer-aided drug development in this paper. Transcriptomics, toxicogenomics, functional genomics, and biological networks are only a few examples of the bioinformatics techniques used to forecast anticancer medications and treatment combinations based on multi-omics data. We believe that a general review of the databases that are now available and the computational techniques used today will be beneficial for the creation of new cancer treatment approaches.