Spatial transcriptomics inferred from pathology whole-slide images links tumor heterogeneity to survival in breast and lung cancer

Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review

Advancements in digital pathology and computing resources have made a significant impact in the field of computational pathology for breast cancer diagnosis and treatment. However, access to high-quality labeled histopathological images of breast cancer is a big challenge that limits the development of accurate and robust deep learning models. In this scoping review, we identified the publicly available datasets of breast H&E-stained whole-slide images (WSIs) that can be used to develop deep learning algorithms. We systematically searched 9 scientific literature databases and 9 research data repositories and found 17 publicly available datasets containing 10 385 H&E WSIs of breast cancer. Moreover, we reported image metadata and characteristics for each dataset to assist researchers in selecting proper datasets for specific tasks in breast cancer computational pathology. In addition, we compiled 2 lists of breast H&E patches and private datasets as supplementary resources for researchers. Notably, only 28% of the included articles utilized multiple datasets, and only 14% used an external validation set, suggesting that the performance of other developed models may be susceptible to overestimation. The TCGA-BRCA was used in 52% of the selected studies. This dataset has a considerable selection bias that can impact the robustness and generalizability of the trained algorithms. There is also a lack of consistent metadata reporting of breast WSI datasets that can be an issue in developing accurate deep learning models, indicating the necessity of establishing explicit guidelines for documenting breast WSI dataset characteristics and metadata.

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Radiology and multi-scale data integration for precision oncology

In this Perspective paper we explore the potential of integrating radiological imaging with other data types, a critical yet underdeveloped area in comparison to the fusion of other multi-omic data. Radiological images provide a comprehensive, three-dimensional view of cancer, capturing features that would be missed by biopsies or other data modalities. This paper explores the complexities and challenges of incorporating medical imaging into data integration models, in the context of precision oncology. We present the different categories of imaging-omics integration and discuss recent progress, highlighting the opportunities that arise from bringing together spatial data on different scales.

Spatial landscapes of cancers: insights and opportunities

Solid tumours comprise many different cell types organized in spatially structured arrangements, with substantial intratumour and intertumour heterogeneity. Advances in spatial profiling technologies over the past decade hold promise to capture the complexity of these cellular architectures to build a holistic view of the intricate molecular mechanisms that shape the tumour ecosystem. Some of these mechanisms act at the cellular scale and are controlled by cell-autonomous programmes or communication between nearby cells, whereas other mechanisms result from coordinated efforts between large networks of cells and extracellular molecules organized into tissues and organs. In this Review we provide insights into the application of single-cell and spatial profiling tools, with a focus on spatially resolved transcriptomic tools developed to understand the cellular architecture of the tumour microenvironment and identify opportunities to use them to improve clinical management of cancers.

A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an indirect two-step approach consisting of (1) DeepPT, a deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response to targeted and immune therapies from the inferred expression values. We show that DeepPT successfully predicts transcriptomics in all 16 The Cancer Genome Atlas cohorts tested and generalizes well to two independent datasets. ENLIGHT-DeepPT successfully predicts true responders in five independent patient cohorts involving four different treatments spanning six cancer types, with an overall odds ratio of 2.28 and a 39.5% increased response rate among predicted responders versus the baseline rate. Notably, its prediction accuracy, obtained without any training on the treatment data, is comparable to that achieved by directly predicting the response from the images, which requires specific training on the treatment evaluation cohorts.

Prediction of DNA methylation-based tumor types from histopathology in central nervous system tumors with deep learning

Precision in the diagnosis of diverse central nervous system (CNS) tumor types is crucial for optimal treatment. DNA methylation profiles, which capture the methylation status of thousands of individual CpG sites, are state-of-the-art data-driven means to enhance diagnostic accuracy but are also time consuming and not widely available. Here, to address these limitations, we developed Deep lEarning from histoPathoLOgy and methYlation (DEPLOY), a deep learning model that classifies CNS tumors to ten major categories from histopathology. DEPLOY integrates three distinct components: the first classifies CNS tumors directly from slide images ('direct model'), the second initially generates predictions for DNA methylation beta values, which are subsequently used for tumor classification ('indirect model'), and the third classifies tumor types directly from routinely available patient demographics. First, we find that DEPLOY accurately predicts beta values from histopathology images. Second, using a ten-class model trained on an internal dataset of 1,796 patients, we predict the tumor categories in three independent external test datasets including 2,156 patients, achieving an overall accuracy of 95% and balanced accuracy of 91% on samples that are predicted with high confidence. These results showcase the potential future use of DEPLOY to assist pathologists in diagnosing CNS tumors within a clinically relevant short time frame.

Advancing Precision Medicine: Algebraic Topology and Differential Geometry in Radiology and Computational Pathology

Precision medicine aims to provide personalized care based on individual patient characteristics, rather than guideline-directed therapies for groups of diseases or patient demographics. Images-both radiology- and pathology-derived-are a major source of information on presence, type, and status of disease. Exploring the mathematical relationship of pixels in medical imaging ("radiomics") and cellular-scale structures in digital pathology slides ("pathomics") offers powerful tools for extracting both qualitative and, increasingly, quantitative data. These analytical approaches, however, may be significantly enhanced by applying additional methods arising from fields of mathematics such as differential geometry and algebraic topology that remain underexplored in this context. Geometry's strength lies in its ability to provide precise local measurements, such as curvature, that can be crucial for identifying abnormalities at multiple spatial levels. These measurements can augment the quantitative features extracted in conventional radiomics, leading to more nuanced diagnostics. By contrast, topology serves as a robust shape descriptor, capturing essential features such as connected components and holes. The field of topological data analysis was initially founded to explore the shape of data, with functional network connectivity in the brain being a prominent example. Increasingly, its tools are now being used to explore organizational patterns of physical structures in medical images and digitized pathology slides. By leveraging tools from both differential geometry and algebraic topology, researchers and clinicians may be able to obtain a more comprehensive, multi-layered understanding of medical images and contribute to precision medicine's armamentarium.

Multiplex Digital Spatial Profiling in Breast Cancer Research: State-of-the-Art Technologies and Applications across the Translational Science Spectrum

While RNA sequencing and multi-omic approaches have significantly advanced cancer diagnosis and treatment, their limitation in preserving critical spatial information has been a notable drawback. This spatial context is essential for understanding cellular interactions and tissue dynamics. Multiplex digital spatial profiling (MDSP) technologies overcome this limitation by enabling the simultaneous analysis of transcriptome and proteome data within the intact spatial architecture of tissues. In breast cancer research, MDSP has emerged as a promising tool, revealing complex biological questions related to disease evolution, identifying biomarkers, and discovering drug targets. This review highlights the potential of MDSP to revolutionize clinical applications, ranging from risk assessment and diagnostics to prognostics, patient monitoring, and the customization of treatment strategies, including clinical trial guidance. We discuss the major MDSP techniques, their applications in breast cancer research, and their integration in clinical practice, addressing both their potential and current limitations. Emphasizing the strategic use of MDSP in risk stratification for women with benign breast disease, we also highlight its transformative potential in reshaping the landscape of breast cancer research and treatment.

Imaging-based characterization of tumoral heterogeneity for personalized cancer treatment

With personalized tumor therapy, understanding and addressing the heterogeneity of malignant tumors is becoming increasingly important. Heterogeneity can be found within one lesion (intralesional) and between several tumor lesions emerging from one primary tumor (interlesional). The heterogeneous tumor cells may show a different response to treatment due to their biology, which in turn influences the outcome of the affected patients and the choice of therapeutic agents. Therefore, both intra- and interlesional heterogeneity should be addressed at the diagnostic stage. While genetic and biological heterogeneity are important parameters in molecular tumor characterization and in histopathology, they are not yet addressed routinely in medical imaging. This article summarizes the recently established markers for tumor heterogeneity in imaging as well as heterogeneous/mixed response to therapy. Furthermore, a look at emerging markers is given. The ultimate goal of this overview is to provide comprehensive understanding of tumor heterogeneity and its implications for radiology and for communication with interdisciplinary teams in oncology. KEY POINTS::   · Tumor heterogeneity can be described within one lesion (intralesional) or between several lesions (interlesional).. · The heterogeneous biology of tumor cells can lead to a mixed therapeutic response and should be addressed in diagnostics and the therapeutic regime.. · Quantitative image diagnostics can be enhanced using AI, improved histopathological methods, and liquid profiling in the future..

Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview

The capability to gather heterogeneous data, alongside the increasing power of artificial intelligence to examine it, leading a revolution in harnessing multimodal data in the life sciences. However, most approaches are limited to unimodal data, leaving integrated approaches across modalities relatively underdeveloped in computational pathology. Pathogenomics, as an invasive method to integrate advanced molecular diagnostics from genomic data, morphological information from histopathological imaging, and codified clinical data enable the discovery of new multimodal cancer biomarkers to propel the field of precision oncology in the coming decade. In this perspective, we offer our opinions on synthesizing complementary modalities of data with emerging multimodal artificial intelligence methods in pathogenomics. It includes correlation between the pathological and genomic profile of cancer, fusion of histology, and genomics profile of cancer. We also present challenges, opportunities, and avenues for future work.

Breaks for Precision Medicine in Cancer: Development and Prospects of Spatiotemporal Transcriptomics

With the development of the social economy and the deepening understanding of cancer, cancer has become a significant cause of death, threatening human health. Although researchers have made rapid progress in cancer treatment strategies in recent years, the overall survival of cancer patients is still not optimistic. Therefore, it is essential to reveal the spatial pattern of gene expression, spatial heterogeneity of cell populations, microenvironment interactions, and other aspects of cancer. Spatiotemporal transcriptomics can help analyze the mechanism of cancer occurrence and development, greatly help precise cancer treatment, and improve clinical prognosis. Here, we review the integration strategies of single-cell RNA sequencing and spatial transcriptomics data, summarize the recent advances in spatiotemporal transcriptomics in cancer studies, and discuss the combined application of spatial multiomics, which provides new directions and strategies for the precise treatment and clinical prognosis of cancer.

An error correction strategy for image reconstruction by DNA sequencing microscopy

By pairing adjacent molecules in situ and then mapping these pairs, DNA microscopy could substantially reduce the workload in spatial omics methods by directly inferring geometry from sequencing data alone. However, experimental artifacts can lead to errors in the adjacency data, which distort the spatial reconstruction. Here we describe a method to correct two such errors: spurious crosslinks formed between any two nodes, and fused nodes that are formed out of multiple molecules. We build on the principle that spatially close molecules should be connected and show that these errors violate this principle, allowing for their detection and correction. Our method corrects errors in simulated data, even in the presence of up to 20% errors, and proves to be more efficient at removing errors from experimental data than a read count filter. Integrating this method in DNA microscopy will substantially improve the accuracy of spatial reconstructions with lower data loss.

ATAT: Automated Tissue Alignment and Traversal in Spatial Transcriptomics with Self-Supervised Learning

Spatial transcriptomics (ST) has enhanced RNA analysis in tissue biopsies, but interpreting these data is challenging without expert input. We present Automated Tissue Alignment and Traversal (ATAT), a novel computational framework designed to enhance ST analysis in the context of multiple and complex tissue architectures and morphologies, such as those found in biopsies of the gastrointestinal tract. ATAT utilizes self-supervised contrastive learning on hematoxylin and eosin (H&E) stained images to automate the alignment and traversal of ST data. This approach addresses a critical gap in current ST analysis methodologies, which rely heavily on manual annotation and pathologist expertise to delineate regions of interest for accurate gene expression modeling. Our framework not only streamlines the alignment of multiple ST samples, but also demonstrates robustness in modeling gene expression transitions across specific regions. Additionally, we highlight the ability of ATAT to traverse complex tissue topologies in real-world cases from various individuals and conditions. Our method successfully elucidates differences in immune infiltration patterns across the intestinal wall, enabling the modeling of transcriptional changes across histological layers. We show that ATAT achieves comparable performance to the state-of-the-art method, while alleviating the burden of manual annotation and enabling alignment of tissue samples with complex morphologies.

Deep topographic proteomics of a human brain tumour

The spatial organisation of cellular protein expression profiles within tissue determines cellular function and is key to understanding disease pathology. To define molecular phenotypes in the spatial context of tissue, there is a need for unbiased, quantitative technology capable of mapping proteomes within tissue structures. Here, we present a workflow for spatially-resolved, quantitative proteomics of tissue that generates maps of protein abundance across tissue slices derived from a human atypical teratoid-rhabdoid tumour at three spatial resolutions, the highest being 40 µm, to reveal distinct abundance patterns of thousands of proteins. We employ spatially-aware algorithms that do not require prior knowledge of the fine tissue structure to detect proteins and pathways with spatial abundance patterns and correlate proteins in the context of tissue heterogeneity and cellular features such as extracellular matrix or proximity to blood vessels. We identify PYGL, ASPH and CD45 as spatial markers for tumour boundary and reveal immune response-driven, spatially-organised protein networks of the extracellular tumour matrix. Overall, we demonstrate spatially-aware deep proteo-phenotyping of tissue heterogeneity, to re-define understanding tissue biology and pathology at the molecular level.

Exploring multisite heterogeneity of human basal cell carcinoma proteome and transcriptome

Basal cell carcinoma (BCC) is the most common type of skin cancer. Due to multiple, potential underlying molecular tumor aberrations, clinical treatment protocols are not well-defined. This study presents multisite molecular heterogeneity profiles of human BCC based on RNA and proteome profiling. Three areas from lesions excised from 9 patients were analyzed. The focus was gene expression profiles based on proteome and RNA measurements of intra-tumor heterogeneity from the same patient and inter-tumor heterogeneity in nodular, infiltrative, and superficial BCC tumor subtypes from different patients. We observed significant overlap in intra- and inter-tumor variability of proteome and RNA expression profiles, showing significant multisite heterogeneity of protein expression in the BCC tumors. Inter-subtype analysis has also identified unique proteins for each BCC subtype. This profiling leads to a deeper understanding of BCC molecular heterogeneity and potentially contributes to developing new sampling tools for personalized diagnostics therapeutic approaches to BCC.

Feasibility of Inferring Spatial Transcriptomics from Single-Cell Histological Patterns for Studying Colon Cancer Tumor Heterogeneity

Background

Spatial transcriptomics involves studying the spatial organization of gene expression within tissues, offering insights into the molecular diversity of tumors. While spatial gene expression is commonly amalgamated from 1-10 cells across 50-micron spots, recent methods have demonstrated the capability to disaggregate this information at subspot resolution by leveraging both expression and histological patterns. However, elucidating such information from histology alone presents a significant challenge but if solved can better permit spatial molecular analysis at cellular resolution for instances where Visium data is not available, reducing study costs. This study explores integrating single-cell histological and transcriptomic data to infer spatial mRNA expression patterns in whole slide images collected from a cohort of stage pT3 colorectal cancer patients. A cell graph neural network algorithm was developed to align histological information extracted from detected cells with single cell RNA patterns through optimal transport methods, facilitating the analysis of cellular groupings and gene relationships. This approach leveraged spot-level expression as an intermediary to co-map histological and transcriptomic information at the single-cell level.

Results

Our study demonstrated that single-cell transcriptional heterogeneity within a spot could be predicted from histological markers extracted from cells detected within a spot. Furthermore, our model exhibited proficiency in delineating overarching gene expression patterns across whole-slide images. This approach compared favorably to traditional patch-based computer vision methods as well as other methods which did not incorporate single cell expression during the model fitting procedures. Topological nuances of single-cell expression within a Visium spot were preserved using the developed methodology.

Conclusion

This innovative approach augments the resolution of spatial molecular assays utilizing histology as a sole input through synergistic co-mapping of histological and transcriptomic datasets at the single-cell level, anchored by spatial transcriptomics. While initial results are promising, they warrant rigorous validation. This includes collaborating with pathologists for precise spatial identification of distinct cell types and utilizing sophisticated assays, such as Xenium, to attain deeper subcellular insights.

The Overlooked Role of Specimen Preparation in Bolstering Deep Learning-Enhanced Spatial Transcriptomics Workflows

The application of deep learning methods to spatial transcriptomics has shown promise in unraveling the complex relationships between gene expression patterns and tissue architecture as they pertain to various pathological conditions. Deep learning methods that can infer gene expression patterns directly from tissue histomorphology can expand the capability to discern spatial molecular markers within tissue slides. However, current methods utilizing these techniques are plagued by substantial variability in tissue preparation and characteristics, which can hinder the broader adoption of these tools. Furthermore, training deep learning models using spatial transcriptomics on small study cohorts remains a costly endeavor. Necessitating novel tissue preparation processes enhance assay reliability, resolution, and scalability. This study investigated the impact of an enhanced specimen processing workflow for facilitating a deep learning-based spatial transcriptomics assessment. The enhanced workflow leveraged the flexibility of the Visium CytAssist assay to permit automated H&E staining (e.g., Leica Bond) of tissue slides, whole-slide imaging at 40x-resolution, and multiplexing of tissue sections from multiple patients within individual capture areas for spatial transcriptomics profiling. Using a cohort of thirteen pT3 stage colorectal cancer (CRC) patients, we compared the efficacy of deep learning models trained on slide prepared using an enhanced workflow as compared to the traditional workflow which leverages manual tissue staining and standard imaging of tissue slides. Leveraging Inceptionv3 neural networks, we aimed to predict gene expression patterns across matched serial tissue sections, each stemming from a distinct workflow but aligned based on persistent histological structures. Findings indicate that the enhanced workflow considerably outperformed the traditional spatial transcriptomics workflow. Gene expression profiles predicted from enhanced tissue slides also yielded expression patterns more topologically consistent with the ground truth. This led to enhanced statistical precision in pinpointing biomarkers associated with distinct spatial structures. These insights can potentially elevate diagnostic and prognostic biomarker detection by broadening the range of spatial molecular markers linked to metastasis and recurrence. Future endeavors will further explore these findings to enrich our comprehension of various diseases and uncover molecular pathways with greater nuance. Combining deep learning with spatial transcriptomics provides a compelling avenue to enrich our understanding of tumor biology and improve clinical outcomes. For results of the highest fidelity, however, effective specimen processing is crucial, and fostering collaboration between histotechnicians, pathologists, and genomics specialists is essential to herald this new era in spatial transcriptomics-driven cancer research.

SMASH: Scalable Method for Analyzing Spatial Heterogeneity of genes in spatial transcriptomics data

In high-throughput spatial transcriptomics (ST) studies, it is of great interest to identify the genes whose level of expression in a tissue covaries with the spatial location of cells/spots. Such genes, also known as spatially variable genes (SVGs), can be crucial to the biological understanding of both structural and functional characteristics of complex tissues. Existing methods for detecting SVGs either suffer from huge computational demand or significantly lack statistical power. We propose a non-parametric method termed SMASH that achieves a balance between the above two problems. We compare SMASH with other existing methods in varying simulation scenarios demonstrating its superior statistical power and robustness. We apply the method to four ST datasets from different platforms uncovering interesting biological insights.

HAVOC: Small-scale histomic mapping of cancer biodiversity across large tissue distances using deep neural networks

Intratumoral heterogeneity can wreak havoc on current precision medicine strategies because of challenges in sufficient sampling of geographically separated areas of biodiversity distributed across centimeter-scale tumor distances. To address this gap, we developed a deep learning pipeline that leverages histomorphologic fingerprints of tissue to create "Histomic Atlases of Variation Of Cancers" (HAVOC). Using a number of objective molecular readouts, we demonstrate that HAVOC can define regional cancer boundaries with distinct biology. Using larger tumor specimens, we show that HAVOC can map biodiversity even across multiple tissue sections. By guiding profiling of 19 partitions across six high-grade gliomas, HAVOC revealed that distinct differentiation states can often coexist and be regionally distributed within these tumors. Last, to highlight generalizability, we benchmark HAVOC on additional tumor types. Together, we establish HAVOC as a versatile tool to generate small-scale maps of tissue heterogeneity and guide regional deployment of molecular resources to relevant biodiverse niches.

Prediction of cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin (H&E)-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an approach for predicting response to multiple targeted and immunotherapies from H&E-slides. In difference from existing approaches that aim to predict treatment response directly from the slides, ENLIGHT-DeepPT is an indirect two-step approach consisting of (1) DeepPT, a new deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response based on the DeepPT inferred expression values. DeepPT successfully predicts transcriptomics in all 16 TCGA cohorts tested and generalizes well to two independent datasets. Our key contribution is showing that ENLIGHT-DeepPT successfully predicts true responders in five independent patients' cohorts involving four different treatments spanning six cancer types with an overall odds ratio of 2.44, increasing the baseline response rate by 43.47% among predicted responders, without the need for any treatment data for training. Furthermore, its prediction accuracy on these datasets is comparable to a supervised approach predicting the response directly from the images, which needs to be trained and tested on the same cohort. ENLIGHT-DeepPT future application could provide clinicians with rapid treatment recommendations to an array of different therapies and importantly, may contribute to advancing precision oncology in developing countries.

Novel insights from spatial transcriptome analysis in solid tumors

Since its first application in 2016, spatial transcriptomics has become a rapidly evolving technology in recent years. Spatial transcriptomics enables transcriptomic data to be acquired from intact tissue sections and provides spatial distribution information and remedies the disadvantage of single-cell RNA sequencing (scRNA-seq), whose data lack spatially resolved information. Presently, spatial transcriptomics has been widely applied to various tissue types, especially for the study of tumor heterogeneity. In this review, we provide a summary of the research progress in utilizing spatial transcriptomics to investigate tumor heterogeneity and the microenvironment with a focus on solid tumors. We summarize the research breakthroughs in various fields and perspectives due to the application of spatial transcriptomics, including cell clustering and interaction, cellular metabolism, gene expression, immune cell programs and combination with other techniques. As a combination of multiple transcriptomics, single-cell multiomics shows its superiority and validity in single-cell analysis. We also discuss the application prospect of single-cell multiomics, and we believe that with the progress of data integration from various transcriptomics, a multilayered subcellular landscape will be revealed.

Deep learning exploration of single-cell and spatially resolved cancer transcriptomics to unravel tumour heterogeneity

Tumour heterogeneity is one of the critical confounding aspects in decoding tumour growth. Malignant cells display variations in their gene transcription profiles and mutation spectra even when originating from a single progenitor cell. Single-cell and spatial transcriptomics sequencing have recently emerged as key technologies for unravelling tumour heterogeneity. Single-cell sequencing promotes individual cell-type identification through transcriptome-wide gene expression measurements of each cell. Spatial transcriptomics facilitates identification of cell-cell interactions and the structural organization of heterogeneous cells within a tumour tissue through associating spatial RNA abundance of cells at distinct spots in the tissue section. However, extracting features and analyzing single-cell and spatial transcriptomics data poses challenges. Single-cell transcriptome data is extremely noisy and its sparse nature and dropouts can lead to misinterpretation of gene expression and the misclassification of cell types. Deep learning predictive power can overcome data challenges, provide high-resolution analysis and enhance precision oncology applications that involve early cancer prognosis, diagnosis, patient survival estimation and anti-cancer therapy planning. In this paper, we provide a background to and review of the recent progress of deep learning frameworks to investigate tumour heterogeneity using both single-cell and spatial transcriptomics data types.

Deep Learning for Lung Cancer Diagnosis, Prognosis and Prediction Using Histological and Cytological Images: A Systematic Review

Lung cancer is one of the deadliest cancers worldwide, with a high incidence rate, especially in tobacco smokers. Lung cancer accurate diagnosis is based on distinct histological patterns combined with molecular data for personalized treatment. Precise lung cancer classification from a single H&E slide can be challenging for a pathologist, requiring most of the time additional histochemical and special immunohistochemical stains for the final pathology report. According to WHO, small biopsy and cytology specimens are the available materials for about 70% of lung cancer patients with advanced-stage unresectable disease. Thus, the limited available diagnostic material necessitates its optimal management and processing for the completion of diagnosis and predictive testing according to the published guidelines. During the new era of Digital Pathology, Deep Learning offers the potential for lung cancer interpretation to assist pathologists' routine practice. Herein, we systematically review the current Artificial Intelligence-based approaches using histological and cytological images of lung cancer. Most of the published literature centered on the distinction between lung adenocarcinoma, lung squamous cell carcinoma, and small cell lung carcinoma, reflecting the realistic pathologist's routine. Furthermore, several studies developed algorithms for lung adenocarcinoma predominant architectural pattern determination, prognosis prediction, mutational status characterization, and PD-L1 expression status estimation.

Recent Advancements in Deep Learning Using Whole Slide Imaging for Cancer Prognosis

This review furnishes an exhaustive analysis of the latest advancements in deep learning techniques applied to whole slide images (WSIs) in the context of cancer prognosis, focusing specifically on publications from 2019 through 2023. The swiftly maturing field of deep learning, in combination with the burgeoning availability of WSIs, manifests significant potential in revolutionizing the predictive modeling of cancer prognosis. In light of the swift evolution and profound complexity of the field, it is essential to systematically review contemporary methodologies and critically appraise their ramifications. This review elucidates the prevailing landscape of this intersection, cataloging major developments, evaluating their strengths and weaknesses, and providing discerning insights into prospective directions. In this paper, a comprehensive overview of the field aims to be presented, which can serve as a critical resource for researchers and clinicians, ultimately enhancing the quality of cancer care outcomes. This review's findings accentuate the need for ongoing scrutiny of recent studies in this rapidly progressing field to discern patterns, understand breakthroughs, and navigate future research trajectories.

Spatial Transcriptomics: Technical Aspects of Recent Developments and Their Applications in Neuroscience and Cancer Research

Spatial transcriptomics is a newly emerging field that enables high-throughput investigation of the spatial localization of transcripts and related analyses in various applications for biological systems. By transitioning from conventional biological studies to "in situ" biology, spatial transcriptomics can provide transcriptome-scale spatial information. Currently, the ability to simultaneously characterize gene expression profiles of cells and relevant cellular environment is a paradigm shift for biological studies. In this review, recent progress in spatial transcriptomics and its applications in neuroscience and cancer studies are highlighted. Technical aspects of existing technologies and future directions of new developments (as of March 2023), computational analysis of spatial transcriptome data, application notes in neuroscience and cancer studies, and discussions regarding future directions of spatial multi-omics and their expanding roles in biomedical applications are emphasized.

Quantifying Intratumoral Heterogeneity and Immunoarchitecture Generated In-Silico by a Spatial Quantitative Systems Pharmacology Model

Spatial heterogeneity is a hallmark of cancer. Tumor heterogeneity can vary with time and location. The tumor microenvironment (TME) encompasses various cell types and their interactions that impart response to therapies. Therefore, a quantitative evaluation of tumor heterogeneity is crucial for the development of effective treatments. Different approaches, such as multiregional sequencing, spatial transcriptomics, analysis of autopsy samples, and longitudinal analysis of biopsy samples, can be used to analyze the intratumoral heterogeneity (ITH) and temporal evolution and to reveal the mechanisms of therapeutic response. However, because of the limitations of these data and the uncertainty associated with the time points of sample collection, having a complete understanding of intratumoral heterogeneity role is challenging. Here, we used a hybrid model that integrates a whole-patient compartmental quantitative-systems-pharmacology (QSP) model with a spatial agent-based model (ABM) describing the TME; we applied four spatial metrics to quantify model-simulated intratumoral heterogeneity and classified the TME immunoarchitecture for representative cases of effective and ineffective anti-PD-1 therapy. The four metrics, adopted from computational digital pathology, included mixing score, average neighbor frequency, Shannon's entropy and area under the curve (AUC) of the G-cross function. A fifth non-spatial metric was used to supplement the analysis, which was the ratio of the number of cancer cells to immune cells. These metrics were utilized to classify the TME as "cold", "compartmentalized" and "mixed", which were related to treatment efficacy. The trends in these metrics for effective and ineffective treatments are in qualitative agreement with the clinical literature, indicating that compartmentalized immunoarchitecture is likely to result in more efficacious treatment outcomes.

Generalization of vision pre-trained models for histopathology

Out-of-distribution (OOD) generalization, especially for medical setups, is a key challenge in modern machine learning which has only recently received much attention. We investigate how different convolutional pre-trained models perform on OOD test data-that is data from domains that have not been seen during training-on histopathology repositories attributed to different trial sites. Different trial site repositories, pre-trained models, and image transformations are examined as specific aspects of pre-trained models. A comparison is also performed among models trained entirely from scratch (i.e., without pre-training) and models already pre-trained. The OOD performance of pre-trained models on natural images, i.e., (1) vanilla pre-trained ImageNet, (2) semi-supervised learning (SSL), and (3) semi-weakly-supervised learning (SWSL) models pre-trained on IG-1B-Targeted are examined in this study. In addition, the performance of a histopathology model (i.e., KimiaNet) trained on the most comprehensive histopathology dataset, i.e., TCGA, has also been studied. Although the performance of SSL and SWSL pre-trained models are conducive to better OOD performance in comparison to the vanilla ImageNet pre-trained model, the histopathology pre-trained model is still the best in overall. In terms of top-1 accuracy, we demonstrate that diversifying the images in the training using reasonable image transformations is effective to avoid learning shortcuts when the distribution shift is significant. In addition, XAI techniques-which aim to achieve high-quality human-understandable explanations of AI decisions-are leveraged for further investigations.

Biological insights and novel biomarker discovery through deep learning approaches in breast cancer histopathology

Breast cancer remains a highly prevalent disease with considerable inter- and intra-tumoral heterogeneity complicating prognostication and treatment decisions. The utilization and depth of genomic, transcriptomic and proteomic data for cancer has exploded over recent times and the addition of spatial context to this information, by understanding the correlating morphologic and spatial patterns of cells in tissue samples, has created an exciting frontier of research, histo-genomics. At the same time, deep learning (DL), a class of machine learning algorithms employing artificial neural networks, has rapidly progressed in the last decade with a confluence of technical developments - including the advent of modern graphic processing units (GPU), allowing efficient implementation of increasingly complex architectures at scale; advances in the theoretical and practical design of network architectures; and access to larger datasets for training - all leading to sweeping advances in image classification and object detection. In this review, we examine recent developments in the application of DL in breast cancer histology with particular emphasis of those producing biologic insights or novel biomarkers, spanning the extraction of genomic information to the use of stroma to predict cancer recurrence, with the aim of suggesting avenues for further advancing this exciting field.

Inferring spatial transcriptomics markers from whole slide images to characterize metastasis-related spatial heterogeneity of colorectal tumors: A pilot study

Over 150 000 Americans are diagnosed with colorectal cancer (CRC) every year, and annually over 50 000 individuals will die from CRC, necessitating improvements in screening, prognostication, disease management, and therapeutic options. Tumor metastasis is the primary factor related to the risk of recurrence and mortality. Yet, screening for nodal and distant metastasis is costly, and invasive and incomplete resection may hamper adequate assessment. Signatures of the tumor-immune microenvironment (TIME) at the primary site can provide valuable insights into the aggressiveness of the tumor and the effectiveness of various treatment options. Spatially resolved transcriptomics technologies offer an unprecedented characterization of TIME through high multiplexing, yet their scope is constrained by cost. Meanwhile, it has long been suspected that histological, cytological, and macroarchitectural tissue characteristics correlate well with molecular information (e.g., gene expression). Thus, a method for predicting transcriptomics data through inference of RNA patterns from whole slide images (WSI) is a key step in studying metastasis at scale. In this work, we collected tissue from 4 stage-III (pT3) matched colorectal cancer patients for spatial transcriptomics profiling. The Visium spatial transcriptomics (ST) assay was used to measure transcript abundance for 17 943 genes at up to 5000 55-micron (i.e., 1-10 cells) spots per patient sampled in a honeycomb pattern, co-registered with hematoxylin and eosin (H&E) stained WSI. The Visium ST assay can measure expression at these spots through tissue permeabilization of mRNAs, which are captured through spatially (i.e., x-y positional coordinates) barcoded, gene specific oligo probes. WSI subimages were extracted around each co-registered Visium spot and were used to predict the expression at these spots using machine learning models. We prototyped and compared several convolutional, transformer, and graph convolutional neural networks to predict spatial RNA patterns at the Visium spots under the hypothesis that the transformer- and graph-based approaches better capture relevant spatial tissue architecture. We further analyzed the model's ability to recapitulate spatial autocorrelation statistics using SPARK and SpatialDE. Overall, the results indicate that the transformer- and graph-based approaches were unable to outperform the convolutional neural network architecture, though they exhibited optimal performance for relevant disease-associated genes. Initial findings suggest that different neural networks that operate on different scales are relevant for capturing distinct disease pathways (e.g., epithelial to mesenchymal transition). We add further evidence that deep learning models can accurately predict gene expression in whole slide images and comment on understudied factors which may increase its external applicability (e.g., tissue context). Our preliminary work will motivate further investigation of inference for molecular patterns from whole slide images as metastasis predictors and in other applications.

Learning to predict RNA sequence expressions from whole slide images with applications for search and classification

Deep learning methods are widely applied in digital pathology to address clinical challenges such as prognosis and diagnosis. As one of the most recent applications, deep models have also been used to extract molecular features from whole slide images. Although molecular tests carry rich information, they are often expensive, time-consuming, and require additional tissue to sample. In this paper, we propose tRNAsformer, an attention-based topology that can learn both to predict the bulk RNA-seq from an image and represent the whole slide image of a glass slide simultaneously. The tRNAsformer uses multiple instance learning to solve a weakly supervised problem while the pixel-level annotation is not available for an image. We conducted several experiments and achieved better performance and faster convergence in comparison to the state-of-the-art algorithms. The proposed tRNAsformer can assist as a computational pathology tool to facilitate a new generation of search and classification methods by combining the tissue morphology and the molecular fingerprint of the biopsy samples.

From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatment

Machine learning (ML) is increasingly used in clinical oncology to diagnose cancers, predict patient outcomes, and inform treatment planning. Here, we review recent applications of ML across the clinical oncology workflow. We review how these techniques are applied to medical imaging and to molecular data obtained from liquid and solid tumor biopsies for cancer diagnosis, prognosis, and treatment design. We discuss key considerations in developing ML for the distinct challenges posed by imaging and molecular data. Finally, we examine ML models approved for cancer-related patient usage by regulatory agencies and discuss approaches to improve the clinical usefulness of ML.

HISMD: A Novel Immune Subtyping System for HNSCC

Immune subtyping is an important way to reveal immune heterogeneity, which may contribute to the diversity of the progression and treatment in head and neck squamous cell carcinoma (HNSCC). However, reported immune subtypes mainly focus on levels of immune infiltration and are mostly based on a mono-omics profile. This study aimed to identify a comprehensive immune subtype for HNSCC via multi-omics clustering and build a novel subtype prediction system for clinical application. Data were obtained from The Cancer Genome Atlas database and our independent multicenter cohort. Multi-omics clustering was performed to identify 3 clusters of 499 patients in The Cancer Genome Atlas based on immune-related gene expression and somatic mutations. The immune characteristics and biological features of the obtained clusters were revealed by bioinformatics, and 3 immune subtypes were identified: 1) adaptive immune activation subtype predominantly enriched in T cells, 2) innate immune activation subtype predominantly enriched in macrophages, and 3) immune desert subtype. Subsequently, the clinical implications of each subtype were analyzed per clinical epidemiology. We found that adaptive immune activation showed better survival outcomes and had a similar response to chemotherapy with innate immune activation, whereas immune desert might be relatively resistant to chemotherapy. Moreover, a subtype prediction system was developed by deep learning with whole slide images and named HISMD: HNSCC Immune Subtypes via Multi-omics and Deep Learning. We endowed HISMD with interpretability through image-based key feature extraction. The clinical implications, biological significances, and predictive stability of HISMD were successfully verified by using our independent multicenter cohort data set. In summary, this study revealed the immune heterogeneity of HNSCC and obtained a novel, highly accurate, and interpretable immune subtyping prediction system. For clinical implementation in the future, additional validation and utility studies are warranted.

Machine Learning in Metastatic Cancer Research: Potentials, Possibilities, and Prospects

Cancer has received extensive recognition for its high mortality rate, with metastatic cancer being the top cause of cancer-related deaths. Metastatic cancer involves the spread of the primary tumor to other body organs. As much as the early detection of cancer is essential, the timely detection of metastasis, the identification of biomarkers, and treatment choice are valuable for improving the quality of life for metastatic cancer patients. This study reviews the existing studies on classical machine learning (ML) and deep learning (DL) in metastatic cancer research. Since the majority of metastatic cancer research data are collected in the formats of PET/CT and MRI image data, deep learning techniques are heavily involved. However, its black-box nature and expensive computational cost are notable concerns. Furthermore, existing models could be overestimated for their generality due to the non-diverse population in clinical trial datasets. Therefore, research gaps are itemized; follow-up studies should be carried out on metastatic cancer using machine learning and deep learning tools with data in a symmetric manner.

Mapping Spatiotemporal Heterogeneity in Tumor Profiles by Integrating High-Throughput Imaging and Omics Analysis

Intratumoral heterogeneity associates with more aggressive disease progression and worse patient outcomes. Understanding the reasons enabling the emergence of such heterogeneity remains incomplete, which restricts our ability to manage it from a therapeutic perspective. Technological advancements such as high-throughput molecular imaging, single-cell omics, and spatial transcriptomics allow recording of patterns of spatiotemporal heterogeneity in a longitudinal manner, thus offering insights into the multiscale dynamics of its evolution. Here, we review the latest technological trends and biological insights from molecular diagnostics as well as spatial transcriptomics, both of which have witnessed burgeoning growth in the recent past in terms of mapping heterogeneity within tumor cell types as well as the stromal constitution. We also discuss ongoing challenges, indicating possible ways to integrate insights across these methods to have a systems-level spatiotemporal map of heterogeneity in each tumor and a more systematic investigation of the implications of heterogeneity for patient outcomes.

Analysis of Spatial Molecular Data

Digital analysis of pathology whole-slide images has been recently gaining interest in the context of cancer diagnosis and treatment. In particular, deep learning methods have demonstrated significant potential in supporting pathology analysis, recently detecting molecular traits never before recognized in pathology H&E whole-slide images (WSIs). Alongside these advancements in the digital analysis of WSIs, it is becoming increasingly evident that both spatial and overall tumor heterogeneity may be significant determinants of cancer prognosis and treatment outcome. In this chapter, we describe methods that aim to support these two elements. We describe both an end-to-end deep learning pipeline for producing limited spatial transcriptomics from WSIs with associated bulk gene expression data, as well as an algorithm for quantifying spatial tumor heterogeneity based on the results of this pipeline.

Tumor infiltrating lymphocytes as an endpoint in cancer vaccine trials

Checkpoint inhibitors have invigorated cancer immunotherapy research, including cancer vaccination. Classic early phase trial design and endpoints used in developing chemotherapy are not suited for evaluating all forms of cancer treatment. Peripheral T cell response dynamics have demonstrated inconsistency in assessing the efficacy of cancer vaccination. Tumor infiltrating lymphocytes (TILs), reflect the local tumor microenvironment and may prove a superior endpoint in cancer vaccination trials. Cancer vaccines may also promote success in combination immunotherapy treatment of weakly immunogenic tumors. This review explores the impact of TILs as an endpoint for cancer vaccination in multiple malignancies, summarizes the current literature regarding TILs analysis, and discusses the challenges of providing validity and a standardized implementation of this approach.

Intratumoral spatial heterogeneity of tumor-infiltrating lymphocytes is a significant factor for precisely stratifying prognostic immune subgroups of microsatellite instability-high colorectal carcinomas

Although the density of tumor-infiltrating lymphocytes (TILs) is known to be linked to prognosis in various cancers, the prognostic impact and immunologic significance of the spatial heterogeneity of TILs have been rarely investigated. In this study, CD3+ and CD8+ TILs were quantified in independent cohorts (discovery, n = 73; and external validation, n = 93) of colorectal carcinomas (CRCs) with microsatellite instability-high (MSI-H) utilizing whole-slide image analysis of CD3/CD8 immunohistochemistry. The Shannon and Simpson indices, which measure intratumoral patch-to-patch evenness of TIL densities, were used to quantitatively assess the spatial heterogeneity of TILs in each case. To uncover immune-related gene expression signatures of spatial heterogeneity-based TIL subgroups of MSI-H CRCs, representative cases were subjected to GeoMx digital spatial profiler (DSP) analysis. As expected, a low density of TILs was significantly associated with poor disease-free survival (DFS) in MSI-H CRCs. The TIL-low tumors were further classified into two subgroups based on the spatial heterogeneity of TILs: TIL-low/heterogeneity-high and TIL-low/heterogeneity-low subgroups. In both discovery and validation cohorts, the TIL-low/heterogeneity-high, TIL-low/heterogeneity-low, and TIL-high subgroups were significantly associated with poor, intermediate, and good DFS, respectively. In the DSP analysis, the TIL-low/heterogeneity-high subgroup showed higher spatial diversity in the expression of immune-related genes than that of the TIL-low/heterogeneity-low subgroup and exhibited upregulation of genes related to immune checkpoints, chemokine/cytokine receptors, and myeloid cells. TIL-low/heterogeneity-high tumors were also enriched with gene sets related to good response to immune checkpoint inhibitor therapy. In conclusion, TIL-low MSI-H CRCs are prognostically heterogeneous and can be divided into prognostically and immunologically distinct subgroups by considering the spatial heterogeneity of TILs. Our data suggest that intratumoral spatial heterogeneity of TILs can be used as a key element for clinically relevant immunologic subtyping of tumors.

Computer-assisted tumor grading, validation of PD-L1 scoring, and quantification of CD8-positive immune cell density in urothelial carcinoma, a visual guide for pathologists using QuPath

Background

Advances in digital imaging in pathology and the new capacity to scan high-quality images have change the way to practice and research in surgical pathology. QuPath is an open-source pathology software that offers a reproducible way to analyze quantified variables. We aimed to present the functionality of biomarker scoring using QuPath and provide a guide for the validation of pathologic grading using a series of cases of urothelial carcinomas.

Methods

Tissue microarrays of urothelial carcinomas were constructed and scanned. The images stained with HE, CD8 and PD-L1 immunohistochemistry were imported into QuPath and dearrayed. Training images were used to build a grade classifier and applied to all cases. Quantification of CD8 and PD-L1 was undertaken for each core using cytoplasmic and membrane color segmentation and output measurement and compared with pathologists semi-quantitative assessments.

Results

There was a good correlation between tumor grade by the pathologist and by QuPath software (Kappa agreement 0.73). For low-grade carcinomas (by the report and pathologist), the concordance was not as high. Of the 32 low-grade tumors, 22 were correctly classified as low-grade, but 11 (34%) were diagnosed as high-grade, with the high-grade to the low-grade ratio in these misclassified cases ranging from 0.41 to 0.58. The median ratio for bona fide high-grade carcinomas was 0.59. Some of the reasons the authors list as potential mimickers for high-grade cases are fulguration artifact, nuclear hyperchromasia, folded tissues, and inconsistency in staining. The correlation analysis between the software and the pathologist showed that the CD8 marker showed a moderate (r = 0.595) and statistically significant (p < 0.001) correlation. The internal consistency of this parameter showed an index of 0.470. The correlation analysis between the software and the pathologist showed that the PDL1 marker showed a robust (r = 0.834) and significant (p < 0.001) correlation. The internal consistency of this parameter showed a CCI of 0.851.

Conclusions

We were able to demonstrate the utility of QuPath in identifying and scoring tumor cells and IHC quantification of two biomarkers. The protocol we present uses a free open-source platform to help researchers deal with imaging and data processing in the surgical pathology field.

Deep learning-based framework for slide-based histopathological image analysis

Digital pathology coupled with advanced machine learning (e.g., deep learning) has been changing the paradigm of whole-slide histopathological images (WSIs) analysis. Major applications in digital pathology using machine learning include automatic cancer classification, survival analysis, and subtyping from pathological images. While most pathological image analyses are based on patch-wise processing due to the extremely large size of histopathology images, there are several applications that predict a single clinical outcome or perform pathological diagnosis per slide (e.g., cancer classification, survival analysis). However, current slide-based analyses are task-dependent, and a general framework of slide-based analysis in WSI has been seldom investigated. We propose a novel slide-based histopathology analysis framework that creates a WSI representation map, called HipoMap, that can be applied to any slide-based problems, coupled with convolutional neural networks. HipoMap converts a WSI of various shapes and sizes to structured image-type representation. Our proposed HipoMap outperformed existing methods in intensive experiments with various settings and datasets. HipoMap showed the Area Under the Curve (AUC) of 0.96±0.026 (5% improved) in the experiments for lung cancer classification, and c-index of 0.787±0.013 (3.5% improved) and coefficient of determination ([Formula: see text]) of 0.978±0.032 (24% improved) in survival analysis and survival prediction with TCGA lung cancer data respectively, as a general framework of slide-based analysis with a flexible capability. The results showed significant improvement comparing to the current state-of-the-art methods on each task. We further discussed experimental results of HipoMap as pathological viewpoints and verified the performance using publicly available TCGA datasets. A Python package is available at https://pypi.org/project/hipomap , and the package can be easily installed using Python PIP. The open-source codes in Python are available at: https://github.com/datax-lab/HipoMap .

Deep learning-based prediction of molecular cancer biomarkers from tissue slides: A new tool for precision oncology

Molecular tests are necessary to stratify cancer patients for targeted therapy. However, high cost and technical barriers limit the application of these tests, hindering optimal treatment. Recently, deep learning (DL) has been applied to predict molecular test results from digitized images of tissue slides. Furthermore, treatment response and prognosis can be predicted from tissue slides using DL. In this review, we summarized DL-based studies regarding the prediction of genetic mutation, microsatellite instability, tumor mutational burden, molecular subtypes, gene expression, treatment response, and prognosis directly from hematoxylin- and eosin-stained tissue slides. Although performance needs to be improved, these studies clearly demonstrated the feasibility of DL-based prediction of key molecular features in cancer tissues. With the accumulation of data and technical advances, the performance of the DL system could be improved in the near future. Therefore, we expect that DL could provide cost- and time-effective alternative tools for patient stratification in the era of precision oncology.

Nondestructive protein sampling with electroporation facilitates profiling of spatial differential protein expression in breast tumors in vivo

Excision tissue biopsy, while central to cancer treatment and precision medicine, presents risks to the patient and does not provide a sufficiently broad and faithful representation of the heterogeneity of solid tumors. Here we introduce e-biopsy—a novel concept for molecular profiling of solid tumors using molecular sampling with electroporation. As e-biopsy provides access to the molecular composition of a solid tumor by permeabilization of the cell membrane, it facilitates tumor diagnostics without tissue resection. Furthermore, thanks to its non tissue destructive characteristics, e-biopsy enables probing the solid tumor multiple times in several distinct locations in the same procedure, thereby enabling the spatial profiling of tumor molecular heterogeneity.We demonstrate e-biopsy in vivo, using the 4T1 breast cancer model in mice to assess its performance, as well as the inferred spatial differential protein expression. In particular, we show that proteomic profiles obtained via e-biopsy in vivo distinguish the tumors from healthy breast tissue and reflect spatial tumor differential protein expression. E-biopsy provides a completely new molecular sampling modality for solid tumors molecular cartography, providing information that potentially enables more rapid and sensitive detection at lesser risk, as well as more precise personalized medicine.

Clinical application of advanced multi-omics tumor profiling: Shaping precision oncology of the future

Next-generation DNA sequencing technology has dramatically advanced clinical oncology through the identification of therapeutic targets and molecular biomarkers, leading to the personalization of cancer treatment with significantly improved outcomes for many common and rare tumor entities. More recent developments in advanced tumor profiling now enable dissection of tumor molecular architecture and the functional phenotype at cellular and subcellular resolution. Clinical translation of high-resolution tumor profiling and integration of multi-omics data into precision treatment, however, pose significant challenges at the level of prospective validation and clinical implementation. In this review, we summarize the latest advances in multi-omics tumor profiling, focusing on spatial genomics and chromatin organization, spatial transcriptomics and proteomics, liquid biopsy, and ex vivo modeling of drug response. We analyze the current stages of translational validation of these technologies and discuss future perspectives for their integration into precision treatment.

Artificial intelligence in histopathology: enhancing cancer research and clinical oncology

Artificial intelligence (AI) methods have multiplied our capabilities to extract quantitative information from digital histopathology images. AI is expected to reduce workload for human experts, improve the objectivity and consistency of pathology reports, and have a clinical impact by extracting hidden information from routinely available data. Here, we describe how AI can be used to predict cancer outcome, treatment response, genetic alterations and gene expression from digitized histopathology slides. We summarize the underlying technologies and emerging approaches, noting limitations, including the need for data sharing and standards. Finally, we discuss the broader implications of AI in cancer research and oncology.

Understanding Breast Cancers through Spatial and High-Resolution Visualization Using Imaging Technologies

Breast cancer is the most common cancer affecting women worldwide. Although many analyses and treatments have traditionally targeted the breast cancer cells themselves, recent studies have focused on investigating entire cancer tissues, including breast cancer cells. To understand the structure of breast cancer tissues, including breast cancer cells, it is necessary to investigate the three-dimensional location of the cells and/or proteins comprising the tissues and to clarify the relationship between the three-dimensional structure and malignant transformation or metastasis of breast cancers. In this review, we aim to summarize the methods for analyzing the three-dimensional structure of breast cancer tissue, paying particular attention to the recent technological advances in the combination of the tissue-clearing method and optical three-dimensional imaging. We also aimed to identify the latest methods for exploring the relationship between the three-dimensional cell arrangement in breast cancer tissues and the gene expression of each cell. Finally, we aimed to describe the three-dimensional imaging features of breast cancer tissues using noninvasive photoacoustic imaging methods.

Deep learning features encode interpretable morphologies within histological images

Convolutional neural networks (CNNs) are revolutionizing digital pathology by enabling machine learning-based classification of a variety of phenotypes from hematoxylin and eosin (H&E) whole slide images (WSIs), but the interpretation of CNNs remains difficult. Most studies have considered interpretability in a post hoc fashion, e.g. by presenting example regions with strongly predicted class labels. However, such an approach does not explain the biological features that contribute to correct predictions. To address this problem, here we investigate the interpretability of H&E-derived CNN features (the feature weights in the final layer of a transfer-learning-based architecture). While many studies have incorporated CNN features into predictive models, there has been little empirical study of their properties. We show such features can be construed as abstract morphological genes (“mones”) with strong independent associations to biological phenotypes. Many mones are specific to individual cancer types, while others are found in multiple cancers especially from related tissue types. We also observe that mone-mone correlations are strong and robustly preserved across related cancers. Importantly, linear mone-based classifiers can very accurately separate 38 distinct classes (19 tumor types and their adjacent normals, AUC = \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$97.1\% \pm 2.8\%$$\end{document}97.1%±2.8% for each class prediction), and linear classifiers are also highly effective for universal tumor detection (AUC = \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$99.2\% \pm 0.12\%$$\end{document}99.2%±0.12%). This linearity provides evidence that individual mones or correlated mone clusters may be associated with interpretable histopathological features or other patient characteristics. In particular, the statistical similarity of mones to gene expression values allows integrative mone analysis via expression-based bioinformatics approaches. We observe strong correlations between individual mones and individual gene expression values, notably mones associated with collagen gene expression in ovarian cancer. Mone-expression comparisons also indicate that immunoglobulin expression can be identified using mones in colon adenocarcinoma and that immune activity can be identified across multiple cancer types, and we verify these findings by expert histopathological review. Our work demonstrates that mones provide a morphological H&E decomposition that can be effectively associated with diverse phenotypes, analogous to the interpretability of transcription via gene expression values. Our work also demonstrates mones can be interpreted without using a classifier as a proxy.

Emerging Artificial Intelligence Applications in Spatial Transcriptomics Analysis

Spatial transcriptomics (ST) has advanced significantly in the last few years. Such advancement comes with the urgent need for novel computational methods to handle the unique challenges of ST data analysis. Many artificial intelligence (AI) methods have been developed to utilize various machine learning and deep learning techniques for computational ST analysis. This review provides a comprehensive and up-to-date survey of current AI methods for ST analysis.

SpatialMap: Spatial Mapping of Unmeasured Gene Expression Profiles in Spatial Transcriptomic Data Using Generalized Linear Spatial Models

Recent advances in various single-cell RNA sequencing (scRNA-seq) technologies have enabled profiling the gene expression level with the whole transcriptome at a single-cell resolution. However, it lacks the spatial context of tissues. The image-based transcriptomics in situ studies (e.g., MERFISH and seqFISH) maintain the cell spatial context at individual cell levels but can only measure a limited number of genes or transcripts (up to roughly 1,000 genes). Therefore, integrating scRNA-seq data and image-based transcriptomics data can potentially gain the complementary benefits of both. Here, we develop a computational method, SpatialMap, to bridge the gap, which primarily facilitates spatial mapping of unmeasured gene profiles in spatial transcriptomic data via integrating with scRNA-seq data from the same tissue. SpatialMap directly models the count nature of spatial gene expression data through generalized linear spatial models, which accounts for the spatial correlation among spatial locations using conditional autoregressive (CAR) prior. With a newly developed computationally efficient penalized quasi-likelihood (PQL)-based algorithm, SpatialMap can scale up to performing large-scale spatial mapping analysis. Finally, we applied the SpatialMap to four publicly available tissue-paired studies (i.e., scRNA-seq studies and image-based transcriptomics studies). The results demonstrate that the proposed method can accurately predict unmeasured gene expression profiles across various spatial and scRNA-seq dataset pairs of different species and technologies.

Reference-free cell type deconvolution of multi-cellular pixel-resolution spatially resolved transcriptomics data

Recent technological advancements have enabled spatially resolved transcriptomic profiling but at multi-cellular pixel resolution, thereby hindering the identification of cell-type-specific spatial patterns and gene expression variation. To address this challenge, we develop STdeconvolve as a reference-free approach to deconvolve underlying cell types comprising such multi-cellular pixel resolution spatial transcriptomics (ST) datasets. Using simulated as well as real ST datasets from diverse spatial transcriptomics technologies comprising a variety of spatial resolutions such as Spatial Transcriptomics, 10X Visium, DBiT-seq, and Slide-seq, we show that STdeconvolve can effectively recover cell-type transcriptional profiles and their proportional representation within pixels without reliance on external single-cell transcriptomics references. STdeconvolve provides comparable performance to existing reference-based methods when suitable single-cell references are available, as well as potentially superior performance when suitable single-cell references are not available. STdeconvolve is available as an open-source R software package with the source code available at https://github.com/JEFworks-Lab/STdeconvolve.

Identifying cell-type-specific spatial patterns in ST data is critical for understanding tissue organization but current methods rely on external references. Here the authors develop a reference-free method to effectively recover cell-type transcriptional profiles and proportions.

AI-enabled in silico immunohistochemical characterization for Alzheimer's disease

We develop a deep learning approach, in silico immunohistochemistry (IHC), which takes routinely collected histochemical-stained samples as input and computationally generates virtual IHC slide images. We apply in silico IHC to Alzheimer's disease samples, where several hallmark changes are conventionally identified using IHC staining across many regions of the brain. In silico IHC computationally identifies neurofibrillary tangles, β-amyloid plaques, and neuritic plaques at a high spatial resolution directly from the histochemical images, with areas under the receiver operating characteristic curve of between 0.88 and 0.92. In silico IHC learns to identify subtle cellular morphologies associated with these lesions and can generate in silico IHC slides that capture key features of the actual IHC.

Artificial intelligence to identify genetic alterations in conventional histopathology

Precision oncology relies on the identification of targetable molecular alterations in tumor tissues. In many tumor types, a limited set of molecular tests is currently part of standard diagnostic workflows. However, universal testing for all targetable alterations, especially rare ones, is limited by the cost and availability of molecular assays. From 2017 to 2021, multiple studies have shown that artificial intelligence (AI) methods can predict the probability of specific genetic alterations directly from conventional hematoxylin and eosin (H&E) tissue slides. Although these methods are currently less accurate than gold-standard testing (e.g. immunohistochemistry, polymerase chain reaction or next-generation sequencing), they could be used as pre-screening tools to reduce the workload of genetic analyses. In this systematic literature review, we summarize the state of the art in predicting molecular alterations from H&E using AI. We found that AI methods perform reasonably well across multiple tumor types, although few algorithms have been broadly validated. In addition, we found that genetic alterations in FGFR, IDH, PIK3CA, BRAF, TP53 and DNA repair pathways are predictable from H&E in multiple tumor types, while many other genetic alterations have rarely been investigated or were only poorly predictable. Finally, we discuss the next steps for the implementation of AI-based surrogate tests in diagnostic workflows. This article is protected by copyright. All rights reserved.