Deep learning-enabled assessment of cardiac allograft rejection from endomyocardial biopsies

Unsupervised mutual transformer learning for multi-gigapixel Whole Slide Image classification

The classification of gigapixel Whole Slide Images (WSIs) is an important task in the emerging area of computational pathology. There has been a surge of interest in deep learning models for WSI classification with clinical applications such as cancer detection or prediction of cellular mutations. Most supervised methods require expensive and labor-intensive manual annotations by expert pathologists. Weakly supervised Multiple Instance Learning (MIL) methods have recently demonstrated excellent performance; however, they still require large-scale slide-level labeled training datasets that require a careful inspection of each slide by an expert pathologist. In this work, we propose a fully unsupervised WSI classification algorithm based on mutual transformer learning. The instances (i.e., patches) from gigapixel WSIs are transformed into a latent space and then inverse-transformed to the original space. Using the transformation loss, pseudo labels are generated and cleaned using a transformer label cleaner. The proposed transformer-based pseudo-label generator and cleaner modules mutually train each other iteratively in an unsupervised manner. A discriminative learning mechanism is introduced to improve normal versus cancerous instance labeling. In addition to the unsupervised learning, we demonstrate the effectiveness of the proposed framework for weakly supervised learning and cancer subtype classification as downstream analysis. Extensive experiments on four publicly available datasets show better performance of the proposed algorithm compared to the existing state-of-the-art methods.

Intrinsically explainable deep learning architecture for semantic segmentation of histological structures in heart tissue

BACKGROUND

Analysis of structures contained in tissue samples and the relevant contextual information is of utmost importance to histopathologists during diagnosis. Cardiac biopsies require in-depth analysis of the relationships between biological structures. Statistical measures are insufficient for determining a model's viability and applicability in the diagnostic process. A deeper understanding of predictions is necessary in order to support histopathologists.

METHODS

We propose a method for providing supporting information in the form of segmentation of histological structures to histopathologists based on these principles. The proposed method utilizes nuclei type and density information in addition to standard image input provided at two different zoom levels for the semantic segmentation of blood vessels, inflammation, and endocardium in heart tissue.

RESULTS

The proposed method was able to reach state-of-the-art segmentation results. The overall quality and viability of the predictions was qualitatively evaluated by two pathologists and a histotechnologist.

CONCLUSIONS

The decision process of the proposed deep learning model utilizes the provided information sources correctly and simulates the decision process of histopathologists via the usage of a custom-designed attention gate that provides a combination of spatial and encoder attention mechanisms. The implementation is available at https://github.com/mathali/IEDL-segmentation-of-heart-tissue.

Revolutionizing Cardiology through Artificial Intelligence-Big Data from Proactive Prevention to Precise Diagnostics and Cutting-Edge Treatment-A Comprehensive Review of the Past 5 Years

BACKGROUND

Artificial intelligence (AI) can radically change almost every aspect of the human experience. In the medical field, there are numerous applications of AI and subsequently, in a relatively short time, significant progress has been made. Cardiology is not immune to this trend, this fact being supported by the exponential increase in the number of publications in which the algorithms play an important role in data analysis, pattern discovery, identification of anomalies, and therapeutic decision making. Furthermore, with technological development, there have appeared new models of machine learning (ML) and deep learning (DP) that are capable of exploring various applications of AI in cardiology, including areas such as prevention, cardiovascular imaging, electrophysiology, interventional cardiology, and many others. In this sense, the present article aims to provide a general vision of the current state of AI use in cardiology.

RESULTS

We identified and included a subset of 200 papers directly relevant to the current research covering a wide range of applications. Thus, this paper presents AI applications in cardiovascular imaging, arithmology, clinical or emergency cardiology, cardiovascular prevention, and interventional procedures in a summarized manner. Recent studies from the highly scientific literature demonstrate the feasibility and advantages of using AI in different branches of cardiology.

CONCLUSIONS

The integration of AI in cardiology offers promising perspectives for increasing accuracy by decreasing the error rate and increasing efficiency in cardiovascular practice. From predicting the risk of sudden death or the ability to respond to cardiac resynchronization therapy to the diagnosis of pulmonary embolism or the early detection of valvular diseases, AI algorithms have shown their potential to mitigate human error and provide feasible solutions. At the same time, limits imposed by the small samples studied are highlighted alongside the challenges presented by ethical implementation; these relate to legal implications regarding responsibility and decision making processes, ensuring patient confidentiality and data security. All these constitute future research directions that will allow the integration of AI in the progress of cardiology.

Computational Pathology Assessments of Cardiac Stromal Remodeling: Clinical Correlates and Prognostic Implications in Heart Transplantation

Both overt and indolent inflammatory insults in heart transplantation can accelerate pathologic cardiac remodeling, but there are few tools for monitoring the speed and severity of remodeling over time. To address this need, we developed an automated computational pathology system to measure pathologic remodeling in transplant biopsy samples in a large, retrospective cohort of n=2167 digitized heart transplant biopsy slides. Biopsy images were analyzed to identify the pathologic stromal changes associated with future allograft loss or advanced allograft vasculopathy. Biopsy images were then analyzed to assess which historical allo-inflammatory events drive progression of these pathologic stromal changes over time in serial biopsy samples. The top-5 features of pathologic stromal remodeling most strongly associated with adverse outcomes were also strongly associated with histories of both overt and indolent inflammatory events. Our findings identify previously unappreciated subgroups of higher- and lower-risk transplant patients, and highlight the translational potential of digital pathology analysis.

Analysis of 3D pathology samples using weakly supervised AI

Human tissue, which is inherently three-dimensional (3D), is traditionally examined through standard-of-care histopathology as limited two-dimensional (2D) cross-sections that can insufficiently represent the tissue due to sampling bias. To holistically characterize histomorphology, 3D imaging modalities have been developed, but clinical translation is hampered by complex manual evaluation and lack of computational platforms to distill clinical insights from large, high-resolution datasets. We present TriPath, a deep-learning platform for processing tissue volumes and efficiently predicting clinical outcomes based on 3D morphological features. Recurrence risk-stratification models were trained on prostate cancer specimens imaged with open-top light-sheet microscopy or microcomputed tomography. By comprehensively capturing 3D morphologies, 3D volume-based prognostication achieves superior performance to traditional 2D slice-based approaches, including clinical/histopathological baselines from six certified genitourinary pathologists. Incorporating greater tissue volume improves prognostic performance and mitigates risk prediction variability from sampling bias, further emphasizing the value of capturing larger extents of heterogeneous morphology.

A machine learning algorithm improves the diagnostic accuracy of the histologic component of antibody mediated rejection (AMR-H) in cardiac transplant endomyocardial biopsies

BACKGROUND

Pathologic antibody mediated rejection (pAMR) remains a major driver of graft failure in cardiac transplant patients. The endomyocardial biopsy remains the primary diagnostic tool but presents with challenges, particularly in distinguishing the histologic component (pAMR-H) defined by 1) intravascular macrophage accumulation in capillaries and 2) activated endothelial cells that expand the cytoplasm to narrow or occlude the vascular lumen. Frequently, pAMR-H is difficult to distinguish from acute cellular rejection (ACR) and healing injury. With the advent of digital slide scanning and advances in machine deep learning, artificial intelligence technology is widely under investigation in the areas of oncologic pathology, but in its infancy in transplant pathology. For the first time, we determined if a machine learning algorithm could distinguish pAMR-H from normal myocardium, healing injury and ACR.

MATERIALS AND METHODS

A total of 4,212 annotations (1,053 regions of normal, 1,053 pAMR-H, 1,053 healing injury and 1,053 ACR) were completed from 300 hematoxylin and eosin slides scanned using a Leica Aperio GT450 digital whole slide scanner at 40X magnification. All regions of pAMR-H were annotated from patients confirmed with a previous diagnosis of pAMR2 (>50% positive C4d immunofluorescence and/or >10% CD68 positive intravascular macrophages). Annotations were imported into a Python 3.7 development environment using the OpenSlide™ package and a convolutional neural network approach utilizing transfer learning was performed.

RESULTS

The machine learning algorithm showed 98% overall validation accuracy and pAMR-H was correctly distinguished from specific categories with the following accuracies: normal myocardium (99.2%), healing injury (99.5%) and ACR (99.5%).

CONCLUSION

Our novel deep learning algorithm can reach acceptable, and possibly surpass, performance of current diagnostic standards of identifying pAMR-H. Such a tool may serve as an adjunct diagnostic aid for improving the pathologist's accuracy and reproducibility, especially in difficult cases with high inter-observer variability. This is one of the first studies that provides evidence that an artificial intelligence machine learning algorithm can be trained and validated to diagnose pAMR-H in cardiac transplant patients. Ongoing studies include multi-institutional verification testing to ensure generalizability.

Demographic bias in misdiagnosis by computational pathology models

Despite increasing numbers of regulatory approvals, deep learning-based computational pathology systems often overlook the impact of demographic factors on performance, potentially leading to biases. This concern is all the more important as computational pathology has leveraged large public datasets that underrepresent certain demographic groups. Using publicly available data from The Cancer Genome Atlas and the EBRAINS brain tumor atlas, as well as internal patient data, we show that whole-slide image classification models display marked performance disparities across different demographic groups when used to subtype breast and lung carcinomas and to predict IDH1 mutations in gliomas. For example, when using common modeling approaches, we observed performance gaps (in area under the receiver operating characteristic curve) between white and Black patients of 3.0% for breast cancer subtyping, 10.9% for lung cancer subtyping and 16.0% for IDH1 mutation prediction in gliomas. We found that richer feature representations obtained from self-supervised vision foundation models reduce performance variations between groups. These representations provide improvements upon weaker models even when those weaker models are combined with state-of-the-art bias mitigation strategies and modeling choices. Nevertheless, self-supervised vision foundation models do not fully eliminate these discrepancies, highlighting the continuing need for bias mitigation efforts in computational pathology. Finally, we demonstrate that our results extend to other demographic factors beyond patient race. Given these findings, we encourage regulatory and policy agencies to integrate demographic-stratified evaluation into their assessment guidelines.

Towards a general-purpose foundation model for computational pathology

Quantitative evaluation of tissue images is crucial for computational pathology (CPath) tasks, requiring the objective characterization of histopathological entities from whole-slide images (WSIs). The high resolution of WSIs and the variability of morphological features present significant challenges, complicating the large-scale annotation of data for high-performance applications. To address this challenge, current efforts have proposed the use of pretrained image encoders through transfer learning from natural image datasets or self-supervised learning on publicly available histopathology datasets, but have not been extensively developed and evaluated across diverse tissue types at scale. We introduce UNI, a general-purpose self-supervised model for pathology, pretrained using more than 100 million images from over 100,000 diagnostic H&E-stained WSIs (>77 TB of data) across 20 major tissue types. The model was evaluated on 34 representative CPath tasks of varying diagnostic difficulty. In addition to outperforming previous state-of-the-art models, we demonstrate new modeling capabilities in CPath such as resolution-agnostic tissue classification, slide classification using few-shot class prototypes, and disease subtyping generalization in classifying up to 108 cancer types in the OncoTree classification system. UNI advances unsupervised representation learning at scale in CPath in terms of both pretraining data and downstream evaluation, enabling data-efficient artificial intelligence models that can generalize and transfer to a wide range of diagnostically challenging tasks and clinical workflows in anatomic pathology.

Adopting artificial intelligence in cardiovascular medicine: a scoping review

Recent years have witnessed significant transformations in cardiovascular medicine, driven by the rapid evolution of artificial intelligence (AI). This scoping review was conducted to capture the breadth of AI applications within cardiovascular science. Employing a structured approach, we sourced relevant articles from PubMed, with an emphasis on journals encompassing general cardiology and digital medicine. We applied filters to highlight cardiovascular articles published in journals focusing on general internal medicine, cardiology and digital medicine, thereby identifying the prevailing trends in the field. Following a comprehensive full-text screening, a total of 140 studies were identified. Over the preceding 5 years, cardiovascular medicine's interplay with AI has seen an over tenfold augmentation. This expansive growth encompasses multiple cardiovascular subspecialties, including but not limited to, general cardiology, ischemic heart disease, heart failure, and arrhythmia. Deep learning emerged as the predominant methodology. The majority of AI endeavors in this domain have been channeled toward enhancing diagnostic and prognostic capabilities, utilizing resources such as hospital datasets, electrocardiograms, and echocardiography. A significant uptrend was observed in AI's application for omics data analysis. However, a clear gap persists in AI's full-scale integration into the clinical decision-making framework. AI, particularly deep learning, has demonstrated robust applications across cardiovascular subspecialties, indicating its transformative potential in this field. As we continue on this trajectory, ensuring the alignment of technological progress with medical ethics becomes crucial. The abundant digital health data today further accentuates the need for meticulous systematic reviews, tailoring them to each cardiovascular subspecialty.

Tissue Contamination Challenges the Credibility of Machine Learning Models in Real World Digital Pathology

Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. Although human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole-slide models. Three operate in placenta for the following functions: (1) detection of decidual arteriopathy, (2) estimation of gestational age, and (3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in the t-distributed stochastic neighbor embedding feature space. Every model showed performance degradation in response to one or more tissue contaminants. Decidual arteriopathy detection--balanced accuracy decreased from 0.74 to 0.69 ± 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant, raised the mean absolute error in estimating gestational age from 1.626 weeks to 2.371 ± 0.003 weeks. Blood, incorporated into placental sections, induced false-negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033 mm2, and resulted in a 97% false-positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.

Regression-based Deep-Learning predicts molecular biomarkers from pathology slides

Deep Learning (DL) can predict biomarkers from cancer histopathology. Several clinically approved applications use this technology. Most approaches, however, predict categorical labels, whereas biomarkers are often continuous measurements. We hypothesize that regression-based DL outperforms classification-based DL. Therefore, we develop and evaluate a self-supervised attention-based weakly supervised regression method that predicts continuous biomarkers directly from 11,671 images of patients across nine cancer types. We test our method for multiple clinically and biologically relevant biomarkers: homologous recombination deficiency score, a clinically used pan-cancer biomarker, as well as markers of key biological processes in the tumor microenvironment. Using regression significantly enhances the accuracy of biomarker prediction, while also improving the predictions' correspondence to regions of known clinical relevance over classification. In a large cohort of colorectal cancer patients, regression-based prediction scores provide a higher prognostic value than classification-based scores. Our open-source regression approach offers a promising alternative for continuous biomarker analysis in computational pathology.

Failing to Make the Grade: Conventional Cardiac Allograft Rejection Grading Criteria Are Inadequate for Predicting Rejection Severity

BACKGROUND

Cardiac allograft rejection is the leading cause of early graft failure and is a major focus of postheart transplant patient care. While histological grading of endomyocardial biopsy samples remains the diagnostic standard for acute rejection, this standard has limited diagnostic accuracy. Discordance between biopsy rejection grade and patient clinical trajectory frequently leads to both overtreatment of indolent processes and delayed treatment of aggressive ones, spurring the need to investigate the adequacy of the current histological criteria for assessing clinically important rejection outcomes.

METHODS

N=2900 endomyocardial biopsy images were assigned a rejection grade label (high versus low grade) and a clinical trajectory label (evident versus silent rejection). Using an image analysis approach, n=370 quantitative morphology features describing the lymphocytes and stroma were extracted from each slide. Two models were constructed to compare the subset of features associated with rejection grades versus those associated with clinical trajectories. A proof-of-principle machine learning pipeline-the cardiac allograft rejection evaluator-was then developed to test the feasibility of identifying the clinical severity of a rejection event.

RESULTS

The histopathologic findings associated with conventional rejection grades differ substantially from those associated with clinically evident allograft injury. Quantitative assessment of a small set of well-defined morphological features can be leveraged to more accurately reflect the severity of rejection compared with that achieved by the International Society of Heart and Lung Transplantation grades.

CONCLUSIONS

Conventional endomyocardial samples contain morphological information that enables accurate identification of clinically evident rejection events, and this information is incompletely captured by the current, guideline-endorsed, rejection grading criteria.

Digital pathology in cardiac transplant diagnostics: from biopsies to algorithms

In the field of heart transplantation, the ability to accurately and promptly diagnose cardiac allograft rejection is crucial. This comprehensive review explores the transformative role of digital pathology and computational pathology, especially through machine learning, in this critical domain. These methodologies harness large datasets to extract subtle patterns and valuable information that extend beyond human perceptual capabilities, potentially enhancing diagnostic outcomes. Current research indicates that these computer-based systems could offer accuracy and performance matching, or even exceeding, that of expert pathologists, thereby introducing more objectivity and reducing observer variability. Despite promising results, several challenges such as limited sample sizes, diverse data sources, and the absence of standardized protocols pose significant barriers to the widespread adoption of these techniques. The future of digital pathology in heart transplantation diagnostics depends on utilizing larger, more diverse patient cohorts, standardizing data collection, processing, and evaluation protocols, and fostering collaborative research efforts. The integration of various data types, including clinical, demographic, and imaging information, could further refine diagnostic precision. As researchers address these challenges and promote collaborative efforts, digital pathology has the potential to become an integral part of clinical practice, ultimately improving patient care in heart transplantation.

An efficient context-aware approach for whole-slide image classification

Computational pathology for gigapixel whole-slide images (WSIs) at slide level is helpful in disease diagnosis and remains challenging. We propose a context-aware approach termed WSI inspection via transformer (WIT) for slide-level classification via holistically modeling dependencies among patches on WSI. WIT automatically learns feature representation of WSI by aggregating features of all image patches. We evaluate classification performance of WIT and state-of-the-art baseline method. WIT achieved an accuracy of 82.1% (95% CI, 80.7%-83.3%) in the detection of 32 cancer types on the TCGA dataset, 0.918 (0.910-0.925) in diagnosis of cancer on the CPTAC dataset, and 0.882 (0.87-0.890) in the diagnosis of prostate cancer from needle biopsy slide, outperforming the baseline by 31.6%, 5.4%, and 9.3%, respectively. WIT can pinpoint the WSI regions that are most influential for its decision. WIT represents a new paradigm for computational pathology, facilitating the development of digital pathology tools.

Shared-Specific Feature Learning With Bottleneck Fusion Transformer for Multi-Modal Whole Slide Image Analysis

The fusion of multi-modal medical data is essential to assist medical experts to make treatment decisions for precision medicine. For example, combining the whole slide histopathological images (WSIs) and tabular clinical data can more accurately predict the lymph node metastasis (LNM) of papillary thyroid carcinoma before surgery to avoid unnecessary lymph node resection. However, the huge-sized WSI provides much more high-dimensional information than low-dimensional tabular clinical data, making the information alignment challenging in the multi-modal WSI analysis tasks. This paper presents a novel transformer-guided multi-modal multi-instance learning framework to predict lymph node metastasis from both WSIs and tabular clinical data. We first propose an effective multi-instance grouping scheme, named siamese attention-based feature grouping (SAG), to group high-dimensional WSIs into representative low-dimensional feature embeddings for fusion. We then design a novel bottleneck shared-specific feature transfer module (BSFT) to explore the shared and specific features between different modalities, where a few learnable bottleneck tokens are utilized for knowledge transfer between modalities. Moreover, a modal adaptation and orthogonal projection scheme were incorporated to further encourage BSFT to learn shared and specific features from multi-modal data. Finally, the shared and specific features are dynamically aggregated via an attention mechanism for slide-level prediction. Experimental results on our collected lymph node metastasis dataset demonstrate the efficiency of our proposed components and our framework achieves the best performance with AUC (area under the curve) of 97.34%, outperforming the state-of-the-art methods by over 1.27%.

Banff Digital Pathology Working Group: Image Bank, Artificial Intelligence Algorithm, and Challenge Trial Developments

The Banff Digital Pathology Working Group (DPWG) was established with the goal to establish a digital pathology repository; develop, validate, and share models for image analysis; and foster collaborations using regular videoconferencing. During the calls, a variety of artificial intelligence (AI)-based support systems for transplantation pathology were presented. Potential collaborations in a competition/trial on AI applied to kidney transplant specimens, including the DIAGGRAFT challenge (staining of biopsies at multiple institutions, pathologists' visual assessment, and development and validation of new and pre-existing Banff scoring algorithms), were also discussed. To determine the next steps, a survey was conducted, primarily focusing on the feasibility of establishing a digital pathology repository and identifying potential hosts. Sixteen of the 35 respondents (46%) had access to a server hosting a digital pathology repository, with 2 respondents that could serve as a potential host at no cost to the DPWG. The 16 digital pathology repositories collected specimens from various organs, with the largest constituent being kidney (n = 12,870 specimens). A DPWG pilot digital pathology repository was established, and there are plans for a competition/trial with the DIAGGRAFT project. Utilizing existing resources and previously established models, the Banff DPWG is establishing new resources for the Banff community.

Neuropathologist-level integrated classification of adult-type diffuse gliomas using deep learning from whole-slide pathological images

Current diagnosis of glioma types requires combining both histological features and molecular characteristics, which is an expensive and time-consuming procedure. Determining the tumor types directly from whole-slide images (WSIs) is of great value for glioma diagnosis. This study presents an integrated diagnosis model for automatic classification of diffuse gliomas from annotation-free standard WSIs. Our model is developed on a training cohort (n = 1362) and a validation cohort (n = 340), and tested on an internal testing cohort (n = 289) and two external cohorts (n = 305 and 328, respectively). The model can learn imaging features containing both pathological morphology and underlying biological clues to achieve the integrated diagnosis. Our model achieves high performance with area under receiver operator curve all above 0.90 in classifying major tumor types, in identifying tumor grades within type, and especially in distinguishing tumor genotypes with shared histological features. This integrated diagnosis model has the potential to be used in clinical scenarios for automated and unbiased classification of adult-type diffuse gliomas.

Multiomics dynamic learning enables personalized diagnosis and prognosis for pancancer and cancer subtypes

Artificial intelligence (AI) approaches in cancer analysis typically utilize a 'one-size-fits-all' methodology characterizing average patient responses. This manner neglects the diverse conditions in the pancancer and cancer subtypes of individual patients, resulting in suboptimal outcomes in diagnosis and treatment. To overcome this limitation, we shift from a blanket application of statistics to a focus on the explicit recognition of patient-specific abnormalities. Our objective is to use multiomics data to empower clinicians with personalized molecular descriptions that allow for customized diagnosis and interventions. Here, we propose a highly trustworthy multiomics learning (HTML) framework that employs multiomics self-adaptive dynamic learning to process each sample with data-dependent architectures and computational flows, ensuring personalized and trustworthy patient-centering of cancer diagnosis and prognosis. Extensive testing on a 33-type pancancer dataset and 12 cancer subtype datasets underscored the superior performance of HTML compared with static-architecture-based methods. Our findings also highlighting the potential of HTML in elucidating complex biological pathogenesis and paving the way for improved patient-specific care in cancer treatment.

Artificial Intelligence Advances in Transplant Pathology

Transplant pathology plays a critical role in ensuring that transplanted organs function properly and the immune systems of the recipients do not reject them. To improve outcomes for transplant recipients, accurate diagnosis and timely treatment are essential. Recent advances in artificial intelligence (AI)-empowered digital pathology could help monitor allograft rejection and weaning of immunosuppressive drugs. To explore the role of AI in transplant pathology, we conducted a systematic search of electronic databases from January 2010 to April 2023. The PRISMA checklist was used as a guide for screening article titles, abstracts, and full texts, and we selected articles that met our inclusion criteria. Through this search, we identified 68 articles from multiple databases. After careful screening, only 14 articles were included based on title and abstract. Our review focuses on the AI approaches applied to four transplant organs: heart, lungs, liver, and kidneys. Specifically, we found that several deep learning-based AI models have been developed to analyze digital pathology slides of biopsy specimens from transplant organs. The use of AI models could improve clinicians' decision-making capabilities and reduce diagnostic variability. In conclusion, our review highlights the advancements and limitations of AI in transplant pathology. We believe that these AI technologies have the potential to significantly improve transplant outcomes and pave the way for future advancements in this field.

A distinct stimulatory cDC1 subpopulation amplifies CD8+ T cell responses in tumors for protective anti-cancer immunity

Type 1 conventional dendritic cells (cDC1) can support T cell responses within tumors but whether this determines protective versus ineffective anti-cancer immunity is poorly understood. Here, we use imaging-based deep learning to identify intratumoral cDC1-CD8+ T cell clustering as a unique feature of protective anti-cancer immunity. These clusters form selectively in stromal tumor regions and constitute niches in which cDC1 activate TCF1+ stem-like CD8+ T cells. We identify a distinct population of immunostimulatory CCR7neg cDC1 that produce CXCL9 to promote cluster formation and cross-present tumor antigens within these niches, which is required for intratumoral CD8+ T cell differentiation and expansion and promotes cancer immune control. Similarly, in human cancers, CCR7neg cDC1 interact with CD8+ T cells in clusters and are associated with patient survival. Our findings reveal an intratumoral phase of the anti-cancer T cell response orchestrated by tumor-residing cDC1 that determines protective versus ineffective immunity and could be exploited for cancer therapy.

An aggregation of aggregation methods in computational pathology

Image analysis and machine learning algorithms operating on multi-gigapixel whole-slide images (WSIs) often process a large number of tiles (sub-images) and require aggregating predictions from the tiles in order to predict WSI-level labels. In this paper, we present a review of existing literature on various types of aggregation methods with a view to help guide future research in the area of computational pathology (CPath). We propose a general CPath workflow with three pathways that consider multiple levels and types of data and the nature of computation to analyse WSIs for predictive modelling. We categorize aggregation methods according to the context and representation of the data, features of computational modules and CPath use cases. We compare and contrast different methods based on the principle of multiple instance learning, perhaps the most commonly used aggregation method, covering a wide range of CPath literature. To provide a fair comparison, we consider a specific WSI-level prediction task and compare various aggregation methods for that task. Finally, we conclude with a list of objectives and desirable attributes of aggregation methods in general, pros and cons of the various approaches, some recommendations and possible future directions.

Overcoming the challenges to implementation of artificial intelligence in pathology

Pathologists worldwide are facing remarkable challenges with increasing workloads and lack of time to provide consistently high-quality patient care. The application of artificial intelligence (AI) to digital whole-slide images has the potential of democratizing the access to expert pathology and affordable biomarkers by supporting pathologists in the provision of timely and accurate diagnosis as well as supporting oncologists by directly extracting prognostic and predictive biomarkers from tissue slides. The long-awaited adoption of AI in pathology, however, has not materialized, and the transformation of pathology is happening at a much slower pace than that observed in other fields (eg, radiology). Here, we provide a critical summary of the developments in digital and computational pathology in the last 10 years, outline key hurdles and ways to overcome them, and provide a perspective for AI-supported precision oncology in the future.

Hierarchical discriminative learning improves visual representations of biomedical microscopy

Learning high-quality, self-supervised, visual representations is essential to advance the role of computer vision in biomedical microscopy and clinical medicine. Previous work has focused on self-supervised representation learning (SSL) methods developed for instance discrimination and applied them directly to image patches, or fields-of-view, sampled from gigapixel whole-slide images (WSIs) used for cancer diagnosis. However, this strategy is limited because it (1) assumes patches from the same patient are independent, (2) neglects the patient-slide-patch hierarchy of clinical biomedical microscopy, and (3) requires strong data augmentations that can degrade downstream performance. Importantly, sampled patches from WSIs of a patient's tumor are a diverse set of image examples that capture the same underlying cancer diagnosis. This motivated HiDisc, a data-driven method that leverages the inherent patient-slide-patch hierarchy of clinical biomedical microscopy to define a hierarchical discriminative learning task that implicitly learns features of the underlying diagnosis. HiDisc uses a self-supervised contrastive learning framework in which positive patch pairs are defined based on a common ancestry in the data hierarchy, and a unified patch, slide, and patient discriminative learning objective is used for visual SSL. We benchmark HiDisc visual representations on two vision tasks using two biomedical microscopy datasets, and demonstrate that (1) HiDisc pretraining outperforms current state-of-the-art self-supervised pretraining methods for cancer diagnosis and genetic mutation prediction, and (2) HiDisc learns high-quality visual representations using natural patch diversity without strong data augmentations.

Algorithmic fairness in artificial intelligence for medicine and healthcare

In healthcare, the development and deployment of insufficiently fair systems of artificial intelligence (AI) can undermine the delivery of equitable care. Assessments of AI models stratified across subpopulations have revealed inequalities in how patients are diagnosed, treated and billed. In this Perspective, we outline fairness in machine learning through the lens of healthcare, and discuss how algorithmic biases (in data acquisition, genetic variation and intra-observer labelling variability, in particular) arise in clinical workflows and the resulting healthcare disparities. We also review emerging technology for mitigating biases via disentanglement, federated learning and model explainability, and their role in the development of AI-based software as a medical device.

Tissue contamination challenges the credibility of machine learning models in real world digital pathology

Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. While human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole slide models. Three operate in placenta for 1) detection of decidual arteriopathy (DA), 2) estimation of gestational age (GA), and 3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in T-distributed Stochastic Neighbor Embedding (tSNE) feature space. Every model showed performance degradation in response to one or more tissue contaminants. DA detection balanced accuracy decreased from 0.74 to 0.69 +/- 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant raised the mean absolute error in estimating gestation age from 1.626 weeks to 2.371 +/ 0.003 weeks. Blood, incorporated into placental sections, induced false negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033mm2, resulted in a 97% false positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.

Prediction of heart transplant rejection from routine pathology slides with self-supervised deep learning

Aims

One of the most important complications of heart transplantation is organ rejection, which is diagnosed on endomyocardial biopsies by pathologists. Computer-based systems could assist in the diagnostic process and potentially improve reproducibility. Here, we evaluated the feasibility of using deep learning in predicting the degree of cellular rejection from pathology slides as defined by the International Society for Heart and Lung Transplantation (ISHLT) grading system.

Methods and results

We collected 1079 histopathology slides from 325 patients from three transplant centres in Germany. We trained an attention-based deep neural network to predict rejection in the primary cohort and evaluated its performance using cross-validation and by deploying it to three cohorts. For binary prediction (rejection yes/no), the mean area under the receiver operating curve (AUROC) was 0.849 in the cross-validated experiment and 0.734, 0.729, and 0.716 in external validation cohorts. For a prediction of the ISHLT grade (0R, 1R, 2/3R), AUROCs were 0.835, 0.633, and 0.905 in the cross-validated experiment and 0.764, 0.597, and 0.913; 0.631, 0.633, and 0.682; and 0.722, 0.601, and 0.805 in the validation cohorts, respectively. The predictions of the artificial intelligence model were interpretable by human experts and highlighted plausible morphological patterns.

Conclusion

We conclude that artificial intelligence can detect patterns of cellular transplant rejection in routine pathology, even when trained on small cohorts.

Convolutional neural network models for automatic diagnosis and graduation in skin frostbite

The study aimed to develop and validate a convolutional neural network (CNN)-based deep learning method for automatic diagnosis and graduation of skin frostbite. A dataset of 71 annotated images was used for the training, the validation, and the testing based on ResNet-50 model. The performances were evaluated with the test set. The diagnosis and graduation performance of our approach was compared with two residents from burns department. The approach correctly identified all the frostbite of IV (18/18, 100%), but with respectively 1 mistake in the diagnosis of degree I (29/30, 96.67%), II (28/29, 96.55%) and III (37/38, 97.37%). The accuracy of the approach on the whole test set was 97.39% (112/115). The accuracy of the two residents were respectively 77.39% and 73.04%. Weighted Kappa of 0.583 indicates good reliability between the two residents (P = .445). Kendall's coefficient of concordance is 0.326 (P = .548), indicating differences in accuracy between the approach and the two residents. Our approach based on CNNs demonstrated an encouraging performance for the automatic diagnosis and graduation of skin frostbite, with higher accuracy and efficiency.

Machine learning classification of placental villous infarction, perivillous fibrin deposition, and intervillous thrombus

INTRODUCTION

Placental parenchymal lesions are commonly encountered and carry significant clinical associations. However, they are frequently missed or misclassified by general practice pathologists. Interpretation of pathology slides has emerged as one of the most successful applications of machine learning (ML) in medicine with applications ranging from cancer detection and prognostication to transplant medicine. The goal of this study was to use a whole-slide learning model to identify and classify placental parenchymal lesions including villous infarctions, intervillous thrombi (IVT), and perivillous fibrin deposition (PVFD).

METHODS

We generated whole slide images from placental discs examined at our institution with infarct, IVT, PVFD, or no macroscopic lesion. Slides were analyzed as a set of overlapping patches. We extracted feature vectors from each patch using a pretrained convolutional neural network (EfficientNetV2L). We trained a model to assign attention to each vector and used the attentions as weights to produce a pooled feature vector. The pooled vector was classified as normal or 1 of 3 lesions using a fully connected network. Patch attention was plotted to highlight informative areas of the slide.

RESULTS

Overall balanced accuracy in a test set of held-out slides was 0.86 with receiver-operator characteristic areas under the curve of 0.917-0.993. Cases of PVFD were frequently miscalled as normal or infarcts, the latter possibly due to the perivillous fibrin found at the periphery of infarctions. We used attention maps to further understand some errors, including one most likely due to poor tissue fixation and processing.

DISCUSSION

We used a whole-slide learning paradigm to train models to recognize three of the most common placental parenchymal lesions. We used attention maps to gain insight into model function, which differed from intuitive explanations.

An automatic entropy method to efficiently mask histology whole-slide images

Tissue segmentation of histology whole-slide images (WSI) remains a critical task in automated digital pathology workflows for both accurate disease diagnosis and deep phenotyping for research purposes. This is especially challenging when the tissue structure of biospecimens is relatively porous and heterogeneous, such as for atherosclerotic plaques. In this study, we developed a unique approach called 'EntropyMasker' based on image entropy to tackle the fore- and background segmentation (masking) task in histology WSI. We evaluated our method on 97 high-resolution WSI of human carotid atherosclerotic plaques in the Athero-Express Biobank Study, constituting hematoxylin and eosin and 8 other staining types. Using multiple benchmarking metrics, we compared our method with four widely used segmentation methods: Otsu's method, Adaptive mean, Adaptive Gaussian and slideMask and observed that our method had the highest sensitivity and Jaccard similarity index. We envision EntropyMasker to fill an important gap in WSI preprocessing, machine learning image analysis pipelines, and enable disease phenotyping beyond the field of atherosclerosis.

Generalizable biomarker prediction from cancer pathology slides with self-supervised deep learning: A retrospective multi-centric study

Deep learning (DL) can predict microsatellite instability (MSI) from routine histopathology slides of colorectal cancer (CRC). However, it is unclear whether DL can also predict other biomarkers with high performance and whether DL predictions generalize to external patient populations. Here, we acquire CRC tissue samples from two large multi-centric studies. We systematically compare six different state-of-the-art DL architectures to predict biomarkers from pathology slides, including MSI and mutations in BRAF, KRAS, NRAS, and PIK3CA. Using a large external validation cohort to provide a realistic evaluation setting, we show that models using self-supervised, attention-based multiple-instance learning consistently outperform previous approaches while offering explainable visualizations of the indicative regions and morphologies. While the prediction of MSI and BRAF mutations reaches a clinical-grade performance, mutation prediction of PIK3CA, KRAS, and NRAS was clinically insufficient.

Explainable synthetic image generation to improve risk assessment of rare pediatric heart transplant rejection

Expert microscopic analysis of cells obtained from frequent heart biopsies is vital for early detection of pediatric heart transplant rejection to prevent heart failure. Detection of this rare condition is prone to low levels of expert agreement due to the difficulty of identifying subtle rejection signs within biopsy samples. The rarity of pediatric heart transplant rejection also means that very few gold-standard images are available for developing machine learning models. To solve this urgent clinical challenge, we developed a deep learning model to automatically quantify rejection risk within digital images of biopsied tissue using an explainable synthetic data augmentation approach. We developed this explainable AI framework to illustrate how our progressive and inspirational generative adversarial network models distinguish between normal tissue images and those containing cellular rejection signs. To quantify biopsy-level rejection risk, we first detect local rejection features using a binary image classifier trained with expert-annotated and synthetic examples. We converted these local predictions into a biopsy-wide rejection score via an interpretable histogram-based approach. Our model significantly improves upon prior works with the same dataset with an area under the receiver operating curve (AUROC) of 98.84% for the local rejection detection task and 95.56% for the biopsy-rejection prediction task. A biopsy-level sensitivity of 83.33% makes our approach suitable for early screening of biopsies to prioritize expert analysis. Our framework provides a solution to rare medical imaging challenges currently limited by small datasets.

Non-invasive detection of cardiac allograft rejection among heart transplant recipients using an electrocardiogram based deep learning model

Aims

Current non-invasive screening methods for cardiac allograft rejection have shown limited discrimination and are yet to be broadly integrated into heart transplant care. Given electrocardiogram (ECG) changes have been reported with severe cardiac allograft rejection, this study aimed to develop a deep-learning model, a form of artificial intelligence, to detect allograft rejection using the 12-lead ECG (AI-ECG).

Methods and results

Heart transplant recipients were identified across three Mayo Clinic sites between 1998 and 2021. Twelve-lead digital ECG data and endomyocardial biopsy results were extracted from medical records. Allograft rejection was defined as moderate or severe acute cellular rejection (ACR) based on International Society for Heart and Lung Transplantation guidelines. The extracted data (7590 unique ECG-biopsy pairs, belonging to 1427 patients) was partitioned into training (80%), validation (10%), and test sets (10%) such that each patient was included in only one partition. Model performance metrics were based on the test set (n = 140 patients; 758 ECG-biopsy pairs). The AI-ECG detected ACR with an area under the receiver operating curve (AUC) of 0.84 [95% confidence interval (CI): 0.78-0.90] and 95% (19/20; 95% CI: 75-100%) sensitivity. A prospective proof-of-concept screening study (n = 56; 97 ECG-biopsy pairs) showed the AI-ECG detected ACR with AUC = 0.78 (95% CI: 0.61-0.96) and 100% (2/2; 95% CI: 16-100%) sensitivity.

Conclusion

An AI-ECG model is effective for detection of moderate-to-severe ACR in heart transplant recipients. Our findings could improve transplant care by providing a rapid, non-invasive, and potentially remote screening option for cardiac allograft function.

Predicting gene mutation status via artificial intelligence technologies based on multimodal integration (MMI) to advance precision oncology

Personalized treatment strategies for cancer frequently rely on the detection of genetic alterations which are determined by molecular biology assays. Historically, these processes typically required single-gene sequencing, next-generation sequencing, or visual inspection of histopathology slides by experienced pathologists in a clinical context. In the past decade, advances in artificial intelligence (AI) technologies have demonstrated remarkable potential in assisting physicians with accurate diagnosis of oncology image-recognition tasks. Meanwhile, AI techniques make it possible to integrate multimodal data such as radiology, histology, and genomics, providing critical guidance for the stratification of patients in the context of precision therapy. Given that the mutation detection is unaffordable and time-consuming for a considerable number of patients, predicting gene mutations based on routine clinical radiological scans or whole-slide images of tissue with AI-based methods has become a hot issue in actual clinical practice. In this review, we synthesized the general framework of multimodal integration (MMI) for molecular intelligent diagnostics beyond standard techniques. Then we summarized the emerging applications of AI in the prediction of mutational and molecular profiles of common cancers (lung, brain, breast, and other tumor types) pertaining to radiology and histology imaging. Furthermore, we concluded that there truly exist multiple challenges of AI techniques in the way of its real-world application in the medical field, including data curation, feature fusion, model interpretability, and practice regulations. Despite these challenges, we still prospect the clinical implementation of AI as a highly potential decision-support tool to aid oncologists in future cancer treatment management.

Artificial intelligence guidance of advanced heart failure therapies: A systematic scoping review

Introduction

Artificial intelligence can recognize complex patterns in large datasets. It is a promising technology to advance heart failure practice, as many decisions rely on expert opinions in the absence of high-quality data-driven evidence.

Methods

We searched Embase, Web of Science, and PubMed databases for articles containing "artificial intelligence," "machine learning," or "deep learning" and any of the phrases "heart transplantation," "ventricular assist device," or "cardiogenic shock" from inception until August 2022. We only included original research addressing post heart transplantation (HTx) or mechanical circulatory support (MCS) clinical care. Review and data extraction were performed in accordance with PRISMA-Scr guidelines.

Results

Of 584 unique publications detected, 31 met the inclusion criteria. The majority focused on outcome prediction post HTx (n = 13) and post durable MCS (n = 7), as well as post HTx and MCS management (n = 7, n = 3, respectively). One study addressed temporary mechanical circulatory support. Most studies advocated for rapid integration of AI into clinical practice, acknowledging potential improvements in management guidance and reliability of outcomes prediction. There was a notable paucity of external data validation and integration of multiple data modalities.

Conclusion

Our review showed mounting innovation in AI application in management of MCS and HTx, with the largest evidence showing improved mortality outcome prediction.

Whole Slide Images in Artificial Intelligence Applications in Digital Pathology: Challenges and Pitfalls

The use of digitized data in pathology research is rapidly increasing. The whole slide image (WSI) is an indispensable part of the visual examination of slides in digital pathology and artificial intelligence applications; therefore, the acquisition of WSI with the highest quality is essential. Unlike the conventional routine of pathology, the digital conversion of tissue slides and the differences in its use pose difficulties for pathologists. We categorized these challenges into three groups: before, during, and after the WSI acquisition. The problems before WSI acquisition are usually related to the quality of the glass slide and reflect all existing problems in the analytical process in pathology laboratories. WSI acquisition problems are dependent on the device used to produce the final image file. They may be related to the parts of the device that create an optical image or the hardware and software that enable digitization. Post-WSI acquisition issues are related to the final image file itself, which is the final form of this data, or the software and hardware that will use this file. Because of the digital nature of the data, most of the difficulties are related to the capabilities of the hardware or software. Being aware of the challenges and pitfalls of using digital pathology and AI will make pathologists' integration to the new technologies easier in their daily practice or research.

The evolving role of morphology in endometrial cancer diagnostics: From histopathology and molecular testing towards integrative data analysis by deep learning

Endometrial cancer (EC) diagnostics is evolving into a system in which molecular aspects are increasingly important. The traditional histological subtype-driven classification has shifted to a molecular-based classification that stratifies EC into DNA polymerase epsilon mutated (POLEmut), mismatch repair deficient (MMRd), and p53 abnormal (p53abn), and the remaining EC as no specific molecular profile (NSMP). The molecular EC classification has been implemented in the World Health Organization 2020 classification and the 2021 European treatment guidelines, as it serves as a better basis for patient management. As a result, the integration of the molecular class with histopathological variables has become a critical focus of recent EC research. Pathologists have observed and described several morphological characteristics in association with specific genomic alterations, but these appear insufficient to accurately classify patients according to molecular subgroups. This requires pathologists to rely on molecular ancillary tests in routine workup. In this new era, it has become increasingly challenging to assign clinically relevant weights to histological and molecular features on an individual patient basis. Deep learning (DL) technology opens new options for the integrative analysis of multi-modal image and molecular datasets with clinical outcomes. Proof-of-concept studies in other cancers showed promising accuracy in predicting molecular alterations from H&E-stained tumor slide images. This suggests that some morphological characteristics that are associated with molecular alterations could be identified in EC, too, expanding the current understanding of the molecular-driven EC classification. Here in this review, we report the morphological characteristics of the molecular EC classification currently identified in the literature. Given the new challenges in EC diagnostics, this review discusses, therefore, the potential supportive role that DL could have, by providing an outlook on all relevant studies using DL on histopathology images in various cancer types with a focus on EC. Finally, we touch upon how DL might shape the management of future EC patients.

Federated Learning in Risk Prediction: A Primer and Application to COVID-19-Associated Acute Kidney Injury

BACKGROUND

Modern machine learning and deep learning algorithms require large amounts of data; however, data sharing between multiple healthcare institutions is limited by privacy and security concerns.

SUMMARY

Federated learning provides a functional alternative to the single-institution approach while avoiding the pitfalls of data sharing. In cross-silo federated learning, the data do not leave a site. The raw data are stored at the site of collection. Models are created at the site of collection and are updated locally to achieve a learning objective. We demonstrate a use case with COVID-19-associated AKI. We showed that federated models outperformed their local counterparts, even when evaluated on local data in the test dataset, and performance was like those being used for pooled data. Increases in performance at a given hospital were inversely proportional to dataset size at a given hospital, which suggests that hospitals with smaller datasets have significant room for growth with federated learning approaches.

KEY MESSAGES

This short article provides an overview of federated learning, gives a use case for COVID-19-associated acute kidney injury, and finally details the issues along with some potential solutions.

Fast and scalable search of whole-slide images via self-supervised deep learning

The adoption of digital pathology has enabled the curation of large repositories of gigapixel whole-slide images (WSIs). Computationally identifying WSIs with similar morphologic features within large repositories without requiring supervised training can have significant applications. However, the retrieval speeds of algorithms for searching similar WSIs often scale with the repository size, which limits their clinical and research potential. Here we show that self-supervised deep learning can be leveraged to search for and retrieve WSIs at speeds that are independent of repository size. The algorithm, which we named SISH (for self-supervised image search for histology) and provide as an open-source package, requires only slide-level annotations for training, encodes WSIs into meaningful discrete latent representations and leverages a tree data structure for fast searching followed by an uncertainty-based ranking algorithm for WSI retrieval. We evaluated SISH on multiple tasks (including retrieval tasks based on tissue-patch queries) and on datasets spanning over 22,000 patient cases and 56 disease subtypes. SISH can also be used to aid the diagnosis of rare cancer types for which the number of available WSIs is often insufficient to train supervised deep-learning models.