Integrative Analysis of Histopathological Images and Genomic Data Predicts Clear Cell Renal Cell Carcinoma Prognosis

Network analysis of histopathological image features and genomics data improving prognosis performance in clear cell renal cell carcinoma

INTRODUCTION

Clear cell renal cell carcinoma is the most common type of kidney cancer, but the prediction of prognosis remains a challenge.

METHODS

We collected whole-slide histopathological images, corresponding clinical and genetic information from the The Cancer Imaging Archive and The Cancer Genome Atlas databases and randomly divided patients into training (n = 197) and validation (n = 84) cohorts. After feature extraction by CellProfiler, we used 2 different machine learning techniques (Least Absolute Shrinkage and Selector Operation-regularized Cox and Support Vector Machine-Recursive Feature Elimination) and weighted gene co-expression network analysis to select prognosis-related image features and genes, respectively. These features and genes were integrated into a joint model using random forest and used to create a nomogram that combines other predictive indicators.

RESULTS

A total of 4 overlapped features were identified, represented by the computed histopathological risk score in the random forest model, and showed predictive value for overall survival (test set: 1-year area under the curves (AUC) = 0.726, 3-year AUC = 0.727, and 5-year AUC = 0.764). The histopathological-genetic risk score (HGRS) integrating the genetic information computed performed better than the model that used image features only (test set: 1-year AUC = 0.682, 3-year AUC = 0.734, and 5-year AUC = 0.78). The nomogram (gender, stage, and HGRS) achieved the highest net benefit according to decision curve analysis compared to HGRS or clinical model.

CONCLUSION

This study developed a histopathological-genetic-related nomogram by combining histopathological features and clinical predictors, providing a more comprehensive prognostic assessment for clear cell renal cell carcinoma patients.

A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an indirect two-step approach consisting of (1) DeepPT, a deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response to targeted and immune therapies from the inferred expression values. We show that DeepPT successfully predicts transcriptomics in all 16 The Cancer Genome Atlas cohorts tested and generalizes well to two independent datasets. ENLIGHT-DeepPT successfully predicts true responders in five independent patient cohorts involving four different treatments spanning six cancer types, with an overall odds ratio of 2.28 and a 39.5% increased response rate among predicted responders versus the baseline rate. Notably, its prediction accuracy, obtained without any training on the treatment data, is comparable to that achieved by directly predicting the response from the images, which requires specific training on the treatment evaluation cohorts.

Deep self-reconstruction driven joint nonnegative matrix factorization model for identifying multiple genomic imaging associations in complex diseases

OBJECTIVE

Comprehensive analysis of histopathology images and transcriptomics data enables the identification of candidate biomarkers and multimodal association patterns. Most existing multimodal data association studies are derived from extensions of the joint nonnegative matrix factorization model for identifying complex data associations, which can make full use of clinical prior information. However, the raw data were usually taken as the input without considering the underlying complex multi-subspace structure, influencing the subsequent integration analysis results.

METHODS

This study proposed a deep-self reconstructed joint nonnegative matrix factorization (DSRJNMF) model to use self-expressive properties to reconstruct the raw data to characterize the similarity structure associated with clinical labels. Then, the sparsity, orthogonality, and regularization constraints constructed from prior information are added to the DSRJNMF model to determine the sparse set of biologically relevant features across modalities.

RESULTS

The algorithm has been applied to identify the imaging genetic association of triple negative breast cancer (TNBC). Multilevel experimental results demonstrate that the proposed algorithm better estimates potential associations between pathological image features and miRNA-gene and identifies consistent multimodal imaging genetic biomarkers to guide the interpretation of TNBC.

CONCLUSION

The propose method provides a novel idea of data association analysis oriented to complex diseases.

A pathologist-AI collaboration framework for enhancing diagnostic accuracies and efficiencies

In pathology, the deployment of artificial intelligence (AI) in clinical settings is constrained by limitations in data collection and in model transparency and interpretability. Here we describe a digital pathology framework, nuclei.io, that incorporates active learning and human-in-the-loop real-time feedback for the rapid creation of diverse datasets and models. We validate the effectiveness of the framework via two crossover user studies that leveraged collaboration between the AI and the pathologist, including the identification of plasma cells in endometrial biopsies and the detection of colorectal cancer metastasis in lymph nodes. In both studies, nuclei.io yielded considerable diagnostic performance improvements. Collaboration between clinicians and AI will aid digital pathology by enhancing accuracies and efficiencies.

Machine learning-based pathomics signature of histology slides as a novel prognostic indicator in primary central nervous system lymphoma

PURPOSE

To develop and validate a pathomics signature for predicting the outcomes of Primary Central Nervous System Lymphoma (PCNSL).

METHODS

In this study, 132 whole-slide images (WSIs) of 114 patients with PCNSL were enrolled. Quantitative features of hematoxylin and eosin (H&E) stained slides were extracted using CellProfiler. A pathomics signature was established and validated. Cox regression analysis, receiver operating characteristic (ROC) curves, Calibration, decision curve analysis (DCA), and net reclassification improvement (NRI) were performed to assess the significance and performance.

RESULTS

In total, 802 features were extracted using a fully automated pipeline. Six machine-learning classifiers demonstrated high accuracy in distinguishing malignant neoplasms. The pathomics signature remained a significant factor of overall survival (OS) and progression-free survival (PFS) in the training cohort (OS: HR 7.423, p < 0.001; PFS: HR 2.143, p = 0.022) and independent validation cohort (OS: HR 4.204, p = 0.017; PFS: HR 3.243, p = 0.005). A significantly lower response rate to initial treatment was found in high Path-score group (19/35, 54.29%) as compared to patients in the low Path-score group (16/70, 22.86%; p < 0.001). The DCA and NRI analyses confirmed that the nomogram showed incremental performance compared with existing models. The ROC curve demonstrated a relatively sensitive and specific profile for the nomogram (1-, 2-, and 3-year AUC = 0.862, 0.932, and 0.927, respectively).

CONCLUSION

As a novel, non-invasive, and convenient approach, the newly developed pathomics signature is a powerful predictor of OS and PFS in PCNSL and might be a potential predictive indicator for therapeutic response.

Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview

The capability to gather heterogeneous data, alongside the increasing power of artificial intelligence to examine it, leading a revolution in harnessing multimodal data in the life sciences. However, most approaches are limited to unimodal data, leaving integrated approaches across modalities relatively underdeveloped in computational pathology. Pathogenomics, as an invasive method to integrate advanced molecular diagnostics from genomic data, morphological information from histopathological imaging, and codified clinical data enable the discovery of new multimodal cancer biomarkers to propel the field of precision oncology in the coming decade. In this perspective, we offer our opinions on synthesizing complementary modalities of data with emerging multimodal artificial intelligence methods in pathogenomics. It includes correlation between the pathological and genomic profile of cancer, fusion of histology, and genomics profile of cancer. We also present challenges, opportunities, and avenues for future work.

Anoikis-related mRNA-lncRNA and DNA methylation profiles for overall survival prediction in breast cancer patients

As a type of programmed cell death, anoikis resistance plays an essential role in tumor metastasis, allowing cancer cells to survive in the systemic circulation and as a key pathway for regulating critical biological processes. We conducted an exploratory analysis to improve risk stratification and optimize adjuvant treatment choices for patients with breast cancer, and identify multigene features in mRNA and lncRNA transcriptome profiles associated with anoikis. First, the variance selection method filters low information content genes in RNA sequence and then extracts the mRNA and lncRNA expression data base on annotation files. Then, the top ten key mRNAs are screened out through the PPI network. Pearson analysis has been employed to identify lncRNAs related to anoikis, and the prognosis-related lncRNAs are selected using Univariate Cox regression and machine learning. Finally, we identified a group of RNAs (including ten mRNAs and six lncRNAs) and integrated the expression data of 16 genes to construct a risk-scoring system for BRCA prognosis and drug sensitivity analysis. The risk score's validity has been evaluated with the ROC curve, Kaplan-Meier survival curve analysis and decision curve analysis (DCA). For the methylation data, we have obtained 169 anoikis-related prognostic methylation sites, integrated these sites with 16 RNA features and further used the deep learning model to evaluate and predict the survival risk of patients. The developed anoikis feature is demonstrated a consistency index (C-index) of 0.778, indicating its potential to predict the survival probability of breast cancer patients using deep learning methods.

Multi-modal fusion network with intra- and inter-modality attention for prognosis prediction in breast cancer

Accurate breast cancer prognosis prediction can help clinicians to develop appropriate treatment plans and improve life quality for patients. Recent prognostic prediction studies suggest that fusing multi-modal data, e.g., genomic data and pathological images, plays a crucial role in improving predictive performance. Despite promising results of existing approaches, there remain challenges in effective multi-modal fusion. First, albeit a powerful fusion technique, Kronecker product produces high-dimensional quadratic expansion of features that may result in high computational cost and overfitting risk, thereby limiting its performance and applicability in cancer prognosis prediction. Second, most existing methods put more attention on learning cross-modality relations between different modalities, ignoring modality-specific relations that are complementary to cross-modality relations and beneficial for cancer prognosis prediction. To address these challenges, in this study we propose a novel attention-based multi-modal network to accurately predict breast cancer prognosis, which efficiently models both modality-specific and cross-modality relations without bringing in high-dimensional features. Specifically, two intra-modality self-attentional modules and an inter-modality cross-attentional module, accompanied by latent space transformation of channel affinity matrix, are developed to successfully capture modality-specific and cross-modality relations for efficient integration of genomic data and pathological images, respectively. Moreover, we design an adaptive fusion block to take full advantage of both modality-specific and cross-modality relations. Comprehensive experiment demonstrates that our method can effectively boost prognosis prediction performance of breast cancer and compare favorably with the state-of-the-art methods.

Machine-learning methods based on the texture and non-texture features of MRI for the preoperative prediction of sentinel lymph node metastasis in breast cancer

Background

The establishment of an accurate, stable, and non-invasive prediction model of sentinel lymph node (SLN) metastasis in breast cancer is difficult nowadays. The aim of this work is to identify the optimal machine learning model based on the three-dimensional (3D) image features of magnetic resonance imaging (MRI) for the preoperative prediction of SLN metastasis in breast cancer patients.

Methods

A total of 172 patients with histologically proven breast cancer were enrolled retrospectively, including 74 SLN metastasis patients and 98 non-SLN metastasis patients. All of them underwent diffusion-weighted imaging (DWI) magnetic resonance imaging (MRI) scan. Firstly, a total of 10,320 texture and four non-texture features were extracted from the region of interests (ROIs) of image. Twenty-four feature selection methods and 11 classification methods were then evaluated by using 10-fold cross-validation to identify the optimal machine learning model in terms of the mean area under the curve (AUC), accuracy (ACC), and stability.

Results

The result showed that the model based on the combination of minimum redundancy maximum relevance (MRMR) + random forest (RF) exhibited the optimal predictive performance (AUC: 0.97±0.03; ACC: 0.89±0.05; stability: 2.94). Moreover, we independently investigated the performance of feature selection methods and classification methods, and observed that L1-support vector machine (L1-SVM) (AUC: 0.80±0.08; ACC: 0.76±0.07) and sequential forward floating selection (SFFS) (stability: 3.04) presented the best average predictive performance and stability among all feature selection methods, respectively. RF (AUC: 0.85±0.11; ACC: 0.80±0.09) and SVM (stability: 8.43) showed the best average predictive performance and stability among all classification methods, respectively.

Conclusions

The identified model based on the 3D image features of MRI provides a non-invasive way for the preoperative prediction of SLN metastasis in breast cancer patients.

Improving Pancreatic Cyst Management: Artificial Intelligence-Powered Prediction of Advanced Neoplasms through Endoscopic Ultrasound-Guided Confocal Endomicroscopy

Despite the increasing rate of detection of incidental pancreatic cystic lesions (PCLs), current standard-of-care methods for their diagnosis and risk stratification remain inadequate. Intraductal papillary mucinous neoplasms (IPMNs) are the most prevalent PCLs. The existing modalities, including endoscopic ultrasound and cyst fluid analysis, only achieve accuracy rates of 65-75% in identifying carcinoma or high-grade dysplasia in IPMNs. Furthermore, surgical resection of PCLs reveals that up to half exhibit only low-grade dysplastic changes or benign neoplasms. To reduce unnecessary and high-risk pancreatic surgeries, more precise diagnostic techniques are necessary. A promising approach involves integrating existing data, such as clinical features, cyst morphology, and data from cyst fluid analysis, with confocal endomicroscopy and radiomics to enhance the prediction of advanced neoplasms in PCLs. Artificial intelligence and machine learning modalities can play a crucial role in achieving this goal. In this review, we explore current and future techniques to leverage these advanced technologies to improve diagnostic accuracy in the context of PCLs.

Comprehensive Systematic Review of Biomarkers in Metastatic Renal Cell Carcinoma: Predictors, Prognostics, and Therapeutic Monitoring

BACKGROUND

Challenges remain in determining the most effective treatment strategies and identifying patients who would benefit from adjuvant or neoadjuvant therapy in renal cell carcinoma. The objective of this review is to provide a comprehensive overview of biomarkers in metastatic renal cell carcinoma (mRCC) and their utility in prediction of treatment response, prognosis, and therapeutic monitoring in patients receiving systemic therapy for metastatic disease.

METHODS

A systematic literature search was conducted using the PubMed database for relevant studies published between January 2017 and December 2022. The search focused on biomarkers associated with mRCC and their relationship to immune checkpoint inhibitors, targeted therapy, and VEGF inhibitors in the adjuvant, neoadjuvant, and metastatic settings.

RESULTS

The review identified various biomarkers with predictive, prognostic, and therapeutic monitoring potential in mRCC. The review also discussed the challenges associated with anti-angiogenic and immune-checkpoint monotherapy trials and highlighted the need for personalized therapy based on molecular signatures.

CONCLUSION

This comprehensive review provides valuable insights into the landscape of biomarkers in mRCC and their potential applications in prediction of treatment response, prognosis, and therapeutic monitoring. The findings underscore the importance of incorporating biomarker assessment into clinical practice to guide treatment decisions and improve patient outcomes in mRCC.

H&E image analysis pipeline for quantifying morphological features

Detecting cell types from histopathological images is essential for various digital pathology applications. However, large number of cells in whole-slide images (WSIs) necessitates automated analysis pipelines for efficient cell type detection. Herein, we present hematoxylin and eosin (H&E) Image Processing pipeline (HEIP) for automatied analysis of scanned H&E-stained slides. HEIP is a flexible and modular open-source software that performs preprocessing, instance segmentation, and nuclei feature extraction. To evaluate the performance of HEIP, we applied it to extract cell types from ovarian high-grade serous carcinoma (HGSC) patient WSIs. HEIP showed high precision in instance segmentation, particularly for neoplastic and epithelial cells. We also show that there is a significant correlation between genomic ploidy values and morphological features, such as major axis of the nucleus.

Prediction of cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin (H&E)-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an approach for predicting response to multiple targeted and immunotherapies from H&E-slides. In difference from existing approaches that aim to predict treatment response directly from the slides, ENLIGHT-DeepPT is an indirect two-step approach consisting of (1) DeepPT, a new deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response based on the DeepPT inferred expression values. DeepPT successfully predicts transcriptomics in all 16 TCGA cohorts tested and generalizes well to two independent datasets. Our key contribution is showing that ENLIGHT-DeepPT successfully predicts true responders in five independent patients' cohorts involving four different treatments spanning six cancer types with an overall odds ratio of 2.44, increasing the baseline response rate by 43.47% among predicted responders, without the need for any treatment data for training. Furthermore, its prediction accuracy on these datasets is comparable to a supervised approach predicting the response directly from the images, which needs to be trained and tested on the same cohort. ENLIGHT-DeepPT future application could provide clinicians with rapid treatment recommendations to an array of different therapies and importantly, may contribute to advancing precision oncology in developing countries.

Computational textural mapping harmonises sampling variation and reveals multidimensional histopathological fingerprints

BACKGROUND

Technical factors can bias H&E digital slides potentially compromising computational histopathology studies. Here, we hypothesised that sample quality and sampling variation can introduce even greater and undocumented technical fallacy.

METHODS

Using The Cancer Genome Atlas (TCGA) clear-cell renal cell carcinoma (ccRCC) as a model disease, we annotated ~78,000 image tiles and trained deep learning models to detect histological textures and lymphocyte infiltration at the tumour core and its surrounding margin and correlated these with clinical, immunological, genomic, and transcriptomic profiles.

RESULTS

The models reached 95% validation accuracy for classifying textures and 95% for lymphocyte infiltration enabling reliable profiling of ccRCC samples. We validated the lymphocyte-per-texture distributions in the Helsinki dataset (n = 64). Texture analysis indicated constitutive sampling bias by TCGA clinical centres and technically suboptimal samples. We demonstrate how computational texture mapping (CTM) can abrogate these issues by normalising textural variance. CTM-harmonised histopathological architecture resonated with both expected associations and novel molecular fingerprints. For instance, tumour fibrosis associated with histological grade, epithelial-to-mesenchymal transition, low mutation burden and metastasis.

CONCLUSIONS

This study highlights texture-based standardisation to resolve technical bias in computational histopathology and understand the molecular basis of tissue architecture. All code, data and models are released as a community resource.

Integrative models of histopathological images and multi-omics data predict prognosis in endometrial carcinoma

Objective

This study aimed to predict the molecular features of endometrial carcinoma (EC) and the overall survival (OS) of EC patients using histopathological imaging.

Methods

The patients from The Cancer Genome Atlas (TCGA) were separated into the training set (n = 215) and test set (n = 214) in proportion of 1:1. By analyzing quantitative histological image features and setting up random forest model verified by cross-validation, we constructed prognostic models for OS. The model performance is evaluated with the time-dependent receiver operating characteristics (AUC) over the test set.

Results

Prognostic models based on histopathological imaging features (HIF) predicted OS in the test set (5-year AUC = 0.803). The performance of combining histopathology and omics transcends that of genomics, transcriptomics, or proteomics alone. Additionally, multi-dimensional omics data, including HIF, genomics, transcriptomics, and proteomics, attained the largest AUCs of 0.866, 0.869, and 0.856 at years 1, 3, and 5, respectively, showcasing the highest discrepancy in survival (HR = 18.347, 95% CI [11.09-25.65], p < 0.001).

Conclusions

The results of this experiment indicated that the complementary features of HIF could improve the prognostic performance of EC patients. Moreover, the integration of HIF and multi-dimensional omics data might ameliorate survival prediction and risk stratification in clinical practice.

Artificial Intelligence in Renal Cell Carcinoma Histopathology: Current Applications and Future Perspectives

Renal cell carcinoma (RCC) is characterized by its diverse histopathological features, which pose possible challenges to accurate diagnosis and prognosis. A comprehensive literature review was conducted to explore recent advancements in the field of artificial intelligence (AI) in RCC pathology. The aim of this paper is to assess whether these advancements hold promise in improving the precision, efficiency, and objectivity of histopathological analysis for RCC, while also reducing costs and interobserver variability and potentially alleviating the labor and time burden experienced by pathologists. The reviewed AI-powered approaches demonstrate effective identification and classification abilities regarding several histopathological features associated with RCC, facilitating accurate diagnosis, grading, and prognosis prediction and enabling precise and reliable assessments. Nevertheless, implementing AI in renal cell carcinoma generates challenges concerning standardization, generalizability, benchmarking performance, and integration of data into clinical workflows. Developing methodologies that enable pathologists to interpret AI decisions accurately is imperative. Moreover, establishing more robust and standardized validation workflows is crucial to instill confidence in AI-powered systems' outcomes. These efforts are vital for advancing current state-of-the-art practices and enhancing patient care in the future.

Artificial Intelligence in the Diagnosis and Treatment of Pancreatic Cystic Lesions and Adenocarcinoma

Pancreatic cancer is projected to become the second leading cause of cancer-related mortality in the United States by 2030. This is in part due to the paucity of reliable screening and diagnostic options for early detection. Amongst known pre-malignant pancreatic lesions, pancreatic intraepithelial neoplasia (PanIN) and intraductal papillary mucinous neoplasms (IPMNs) are the most prevalent. The current standard of care for the diagnosis and classification of pancreatic cystic lesions (PCLs) involves cross-sectional imaging studies and endoscopic ultrasound (EUS) and, when indicated, EUS-guided fine needle aspiration and cyst fluid analysis. However, this is suboptimal for the identification and risk stratification of PCLs, with accuracy of only 65-75% for detecting mucinous PCLs. Artificial intelligence (AI) is a promising tool that has been applied to improve accuracy in screening for solid tumors, including breast, lung, cervical, and colon cancer. More recently, it has shown promise in diagnosing pancreatic cancer by identifying high-risk populations, risk-stratifying premalignant lesions, and predicting the progression of IPMNs to adenocarcinoma. This review summarizes the available literature on artificial intelligence in the screening and prognostication of precancerous lesions in the pancreas, and streamlining the diagnosis of pancreatic cancer.

A joint multi-modal learning method for early-stage knee osteoarthritis disease classification

Osteoarthritis (OA) is a progressive and chronic disease. Identifying the early stages of OA disease is important for the treatment and care of patients. However, most state-of-the-art methods only use single-modal data to predict disease status, so that these methods usually ignore complementary information in multi-modal data. In this study, we develop an integrated multi-modal learning method (MMLM) that uses an interpretable strategy to select and fuse clinical, imaging, and demographic features to classify the grade of early-stage knee OA disease. MMLM applies XGboost and ResNet50 to extract two heterogeneous features from the clinical data and imaging data, respectively. And then we integrate these extracted features with demographic data. To avoid the negative effects of redundant features in a direct integration of multiple features, we propose a L1-norm-based optimization method (MMLM) to regularize the inter-correlations among the multiple features. MMLM was assessed using the Osteoarthritis Initiative (OAI) data set with machine learning classifiers. Extensive experiments demonstrate that MMLM improves the performance of the classifiers. Furthermore, a visual analysis of the important features in the multimodal data verified the relations among the modalities when classifying the grade of knee OA disease.

Multimodal data fusion for cancer biomarker discovery with deep learning

Technological advances now make it possible to study a patient from multiple angles with high-dimensional, high-throughput multi-scale biomedical data. In oncology, massive amounts of data are being generated ranging from molecular, histopathology, radiology to clinical records. The introduction of deep learning has significantly advanced the analysis of biomedical data. However, most approaches focus on single data modalities leading to slow progress in methods to integrate complementary data types. Development of effective multimodal fusion approaches is becoming increasingly important as a single modality might not be consistent and sufficient to capture the heterogeneity of complex diseases to tailor medical care and improve personalised medicine. Many initiatives now focus on integrating these disparate modalities to unravel the biological processes involved in multifactorial diseases such as cancer. However, many obstacles remain, including lack of usable data as well as methods for clinical validation and interpretation. Here, we cover these current challenges and reflect on opportunities through deep learning to tackle data sparsity and scarcity, multimodal interpretability, and standardisation of datasets.

Multimodal deep learning to predict prognosis in adult and pediatric brain tumors

BACKGROUND

The introduction of deep learning in both imaging and genomics has significantly advanced the analysis of biomedical data. For complex diseases such as cancer, different data modalities may reveal different disease characteristics, and the integration of imaging with genomic data has the potential to unravel additional information than when using these data sources in isolation. Here, we propose a DL framework that combines these two modalities with the aim to predict brain tumor prognosis.

METHODS

Using two separate glioma cohorts of 783 adults and 305 pediatric patients we developed a DL framework that can fuse histopathology images with gene expression profiles. Three strategies for data fusion were implemented and compared: early, late, and joint fusion. Additional validation of the adult glioma models was done on an independent cohort of 97 adult patients.

RESULTS

Here we show that the developed multimodal data models achieve better prediction results compared to the single data models, but also lead to the identification of more relevant biological pathways. When testing our adult models on a third brain tumor dataset, we show our multimodal framework is able to generalize and performs better on new data from different cohorts. Leveraging the concept of transfer learning, we demonstrate how our pediatric multimodal models can be used to predict prognosis for two more rare (less available samples) pediatric brain tumors.

CONCLUSIONS

Our study illustrates that a multimodal data fusion approach can be successfully implemented and customized to model clinical outcome of adult and pediatric brain tumors.

Predicting peritoneal recurrence in gastric cancer with serosal invasion using a pathomics nomogram

Peritoneal recurrence is the most frequent and lethal recurrence pattern in gastric cancer (GC) with serosal invasion after radical surgery. However, current evaluation methods are not adequate for predicting peritoneal recurrence in GC with serosal invasion. Emerging evidence shows that pathomics analyses could be advantageous for risk stratification and outcome prediction. Herein, we propose a pathomics signature composed of multiple pathomics features extracted from digital hematoxylin and eosin-stained images. We found that the pathomics signature was significantly associated with peritoneal recurrence. A competing-risk pathomics nomogram including carbohydrate antigen 19-9 level, depth of invasion, lymph node metastasis, and pathomics signature was developed for predicting peritoneal recurrence. The pathomics nomogram had favorable discrimination and calibration. Thus, the pathomics signature is a predictive indicator of peritoneal recurrence, and the pathomics nomogram may provide a helpful reference for predicting an individual's risk in peritoneal recurrence of GC with serosal invasion.

Artificial intelligence reveals features associated with breast cancer neoadjuvant chemotherapy responses from multi-stain histopathologic images

Advances in computational algorithms and tools have made the prediction of cancer patient outcomes using computational pathology feasible. However, predicting clinical outcomes from pre-treatment histopathologic images remains a challenging task, limited by the poor understanding of tumor immune micro-environments. In this study, an automatic, accurate, comprehensive, interpretable, and reproducible whole slide image (WSI) feature extraction pipeline known as, IMage-based Pathological REgistration and Segmentation Statistics (IMPRESS), is described. We used both H&E and multiplex IHC (PD-L1, CD8+, and CD163+) images, investigated whether artificial intelligence (AI)-based algorithms using automatic feature extraction methods can predict neoadjuvant chemotherapy (NAC) outcomes in HER2-positive (HER2+) and triple-negative breast cancer (TNBC) patients. Features are derived from tumor immune micro-environment and clinical data and used to train machine learning models to accurately predict the response to NAC in breast cancer patients (HER2+ AUC = 0.8975; TNBC AUC = 0.7674). The results demonstrate that this method outperforms the results trained from features that were manually generated by pathologists. The developed image features and algorithms were further externally validated by independent cohorts, yielding encouraging results, especially for the HER2+ subtype.

Predicting melanoma survival and metastasis with interpretable histopathological features and machine learning models

Introduction

Melanoma is the fifth most common cancer in US, and the incidence is increasing 1.4% annually. The overall survival rate for early-stage disease is 99.4%. However, melanoma can recur years later (in the same region of the body or as distant metastasis), and results in a dramatically lower survival rate. Currently there is no reliable method to predict tumor recurrence and metastasis on early primary tumor histological images.

Methods

To identify rapid, accurate, and cost-effective predictors of metastasis and survival, in this work, we applied various interpretable machine learning approaches to analyze melanoma histopathological H&E images. The result is a set of image features that can help clinicians identify high-risk-of-metastasis patients for increased clinical follow-up and precision treatment. We use simple models (i.e., logarithmic classification and KNN) and "human-interpretable" measures of cell morphology and tissue architecture (e.g., cell size, staining intensity, and cell density) to predict the melanoma survival on public and local Stage I-III cohorts as well as the metastasis risk on a local cohort.

Results

We use penalized survival regression to limit features available to downstream classifiers and investigate the utility of convolutional neural networks in isolating tumor regions to focus morphology extraction on only the tumor region. This approach allows us to predict survival and metastasis with a maximum F1 score of 0.72 and 0.73, respectively, and to visualize several high-risk cell morphologies.

Discussion

This lays the foundation for future work, which will focus on using our interpretable pipeline to predict metastasis in Stage I & II melanoma.

Imaging genetic association analysis of triple-negative breast cancer based on the integration of prior sample information

Triple-negative breast cancer (TNBC) is one of the more aggressive subtypes of breast cancer. The prognosis of TNBC patients remains low. Therefore, there is still a need to continue identifying novel biomarkers to improve the prognosis and treatment of TNBC patients. Research in recent years has shown that the effective use and integration of information in genomic data and image data will contribute to the prediction and prognosis of diseases. Considering that imaging genetics can deeply study the influence of microscopic genetic variation on disease phenotype, this paper proposes a sample prior information-induced multidimensional combined non-negative matrix factorization (SPID-MDJNMF) algorithm to integrate the Whole-slide image (WSI), mRNAs expression data, and miRNAs expression data. The algorithm effectively fuses high-dimensional data of three modalities through various constraints. In addition, this paper constructs an undirected graph between samples, uses an adjacency matrix to constrain the similarity, and embeds the clinical stage information of patients in the algorithm so that the algorithm can identify the co-expression patterns of samples with different labels. We performed univariate and multivariate Cox regression analysis on the mRNAs and miRNAs in the screened co-expression modules to construct a TNBC-related prognostic model. Finally, we constructed prognostic models for 2-mRNAs (IL12RB2 and CNIH2) and 2-miRNAs (miR-203a-3p and miR-148b-3p), respectively. The prognostic model can predict the survival time of TNBC patients with high accuracy. In conclusion, our proposed SPID-MDJNMF algorithm can efficiently integrate image and genomic data. Furthermore, we evaluated the prognostic value of mRNAs and miRNAs screened by the SPID-MDJNMF algorithm in TNBC, which may provide promising targets for the prognosis of TNBC patients.

A Review of Fusion Methods for Omics and Imaging Data

The development of omics data and biomedical images has greatly advanced the progress of precision medicine in diagnosis, treatment, and prognosis. The fusion of omics and imaging data, i.e., omics-imaging fusion, offers a new strategy for understanding complex diseases. However, due to a variety of issues such as the limited number of samples, high dimensionality of features, and heterogeneity of different data types, efficiently learning complementary or associated discriminative fusion information from omics and imaging data remains a challenge. Recently, numerous machine learning methods have been proposed to alleviate these problems. In this review, from the perspective of fusion levels and fusion methods, we first provide an overview of preprocessing and feature extraction methods for omics and imaging data, and comprehensively analyze and summarize the basic forms and variations of commonly used and newly emerging fusion methods, along with their advantages, disadvantages and the applicable scope. We then describe public datasets and compare experimental results of various fusion methods on the ADNI and TCGA datasets. Finally, we discuss future prospects and highlight remaining challenges in the field.

CAMR: cross-aligned multimodal representation learning for cancer survival prediction

MOTIVATION

Accurately predicting cancer survival is crucial for helping clinicians to plan appropriate treatments, which largely improves the life quality of cancer patients and spares the related medical costs. Recent advances in survival prediction methods suggest that integrating complementary information from different modalities, e.g. histopathological images and genomic data, plays a key role in enhancing predictive performance. Despite promising results obtained by existing multimodal methods, the disparate and heterogeneous characteristics of multimodal data cause the so-called modality gap problem, which brings in dramatically diverse modality representations in feature space. Consequently, detrimental modality gaps make it difficult for comprehensive integration of multimodal information via representation learning and therefore pose a great challenge to further improvements of cancer survival prediction.

RESULTS

To solve the above problems, we propose a novel method called cross-aligned multimodal representation learning (CAMR), which generates both modality-invariant and -specific representations for more accurate cancer survival prediction. Specifically, a cross-modality representation alignment learning network is introduced to reduce modality gaps by effectively learning modality-invariant representations in a common subspace, which is achieved by aligning the distributions of different modality representations through adversarial training. Besides, we adopt a cross-modality fusion module to fuse modality-invariant representations into a unified cross-modality representation for each patient. Meanwhile, CAMR learns modality-specific representations which complement modality-invariant representations and therefore provides a holistic view of the multimodal data for cancer survival prediction. Comprehensive experiment results demonstrate that CAMR can successfully narrow modality gaps and consistently yields better performance than other survival prediction methods using multimodal data.

AVAILABILITY AND IMPLEMENTATION

CAMR is freely available at https://github.com/wxq-ustc/CAMR.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Prognostic and predictive value of a pathomics signature in gastric cancer

The current tumour-node-metastasis (TNM) staging system alone cannot provide adequate information for prognosis and adjuvant chemotherapy benefits in patients with gastric cancer (GC). Pathomics, which is based on the development of digital pathology, is an emerging field that might improve clinical management. Herein, we propose a pathomics signature (PS_GC) that is derived from multiple pathomics features of haematoxylin and eosin-stained slides. We find that the PS_GC is an independent predictor of prognosis. A nomogram incorporating the PS_GC and TNM staging system shows significantly improved accuracy in predicting the prognosis compared to the TNM staging system alone. Moreover, in stage II and III GC patients with a low PS_GC (but not in those with a high PS_GC), satisfactory chemotherapy benefits are observed. Therefore, the PS_GC could serve as a prognostic predictor in patients with GC and might be a potential predictive indicator for decision-making regarding adjuvant chemotherapy.

Developmental Deconvolution for Classification of Cancer Origin

Cancer is partly a developmental disease, with malignancies named based on cell or tissue of origin. However, a systematic atlas of tumor origins is lacking. Here we map the single-cell organogenesis of 56 developmental trajectories to the transcriptomes of over 10,000 tumors across 33 cancer types. We deconvolute tumor transcriptomes into signals for individual developmental trajectories. Using these signals as inputs, we construct a developmental multilayer perceptron (D-MLP) classifier that outputs cancer origin. D-MLP (ROC-AUC: 0.974 for top prediction) outperforms benchmark classifiers. We analyze tumors from patients with cancer of unknown primary (CUP), selecting the most difficult cases in which extensive multimodal workup yielded no definitive tumor type. Interestingly, CUPs form groups distinguished by developmental trajectories, and classification reveals diagnosis for patient tumors. Our results provide an atlas of tumor developmental origins, provide a tool for diagnostic pathology, and suggest developmental classification may be a useful approach for patient tumors.

SIGNIFICANCE

Here we map the developmental trajectories of tumors. We deconvolute tumor transcriptomes into signals for mammalian developmental programs and use this information to construct a deep learning classifier that outputs tumor type. We apply the classifier to CUP and reveal the developmental origins of patient tumors. See related commentary by Wang, p. 2498. This article is highlighted in the In This Issue feature, p. 2483.

Deep learning with biopsy whole slide images for pretreatment prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer：A multicenter study

INTRODUCTION

Predicting pathological complete response (pCR) for patients receiving neoadjuvant chemotherapy (NAC) is crucial in establishing individualized treatment. Whole-slide images (WSIs) of tumor tissues reflect the histopathologic information of the tumor, which is important for therapeutic response effectiveness. In this study, we aimed to investigate whether predictive information for pCR could be detected from WSIs.

MATERIALS AND METHODS

We retrospectively collected data from four cohorts of 874 patients diagnosed with biopsy-proven breast cancer. A deep learning pathological model (DLPM) was constructed to predict pCR using biopsy WSIs in the primary cohort, and it was then validated in three external cohorts. The DLPM could generate a deep learning pathological score (DLPs) for each patient; stromal tumor-infiltrating lymphocytes (TILs) were selected for comparison with DLPs.

RESULTS

The WSI feature-based DLPM showed good predictive performance with the highest area under the curve (AUC) of 0.72 among the cohorts. Alternatively, the combination of the DLPM and clinical characteristics offered a better prediction performance (AUC >0.70) in all cohorts. We also evaluated the performance of DLPM in three different breast subtypes with the best prediction for the triple-negative breast cancer (TNBC) subtype (AUC: 0.73). Moreover, DLPM combined with clinical characteristics and stromal TILs achieved the highest AUC in the primary cohort (AUC: 0.82) and validation cohort 1 (AUC: 0.80).

CONCLUSION

Our study suggested that WSIs integrated with deep learning could potentially predict pCR to NAC in breast cancer. The predictive performance will be improved by combining clinical characteristics. DLPs from DLPM can provide more information compared to stromal TILs for pCR prediction.

Spatial Transcriptomic Analysis Reveals Associations between Genes and Cellular Topology in Breast and Prostate Cancers

BACKGROUND

Cancer is the leading cause of death worldwide with breast and prostate cancer the most common among women and men, respectively. Gene expression and image features are independently prognostic of patient survival; but until the advent of spatial transcriptomics (ST), it was not possible to determine how gene expression of cells was tied to their spatial relationships (i.e., topology).

METHODS

We identify topology-associated genes (TAGs) that correlate with 700 image topological features (ITFs) in breast and prostate cancer ST samples. Genes and image topological features are independently clustered and correlated with each other. Themes among genes correlated with ITFs are investigated by functional enrichment analysis.

RESULTS

Overall, topology-associated genes (TAG) corresponding to extracellular matrix (ECM) and Collagen Type I Trimer gene ontology terms are common to both prostate and breast cancer. In breast cancer specifically, we identify the ZAG-PIP Complex as a TAG. In prostate cancer, we identify distinct TAGs that are enriched for GI dysmotility and the IgA immunoglobulin complex. We identified TAGs in every ST slide regardless of cancer type.

CONCLUSIONS

These TAGs are enriched for ontology terms, illustrating the biological relevance to our image topology features and their potential utility in diagnostic and prognostic models.

Integrative Histology-Genomic Analysis Predicts Hepatocellular Carcinoma Prognosis Using Deep Learning

Cancer prognosis analysis is of essential interest in clinical practice. In order to explore the prognostic power of computational histopathology and genomics, this paper constructs a multi-modality prognostic model for survival prediction. We collected 346 patients diagnosed with hepatocellular carcinoma (HCC) from The Cancer Genome Atlas (TCGA), each patient has 1-3 whole slide images (WSIs) and an mRNA expression file. WSIs were processed by a multi-instance deep learning model to obtain the patient-level survival risk scores; mRNA expression data were processed by weighted gene co-expression network analysis (WGCNA), and the top hub genes of each module were extracted as risk factors. Information from two modalities was integrated by Cox proportional hazard model to predict patient outcomes. The overall survival predictions of the multi-modality model (Concordance index (C-index): 0.746, 95% confidence interval (CI): ±0.077) outperformed these based on histopathology risk score or hub genes, respectively. Furthermore, in the prediction of 1-year and 3-year survival, the area under curve of the model achieved 0.816 and 0.810. In conclusion, this paper provides an effective workflow for multi-modality prognosis of HCC, the integration of histopathology and genomic information has the potential to assist clinical prognosis management.

Pathological prognosis classification of patients with neuroblastoma using computational pathology analysis

Neuroblastoma is the most common extracranial solid tumor in early childhood. International Neuroblastoma Pathology Classification (INPC) is a commonly used classification system that provides clinicians with a reference for treatment stratification. However, given the complex and subjective assessment of the INPC, there will be inconsistencies in the analysis of the same patient by multiple pathologists. An automated, comprehensive and objective classification method is needed to identify different prognostic groups in patients with neuroblastoma. In this study, we collected 563 hematoxylin and eosin-stained histopathology whole-slide images from 107 patients with neuroblastoma who underwent surgical resection. We proposed a novel processing pipeline for nuclear segmentation, cell-level image feature extraction, and patient-level feature aggregation. Logistic regression model was built to classify patients with favorable histology (FH) and patients with unfavorable histology (UH). On the training/test dataset, patient-level of nucleus morphological/intensity features and age could correctly classify patients with a mean area under the receiver operating characteristic curve (AUC) of 0.946, a mean accuracy of 0.856, and a mean Matthews Correlation Coefficient (MCC) of 0.703,respectively. On the independent validation dataset, the classification model achieved a mean AUC of 0.938, a mean accuracy of 0.865 and a mean MCC of 0.630, showing good generalizability. Our results suggested that automatically derived image features could identify the differences in nuclear morphological and intensity between different prognostic groups, which could provide a reference to pathologists and facilitate the evaluation of the pathological prognosis in patients with neuroblastoma.

iCEMIGE: Integration of CEll-morphometrics, MIcrobiome, and GEne biomarker signatures for risk stratification in breast cancers

BACKGROUND

The development of precision medicine is essential for personalized treatment and improved clinical outcome, whereas biomarkers are critical for the success of precision therapies.

AIM

To investigate whether iCEMIGE (integration of CEll-morphometrics, MIcro biome, and GEne biomarker signatures) improves risk stratification of breast cancer (BC) patients.

METHODS

We used our recently developed machine learning technique to identify cellular morphometric biomarkers (CMBs) from the whole histological slide images in The Cancer Genome Atlas (TCGA) breast cancer (TCGA-BRCA) cohort. Multivariate Cox regression was used to assess whether cell-morphometrics prognosis score (CMPS) and our previously reported 12-gene expression prognosis score (GEPS) and 15-microbe abundance prognosis score (MAPS) were independent prognostic factors. iCEMIGE was built upon the sparse representation learning technique. The iCEMIGE scoring model performance was measured by the area under the receiver operating characteristic curve compared to CMPS, GEPS, or MAPS alone. Nomogram models were created to predict overall survival (OS) and progress-free survival (PFS) rates at 5- and 10-year in the TCGA-BRCA cohort.

RESULTS

We identified 39 CMBs that were used to create a CMPS system in BCs. CMPS, GEPS, and MAPS were found to be significantly independently associated with OS. We then established an iCEMIGE scoring system for risk stratification of BC patients. The iGEMIGE score has a significant prognostic value for OS and PFS independent of clinical factors (age, stage, and estrogen and progesterone receptor status) and PAM50-based molecular subtype. Importantly, the iCEMIGE score significantly increased the power to predict OS and PFS compared to CMPS, GEPS, or MAPS alone.

CONCLUSION

Our study demonstrates a novel and generic artificial intelligence framework for multimodal data integration toward improving prognosis risk stratification of BC patients, which can be extended to other types of cancer.

TSDLPP: A Novel Two-Stage Deep Learning Framework For Prognosis Prediction Based on Whole Slide Histopathological Images

Recently, digital pathology image-based prognosis prediction has become a hot topic in healthcare research to make early decisions on therapy and improve the treatment quality of patients. Therefore, there has been a recent surge of interest in designing deep learning method solving the problem of prognosis prediction with digital pathology images. However, whole slide histopathological images (WSIs) based prognosis prediction is still a challenge due to the large size of pathological images, the heterogeneity of tumors and the high cost of region of interests (ROIs) labeling. In this study, we design a novel two-stage deep learning framework for prognosis prediction (TSDLPP) based on WSIs. Our proposed framework consists of two-stage paradigms: 1) training tissue decomposition network (TDNet) to divide WSIs into cancerous and non-cancerous regions, 2) integrating general prognosis-related densely connected CNN (GPR-DCCNN) and morphology-specific prognosis-related densely connected CNNs (MSPR-DCCNNs) to extract different level features of pathological images. In the end, we apply TSDLPP to the prognosis prediction of breast cancer using The Cancer Genome Atlas (TCGA) datasets. Experiment results demonstrate that TSDLPP obtains superior performance of prognosis prediction compared with the existing state-of-arts methods.

Combining Molecular, Imaging, and Clinical Data Analysis for Predicting Cancer Prognosis

Simple Summary

The rise of Big Data, the widespread use of Machine Learning, and the cheapening of omics techniques have allowed for the creation of more sophisticated and accurate models in biomedical research. This article presents the state-of-the-art predictive models of cancer prognosis that use multimodal data, considering clinical, molecular (omics and non-omics), and image data. The subject of study, the data modalities used, the data processing and modelling methods applied, the validation strategies involved, the integration strategies encompassed, and the evolution of prognostic predictive models are discussed. Finally, we discuss challenges and opportunities in this field of cancer research, with great potential impact on the clinical management of patients and, by extension, on the implementation of personalised and precision medicine.

Abstract

Cancer is one of the most detrimental diseases globally. Accordingly, the prognosis prediction of cancer patients has become a field of interest. In this review, we have gathered 43 state-of-the-art scientific papers published in the last 6 years that built cancer prognosis predictive models using multimodal data. We have defined the multimodality of data as four main types: clinical, anatomopathological, molecular, and medical imaging; and we have expanded on the information that each modality provides. The 43 studies were divided into three categories based on the modelling approach taken, and their characteristics were further discussed together with current issues and future trends. Research in this area has evolved from survival analysis through statistical modelling using mainly clinical and anatomopathological data to the prediction of cancer prognosis through a multi-faceted data-driven approach by the integration of complex, multimodal, and high-dimensional data containing multi-omics and medical imaging information and by applying Machine Learning and, more recently, Deep Learning techniques. This review concludes that cancer prognosis predictive multimodal models are capable of better stratifying patients, which can improve clinical management and contribute to the implementation of personalised medicine as well as provide new and valuable knowledge on cancer biology and its progression.

Multimodal data analysis reveals that pancreatobiliary-type ampullary adenocarcinoma resembles pancreatic adenocarcinoma and differs from cholangiocarcinoma

BACKGROUND

Ampullary adenocarcinoma (AAC) arises from the ampulla of Vater where the pancreatic duct and bile duct join and empty into the duodenum. It can be classified into intestinal and pancreatobiliary types based on histopathology or immunohistochemistry. However, there are no biomarkers for further classification of pancreatobiliary-type AAC which has important implications for its treatment. We aimed to identify the tumor origin of pancreatobiliary-type AAC by systematically analyzing whole-slide images (WSIs), survival data, and genome sequencing data collected from multiple centers.

METHODS

This study involved three experiments. First, we extracted quantitative and highly interpretable features from the tumor region in WSIs and constructed a histologic classifier to differentiate between pancreatic adenocarcinoma (PAC) and cholangiocarcinoma. The histologic classifier was then applied to patients with pancreatobiliary-type AAC to infer the tumor origin. Secondly, we compared the overall survival of patients with pancreatobiliary-type AAC stratified by the adjuvant chemotherapy regimens designed for PAC or cholangiocarcinoma. Finally, we compared the mutation landscape of pancreatobiliary-type AAC with those of PAC and cholangiocarcinoma.

RESULTS

The histologic classifier accurately classified PAC and cholangiocarcinoma in both the internal and external validation sets (AUC > 0.99). All pancreatobiliary-type AACs (n = 45) were classified as PAC. The patients with pancreatobiliary-type AAC receiving regimens designed for PAC showed more favorable overall survival than those receiving regimens designed for cholangiocarcinoma in a multivariable Cox regression (hazard ratio = 7.24, 95% confidence interval: 1.28-40.78, P = 0.025). The results of mutation analysis showed that the mutation landscape of AAC was very similar to that of PAC but distinct from that of cholangiocarcinoma.

CONCLUSIONS

This multi-center study provides compelling evidence that pancreatobiliary-type AAC resembles PAC instead of cholangiocarcinoma in different aspects, which can guide the treatment selection and clinical trials planning for pancreatobiliary-type AAC.

Exploring Histological Similarities Across Cancers From a Deep Learning Perspective

Histopathology image analysis is widely accepted as a gold standard for cancer diagnosis. The Cancer Genome Atlas (TCGA) contains large repositories of histopathology whole slide images spanning several organs and subtypes. However, not much work has gone into analyzing all the organs and subtypes and their similarities. Our work attempts to bridge this gap by training deep learning models to classify cancer vs. normal patches for 11 subtypes spanning seven organs (9,792 tissue slides) to achieve high classification performance. We used these models to investigate their performances in the test set of other organs (cross-organ inference). We found that every model had a good cross-organ inference accuracy when tested on breast, colorectal, and liver cancers. Further, high accuracy is observed between models trained on the cancer subtypes originating from the same organ (kidney and lung). We also validated these performances by showing the separability of cancer and normal samples in a high-dimensional feature space. We further hypothesized that the high cross-organ inferences are due to shared tumor morphologies among organs. We validated the hypothesis by showing the overlap in the Gradient-weighted Class Activation Mapping (GradCAM) visualizations and similarities in the distributions of nuclei features present within the high-attention regions.

Workflow to Mine Frequent DNA Co-methylation Clusters in DNA Methylome Data

The advances in high-throughput nucleotide sequencing technology revolutionized biomedical research. Vast amount of genomic data rapidly accumulates in a daily basis, which in turn calls for the development of powerful bioinformatics tools and efficient workflows to analyze them. One of the approaches to address the "big data" issue is to mine highly correlated clusters/networks of biological molecules, which may provide rich yet hidden information about the underlying functional, regulatory, or structural relationships among genes, proteins, genomic loci or various types of biological molecules or events. A network mining algorithm lmQCM has recently been developed, which can be applied to mine tightly connected correlation clusters (networks) in large biological data with big sample size, and it guarantees a lower bound of the cluster density. This algorithm has been used in a variety of cancer transcriptomic data to mine gene co-expression networks (GCNs), but it can be applied to any correlational matrix. lmQCM is available through R package lmQCM as well as the online tool package TSUNAMI ( https://biolearns.medicine.iu.edu ). In this study, the purpose is to establish an analytical workflow to apply lmQCM for frequent (consensus) cluster mining in multiple DNA methylation datasets in different cancers and extract the underlying common co-methylation networks for genes.Specifically, the workflow is applied to analyze DNA methylome data across different cancer types using lmQCM. It mines frequent DNA methylation clusters based on individual clustering mining results, identifying common as well as distinctive DNA methylation patterns among different cancer types. This workflow has successfully identified frequent GCNs in 33 types of cancers, thus proven to be a powerful tool to analyze large biological data. It helps to identify common features as well as distinctions among different diseases, disease subtypes, or among different biological processes. The resulted frequent clusters may provide new insights on the pathway/function networks. In the case of disease study, the results lead to new directions for biomarker and drug target discovery. The advantages of this workflow include the highly efficient processing of large biological data generated from high-throughput experiments, quick identification of highly correlated interaction networks, substantial reduction of the data dimensionality to a manageable number of variables for downstream comparative analysis, and consequently increased statistical power for detecting differences between conditions.

Estimating the time of skeletal muscle contusion based on the spatial distribution of neutrophils: a practical approach to forensic problems

The study aimed to explore the neutrophil's spatial distributions used to estimate the histological age of contused skeletal muscle, and assessed the accuracy of various indicators, such as the proportion of neutrophils, "neutrophil mean distance," and distribution of neutrophils in areas of "contiguous contour lines." Fifty-five Sprague-Dawley rats were divided randomly into a control group and contusion groups at 1, 1.5, 2, 3, 4, and 6 h, as well as 1, 3, 5, and 15 days, post-injury (n = 5 per group). Nuclei and neutrophils were detected by hematoxylin and eosin (HE) staining and immunohistochemical (IHC) staining. At 0-24 h after injury, the distribution of neutrophils at distances of 100, 200, 300, 400, 500, and 600 µm from adjacent blood vessels was determined, and the best samples were screened to estimate wound age. To estimate wound age as accurately as possible, Fisher discriminant analysis (FDA) of the proportion of neutrophils, neutrophil mean distance, and distribution of neutrophils was performed, and 100.0% and 95.0% of the original and cross-validated cases were correctly classified, respectively. The spatial distribution of neutrophils at different distances from adjacent blood vessels showed a strong correlation with the histological age of contusion skeletal muscle, and the combination of the proportion of neutrophils, neutrophil mean distance, and distribution of neutrophils could be used to accurately estimate wound age.

Relation-Aware Shared Representation Learning for Cancer Prognosis Analysis With Auxiliary Clinical Variables and Incomplete Multi-Modality Data

The integrative analysis of complementary phenotype information contained in multi-modality data (e.g., histopathological images and genomic data) has advanced the prognostic evaluation of cancers. However, multi-modality based prognosis analysis confronts two challenges: (1) how to explore underlying relations inherent in different modalities data for learning compact and discriminative multi-modality representations; (2) how to take full consideration of incomplete multi-modality data for constructing accurate and robust prognostic model, since a host of complete multi-modality data are not always available. Additionally, many existing multi-modality based prognostic methods commonly ignore relevant clinical variables (e.g., grade and stage), which, however, may provide supplemental information to promote the performance of model. In this paper, we propose a relation-aware shared representation learning method for prognosis analysis of cancers, which makes full use of clinical information and incomplete multi-modality data. The proposed method learns multi-modal shared space tailored for prognostic model via a dual mapping. Within the shared space, it equips with relational regularizers to explore the potential relations (i.e., feature-label and feature-feature relations) among multi-modality data for inducing discriminatory representations and simultaneously obtaining extra sparsity for alleviating overfitting. Moreover, it regresses and incorporates multiple auxiliary clinical attributes with dynamic coefficients to meliorate performance. Furthermore, in training stage, a partial mapping strategy is employed to extend and train a more reliable model with incomplete multi-modality data. We have evaluated our method on three public datasets derived from The Cancer Genome Atlas (TCGA) project, and the experimental results demonstrate the superior performance of the proposed method.

OUP accepted manuscript

Motivation

Cancer survival prediction can greatly assist clinicians in planning patient treatments and improving their life quality. Recent evidence suggests the fusion of multimodal data, such as genomic data and pathological images, is crucial for understanding cancer heterogeneity and enhancing survival prediction. As a powerful multimodal fusion technique, Kronecker product has shown its superiority in predicting survival. However, this technique introduces a large number of parameters that may lead to high computational cost and a risk of overfitting, thus limiting its applicability and improvement in performance. Another limitation of existing approaches using Kronecker product is that they only mine relations for one single time to learn multimodal representation and therefore face significant challenges in deeply mining rich information from multimodal data for accurate survival prediction.

Results

To address the above limitations, we present a novel hierarchical multimodal fusion approach named HFBSurv by employing factorized bilinear model to fuse genomic and image features step by step. Specifically, with a multiple fusion strategy HFBSurv decomposes the fusion problem into different levels and each of them integrates and passes information progressively from the low level to the high level, thus leading to the more specialized fusion procedure and expressive multimodal representation. In this hierarchical framework, both modality-specific and cross-modality attentional factorized bilinear modules are designed to not only capture and quantify complex relations from multimodal data, but also dramatically reduce computational complexity. Extensive experiments demonstrate that our method performs an effective hierarchical fusion of multimodal data and achieves consistently better performance than other methods for survival prediction.

Availability and implementation

HFBSurv is freely available at https://github.com/Liruiqing-ustc/HFBSurv.

Supplementary information

Supplementary data are available at Bioinformatics online.

Integrating pathomics with radiomics and genomics for cancer prognosis: A brief review

In the last decade, the focus of computational pathology research community has shifted from replicating the pathological examination for diagnosis done by pathologists to unlocking and discovering "sub-visual" prognostic image cues from the histopathological image. While we are getting more knowledge and experience in digital pathology, the emerging goal is to integrate other-omics or modalities that will contribute for building a better prognostic assay. In this paper, we provide a brief review of representative works that focus on integrating pathomics with radiomics and genomics for cancer prognosis. It includes: correlation of pathomics and genomics; fusion of pathomics and genomics; fusion of pathomics and radiomics. We also present challenges, potential opportunities, and avenues for future work.

Histopathological Images and Multi-Omics Integration Predict Molecular Characteristics and Survival in Lung Adenocarcinoma

Histopathological images and omics profiles play important roles in prognosis of cancer patients. Here, we extracted quantitative features from histopathological images to predict molecular characteristics and prognosis, and integrated image features with mutations, transcriptomics, and proteomics data for prognosis prediction in lung adenocarcinoma (LUAD). Patients obtained from The Cancer Genome Atlas (TCGA) were divided into training set (n = 235) and test set (n = 235). We developed machine learning models in training set and estimated their predictive performance in test set. In test set, the machine learning models could predict genetic aberrations: ALK (AUC = 0.879), BRAF (AUC = 0.847), EGFR (AUC = 0.855), ROS1 (AUC = 0.848), and transcriptional subtypes: proximal-inflammatory (AUC = 0.897), proximal-proliferative (AUC = 0.861), and terminal respiratory unit (AUC = 0.894) from histopathological images. Moreover, we obtained tissue microarrays from 316 LUAD patients, including four external validation sets. The prognostic model using image features was predictive of overall survival in test and four validation sets, with 5-year AUCs from 0.717 to 0.825. High-risk and low-risk groups stratified by the model showed different survival in test set (HR = 4.94, p < 0.0001) and three validation sets (HR = 1.64–2.20, p < 0.05). The combination of image features and single omics had greater prognostic power in test set, such as histopathology + transcriptomics model (5-year AUC = 0.840; HR = 7.34, p < 0.0001). Finally, the model integrating image features with multi-omics achieved the best performance (5-year AUC = 0.908; HR = 19.98, p < 0.0001). Our results indicated that the machine learning models based on histopathological image features could predict genetic aberrations, transcriptional subtypes, and survival outcomes of LUAD patients. The integration of histopathological images and multi-omics may provide better survival prediction for LUAD.

Integrative Analysis of Histopathological Images and Genomic Data in Colon Adenocarcinoma

Background

Colon adenocarcinoma (COAD) is one of the most common malignant tumors in the world. The histopathological features are crucial for the diagnosis, prognosis, and therapy of COAD.

Methods

We downloaded 719 whole-slide histopathological images from TCIA, and 459 corresponding HTSeq-counts mRNA expression and clinical data were obtained from TCGA. Histopathological image features were extracted by CellProfiler. Prognostic image features were selected by the least absolute shrinkage and selection operator (LASSO) and support vector machine (SVM) algorithms. The co-expression gene module correlated with prognostic image features was identified by weighted gene co-expression network analysis (WGCNA). Random forest was employed to construct an integrative prognostic model and calculate the histopathological-genomic prognosis factor (HGPF).

Results

There were five prognostic image features and one co-expression gene module involved in the model construction. The time-dependent receiver operating curve showed that the prognostic model had a significant prognostic value. Patients were divided into high-risk group and low-risk group based on the HGPF. Kaplan-Meier analysis indicated that the overall survival of the low-risk group was significantly better than the high-risk group.

Conclusions

These results suggested that the histopathological image features had a certain ability to predict the survival of COAD patients. The integrative prognostic model based on the histopathological images and genomic features could further improve the prognosis prediction in COAD, which may assist the clinical decision in the future.

Building robust pathology image analyses with uncertainty quantification

BACKGROUND AND OBJECTIVE

Computerized pathology image analysis is an important tool in research and clinical settings, which enables quantitative tissue characterization and can assist a pathologist's evaluation. The aim of our study is to systematically quantify and minimize uncertainty in output of computer based pathology image analysis.

METHODS

Uncertainty quantification (UQ) and sensitivity analysis (SA) methods, such as Variance-Based Decomposition (VBD) and Morris One-At-a-Time (MOAT), are employed to track and quantify uncertainty in a real-world application with large Whole Slide Imaging datasets - 943 Breast Invasive Carcinoma (BRCA) and 381 Lung Squamous Cell Carcinoma (LUSC) patients. Because these studies are compute intensive, high-performance computing systems and efficient UQ/SA methods were combined to provide efficient execution. UQ/SA has been able to highlight parameters of the application that impact the results, as well as nuclear features that carry most of the uncertainty. Using this information, we built a method for selecting stable features that minimize application output uncertainty.

RESULTS

The results show that input parameter variations significantly impact all stages (segmentation, feature computation, and survival analysis) of the use case application. We then identified and classified features according to their robustness to parameter variation, and using the proposed features selection strategy, for instance, patient grouping stability in survival analysis has been improved from in 17% and 34% for BRCA and LUSC, respectively.

CONCLUSIONS

This strategy created more robust analyses, demonstrating that SA and UQ are important methods that may increase confidence digital pathology.

BrcaSeg: A Deep Learning Approach for Tissue Quantification and Genomic Correlations of Histopathological Images

Epithelial and stromal tissues are components of the tumor microenvironment and play a major role in tumor initiation and progression. Distinguishing stroma from epithelial tissues is critically important for spatial characterization of the tumor microenvironment. Here, we propose BrcaSeg, an image analysis pipeline based on a convolutional neural network (CNN) model to classify epithelial and stromal regions in whole-slide hematoxylin and eosin (H&E) stained histopathological images. The CNN model is trained using well-annotated breast cancer tissue microarrays and validated with images from The Cancer Genome Atlas (TCGA) Program. BrcaSeg achieves a classification accuracy of 91.02%, which outperforms other state-of-the-art methods. Using this model, we generate pixel-level epithelial/stromal tissue maps for 1000 TCGA breast cancer slide images that are paired with gene expression data. We subsequently estimate the epithelial and stromal ratios and perform correlation analysis to model the relationship between gene expression and tissue ratios. Gene Ontology (GO) enrichment analyses of genes that are highly correlated with tissue ratios suggest that the same tissue is associated with similar biological processes in different breast cancer subtypes, whereas each subtype also has its own idiosyncratic biological processes governing the development of these tissues. Taken all together, our approach can lead to new insights in exploring relationships between image-based phenotypes and their underlying genomic events and biological processes for all types of solid tumors. BrcaSeg can be accessed at https://github.com/Serian1992/ImgBio.

Integration of histopathological images and multi-dimensional omics analyses predicts molecular features and prognosis in high-grade serous ovarian cancer

OBJECTIVE

This study used histopathological image features to predict molecular features, and combined with multi-dimensional omics data to predict overall survival (OS) in high-grade serous ovarian cancer (HGSOC).

METHODS

Patients from The Cancer Genome Atlas (TCGA) were distributed into training set (n = 115) and test set (n = 114). In addition, we collected tissue microarrays of 92 patients as an external validation set. Quantitative features were extracted from histopathological images using CellProfiler, and utilized to establish prediction models by machine learning methods in training set. The prediction performance was assessed in test set and validation set.

RESULTS

The prediction models were able to identify BRCA1 mutation (AUC = 0.952), BRCA2 mutation (AUC = 0.912), microsatellite instability-high (AUC = 0.919), microsatellite stable (AUC = 0.924), and molecular subtypes: proliferative (AUC = 0.961), differentiated (AUC = 0.952), immunoreactive (AUC = 0.941), mesenchymal (AUC = 0.918) in test set. The prognostic model based on histopathological image features could predict OS in test set (5-year AUC = 0.825) and validation set (5-year AUC = 0.703). We next explored the integrative prognostic models of image features, genomics, transcriptomics and proteomics. In test set, the models combining two omics had higher prediction accuracy, such as image features and genomics (5-year AUC = 0.834). The multi-omics model including all features showed the best prediction performance (5-year AUC = 0.911). According to risk score of multi-omics model, the high-risk and low-risk groups had significant survival differences (HR = 18.23, p < 0.001).

CONCLUSIONS

These results indicated the potential ability of histopathological image features to predict above molecular features and survival risk of HGSOC patients. The integration of image features and multi-omics data may improve prognosis prediction in HGSOC patients.