A plea for taking all available clinical information into account when assessing the predictive value of omics data

Does combining numerous data types in multi-omics data improve or hinder performance in survival prediction? Insights from a large-scale benchmark study

BACKGROUND

Predictive modeling based on multi-omics data, which incorporates several types of omics data for the same patients, has shown potential to outperform single-omics predictive modeling. Most research in this domain focuses on incorporating numerous data types, despite the complexity and cost of acquiring them. The prevailing assumption is that increasing the number of data types necessarily improves predictive performance. However, the integration of less informative or redundant data types could potentially hinder this performance. Therefore, identifying the most effective combinations of omics data types that enhance predictive performance is critical for cost-effective and accurate predictions.

METHODS

In this study, we systematically evaluated the predictive performance of all 31 possible combinations including at least one of five genomic data types (mRNA, miRNA, methylation, DNAseq, and copy number variation) using 14 cancer datasets with right-censored survival outcomes, publicly available from the TCGA database. We employed various prediction methods and up-weighted clinical data in every model to leverage their predictive importance. Harrell's C-index and the integrated Brier Score were used as performance measures. To assess the robustness of our findings, we performed a bootstrap analysis at the level of the included datasets. Statistical testing was conducted for key results, limiting the number of tests to ensure a low risk of false positives.

RESULTS

Contrary to expectations, we found that using only mRNA data or a combination of mRNA and miRNA data was sufficient for most cancer types. For some cancer types, the additional inclusion of methylation data led to improved prediction results. Far from enhancing performance, the introduction of more data types most often resulted in a decline in performance, which varied between the two performance measures.

CONCLUSIONS

Our findings challenge the prevailing notion that combining multiple omics data types in multi-omics survival prediction improves predictive performance. Thus, the widespread approach in multi-omics prediction of incorporating as many data types as possible should be reconsidered to avoid suboptimal prediction results and unnecessary expenditure.

Multivariable prognostic modelling to improve prediction of colorectal cancer recurrence: the PROSPeCT trial

OBJECTIVE

Improving prognostication to direct personalised therapy remains an unmet need. This study prospectively investigated promising CT, genetic, and immunohistochemical markers to improve the prediction of colorectal cancer recurrence.

MATERIAL AND METHODS

This multicentre trial (ISRCTN 95037515) recruited patients with primary colorectal cancer undergoing CT staging from 13 hospitals. Follow-up identified cancer recurrence and death. A baseline model for cancer recurrence at 3 years was developed from pre-specified clinicopathological variables (age, sex, tumour-node stage, tumour size, location, extramural venous invasion, and treatment). Then, CT perfusion (blood flow, blood volume, transit time and permeability), genetic (RAS, RAF, and DNA mismatch repair), and immunohistochemical markers of angiogenesis and hypoxia (CD105, vascular endothelial growth factor, glucose transporter protein, and hypoxia-inducible factor) were added to assess whether prediction improved over tumour-node staging alone as the main outcome measure.

RESULTS

Three hundred twenty-six of 448 participants formed the final cohort (226 male; mean 66 ± 10 years. 227 (70%) had ≥ T3 stage cancers; 151 (46%) were node-positive; 81 (25%) developed subsequent recurrence. The sensitivity and specificity of staging alone for recurrence were 0.56 [95% CI: 0.44, 0.67] and 0.58 [0.51, 0.64], respectively. The baseline clinicopathologic model improved specificity (0.74 [0.68, 0.79], with equivalent sensitivity of 0.57 [0.45, 0.68] for high vs medium/low-risk participants. The addition of prespecified CT perfusion, genetic, and immunohistochemical markers did not improve prediction over and above the clinicopathologic model (sensitivity, 0.58-0.68; specificity, 0.75-0.76).

CONCLUSION

A multivariable clinicopathological model outperformed staging in identifying patients at high risk of recurrence. Promising CT, genetic, and immunohistochemical markers investigated did not further improve prognostication in rigorous prospective evaluation.

CLINICAL RELEVANCE STATEMENT

A prognostic model based on clinicopathological variables including age, sex, tumour-node stage, size, location, and extramural venous invasion better identifies colorectal cancer patients at high risk of recurrence for neoadjuvant/adjuvant therapy than stage alone.

KEY POINTS

Identification of colorectal cancer patients at high risk of recurrence is an unmet need for treatment personalisation. This model for recurrence, incorporating many patient variables, had higher specificity than staging alone. Continued optimisation of risk stratification schema will help individualise treatment plans and follow-up schedules.

Statistical analysis of high-dimensional biomedical data: a gentle introduction to analytical goals, common approaches and challenges

BACKGROUND

In high-dimensional data (HDD) settings, the number of variables associated with each observation is very large. Prominent examples of HDD in biomedical research include omics data with a large number of variables such as many measurements across the genome, proteome, or metabolome, as well as electronic health records data that have large numbers of variables recorded for each patient. The statistical analysis of such data requires knowledge and experience, sometimes of complex methods adapted to the respective research questions.

METHODS

Advances in statistical methodology and machine learning methods offer new opportunities for innovative analyses of HDD, but at the same time require a deeper understanding of some fundamental statistical concepts. Topic group TG9 "High-dimensional data" of the STRATOS (STRengthening Analytical Thinking for Observational Studies) initiative provides guidance for the analysis of observational studies, addressing particular statistical challenges and opportunities for the analysis of studies involving HDD. In this overview, we discuss key aspects of HDD analysis to provide a gentle introduction for non-statisticians and for classically trained statisticians with little experience specific to HDD.

RESULTS

The paper is organized with respect to subtopics that are most relevant for the analysis of HDD, in particular initial data analysis, exploratory data analysis, multiple testing, and prediction. For each subtopic, main analytical goals in HDD settings are outlined. For each of these goals, basic explanations for some commonly used analysis methods are provided. Situations are identified where traditional statistical methods cannot, or should not, be used in the HDD setting, or where adequate analytic tools are still lacking. Many key references are provided.

CONCLUSIONS

This review aims to provide a solid statistical foundation for researchers, including statisticians and non-statisticians, who are new to research with HDD or simply want to better evaluate and understand the results of HDD analyses.

Optimal microRNA Sequencing Depth to Predict Cancer Patient Survival with Random Forest and Cox Models

(1) Background: tumor profiling enables patient survival prediction. The two essential parameters to be calibrated when designing a study based on tumor profiles from a cohort are the sequencing depth of RNA-seq technology and the number of patients. This calibration is carried out under cost constraints, and a compromise has to be found. In the context of survival data, the goal of this work is to benchmark the impact of the number of patients and of the sequencing depth of miRNA-seq and mRNA-seq on the predictive capabilities for both the Cox model with elastic net penalty and random survival forest. (2) Results: we first show that the Cox model and random survival forest provide comparable prediction capabilities, with significant differences for some cancers. Second, we demonstrate that miRNA and/or mRNA data improve prediction over clinical data alone. mRNA-seq data leads to slightly better prediction than miRNA-seq, with the notable exception of lung adenocarcinoma for which the tumor miRNA profile shows higher predictive power. Third, we demonstrate that the sequencing depth of RNA-seq data can be reduced for most of the investigated cancers without degrading the prediction abilities, allowing the creation of independent validation sets at a lower cost. Finally, we show that the number of patients in the training dataset can be reduced for the Cox model and random survival forest, allowing the use of different models on different patient subgroups.

Prognosis of lasso-like penalized Cox models with tumor profiling improves prediction over clinical data alone and benefits from bi-dimensional pre-screening

BACKGROUND

Prediction of patient survival from tumor molecular '-omics' data is a key step toward personalized medicine. Cox models performed on RNA profiling datasets are popular for clinical outcome predictions. But these models are applied in the context of "high dimension", as the number p of covariates (gene expressions) greatly exceeds the number n of patients and e of events. Thus, pre-screening together with penalization methods are widely used for dimensional reduction.

METHODS

In the present paper, (i) we benchmark the performance of the lasso penalization and three variants (i.e., ridge, elastic net, adaptive elastic net) on 16 cancers from TCGA after pre-screening, (ii) we propose a bi-dimensional pre-screening procedure based on both gene variability and p-values from single variable Cox models to predict survival, and (iii) we compare our results with iterative sure independence screening (ISIS).

RESULTS

First, we show that integration of mRNA-seq data with clinical data improves predictions over clinical data alone. Second, our bi-dimensional pre-screening procedure can only improve, in moderation, the C-index and/or the integrated Brier score, while excluding irrelevant genes for prediction. We demonstrate that the different penalization methods reached comparable prediction performances, with slight differences among datasets. Finally, we provide advice in the case of multi-omics data integration.

CONCLUSIONS

Tumor profiles convey more prognostic information than clinical variables such as stage for many cancer subtypes. Lasso and Ridge penalizations perform similarly than Elastic Net penalizations for Cox models in high-dimension. Pre-screening of the top 200 genes in term of single variable Cox model p-values is a practical way to reduce dimension, which may be particularly useful when integrating multi-omics.

Challenges in translational machine learning

Machine learning (ML) algorithms are increasingly being used to help implement clinical decision support systems. In this new field, we define as "translational machine learning", joint efforts and strong communication between data scientists and clinicians help to span the gap between ML and its adoption in the clinic. These collaborations also improve interpretability and trust in translational ML methods and ultimately aim to result in generalizable and reproducible models. To help clinicians and bioinformaticians refine their translational ML pipelines, we review the steps from model building to the use of ML in the clinic. We discuss experimental setup, computational analysis, interpretability and reproducibility, and emphasize the challenges involved. We highly advise collaboration and data sharing between consortia and institutes to build multi-centric cohorts that facilitate ML methodologies that generalize across centers. In the end, we hope that this review provides a way to streamline translational ML and helps to tackle the challenges that come with it.

Why did European Radiology reject my radiomic biomarker paper? How to correctly evaluate imaging biomarkers in a clinical setting

This review explains in simple terms, accessible to the non-statistician, general principles regarding the correct research methods to develop and then evaluate imaging biomarkers in a clinical setting, including radiomic biomarkers. The distinction between diagnostic and prognostic biomarkers is made and emphasis placed on the need to assess clinical utility within the context of a multivariable model. Such models should not be restricted to imaging biomarkers and must include relevant disease and patient characteristics likely to be clinically useful. Biomarker utility is based on whether its addition to the basic clinical model improves diagnosis or prediction. Approaches to both model development and evaluation are explained and the need for adequate amounts of representative data stressed so as to avoid underpowering and overfitting. Advice is provided regarding how to report the research correctly. KEY POINTS: • Imaging biomarker research is common but methodological errors are encountered frequently that may mean the research is not clinically useful. • The clinical utility of imaging biomarkers is best assessed by their additive effect on multivariable models based on clinical factors known to be important. • The data used to develop such models should be sufficient for the number of variables investigated and the model should be evaluated, preferably using data unrelated to development.

Time to reality check the promises of machine learning-powered precision medicine

Machine learning methods, combined with large electronic health databases, could enable a personalised approach to medicine through improved diagnosis and prediction of individual responses to therapies. If successful, this strategy would represent a revolution in clinical research and practice. However, although the vision of individually tailored medicine is alluring, there is a need to distinguish genuine potential from hype. We argue that the goal of personalised medical care faces serious challenges, many of which cannot be addressed through algorithmic complexity, and call for collaboration between traditional methodologists and experts in medical machine learning to avoid extensive research waste.

Differential MicroRNA Signatures in the Pathogenesis of Barrett's Esophagus

OBJECTIVES:

Barrett's esophagus (BE) is the precursor lesion and a major risk factor for esophageal adenocarcinoma (EAC). Although patients with BE undergo routine endoscopic surveillance, current screening methodologies have proven ineffective at identifying individuals at risk of EAC. Since microRNAs (miRNAs) have potential diagnostic and prognostic value as disease biomarkers, we sought to identify an miRNA signature of BE and EAC.

METHODS:

High-throughput sequencing of miRNAs was performed on serum and tissue biopsies from 31 patients identified either as normal, gastroesophageal reflux disease (GERD), BE, BE with low-grade dysplasia (LGD), or EAC. Logistic regression modeling of miRNA profiles with Lasso regularization was used to identify discriminating miRNA. Quantitative reverse transcription polymerase chain reaction was used to validate changes in miRNA expression using 46 formalin-fixed, paraffin-embedded specimens obtained from normal, GERD, BE, BE with LGD or HGD, and EAC subjects.

RESULTS:

A 3-class predictive model was able to classify tissue samples into normal, GERD/BE, or LGD/EAC classes with an accuracy of 80%. Sixteen miRNAs were identified that predicted 1 of the 3 classes. Our analysis confirmed previous reports indicating that miR-29c-3p and miR-193b-5p expressions are altered in BE and EAC and identified miR-4485-5p as a novel biomarker of esophageal dysplasia. Quantitative reverse transcription polymerase chain reaction validated 11 of 16 discriminating miRNAs.

DISCUSSION:

Our data provide an miRNA signature of normal, precancerous, and cancerous tissue that may stratify patients at risk of progressing to EAC. We found that serum miRNAs have a limited ability to distinguish between disease states, thus limiting their potential utility in early disease detection.

Partially linear monotone methods with automatic variable selection and monotonicity direction discovery

In many statistical regression and prediction problems, it is reasonable to assume monotone relationships between certain predictor variables and the outcome. Genomic effects on phenotypes are, for instance, often assumed to be monotone. However, in some settings, it may be reasonable to assume a partially linear model, where some of the covariates can be assumed to have a linear effect. One example is a prediction model using both high-dimensional gene expression data, and low-dimensional clinical data, or when combining continuous and categorical covariates. We study methods for fitting the partially linear monotone model, where some covariates are assumed to have a linear effect on the response, and some are assumed to have a monotone (potentially nonlinear) effect. Most existing methods in the literature for fitting such models are subject to the limitation that they have to be provided the monotonicity directions a priori for the different monotone effects. We here present methods for fitting partially linear monotone models which perform both automatic variable selection, and monotonicity direction discovery. The proposed methods perform comparably to, or better than, existing methods, in terms of estimation, prediction, and variable selection performance, in simulation experiments in both classical and high-dimensional data settings.