A comparison of estimators to evaluate the discriminatory power of time-to-event models

Genetic Prediction Modeling in Large Cohort Studies via Boosting Targeted Loss Functions

Polygenic risk scores (PRS) aim to predict a trait from genetic information, relying on common genetic variants with low to medium effect sizes. As genotype data are high-dimensional in nature, it is crucial to develop methods that can be applied to large-scale data (largen $$ n $$ and largep $$ p $$ ). Many PRS tools aggregate univariate summary statistics from genome-wide association studies into a single score. Recent advancements allow simultaneous modeling of variant effects from individual-level genotype data. In this context, we introduced snpboost, an algorithm that applies statistical boosting on individual-level genotype data to estimate PRS via multivariable regression models. By processing variants iteratively in batches, snpboost can deal with large-scale cohort data. Having solved the technical obstacles due to data dimensionality, the methodological scope can now be broadened-focusing on key objectives for the clinical application of PRS. Similar to most methods in this context, snpboost has, so far, been restricted to quantitative and binary traits. Now, we incorporate more advanced alternatives-targeted to the particular aim and outcome. Adapting the loss function extends the snpboost framework to further data situations such as time-to-event and count data. Furthermore, alternative loss functions for continuous outcomes allow us to focus not only on the mean of the conditional distribution but also on other aspects that may be more helpful in the risk stratification of individual patients and can quantify prediction uncertainty, for example, median or quantile regression. This work enhances PRS fitting across multiple model classes previously unfeasible for this data type.

Does combining numerous data types in multi-omics data improve or hinder performance in survival prediction? Insights from a large-scale benchmark study

BACKGROUND

Predictive modeling based on multi-omics data, which incorporates several types of omics data for the same patients, has shown potential to outperform single-omics predictive modeling. Most research in this domain focuses on incorporating numerous data types, despite the complexity and cost of acquiring them. The prevailing assumption is that increasing the number of data types necessarily improves predictive performance. However, the integration of less informative or redundant data types could potentially hinder this performance. Therefore, identifying the most effective combinations of omics data types that enhance predictive performance is critical for cost-effective and accurate predictions.

METHODS

In this study, we systematically evaluated the predictive performance of all 31 possible combinations including at least one of five genomic data types (mRNA, miRNA, methylation, DNAseq, and copy number variation) using 14 cancer datasets with right-censored survival outcomes, publicly available from the TCGA database. We employed various prediction methods and up-weighted clinical data in every model to leverage their predictive importance. Harrell's C-index and the integrated Brier Score were used as performance measures. To assess the robustness of our findings, we performed a bootstrap analysis at the level of the included datasets. Statistical testing was conducted for key results, limiting the number of tests to ensure a low risk of false positives.

RESULTS

Contrary to expectations, we found that using only mRNA data or a combination of mRNA and miRNA data was sufficient for most cancer types. For some cancer types, the additional inclusion of methylation data led to improved prediction results. Far from enhancing performance, the introduction of more data types most often resulted in a decline in performance, which varied between the two performance measures.

CONCLUSIONS

Our findings challenge the prevailing notion that combining multiple omics data types in multi-omics survival prediction improves predictive performance. Thus, the widespread approach in multi-omics prediction of incorporating as many data types as possible should be reconsidered to avoid suboptimal prediction results and unnecessary expenditure.

Oxidative damage in the gastrocnemius predicts long-term survival in patients with peripheral artery disease

Patients with peripheral artery disease (PAD) have increased mortality rates and a myopathy in their affected legs which is characterized by increased oxidative damage, reduced antioxidant enzymatic activity and defective mitochondrial bioenergetics. This study evaluated the hypothesis that increased levels of oxidative damage in gastrocnemius biopsies from patients with PAD predict long-term mortality rates. Oxidative damage was quantified as carbonyl adducts in myofibers of the gastrocnemius of PAD patients. The oxidative stress data were grouped into tertiles and the 5-year, all-cause mortality for each tertile was determined by Kaplan-Meier curves and compared by the Modified Peto test. A Cox-regression model was used to control the effects of clinical characteristics. Results were adjusted for age, sex, race, body mass index, ankle-brachial index, smoking, physical activity, and comorbidities. Of the 240 study participants, 99 died during a mean follow up of 37.8 months. Patients in the highest tertile of oxidative damage demonstrated the highest 5-year mortality rate. The mortality hazard ratios (HR) from the Cox analysis were statistically significant for oxidative damage (lowest vs middle tertile; HR = 6.33; p = 0.0001 and lowest vs highest; HR = 8.37; p < 0.0001). Survival analysis of a contemporaneous population of PAD patients identifies abundance of carbonyl adducts in myofibers of their gastrocnemius as a predictor of mortality rate independently of ankle-brachial index, disease stage and other clinical and myopathy-related covariates.

Accounting for time dependency in meta-analyses of concordance probability estimates

Recent years have seen the development of many novel scoring tools for disease prognosis and prediction. To become accepted for use in clinical applications, these tools have to be validated on external data. In practice, validation is often hampered by logistical issues, resulting in multiple small-sized validation studies. It is therefore necessary to synthesize the results of these studies using techniques for meta-analysis. Here we consider strategies for meta analyzing the concordance probability for time-to-event data ("C-index"), which has become a popular tool to evaluate the discriminatory power of prediction models with a right-censored outcome. We show that standard meta-analysis of the C-index may lead to biased results, as the magnitude of the concordance probability depends on the length of the time interval used for evaluation (defined e.g., by the follow-up time, which might differ considerably between studies). To address this issue, we propose a set of methods for random-effects meta-regression that incorporate time directly as covariate in the model equation. In addition to analyzing nonlinear time trends via fractional polynomial, spline, and exponential decay models, we provide recommendations on suitable transformations of the C-index before meta-regression. Our results suggest that the C-index is best meta-analyzed using fractional polynomial meta-regression with logit-transformed C-index values. Classical random-effects meta-analysis (not considering time as covariate) is demonstrated to be a suitable alternative when follow-up times are small. Our findings have implications for the reporting of C-index values in future studies, which should include information on the length of the time interval underlying the calculations.

Estimation and inference of predictive discrimination for survival outcome risk prediction models

Accurate risk prediction has been the central goal in many studies of survival outcomes. In the presence of multiple risk factors, a censored regression model can be employed to estimate a risk prediction rule. Before the prediction tool can be popularized for practical use, it is crucial to rigorously assess its prediction performance. In our motivating example, researchers are interested in developing and validating a risk prediction tool to identify future lung cancer cases by integrating demographic information, disease characteristics and smoking-related data. Considering the long latency period of cancer, it is desirable for a prediction tool to achieve discriminative performance that does not weaken over time. We propose estimation and inferential procedures to comprehensively assess both the overall predictive discrimination and the temporal pattern of an estimated prediction rule. The proposed methods readily accommodate commonly used censored regression models, including the Cox proportional hazards model and the accelerated failure time model. The estimators are consistent and asymptotically normal, and reliable variance estimators are also developed. The proposed methods offer an informative tool for inferring time-dependent predictive discrimination, as well as for comparing the discrimination performance between candidate models. Applications of the proposed methods demonstrate enduring performance of the risk prediction tool in the PLCO study and detected decaying performance in a study of liver disease.

Machine Learning Based Prediction of COVID-19 Mortality Suggests Repositioning of Anticancer Drug for Treating Severe Cases

Despite available vaccinations COVID-19 case numbers around the world are still growing, and effective medications against severe cases are lacking. In this work, we developed a machine learning model which predicts mortality for COVID-19 patients using data from the multi-center 'Lean European Open Survey on SARS-CoV-2-infected patients' (LEOSS) observational study (>100 active sites in Europe, primarily in Germany), resulting into an AUC of almost 80%. We showed that molecular mechanisms related to dementia, one of the relevant predictors in our model, intersect with those associated to COVID-19. Most notably, among these molecules was tyrosine kinase 2 (TYK2), a protein that has been patented as drug target in Alzheimer's Disease but also genetically associated with severe COVID-19 outcomes. We experimentally verified that anti-cancer drugs Sorafenib and Regorafenib showed a clear anti-cytopathic effect in Caco2 and VERO-E6 cells and can thus be regarded as potential treatments against COVID-19. Altogether, our work demonstrates that interpretation of machine learning based risk models can point towards drug targets and new treatment options, which are strongly needed for COVID-19.

Bias in Cross-Entropy-Based Training of Deep Survival Networks

Over the last years, utilizing deep learning for the analysis of survival data has become attractive to many researchers. This has led to the advent of numerous network architectures for the prediction of possibly censored time-to-event variables. Unlike networks for cross-sectional data (used e.g., in classification), deep survival networks require the specification of a suitably defined loss function that incorporates typical characteristics of survival data such as censoring and time-dependent features. Here, we provide an in-depth analysis of the cross-entropy loss function, which is a popular loss function for training deep survival networks. For each time point t, the cross-entropy loss is defined in terms of a binary outcome with levels "event at or before t" and "event after t". Using both theoretical and empirical approaches, we show that this definition may result in a high prediction error and a heavy bias in the predicted survival probabilities. To overcome this problem, we analyze an alternative loss function that is derived from the negative log-likelihood function of a discrete time-to-event model. We show that replacing the cross-entropy loss by the negative log-likelihood loss results in much better calibrated prediction rules and also in an improved discriminatory power, as measured by the concordance index.

A reduced panel of eight genes (ATM, SF3B1, NOTCH1, BIRC3, XPO1, MYD88, TNFAIP3, and TP53) as an estimator of the tumor mutational burden in chronic lymphocytic leukemia

INTRODUCTION

Mutational complexity or tumor mutational burden (TMB) influences the course of chronic lymphocytic leukemia (CLL). However, this information is not routinely used because TMB is usually obtained from whole genome or exome, or from large gene panel high-throughput sequencing.

METHODS

Here, we used the C-Harrel concordance index to determine the minimum panel of genes for which mutations predict treatment-free survival (TFS) as well as large resequencing panels.

RESULTS

An eight gene estimator was defined encompassing ATM, SF3B1, NOTCH1, BIRC3, XPO1, MYD88, TNFAIP3, and TP53. TMB estimated from either a large panel of genes or the eight gene estimator was increased in treated patients or in those with a short TFS (<2 years), unmutated IGHV gene or with an unfavorable karyotype. Being an independent prognostic parameter, any mutation in the eight gene estimator predicted a shorter TFS better than Binet stage and IGHV mutational status among patients with an apparently non-progressive disease (TFS >6 months). Strikingly, the eight gene estimator was also highly informative for patients with Binet stage A CLL or with a good prognosis karyotype.

CONCLUSION

These results suggest that the eight gene estimator, that is easily achievable by high-throughput resequencing, brings robust and valuable information that predicts evolution of untreated patients at diagnosis better than any other parameter.

Acid-base disorders in sick goats and their association with mortality: A simplified strong ion difference approach

OBJECTIVES

To investigate the acid-base status of sick goats using the simplified strong ion difference (sSID) approach, to establish the quantitative contribution of sSID variables to changes in blood pH and HCO3 - and to determine whether clinical, acid-base, and biochemical variables on admission are associated with the mortality of sick goats.

ANIMALS

One hundred forty-three sick goats.

METHODS

Retrospective study. Calculated sSID variables included SID using 6 electrolytes unmeasured strong ions (USI) and the total nonvolatile buffer ion concentration in plasma (Atot ). The relationship between measured blood pH and HCO3 - , and the sSID variables was examined using forward stepwise linear regression. Cox proportional hazard models were constructed to assess associations between potential predictor variables and mortality of goats during hospitalization.

RESULTS

Hypocapnia, hypokalemia, hyperchloremia, hyperlactatemia, and hyperproteinemia were common abnormalities identified in sick goats. Respiratory alkalosis, strong ion acidosis, and Atot acidosis were acid-base disorders frequently encountered in sick goats. In sick goats, the sSID variables explained 97% and 100% of the changes in blood pH and HCO3 - , respectively. The results indicated that changes in the respiratory rate (<16 respirations per minute), USI, and pH at admission were associated with increased hazard of hospital mortality in sick goats.

CONCLUSIONS AND CLINICAL IMPORTANCE

The sSID approach is a useful methodology to quantify acid-base disorders in goats and to determine the mechanisms of their development. Clinicians should consider calculation of USI in sick goats as part of the battery of information required to establish prognosis.

Inference about time-dependent prognostic accuracy measures in the presence of competing risks

BACKGROUND

Evaluating a candidate marker or developing a model for predicting risk of future conditions is one of the major goals in medicine. However, model development and assessment for a time-to-event outcome may be complicated in the presence of competing risks. In this manuscript, we propose a local and a global estimators of cause-specific AUC for right-censored survival times in the presence of competing risks.

METHODS

The local estimator - cause-specific weighted mean rank (cWMR) - is a local average of time-specific observed cause-specific AUCs within a neighborhood of given time t. The global estimator - cause-specific fractional polynomials (cFPL) - is based on modelling the cause-specific AUC as a function of t through fractional polynomials.

RESULTS

We investigated the performance of the proposed cWMR and cFPL estimators through simulation studies and real-life data analysis. The estimators perform well in small samples, have minimal bias and appropriate coverage.

CONCLUSIONS

The local estimator cWMR and the global estimator cFPL will provide computationally efficient options for assessing the prognostic accuracy of markers for time-to-event outcome in the presence of competing risks in many practical settings.

A practical perspective on the concordance index for the evaluation and selection of prognostic time-to-event models

Developing a prognostic model for biomedical applications typically requires mapping an individual's set of covariates to a measure of the risk that he or she may experience the event to be predicted. Many scenarios, however, especially those involving adverse pathological outcomes, are better described by explicitly accounting for the timing of these events, as well as their probability. As a result, in these cases, traditional classification or ranking metrics may be inadequate to inform model evaluation or selection. To address this limitation, it is common practice to reframe the problem in the context of survival analysis, and resort, instead, to the concordance index (C-index), which summarises how well a predicted risk score describes an observed sequence of events. A practically meaningful interpretation of the C-index, however, may present several difficulties and pitfalls. Specifically, we identify two main issues: i) the C-index remains implicitly, and subtly, dependent on time, and ii) its relationship with the number of subjects whose risk was incorrectly predicted is not straightforward. Failure to consider these two aspects may introduce undesirable and unwanted biases in the evaluation process, and even result in the selection of a suboptimal model. Hence, here, we discuss ways to obtain a meaningful interpretation in spite of these difficulties. Aiming to assist experimenters regardless of their familiarity with the C-index, we start from an introductory-level presentation of its most popular estimator, highlighting the latter's temporal dependency, and suggesting how it might be correctly used to inform model selection. We also address the nonlinearity of the C-index with respect to the number of correct risk predictions, elaborating a simplified framework that may enable an easier interpretation and quantification of C-index improvements or deteriorations.

Hypothesis-free deep survival learning applied to the tumour microenvironment in gastric cancer

The biological complexity reflected in histology images requires advanced approaches for unbiased prognostication. Machine learning and particularly deep learning methods are increasingly applied in the field of digital pathology. In this study, we propose new ways to predict risk for cancer‐specific death from digital images of immunohistochemically (IHC) stained tissue microarrays (TMAs). Specifically, we evaluated a cohort of 248 gastric cancer patients using convolutional neural networks (CNNs) in an end‐to‐end weakly supervised scheme independent of subjective pathologist input. To account for the time‐to‐event characteristic of the outcome data, we developed new survival models to guide the network training. In addition to the standard H&E staining, we investigated the prognostic value of a panel of immune cell markers (CD8, CD20, CD68) and a proliferation marker (Ki67). Our CNN‐derived risk scores provided additional prognostic value when compared to the gold standard prognostic tool TNM stage. The CNN‐derived risk scores were also shown to be superior when systematically compared to cell density measurements or a CNN score derived from binary 5‐year survival classification, which ignores time‐to‐event. To better understand the underlying biological mechanisms, we qualitatively investigated risk heat maps for each marker which visualised the network output. We identified patterns of biological interest that were related to low risk of cancer‐specific death such as the presence of B‐cell predominated clusters and Ki67 positive sub‐regions and showed that the corresponding risk scores had prognostic value in multivariate Cox regression analyses (Ki67&CD20 risks: hazard ratio (HR) = 1.47, 95% confidence interval (CI) = 1.15–1.89, p = 0.002; CD20&CD68 risks: HR = 1.33, 95% CI = 1.07–1.67, p = 0.009). Our study demonstrates the potential additional value that deep learning in combination with a panel of IHC markers can bring to the field of precision oncology.

Clinical-learning versus machine-learning for transdiagnostic prediction of psychosis onset in individuals at-risk

Predicting the onset of psychosis in individuals at-risk is based on robust prognostic model building methods including a priori clinical knowledge (also termed clinical-learning) to preselect predictors or machine-learning methods to select predictors automatically. To date, there is no empirical research comparing the prognostic accuracy of these two methods for the prediction of psychosis onset. In a first experiment, no improved performance was observed when machine-learning methods (LASSO and RIDGE) were applied-using the same predictors-to an individualised, transdiagnostic, clinically based, risk calculator previously developed on the basis of clinical-learning (predictors: age, gender, age by gender, ethnicity, ICD-10 diagnostic spectrum), and externally validated twice. In a second experiment, two refined versions of the published model which expanded the granularity of the ICD-10 diagnosis were introduced: ICD-10 diagnostic categories and ICD-10 diagnostic subdivisions. Although these refined versions showed an increase in apparent performance, their external performance was similar to the original model. In a third experiment, the three refined models were analysed under machine-learning and clinical-learning with a variable event per variable ratio (EPV). The best performing model under low EPVs was obtained through machine-learning approaches. The development of prognostic models on the basis of a priori clinical knowledge, large samples and adequate events per variable is a robust clinical prediction method to forecast psychosis onset in patients at-risk, and is comparable to machine-learning methods, which are more difficult to interpret and implement. Machine-learning methods should be preferred for high dimensional data when no a priori knowledge is available.

On the validity of time-dependent AUC estimation in the presence of cure fraction

During the last decades, several approaches have been proposed to estimate the time-dependent area under the receiver operating characteristic curve (AUC) of risk tools derived from survival data. The validity of these estimators relies on some regularity assumptions among which a survival function being proper. In practice, this assumption is not always satisfied because a fraction of the population may not be susceptible to experience the event of interest even for long follow-up. Studying the sensitivity of the proposed estimators to the violation of this assumption is of substantial interest. In this paper, we investigate the performance of a nonparametric simple estimator, developed for classical survival data, in the case when the population exhibits a cure fraction. Motivated from the current practice of deriving risk tools in oncology and cardiovascular disease prevention, we also assess the loss, in terms of predictive performance, when deriving risk tools from survival models that do not acknowledge the presence of cure. The simulation results show that the investigated method is valid even under the presence of cure. They also show that risk tools derived from survival models that ignore the presence of cure have smaller AUC compared to those derived from survival models that acknowledge the presence of cure. This was also attested with a real data analysis from a breast cancer study.

Explained variation in shared frailty models

Explained variation measures the relative gain in predictive accuracy when prediction based on prognostic factors replaces unconditional prediction. The factors may be measured on different scales or may be of different types (dichotomous, qualitative, or continuous). Thus, explained variation permits to establish a ranking of the importance of factors, even if predictive accuracy is too low to be helpful in clinical practice. In this contribution, the explained variation measure by Schemper and Henderson (2000) is extended to accommodate random factors, such as center effects in multicenter studies. This permits a direct comparison of the importance of centers and of other prognostic factors. We develop this extension for a shared frailty Cox model and provide an SAS macro and an R function to facilitate its application. Interesting empirical properties of the variation explained by a random factor are explored by a Monte Carlo study. Advantages of the approach are exemplified by an Austrian multicenter study of colon cancer.

Survival Forests with R-Squared Splitting Rules

In modeling censored data, survival forest models are a competitive nonparametric alternative to traditional parametric or semiparametric models when the function forms are possibly misspecified or the underlying assumptions are violated. In this work, we propose a survival forest approach with trees constructed using a novel pseudo R2 splitting rules. By studying the well-known benchmark data sets, we find that the proposed model generally outperforms popular survival models such as random survival forest with different splitting rules, Cox proportional hazard model, and generalized boosted model in terms of C-index metric.

Extending Statistical Boosting

Background: Boosting algorithms to simultaneously estimate and select predictor effects in statistical models have gained substantial interest during the last decade.

Objectives: This review highlights recent methodological developments regarding boosting algorithms for statistical modelling especially focusing on topics relevant for biomedical research.

Methods: We suggest a unified framework for gradient boosting and likelihood-based boosting (statistical boosting) which have been addressed separately in the literature up to now.

Results: The methodological developments on statistical boosting during the last ten years can be grouped into three different lines of research: i) efforts to ensure variable selection leading to sparser models, ii) developments regarding different types of predictor effects and how to choose them, iii) approaches to extend the statistical boosting framework to new regression settings.

Conclusions: Statistical boosting algorithms have been adapted to carry out unbiased variable selection and automated model choice during the fitting process and can nowadays be applied in almost any regression setting in combination with a large amount of different types of predictor effects.

Review and evaluation of performance measures for survival prediction models in external validation settings

BACKGROUND

When developing a prediction model for survival data it is essential to validate its performance in external validation settings using appropriate performance measures. Although a number of such measures have been proposed, there is only limited guidance regarding their use in the context of model validation. This paper reviewed and evaluated a wide range of performance measures to provide some guidelines for their use in practice.

METHODS

An extensive simulation study based on two clinical datasets was conducted to investigate the performance of the measures in external validation settings. Measures were selected from categories that assess the overall performance, discrimination and calibration of a survival prediction model. Some of these have been modified to allow their use with validation data, and a case study is provided to describe how these measures can be estimated in practice. The measures were evaluated with respect to their robustness to censoring and ease of interpretation. All measures are implemented, or are straightforward to implement, in statistical software.

RESULTS

Most of the performance measures were reasonably robust to moderate levels of censoring. One exception was Harrell's concordance measure which tended to increase as censoring increased.

CONCLUSIONS

We recommend that Uno's concordance measure is used to quantify concordance when there are moderate levels of censoring. Alternatively, Gönen and Heller's measure could be considered, especially if censoring is very high, but we suggest that the prediction model is re-calibrated first. We also recommend that Royston's D is routinely reported to assess discrimination since it has an appealing interpretation. The calibration slope is useful for both internal and external validation settings and recommended to report routinely. Our recommendation would be to use any of the predictive accuracy measures and provide the corresponding predictive accuracy curves. In addition, we recommend to investigate the characteristics of the validation data such as the level of censoring and the distribution of the prognostic index derived in the validation setting before choosing the performance measures.

A Selective Review on Random Survival Forests for High Dimensional Data

Over the past decades, there has been considerable interest in applying statistical machine learning methods in survival analysis. Ensemble based approaches, especially random survival forests, have been developed in a variety of contexts due to their high precision and non-parametric nature. This article aims to provide a timely review on recent developments and applications of random survival forests for time-to-event data with high dimensional covariates. This selective review begins with an introduction to the random survival forest framework, followed by a survey of recent developments on splitting criteria, variable selection, and other advanced topics of random survival forests for time-to-event data in high dimensional settings. We also discuss potential research directions for future research.

Boomerang: A method for recursive reclassification

While there are many validated prognostic classifiers used in practice, often their accuracy is modest and heterogeneity in clinical outcomes exists in one or more risk subgroups. Newly available markers, such as genomic mutations, may be used to improve the accuracy of an existing classifier by reclassifying patients from a heterogenous group into a higher or lower risk category. The statistical tools typically applied to develop the initial classifiers are not easily adapted toward this reclassification goal. In this article, we develop a new method designed to refine an existing prognostic classifier by incorporating new markers. The two-stage algorithm called Boomerang first searches for modifications of the existing classifier that increase the overall predictive accuracy and then merges to a prespecified number of risk groups. Resampling techniques are proposed to assess the improvement in predictive accuracy when an independent validation data set is not available. The performance of the algorithm is assessed under various simulation scenarios where the marker frequency, degree of censoring, and total sample size are varied. The results suggest that the method selects few false positive markers and is able to improve the predictive accuracy of the classifier in many settings. Lastly, the method is illustrated on an acute myeloid leukemia data set where a new refined classifier incorporates four new mutations into the existing three category classifier and is validated on an independent data set.

Boosting the discriminatory power of sparse survival models via optimization of the concordance index and stability selection

Background

When constructing new biomarker or gene signature scores for time-to-event outcomes, the underlying aims are to develop a discrimination model that helps to predict whether patients have a poor or good prognosis and to identify the most influential variables for this task. In practice, this is often done fitting Cox models. Those are, however, not necessarily optimal with respect to the resulting discriminatory power and are based on restrictive assumptions. We present a combined approach to automatically select and fit sparse discrimination models for potentially high-dimensional survival data based on boosting a smooth version of the concordance index (C-index). Due to this objective function, the resulting prediction models are optimal with respect to their ability to discriminate between patients with longer and shorter survival times. The gradient boosting algorithm is combined with the stability selection approach to enhance and control its variable selection properties.

Results

The resulting algorithm fits prediction models based on the rankings of the survival times and automatically selects only the most stable predictors. The performance of the approach, which works best for small numbers of informative predictors, is demonstrated in a large scale simulation study: C-index boosting in combination with stability selection is able to identify a small subset of informative predictors from a much larger set of non-informative ones while controlling the per-family error rate. In an application to discover biomarkers for breast cancer patients based on gene expression data, stability selection yielded sparser models and the resulting discriminatory power was higher than with lasso penalized Cox regression models.

Conclusion

The combination of stability selection and C-index boosting can be used to select small numbers of informative biomarkers and to derive new prediction rules that are optimal with respect to their discriminatory power. Stability selection controls the per-family error rate which makes the new approach also appealing from an inferential point of view, as it provides an alternative to classical hypothesis tests for single predictor effects. Due to the shrinkage and variable selection properties of statistical boosting algorithms, the latter tests are typically unfeasible for prediction models fitted by boosting.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-016-1149-8) contains supplementary material, which is available to authorized users.

Statistical controversies in clinical research: the importance of importance

We define the notion of 'importance' of prognostic factors in studies of survival and suggest quantifying it by the Schemper-Henderson measure of explained variation. Conceptual differences to the standard approach for the statistical analysis of oncologic studies of survival are discussed and exemplified by means of a study of ovarian cancer. Explained variation permits to establish a ranking of the importance of factors, also if measured on different scales, or of different types (dichotomous, qualitative or continuous), and permits to compare groups of related factors. In practice, the importance of prognostic factors often is disappointingly low. From this, it follows that even strong and highly significant prognostic factors often do not translate into close determination of individual survival of patients.

Empirical extensions of the lasso penalty to reduce the false discovery rate in high-dimensional Cox regression models

Correct selection of prognostic biomarkers among multiple candidates is becoming increasingly challenging as the dimensionality of biological data becomes higher. Therefore, minimizing the false discovery rate (FDR) is of primary importance, while a low false negative rate (FNR) is a complementary measure. The lasso is a popular selection method in Cox regression, but its results depend heavily on the penalty parameter λ. Usually, λ is chosen using maximum cross-validated log-likelihood (max-cvl). However, this method has often a very high FDR. We review methods for a more conservative choice of λ. We propose an empirical extension of the cvl by adding a penalization term, which trades off between the goodness-of-fit and the parsimony of the model, leading to the selection of fewer biomarkers and, as we show, to the reduction of the FDR without large increase in FNR. We conducted a simulation study considering null and moderately sparse alternative scenarios and compared our approach with the standard lasso and 10 other competitors: Akaike information criterion (AIC), corrected AIC, Bayesian information criterion (BIC), extended BIC, Hannan and Quinn information criterion (HQIC), risk information criterion (RIC), one-standard-error rule, adaptive lasso, stability selection, and percentile lasso. Our extension achieved the best compromise across all the scenarios between a reduction of the FDR and a limited raise of the FNR, followed by the AIC, the RIC, and the adaptive lasso, which performed well in some settings. We illustrate the methods using gene expression data of 523 breast cancer patients. In conclusion, we propose to apply our extension to the lasso whenever a stringent FDR with a limited FNR is targeted. Copyright © 2016 John Wiley & Sons, Ltd.

A scoring model for predicting survival following primary tumour resection in stage IV colorectal cancer patients with unresectable metastasis

BACKGROUND

Stage IV colorectal cancer patients with unresectable metastasis who undergo elective primary tumour resection experience heterogeneous post-operative survival. We aimed to develop a scoring model for predicting post-operative survival using pre-operative variables to identify patients who are least likely to experience extended survival following the procedure.

METHODS

Survival data were collected from stage IV colorectal cancer patients who had undergone elective primary tumour resection between January 1999 and December 2007. Coefficients of significant covariates from the multivariate Cox regression model were used to compute individual survival scores to classify patients into three prognostic groups. A survival function was derived for each group via Kaplan-Meier estimation. Internal validation was performed.

RESULTS

Advanced age (hazard ratio, HR 1.43 (1.16-1.78)); poorly differentiated tumour (HR 2.72 (1.49-5.04)); metastasis to liver (HR 1.76 (1.33-2.33)), lung (HR 1.37 (1.10-1.71)) and bone (HR 2.08 ((1.16-3.71)); carcinomatosis (HR 1.68 (1.30-2.16)); hypoalbuminaemia (HR 1.30 (1.04-1.61) and elevated carcinoembryonic antigen levels (HR 1.89 (1.49-2.39)) significantly shorten post-operative survival. The scoring model separated patients into three prognostic groups with distinct median survival lengths of 4.8, 12.4 and 18.6 months (p < 0.0001). Internal validation revealed a concordance probability estimate of 0.65 and a time-dependent area under receiver operating curve of 0.75 at 6 months. Temporal split-sample validation implied good local generalizability to future patient populations (p < 0.0001).

CONCLUSION

Predicting survival following elective primary tumour resection using pre-operative variables has been demonstrated with the scoring model developed. Model-based survival prognostication can support clinical decisions on elective primary tumour resection eligibility.

A novel nonparametric measure of explained variation for survival data with an easy graphical interpretation

INTRODUCTION

For survival data the coefficient of determination cannot be used to describe how good a model fits to the data. Therefore, several measures of explained variation for survival data have been proposed in recent years.

METHODS

We analyse an existing measure of explained variation with regard to minimisation aspects and demonstrate that these are not fulfilled for the measure.

RESULTS

In analogy to the least squares method from linear regression analysis we develop a novel measure for categorical covariates which is based only on the Kaplan-Meier estimator. Hence, the novel measure is a completely nonparametric measure with an easy graphical interpretation. For the novel measure different weighting possibilities are available and a statistical test of significance can be performed. Eventually, we apply the novel measure and further measures of explained variation to a dataset comprising persons with a histopathological papillary thyroid carcinoma.

CONCLUSION

We propose a novel measure of explained variation with a comprehensible derivation as well as a graphical interpretation, which may be used in further analyses with survival data.

The extension of total gain (TG) statistic in survival models: properties and applications

BACKGROUND

The results of multivariable regression models are usually summarized in the form of parameter estimates for the covariates, goodness-of-fit statistics, and the relevant p-values. These statistics do not inform us about whether covariate information will lead to any substantial improvement in prediction. Predictive ability measures can be used for this purpose since they provide important information about the practical significance of prognostic factors. R (2)-type indices are the most familiar forms of such measures in survival models, but they all have limitations and none is widely used.

METHODS

In this paper, we extend the total gain (TG) measure, proposed for a logistic regression model, to survival models and explore its properties using simulations and real data. TG is based on the binary regression quantile plot, otherwise known as the predictiveness curve. Standardised TG ranges from 0 (no explanatory power) to 1 ('perfect' explanatory power).

RESULTS

The results of our simulations show that unlike many of the other R (2)-type predictive ability measures, TG is independent of random censoring. It increases as the effect of a covariate increases and can be applied to different types of survival models, including models with time-dependent covariate effects. We also apply TG to quantify the predictive ability of multivariable prognostic models developed in several disease areas.

CONCLUSIONS

Overall, TG performs well in our simulation studies and can be recommended as a measure to quantify the predictive ability in survival models.

Significance of lymph node metastasis in cancer dissemination of head and neck cancer

Lymph node metastasis (LNM) in many solid cancers is a well-known prognostic factor; however, it has been debated whether regional LNM simply reflects tumor aggressiveness or is a source for further tumor dissemination. Similarly, the metastatic process in head and neck cancer (HNC) has not been fully evaluated. Thus, we aimed to investigate the relative significance of LNM in metastatic cascade of HNC using functional imaging of HNC patients and molecular imaging in in vivo models. First, we analyzed ¹⁸Fluorodeoxyglucose positron emission tomography (PET) parameters of 117 patients with oral cancer. The primary tumor and nodal PET parameters were measured separately, and survival analyses were conducted on the basis of clinical and PET variables to identify significant prognostic factors. In multivariate analyses, we found that only the metastatic node PET values were significant. Next, we compared the relative frequency of lung metastasis in primary ear tumors versus lymph node (LN) tumors, and we tested the rate of lung metastasis in another animal model, in which each animal had both primary and LN tumors that were expressing different colors. As a result, LN tumors showed higher frequencies of lung metastasis compared to orthotopic primary tumors. In color-matched comparisons, the relative contribution to lung metastasis was higher in LN tumors than in primary tumors, although both primary and LN tumors caused lung metastases. In summary, tumors growing in the LN microenvironment spread to systemic sites more commonly than primary tumors in HNC, suggesting that the adequate management of LNM can reduce further systemic metastasis.

Boosting the concordance index for survival data--a unified framework to derive and evaluate biomarker combinations

The development of molecular signatures for the prediction of time-to-event outcomes is a methodologically challenging task in bioinformatics and biostatistics. Although there are numerous approaches for the derivation of marker combinations and their evaluation, the underlying methodology often suffers from the problem that different optimization criteria are mixed during the feature selection, estimation and evaluation steps. This might result in marker combinations that are suboptimal regarding the evaluation criterion of interest. To address this issue, we propose a unified framework to derive and evaluate biomarker combinations. Our approach is based on the concordance index for time-to-event data, which is a non-parametric measure to quantify the discriminatory power of a prediction rule. Specifically, we propose a gradient boosting algorithm that results in linear biomarker combinations that are optimal with respect to a smoothed version of the concordance index. We investigate the performance of our algorithm in a large-scale simulation study and in two molecular data sets for the prediction of survival in breast cancer patients. Our numerical results show that the new approach is not only methodologically sound but can also lead to a higher discriminatory power than traditional approaches for the derivation of gene signatures.

On the validity of time-dependent AUC estimators

Recent developments in molecular biology have led to the massive discovery of new marker candidates for the prediction of patient survival. To evaluate the predictive value of these markers, statistical tools for measuring the performance of survival models are needed. We consider estimators of discrimination measures, which are a popular approach to evaluate survival predictions in biomarker studies. Estimators of discrimination measures are usually based on regularity assumptions such as the proportional hazards assumption. Based on two sets of molecular data and a simulation study, we show that violations of the regularity assumptions may lead to over-optimistic estimates of prediction accuracy and may therefore result in biased conclusions regarding the clinical utility of new biomarkers. In particular, we demonstrate that biased medical decision making is possible even if statistical checks indicate that all regularity assumptions are satisfied.