Quantifying the predictive accuracy of time-to-event models in the presence of competing risks

Explained variation and degrees of necessity and of sufficiency for competing risks survival data

In this contribution, the Schemper-Henderson measure of explained variation for survival outcomes is extended to accommodate competing events (CEs) in addition to events of interest. The extension is achieved by moving from the unconditional and conditional survival functions of the original measure to unconditional and conditional cumulative incidence functions, the latter obtained, for example, from Fine and Gray models. In the absence of CEs, the original measure is obtained as a special case. We define explained variation on the population level and provide two different types of estimates. Recently, the authors have achieved a multiplicative decomposition of explained variation into degrees of necessity and degrees of sufficiency. These measures are also extended to the case of competing risks survival data. A SAS macro and an R function are provided to facilitate application. Interesting empirical properties of the measures are explored on the population level and by an extensive simulation study. Advantages of the approach are exemplified by an Austrian study of breast cancer with a high proportion of CEs.

A review on statistical and machine learning competing risks methods

When modeling competing risks (CR) survival data, several techniques have been proposed in both the statistical and machine learning literature. State-of-the-art methods have extended classical approaches with more flexible assumptions that can improve predictive performance, allow high-dimensional data and missing values, among others. Despite this, modern approaches have not been widely employed in applied settings. This article aims to aid the uptake of such methods by providing a condensed compendium of CR survival methods with a unified notation and interpretation across approaches. We highlight available software and, when possible, demonstrate their usage via reproducible R vignettes. Moreover, we discuss two major concerns that can affect benchmark studies in this context: the choice of performance metrics and reproducibility.

Dynamic prediction of survival using multivariate functional principal component analysis: A strict landmarking approach

Dynamically predicting patient survival probabilities using longitudinal measurements has become of great importance with routine data collection becoming more common. Many existing models utilize a multi-step landmarking approach for this problem, mostly due to its ease of use and versatility but unfortunately most fail to do so appropriately. In this article we make use of multivariate functional principal component analysis to summarize the available longitudinal information, and employ a Cox proportional hazards model for prediction. Additionally, we consider a centred functional principal component analysis procedure in an attempt to remove the natural variation incurred by the difference in age of the considered subjects. We formalize the difference between a 'relaxed' landmarking approach where only validation data is landmarked and a 'strict' landmarking approach where both the training and validation data are landmarked. We show that a relaxed landmarking approach fails to effectively use the information contained in the longitudinal outcomes, thereby producing substantially worse prediction accuracy than a strict landmarking approach.

Predictive abilities comparison from multiple dynamic prediction models

With the development of personalized medicine, the study of individual prognosis appears to be a major contemporary scientific issue. Dynamic models are particularly well adapted to such studies by allowing some potential changes in the follow-up to be taken into account. In particular, this leads to more accurate predictions by updating the available information throughout the patient monitoring. Some mathematical tools have been developed to quantify and compare the effectiveness of dynamic predictions using dynamic versions of the area under the receiver operating characteristic curve and the Brier score in the competing risks setting. Nevertheless, only two predictive abilities can be compared. This may be too restrictive in a clinical context where more and more information can be collected during patient follow-up thanks to recent technological advances. Here we propose a new procedure that allows multiple comparisons of the predictive abilities of different biomarkers, based on the dynamic area under the receiver operating characteristic curve or Brier score. Performances of our testing procedure were assessed by simulations. Moreover, a motivating application in hepatology will be presented. Finally, this work compares more than two dynamic predictive abilities of biomarkers and is available via R functions on GitHub.

Development and Validation of Multivariable Prediction Models for In-Hospital Death, 30-Day Death, and Change in Residence After Hip Fracture Surgery and the "Stratify-Hip" Algorithm

BACKGROUND

To develop and validate the stratify-hip algorithm (multivariable prediction models to predict those at low, medium, and high risk across in-hospital death, 30-day death, and residence change after hip fracture).

METHODS

Multivariable Fine-Gray and logistic regression of audit data linked to hospital records for older adults surgically treated for hip fracture in England/Wales 2011-14 (development n = 170 411) and 2015-16 (external validation, n = 90 102). Outcomes included time to in-hospital death, death at 30 days, and time to residence change. Predictors included age, sex, pre-fracture mobility, dementia, and pre-fracture residence (not for residence change). Model assumptions, performance, and sensitivity to missingness were assessed. Models were incorporated into the stratify-hip algorithm assigning patients to overall low (low risk across outcomes), medium (low death risk, medium/high risk of residence change), or high (high risk of in-hospital death, high/medium risk of 30-day death) risk.

RESULTS

For complete-case analysis, 6 780 of 141 158 patients (4.8%) died in-hospital, 8 693 of 149 258 patients (5.8%) died by 30 days, and 4 461 of 119 420 patients (3.7%) had residence change. Models demonstrated acceptable calibration (observed:expected ratio 0.90, 0.99, and 0.94), and discrimination (area under curve 73.1, 71.1, and 71.5; Brier score 5.7, 5.3, and 5.6) for in-hospital death, 30-day death, and residence change, respectively. Overall, 31%, 28%, and 41% of patients were assigned to overall low, medium, and high risk. External validation and missing data analyses elicited similar findings. The algorithm is available at https://stratifyhip.co.uk.

CONCLUSIONS

The current study developed and validated the stratify-hip algorithm as a new tool to risk stratify patients after hip fracture.

Statistical models versus machine learning for competing risks: development and validation of prognostic models

BACKGROUND

In health research, several chronic diseases are susceptible to competing risks (CRs). Initially, statistical models (SM) were developed to estimate the cumulative incidence of an event in the presence of CRs. As recently there is a growing interest in applying machine learning (ML) for clinical prediction, these techniques have also been extended to model CRs but literature is limited. Here, our aim is to investigate the potential role of ML versus SM for CRs within non-complex data (small/medium sample size, low dimensional setting).

METHODS

A dataset with 3826 retrospectively collected patients with extremity soft-tissue sarcoma (eSTS) and nine predictors is used to evaluate model-predictive performance in terms of discrimination and calibration. Two SM (cause-specific Cox, Fine-Gray) and three ML techniques are compared for CRs in a simple clinical setting. ML models include an original partial logistic artificial neural network for CRs (PLANNCR original), a PLANNCR with novel specifications in terms of architecture (PLANNCR extended), and a random survival forest for CRs (RSFCR). The clinical endpoint is the time in years between surgery and disease progression (event of interest) or death (competing event). Time points of interest are 2, 5, and 10 years.

RESULTS

Based on the original eSTS data, 100 bootstrapped training datasets are drawn. Performance of the final models is assessed on validation data (left out samples) by employing as measures the Brier score and the Area Under the Curve (AUC) with CRs. Miscalibration (absolute accuracy error) is also estimated. Results show that the ML models are able to reach a comparable performance versus the SM at 2, 5, and 10 years regarding both Brier score and AUC (95% confidence intervals overlapped). However, the SM are frequently better calibrated.

CONCLUSIONS

Overall, ML techniques are less practical as they require substantial implementation time (data preprocessing, hyperparameter tuning, computational intensity), whereas regression methods can perform well without the additional workload of model training. As such, for non-complex real life survival data, these techniques should only be applied complementary to SM as exploratory tools of model's performance. More attention to model calibration is urgently needed.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression.

As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker.

The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation.

The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

Public and occupational health risks related to lead exposure updated according to present-day blood lead levels

Lead is an environmental hazard that should be addressed worldwide. Over time, human lead exposure in the western world has decreased drastically to levels comparable to those among humans living in the preindustrial era, who were mainly exposed to natural sources of lead. To re-evaluate the potential health risks associated with present-day lead exposure, a two-pronged approach was applied. First, recently published population metrics describing the adverse health effects associated with lead exposure at the population level were critically assessed. Next, the key results of the Study for Promotion of Health in Recycling Lead (SPHERL; NCT02243904) were summarized and put in perspective with those of the published population metrics. To our knowledge, SPHERL is the first prospective study that accounted for interindividual variability between people with respect to their vulnerability to the toxic effects of lead exposure by assessing the participants' health status before and after occupational lead exposure. The overall conclusion of this comprehensive review is that mainstream ideas about the public and occupational health risks related to lead exposure urgently need to be updated because a large portion of the available literature became obsolete given the sharp decrease in exposure levels over the past 40 years.

Computer-aided diagnostic models to classify lymph node metastasis and lymphoma involvement in enlarged cervical lymph nodes using PET/CT

BACKGROUND

It is a clinical problem to identify histological component in enlarged cervical lymph nodes, particularly in differentiation between lymph node metastasis and lymphoma involvement.

PURPOSE

To construct two kinds of deep learning (DL)-based computer-aided diagnosis (CAD) systems including DL-convolutional neural networks (DL-CNN) and DL-machine learning for pathological diagnosis of cervical lymph nodes by positron emission tomography (PET)/computed tomography (CT) images.

METHODS

We collected CT, PET, and PET/CT images series from 165 patients with enlarged cervical lymph nodes receiving examinations from January 2014 to June 2018. Six CNNs pretrained on ImageNet as DL architectures were used for two kinds of DL-based CAD models, including DL-CNN and DL-machine learning models. The DL-CNN models were constructed via transfer learning for classification of lymphomatous and metastatic lymph nodes. The DL-machine learning models were developed by DL-based features extractors and support vector machine (SVM) classifier. As for DL-SVM models, we also evaluate the effect of handcrafted radiomics features in combination of DL-based features.

RESULTS

The DL-CNN model with ResNet50 architecture on PET/CT images had the best diagnostic performance among all six algorithms with an area under the receiver operating characteristic curve (AUC) of 0.845 and accuracy of 78.13% in the testing cohort. The DL-SVM model on ResNet50 extractor showed great performance for the testing cohort with an AUC of 0.901, accuracy of 86.96%, sensitivity of 76.09%, and specificity of 94.20%. The combination of DL-based and handcrafted features yielded the improvement of diagnostic performance.

CONCLUSIONS

Our DL-based CAD systems on PET/CT images were developed for classifying metastatic and lymphomatous involvement with favorable diagnostic performance in enlarged cervical lymph nodes. Further clinical practice of our systems may improve quality of the following therapeutic interventions and optimize patients' outcomes.

MRI-Based Computer-Aided Diagnostic Model to Predict Tumor Grading and Clinical Outcomes in Patients With Soft Tissue Sarcoma

BACKGROUND

MRI acts as a potential resource for exploration and interpretation to identify tumor characterization by advanced computer-aided diagnostic (CAD) methods.

PURPOSE

To evaluate and validate the performance of MRI-based CAD models for identifying low-grade and high-grade soft tissue sarcoma (STS) and for investigating survival prognostication.

STUDY TYPE

Retrospective.

SUBJECTS

A total of 540 patients (295 male/female: 295/245, median age: 42 years) with STSs.

FIELD SEQUENCE

5-T MRI with T₁ WI sequence and fat-suppressed T₂ -weighted (T₂ FS) sequence.

ASSESSMENT

Manual regions of interests (ROIs) were delineated for generation of radiomic features. Automatic segmentation and pretrained convolutional neural networks (CNNs) were performed for deep learning (DL) analysis. The last fully connected layer at the top of CNNs was removed, and the global max pooling was added to transform feature maps to numeric values. Tumor grade was determined on histological specimens.

STATISTICAL TESTS

The support vector machine was adopted as the classifier for all MRI-based models. The DL signature was derived from the DL-MRI model with the highest area under the curve (AUC). The significant clinical variables, tumor location and size, integrated with radiomics and DL signatures were ready for construction of clinical-MRI nomogram to identify tumor grading. The prognostic value of clinical variables and these MRI-based signatures for overall survival (OS) was evaluated via Cox proportional hazard.

RESULTS

The clinical-MRI differentiation nomogram represented an AUC of 0.870 in the training cohort, and an AUC of 0.855, accuracy of 79.01%, sensitivity of 79.03%, and specificity of 78.95% in the validation cohort. The prognostic model showed good performance for OS with 3-year C-index of 0.681 and 0.642 and 5-year C-index of 0.722 and 0.676 in the training and validation cohorts.

DATA CONCLUSION

MRI-based CAD nomogram represents effective abilities in classification of low-grade and high-grade STSs. The MRI-based prognostic model yields favorable preoperative capacities to identify long-term survivals for STSs.

EVIDENCE LEVEL

3 TECHNICAL EFFICACY: Stage 4.

Predicting COVID-19 progression in hospitalized patients in Belgium from a multi-state model

Objectives

To adopt a multi-state risk prediction model for critical disease/mortality outcomes among hospitalised COVID-19 patients using nationwide COVID-19 hospital surveillance data in Belgium.

Materials and methods

Information on 44,659 COVID-19 patients hospitalised between March 2020 and June 2021 with complete data on disease outcomes and candidate predictors was used to adopt a multi-state, multivariate Cox model to predict patients' probability of recovery, critical [transfer to intensive care units (ICU)] or fatal outcomes during hospital stay.

Results

Median length of hospital stay was 9 days (interquartile range: 5-14). After admission, approximately 82% of the COVID-19 patients were discharged alive, 15% of patients were admitted to ICU, and 15% died in the hospital. The main predictors of an increased probability for recovery were younger age, and to a lesser extent, a lower number of prevalent comorbidities. A patient's transition to ICU or in-hospital death had in common the following predictors: high levels of c-reactive protein (CRP) and lactate dehydrogenase (LDH), reporting lower respiratory complaints and male sex. Additionally predictors for a transfer to ICU included middle-age, obesity and reporting loss of appetite and staying at a university hospital, while advanced age and a higher number of prevalent comorbidities for in-hospital death. After ICU, younger age and low levels of CRP and LDH were the main predictors for recovery, while in-hospital death was predicted by advanced age and concurrent comorbidities.

Conclusion

As one of the very few, a multi-state model was adopted to identify key factors predicting COVID-19 progression to critical disease, and recovery or death.

Regression trees and ensembles for cumulative incidence functions

The use of cumulative incidence functions for characterizing the risk of one type of event in the presence of others has become increasingly popular over the past two decades. The problems of modeling, estimation and inference have been treated using parametric, nonparametric and semi-parametric methods. Efforts to develop suitable extensions of machine learning methods, such as regression trees and ensemble methods, have begun comparatively recently. In this paper, we propose a novel approach to estimating cumulative incidence curves in a competing risks setting using regression trees and associated ensemble estimators. The proposed methods use augmented estimators of the Brier score risk as the primary basis for building and pruning trees, and lead to methods that are easily implemented using existing R packages. Data from the Radiation Therapy Oncology Group (trial 9410) is used to illustrate these new methods.

Novel computer aided diagnostic models on multimodality medical images to differentiate well differentiated liposarcomas from lipomas approached by deep learning methods

BACKGROUND

Deep learning methods have great potential to predict tumor characterization, such as histological diagnosis and genetic aberration. The objective of this study was to evaluate and validate the predictive performance of multimodality imaging-derived models using computer-aided diagnostic (CAD) methods for prediction of MDM2 gene amplification to identify well-differentiated liposarcoma (WDLPS) and lipoma.

MATERIALS AND METHODS

All 127 patients from two institutions were included with 89 patients in one institution for model training and 38 patients in the other institution for external validation between January 2012 and December 2018. For each modality, handcrafted radiomics analysis with manual segmentation was applied to extract 851 features for each modality, and six pretrained convolutional neural networks (CNNs) extracted 512-2048 deep learning features automatically. Extracted imaging-based features were selected via univariate filter selection methods and the recursive feature elimination algorithm, which were then classified by support vector machine for model construction. Integrated with two significant clinical variables, age and LDH level, a clinical-radiological model was constructed for identification WDLPS and lipoma. All differentiation models were evaluated using the area under the receiver operating characteristics curve (AUC) and their 95% confidence interval (CI).

RESULTS

The multimodality model on deep learning features extracted from ResNet50 algorithm (RN-DL model) performed great differentiation performance with an AUC of 0.995 (95% CI 0.987-1.000) for the training cohort, and an AUC of 0.950 (95% CI 0.886-1.000), accuracy of 92.11%, sensitivity of 95.00% (95% CI 73.06-99.74%), specificity of 88.89% (95% CI 63.93-98.05%) in external validation. The integrated clinical-radiological model represented an AUC of 0.996 (95% CI 0.989-1.000) for the training cohort, and an AUC of 0.942 (95% CI 0.867-1.000), accuracy of 86.84%, sensitivity of 95.00% (95% CI 73.06-99.74%), and specificity of 77.78% (95% CI 51.92-92.63%) in external validation.

CONCLUSIONS

Imaging-based multimodality models represent effective discrimination abilities between WDLPS and lipoma via CAD methods, and might be a practicable approach in assistance of treatment decision.

Lessons learnt when accounting for competing events in the external validation of time-to-event prognostic models

Background

External validation of prognostic models is necessary to assess the accuracy and generalizability of the model to new patients. If models are validated in a setting in which competing events occur, these competing risks should be accounted for when comparing predicted risks to observed outcomes.

Methods

We discuss existing measures of calibration and discrimination that incorporate competing events for time-to-event models. These methods are illustrated using a clinical-data example concerning the prediction of kidney failure in a population with advanced chronic kidney disease (CKD), using the guideline-recommended Kidney Failure Risk Equation (KFRE). The KFRE was developed using Cox regression in a diverse population of CKD patients and has been proposed for use in patients with advanced CKD in whom death is a frequent competing event.

Results

When validating the 5-year KFRE with methods that account for competing events, it becomes apparent that the 5-year KFRE considerably overestimates the real-world risk of kidney failure. The absolute overestimation was 10%age points on average and 29%age points in older high-risk patients.

Conclusions

It is crucial that competing events are accounted for during external validation to provide a more reliable assessment the performance of a model in clinical settings in which competing risks occur.

Deep learning radiomics based on contrast enhanced computed tomography predicts microvascular invasion and survival outcome in early stage hepatocellular carcinoma

OBJECTIVE

To evaluate the performance of a deep learning (DL)-based radiomics strategy on contrast-enhanced computed tomography (CT) to predict microvascular invasion (MVI) status and clinical outcomes, recurrence-free survival (RFS) and overall survival (OS) in patients with early stage hepatocellular carcinoma (HCC) receiving surgical resection.

METHODS

All 283 eligible patients were included retrospectively between January 2008 and December 2015, and assigned into the training cohort (n = 198) and the testing cohort (n = 85). We extracted radiomics features via handcrafted radiomics analysis manually and DL analysis of pretrained convolutional neural networks via transfer learning automatically. Support vector machine was adopted as the classifier. A clinical-radiological model for MVI status integrated significant clinical features and the radiological signature generated from the radiological model with the optimal area under the receiver operating characteristics curve (AUC) in the testing cohort. Otherwise, DL-based prognostic models were constructed in prediction of recurrence and mortality via Cox proportional hazard analysis.

RESULTS

The clinical-radiological model for MVI represented an AUC of 0.909, accuracy of 96.47%, sensitivity of 90.91%, specificity of 97.30%, positive predictive value of 83.33%, and negative predictive value of 98.63% in the testing cohort. The clinical-radiological models for identification of RFS and OS outperformed prediction performance of the clinical model or the DL signature alone. The DL-based integrated model for prognostication showed great predictive value with significant classification and discrimination abilities after validation.

CONCLUSIONS

The integrated DL-based radiomics models achieved accurate preoperative prediction of MVI status, and might facilitate predicting tumor recurrence and mortality in order to optimize clinical decisions for patients with early stage HCC.

Evaluation of competing risks prediction models using polytomous discrimination index

For competing risks data, it is often important to predict a patient's outcome status at a clinically meaningful time point after incorporating the informative censoring due to competing risks. This can be done by adopting a regression model that relates the cumulative incidence probabilities to a set of covariates. To assess the performance of the resulting prediction tool, we propose an estimator of the polytomous discrimination index applicable to competing risks data, which can quantify a prognostic model's ability to discriminate among subjects from different outcome groups. The proposed estimator allows the prediction model to be subject to model misspecification and enjoys desirable asymptotic properties. We also develop an efficient computation algorithm that features a computational complexity of O(n log n). A perturbation resampling scheme is developed to achieve consistent variance estimation. Numerical results suggest that the estimator performs well under realistic sample sizes. We apply the proposed methods to a study of monoclonal gammopathy of undetermined significance.

Prognosis prediction of extremity and trunk wall soft-tissue sarcomas treated with surgical resection with radiomic analysis based on random survival forest

Many researches have applied machine learning methods to find associations between radiomic features and clinical outcomes. Random survival forests (RSF), as an accurate classifier, sort all candidate variables as the rank of importance values. There was no study concerning on finding radiomic predictors in patients with extremity and trunk wall soft-tissue sarcomas using RSF. This study aimed to determine associations between radiomic features and overall survival (OS) by RSF analysis. To identify radiomic features with important values by RSF analysis, construct predictive models for OS incorporating clinical characteristics, and evaluate models' performance with different method. We collected clinical characteristics and radiomic features extracted from plain and contrast-enhanced computed tomography (CT) from 353 patients with extremity and trunk wall soft-tissue sarcomas treated with surgical resection. All radiomic features were analyzed by Cox proportional hazard (CPH) and followed RSF analysis. The association between radiomics-predicted risks and OS was assessed by Kaplan-Meier analysis. All clinical features were screened by CPH analysis. Prognostic clinical and radiomic parameters were fitted into RSF and CPH integrative models for OS in the training cohort, respectively. The concordance indexes (C-index) and Brier scores of both two models were evaluated in both training and testing cohorts. The model with better predictive performance was interpreted with nomogram and calibration plots. Among all 86 radiomic features, there were three variables selected with high importance values. The RSF on these three features distinguished patients with high predicted risks from patients with low predicted risks for OS in the training set (P < 0.001) using Kaplan-Meier analysis. Age, lymph node involvement and grade were incorporated into the combined models for OS (P < 0.05). The C-indexes in both two integrative models fluctuated above 0.80 whose Brier scores maintained less than 15.0 in the training and testing datasets. The RSF model performed little advantages over the CPH model that the calibration curve of the RSF model showed favorable agreement between predicted and actual survival probabilities for the 3-year and 5-year survival prediction. The multimodality RSF model including clinical and radiomic characteristics conducted high capacity in prediction of OS which might assist individualized therapeutic regimens. Level III, prognostic study.

Predicting in-hospital mortality from Coronavirus Disease 2019: A simple validated app for clinical use

BACKGROUNDS

Validated tools for predicting individual in-hospital mortality of COVID-19 are lacking. We aimed to develop and to validate a simple clinical prediction rule for early identification of in-hospital mortality of patients with COVID-19.

METHODS AND FINDINGS

We enrolled 2191 consecutive hospitalized patients with COVID-19 from three Italian dedicated units (derivation cohort: 1810 consecutive patients from Bergamo and Pavia units; validation cohort: 381 consecutive patients from Rome unit). The outcome was in-hospital mortality. Fine and Gray competing risks multivariate model (with discharge as a competing event) was used to develop a prediction rule for in-hospital mortality. Discrimination and calibration were assessed by the area under the receiver operating characteristic curve (AUC) and by Brier score in both the derivation and validation cohorts. Seven variables were independent risk factors for in-hospital mortality: age (Hazard Ratio [HR] 1.08, 95% Confidence Interval [CI] 1.07-1.09), male sex (HR 1.62, 95%CI 1.30-2.00), duration of symptoms before hospital admission <10 days (HR 1.72, 95%CI 1.39-2.12), diabetes (HR 1.21, 95%CI 1.02-1.45), coronary heart disease (HR 1.40 95% CI 1.09-1.80), chronic liver disease (HR 1.78, 95%CI 1.16-2.72), and lactate dehydrogenase levels at admission (HR 1.0003, 95%CI 1.0002-1.0005). The AUC was 0.822 (95%CI 0.722-0.922) in the derivation cohort and 0.820 (95%CI 0.724-0.920) in the validation cohort with good calibration. The prediction rule is freely available as a web-app (COVID-CALC: https://sites.google.com/community.unipa.it/covid-19riskpredictions/c19-rp).

CONCLUSIONS

A validated simple clinical prediction rule can promptly and accurately assess the risk for in-hospital mortality, improving triage and the management of patients with COVID-19.

Inference about time-dependent prognostic accuracy measures in the presence of competing risks

BACKGROUND

Evaluating a candidate marker or developing a model for predicting risk of future conditions is one of the major goals in medicine. However, model development and assessment for a time-to-event outcome may be complicated in the presence of competing risks. In this manuscript, we propose a local and a global estimators of cause-specific AUC for right-censored survival times in the presence of competing risks.

METHODS

The local estimator - cause-specific weighted mean rank (cWMR) - is a local average of time-specific observed cause-specific AUCs within a neighborhood of given time t. The global estimator - cause-specific fractional polynomials (cFPL) - is based on modelling the cause-specific AUC as a function of t through fractional polynomials.

RESULTS

We investigated the performance of the proposed cWMR and cFPL estimators through simulation studies and real-life data analysis. The estimators perform well in small samples, have minimal bias and appropriate coverage.

CONCLUSIONS

The local estimator cWMR and the global estimator cFPL will provide computationally efficient options for assessing the prognostic accuracy of markers for time-to-event outcome in the presence of competing risks in many practical settings.

Prediction meets causal inference: the role of treatment in clinical prediction models

In this paper we study approaches for dealing with treatment when developing a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a 'predictimand' framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference.

Validation of discrete time-to-event prediction models in the presence of competing risks

Clinical prediction models play a key role in risk stratification, therapy assignment and many other fields of medical decision making. Before they can enter clinical practice, their usefulness has to be demonstrated using systematic validation. Methods to assess their predictive performance have been proposed for continuous, binary, and time-to-event outcomes, but the literature on validation methods for discrete time-to-event models with competing risks is sparse. The present paper tries to fill this gap and proposes new methodology to quantify discrimination, calibration, and prediction error (PE) for discrete time-to-event outcomes in the presence of competing risks. In our case study, the goal was to predict the risk of ventilator-associated pneumonia (VAP) attributed to Pseudomonas aeruginosa in intensive care units (ICUs). Competing events are extubation, death, and VAP due to other bacteria. The aim of this application is to validate complex prediction models developed in previous work on more recently available validation data.

Assessment of ST2 for risk of death following graft-versus-host disease in pediatric and adult age groups

Assessment of prognostic biomarkers of nonrelapse mortality (NRM) after allogeneic hematopoietic cell transplantation (HCT) in the pediatric age group is lacking. To address this need, we conducted a prospective cohort study with 415 patients at 6 centers: 170 were children age 10 years or younger and 245 were patients older than age 10 years (both children and adults were accrued from 2013 to 2018). The following 4 plasma biomarkers were assessed pre-HCT and at days +7, +14, and +21 post-HCT: stimulation-2 (ST2), tumor necrosis factor receptor 1 (TNFR1), regenerating islet-derived protein 3α (REG3α), and interleukin-6 (IL-6). We performed landmark analyses for NRM, dichotomizing the cohort at age 10 years or younger and using each biomarker median as a cutoff for high- and low-risk groups. Post-HCT biomarker analysis showed that ST2 (>26 ng/mL), TNFR1 (>3441 pg/mL), and REG3α (>25 ng/mL) are associated with NRM in children age 10 years or younger (ST2: hazard ratio [HR], 9.13; 95% confidence interval [CI], 2.74-30.38; P = .0003; TNFR1: HR, 4.29; 95% CI, 1.48-12.48; P = .0073; REG3α: HR, 7.28; 95% CI, 2.05-25.93; P = .0022); and in children and adults older than age 10 years (ST2: HR, 2.60; 95% CI, 1.15-5.86; P = .021; TNFR1: HR, 2.09; 95% CI, 0.96-4.58; P = .06; and REG3α: HR, 2.57; 95% CI, 1.19-5.55; P = .016). When pre-HCT biomarkers were included, only ST2 remained significant in both cohorts. After adjustment for significant covariates (race/ethnicity, malignant disease, graft, and graft-versus-host-disease prophylaxis), ST2 remained associated with NRM only in recipients age 10 years or younger (HR, 4.82; 95% CI, 1.89-14.66; P = .0056). Assays of ST2, TNFR1, and REG3α in the first 3 weeks after HCT have prognostic value for NRM in both children and adults. The presence of ST2 before HCT is a prognostic biomarker for NRM in children age 10 years or younger allowing for additional stratification. This trial was registered at www.clinicaltrials.gov as #NCT02194439.

Quantifying diagnostic accuracy improvement of new biomarkers for competing risk outcomes

The net reclassification improvement (NRI) and the integrated discrimination improvement (IDI) were originally proposed to characterize accuracy improvement in predicting a binary outcome, when new biomarkers are added to regression models. These two indices have been extended from binary outcomes to multi-categorical and survival outcomes. Working on an AIDS study where the onset of cognitive impairment is competing risk censored by death, we extend the NRI and the IDI to competing risk outcomes, by using cumulative incidence functions to quantify cumulative risks of competing events, and adopting the definitions of the two indices for multi-category outcomes. The "missing" category due to independent censoring is handled through inverse probability weighting. Various competing risk models are considered, such as the Fine and Gray, multistate, and multinomial logistic models. Estimation methods for the NRI and the IDI from competing risk data are presented. The inference for the NRI is constructed based on asymptotic normality of its estimator, and the bias-corrected and accelerated bootstrap procedure is used for the IDI. Simulations demonstrate that the proposed inferential procedures perform very well. The Multicenter AIDS Cohort Study is used to illustrate the practical utility of the extended NRI and IDI for competing risk outcomes.

Assessing predictive accuracy of survival regressions subject to nonindependent censoring

Survival regression is commonly applied in biomedical studies or clinical trials, and evaluating their predictive performance plays an essential role for model diagnosis and selection. The presence of censored data, particularly if informative, may pose more challenges for the assessment of predictive accuracy. Existing literature mainly focuses on prediction for survival probabilities with limitation work for survival time. In this work, we focus on accuracy measures of predicted survival times adjusted for a potentially informative censoring mechanism (ie, coarsening at random (CAR); non-CAR) by adopting the technique of inverse probability of censoring weighting. Our proposed predictive metric can be adaptive to various survival regression frameworks including but not limited to accelerated failure time models and proportional hazards models. Moreover, we provide the asymptotic properties of the inverse probability of censoring weighting estimators under CAR. We consider the settings of high-dimensional data under CAR or non-CAR for extensions. The performance of the proposed method is evaluated through extensive simulation studies and analysis of real data from the Critical Assessment of Microarray Data Analysis.

Interpretation of Population Health Metrics: Environmental Lead Exposure as Exemplary Case

Our objective was to gain insight in the calculation and interpretation of population health metrics that inform disease prevention. Using as model environmental exposure to lead (ELE), a global pollutant, we assessed population health metrics derived from the Third National Health and Nutrition Examination Survey (1988 to 1994), the GBD (Global Burden of Disease Study 2010), and the Organization for Economic Co-operation and Development. In the National Health and Nutrition Examination Survey, the hazard ratio relating mortality over 19.3 years of follow-up to a blood lead increase at baseline from 1.0 to 6.7 µg/dL (10th–90th percentile interval) was 1.37 (95% CI, 1.17–1.60). The population-attributable fraction of blood lead was 18.0% (10.9%–26.1%). The number of preventable ELE-related deaths in the United States would be 412 000 per year (250 000–598 000). In GBD 2010, deaths and disability-adjusted life-years globally lost due to ELE were 0.67 million (0.58–0.78 million) and 0.56% (0.47%–0.66%), respectively. According to the 2017 Organization for Economic Co-operation and Development statistics, ELE-related welfare costs were $1 676 224 million worldwide. Extrapolations from the foregoing metrics assumed causality and reversibility of the association between mortality and blood lead, which at present-day ELE levels in developed nations is not established. Other issues limiting the interpretation of ELE-related population health metrics are the inflation of relative risk based on outdated blood lead levels, not differentiating relative from absolute risk, clustering of risk factors and exposures within individuals, residual confounding, and disregarding noncardiovascular disease and immigration in national ELE-associated welfare estimates. In conclusion, this review highlights the importance of critical thinking in translating population health metrics into cost-effective preventive strategies.

Dynamic prediction of repeated events data based on landmarking model: application to colorectal liver metastases data

BACKGROUND

In some clinical situations, patients experience repeated events of the same type. Among these, cancer recurrences can result in terminal events such as death. Therefore, here we dynamically predicted the risks of repeated and terminal events given longitudinal histories observed before prediction time using dynamic pseudo-observations (DPOs) in a landmarking model.

METHODS

The proposed DPOs were calculated using Aalen-Johansen estimator for the event processes described in the multi-state model. Furthermore, in the absence of a terminal event, a more convenient approach without matrix operation was described using the ordering of repeated events. Finally, generalized estimating equations were used to calculate probabilities of repeated and terminal events, which were treated as multinomial outcomes.

RESULTS

Simulation studies were conducted to assess bias and investigate the efficiency of the proposed DPOs in a finite sample. Little bias was detected in DPOs even under relatively heavy censoring, and the method was applied to data from patients with colorectal liver metastases.

CONCLUSIONS

The proposed method enabled intuitive interpretations of terminal event settings.

Overview of model validation for survival regression model with competing risks using melanoma study data

The article introduces how to validate regression models in the analysis of competing risks. The prediction accuracy of competing risks regression models can be assessed by discrimination and calibration. The area under receiver operating characteristic curve (AUC) or Concordance-index, and calibration plots have been widely used as measures of discrimination and calibration, respectively. One-time splitting method can be used for randomly splitting original data into training and test datasets. However, this method reduces sample sizes of both training and testing datasets, and the results can be different by different splitting processes. Thus, the cross-validation method is more appealing. For time-to-event data, model validation is performed at each analysis time point. In this article, we review how to perform model validation using the riskRegression package in R, along with plotting a nomogram for competing risks regression models using the regplot() package.

Quantifying and estimating the predictive accuracy for censored time-to-event data with competing risks

This paper focuses on quantifying and estimating the predictive accuracy of prognostic models for time-to-event outcomes with competing events. We consider the time-dependent discrimination and calibration metrics, including the receiver operating characteristics curve and the Brier score, in the context of competing risks. To address censoring, we propose a unified nonparametric estimation framework for both discrimination and calibration measures, by weighting the censored subjects with the conditional probability of the event of interest given the observed data. The proposed method can be extended to time-dependent predictive accuracy metrics constructed from a general class of loss functions. We apply the methodology to a data set from the African American Study of Kidney Disease and Hypertension to evaluate the predictive accuracy of a prognostic risk score in predicting end-stage renal disease, accounting for the competing risk of pre-end-stage renal disease death, and evaluate its numerical performance in extensive simulation studies.

Prediction errors for state occupation and transition probabilities in multi-state models

In this paper, we consider the estimation of prediction errors for state occupation probabilities and transition probabilities for multistate time-to-event data. We study prediction errors based on the Brier score and on the Kullback-Leibler score and prove their properness. In the presence of right-censored data, two classes of estimators, based on inverse probability weighting and pseudo-values, respectively, are proposed, and consistency properties of the proposed estimators are investigated. The second part of the paper is devoted to the estimation of dynamic prediction errors for state occupation probabilities for multistate models, conditional on being alive, and for transition probabilities. Cross-validated versions are proposed. Our methods are illustrated on the CSL1 randomized clinical trial comparing prednisone versus placebo for liver cirrhosis patients.

Prediction accuracy and variable selection for penalized cause-specific hazards models

We consider modeling competing risks data in high dimensions using a penalized cause-specific hazards (CSHs) approach. CSHs have conceptual advantages that are useful for analyzing molecular data. First, working on hazards level can further understanding of the underlying biological mechanisms that drive transition hazards. Second, CSH models can be used to extend the multistate framework for high-dimensional data. The CSH approach is implemented by fitting separate proportional hazards models for each event type (iCS). In the high-dimensional setting, this might seem too complex and possibly prone to overfitting. Therefore, we consider an extension, namely "linking" the separate models by choosing penalty tuning parameters that in combination yield best prediction of the incidence of the event of interest (penCR). We investigate whether this extension is useful with respect to prediction accuracy and variable selection. The two approaches are compared to the subdistribution hazards (SDH) model, which is an established method that naturally achieves "linking" by working on incidence level, but loses interpretability of the covariate effects. Our simulation studies indicate that in many aspects, iCS is competitive to penCR and the SDH approach. There are some instances that speak in favor of linking the CSH models, for example, in the presence of opposing effects on the CSHs. We conclude that penalized CSH models are a viable solution for competing risks models in high dimensions. Linking the CSHs can be useful in some particular cases; however, simple models using separately penalized CSH are often justified.

Penalized variable selection in competing risks regression

Penalized variable selection methods have been extensively studied for standard time-to-event data. Such methods cannot be directly applied when subjects are at risk of multiple mutually exclusive events, known as competing risks. The proportional subdistribution hazard (PSH) model proposed by Fine and Gray (J Am Stat Assoc 94:496-509, 1999) has become a popular semi-parametric model for time-to-event data with competing risks. It allows for direct assessment of covariate effects on the cumulative incidence function. In this paper, we propose a general penalized variable selection strategy that simultaneously handles variable selection and parameter estimation in the PSH model. We rigorously establish the asymptotic properties of the proposed penalized estimators and modify the coordinate descent algorithm for implementation. Simulation studies are conducted to demonstrate the good performance of the proposed method. Data from deceased donor kidney transplants from the United Network of Organ Sharing illustrate the utility of the proposed method.

Combined dynamic predictions using joint models of two longitudinal outcomes and competing risk data

Nowadays there is an increased medical interest in personalized medicine and tailoring decision making to the needs of individual patients. Within this context our developments are motivated from a Dutch study at the Cardio-Thoracic Surgery Department of the Erasmus Medical Center, consisting of patients who received a human tissue valve in aortic position and who were thereafter monitored echocardiographically. Our aim is to utilize the available follow-up measurements of the current patients to produce dynamically updated predictions of both survival and freedom from re-intervention for future patients. In this paper, we propose to jointly model multiple longitudinal measurements combined with competing risk survival outcomes and derive the dynamically updated cumulative incidence functions. Moreover, we investigate whether different features of the longitudinal processes would change significantly the prediction for the events of interest by considering different types of association structures, such as time-dependent trajectory slopes and time-dependent cumulative effects. Our final contribution focuses on optimizing the quality of the derived predictions. In particular, instead of choosing one final model over a list of candidate models which ignores model uncertainty, we propose to suitably combine predictions from all considered models using Bayesian model averaging.

Explained variation for recurrent event data

Although there are many suggested measures of explained variation for single-event survival data, there has been little attention to explained variation for recurrent event data. We describe an existing rank-based measure and we investigate a new statistic based on observed and expected event count processes. Both methods can be used for all models. Adjustments for missing data are proposed and demonstrated through simulation to be effective. We compare the population values of the two statistics and illustrate their use in comparing an array of non-nested models for data on recurrent episodes of infant diarrhoea.

Quantifying and comparing dynamic predictive accuracy of joint models for longitudinal marker and time-to-event in presence of censoring and competing risks

Thanks to the growing interest in personalized medicine, joint modeling of longitudinal marker and time-to-event data has recently started to be used to derive dynamic individual risk predictions. Individual predictions are called dynamic because they are updated when information on the subject's health profile grows with time. We focus in this work on statistical methods for quantifying and comparing dynamic predictive accuracy of this kind of prognostic models, accounting for right censoring and possibly competing events. Dynamic area under the ROC curve (AUC) and Brier Score (BS) are used to quantify predictive accuracy. Nonparametric inverse probability of censoring weighting is used to estimate dynamic curves of AUC and BS as functions of the time at which predictions are made. Asymptotic results are established and both pointwise confidence intervals and simultaneous confidence bands are derived. Tests are also proposed to compare the dynamic prediction accuracy curves of two prognostic models. The finite sample behavior of the inference procedures is assessed via simulations. We apply the proposed methodology to compare various prediction models using repeated measures of two psychometric tests to predict dementia in the elderly, accounting for the competing risk of death. Models are estimated on the French Paquid cohort and predictive accuracies are evaluated and compared on the French Three-City cohort.

Concordance for prognostic models with competing risks

The concordance probability is a widely used measure to assess discrimination of prognostic models with binary and survival endpoints. We formally define the concordance probability for a prognostic model of the absolute risk of an event of interest in the presence of competing risks and relate it to recently proposed time-dependent area under the receiver operating characteristic curve measures. For right-censored data, we investigate inverse probability of censoring weighted (IPCW) estimates of a truncated concordance index based on a working model for the censoring distribution. We demonstrate consistency and asymptotic normality of the IPCW estimate if the working model is correctly specified and derive an explicit formula for the asymptotic variance under independent censoring. The small sample properties of the estimator are assessed in a simulation study also against misspecification of the working model. We further illustrate the methods by computing the concordance probability for a prognostic model of coronary heart disease (CHD) events in the presence of the competing risk of non-CHD death.

A K-nearest neighbors survival probability prediction method

We introduce a nonparametric survival prediction method for right-censored data. The method generates a survival curve prediction by constructing a (weighted) Kaplan-Meier estimator using the outcomes of the K most similar training observations. Each observation has an associated set of covariates, and a metric on the covariate space is used to measure similarity between observations. We apply our method to a kidney transplantation data set to generate patient-specific distributions of graft survival and to a simulated data set in which the proportional hazards assumption is explicitly violated. We compare the performance of our method with the standard Cox model and the random survival forests method.

Model selection in competing risks regression

In the analysis of time-to-event data, the problem of competing risks occurs when an individual may experience one, and only one, of m different types of events. The presence of competing risks complicates the analysis of time-to-event data, and standard survival analysis techniques such as Kaplan-Meier estimation, log-rank test and Cox modeling are not always appropriate and should be applied with caution. Fine and Gray developed a method for regression analysis that models the hazard that corresponds to the cumulative incidence function. This model is becoming widely used by clinical researchers and is now available in all the major software environments. Although model selection methods for Cox proportional hazards models have been developed, few methods exist for competing risks data. We have developed stepwise regression procedures, both forward and backward, based on AIC, BIC, and BICcr (a newly proposed criteria that is a modified BIC for competing risks data subject to right censoring) as selection criteria for the Fine and Gray model. We evaluated the performance of these model selection procedures in a large simulation study and found them to perform well. We also applied our procedures to assess the importance of bone mineral density in predicting the absolute risk of hip fracture in the Women's Health Initiative-Observational Study, where mortality was the competing risk. We have implemented our method as a freely available R package called crrstep.

Absolute risk regression for competing risks: interpretation, link functions, and prediction

In survival analysis with competing risks, the transformation model allows different functions between the outcome and explanatory variables. However, the model's prediction accuracy and the interpretation of parameters may be sensitive to the choice of link function. We review the practical implications of different link functions for regression of the absolute risk (or cumulative incidence) of an event. Specifically, we consider models in which the regression coefficients β have the following interpretation: The probability of dying from cause D during the next t years changes with a factor exp(β) for a one unit change of the corresponding predictor variable, given fixed values for the other predictor variables. The models have a direct interpretation for the predictive ability of the risk factors. We propose some tools to justify the models in comparison with traditional approaches that combine a series of cause-specific Cox regression models or use the Fine-Gray model. We illustrate the methods with the use of bone marrow transplant data.

Extension of the survival dimensionality reduction algorithm to detect epistasis in competing risks models (SDR-CR)

BACKGROUND

The discovery and the description of the genetic background of common human diseases is hampered by their complexity and dynamic behavior. Appropriate bioinformatic tools are needed to account all the facets of complex diseases and to this end we recently described the survival dimensionality reduction (SDR) algorithm in the effort to model gene-gene interactions in the context of survival analysis. When one event precludes the occurrence of another event under investigation in the 'competing risk model', survival algorithms require particular adjustment to avoid the risk of reporting wrong or biased conclusions.

METHODS

The SDR algorithm was modified to incorporate the cumulative incidence function as well as an adapted version of the Brier score for mutually exclusive outcomes, to better search for epistatic models in the competing risk setting. The applicability of the new SDR algorithm (SDR-CR) was evaluated using synthetic lifetime epistatic datasets with competing risks and on a dataset of scleroderma patients.

RESULTS/CONCLUSIONS

The SDR-CR algorithms retains a satisfactory power to detect the causative variants in simulated datasets under different scenarios of sample size and degrees of type I or type II censoring. In the real-world dataset, SDR-CR was capable of detecting a significant interaction between the IL-1α C-889T and the IL-1β C-511T single-nucleotide polymorphisms to predict the occurrence of restrictive lung disease vs. isolated pulmonary hypertension. We provide an useful extension of the SDR algorithm to analyze epistatic interactions in the competing risk settings that may be of use to unveil the genetic background of complex human diseases.

AVAILABILITY

http://sourceforge.net/projects/sdrproject/files/.