Validation of discrete time-to-event prediction models in the presence of competing risks

Lessons learnt when accounting for competing events in the external validation of time-to-event prognostic models

Background

External validation of prognostic models is necessary to assess the accuracy and generalizability of the model to new patients. If models are validated in a setting in which competing events occur, these competing risks should be accounted for when comparing predicted risks to observed outcomes.

Methods

We discuss existing measures of calibration and discrimination that incorporate competing events for time-to-event models. These methods are illustrated using a clinical-data example concerning the prediction of kidney failure in a population with advanced chronic kidney disease (CKD), using the guideline-recommended Kidney Failure Risk Equation (KFRE). The KFRE was developed using Cox regression in a diverse population of CKD patients and has been proposed for use in patients with advanced CKD in whom death is a frequent competing event.

Results

When validating the 5-year KFRE with methods that account for competing events, it becomes apparent that the 5-year KFRE considerably overestimates the real-world risk of kidney failure. The absolute overestimation was 10%age points on average and 29%age points in older high-risk patients.

Conclusions

It is crucial that competing events are accounted for during external validation to provide a more reliable assessment the performance of a model in clinical settings in which competing risks occur.

Dementia risk in the general population: large-scale external validation of prediction models in the AGES-Reykjavik study

We aimed to evaluate the external performance of prediction models for all-cause dementia or AD in the general population, which can aid selection of high-risk individuals for clinical trials and prevention. We identified 17 out of 36 eligible published prognostic models for external validation in the population-based AGES-Reykjavik Study. Predictive performance was assessed with c statistics and calibration plots. All five models with a c statistic > .75 (.76-.81) contained cognitive testing as a predictor, while all models with lower c statistics (.67-.75) did not. Calibration ranged from good to poor across all models, including systematic risk overestimation or overestimation for particularly the highest risk group. Models that overestimate risk may be acceptable for exclusion purposes, but lack the ability to accurately identify individuals at higher dementia risk. Both updating existing models or developing new models aimed at identifying high-risk individuals, as well as more external validation studies of dementia prediction models are warranted.

Development and validation of a nomogram for predicting severity in patients with hemorrhagic fever with renal syndrome: A retrospective study

Background

Hemorrhagic fever with renal syndrome (HFRS) is a zoonotic disease caused by hantavirus infection. Patients with severe HFRS may develop multiple organ failure or even death, which makes HFRS a serious public health problem.

Methods

In this retrospective study, we included a total of 155 consecutive patients who were diagnosed with HFRS, of whom 109 patients served as a training cohort and 46 patients as an independent verification cohort. In the training set, the least absolute shrinkage and selection operator (LASSO) regression was used to screen the characteristic variables of the risk model. Multivariate logistic regression analysis was used to construct a nomogram containing the characteristic variables selected in the LASSO regression model.

Results

The area under the receiver operating characteristic curve (AUC) of the nomogram indicated that the model had good discrimination. The calibration curve exhibited that the nomogram was in good agreement between the prediction and the actual observation. Decision curve analysis and clinical impact curve suggested that the predictive nomogram had clinical utility.

Conclusion

In this study, we established a simple and feasible model to predict severity in patients with HFRS, with which HFRS would be better identified and patients can be treated early.

A Predicting Nomogram for Mortality in Patients With COVID-19

Background: The global COVID-19 epidemic remains severe, with the cumulative global death toll reaching more than 207,170 as of May 2, 2020 (1). Purpose: Our research objective is to establish a reliable nomogram to predict mortality in COVID-19 patients. The nomogram can help us distinguish between patients who are at high risk of death and need close attention. Patients and Methods: For the single-center retrospective study, we collected 21 cases of patients who died in the critical illness area of the Optical Valley Branch of Tongji Hospital, Huazhong University of Science and Technology, from February 9 to March 10. Additionally, we selected 99 patients discharged during this period for analysis. The nomogram was constructed to predict the mortality for COVID-19 patients using the primary group of 120 patients and was validated using an independent cohort of 84 patients. We used multivariable logistic regression analysis to construct the prediction model. The nomogram was evaluated for calibration, differentiation, and clinical usefulness. Results: The predictors included in the nomogram were c-reactive protein, PaO2/FiO2, and cTnI. The areas under the curves of the nomogram were 0.988 (95% CI: 0.972-1.000) and 0.956 (95% CI, 0.874-1.000) in the primary and validation groups, respectively. Decision curve analysis suggests that the nomogram may have clinical usefulness. Conclusion: This study provides a nomogram containing c-reactive protein, PaO2/FiO2, and cTnI that can be conveniently used to predict individual mortality in COVID-19 patients. Next, we will collect as many cases as possible from multiple centers to build a more reliable nomogram to predict mortality for COVID-19 patients.

Validation of discrete time-to-event prediction models in the presence of competing risks

Clinical prediction models play a key role in risk stratification, therapy assignment and many other fields of medical decision making. Before they can enter clinical practice, their usefulness has to be demonstrated using systematic validation. Methods to assess their predictive performance have been proposed for continuous, binary, and time-to-event outcomes, but the literature on validation methods for discrete time-to-event models with competing risks is sparse. The present paper tries to fill this gap and proposes new methodology to quantify discrimination, calibration, and prediction error (PE) for discrete time-to-event outcomes in the presence of competing risks. In our case study, the goal was to predict the risk of ventilator-associated pneumonia (VAP) attributed to Pseudomonas aeruginosa in intensive care units (ICUs). Competing events are extubation, death, and VAP due to other bacteria. The aim of this application is to validate complex prediction models developed in previous work on more recently available validation data.