Amino-Terminal Pro-B-Type Natriuretic Peptide for Risk Prediction in Acute Decompensated Heart Failure

Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

Background:

While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood.

Methods:

We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed.

Results:

Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73–0.78]) and validation (0.64 [interquartile range 0.60–0.67], P<0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59–0.90]) than distantly related cohorts (0.59 [interquartile range 0.43–0.73], P=0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation.

Conclusions:

There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.

Clinical utility of N-terminal pro-B-type natriuretic peptide for risk stratification of patients with acute decompensated heart failure. Derivation and validation of the ADHF/NT-proBNP risk score

BACKGROUND

NT-proBNP has been associated with prognosis in acute decompensated heart failure (ADHF). Whether NT-proBNP provides additional prognostic information beyond that obtained from standard clinical variables is uncertain. We sought to assess whether N-terminal pro-B-type natriuretic peptide (NT-proBNP) determination improves risk reclassification of patients with ADHF and to develop and validate a point-based NT-proBNP risk score.

METHODS

This study included 824 patients with ADHF (453 in the derivation cohort, 371 in the validation cohort). We compared two multivariable models predicting 1-year all-cause mortality, including clinical variables and clinical variables plus NT-proBNP. We calculated the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI). Then, we developed and externally validated the NT-proBNP risk score.

RESULTS

One-year mortalities for the derivation and validation cohorts were 28.3% and 23.4%, respectively. Multivariable predictors of mortality included chronic obstructive pulmonary disease, estimated glomerular filtration rate, sodium, hemoglobin, left ventricular ejection fraction, and moderate to severe tricuspid regurgitation. Adding NT-proBNP to the clinical variables only model significantly improved the NRI (0.129; p=0.0027) and the IDI (0.037; p=0.0005). In the derivation cohort, the NT-proBNP risk score had a C index of 0.839 (95% CI: 0.798-0.880) and the Hosmer-Lemeshow statistic was 1.23 (p=0.542), indicating good calibration. In the validation cohort, the risk score had a C index of 0.768 (95% CI: 0.711-0.817); the Hosmer-Lemeshow statistic was 2.76 (p=0.251), after recalibration.

CONCLUSIONS

The NT-proBNP risk score provides clinicians with a contemporary, accurate, easy-to-use, and validated predictive tool. Further validation in other datasets is advisable.