Use of reclassification for assessment of improved prediction: an empirical evaluation

Predicting Heart Failure in Patients with Atrial Fibrillation: A Report from the Prospective COOL-AF Registry

BACKGROUND

This study aimed to determine risk factors and incidence rate and develop a predictive risk model for heart failure for Asian patients with atrial fibrillation (AF).

METHODS

This is a prospective multicenter registry of patients with non-valvular AF in Thailand conducted between 2014 and 2017. The primary outcome was the occurrence of an HF event. A predictive model was developed using a multivariable Cox-proportional model. The predictive model was assessed using C-index, D-statistics, Calibration plot, Brier test, and survival analysis.

RESULTS

There were a total of 3402 patients (average age 67.4 years, 58.2% male) with mean follow-up duration of 25.7 ± 10.6 months. Heart failure occurred in 218 patients during follow-up, representing an incidence rate of 3.03 (2.64-3.46) per 100 person-years. There were ten HF clinical factors in the model. The predictive model developed from these factors had a C-index and D-statistic of 0.756 (95% CI: 0.737-0.775) and 1.503 (95% CI: 1.372-1.634), respectively. The calibration plots showed a good agreement between the predicted and observed model with the calibration slope of 0.838. The internal validation was confirmed using the bootstrap method. The Brier score indicated that the model had a good prediction for HF.

CONCLUSIONS

We provide a validated clinical HF predictive model for patients with AF, with good prediction and discrimination values.

Polygenic risk score improves the accuracy of a clinical risk score for coronary artery disease

BACKGROUND

The value of polygenic risk scores (PRSs) towards improving guideline-recommended clinical risk models for coronary artery disease (CAD) prediction is controversial. Here we examine whether an integrated polygenic risk score improves the prediction of CAD beyond pooled cohort equations.  METHODS: An observation study of 291,305 unrelated White British UK Biobank participants enrolled from 2006 to 2010 was conducted. A case-control sample of 9499 prevalent CAD cases and an equal number of randomly selected controls was used for tuning and integrating of the polygenic risk scores. A separate cohort of 272,307 individuals (with follow-up to 2020) was used to examine the risk prediction performance of pooled cohort equations, integrated polygenic risk score, and PRS-enhanced pooled cohort equation for incident CAD cases. The performance of each model was analyzed by discrimination and risk reclassification using a 7.5% threshold.

RESULTS

In the cohort of 272,307 individuals (mean age, 56.7 years) used to analyze predictive accuracy, there were 7036 incident CAD cases over a 12-year follow-up period. Model discrimination was tested for integrated polygenic risk score, pooled cohort equation, and PRS-enhanced pooled cohort equation with reported C-statistics of 0.640 (95% CI, 0.634-0.646), 0.718 (95% CI, 0.713-0.723), and 0.753 (95% CI, 0.748-0.758), respectively. Risk reclassification for the addition of the integrated polygenic risk score to the pooled cohort equation at a 7.5% risk threshold resulted in a net reclassification improvement of 0.117 (95% CI, 0.102 to 0.129) for cases and - 0.023 (95% CI, - 0.025 to - 0.022) for noncases [overall: 0.093 (95% CI, 0.08 to 0.104)]. For incident CAD cases, this represented 14.2% correctly reclassified to the higher-risk category and 2.6% incorrectly reclassified to the lower-risk category.

CONCLUSIONS

Addition of the integrated polygenic risk score for CAD to the pooled cohort questions improves the predictive accuracy for incident CAD and clinical risk classification in the White British from the UK Biobank. These findings suggest that an integrated polygenic risk score may enhance CAD risk prediction and screening in the White British population.

Preoperative Prediction Nomogram Based on Integrated Profiling for Glioblastoma Multiforme in Glioma Patients

Introduction: Traditional classification that divided gliomas into glioblastoma multiformes (GBM) and lower grade gliomas (LGG) based on pathological morphology has been challenged over the past decade by improvements in molecular stratification, however, the reproducibility and diagnostic accuracy of glioma classification still remains poor. This study aimed to establish and validate a novel nomogram for the preoperative diagnosis of GBM by using integrated data combined with feasible baseline characteristics and preoperative tests. Material and method: The models were established in a primary cohort that included 259 glioma patients who had undergone surgical resection and were pathologically diagnosed from March 2014 to May 2016 in the First Affiliated Hospital of Xi'an Jiaotong University. The preoperative data were used to construct three models by the best subset regression, the forward stepwise regression, and the least absolute shrinkage and selection operator, and to furthermore establish the nomogram among those models. The assessment of nomogram was carried out by the discrimination and calibration in internal cohorts and external cohorts. Results and discussion: Out of all three models, model 2 contained eight clinical-related variables, which exhibited the minimum Akaike Information Criterion (173.71) and maximum concordance index (0.894). Compared with the other two models, the integrated discrimination index for model 2 was significantly improved, indicating that the nomogram obtained from model 2 was the most appropriate model. Likewise, the nomogram showed great calibration and significant clinical benefit according to calibration curves and the decision curve analysis. Conclusion: In conclusion, our study showed a novel preoperative model that incorporated clinically relevant variables and imaging features with laboratory data that could be used for preoperative prediction in glioma patients, thus providing more reliable evidence for surgical decision-making.

Application of net reclassification index to non-nested and point-based risk prediction models: a review

Much of medical risk prediction involves externally derived prediction equations, nomograms, and point-based risk scores. These settings are vulnerable to misleading findings of incremental value based on versions of the net reclassification index (NRI) in common use. By applying non-nested models and point-based risk scores in the setting of stroke risk prediction in patients with atrial fibrillation (AF), we demonstrate current recommendations for presentation and interpretation of the NRI. We emphasize pitfalls that are likely to occur with point-based risk scores that are easy to neglect when statistical methodology is focused on continuous models. In order to make appropriate decisions about risk prediction and personalized medicine, physicians, researchers, and policy makers need to understand the strengths and limitations of the NRI.

A methodology review on the incremental prognostic value of computed tomography biomarkers in addition to Framingham risk score in predicting cardiovascular disease: the use of association, discrimination and reclassification

BACKGROUND

Computed tomography (CT) biomarkers claim to improve cardiovascular risk stratification. This review focuses on significant differences in incremental measures between adequate and inadequate reporting practise.

METHODS

Studies included were those that used Framingham Risk Score as a baseline and described the incremental value of adding calcium score or CT coronary angiogram in predicting cardiovascular risk. Searches of MEDLINE, EMBASE, Web of Science and Cochrane Central were performed with no language restriction.

RESULTS

Thirty five studies consisting of 206,663 patients (men = 118,114, 55.1%) were included. The baseline Framingham Risk Score included the 1998, 2002 and 2008 iterations. Selective reporting, inconsistent reference groupings and thresholds were found. Twelve studies (34.3%) had major and 23 (65.7%) had minor alterations and the respective Δ AUC were significantly different (p = 0.015). When the baseline model performed well, the Δ AUC was relatively lower with the addition of a CT biomarker (Spearman coefficient = - 0.46, p < 0.0001; n = 33; 76 pairs of data). Other factors that influenced AUC performance included exploration of data analysis, calibration, validation, multivariable and AUC documentation (all p < 0.05). Most studies (68.7%) that reported categorical NRI (n = 16; 46 pairs of data) subjectively drew strong conclusions along with other poor reporting practices. However, no significant difference in values of NRI was found between adequate and inadequate reporting.

CONCLUSIONS

The widespread practice of poor reporting particularly association, discrimination, reclassification, calibration and validation undermines the claimed incremental value of CT biomarkers over the Framingham Risk Score alone. Inadequate reporting of discrimination inflates effect estimate, however, that is not necessarily the case for reclassification.

Reclassification calibration test for censored survival data: performance and comparison to goodness-of-fit criteria

BACKGROUND

The risk reclassification table assesses clinical performance of a biomarker in terms of movements across relevant risk categories. The Reclassification-Calibration (RC) statistic has been developed for binary outcomes, but its performance for survival data with moderate to high censoring rates has not been evaluated.

METHODS

We develop an RC statistic for survival data with higher censoring rates using the Greenwood-Nam-D'Agostino approach (RC-GND). We examine its performance characteristics and compare its performance and utility to the Hosmer-Lemeshow goodness-of-fit test under various assumptions about the censoring rate and the shape of the baseline hazard.

RESULTS

The RC-GND test was robust to high (up to 50%) censoring rates and did not exceed the targeted 5% Type I error in a variety of simulated scenarios. It achieved 80% power to detect better calibration with respect to clinical categories when an important predictor with a hazard ratio of at least 1.7 to 2.2 was added to the model, while the Hosmer-Lemeshow goodness of fit (gof) test had power of 5% in this scenario.

CONCLUSIONS

The RC-GND test should be used to test the improvement in calibration with respect to clinically-relevant risk strata. When an important predictor is omitted, the Hosmer-Lemeshow goodness-of-fit test is usually not significant, while the RC-GND test is sensitive to such an omission.

Activated factor VII-antithrombin complex predicts mortality in patients with stable coronary artery disease: a cohort study

BACKGROUND

Plasma concentration of activated factor VII (FVIIa)-antithrombin (AT) complex has been proposed as an indicator of intravascular exposure of tissue factor.

OBJECTIVES

The aims of this observational study were to evaluate (i) FVIIa-AT plasma concentration in subjects with or without coronary artery disease (CAD) and (ii) its association with mortality in a prospective cohort of patients with CAD.

METHODS

FVIIa-AT levels were measured by elisa in 686 subjects with (n = 546) or without (n = 140) angiographically proven CAD. Subjects with acute coronary syndromes and those taking anticoagulant drugs at the time of enrollment were excluded. CAD patients were followed for total and cardiovascular mortality.

RESULTS

There was no difference in FVIIa-AT levels between CAD (84.8 with 95% confidence interval [CI] 80.6-88.2 pmol L(-1) ) and CAD-free subjects (83.9 with 95% CI 76.7-92.8 pmol L(-1) ). Within the CAD population, during a 64-month median follow-up, patients with FVIIa-AT levels higher than the median value at baseline (≥ 79 pmol L(-1) ) had a two-fold greater risk of both total and cardiovascular mortality. Results were confirmed after adjustment for sex, age, the other predictors of mortality (hazard ratio for total mortality: 2.05 with 95% CI 1.22-3.45, hazard ratio for cardiovascular mortality 1.94 with 95% CI 1.01-3.73, with a slight improvement of C-statistic over traditional risk factors), FVIIa levels, drug therapy at discharge, and even patients using all the usual medications for CAD treatment. High FVIIa-AT levels also correlated with increased thrombin generation.

CONCLUSIONS

This preliminary study suggests that plasma concentration of FVIIa-AT is a thrombophilic marker of total and cardiovascular mortality risk in patients with clinically stable CAD.

Simpson's paradox in the integrated discrimination improvement

The integrated discrimination improvement (IDI) is commonly used to compare two risk prediction models; it summarizes the extent a new model increases risk in events and decreases risk in non-events. The IDI averages risks across events and non-events and is therefore susceptible to Simpson's paradox. In some settings, adding a predictive covariate to a well calibrated model results in an overall negative (positive) IDI. However, if stratified by that same covariate, the strata-specific IDIs are positive (negative). Meanwhile, the calibration (observed to expected ratio and Hosmer-Lemeshow Goodness of Fit Test), area under the receiver operating characteristic curve, and Brier score improve overall and by stratum. We ran extensive simulations to investigate the impact of an imbalanced covariate upon metrics (IDI, area under the receiver operating characteristic curve, Brier score, and R²), provide an analytic explanation for the paradox in the IDI, and use an investigative metric, a Weighted IDI, to better understand the paradox. In simulations, all instances of the paradox occurred under stratum-specific mis-calibration, yet there were mis-calibrated settings in which the paradox did not occur. The paradox is illustrated on Cancer Genomics Network data by calculating predictions based on two versions of BRCAPRO, a Mendelian risk prediction model for breast and ovarian cancer. In both simulations and the Cancer Genomics Network data, overall model calibration did not guarantee stratum-level calibration. We conclude that the IDI should only assess model performance among a clinically relevant subset when stratum-level calibration is strictly met and recommend calculating additional metrics to confirm the direction and conclusions of the IDI. Copyright © 2016 John Wiley & Sons, Ltd.

The use of genetic information in the prediction of Type 2 diabetes

Type 2 diabetes (T2D) is a metabolic disorder characterized by high blood glucose levels and elevated risk of cardiovascular events. The progression of T2D can be delayed, or prevented, so early prediction is of high importance. More than 70 genetic loci are associated with T2D risk, raising the possibility of early identification of future cases. Results show that the benefits in discrimination by including genes in current risk models are uncertain. Improvements have been shown in reclassification but are too modest for clinical use. Given the current guidelines for T2D risk assessment and the increasing availability of genotyped individuals, we could soon be able to use genetics, not to quantify risk, but to inform clinicians on those requiring earlier observation.

Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement

CONTEXT

Prediction models are developed to aid health care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed.

OBJECTIVE

The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes.

EVIDENCE ACQUISITION

This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, health care professionals, and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors.

EVIDENCE SYNTHESIS

The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document.

CONCLUSIONS

To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org).

PATIENT SUMMARY

The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes.

Cardiac biomarkers and acute kidney injury after cardiac surgery

OBJECTIVES

To examine the relationship of cardiac biomarkers with postoperative acute kidney injury (AKI) among pediatric patients undergoing cardiac surgery.

METHODS

Data from TRIBE-AKI, a prospective study of children undergoing cardiac surgery, were used to examine the association of cardiac biomarkers (N-type pro-B-type natriuretic peptide, creatine kinase-MB [CK-MB], heart-type fatty acid binding protein [h-FABP], and troponins I and T) with the development of postoperative AKI. Cardiac biomarkers were collected before and 0 to 6 hours after surgery. AKI was defined as a ≥ 50% or 0.3 mg/dL increase in serum creatinine, within 7 days of surgery.

RESULTS

Of the 106 patients included in this study, 55 (52%) developed AKI after cardiac surgery. Patients who developed AKI had higher median levels of pre- and postoperative cardiac biomarkers compared with patients without AKI (all P < .01). Preoperatively, higher levels of CK-MB and h-FABP were associated with increased odds of developing AKI (CK-MB: adjusted odds ratio 4.58, 95% confidence interval [CI] 1.56-13.41; h-FABP: adjusted odds ratio 2.76, 95% CI 1.27-6.03). When combined with clinical models, both preoperative CK-MB and h-FABP provided good discrimination (area under the curve 0.77, 95% CI 0.68-0.87, and 0.78, 95% CI 0.68-0.87, respectively) and improved reclassification indices. Cardiac biomarkers collected postoperatively did not significantly improve the prediction of AKI beyond clinical models.

CONCLUSIONS

Preoperative CK-MB and h-FABP are associated with increased risk of postoperative AKI and provide good discrimination of patients who develop AKI. These biomarkers may be useful for risk stratifying patients undergoing cardiac surgery.

Risk reclassification analysis investigating the added value of fatigue to sickness absence predictions

Background

Prognostic models including age, self-rated health and prior sickness absence (SA) have been found to predict high (≥30) SA days and high (≥3) SA episodes during 1-year follow-up. More predictors of high SA are needed to improve these SA prognostic models. The purpose of this study was to investigate fatigue as new predictor in SA prognostic models by using risk reclassification methods and measures.

Methods

This was a prospective cohort study with 1-year follow-up of 1,137 office workers. Fatigue was measured at baseline with the 20-item checklist individual strength and added to the existing SA prognostic models. SA days and episodes during 1-year follow-up were retrieved from an occupational health service register. The added value of fatigue was investigated with Net Reclassification Index (NRI) and integrated discrimination improvement (IDI) measures.

Results

In total, 579 (51 %) office workers had complete data for analysis. Fatigue was prospectively associated with both high SA days and episodes. The NRI revealed that adding fatigue to the SA days model correctly reclassified workers with high SA days, but incorrectly reclassified workers without high SA days. The IDI indicated no improvement in risk discrimination by the SA days model. Both NRI and IDI showed that the prognostic model predicting high SA episodes did not improve when fatigue was added as predictor variable.

Conclusion

In the present study, fatigue increased false-positive rates which may reduce the cost-effectiveness of interventions for preventing SA.

Transparent reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement

Prediction models are developed to aid health care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, health care professionals, and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors. The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document. To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org).

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement

Prediction models are developed to aid health care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed. The Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, health care professionals, and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors. The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document. To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org).

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement

Prediction models are developed to aid healthcare providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision-making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed. The Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) initiative developed a set of recommendations for the reporting of studies developing, validating or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a web-based survey and revised during a 3-day meeting in June 2011 with methodologists, healthcare professionals and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors. The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study, regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document. To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org).

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement

BACKGROUND

Prediction models are developed to aid healthcare providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision-making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed.

MATERIALS AND METHODS

The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) initiative developed a set of recommendations for the reporting of studies developing, validating or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, healthcare professionals and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors.

RESULTS

The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document.

CONCLUSIONS

To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org).

Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD Statement

BACKGROUND

Prediction models are developed to aid healthcare providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision-making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed.

METHODS

An extensive list of items based on a review of the literature was created, which was reduced after a web-based survey and revised during a 3-day meeting in June 2011 with methodologists, healthcare professionals and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors.

RESULTS

The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study.

CONCLUSION

The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document. A complete checklist is available at http://www.tripod-statement.org.

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement

Prediction models are developed to aid health-care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, health-care professionals, and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors. The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document. To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org).

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. The TRIPOD Group

BACKGROUND

Prediction models are developed to aid health care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed.

METHODS

The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, health care professionals, and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors.

RESULTS

The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document.

CONCLUSIONS

To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org).

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement

Prediction models are developed to aid health care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, health care professionals, and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors. The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document. To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org).

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement

Prediction models are developed to aid health care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, health care professionals, and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors. The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document. To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org).

What to expect from net reclassification improvement with three categories

The net reclassification improvement (NRI) has become a popular measure of incremental usefulness of markers added to risk prediction models. However, the expected magnitude of the three-category NRI is not well understood, leading researchers to rely on statistical significance. In this paper, we describe a slight modification to the original definition of the NRI, which weighs each reclassification by the number of categories by which a given individual is reclassified. This modification resolves some recent criticisms of the three-category NRI and at the same time has a minimal impact on its magnitude. Then we show that using this modified definition, the event and nonevent NRIs have simple interpretations as sums of changes in sensitivities and specificities calculated at the risk thresholds. We exploit this relationship to arrive at closed-form solutions for the NRI under normality within the event and nonevent subgroups. We observe that the size of the intermediate risk category and the event rate have limited impact on the magnitude of the NRI. As expected, the NRI increases with the strength of the added marker, and this relationship appears fairly proportional for markers with non-weak net effect size (above 0.25). Furthermore, we conclude that using the NRI as a metric, it is harder to improve models that already perform well.

Design and analysis of metabolomics studies in epidemiologic research: a primer on -omic technologies

Metabolomics is the field of "-omics" research concerned with the comprehensive characterization of the small low-molecular-weight metabolites in biological samples. In epidemiology, it represents an emerging technology and an unprecedented opportunity to measure environmental and other exposures with improved precision and far less measurement error than with standard epidemiologic methods. Advances in the application of metabolomics in large-scale epidemiologic research are now being realized through a combination of improved sample preparation and handling, automated laboratory and processing methods, and reduction in costs. The number of epidemiologic studies that use metabolic profiling is still limited, but it is fast gaining popularity in this area. In the present article, we present a roadmap for metabolomic analyses in epidemiologic studies and discuss the various challenges these data pose to large-scale studies. We discuss the steps of data preprocessing, univariate and multivariate data analysis, correction for multiplicity of comparisons with correlated data, and finally the steps of cross-validation and external validation. As data from metabolomic studies accumulate in epidemiology, there is a need for large-scale replication and synthesis of findings, increased availability of raw data, and a focus on good study design, all of which will highlight the potential clinical impact of metabolomics in this field.

Bleeding risk prediction models in atrial fibrillation

Novel, nonvitamin K antagonist oral anticoagulants (OACs) have demonstrated similar or superior efficacy to warfarin for ischemic stroke prevention in patients with atrial fibrillation (AF). As the prevalence of AF rises in a growing elderly population, these agents are becoming central to the routine practice of clinicians caring for these patients. Though the benefits are clear, the decision to treat the elderly patient with AF with long-term oral OACs is often a dilemma for the clinician mindful of the risk of major bleeding. Several bleeding risk prediction models have been created to help the clinician identify patients for whom the risk of bleeding is high, and would potentially outweigh the benefits of OAC therapy. In this review, we discuss the features of 8 bleeding risk prediction models, including the recently described HEMORR2HAGES, HAS-BLED, and ATRIA models, and approaches to assessing bleeding risk in clinical practice.

Perioperative cardiac biomarkers: the utility and timing

PURPOSE OF REVIEW

We presume that biomarkers will improve identification of patients at risk, leading to interventions and treatments that reduce perioperative adverse events. Risk stratification is multifactorial, and a biomarker must add information to this process, thereby redistributing patients to either higher or lower risk categories, to improve the allocation of expensive and risky interventions. This review focuses on the utility of three cardiac biomarkers in perioperative management.

RECENT FINDINGS

Using newly defined epidemiologic criteria, three distinct molecules, brain natriuretic peptide (BNP), troponin (cTn), and glycosylated hemoglobin (HbA1c) emerge as potentially useful in perioperative medicine. A meta-analysis shows, in vascular surgery, BNP improves risk stratification. Four articles highlight the utility of postoperative cTn measurements in cases of myocardial injury. These articles show that most injury is not infarction, and they present preliminary evidence of the populations that will benefit from structured surveillance protocols. HbA1c is shown to improve the prediction of mortality, but there are questions whether this risk is modifiable.

SUMMARY

The findings here suggest an expanded role for postoperative cTn surveillance; however, the precise populations that benefit, or the interventions required, are not yet defined. The encouraging data for the other two biomarkers need more investigations before adopting them into routine clinical use.

The accuracy of predicting cardiovascular death based on one compared to several albuminuria values

Albuminuria is a well-documented predictor of cardiovascular (CV) mortality. However, day-to-day variability is substantial, and there is no consensus on the number of urine samples required for risk prediction. To resolve this we followed 9158 adults from the population-based Nord-Trøndelag Health Study for 13 years (Second HUNT Study). The predictive performance of models for CV death based on Framingham variables plus 1 versus 3 albumin-creatinine ratio (ACR) was assessed in participants who provided 3 urine samples. There was no improvement in discrimination, calibration, or reclassification when using ACR as a continuous variable. Difference in Akaike information criterion indicated an uncertain improvement in overall fit for the model with the mean of 3 urine samples. Criterion analyses on dichotomized albuminuria information sustained 1 sample as sufficient for ACR levels down to 1.7 mg/mmol. At lower levels, models with 3 samples had a better overall fit. Likewise, in survival analyses, 1 sample was enough to show a significant association to CV mortality for ACR levels above 1.7 mg/mmol (adjusted hazard ratio 1.37; 95% CI 1.15-1.63). For lower ACR levels, 2 or 3 positive urine samples were needed for significance. Thus, multiple urine sampling did not improve CV death prediction when using ACR as a continuous variable. For cutoff ACR levels of 1.0 mg/mmol or less, additional urine samples were required, and associations were stronger with increasing number of samples.

Predicting risk of mortality in dialysis patients: a retrospective cohort study evaluating the prognostic value of a simple chest X-ray

BACKGROUND

Clinical outcomes of dialysis patients are variable, and improved knowledge of prognosis would inform decisions regarding patient management. We assessed the value of simple, chest X-ray derived measures of cardiac size (cardiothoracic ratio (CTR)) and vascular calcification (Aortic Arch Calcification (AAC)), in predicting death and improving multivariable prognostic models in a prevalent cohort of hemodialysis patients.

METHODS

Eight hundred and twenty-four dialysis patients with one or more postero-anterior (PA) chest X-ray were included in the study. Using a validated calcification score, the AAC was graded from 0 to 3. Cox proportional hazards models were used to assess the association between AAC score, CTR, and mortality. AAC was treated as a categorical variable with 4 levels (0,1,2, or 3). Age, race, diabetes, and heart failure were adjusted for in the multivariable analysis. The criterion for statistical significance was p<0.05.

RESULTS

The median CTR of the sample was 0.53 [IQR=0.48,0.58] with calcification scores as follows: 0 (54%), 1 (24%), 2 (17%), and 3 (5%). Of 824 patients, 152 (18%) died during follow-up. Age, sex, race, duration of dialysis, diabetes, heart failure, ischemic heart disease and baseline serum creatinine and phosphate were included in a base Cox model. Both CTR (HR 1.78[1.40,2.27] per 0.1 unit change), area under the curve (AUC)=0.60[0.55,0.65], and AAC (AAC 3 vs 0 HR 4.35[2.38,7.66], AAC 2 vs 0 HR 2.22[1.41,3.49], AAC 1 vs 0 HR 2.43[1.64,3.61]), AUC=0.63[0.58,0.68]) were associated with death in univariate Cox analysis. CTR remained significant after adjustment for base model variables (adjusted HR 1.46[1.11,1.92]), but did not increase the AUC of the base model (0.71[0.66,0.76] vs. 0.71[0.66,0.76]) and did not improve net reclassification performance (NRI=0). AAC also remained significant on multivariable analysis, but did not improve net reclassification (NRI=0). All ranges were based on 95% confidence intervals.

CONCLUSIONS

Neither CTR nor AAC assessed on chest x-ray improved prediction of mortality in this prevalent cohort of dialysis patients. Our data do not support the clinical utility of X-ray measures of cardiac size and vascular calcification for the purpose of mortality prediction in prevalent hemodialysis patients. More advanced imaging techniques may be needed to improve prognostication in this population.

Predicting risk of type 2 diabetes mellitus with genetic risk models on the basis of established genome-wide association markers: a systematic review

This study aimed to evaluate the predictive performance of genetic risk models based on risk loci identified and/or confirmed in genome-wide association studies for type 2 diabetes mellitus. A systematic literature search was conducted in the PubMed/MEDLINE and EMBASE databases through April 13, 2012, and published data relevant to the prediction of type 2 diabetes based on genome-wide association marker-based risk models (GRMs) were included. Of the 1,234 potentially relevant articles, 21 articles representing 23 studies were eligible for inclusion. The median area under the receiver operating characteristic curve (AUC) among eligible studies was 0.60 (range, 0.55-0.68), which did not differ appreciably by study design, sample size, participants' race/ethnicity, or the number of genetic markers included in the GRMs. In addition, the AUCs for type 2 diabetes did not improve appreciably with the addition of genetic markers into conventional risk factor-based models (median AUC, 0.79 (range, 0.63-0.91) vs. median AUC, 0.78 (range, 0.63-0.90), respectively). A limited number of included studies used reclassification measures and yielded inconsistent results. In conclusion, GRMs showed a low predictive performance for risk of type 2 diabetes, irrespective of study design, participants' race/ethnicity, and the number of genetic markers included. Moreover, the addition of genome-wide association markers into conventional risk models produced little improvement in predictive performance.

Biomarkers in sepsis

There is much enthusiasm and interest in sepsis biomarkers, particularly because sepsis is a highly lethal condition, its diagnosis is challenging, and even simple treatment with antibiotics has led to serious adverse consequences such as emergence of resistant pathogens. Yet development of a sepsis biomarker requires many more steps than simply finding an association between a particular molecule and a clinical state or outcome. Demonstration of improvement of therapeutic practice using receiver-operating characteristic and other analyses is important. Validation in independent, prospective and, preferably, multicenter trials is essential. Many promising candidate sepsis biomarkers have recently been proposed. While procalcitonin (PCT) is currently the most studied sepsis biomarker, evidence of potential value has been found for a wide array of blood biomarkers including proteins, mRNA expression in whole blood or leukocytes, micro-RNAs (miRNA), pathogen and host DNA, pathogen and host genetic variants and metabolomic panels, and even in the novel use of currently available clinical data. While the most common early reports link putative sepsis biomarker levels to severity of illness and outcome (prognostic), this is not anticipated to be their primary use. More important is the distinction between infection and noninfectious inflammatory responses (diagnostic) and the use of sepsis biomarkers to direct therapy (predictive).

Clinical utility of N-terminal pro-B-type natriuretic peptide for risk stratification of patients with acute decompensated heart failure. Derivation and validation of the ADHF/NT-proBNP risk score

BACKGROUND

NT-proBNP has been associated with prognosis in acute decompensated heart failure (ADHF). Whether NT-proBNP provides additional prognostic information beyond that obtained from standard clinical variables is uncertain. We sought to assess whether N-terminal pro-B-type natriuretic peptide (NT-proBNP) determination improves risk reclassification of patients with ADHF and to develop and validate a point-based NT-proBNP risk score.

METHODS

This study included 824 patients with ADHF (453 in the derivation cohort, 371 in the validation cohort). We compared two multivariable models predicting 1-year all-cause mortality, including clinical variables and clinical variables plus NT-proBNP. We calculated the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI). Then, we developed and externally validated the NT-proBNP risk score.

RESULTS

One-year mortalities for the derivation and validation cohorts were 28.3% and 23.4%, respectively. Multivariable predictors of mortality included chronic obstructive pulmonary disease, estimated glomerular filtration rate, sodium, hemoglobin, left ventricular ejection fraction, and moderate to severe tricuspid regurgitation. Adding NT-proBNP to the clinical variables only model significantly improved the NRI (0.129; p=0.0027) and the IDI (0.037; p=0.0005). In the derivation cohort, the NT-proBNP risk score had a C index of 0.839 (95% CI: 0.798-0.880) and the Hosmer-Lemeshow statistic was 1.23 (p=0.542), indicating good calibration. In the validation cohort, the risk score had a C index of 0.768 (95% CI: 0.711-0.817); the Hosmer-Lemeshow statistic was 2.76 (p=0.251), after recalibration.

CONCLUSIONS

The NT-proBNP risk score provides clinicians with a contemporary, accurate, easy-to-use, and validated predictive tool. Further validation in other datasets is advisable.

Assessing improvement in disease prediction using net reclassification improvement: impact of risk cut-offs and number of risk categories

Net reclassification improvement (NRI) has received much attention for comparing risk prediction models, and might be preferable over the area under the receiver operating characteristics (ROC) curve to indicate changes in predictive ability. We investigated the influence of the choice of risk cut-offs and number of risk categories on the NRI. Using data of the European Prospective Investigation into Cancer and Nutrition-Potsdam study, three diabetes prediction models were compared according to ROC area and NRI with varying cut-offs for two and three risk categories and varying numbers of risk categories. When compared with a basic model, including age, anthropometry, and hypertension status, a model extension by waist circumference improved discrimination from 0.720 to 0.831 (0.111 [0.097-0.125]) while increase in ROC-AUC from 0.831 to 0.836 (0.006 [0.002-0.009]) indicated moderate improvement when additionally considering diet and physical activity. However, NRI based on these two model comparisons varied with varying cut-offs for two (range: 5.59-23.20%; -0.79 to 4.09%) and three risk categories (20.37-40.15%; 1.22-4.34%). This variation was more pronounced in the model extension showing a larger difference in ROC-AUC. NRI increased with increasing numbers of categories from minimum NRIs of 18.41 and 0.46% to approximately category-free NRIs of 79.61 and 19.22%, but not monotonically. There was a similar pattern for this increase in both model comparisons. In conclusion, the choice of risk cut-offs and number of categories has a substantial impact on NRI. A limited number of categories should only be used if categories have strong clinical importance.

A minimal net reclassification improvement to assess predictions of intensive care mortality

OBJECTIVE

In assessing the improved discrimination of a new prognostic score, the "Net Reclassification Improvement" from reclassification methods appears of interest. We propose a measure that takes into account improvements in predicted probabilities to assess and allows testing the additional predictive ability of a new scoring system Y compared to a reference score X.

STUDY DESIGN AND SETTINGS

To assess and test the improvement in mortality prediction of (X + Y) compared to X, we defined a minimal net reclassification improvement that restricted improvements in predicted probabilities according to some positive threshold δ. Both absolute and relative improvements were considered. A simulation study was performed to assess its performances in a range of practical situations. We then applied our measures to real intensive care unit data.

RESULTS

Expectedly, minimal net reclassification improvement increased with the effect size of Y and decreased with the value of δ. Using relative improvements allowed erasing the influence of the population mortality. For given effect sizes of X and Y, the difference in all measures of reclassification decreased when a correlation between X and Y was introduced.

CONCLUSION

Reclassification methods, particularly the minimal net reclassification improvement, seem to be clinically relevant when used with continuous clinical data with no known threshold.

Further insight into the incremental value of new markers: the interpretation of performance measures and the importance of clinical context

In this issue of the Journal, Pencina and et al. (Am J Epidemiol. 2012;176(6):492-494) examine the operating characteristics of measures of incremental value. Their goal is to provide benchmarks for the measures that can help identify the most promising markers among multiple candidates. They consider a setting in which new predictors are conditionally independent of established predictors. In the present article, the authors consider more general settings. Their results indicate that some of the conclusions made by Pencina et al. are limited to the specific scenarios the authors considered. For example, Pencina et al. observed that continuous net reclassification improvement was invariant to the strength of the baseline model, but the authors of the present study show this invariance does not hold generally. Further, they disagree with the suggestion that such invariance would be desirable for a measure of incremental value. They also do not see evidence to support the claim that the measures provide complementary information. In addition, they show that correlation with baseline predictors can lead to much bigger gains in performance than the conditional independence scenario studied by Pencina et al. Finally, the authors note that the motivation of providing benchmarks actually reinforces previous observations that the problem with these measures is they do not have useful clinical interpretations. If they did, researchers could use the measures directly and benchmarks would not be needed.

Prediction of vascular risk after stroke - protocol and pilot data of the Prospective Cohort with Incident Stroke (PROSCIS)

RATIONALE

Long-term risk of vascular disease is substantially increased after stroke with several models proposed to predict subsequent stroke and other vascular events after an index event. However, recent validation studies demonstrate limited predictive properties of available prognostic models.

AIMS

We aim to determine prediction models of different complexity for the combined vascular end-point of stroke, myocardial infarction, and vascular death at three-years after first-ever stroke. An independent external validation of the developed models will be performed.

DESIGN

Prospective observational hospital-based cohort study of patients after first-ever stroke.

METHODS

The new predictive models will be developed using the following steps: (1) Development of a basic score based on clinical history data (e.g. hypertension, myocardial infarction, and atrial fibrillation); (2) Development of an advanced score including additional factors such as blood-based biomarkers and results of vascular imaging; (3) Comparing the models fit using different methods (discrimination, calibration); (4) Assessment of clinical utility of an advanced score using methods based on reclassification tables (e.g. net reclassification improvement, integrated discrimination improvement, decision curve analysis); and (5) Investigation of external validity.

OUTCOMES

Primary outcome is a combined vascular end-point composed of stroke, myocardial infarction, and vascular death at three-years after stroke. Furthermore, each component of the composite end-point will be investigated individually and the patterns and time points of risk transitions between vascular end-points and stroke sub-types will be determined.

Reclassification of risk of death with the knowledge of D-dimer in a cohort of treated HIV-infected individuals

OBJECTIVE

To evaluate the change in categories of risk of death by adding D-dimer to conventional mortality risk factors.

DESIGN

Cohort study.

METHODS

Data on HIV-infected participants receiving standard combination antiretroviral therapy in two clinical trials (Evaluation of Subcutaneous Proleukin in a Randomized International Trial and Strategic Management of antiretroviral therapy), who had baseline D-dimer measured, were randomly split into two equal training and a validation datasets. A multivariable survival model was built using the training dataset and included only conventional mortality risk factors measured at baseline. D-dimer was added to create the comparison model. The level of reclassification of mortality risk, for those with at least 5-years of follow-up, was then assessed by tabulating mortality risk defined as low (≤2% predicted rate), moderate (2-5%) or high (>5%). Reclassification analyses were then repeated on the validation dataset.

RESULTS

The analysis population at baseline had a mean age of 43 years, median CD4(+) cell count of 535 cells/μl (IQR: 420-712), and 83% had HIV RNA of at least 500 copies/ml. In the training dataset (n=1946, 8939 person-years), there were 83 deaths at a rate of 0.93 per 100 person-years. Addition of D-dimer to the reference model resulted in 6% or fewer (P>0.05) being correctly reassigned, either up or down, to a new risk category, in both, training and validation datasets. The integrated discrimination improvement in training and validation datasets was 0.60% (P=0.084) and 0.45% (P=0.168), respectively.

CONCLUSION

In this relatively well population, at the given risk cutoffs, D-dimer appeared to only modestly improve the discernment of risk. Risk reclassification provides a method for assessing the clinical utility of biomarkers in HIV cohort studies.

New metrics for assessing diagnostic potential of candidate biomarkers

New tests should improve the diagnostic performance of available tests. The area under the receiver operator characteristic curve has been the "metric of choice" to quantify new biomarker performance. Two new metrics, the integrated discrimination improvement (IDI) and net reclassification improvement (NRI), have been rapidly adopted to quantify the added value of a biomarker to an existing test. These metrics require the development of risk prediction models that calculate the probability of an event for each individual. This study demonstrates the application of these metrics in 528 critically ill patients with risk models of AKI, sepsis, and 30-day mortality to which the biomarker urinary cystatin C was added. Analogous to the receiver operator characteristic curve, we present a new risk assessment plot for visualizing these metrics. The results showed that the NRI was sensitive to the choice of risk threshold. The risk assessment plot identified that the addition of urinary cystatin C to the model decreased the calculated risk for some who did not have sepsis but increased it for others. The category-free NRI for each outcome indicated that most of those without the event had reduced calculated risk. This was driven by very small changes in calculated risk in the AKI and death models. The IDI reflected those small changes. Of the new metrics, the IDI, reported separately for those with and without the events, best represents the value of a new test. The risk assessment plot identified differences in the models not apparent in any of the metrics.

Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: meta-epidemiology study

OBJECTIVE

To compare the reported effect sizes of cardiovascular biomarkers in datasets from observational studies with those in datasets from randomised controlled trials.

DESIGN

Review of meta-analyses.

STUDY SELECTION

Meta-analyses of emerging cardiovascular biomarkers (not part of the Framingham risk score) that included datasets from at least one observational study and at least one randomised controlled trial were identified through Medline (last update, January 2011).

DATA EXTRACTION

Study-specific risk ratios were extracted from all identified meta-analyses and synthesised with random effects for (a) all studies, and (b) separately for observational and for randomised controlled trial populations for comparison.

RESULTS

31 eligible meta-analyses were identified. For seven major biomarkers (C reactive protein, non-HDL cholesterol, lipoprotein(a), post-load glucose, fibrinogen, B-type natriuretic peptide, and troponins), the prognostic effect was significantly stronger in datasets from observational studies than in datasets from randomised controlled trials. For five of the biomarkers the effect was less than half as strong in the randomised controlled trial datasets. Across all 31 meta-analyses, on average datasets from observational studies suggested larger prognostic effects than those from randomised controlled trials; from a random effects meta-analysis, the estimated average difference in the effect size was 24% (95% CI 7% to 40%) of the overall biomarker effect.

CONCLUSIONS

Cardiovascular biomarkers often have less promising results in the evidence derived from randomised controlled trials than from observational studies.