Accounting for treatment use when validating a prognostic model: a simulation study

Prediction Under Interventions: Evaluation of Counterfactual Performance Using Longitudinal Observational Data

Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision-making. However, evaluating the predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different from those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting that involves creating a validation dataset mimicking the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.

Performance of the ACC/AHA Pooled Cohort Cardiovascular Risk Equations in Clinical Practice

BACKGROUND

The performance of the American College of Cardiology/American Heart Association pooled cohort equation (PCE) for atherosclerotic cardiovascular disease (ASCVD) in real-world clinical practice has not been evaluated extensively.

OBJECTIVES

The goal of this study was to test the performance of PCE to predict ASCVD risk in the community, and determine if including individuals with values outside the PCE range (ie, age, blood pressure, cholesterol) or statin therapy initiation over follow-up would significantly affect PCE predictive capabilities.

METHODS

The PCE was validated in a community-based cohort of consecutive patients who sought primary care in Olmsted County, Minnesota, between 1997 and 2000, followed-up through 2016. Inclusion criteria were similar to those of PCE derivation. Patient information was ascertained by using the record linkage system of the Rochester Epidemiology Project. ASCVD events (nonfatal and fatal myocardial infarction and ischemic stroke) were validated in duplicate. Calculated and observed ASCVD risk and c-statistics were compared across predefined groups.

RESULTS

This study included 30,042 adults, with a mean age of 48.5 ± 12.2 years; 46% were male. Median follow-up was 16.5 years, truncated at 10 years for this analysis. Mean ASCVD risk was 5.6% ± 8.73%. There were 1,555 ASCVD events (5.2%). The PCE revealed good performance overall (c-statistic 0.78) and in sex and race subgroups; it was highest among non-White female subjects (c-statistic 0.81) and lowest in White male subjects (c-statistic 0.77). Out-of-range values and initiation of statin medication did not affect model performance.

CONCLUSIONS

The PCE performed well in a community cohort representing real-world clinical practice. Values outside PCE ranges and initiation of statin medication did not affect performance. These results have implications for the applicability of current strategies for the prevention of ASCVD.

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

How to conduct a systematic review and meta-analysis of prognostic model studies

BACKGROUND

Prognostic models are typically developed to estimate the risk that an individual in a particular health state will develop a particular health outcome, to support (shared) decision making. Systematic reviews of prognostic model studies can help identify prognostic models that need to further be validated or are ready to be implemented in healthcare.

OBJECTIVES

To provide a step-by-step guidance on how to conduct and read a systematic review of prognostic model studies and to provide an overview of methodology and guidance available for every step of the review progress.

SOURCES

Published, peer-reviewed guidance articles.

CONTENT

We describe the following steps for conducting a systematic review of prognosis studies: 1) Developing the review question using the Population, Index model, Comparator model, Outcome(s), Timing, Setting format, 2) Searching and selection of articles, 3) Data extraction using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist, 4) Quality and risk of bias assessment using the Prediction model Risk Of Bias ASsessment (PROBAST) tool, 5) Analysing data and undertaking quantitative meta-analysis, and 6) Presenting summary of findings, interpreting results, and drawing conclusions. Guidance for each step is described and illustrated using a case study on prognostic models for patients with COVID-19.

IMPLICATIONS

Guidance for conducting a systematic review of prognosis studies is available, but the implications of these reviews for clinical practice and further research highly depend on complete reporting of primary studies.

Comprehensive comparison of stroke risk score performance: a systematic review and meta-analysis among 6 267 728 patients with atrial fibrillation

AIMS

Multiple risk scores to predict ischaemic stroke (IS) in patients with atrial fibrillation (AF) have been developed. This study aims to systematically review these scores, their validations and updates, assess their methodological quality, and calculate pooled estimates of the predictive performance.

METHODS AND RESULTS

We searched PubMed and Web of Science for studies developing, validating, or updating risk scores for IS in AF patients. Methodological quality was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). To assess discrimination, pooled c-statistics were calculated using random-effects meta-analysis. We identified 19 scores, which were validated and updated once or more in 70 and 40 studies, respectively, including 329 validations and 76 updates-nearly all on the CHA2DS2-VASc and CHADS2. Pooled c-statistics were calculated among 6 267 728 patients and 359 373 events of IS. For the CHA2DS2-VASc and CHADS2, pooled c-statistics were 0.644 [95% confidence interval (CI) 0.635-0.653] and 0.658 (0.644-0.672), respectively. Better discriminatory abilities were found in the newer risk scores, with the modified-CHADS2 demonstrating the best discrimination [c-statistic 0.715 (0.674-0.754)]. Updates were found for the CHA2DS2-VASc and CHADS2 only, showing improved discrimination. Calibration was reasonable but available for only 17 studies. The PROBAST indicated a risk of methodological bias in all studies.

CONCLUSION

Nineteen risk scores and 76 updates are available to predict IS in patients with AF. The guideline-endorsed CHA2DS2-VASc shows inferior discriminative abilities compared with newer scores. Additional external validations and data on calibration are required before considering the newer scores in clinical practice.

CLINICAL TRIAL REGISTRATION

ID CRD4202161247 (PROSPERO).

Validation of Cardiovascular Risk Prediction by the Arriba Instrument

BACKGROUND

It is recommended in cardiovascular prevention guidelines that treatment should be based on overall cardiovascular risk. The arriba instrument has been widely used for this purpose in Germany. The aim of this study is to validate risk prediction by arriba with the aid of morbidity and mortality data from the population- based Study of Health in Pomerania.

METHODS

In a longitudinal analysis, the arriba instrument was used to calculate the 10-year overall cardiovascular risk at baseline for subjects who had not sustained any prior cardiovascular event. Cardiovascular event rates were determined from follow-up data, and discrimination and calibration measures for the risk determination algorithm were calculated.

RESULTS

Data from 1973 subjects (mean age 51 ± 13 years, 48% men) were included in the analysis. After a median follow-up of 10.9 years, cardiovascular events had occurred in 196 subjects, or 9.8%. The ratio of predicted to observed event rate was 0.8 (95% confidence interval: [0.5; 1.1]), 1.3 [1.0; 1.8], and 1.1 [0.8; 1.4] for subjects at low, intermediate, and high cardiovascular risk, respectively. Arriba underestimated cardiovascular event rates in women and overestimated them in persons aged 30-44 and 45-59. The area under curve was 0.84 [95% CI 0.81; 0.86].

CONCLUSION

The discrimination scores of the arriba instrument resemble those of SCORE-Germany and PROCAM, but a better adjustment to the target population would be desirable. The results support the recommendation of the German Guideline for Cardiovascular Risk Counseling in General Practice for the use of the arriba instrument. An unresolved problem is the failure to consider intervention effects, resulting in an overall mild overestimation of risk.

Predicting counterfactual risks under hypothetical treatment strategies: an application to HIV

The accuracy of a prediction algorithm depends on contextual factors that may vary across deployment settings. To address this inherent limitation of prediction, we propose an approach to counterfactual prediction based on the g-formula to predict risk across populations that differ in their distribution of treatment strategies. We apply this to predict 5-year risk of mortality among persons receiving care for HIV in the U.S. Veterans Health Administration under different hypothetical treatment strategies. First, we implement a conventional approach to develop a prediction algorithm in the observed data and show how the algorithm may fail when transported to new populations with different treatment strategies. Second, we generate counterfactual data under different treatment strategies and use it to assess the robustness of the original algorithm's performance to these differences and to develop counterfactual prediction algorithms. We discuss how estimating counterfactual risks under a particular treatment strategy is more challenging than conventional prediction as it requires the same data, methods, and unverifiable assumptions as causal inference. However, this may be required when the alternative assumption of constant treatment patterns across deployment settings is unlikely to hold and new data is not yet available to retrain the algorithm.

Development of a clinical prediction model for perinatal deaths in low resource settings

BACKGROUND

Most pregnancy-related deaths in low and middle income countries occur around the time of birth and are avoidable with timely care. This study aimed to develop a prognostic model to identify women at risk of intrapartum-related perinatal deaths in low-resourced settings, by (1) external validation of an existing prediction model, and subsequently (2) development of a novel model.

METHODS

A prospective cohort study was conducted among pregnant women who presented consecutively for delivery at the maternity unit of Zanzibar's tertiary hospital, Mnazi Mmoja Hospital, the Republic of Tanzania between October 2017 and May 2018. Candidate predictors of perinatal deaths included maternal and foetal characteristics obtained from routine history and physical examination at the time of admission to the labour ward. The outcomes were intrapartum stillbirths and neonatal death before hospital discharge. An existing stillbirth prediction model with six predictors from Nigeria was applied to the Zanzibar cohort to assess its discrimination and calibration performance. Subsequently, a new prediction model was developed using multivariable logistic regression. Model performance was evaluated through internal validation and corrected for overfitting using bootstrapping methods.

FINDINGS

5747 mother-baby pairs were analysed. The existing model showed poor discrimination performance (c-statistic 0·57). The new model included 15 clinical predictors and showed promising discriminative and calibration performance after internal validation (optimism adjusted c-statistic of 0·78, optimism adjusted calibration slope =0·94).

INTERPRETATION

The new model consisted of predictors easily obtained through history-taking and physical examination at the time of admission to the labour ward. It had good performance in predicting risk of perinatal death in women admitted in labour wards. Therefore, it has the potential to assist skilled birth attendance to triage women for appropriate management during labour. Before routine implementation, external validation and usefulness should be determined in future studies.

FUNDING

The study received funding from Laerdal Foundation, Otto Kranendonk Fund and UMC Global Health Fellowship. TD acknowledges financial support from the Netherlands Organisation for Health Research and Development (grant 91617050).

Invited Commentary: Treatment Drop-in-Making the Case for Causal Prediction

Clinical prediction models (CPMs) are often used to guide treatment initiation, with individuals at high risk offered treatment. This implicitly assumes that the probability quoted from a CPM represents the risk to an individual of an adverse outcome in absence of treatment. However, for a CPM to correctly target this estimand requires careful causal thinking. One problem that needs to be overcome is treatment drop-in: where individuals in the development data commence treatment after the time of prediction but before the outcome occurs. In this issue of the Journal, Xu et al. (Am J Epidemiol. 2021;190(10):2000-2014) use causal estimates from external data sources, such as clinical trials, to adjust CPMs for treatment drop-in. This represents a pragmatic and promising approach to address this issue, and it illustrates the value of utilizing causal inference in prediction. Building causality into the prediction pipeline can also bring other benefits. These include the ability to make and compare hypothetical predictions under different interventions, to make CPMs more explainable and transparent, and to improve model generalizability. Enriching CPMs with causal inference therefore has the potential to add considerable value to the role of prediction in healthcare.

Prediction of Cardiovascular Disease Risk Accounting for Future Initiation of Statin Treatment

Cardiovascular disease (CVD) risk-prediction models are used to identify high-risk individuals and guide statin initiation. However, these models are usually derived from individuals who might initiate statins during follow-up. We present a simple approach to address statin initiation to predict “statin-naive” CVD risk. We analyzed primary care data (2004–2017) from the UK Clinical Practice Research Datalink for 1,678,727 individuals (aged 40–85 years) without CVD or statin treatment history at study entry. We derived age- and sex-specific prediction models including conventional risk factors and a time-dependent effect of statin initiation constrained to 25% risk reduction (from trial results). We compared predictive performance and measures of public-health impact (e.g., number needed to screen to prevent 1 event) against models ignoring statin initiation. During a median follow-up of 8.9 years, 103,163 individuals developed CVD. In models accounting for (versus ignoring) statin initiation, 10-year CVD risk predictions were slightly higher; predictive performance was moderately improved. However, few individuals were reclassified to a high-risk threshold, resulting in negligible improvements in number needed to screen to prevent 1 event. In conclusion, incorporating statin effects from trial results into risk-prediction models enables statin-naive CVD risk estimation and provides moderate gains in predictive ability but had a limited impact on treatment decision-making under current guidelines in this population.

The development and validation of a prognostic model to PREDICT Relapse of depression in adult patients in primary care: protocol for the PREDICTR study

BACKGROUND

Most patients who present with depression are treated in primary care by general practitioners (GPs). Relapse of depression is common (at least 50% of patients treated for depression will relapse after a single episode) and leads to considerable morbidity and decreased quality of life for patients. The majority of patients will relapse within 6 months, and those with a history of relapse are more likely to relapse in the future than those with no such history. GPs see a largely undifferentiated case-mix of patients, and once patients with depression reach remission, there is limited guidance to help GPs stratify patients according to risk of relapse. We aim to develop a prognostic model to predict an individual's risk of relapse within 6-8 months of entering remission. The long-term objective is to inform the clinical management of depression after the acute phase.

METHODS

We will develop a prognostic model using secondary analysis of individual participant data drawn from seven RCTs and one longitudinal cohort study in primary or community care settings. We will use logistic regression to predict the outcome of relapse of depression within 6-8 months. We plan to include the following established relapse predictors in the model: residual depressive symptoms, number of previous depressive episodes, co-morbid anxiety and severity of index episode. We will use a "full model" development approach, including all available predictors. Performance statistics (optimism-adjusted C-statistic, calibration-in-the-large, calibration slope) and calibration plots (with smoothed calibration curves) will be calculated. Generalisability of predictive performance will be assessed through internal-external cross-validation. Clinical utility will be explored through net benefit analysis.

DISCUSSION

We will derive a statistical model to predict relapse of depression in remitted depressed patients in primary care. Assuming the model has sufficient predictive performance, we outline the next steps including independent external validation and further assessment of clinical utility and impact.

STUDY REGISTRATION

ClinicalTrials.gov ID: NCT04666662.

Retrospective study on the possible existence of a treatment paradox in sepsis scores in the emergency department

OBJECTIVE

The quick Sequential Organ Failure Assessment (qSOFA) is developed as a tool to identify patients with infection with increased risk of dying from sepsis in non-intensive care unit settings, like the emergency department (ED). An abnormal score may trigger the initiation of appropriate therapy to reduce that risk. This study assesses the risk of a treatment paradox: the effect of a strong predictor for mortality will be reduced if that predictor also acts as a trigger for initiating treatment to prevent mortality.

DESIGN

Retrospective analysis on data from a large observational cohort.

SETTING

ED of a tertiary medical centre in the Netherlands.

PARTICIPANTS

3178 consecutive patients with suspected infection.

PRIMARY OUTCOME

To evaluate the existence of a treatment paradox by determining the influence of baseline qSOFA on treatment decisions within the first 24 hours after admission.

RESULTS

226 (7.1%) had a qSOFA ≥2, of which 51 (22.6%) died within 30 days. Area under receiver operating characteristics of qSOFA for 30-day mortality was 0.68 (95% CI 0.61 to 0.75). Patients with a qSOFA ≥2 had higher odds of receiving any form of intensive therapy (OR 11.4 (95% CI 7.5 to 17.1)), such as aggressive fluid resuscitation (OR 8.8 95% CI 6.6 to 11.8), fast antibiotic administration (OR 8.5, 95% CI 5.7 to 12.3) or vasopressic therapy (OR 17.3, 95% CI 11.2 to 26.8), compared with patients with qSOFA <2.

CONCLUSION

In ED patients with suspected infection, a qSOFA ≥2 was associated with more intensive treatment. This could lead to inadequate prediction of 30-day mortality due to the presence of a treatment paradox.

TRIAL REGISTRATION NUMBER

6916.

A scoping review of causal methods enabling predictions under hypothetical interventions

BACKGROUND

The methods with which prediction models are usually developed mean that neither the parameters nor the predictions should be interpreted causally. For many applications, this is perfectly acceptable. However, when prediction models are used to support decision making, there is often a need for predicting outcomes under hypothetical interventions.

AIMS

We aimed to identify published methods for developing and validating prediction models that enable risk estimation of outcomes under hypothetical interventions, utilizing causal inference. We aimed to identify the main methodological approaches, their underlying assumptions, targeted estimands, and potential pitfalls and challenges with using the method. Finally, we aimed to highlight unresolved methodological challenges.

METHODS

We systematically reviewed literature published by December 2019, considering papers in the health domain that used causal considerations to enable prediction models to be used for predictions under hypothetical interventions. We included both methodologies proposed in statistical/machine learning literature and methodologies used in applied studies.

RESULTS

We identified 4919 papers through database searches and a further 115 papers through manual searches. Of these, 87 papers were retained for full-text screening, of which 13 were selected for inclusion. We found papers from both the statistical and the machine learning literature. Most of the identified methods for causal inference from observational data were based on marginal structural models and g-estimation.

CONCLUSIONS

There exist two broad methodological approaches for allowing prediction under hypothetical intervention into clinical prediction models: (1) enriching prediction models derived from observational studies with estimated causal effects from clinical trials and meta-analyses and (2) estimating prediction models and causal effects directly from observational data. These methods require extending to dynamic treatment regimes, and consideration of multiple interventions to operationalise a clinical decision support system. Techniques for validating 'causal prediction models' are still in their infancy.

Prediction meets causal inference: the role of treatment in clinical prediction models

In this paper we study approaches for dealing with treatment when developing a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a 'predictimand' framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference.

Effect of tailoring anticoagulant treatment duration by applying a recurrence risk prediction model in patients with venous thromboembolism compared to usual care: A randomized controlled trial

BACKGROUND

Patients with unprovoked (i.e., without the presence of apparent transient risk factors such as recent surgery) venous thromboembolism (VTE) are at risk of recurrence if anticoagulants are stopped after 3-6 months, yet their risk remains heterogeneous. Thus, prolonging anticoagulant treatment should be considered in high-risk patients, whereas stopping is likely preferred in those with a low predicted risk. The Vienna Prediction Model (VPM) could aid clinicians in estimating this risk, yet its clinical effects and external validity are currently unknown. The aim of this study was to investigate the clinical impact of this model on reducing recurrence risk in patients with unprovoked VTE, compared to usual care.

METHODS AND FINDINGS

In a randomized controlled trial, the decision to prolong or stop anticoagulant treatment was guided by predicted recurrence risk using the VPM (n = 441), which was compared with usual care (n = 442). Patients with unprovoked VTE were recruited from local thrombosis services in the Netherlands (in Utrecht, Harderwijk, Ede, Amersfoort, Zwolle, Hilversum, Rotterdam, Deventer, and Enschede) between 22 July 2011 and 30 November 2015, with 24-month follow-up complete for all patients by early 2018. The primary outcome was recurrent VTE during 24 months of follow-up. Secondary outcomes included major bleeding and clinically relevant non-major (CRNM) bleeding. In the total study population of 883 patients, mean age was 55 years, and 507 (57.4%) were men. A total of 96 recurrent VTE events (10.9%) were observed, 46 in the intervention arm and 50 in the control arm (risk ratio 0.92, 95% CI 0.63-1.35, p = 0.67). Major bleeding occurred in 4 patients, 2 in each treatment arm, whereas CRNM bleeding occurred in 20 patients (12 in intervention arm versus 8 in control arm). The VPM showed good discriminative power (c-statistic 0.76, 95% CI 0.69-0.83) and moderate to good calibration, notably at the lower spectrum of predicted risk. For instance, in 284 patients with a predicted risk of >2% to 4%, the observed rate of recurrence was 2.5% (95% CI 0.7% to 4.3%). The main limitation of this study is that it did not enroll the preplanned number of 750 patients in each study arm due to declining recruitment rate.

CONCLUSIONS

Our results show that application of the VPM in all patients with unprovoked VTE is unlikely to reduce overall recurrence risk. Yet, in those with a low predicted risk of recurrence, the observed rate was also low, suggesting that it might be safe to stop anticoagulant treatment in these patients.

TRIAL REGISTRATION

Netherlands Trial Register NTR2680.

Predictive value of traditional risk factors for cardiovascular disease in older people: A systematic review

With increasing age, associations between traditional risk factors (TRFs) and cardiovascular disease (CVD) shift. It is unknown which mid-life risk factors remain relevant predictors for CVD in older people. We systematically searched PubMed and EMBASE on August 16th 2019 for studies assessing predictive ability of >1 of fourteen TRFs for fatal and non-fatal CVD, in the general population aged 60+. We included 12 studies, comprising 11 unique cohorts. TRF were evaluated in 2 to 11 cohorts, and retained in 0-70% of the cohorts: age (70%), diabetes (64%), male sex (57%), systolic blood pressure (SBP) (50%), smoking (36%), high-density lipoprotein cholesterol (HDL) (33%), left ventricular hypertrophy (LVH) (33%), total cholesterol (22%), diastolic blood pressure (20%), antihypertensive medication use (AHM) (20%), body mass index (BMI) (0%), hypertension (0%), low-density lipoprotein cholesterol (0%). In studies with low to moderate risk of bias, systolic blood pressure (SBP) (80%), smoking (80%) and HDL cholesterol (60%) were more often retained. Model performance was moderate with C-statistics ranging from 0.61 to 0.77. Compared to middle-aged adults, in people aged 60+ different risk factors predict CVD and current prediction models perform only moderate at best. According to most studies, age, sex and diabetes seem valuable predictors of CVD in old-age. SBP, HDL cholesterol and smoking may also have predictive value. Other blood pressure and cholesterol related variables, BMI, and LVH seem of very limited or no additional value. Without competing risk analysis, predictors are overestimated.

Informative missingness in electronic health record systems: the curse of knowing

Electronic health records provide a potentially valuable data source of information for developing clinical prediction models. However, missing data are common in routinely collected health data and often missingness is informative. Informative missingness can be incorporated in a clinical prediction model, for example by including a separate category of a predictor variable that has missing values. The predictive performance of such a model depends on the transportability of the missing data mechanism, which may be compromised once the model is deployed in practice and the predictive value of certain variables becomes known. Using synthetic data, this phenomenon is explained and illustrated.

Multidimensional screening for predicting pain problems in adults: a systematic review of screening tools and validation studies

Screening tools allowing to predict poor pain outcomes are widely used. Often these screening tools contain psychosocial risk factors. This review (1) identifies multidimensional screening tools that include psychosocial risk factors for the development or maintenance of pain, pain-related distress, and pain-related disability across pain problems in adults, (2) evaluates the quality of the validation studies using Prediction model Risk Of Bias ASsessment Tool (PROBAST), and (3) synthesizes methodological concerns. We identified 32 articles, across 42 study samples, validating 7 screening tools. All tools were developed in the context of musculoskeletal pain, most often back pain, and aimed to predict the maintenance of pain or pain-related disability, not pain-related distress. Although more recent studies design, conduct, analyze, and report according to best practices in prognosis research, risk of bias was most often moderate. Common methodological concerns were identified, related to participant selection (eg, mixed populations), predictors (eg, predictors were administered differently to predictors in the development study), outcomes (eg, overlap between predictors and outcomes), sample size and participant flow (eg, unknown or inappropriate handling of missing data), and analysis (eg, wide variety of performance measures). Recommendations for future research are provided.

A cardiovascular risk prediction model for older people: Development and validation in a primary care population

Cardiovascular risk prediction is mainly based on traditional risk factors that have been validated in middle‐aged populations. However, associations between these risk factors and cardiovascular disease (CVD) attenuate with increasing age. Therefore, for older people the authors developed and internally validated risk prediction models for fatal and non‐fatal CVD, (re)evaluated the predictive value of traditional and new factors, and assessed the impact of competing risks of non‐cardiovascular death. Post hoc analyses of 1811 persons aged 70‐78 year and free from CVD at baseline from the preDIVA study (Prevention of Dementia by Intensive Vascular care, 2006‐2015), a primary care‐based trial that included persons free from dementia and conditions likely to hinder successful long‐term follow‐up, were performed. In 2017‐2018, Cox‐regression analyses were performed for a model including seven traditional risk factors only, and a model to assess incremental predictive ability of the traditional and eleven new factors. Analyses were repeated accounting for competing risk of death, using Fine‐Gray models. During an average of 6.2 years of follow‐up, 277 CVD events occurred. Age, sex, smoking, and type 2 diabetes mellitus were traditional predictors for CVD, whereas total cholesterol, HDL‐cholesterol, and systolic blood pressure (SBP) were not. Of the eleven new factors, polypharmacy and apathy symptoms were predictors. Discrimination was moderate (concordance statistic 0.65). Accounting for competing risks resulted in slightly smaller predicted absolute risks. In conclusion, we found, SBP, HDL, and total cholesterol no longer predict CVD in older adults, whereas polypharmacy and apathy symptoms are two new relevant predictors. Building on the selected risk factors in this study may improve CVD prediction in older adults and facilitate targeting preventive interventions to those at high risk.

Performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis

BACKGROUND

The Framingham risk models and pooled cohort equations (PCE) are widely used and advocated in guidelines for predicting 10-year risk of developing coronary heart disease (CHD) and cardiovascular disease (CVD) in the general population. Over the past few decades, these models have been extensively validated within different populations, which provided mounting evidence that local tailoring is often necessary to obtain accurate predictions. The objective is to systematically review and summarize the predictive performance of three widely advocated cardiovascular risk prediction models (Framingham Wilson 1998, Framingham ATP III 2002 and PCE 2013) in men and women separately, to assess the generalizability of performance across different subgroups and geographical regions, and to determine sources of heterogeneity in the findings across studies.

METHODS

A search was performed in October 2017 to identify studies investigating the predictive performance of the aforementioned models. Studies were included if they externally validated one or more of the original models in the general population for the same outcome as the original model. We assessed risk of bias for each validation and extracted data on population characteristics and model performance. Performance estimates (observed versus expected (OE) ratio and c-statistic) were summarized using a random effects models and sources of heterogeneity were explored with meta-regression.

RESULTS

The search identified 1585 studies, of which 38 were included, describing a total of 112 external validations. Results indicate that, on average, all models overestimate the 10-year risk of CHD and CVD (pooled OE ratio ranged from 0.58 (95% CI 0.43-0.73; Wilson men) to 0.79 (95% CI 0.60-0.97; ATP III women)). Overestimation was most pronounced for high-risk individuals and European populations. Further, discriminative performance was better in women for all models. There was considerable heterogeneity in the c-statistic between studies, likely due to differences in population characteristics.

CONCLUSIONS

The Framingham Wilson, ATP III and PCE discriminate comparably well but all overestimate the risk of developing CVD, especially in higher risk populations. Because the extent of miscalibration substantially varied across settings, we highly recommend that researchers further explore reasons for overprediction and that the models be updated for specific populations.

Using marginal structural models to adjust for treatment drop-in when developing clinical prediction models

Clinical prediction models (CPMs) can inform decision making about treatment initiation, which requires predicted risks assuming no treatment is given. However, this is challenging since CPMs are usually derived using data sets where patients received treatment, often initiated postbaseline as "treatment drop-ins." This study proposes the use of marginal structural models (MSMs) to adjust for treatment drop-in. We illustrate the use of MSMs in the CPM framework through simulation studies that represent randomized controlled trials and real-world observational data and the example of statin initiation for cardiovascular disease prevention. The simulations include a binary treatment and a covariate, each recorded at two timepoints and having a prognostic effect on a binary outcome. The bias in predicted risk was examined in a model ignoring treatment, a model fitted on treatment-naïve patients (at baseline), a model including baseline treatment, and the MSM. In all simulation scenarios, all models except the MSM underestimated the risk of outcome given absence of treatment. These results were supported in the statin initiation example, which showed that ignoring statin initiation postbaseline resulted in models that significantly underestimated the risk of a cardiovascular disease event occurring within 10 years. Consequently, CPMs that do not acknowledge treatment drop-in can lead to underallocation of treatment. In conclusion, when developing CPMs to predict treatment-naïve risk, researchers should consider using MSMs to adjust for treatment drop-in, and also seek to exploit the ability of MSMs to allow estimation of individual treatment effects.

Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation

Introduction

Prediction models may contribute to personalized risk‐based management of women at high risk of spontaneous preterm delivery. Although prediction models are published frequently, often with promising results, external validation generally is lacking. We performed a systematic review of prediction models for the risk of spontaneous preterm birth based on routine clinical parameters. Additionally, we externally validated and evaluated the clinical potential of the models.

Material and methods

Prediction models based on routinely collected maternal parameters obtainable during first 16 weeks of gestation were eligible for selection. Risk of bias was assessed according to the CHARMS guidelines. We validated the selected models in a Dutch multicenter prospective cohort study comprising 2614 unselected pregnant women. Information on predictors was obtained by a web‐based questionnaire. Predictive performance of the models was quantified by the area under the receiver operating characteristic curve (AUC) and calibration plots for the outcomes spontaneous preterm birth <37 weeks and <34 weeks of gestation. Clinical value was evaluated by means of decision curve analysis and calculating classification accuracy for different risk thresholds.

Results

Four studies describing five prediction models fulfilled the eligibility criteria. Risk of bias assessment revealed a moderate to high risk of bias in three studies. The AUC of the models ranged from 0.54 to 0.67 and from 0.56 to 0.70 for the outcomes spontaneous preterm birth <37 weeks and <34 weeks of gestation, respectively. A subanalysis showed that the models discriminated poorly (AUC 0.51–0.56) for nulliparous women. Although we recalibrated the models, two models retained evidence of overfitting. The decision curve analysis showed low clinical benefit for the best performing models.

Conclusions

This review revealed several reporting and methodological shortcomings of published prediction models for spontaneous preterm birth. Our external validation study indicated that none of the models had the ability to predict spontaneous preterm birth adequately in our population. Further improvement of prediction models, using recent knowledge about both model development and potential risk factors, is necessary to provide an added value in personalized risk assessment of spontaneous preterm birth.

Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies

BACKGROUND

Ignoring treatments in prognostic model development or validation can affect the accuracy and transportability of models. We aim to quantify the extent to which the effects of treatment have been addressed in existing prognostic model research and provide recommendations for the handling and reporting of treatment use in future studies.

METHODS

We first describe how and when the use of treatments by individuals in a prognostic study can influence the development or validation of a prognostic model. We subsequently conducted a systematic review of the handling and reporting of treatment use in prognostic model studies in cardiovascular medicine. Data on treatment use (e.g. medications, surgeries, lifestyle interventions), the timing of their use, and the handling of such treatment use in the analyses were extracted and summarised.

RESULTS

Three hundred two articles were included in the review. Treatment use was not mentioned in 91 (30%) articles. One hundred forty-six (48%) reported specific information about treatment use in their studies; 78 (26%) provided information about multiple treatments. Three articles (1%) reported changes in medication use ("treatment drop-in") during follow-up. Seventy-nine articles (26%) excluded treated individuals from their analysis, 80 articles (26%) modelled treatment as an outcome, and of the 155 articles that developed a model, 86 (55%) modelled treatment use, almost exclusively at baseline, as a predictor.

CONCLUSIONS

The use of treatments has been partly considered by the majority of CVD prognostic model studies. Detailed accounts including, for example, information on treatment drop-in were rare. Where relevant, the use of treatments should be considered in the analysis of prognostic model studies, particularly when a prognostic model is designed to guide the use of certain treatments and these treatments have been used by the study participants. Future prognostic model studies should clearly report the use of treatments by study participants and consider the potential impact of treatment use on the study findings.