Prognosis Research Strategy (PROGRESS) 3: prognostic model research

A study for proposal of use of regulatory T cells as a prognostic marker and establishing an optimal threshold level for their expression in chronic lymphocytic leukemia

Although regulatory T cells (Tregs) have been extensively studied in chronic lymphocytic leukemia, there is no uniform guideline or consensus regarding their use as a prognostic marker. This study describes the methodology used to develop an optimal threshold level for Tregs in these patients. Treg levels were assessed in the peripheral blood of 130 patients and 150 controls. Treg frequencies were linked to established prognostic markers as well as overall survival and time to first treatment. The cut-offs for Treg positivity were assessed by receiver operating characteristic (ROC) analysis. A cut-off of 5.7% for Treg cell percentage and of 35 cells/μL for absolute Treg cell count were determined as optimal in patients with CLL along with a median Treg percentage of 15.5% used to separate patients with low- and high-risk disease. The experiments presented here will possibly aid in the use of Treg frequencies as a potential prognostic marker in CLL.

External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination

OBJECTIVES

To evaluate how often newly developed risk prediction models undergo external validation and how well they perform in such validations.

STUDY DESIGN AND SETTING

We reviewed derivation studies of newly proposed risk models and their subsequent external validations. Study characteristics, outcome(s), and models' discriminatory performance [area under the curve, (AUC)] in derivation and validation studies were extracted. We estimated the probability of having a validation, change in discriminatory performance with more stringent external validation by overlapping or different authors compared to the derivation estimates.

RESULTS

We evaluated 127 new prediction models. Of those, for 32 models (25%), at least an external validation study was identified; in 22 models (17%), the validation had been done by entirely different authors. The probability of having an external validation by different authors within 5 years was 16%. AUC estimates significantly decreased during external validation vs. the derivation study [median AUC change: -0.05 (P < 0.001) overall; -0.04 (P = 0.009) for validation by overlapping authors; -0.05 (P < 0.001) for validation by different authors]. On external validation, AUC decreased by at least 0.03 in 19 models and never increased by at least 0.03 (P < 0.001).

CONCLUSION

External independent validation of predictive models in different studies is uncommon. Predictive performance may worsen substantially on external validation.

Cardiovascular Disease Population Risk Tool (CVDPoRT): predictive algorithm for assessing CVD risk in the community setting. A study protocol

INTRODUCTION

Recent publications have called for substantial improvements in the design, conduct, analysis and reporting of prediction models. Publication of study protocols, with prespecification of key aspects of the analysis plan, can help to improve transparency, increase quality and protect against increased type I error. Valid population-based risk algorithms are essential for population health planning and policy decision-making. The purpose of this study is to develop, evaluate and apply cardiovascular disease (CVD) risk algorithms for the population setting.

METHODS AND ANALYSIS

The Ontario sample of the Canadian Community Health Survey (2001, 2003, 2005; 77,251 respondents) will be used to assess risk factors focusing on health behaviours (physical activity, diet, smoking and alcohol use). Incident CVD outcomes will be assessed through linkage to administrative healthcare databases (619,886 person-years of follow-up until 31 December 2011). Sociodemographic factors (age, sex, immigrant status, education) and mediating factors such as presence of diabetes and hypertension will be included as predictors. Algorithms will be developed using competing risks survival analysis. The analysis plan adheres to published recommendations for the development of valid prediction models to limit the risk of overfitting and improve the quality of predictions. Key considerations are fully prespecifying the predictor variables; appropriate handling of missing data; use of flexible functions for continuous predictors; and avoiding data-driven variable selection procedures. The 2007 and 2009 surveys (approximately 50,000 respondents) will be used for validation. Calibration will be assessed overall and in predefined subgroups of importance to clinicians and policymakers.

ETHICS AND DISSEMINATION

This study has been approved by the Ottawa Health Science Network Research Ethics Board. The findings will be disseminated through professional and scientific conferences, and in peer-reviewed journals. The algorithm will be accessible electronically for population and individual uses.

TRIAL REGISTRATION NUMBER

ClinicalTrials.gov NCT02267447.

Towards indicated prevention of psychosis: using probabilistic assessments of transition risk in psychosis prodrome

The concept of indicated prevention has proliferated in psychiatry, and accumulating evidence suggests that it may indeed be possible to prevent or delay the onset of a first episode of psychosis though adequate interventions in individuals deemed at clinical high risk (CHR) for such an event. One challenge undermining these efforts is the relatively poor predictive accuracy of clinical assessments used in practice for CHR individuals, often leading to diagnostic and therapeutic uncertainty reflected in clinical guidelines promoting a 'watch and wait' approach to CHR patients. Using data from published studies, and employing predictive models based on the odds-ratio form of Bayes' rule, we simulated scenarios where clinical interview, neurocognitive testing, structural magnetic resonance imaging and electrophysiology are part of the initial assessment process of a CHR individual (extended diagnostic approach). Our findings indicate that for most at-risk patients, at least three of these assessments are necessary to arrive at a clinically meaningful differentiation into high- intermediate-, and low-risk groups. In particular, patients with equivocal results in the initial assessments require additional diagnostic testing to produce an accurate risk profile forming part of the comprehensive initial assessment. The findings may inform future research into reliable identification and personalized therapeutic targeting of CHR patients, to prevent transition to full-blown psychosis.

Observational research on NCDs in HIV-positive populations: conceptual and methodological considerations

Noncommunicable diseases (NCDs) account for a growing burden of morbidity and mortality among people living with HIV in low- and middle-income countries (LMICs). HIV infection and antiretroviral therapy interact with NCD risk factors in complex ways, and research into this "web of causation" has so far been largely based on data from high-income countries. However, improving the understanding, treatment, and prevention of NCDs in LMICs requires region-specific evidence. Priority research areas include: (1) defining the burden of NCDs among people living with HIV, (2) understanding the impact of modifiable risk factors, (3) evaluating effective and efficient care strategies at individual and health systems levels, and (4) evaluating cost-effective prevention strategies. Meeting these needs will require observational data, both to inform the design of randomized trials and to replace trials that would be unethical or infeasible. Focusing on Sub-Saharan Africa, we discuss data resources currently available to inform this effort and consider key limitations and methodological challenges. Existing data resources often lack population-based samples; HIV-negative, HIV-positive, and antiretroviral therapy-naive comparison groups; and measurements of key NCD risk factors and outcomes. Other challenges include loss to follow-up, competing risk of death, incomplete outcome ascertainment and measurement of factors affecting clinical decision making, and the need to control for (time-dependent) confounding. We review these challenges and discuss strategies for overcoming them through augmented data collection and appropriate analysis. We conclude with recommendations to improve the quality of data and analyses available to inform the response to HIV and NCD comorbidity in LMICs.

Psychometric properties and validation of the Italian version of the Mainz pain staging system as a tool for pain-patients referral selection

RATIONALE, AIMS AND OBJECTIVES

Indications are lacking on which patient to refer to pain facilities. Pain-chronicity stage and outcome prognosis may be used for such aims. The Mainz pain-staging system (MPSS) classifies pain patients in three chronicity stages that respectively require more extensive management. We explored the psychometric and validation properties of its Italian version towards its application as screening/referral tool.

METHODS

I-MPSS was administered to n=120 mixed non-cancer-pain outpatients. Psychometric analyses and formal validation included: content validity, by assessing the hypothesis of an existing relationship between the I-MPSS classes and criteria derived from an operational case definition of chronic pain; construct validity, by principle component analysis (PCA); the autonomous construct of the I-MPSS was assessed by the strength of the Spearman correlation between its classes and the brief pain inventory (BPI) items; and reliability, by applying Cronbach's alpha statistics. Associations between psychosocial moderators and the I-MPSS were assessed applying χ(2) analyses.

RESULTS

Quantitative and qualitative analyses showed significant differences between I-MPSS classes for health care and drug utilization; BPI item scores significantly differed between the classes; Spearman correlation between I-MPSS classes and BPI items was mostly moderate or mild. PCA and scree test identified four components accounting for 63.7% of the variance. Cronbach's alpha was 0.842.

CONCLUSIONS

The I-MPSS showed satisfactory psychometric and validation properties. With adequate feasibility, it enabled the screening of mixed non-cancer-pain outpatients in three chronicity/prognostic stages. Results are sufficient to warrant its use for a subsequent impact study as a prognostic model and screening tool for referring pain patients.

Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist

Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary.

Impact of adding therapeutic recommendations to risk assessments from a prediction model for postoperative nausea and vomiting

BACKGROUND

In a large cluster-randomized trial on the impact of a prediction model, presenting the calculated risk of postoperative nausea and vomiting (PONV) on-screen (assistive approach) increased the administration of risk-dependent PONV prophylaxis by anaesthetists. This change in therapeutic decision-making did not improve the patient outcome; that is, the incidence of PONV. The present study aimed to quantify the effects of adding a specific therapeutic recommendation to the predicted risk (directive approach) on PONV prophylaxis decision-making and the incidence of PONV.

METHODS

A prospective before-after study was conducted in 1483 elective surgical inpatients. The before-period included care-as-usual and the after-period included the directive risk-based (intervention) strategy. Risk-dependent effects on the administered number of prophylactic antiemetics and incidence of PONV were analysed by mixed-effects regression analysis.

RESULTS

During the intervention period anaesthetists administered 0.5 [95% confidence intervals (CIs): 0.4-0.6] more antiemetics for patients identified as being at greater risk of PONV. This directive approach led to a reduction in PONV [odds ratio (OR): 0.60, 95% CI: 0.43-0.83], with an even greater reduction in PONV in high-risk patients (OR: 0.45, 95% CI: 0.28-0.72).

CONCLUSIONS

Anaesthetists administered more prophylactic antiemetics when a directive approach was used for risk-tailored intervention compared with care-as-usual. In contrast to the previously studied assistive approach, the increase in PONV prophylaxis now resulted in a lower PONV incidence, particularly in high-risk patients. When one aims for a truly 'PONV-free hospital', a more liberal use of prophylactic antiemetics must be accepted and lower-risk thresholds should be set for the actionable recommendations.

Predicting adverse drug reactions in older adults; a systematic review of the risk prediction models

Adverse drug reaction (ADR) risk-prediction models for use in older adults have been developed, but it is not clear if they are suitable for use in clinical practice. This systematic review aimed to identify and investigate the quality of validated ADR risk-prediction models for use in older adults. Standard computerized databases, the gray literature, bibliographies, and citations were searched (2012) to identify relevant peer-reviewed studies. Studies that developed and validated an ADR prediction model for use in patients over 65 years old, using a multivariable approach in the design and analysis, were included. Data were extracted and their quality assessed by independent reviewers using a standard approach. Of the 13,423 titles identified, only 549 were associated with adverse outcomes of medicines use. Four met the inclusion criteria. All were conducted in inpatient cohorts in Western Europe. None of the models satisfied the four key stages in the creation of a quality risk prediction model; development and validation were completed, but impact and implementation were not assessed. Model performance was modest; area under the receiver operator curve ranged from 0.623 to 0.73. Study quality was difficult to assess due to poor reporting, but inappropriate methods were apparent. Further work needs to be conducted concerning the existing models to enable the development of a robust ADR risk-prediction model that is externally validated, with practical design and good performance. Only then can implementation and impact be assessed with the aim of generating a model of high enough quality to be considered for use in clinical care to prioritize older people at high risk of suffering an ADR.

Risk prediction in patients with heart failure: a systematic review and analysis

OBJECTIVES

This study sought to review the literature for risk prediction models in patients with heart failure and to identify the most consistently reported independent predictors of risk across models.

BACKGROUND

Risk assessment provides information about patient prognosis, guides decision making about the type and intensity of care, and enables better understanding of provider performance.

METHODS

MEDLINE and EMBASE were searched from January 1995 to March 2013, followed by hand searches of the retrieved reference lists. Studies were eligible if they reported at least 1 multivariable model for risk prediction of death, hospitalization, or both in patients with heart failure and reported model performance. We ranked reported individual risk predictors by their strength of association with the outcome and assessed the association of model performance with study characteristics.

RESULTS

Sixty-four main models and 50 modifications from 48 studies met the inclusion criteria. Of the 64 main models, 43 models predicted death, 10 hospitalization, and 11 death or hospitalization. The discriminatory ability of the models for prediction of death appeared to be higher than that for prediction of death or hospitalization or prediction of hospitalization alone (p = 0.0003). A wide variation between studies in clinical settings, population characteristics, sample size, and variables used for model development was observed, but these features were not significantly associated with the discriminatory performance of the models. A few strong predictors emerged for prediction of death; the most consistently reported predictors were age, renal function, blood pressure, blood sodium level, left ventricular ejection fraction, sex, brain natriuretic peptide level, New York Heart Association functional class, diabetes, weight or body mass index, and exercise capacity.

CONCLUSIONS

There are several clinically useful and well-validated death prediction models in patients with heart failure. Although the studies differed in many respects, the models largely included a few common markers of risk.

A new framework to enhance the interpretation of external validation studies of clinical prediction models

OBJECTIVES

It is widely acknowledged that the performance of diagnostic and prognostic prediction models should be assessed in external validation studies with independent data from "different but related" samples as compared with that of the development sample. We developed a framework of methodological steps and statistical methods for analyzing and enhancing the interpretation of results from external validation studies of prediction models.

STUDY DESIGN AND SETTING

We propose to quantify the degree of relatedness between development and validation samples on a scale ranging from reproducibility to transportability by evaluating their corresponding case-mix differences. We subsequently assess the models' performance in the validation sample and interpret the performance in view of the case-mix differences. Finally, we may adjust the model to the validation setting.

RESULTS

We illustrate this three-step framework with a prediction model for diagnosing deep venous thrombosis using three validation samples with varying case mix. While one external validation sample merely assessed the model's reproducibility, two other samples rather assessed model transportability. The performance in all validation samples was adequate, and the model did not require extensive updating to correct for miscalibration or poor fit to the validation settings.

CONCLUSION

The proposed framework enhances the interpretation of findings at external validation of prediction models.

Risk factors for adverse prognosis and death in American visceral leishmaniasis: a meta-analysis

BACKGROUND

In the current context of high fatality rates associated with American visceral leishmaniasis (VL), the appropriate use of prognostic factors to identify patients at higher risk of unfavorable outcomes represents a potential tool for clinical practice. This systematic review brings together information reported in studies conducted in Latin America, on the potential predictors of adverse prognosis (continued evolution of the initial clinical conditions of the patient despite the implementation of treatment, independent of the occurrence of death) and death from VL. The limitations of the existing knowledge, the advances achieved and the approaches to be used in future research are presented.

METHODS/PRINCIPAL FINDINGS

The full texts of 14 studies conforming to the inclusion criteria were analyzed and their methodological quality examined by means of a tool developed in the light of current research tools. Information regarding prognostic variables was synthesized using meta-analysis. Variables were grouped according to the strength of evidence considering summary measures, patterns and heterogeneity of effect-sizes, and the results of multivariate analyses. The strongest predictors identified in this review were jaundice, thrombocytopenia, hemorrhage, HIV coinfection, diarrhea, age <5 and age >40-50 years, severe neutropenia, dyspnoea and bacterial infections. Edema and low hemoglobin concentration were also associated with unfavorable outcomes. The main limitation identified was the absence of validation procedures for the few prognostic models developed so far.

CONCLUSIONS/SIGNIFICANCE

Integration of the results from different investigations conducted over the last 10 years enabled the identification of consistent prognostic variables that could be useful in recognizing and handling VL patients at higher risk of unfavorable outcomes. The development of externally validated prognostic models must be prioritized in future investigations.

Graphical assessment of incremental value of novel markers in prediction models: From statistical to decision analytical perspectives

New markers may improve prediction of diagnostic and prognostic outcomes. We aimed to review options for graphical display and summary measures to assess the predictive value of markers over standard, readily available predictors. We illustrated various approaches using previously published data on 3264 participants from the Framingham Heart Study, where 183 developed coronary heart disease (10-year risk 5.6%). We considered performance measures for the incremental value of adding HDL cholesterol to a prediction model. An initial assessment may consider statistical significance (HR = 0.65, 95% confidence interval 0.53 to 0.80; likelihood ratio p < 0.001), and distributions of predicted risks (densities or box plots) with various summary measures. A range of decision thresholds is considered in predictiveness and receiver operating characteristic curves, where the area under the curve (AUC) increased from 0.762 to 0.774 by adding HDL. We can furthermore focus on reclassification of participants with and without an event in a reclassification graph, with the continuous net reclassification improvement (NRI) as a summary measure. When we focus on one particular decision threshold, the changes in sensitivity and specificity are central. We propose a net reclassification risk graph, which allows us to focus on the number of reclassified persons and their event rates. Summary measures include the binary AUC, the two-category NRI, and decision analytic variants such as the net benefit (NB). Various graphs and summary measures can be used to assess the incremental predictive value of a marker. Important insights for impact on decision making are provided by a simple graph for the net reclassification risk.

Developing an international register of clinical prediction rules for use in primary care: a descriptive analysis

PURPOSE

We describe the methodology used to create a register of clinical prediction rules relevant to primary care. We also summarize the rules included in the register according to various characteristics.

METHODS

To identify relevant articles, we searched the MEDLINE database (PubMed) for the years 1980 to 2009 and supplemented the results with searches of secondary sources (books on clinical prediction rules) and personal resources (eg, experts in the field). The rules described in relevant articles were classified according to their clinical domain, the stage of development, and the clinical setting in which they were studied.

RESULTS

Our search identified clinical prediction rules reported between 1965 and 2009. The largest share of rules (37.2%) were retrieved from PubMed. The number of published rules increased substantially over the study decades. We included 745 articles in the register; many contained more than 1 clinical prediction rule study (eg, both a derivation study and a validation study), resulting in 989 individual studies. In all, 434 unique rules had gone through derivation; however, only 54.8% had been validated and merely 2.8% had undergone analysis of their impact on either the process or outcome of clinical care. The rules most commonly pertained to cardiovascular disease, respiratory, and musculoskeletal conditions. They had most often been studied in the primary care or emergency department settings.

CONCLUSIONS

Many clinical prediction rules have been derived, but only about half have been validated and few have been assessed for clinical impact. This lack of thorough evaluation for many rules makes it difficult to retrieve and identify those that are ready for use at the point of patient care. We plan to develop an international web-based register of clinical prediction rules and computer-based clinical decision support systems.

Improving the transparency of prognosis research: the role of reporting, data sharing, registration, and protocols

George Peat and colleagues review and discuss current approaches to transparency and published debates and concerns about efforts to standardize prognosis research practice, and make five recommendations.

Please see later in the article for the Editors' Summary

How diabetes risk assessment tools are implemented in practice: a systematic review

This review aimed to explore the extent of the use of diabetes risk assessment tools and to determine influential variables associated with the implementation of these tools. CINAHL, Google Scholar, ISI Citation Indexes, PubMed, and Scopus were searched from inception to January 2013. Studies that reported the use of diabetes risk assessment tools to identify individuals at risk of diabetes were included. Of the 1719 articles identified, 24 were included. Follow-up of high risk individuals for diagnosis of diabetes was conducted in 5 studies. Barriers to the uptake of diabetes risk assessment tools by healthcare practitioners included (1) attitudes toward the tools; (2) impracticality of using the tools and (3) lack of reimbursement and regulatory support. Individuals were reluctant to undertake self-assessment of diabetes risk due to (1) lack of perceived severity of type 2 diabetes; (2) impracticality of the tools; and (3) concerns related to finding out the results. The current use of non-invasive diabetes risk assessment scores as screening tools appears to be limited. Practical follow up systems as well as strategies to address other barriers to the implementation of diabetes risk assessment tools are essential and need to be developed.

Acute liver failure in Chinese children: a multicenter investigation

BACKGROUND

Currently, no documentation is available regarding Chinese children with acute liver failure (ALF). This study was undertaken to investigate etiologies and outcomes of Chinese children with ALF.

METHODS

We retrospectively enrolled 32 pediatric patients with ALF admitted in five hospitals in different areas of China from January 2007 to December 2012. The coagulation indices, serum creatinine, serum lactate dehydrogenase, blood ammonia and prothrombin activity were analyzed; the relationship between these indices and mortality was evaluated by multivariate analysis.

RESULTS

The most common causes of Chinese children with ALF were indeterminate etiology (15/32), drug toxicity (8/32), and acute cytomegalovirus hepatitis (6/32). Only 1 patient (3.13%) received liver transplantation and the spontaneous mortality of Chinese children with ALF was 58.06% (18/31). Patients who eventually died had higher baseline levels of international normalized ratio (P=0.01), serum creatinine (P=0.04), serum lactate dehydrogenase (P=0.01), blood ammonia (P<0.01) and lower prothrombin activity (P=0.01) than those who survived. Multivariate analysis showed that the entry blood ammonia was the only independent factor significantly associated with mortality (odds ratio=1.069, 95% confidence interval 1.023-1.117, P<0.01) and it had a sensitivity of 94.74%, a specificity of 84.62% and an accuracy of 90.63% for predicting the death. Based on the established model, with an increase of blood ammonia level, the risk of mortality would increase by 6.9%.

CONCLUSIONS

The indeterminate causes predominated in the etiologies of ALF in Chinese children. The spontaneous mortality of pediatric patients with ALF was high, whereas the proportion of patients undergoing liver transplantation was significantly low. Entry blood ammonia was a reliable predictor for the death of pediatric patients with ALF.

A prospective study comparing the predictions of doctors versus models for treatment outcome of lung cancer patients: a step toward individualized care and shared decision making

BACKGROUND

Decision Support Systems, based on statistical prediction models, have the potential to change the way medicine is being practiced, but their application is currently hampered by the astonishing lack of impact studies. Showing the theoretical benefit of using these models could stimulate conductance of such studies. In addition, it would pave the way for developing more advanced models, based on genomics, proteomics and imaging information, to further improve the performance of the models.

PURPOSE

In this prospective single-center study, previously developed and validated statistical models were used to predict the two-year survival (2yrS), dyspnea (DPN), and dysphagia (DPH) outcomes for lung cancer patients treated with chemo radiation. These predictions were compared to probabilities provided by doctors and guideline-based recommendations currently used. We hypothesized that model predictions would significantly outperform predictions from doctors.

MATERIALS AND METHODS

Experienced radiation oncologists (ROs) predicted all outcomes at two timepoints: (1) after the first consultation of the patient, and (2) after the radiation treatment plan was made. Differences in the performances of doctors and models were assessed using Area Under the Curve (AUC) analysis.

RESULTS

A total number of 155 patients were included. At timepoint #1 the differences in AUCs between the ROs and the models were 0.15, 0.17, and 0.20 (for 2yrS, DPN, and DPH, respectively), with p-values of 0.02, 0.07, and 0.03. Comparable differences at timepoint #2 were not statistically significant due to the limited number of patients. Comparison to guideline-based recommendations also favored the models.

CONCLUSION

The models substantially outperformed ROs' predictions and guideline-based recommendations currently used in clinical practice. Identification of risk groups on the basis of the models facilitates individualized treatment, and should be further investigated in clinical impact studies.

Development and validation of a prediction algorithm for use by health professionals in prediction of recurrence of major depression

BACKGROUND

There exists very little evidence to guide clinical management for preventing recurrence of major depression. The objective of this study was to develop and validate a prediction algorithm for recurrence of major depression.

METHODS

Wave 1 and wave 2 longitudinal data from the U.S. National Epidemiological Survey on Alcohol and Related Condition (2001/2002–2003/2004) were used. Participants with a major depressive episode at baseline and who had visited health professionals for depression were included in this analysis (n = 2,711). Mental disorders were assessed based on the DSM-IV criteria.

RESULTS

With the development data (n = 1,518), a prediction model with 19 unique factors had a C statistics of 0.7504 and excellent calibration (P = .23). The model had a C statistics of 0.7195 in external validation data (n = 1,195) and 0.7365 in combined data. The algorithm calibrated very well in validation data. In the combined data, the 3-year observed and predicted risk of recurrence was 25.40% (95% CI: 23.76%, 27.04%) and 25.34% (95% CI: 24.73%, 25.95%), respectively. The predicted risk in the 1st and 10th decile risk group was 5.68% and 60.21%, respectively.

CONCLUSIONS

The developed prediction model for recurrence of major depression has acceptable discrimination and excellent calibration, and is feasible to be used by physicians. The prognostic model may assist physicians and patients in quantifying the probability of recurrence so that physicians can develop specific treatment plans for those who are at high risk of recurrence, leading to personalized treatment and better use of resources.

Predicting non return to work after orthopaedic trauma: the Wallis Occupational Rehabilitation RisK (WORRK) model

Background

Workers with persistent disabilities after orthopaedic trauma may need occupational rehabilitation. Despite various risk profiles for non-return-to-work (non-RTW), there is no available predictive model. Moreover, injured workers may have various origins (immigrant workers), which may either affect their return to work or their eligibility for research purposes. The aim of this study was to develop and validate a predictive model that estimates the likelihood of non-RTW after occupational rehabilitation using predictors which do not rely on the worker’s background.

Methods

Prospective cohort study (3177 participants, native (51%) and immigrant workers (49%)) with two samples: a) Development sample with patients from 2004 to 2007 with Full and Reduced Models, b) External validation of the Reduced Model with patients from 2008 to March 2010. We collected patients’ data and biopsychosocial complexity with an observer rated interview (INTERMED). Non-RTW was assessed two years after discharge from the rehabilitation. Discrimination was assessed by the area under the receiver operating curve (AUC) and calibration was evaluated with a calibration plot. The model was reduced with random forests.

Results

At 2 years, the non-RTW status was known for 2462 patients (77.5% of the total sample). The prevalence of non-RTW was 50%. The full model (36 items) and the reduced model (19 items) had acceptable discrimination performance (AUC 0.75, 95% CI 0.72 to 0.78 and 0.74, 95% CI 0.71 to 0.76, respectively) and good calibration. For the validation model, the discrimination performance was acceptable (AUC 0.73; 95% CI 0.70 to 0.77) and calibration was also adequate.

Conclusions

Non-RTW may be predicted with a simple model constructed with variables independent of the patient’s education and language fluency. This model is useful for all kinds of trauma in order to adjust for case mix and it is applicable to vulnerable populations like immigrant workers.

Recommendations for reporting the results of studies of instrument and scale development and testing

Scales and instruments play an important role in health research and practice. It is important that studies that report on their psychometric properties do so in a way such that readers can understand what was done and what was found. This paper is a guide to writing articles about the development and assessment of these tools. It covers what should be in the abstract and how key words should be chosen. The article then discusses what should be in the main parts of the paper: the introduction, methods, results and discussion. In each of these parts, it suggests the statistical tests that should be used and how to report them. The emphasis throughout the paper is that reliability and validity are not fixed properties of a scale, but depend on an interaction among it, the population being evaluated and the circumstances under which the instrument is administered.

End-of-life decisions in patients with severe acute brain injury

Most in-hospital deaths of patients with stroke, traumatic brain injury, or postanoxic encephalopathy after cardiac arrest occur after a decision to withhold or withdraw life-sustaining treatments. Decisions on treatment restrictions in these patients are generally complex and are based only in part on evidence from published work. Prognostic models to be used in this decision-making process should have a strong discriminative power. However, for most causes of acute brain injury, prognostic models are not sufficiently accurate to serve as the sole basis of decisions to limit treatment. These decisions are also complicated because patients often do not have the capacity to communicate their preferences. Additionally, surrogate decision makers might not accurately represent the patient's preferences. Finally, in the acute stage, prediction of how a patient would adapt to a life with major disability is difficult.

External validation of multivariable prediction models: a systematic review of methodological conduct and reporting

BACKGROUND

Before considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.

METHODS

We conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures.

RESULTS

11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models.

CONCLUSIONS

The vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

Beyond the diagnosis of idiopathic pulmonary fibrosis; the growing role of systems biology and stratified medicine

PURPOSE OF REVIEW

Idiopathic pulmonary fibrosis (IPF) is a progressive, invariably fatal condition with a median survival from diagnosis of only 3 years. Despite improvements in disease understanding, challenges remain in establishing a diagnosis and predicting prognosis in individual patients. Furthermore, limited understanding of the key pathogenetic mechanisms driving disease is hampering development of new therapies. This review outlines progress that has been made in applying systems biology to IPF and the insights into disease pathogenesis, diagnosis and monitoring that this research is providing.

RECENT FINDINGS

Large-scale genome-wide association studies have highlighted polymorphisms in genes involved in epithelial integrity and host defense including MUC5B and TOLLIP. Whole blood transcriptomics points towards changes in immune cell regulation that influence the progression of fibrosis. Proteomic studies have identified serum proteins, including matrix metalloproteinase 7 and CC chemokine ligand (CCL)-18, which associate with disease severity and predict prognosis.

SUMMARY

Use of molecular research techniques in large populations of well-phenotyped patients is leading to major advances in understanding of IPF. As new treatments for IPF emerge, it is to be hoped that careful application of these findings will enable the targeting of therapy to individuals based on the predominant mechanisms driving progression of their disease.

Integrating pretreatment diffusion weighted MRI into a multivariable prognostic model for head and neck squamous cell carcinoma

INTRODUCTION

In head and neck squamous cell carcinoma (HNSCC) the ability to anticipate an individual patient's outcome is very valuable. With this study we wanted to assess the prognostic value of pretreatment apparent diffusion coefficient (ADC) in a large patient population and integrate it into a multivariable prognostic model.

METHODS

From 2004 to 2010 175 patients with pathology proven HNSCC were included in this study. All patients underwent a pretreatment MRI with diffusion weighted imaging (DWI) using six b-values. For each tumor, three ADC values were calculated using different b-value combinations: ADC(low) (b 0-50-100 s/mm(2)), ADChigh (b 500-750-1000 s/mm(2)) and ADC(avg) (all b-values). The clinical and radiological variables included: tumor and nodal volume, tumor location and age. Disease recurrence was analyzed using competing risk regression. A prognostic model for disease recurrence was developed, and internal validation was performed using bootstrapping and by dividing patients in three equal sized groups based on prognosis.

RESULTS

One hundred and sixty-one patients were eligible for analysis. Median follow-up was 50 months (range 4-86). A total of 67 patients experienced disease recurrence during follow-up (42%). ADC(high) was a prognostic factor for disease recurrence (adjusted hazard ratio: 1.14 per 10(-4) mm(2)/s, 95% CI 1.04-1.25). Harrell's c-index of the multivariable prognostic model was 0.62 (95% CI 0.56-0.70) after internal validation. The validated 3-year disease recurrence rates for the groups with worst, intermediate, and best prognosis were 56%, 33% and 31% respectively.

CONCLUSION

Pretreatment ADC value derived from high b-values is an independent prognostic factor in HNSCC and increases the performance of a multivariable prognostic model in addition to known clinical and radiological variables. Integration of other biomarkers and external validation is necessary to ensure its clinical applicability.

How to derive and validate clinical prediction models for use in intensive care medicine

BACKGROUND

Clinical prediction models are formal combinations of historical, physical examination and laboratory or radiographic test data elements designed to accurately estimate the probability that a specific illness is present (diagnostic model), will respond to a form of treatment (therapeutic model) or will have a well-defined outcome (prognostic model) in an individual patient. They are derived and validated using empirical data and used to assist physicians in their clinical decision-making that requires a quantitative assessment of diagnostic, therapeutic or prognostic probabilities at the bedside.

PURPOSE

To provide intensivists with a comprehensive overview of the empirical development and testing phases that a clinical prediction model must satisfy before its implementation into clinical practice.

RESULTS

The development of a clinical prediction model encompasses three consecutive phases, namely derivation, (external) validation and impact analysis. The derivation phase consists of building a multivariable model, estimating its apparent predictive performance in terms of both calibration and discrimination, and assessing the potential for statistical over-fitting using internal validation techniques (i.e. split-sampling, cross-validation or bootstrapping). External validation consists of testing the predictive performance of a model by assessing its calibration and discrimination in different but plausibly related patients. Impact analysis involves comparative research [i.e. (cluster) randomized trials] to determine whether clinical use of a prediction model affects physician practices, patient outcomes or the cost of healthcare delivery.

CONCLUSIONS

This narrative review introduces a checklist of 19 items designed to help intensivists develop and transparently report valid clinical prediction models.

Risk prediction with machine learning and regression methods

This is a discussion of issues in risk prediction based on the following papers: "Probability estimation with machine learning methods for dichotomous and multicategory outcome: Theory" by Jochen Kruppa, Yufeng Liu, Gérard Biau, Michael Kohler, Inke R. König, James D. Malley, and Andreas Ziegler; and "Probability estimation with machine learning methods for dichotomous and multicategory outcome: Applications" by Jochen Kruppa, Yufeng Liu, Hans-Christian Diener, Theresa Holste, Christian Weimar, Inke R. König, and Andreas Ziegler.

Vaginal birth after a caesarean section: the development of a Western European population-based prediction model for deliveries at term

OBJECTIVE

To develop and internally validate a model that predicts the outcome of an intended vaginal birth after caesarean (VBAC) for a Western European population that can be used to personalise counselling for deliveries at term.

DESIGN

Registration-based retrospective cohort study.

SETTING

Five university teaching hospitals, seven non-university teaching hospitals, and five non-university non-teaching hospitals in the Netherlands.

POPULATION

A cohort of 515 women with a history of one caesarean section and a viable singleton pregnancy, without a contraindication for intended VBAC, who delivered at term.

METHODS

Potential predictors for a vaginal delivery after caesarean section were chosen based on literature and expert opinions. We internally validated the prediction model using bootstrapping techniques.

MAIN OUTCOME MEASURES

Predictors for VBAC. For model validation, the area under the receiver operating characteristic curve (AUC) for discriminative capacity and calibration-per-risk-quantile for accuracy were calculated.

RESULTS

A total of 371 out of 515 women had a VBAC (72%). Variables included in the model were: estimated fetal weight greater than the 90(th) percentile in the third trimester; previous non-progressive labour; previous vaginal delivery; induction of labour; pre-pregnancy body mass index; and ethnicity. The AUC was 71% (95% confidence interval, 95% CI = 69-73%), indicating a good discriminative ability. The calibration plot shows that the predicted probabilities are well calibrated, especially from 65% up, which accounts for 77% of the total study population.

CONCLUSION

We developed an appropriate Western European population-based prediction model that is aimed to personalise counselling for term deliveries.

Cardiac magnetic resonance imaging findings and the risk of cardiovascular events in patients with recent myocardial infarction or suspected or known coronary artery disease: a systematic review of prognostic studies

The goal of this study was to review the prognostic value of cardiac magnetic resonance (CMR) imaging findings for future cardiovascular events in patients with a recent myocardial infarction (MI) and patients with suspected or known coronary artery disease (CAD). Although the diagnostic value of CMR findings is established, the independent prognostic association with future cardiovascular events remains largely unclear. Studies published by February 2013, identified by systematic MEDLINE and EMBASE searches, were reviewed for associations between CMR findings (left ventricular ejection fraction [LVEF], wall motion abnormalities [WMA], abnormal myocardial perfusion, microvascular obstruction, late gadolinium enhancement, edema, and intramyocardial hemorrhage) and hard events (all-cause mortality, cardiac death, cardiac transplantation, and MI) or major adverse cardiovascular events (MACE) (hard events and other cardiovascular events defined by the authors of the evaluated papers). Fifty-six studies (n = 25,497) were evaluated. For patients with recent MI, too few patients were evaluated to establish associations between CMR findings and hard events. LVEF (range of adjusted hazard ratios [HRs]: 1.03 to 1.05 per % decrease) was independently associated with MACE. In patients with suspected or known CAD, WMA (adjusted HRs: 1.87 to 2.99), inducible perfusion defects (adjusted HRs: 3.02 to 7.77), LVEF (adjusted HRs: 0.72 to 0.82 per 10% increase), and infarction (adjusted HRs: 2.82 to 9.43) were independently associated with hard events, and the presence of inducible perfusion defects was associated with MACE (adjusted HRs: 1.76 to 3.21). The independent predictor of future cardiovascular events for patients with a recent MI was LVEF, and the predictors for patients with suspected or known CAD were WMA, inducible perfusion defects, LVEF, and presence of infarction.

Developing and validating risk prediction models in an individual participant data meta-analysis

Background

Risk prediction models estimate the risk of developing future outcomes for individuals based on one or more underlying characteristics (predictors). We review how researchers develop and validate risk prediction models within an individual participant data (IPD) meta-analysis, in order to assess the feasibility and conduct of the approach.

Methods

A qualitative review of the aims, methodology, and reporting in 15 articles that developed a risk prediction model using IPD from multiple studies.

Results

The IPD approach offers many opportunities but methodological challenges exist, including: unavailability of requested IPD, missing patient data and predictors, and between-study heterogeneity in methods of measurement, outcome definitions and predictor effects. Most articles develop their model using IPD from all available studies and perform only an internal validation (on the same set of data). Ten of the 15 articles did not allow for any study differences in baseline risk (intercepts), potentially limiting their model’s applicability and performance in some populations. Only two articles used external validation (on different data), including a novel method which develops the model on all but one of the IPD studies, tests performance in the excluded study, and repeats by rotating the omitted study.

Conclusions

An IPD meta-analysis offers unique opportunities for risk prediction research. Researchers can make more of this by allowing separate model intercept terms for each study (population) to improve generalisability, and by using ‘internal-external cross-validation’ to simultaneously develop and validate their model. Methodological challenges can be reduced by prospectively planned collaborations that share IPD for risk prediction.

Prognostic models for stable coronary artery disease based on electronic health record cohort of 102 023 patients

Aims

The population with stable coronary artery disease (SCAD) is growing but validated models to guide their clinical management are lacking. We developed and validated prognostic models for all-cause mortality and non-fatal myocardial infarction (MI) or coronary death in SCAD.

Methods and results

Models were developed in a linked electronic health records cohort of 102 023 SCAD patients from the CALIBER programme, with mean follow-up of 4.4 (SD 2.8) years during which 20 817 deaths and 8856 coronary outcomes were observed. The Kaplan–Meier 5-year risk was 20.6% (95% CI, 20.3, 20.9) for mortality and 9.7% (95% CI, 9.4, 9.9) for non-fatal MI or coronary death. The predictors in the models were age, sex, CAD diagnosis, deprivation, smoking, hypertension, diabetes, lipids, heart failure, peripheral arterial disease, atrial fibrillation, stroke, chronic kidney disease, chronic pulmonary disease, liver disease, cancer, depression, anxiety, heart rate, creatinine, white cell count, and haemoglobin. The models had good calibration and discrimination in internal (external) validation with C-index 0.811 (0.735) for all-cause mortality and 0.778 (0.718) for non-fatal MI or coronary death. Using these models to identify patients at high risk (defined by guidelines as 3% annual mortality) and support a management decision associated with hazard ratio 0.8 could save an additional 13–16 life years or 15–18 coronary event-free years per 1000 patients screened, compared with models with just age, sex, and deprivation.

Conclusion

These validated prognostic models could be used in clinical practice to support risk stratification as recommended in clinical guidelines.

Prognosis and course of disability in patients with chronic nonspecific low back pain: a 5- and 12-month follow-up cohort study

BACKGROUND

Few data are available on the course of and predictors for disability in patients with chronic nonspecific low back pain (CNSLBP).

OBJECTIVE

The purpose of this study was to describe the course of disability and identify clinically important prognostic factors of low-back-pain-specific disability in patients with CNSLBP receiving multidisciplinary therapy.

DESIGN

A prospective cohort study was conducted.

METHODS

A total of 1,760 patients with CNSLBP who received multidisciplinary therapy were evaluated for their course of disability and prognostic factors at baseline and at 2-, 5-, and 12-month follow-ups. Recovery was defined as 30% reduction in low back pain-specific disability at follow-up compared with baseline and as absolute recovery if the score on the Quebec Back Pain Disability Scale (QBPDS) was ≤20 points at follow-up. Potential prognostic factors were identified using multivariable logistic regression analysis.

RESULTS

Mean patient-reported disability scores on the QBPDS ranged from 51.7 (SD=15.6) at baseline to 31.7 (SD=15.2), 31.1 (SD=18.2), and 29.1 (SD=20.0) at 2, 5, and 12 months, respectively. The prognostic factors identified for recovery at 5 and 12 months were younger age and high scores on disability and on the 36-Item Short-Form Health Survey (SF-36) (Physical and Mental Component Summaries) at baseline. In addition, at 5-month follow-up, a shorter duration of complaints was a positive predictor, and having no comorbidity and less pain at baseline were additional predictors at 12-month follow-up.

LIMITATIONS

Missing values at 5- and 12-month follow-ups were 11.1% and 45.2%, respectively.

CONCLUSION

After multidisciplinary treatment, the course of disability in patients with CNSLBP continued to decline over a 12-month period. At 5- and 12-month follow-ups, prognostic factors were identified for a clinically relevant decrease in disability scores on the QBPDS.

Prediction of time to return to work after gynaecological surgery: a prospective cohort study in the Netherlands

OBJECTIVE

To measure the impact of the level of invasiveness of gynaecological procedures on time to full Return to Work (RTW) and to identify the most important preoperative sociodemographic, medical and work-related factors that predict the risk of prolonged sick leave.

DESIGN

Prospective cohort study.

SETTING

Dutch university hospital.

POPULATION

A total of 148 women aged 18-65 years scheduled for gynaecological surgery for benign indications.

METHODS

A questionnaire regarding the surgical procedure as well as perioperative and postoperative complications was completed by the attending resident at baseline and 6 weeks after surgery. All other outcome measures were assessed using self-reported patient questionnaires at baseline and 12 weeks post-surgery. The follow-up period was extended up to 1 year after surgery in women failing to return to work. Surgical procedures were categorised into diagnostic, minor, intermediate and major surgery.

MAIN OUTCOME MEASURES

Time to RTW and important predictors for prolonged sick leave after surgery.

RESULTS

Median time to RTW was 7 days (interquartile range [IQR] 5-14) for diagnostic surgery, 14 days (IQR 9-28) for minor surgery, 60 days (IQR 28-101) for intermediate surgery and 69 days (IQR 56-135) for major surgery. Multivariable analysis showed a strongest predictive value of RTW 1 year after surgery for level of invasiveness of surgery (minor surgery hazard ratio [HR] 0.51, 95% CI 0.32-0.81; intermediate surgery HR 0.20, 95% CI 0.12-0.34; major surgery HR 0.09, 95% CI 0.06-0.16), RTW expectations before surgery (HR 0.55, 95% CI 0.36-0.84), and preoperative functional status (HR 1.09, 95% CI 1.04-1.13). A prediction model regarding the probability of prolonged sick leave at 6 weeks was developed, with a sensitivity of 89% and a specificity of 86%.

CONCLUSIONS

RTW often takes a long time, especially after intermediate and major surgery. This study reveals important predictors for prolonged sick leave and provides a prediction model for the risk of sick leave extending 6 weeks after benign gynaecological surgery in the Netherlands.

Protocol for a systematic review of prognostic models for the recurrence of venous thromboembolism (VTE) following treatment for a first unprovoked VTE

BACKGROUND

Venous thromboembolism (VTE) is a chronic disease, with fatal recurrences occurring in 5% to 9% of patients, yet it is also one of the best examples of preventable disease. Prognostic models that utilise multiple prognostic factors (demographic, clinical and laboratory patient characteristics) in combination to predict individual outcome risk may allow the identification of patients who would benefit from long-term anticoagulation therapy, and conversely those that would benefit from stopping such therapy due to a low risk of recurrence. The study will systematically review the evidence on potential prognostic models for the recurrence of VTE or adverse outcomes following the cessation of therapy, and synthesise and summarise each model's prognostic value. The review has been registered with PROSPERO (CRD42013003494).

METHODS/DESIGN

Articles will be sought from the Cochrane library (CENTRAL, CDSR, DARE, HTA databases), MEDLINE and EMBASE. Trial registers will be searched for ongoing studies, and conference abstracts will be sought. Reference lists and subject experts will be utilised. No restrictions on language of publications will be applied. Studies of any design will be included if they examine, in patients ceasing therapy after at least three months' treatment with an oral anticoagulant therapy, whether more than one factor in combination is associated with the risk of VTE recurrence or another adverse outcome. Study quality will be assessed using appropriate guidelines for prognostic models. Prognostic models will be summarised qualitatively and, if tested in multiple validation studies, their predictive performance will be summarised using a random-effects meta-analysis model to account for any between-study heterogeneity.

DISCUSSION

The results of the review will identify prognostic models for the risk of VTE recurrence or adverse outcome following cessation of therapy for a first unprovoked VTE. These will be informative for clinicians currently treating patients for a first unprovoked VTE and considering whether to stop treatment or not for particular individuals. The conclusions of the review will also inform the potential development of new prognostic models and clinical prediction rules to identify those at high or low risk of VTE recurrence or adverse outcome following a first unprovoked VTE.

Diagnostic and prognostic prediction models

Risk prediction models can be used to estimate the probability of either having (diagnostic model) or developing a particular disease or outcome (prognostic model). In clinical practice, these models are used to inform patients and guide therapeutic management. Examples from the field of venous thrombo-embolism (VTE) include the Wells rule for patients suspected of deep venous thrombosis and pulmonary embolism, and more recently prediction rules to estimate the risk of recurrence after a first episode of unprovoked VTE. In this paper, the three phases that are recommended before a prediction model may be used in daily practice are described: development, validation, and impact assessment. In the development phase, the focus is on model development commonly using a multivariable logistic (diagnostic) or survival (prognostic) regression analysis. The performance of the developed model is expressed by discrimination, calibration and (re-) classification. In the validation phase, the developed model is tested in a new set of patients using these same performance measures. This is important, as model performance is commonly poorer in a new set of patients, e.g. due to case-mix or domain differences. Finally, in the impact phase the ability of a prediction model to actually guide patient management is evaluated. Whereas in the development and validation phase single cohort designs are preferred, this last phase asks for comparative designs, ideally randomized designs; therapeutic management and outcomes after using the prediction model is compared to a control group not using the model (e.g. usual care).