Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews

Childhood asthma prediction models: a systematic review

Early identification of children at risk of developing asthma at school age is crucial, but the usefulness of childhood asthma prediction models in clinical practice is still unclear. We systematically reviewed all existing prediction models to identify preschool children with asthma-like symptoms at risk of developing asthma at school age. Studies were included if they developed a new prediction model or updated an existing model in children aged 4 years or younger with asthma-like symptoms, with assessment of asthma done between 6 and 12 years of age. 12 prediction models were identified in four types of cohorts of preschool children: those with health-care visits, those with parent-reported symptoms, those at high risk of asthma, or children in the general population. Four basic models included non-invasive, easy-to-obtain predictors only, notably family history, allergic disease comorbidities or precursors of asthma, and severity of early symptoms. Eight extended models included additional clinical tests, mostly specific IgE determination. Some models could better predict asthma development and other models could better rule out asthma development, but the predictive performance of no single model stood out in both aspects simultaneously. This finding suggests that there is a large proportion of preschool children with wheeze for which prediction of asthma development is difficult.

Building a gold standard to construct search filters: a case study with biomarkers for oral cancer

OBJECTIVE

To support clinical researchers, librarians and informationists may need search filters for particular tasks. Development of filters typically depends on a "gold standard" dataset. This paper describes generalizable methods for creating a gold standard to support future filter development and evaluation using oral squamous cell carcinoma (OSCC) as a case study. OSCC is the most common malignancy affecting the oral cavity. Investigation of biomarkers with potential prognostic utility is an active area of research in OSCC. The methods discussed here should be useful for designing quality search filters in similar domains.

METHODS

The authors searched MEDLINE for prognostic studies of OSCC, developed annotation guidelines for screeners, ran three calibration trials before annotating the remaining body of citations, and measured inter-annotator agreement (IAA).

RESULTS

We retrieved 1,818 citations. After calibration, we screened the remaining citations (n = 1,767; 97.2%); IAA was substantial (kappa = 0.76). The dataset has 497 (27.3%) citations representing OSCC studies of potential prognostic biomarkers.

CONCLUSIONS

The gold standard dataset is likely to be high quality and useful for future development and evaluation of filters for OSCC studies of potential prognostic biomarkers.

IMPLICATIONS

The methodology we used is generalizable to other domains requiring a reference standard to evaluate the performance of search filters. A gold standard is essential because the labels regarding relevance enable computation of diagnostic metrics, such as sensitivity and specificity. Librarians and informationists with data analysis skills could contribute to developing gold standard datasets and subsequent filters tuned for their patrons' domains of interest.

Non-cardiovascular comorbidity, severity and prognosis in non-selected heart failure populations: A systematic review and meta-analysis

Background

Non-cardiovascular comorbidities are recognised as independent prognostic factors in selected heart failure (HF) populations, but the evidence on non-selected HF and how comorbid disease severity and change impacts on outcomes has not been synthesised. We identified primary HF comorbidity follow-up studies to compare the impact of non-cardiovascular comorbidity, severity and change on the outcomes of quality of life, all-cause hospital admissions and all-cause mortality.

Methods

Literature databases (Jan 1990–May 2013) were screened using validated strategies and quality appraisal (QUIPS tool). Adjusted hazard ratios for the main HF outcomes were combined using random effects meta-analysis and inclusion of comorbidity in prognostic models was described.

Results

There were 68 primary HF studies covering nine non-cardiovascular comorbidities. Most were on diabetes mellitus (DM), chronic obstructive pulmonary disease (COPD) and renal dysfunction (RD) for the outcome of mortality (93%) and hospital admissions (16%), median follow-up of 4 years. The adjusted associations between HF comorbidity and mortality were DM (HR 1.34; 95% CI 1.2, 1.5), COPD (1.39; 1.2, 1.6) and RD (1.52; 1.3, 1.7). Comorbidity severity increased mortality from moderate to severe disease by an estimated 78%, 42% and 80% respectively. The risk of hospital admissions increased up to 50% for each disease. Few studies or prognostic models included comorbidity change.

Conclusions

Non-cardiovascular comorbidity and severity significantly increases the prognostic risk of poor outcomes in non-selected HF populations but there is a major gap in investigating change in comorbid status over time. The evidence supports a step-change for the inclusion of comorbidity severity in new HF interventions to improve prognostic outcomes.

•

We synthesise the prognosis evidence on non-CVD comorbidity and severity in non-selected HF

•

Most studies focused on three comorbid diseases for mortality and admissions and none for QoL

•

COPD, diabetes and CKD increased mortality and admission risk in non-selected HF

•

Severity studies were few but where available, risk increased with disease severity

•

Comorbidity severity is important but has yet to be included in HF prognostic models

Validation and impact analysis of prognostic clinical prediction rules for low back pain is needed: a systematic review

OBJECTIVES

To identify prognostic forms of clinical prediction rules (CPRs) related to the nonsurgical management of adults with low back pain (LBP) and to evaluate their current stage of development.

STUDY DESIGN AND SETTING

Systematic review using a sensitive search strategy across seven databases with hand searching and citation tracking.

RESULTS

A total of 10,005 records were screened for eligibility with 35 studies included in the review. The included studies report on the development of 30 prognostic LBP CPRs. Most of the identified CPRs are in their initial phase of development. Three CPRs were found to have undergone validation--the Cassandra rule for predicting long-term significant functional limitations and the five-item and two-item Flynn manipulation CPRs for predicting a favorable functional prognosis in patients being treated with lumbopelvic manipulation. No studies were identified that investigated whether the implementation of a CPR resulted in beneficial patient outcomes or improved resource efficiencies.

CONCLUSION

Most of the identified prognostic CPRs for LBP are in the initial phase of development and are consequently not recommended for direct application in clinical practice at this time. The body of evidence provides emergent confidence in the limited predictive performance of the Cassandra rule and the five-item Flynn manipulation CPR in comparable clinical settings and patient populations.

Diagnostic clinical prediction rules for specific subtypes of low back pain: a systematic review

STUDY DESIGN

Systematic review.

OBJECTIVES

To identify diagnostic clinical prediction rules (CPRs) for low back pain (LBP) and to assess their readiness for clinical application.

BACKGROUND

Significant research has been invested into the development of CPRs that may assist in the meaningful subgrouping of patients with LBP. To date, very little is known about diagnostic forms of CPRs for LBP, which relate to the present status or classification of an individual, and whether they have been developed sufficiently to enable their application in clinical practice.

METHODS

A sensitive electronic search strategy using 7 databases was combined with hand searching and citation tracking to identify eligible studies. Two independent reviewers identified relevant studies for inclusion using a 2-stage selection process. The quality appraisal of included studies was conducted by 2 independent raters using the Quality Assessment of Diagnostic Accuracy Studies-2 and checklists composed of accepted methodological standards for the development of CPRs.

RESULTS

Of 10 014 studies screened for eligibility, the search identified that 13 diagnostic CPRs for LBP have been derived. Among those, 1 tool for identifying lumbar spinal stenosis and 2 tools for identifying inflammatory back pain have undergone validation. No impact analysis studies were identified.

CONCLUSION

Most diagnostic CPRs for LBP are in their initial development phase and cannot be recommended for use in clinical practice at this time. Validation and impact analysis of the diagnostic CPRs identified in this review are warranted, particularly for those tools that meet an identified unmet need of clinicians who manage patients with LBP.

LEVEL OF EVIDENCE

Diagnosis, level 2a-.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

Methodological issues and recommendations for systematic reviews of prognostic studies: an example from cardiovascular disease

BACKGROUND

Prognostic factors are associated with the risk of future health outcomes in individuals with a particular health condition. The prognostic ability of such factors is increasingly being assessed in both primary research and systematic reviews. Systematic review methodology in this area is continuing to evolve, reflected in variable approaches to key methodological aspects. The aim of this article was to (i) explore and compare the methodology of systematic reviews of prognostic factors undertaken for the same clinical question, (ii) to discuss implications for review findings, and (iii) to present recommendations on what might be considered to be 'good practice' approaches.

METHODS

The sample was comprised of eight systematic reviews addressing the same clinical question, namely whether 'aspirin resistance' (a potential prognostic factor) has prognostic utility relative to future vascular events in patients on aspirin therapy for secondary prevention. A detailed comparison of methods around study identification, study selection, quality assessment, approaches to analysis, and reporting of findings was undertaken and the implications discussed. These were summarised into key considerations that may be transferable to future systematic reviews of prognostic factors.

RESULTS

Across systematic reviews addressing the same clinical question, there were considerable differences in the numbers of studies identified and overlap between included studies, which could only partially be explained by different study eligibility criteria. Incomplete reporting and differences in terminology within primary studies hampered study identification and selection process across reviews. Quality assessment was highly variable and only one systematic review considered a checklist for studies of prognostic questions. There was inconsistency between reviews in approaches towards analysis, synthesis, addressing heterogeneity and reporting of results.

CONCLUSIONS

Different methodological approaches may ultimately affect the findings and interpretation of systematic reviews of prognostic research, with implications for clinical decision-making.

External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination

OBJECTIVES

To evaluate how often newly developed risk prediction models undergo external validation and how well they perform in such validations.

STUDY DESIGN AND SETTING

We reviewed derivation studies of newly proposed risk models and their subsequent external validations. Study characteristics, outcome(s), and models' discriminatory performance [area under the curve, (AUC)] in derivation and validation studies were extracted. We estimated the probability of having a validation, change in discriminatory performance with more stringent external validation by overlapping or different authors compared to the derivation estimates.

RESULTS

We evaluated 127 new prediction models. Of those, for 32 models (25%), at least an external validation study was identified; in 22 models (17%), the validation had been done by entirely different authors. The probability of having an external validation by different authors within 5 years was 16%. AUC estimates significantly decreased during external validation vs. the derivation study [median AUC change: -0.05 (P < 0.001) overall; -0.04 (P = 0.009) for validation by overlapping authors; -0.05 (P < 0.001) for validation by different authors]. On external validation, AUC decreased by at least 0.03 in 19 models and never increased by at least 0.03 (P < 0.001).

CONCLUSION

External independent validation of predictive models in different studies is uncommon. Predictive performance may worsen substantially on external validation.

Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist

Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary.

Predicting adverse drug reactions in older adults; a systematic review of the risk prediction models

Adverse drug reaction (ADR) risk-prediction models for use in older adults have been developed, but it is not clear if they are suitable for use in clinical practice. This systematic review aimed to identify and investigate the quality of validated ADR risk-prediction models for use in older adults. Standard computerized databases, the gray literature, bibliographies, and citations were searched (2012) to identify relevant peer-reviewed studies. Studies that developed and validated an ADR prediction model for use in patients over 65 years old, using a multivariable approach in the design and analysis, were included. Data were extracted and their quality assessed by independent reviewers using a standard approach. Of the 13,423 titles identified, only 549 were associated with adverse outcomes of medicines use. Four met the inclusion criteria. All were conducted in inpatient cohorts in Western Europe. None of the models satisfied the four key stages in the creation of a quality risk prediction model; development and validation were completed, but impact and implementation were not assessed. Model performance was modest; area under the receiver operator curve ranged from 0.623 to 0.73. Study quality was difficult to assess due to poor reporting, but inappropriate methods were apparent. Further work needs to be conducted concerning the existing models to enable the development of a robust ADR risk-prediction model that is externally validated, with practical design and good performance. Only then can implementation and impact be assessed with the aim of generating a model of high enough quality to be considered for use in clinical care to prioritize older people at high risk of suffering an ADR.

External validation of multivariable prediction models: a systematic review of methodological conduct and reporting

BACKGROUND

Before considering whether to use a multivariable (diagnostic or prognostic) prediction model, it is essential that its performance be evaluated in data that were not used to develop the model (referred to as external validation). We critically appraised the methodological conduct and reporting of external validation studies of multivariable prediction models.

METHODS

We conducted a systematic review of articles describing some form of external validation of one or more multivariable prediction models indexed in PubMed core clinical journals published in 2010. Study data were extracted in duplicate on design, sample size, handling of missing data, reference to the original study developing the prediction models and predictive performance measures.

RESULTS

11,826 articles were identified and 78 were included for full review, which described the evaluation of 120 prediction models. in participant data that were not used to develop the model. Thirty-three articles described both the development of a prediction model and an evaluation of its performance on a separate dataset, and 45 articles described only the evaluation of an existing published prediction model on another dataset. Fifty-seven percent of the prediction models were presented and evaluated as simplified scoring systems. Sixteen percent of articles failed to report the number of outcome events in the validation datasets. Fifty-four percent of studies made no explicit mention of missing data. Sixty-seven percent did not report evaluating model calibration whilst most studies evaluated model discrimination. It was often unclear whether the reported performance measures were for the full regression model or for the simplified models.

CONCLUSIONS

The vast majority of studies describing some form of external validation of a multivariable prediction model were poorly reported with key details frequently not presented. The validation studies were characterised by poor design, inappropriate handling and acknowledgement of missing data and one of the most key performance measures of prediction models i.e. calibration often omitted from the publication. It may therefore not be surprising that an overwhelming majority of developed prediction models are not used in practice, when there is a dearth of well-conducted and clearly reported (external validation) studies describing their performance on independent participant data.

Exclusion of deep vein thrombosis using the Wells rule in clinically important subgroups: individual patient data meta-analysis

OBJECTIVE

To assess the accuracy of the Wells rule for excluding deep vein thrombosis and whether this accuracy applies to different subgroups of patients.

DESIGN

Meta-analysis of individual patient data.

DATA SOURCES

Authors of 13 studies (n = 10,002) provided their datasets, and these individual patient data were merged into one dataset.

ELIGIBILITY CRITERIA

Studies were eligible if they enrolled consecutive outpatients with suspected deep vein thrombosis, scored all variables of the Wells rule, and performed an appropriate reference standard.

MAIN OUTCOME MEASURES

Multilevel logistic regression models, including an interaction term for each subgroup, were used to estimate differences in predicted probabilities of deep vein thrombosis by the Wells rule. In addition, D-dimer testing was added to assess differences in the ability to exclude deep vein thrombosis using an unlikely score on the Wells rule combined with a negative D-dimer test result.

RESULTS

Overall, increasing scores on the Wells rule were associated with an increasing probability of having deep vein thrombosis. Estimated probabilities were almost twofold higher in patients with cancer, in patients with suspected recurrent events, and (to a lesser extent) in males. An unlikely score on the Wells rule (≤ 1) combined with a negative D-dimer test result was associated with an extremely low probability of deep vein thrombosis (1.2%, 95% confidence interval 0.7% to 1.8%). This combination occurred in 29% (95% confidence interval 20% to 40%) of patients. These findings were consistent in subgroups defined by type of D-dimer assay (quantitative or qualitative), sex, and care setting (primary or hospital care). For patients with cancer, the combination of an unlikely score on the Wells rule and a negative D-dimer test result occurred in only 9% of patients and was associated with a 2.2% probability of deep vein thrombosis being present. In patients with suspected recurrent events, only the modified Wells rule (adding one point for the previous event) is safe.

CONCLUSION

Combined with a negative D-dimer test result (both quantitative and qualitative), deep vein thrombosis can be excluded in patients with an unlikely score on the Wells rule. This finding is true for both sexes, as well as for patients presenting in primary and hospital care. In patients with cancer, the combination is neither safe nor efficient. For patients with suspected recurrent disease, one extra point should be added to the rule to enable a safe exclusion.

Meta-analysis and aggregation of multiple published prediction models

Published clinical prediction models are often ignored during the development of novel prediction models despite similarities in populations and intended usage. The plethora of prediction models that arise from this practice may still perform poorly when applied in other populations. Incorporating prior evidence might improve the accuracy of prediction models and make them potentially better generalizable. Unfortunately, aggregation of prediction models is not straightforward, and methods to combine differently specified models are currently lacking. We propose two approaches for aggregating previously published prediction models when a validation dataset is available: model averaging and stacked regressions. These approaches yield user-friendly stand-alone models that are adjusted for the new validation data. Both approaches rely on weighting to account for model performance and between-study heterogeneity but adopt a different rationale (averaging versus combination) to combine the models. We illustrate their implementation in a clinical example and compare them with established methods for prediction modeling in a series of simulation studies. Results from the clinical datasets and simulation studies demonstrate that aggregation yields prediction models with better discrimination and calibration in a vast majority of scenarios, and results in equivalent performance (compared to developing a novel model from scratch) when validation datasets are relatively large. In conclusion, model aggregation is a promising strategy when several prediction models are available from the literature and a validation dataset is at hand. The aggregation methods do not require existing models to have similar predictors and can be applied when relatively few data are at hand.

Clinical prediction models for bronchopulmonary dysplasia: a systematic review and external validation study

Background

Bronchopulmonary dysplasia (BPD) is a common complication of preterm birth. Very different models using clinical parameters at an early postnatal age to predict BPD have been developed with little extensive quantitative validation. The objective of this study is to review and validate clinical prediction models for BPD.

Methods

We searched the main electronic databases and abstracts from annual meetings. The STROBE instrument was used to assess the methodological quality. External validation of the retrieved models was performed using an individual patient dataset of 3229 patients at risk for BPD. Receiver operating characteristic curves were used to assess discrimination for each model by calculating the area under the curve (AUC). Calibration was assessed for the best discriminating models by visually comparing predicted and observed BPD probabilities.

Results

We identified 26 clinical prediction models for BPD. Although the STROBE instrument judged the quality from moderate to excellent, only four models utilised external validation and none presented calibration of the predictive value. For 19 prediction models with variables matched to our dataset, the AUCs ranged from 0.50 to 0.76 for the outcome BPD. Only two of the five best discriminating models showed good calibration.

Conclusions

External validation demonstrates that, except for two promising models, most existing clinical prediction models are poor to moderate predictors for BPD. To improve the predictive accuracy and identify preterm infants for future intervention studies aiming to reduce the risk of BPD, additional variables are required. Subsequently, that model should be externally validated using a proper impact analysis before its clinical implementation.