Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish a Web-based register

Prognostic models of quality of life after total knee replacement: A systematic review

OBJECTIVE

To systematically review and critically appraise prognostic models for quality of life (QOL) in patients with total knee replacement (TKA).

METHODS

Subjects were TKA recipients recruited from inpatient postoperative settings. Searches were made on June 2022 and updated on April 2023. Databases included PubMed.gov, CINAHL, The Cochrane Library, Web of Science. Two authors performed all review stages independently. Risk of bias assessments on participants, predictors, outcomes and analysis methods followed the Prediction study Risk Of Bias ASsessment Tool (PROBAST).

RESULTS

After screening 2204 studies, 9 were eligible for inclusion. Twelve prognostic models were reported, of which 10 models were developed from data without validation and 2 were both developed and validated. The most frequently applied predictor was the pre-TKA QOL score. Discriminatory measures were reported for 9 (75.0%) models with areas under the curve values of 0.66-0.95. All models showed a high risk of bias, mostly due to limitations in statistical methods and outcome assessments.

CONCLUSION

Several prognostic models have been developed for QOL in patients with TKA, but all models show a high risk of bias and are unreliable in clinical practice. Future, prognostic models overcoming the risk of bias identified in this study are needed.

Multivariable prediction models for long-term outcomes after hip fracture: A protocol for a systematic review

Background: Hip fracture results in high mortality and, for many survivors, long-term functional limitations. Multivariable prediction models for hip fracture outcomes have the potential to aid clinical-decision making as well as risk-adjustment in national audits of care. The aim of this study is to identify, critically appraise and synthesise published multivariable prediction models for long-term outcomes after hip fracture. Protocol: The systematic review will include a literature search of electronic databases (MEDLINE, Embase, Scopus, Web of Science and CINAHL) for journal articles. Search terms related to hip fracture, prognosis and outcomes will be included. Study selection criteria includes studies of people with hip fracture where the study aimed to predict one or more long-term outcomes through derivation or validation of a multivariable prediction model. Studies will be excluded if they focus only on the predictive value of individual factors, or only include patients with periprosthetic fractures, fractures managed non-surgically or younger patients. Covidence software will be used for data management. Two review authors will independently conduct study selection, data extraction and appraisal. Data will be extracted based on the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist. Risk of bias assessment will be conducted using the Prediction model Risk of Bias Assessment Tool (PROBAST). Characteristics and results of all studies will be narratively synthesised and presented in tables. Where the same model has been validated in multiple studies, a meta-analysis of discrimination and calibration will be conducted. Conclusions: This systematic review will aim to identify multivariable models for hip fracture outcome prognosis that have been derived using high quality methods. Results will highlight if current models have the potential for further assessment for use in both clinical decision making and improving methods of national hip fracture audits. PROSPERO registration: CRD42022330019 (25th May 2022).

A Deep Learning Approach to Refine the Identification of High-Quality Clinical Research Articles From the Biomedical Literature: Protocol for Algorithm Development and Validation

Background

A barrier to practicing evidence-based medicine is the rapidly increasing body of biomedical literature. Use of method terms to limit the search can help reduce the burden of screening articles for clinical relevance; however, such terms are limited by their partial dependence on indexing terms and usually produce low precision, especially when high sensitivity is required. Machine learning has been applied to the identification of high-quality literature with the potential to achieve high precision without sacrificing sensitivity. The use of artificial intelligence has shown promise to improve the efficiency of identifying sound evidence.

Objective

The primary objective of this research is to derive and validate deep learning machine models using iterations of Bidirectional Encoder Representations from Transformers (BERT) to retrieve high-quality, high-relevance evidence for clinical consideration from the biomedical literature.

Methods

Using the HuggingFace Transformers library, we will experiment with variations of BERT models, including BERT, BioBERT, BlueBERT, and PubMedBERT, to determine which have the best performance in article identification based on quality criteria. Our experiments will utilize a large data set of over 150,000 PubMed citations from 2012 to 2020 that have been manually labeled based on their methodological rigor for clinical use. We will evaluate and report on the performance of the classifiers in categorizing articles based on their likelihood of meeting quality criteria. We will report fine-tuning hyperparameters for each model, as well as their performance metrics, including recall (sensitivity), specificity, precision, accuracy, F-score, the number of articles that need to be read before finding one that is positive (meets criteria), and classification probability scores.

Results

Initial model development is underway, with further development planned for early 2022. Performance testing is expected to star in February 2022. Results will be published in 2022.

Conclusions

The experiments will aim to improve the precision of retrieving high-quality articles by applying a machine learning classifier to PubMed searching.

International Registered Report Identifier (IRRID)

DERR1-10.2196/29398

The potential of prediction models of functioning remains to be fully exploited: A scoping review in the field of spinal cord injury rehabilitation

OBJECTIVE

The study aimed to explore existing prediction models of functioning in spinal cord injury (SCI).

STUDY DESIGN AND SETTING

The databases PubMed, EBSCOhost CINAHL Complete, and IEEE Xplore were searched for relevant literature. The search strategy included published search filters for prediction model and impact studies, index terms and keywords for SCI, and relevant outcome measures able to assess functioning as reflected in the International Classification of Functioning, Disability and Health (ICF). The search was completed in October 2020.

RESULTS

We identified seven prediction model studies reporting twelve prediction models of functioning. The identified prediction models were mainly envisioned to be used for rehabilitation planning, however, also other possible applications were stated. The method predominantly used was regression analysis and the investigated predictors covered mainly the ICF-components of body functions and activities and participation, next to characteristics of the health condition and health interventions.

CONCLUSION

Findings suggest that the development of prediction models of functioning for use in clinical practice remains to be fully exploited. By providing a comprehensive overview of what has been done, this review informs future research on prediction models of functioning in SCI and contributes to an efficient use of research evidence.

Search methods for prognostic factor systematic reviews: a methodologic investigation

Objective

This study retroactively investigated the search used in a 2019 review by Hayden et al., one of the first systematic reviews of prognostic factors that was published in the Cochrane Library. The review was designed to address recognized weaknesses in reviews of prognosis by using multiple supplementary search methods in addition to traditional electronic database searching.

Methods

The authors used four approaches to comprehensively assess aspects of systematic review literature searching for prognostic factor studies: (1) comparison of search recall of broad versus focused electronic search strategies, (2) linking of search methods of origin for eligible studies, (3) analysis of impact of supplementary search methods on meta-analysis conclusions, and (4) analysis of prognosis filter performance.

Results

The review's focused electronic search strategy resulted in a 91% reduction in recall, compared to a broader version. Had the team relied on the focused search strategy without using supplementary search methods, they would have missed 23 of 58 eligible studies that were indexed in MEDLINE; additionally, the number of included studies in 2 of the review's primary outcome meta-analyses would have changed. Using a broader strategy without supplementary searches would still have missed 5 studies. The prognosis filter used in the review demonstrated the highest sensitivity of any of the filters tested.

Conclusions

Our study results support recommendations for supplementary search methods made by prominent systematic review methodologists. Leaving out any supplemental search methods would have resulted in missed studies, and these omissions would not have been prevented by using a broader search strategy or any of the other prognosis filters tested.

Optimizing a literature surveillance strategy to retrieve sound overall prognosis and risk assessment model papers

OBJECTIVE

Our aim was to develop an efficient search strategy for prognostic studies and clinical prediction guides (CPGs), optimally balancing sensitivity and precision while independent of MeSH terms, as relying on them may miss the most current literature.

MATERIALS AND METHODS

We combined 2 Hedges-based search strategies, modified to remove MeSH terms for overall prognostic studies and CPGs, and ran the search on 269 journals. We read abstracts from a random subset of retrieved references until ≥ 20 per journal were reviewed and classified them as positive when fulfilling standardized quality criteria, thereby assembling a standard dataset used to calibrate the search strategy. We determined performance characteristics of our new search strategy against the Hedges standard and performance characteristics of published search strategies against the standard dataset.

RESULTS

Our search strategy retrieved 16 089 references from 269 journals during our study period. One hundred fifty-four journals yielded ≥ 20 references and ≥ 1 prognostic study or CPG. Against the Hedges standard, the new search strategy had sensitivity/specificity/precision/accuracy of 84%/80%/2%/80%, respectively. Existing published strategies tested against our standard dataset had sensitivities of 36%-94% and precision of 5%-10%.

DISCUSSION

We developed a new search strategy to identify overall prognosis studies and CPGs independent of MeSH terms. These studies are important for medical decision-making, as they identify specific populations and individuals who may benefit from interventions.

CONCLUSION

Our results may benefit literature surveillance and clinical guideline efforts, as our search strategy performs as well as published search strategies while capturing literature at the time of publication.

Existing validated clinical prediction rules for predicting response to physiotherapy interventions for musculoskeletal conditions have limited clinical value: A systematic review

OBJECTIVE

To systematically review clinical prediction rules (CPRs) that have undergone validation testing for predicting response to physiotherapy-related interventions for musculoskeletal conditions.

STUDY DESIGN AND SETTING

PubMed, EMBASE, CINAHL and Cochrane Library were systematically searched to September 2020. Search terms included musculoskeletal (MSK) conditions, physiotherapy interventions and clinical prediction rules. Controlled studies that validated a prescriptive CPR for physiotherapy treatment response in musculoskeletal conditions were included. Two independent reviewers assessed eligibility. Original derivation studies of each CPR were identified. Risk of bias was assessed with the PROBAST tool (derivation studies) and the Cochrane Effective Practice and Organisation of Care group criteria (validation studies).

RESULTS

Nine studies aimed to validate seven prescriptive CPRs for treatment response for MSK conditions including back pain, neck pain, shoulder pain and carpal tunnel syndrome. Treatments included manipulation, traction and exercise. Seven studies failed to demonstrate an association between CPR prediction and outcome. Methodological quality of derivation studies was poor and for validation studies was good overall.

CONCLUSION

Results do not support the use of any CPRs identified to aid physiotherapy treatment selection for common musculoskeletal conditions, due to methodological shortcomings in the derivation studies and lack of association between CPR and outcome in validation studies.

Predictive models of infection in patients with systemic lupus erythematosus: A systematic literature review

INTRODUCTION

Having reliable predictive models of prognosis/the risk of infection in systemic lupus erythematosus (SLE) patients would allow this problem to be addressed on an individual basis to study and implement possible preventive or therapeutic interventions.

OBJECTIVE

To identify and analyze all predictive models of prognosis/the risk of infection in patients with SLE that exist in medical literature.

METHODS

A structured search in PubMed, Embase, and LILACS databases was carried out until May 9, 2020. In addition, a search for abstracts in the American Congress of Rheumatology (ACR) and European League Against Rheumatism (EULAR) annual meetings' archives published over the past eight years was also conducted. Studies on developing, validating or updating predictive prognostic models carried out in patients with SLE, in which the outcome to be predicted is some type of infection, that were generated in any clinical context and with any time horizon were included. There were no restrictions on language, date, or status of the publication. To carry out the systematic review, the CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) guideline recommendations were followed. The PROBAST tool (A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies) was used to assess the risk of bias and the applicability of each model.

RESULTS

We identified four models of infection prognosis in patients with SLE. Mostly, there were very few events per candidate predictor. In addition, to construct the models, an initial selection was made based on univariate analyses with no contraction of the estimated coefficients being carried out. This suggests that the proposed models have a high probability of overfitting and being optimistic.

CONCLUSIONS

To date, very few prognostic models have been published on the infection of SLE patients. These models are very heterogeneous and are rated as having a high risk of bias and methodological weaknesses. Despite the widespread recognition of the frequency and severity of infections in SLE patients, there is no reliable predictive prognostic model that facilitates the study and implementation of personalized preventive or therapeutic measures.Protocol registration number: PROSPERO CRD42020171638.

An updated and more efficient search strategy to identify primary care-relevant clinical prediction rules

OBJECTIVES

The aim of the study was to develop an improved search strategy for clinical prediction rules.

STUDY DESIGN AND SETTING

We first refined a list of 30 primary care-relevant journals and improved the efficiency of the Haynes Narrow Filter/Teljour/Murphy Inclusion Filter with 26 items by removing one term (Modified Haynes 26 filter). We then developed the "Royal College of Surgeons in Ireland (RCSI) filter" and compared it with the modified HNF/TMIF26 for its ability to detect prediction rules in the primary care literature. All abstracts and, if necessary, full text were reviewed independently in parallel by primary care physicians. The key outcomes were the percentage of prediction rules identified out of the total identified by both search strategies (sensitivity) and the number of articles that had to be reviewed to identify them (efficiency).

RESULTS

The Modified Haynes 26 filter returned 1,701 abstracts vs. 1,062 for the RCSI filter. The RCSI filter identified 105 of 111 of all prediction rules identified by either filter, compared with 107 of 111 by the Modified Haynes 26 filter (94.6% vs. 96.4%; P = 0.52). In addition, 9.9% of abstracts found using the RCSI filter were prediction rules, compared with only 6.3% using the Modified Haynes 25 filter (P = 0.001).

CONCLUSION

We have developed a novel "RCSI filter" that more efficiently identifies prediction rules in the medical literature.

PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration

Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.

Impact analysis studies of clinical prediction rules relevant to primary care: a systematic review

OBJECTIVES

Following appropriate validation, clinical prediction rules (CPRs) should undergo impact analysis to evaluate their effect on patient care. The aim of this systematic review is to narratively review and critically appraise CPR impact analysis studies relevant to primary care.

SETTING

Primary care.

PARTICIPANTS

Adults and children.

INTERVENTION

Studies that implemented the CPR compared to usual care were included.

STUDY DESIGN

Randomised controlled trial (RCT), controlled before-after, and interrupted time series.

PRIMARY OUTCOME

Physician behaviour and/or patient outcomes.

RESULTS

A total of 18 studies, incorporating 14 unique CPRs, were included. The main study design was RCT (n=13). Overall, 10 studies reported an improvement in primary outcome with CPR implementation. Of 6 musculoskeletal studies, 5 were effective in altering targeted physician behaviour in ordering imaging for patients presenting with ankle, knee and neck musculoskeletal injuries. Of 6 cardiovascular studies, 4 implemented cardiovascular risk scores, and 3 reported no impact on physician behaviour outcomes, such as prescribing and referral, or patient outcomes, such as reduction in serum lipid levels. 2 studies examined CPRs in decision-making for patients presenting with chest pain and reduced inappropriate admissions. Of 5 respiratory studies, 2 were effective in reducing antibiotic prescribing for sore throat following CPR implementation. Overall, study methodological quality was often unclear due to incomplete reporting.

CONCLUSIONS

Despite increasing interest in developing and validating CPRs relevant to primary care, relatively few have gone through impact analysis. To date, research has focused on a small number of CPRs across few clinical domains only.

A Multistep Maturity Model for the Implementation of Electronic and Computable Diagnostic Clinical Prediction Rules (eCPRs)

Introduction:

The use of Clinical Prediction Rules (CPRs) has been advocated as one way of implementing actionable evidence-based rules in clinical practice. The current highly manual nature of deriving CPRs makes them difficult to use and maintain. Addressing the known limitations of CPRs requires implementing more flexible and dynamic models of CPR development. We describe the application of Information and Communication Technology (ICT) to provide a platform for the derivation and dissemination of CPRs derived through analysis and continual learning from electronic patient data.

Model Components:

We propose a multistep maturity model for constructing electronic and computable CPRs (eCPRs). The model has six levels – from the lowest level of CPR maturity (literaturebased CPRs) to a fully electronic and computable service-oriented model of CPRs that are sensitive to specific demographic patient populations. We describe examples of implementations of the core model components – focusing on CPR representation, interoperability, electronic dissemination, CPR learning, and user interface requirements.

Conclusion:

The traditional focus on derivation and narrow validation of CPRs has severely limited their wider acceptance. The evolution and maturity model described here outlines a progression toward eCPRs consistent with the vision of a learning health system (LHS) – using central repositories of CPR knowledge, accessible open standards, and generalizable models to avoid repetition of previous work. This is useful for developing more ambitious strategies to address limitations of the traditional CPR development life cycle. The model described here is a starting point for promoting discussion about what a more dynamic CPR development process should look like.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

An assessment of the cost-effectiveness of magnetic resonance, including diffusion-weighted imaging, in patients with transient ischaemic attack and minor stroke: a systematic review, meta-analysis and economic evaluation

BACKGROUND

Patients with transient ischaemic attack (TIA) or minor stroke need rapid treatment of risk factors to prevent recurrent stroke. ABCD2 score or magnetic resonance diffusion-weighted brain imaging (MR DWI) may help assessment and treatment.

OBJECTIVES

Is MR with DWI cost-effective in stroke prevention compared with computed tomography (CT) brain scanning in all patients, in specific subgroups or as 'one-stop' brain-carotid imaging? What is the current UK availability of services for stroke prevention?

DATA SOURCES

Published literature; stroke registries, audit and randomised clinical trials; national databases; survey of UK clinical and imaging services for stroke; expert opinion.

REVIEW METHODS

Systematic reviews and meta-analyses of published/unpublished data. Decision-analytic model of stroke prevention including on a 20-year time horizon including nine representative imaging scenarios.

RESULTS

The pooled recurrent stroke rate after TIA (53 studies, 30,558 patients) is 5.2% [95% confidence interval (CI) 3.9% to 5.9%] by 7 days, and 6.7% (5.2% to 8.7%) at 90 days. ABCD2 score does not identify patients with key stroke causes or identify mimics: 66% of specialist-diagnosed true TIAs and 35-41% of mimics had an ABCD2 score of ≥ 4; 20% of true TIAs with ABCD2 score of < 4 had key risk factors. MR DWI (45 studies, 9078 patients) showed an acute ischaemic lesion in 34.3% (95% CI 30.5% to 38.4%) of TIA, 69% of minor stroke patients, i.e. two-thirds of TIA patients are DWI negative. TIA mimics (16 studies, 14,542 patients) make up 40-45% of patients attending clinics. UK survey (45% response) showed most secondary prevention started prior to clinic, 85% of primary brain imaging was same-day CT; 51-54% of patients had MR, mostly additional to CT, on average 1 week later; 55% omitted blood-sensitive MR sequences. Compared with 'CT scan all patients' MR was more expensive and no more cost-effective, except for patients presenting at > 1 week after symptoms to diagnose haemorrhage; strategies that triaged patients with low ABCD2 scores for slow investigation or treated DWI-negative patients as non-TIA/minor stroke prevented fewer strokes and increased costs. 'One-stop' CT/MR angiographic-plus-brain imaging was not cost-effective.

LIMITATIONS

Data on sensitivity/specificity of MR in TIA/minor stroke, stroke costs, prognosis of TIA mimics and accuracy of ABCD2 score by non-specialists are sparse or absent; all analysis had substantial heterogeneity.

CONCLUSIONS

Magnetic resonance with DWI is not cost-effective for secondary stroke prevention. MR was most helpful in patients presenting at > 1 week after symptoms if blood-sensitive sequences were used. ABCD2 score is unlikely to facilitate patient triage by non-stroke specialists. Rapid specialist assessment, CT brain scanning and identification of serious underlying stroke causes is the most cost-effective stroke prevention strategy.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

Development and validation of a clinical prediction rule to identify suspected breast cancer: a prospective cohort study

Background

The number of primary care referrals of women with breast symptoms to symptomatic breast units (SBUs) has increased exponentially in the past decade in Ireland. The aim of this study is to develop and validate a clinical prediction rule (CPR) to identify women with breast cancer so that a more evidence based approach to referral from primary care to these SBUs can be developed.

Methods

We analysed routine data from a prospective cohort of consecutive women reviewed at a SBU with breast symptoms. The dataset was split into a derivation and validation cohort. Regression analysis was used to derive a CPR from the patient’s history and clinical findings. Validation of the CPR consisted of estimating the number of breast cancers predicted to occur compared with the actual number of observed breast cancers across deciles of risk.

Results

A total of 6,590 patients were included in the derivation study and 4.9% were diagnosed with breast cancer. Independent clinical predictors for breast cancer were: increasing age by year (adjusted odds ratio 1.08, 95% CI 1.07-1.09); presence of a lump (5.63, 95% CI 4.2-7.56); nipple change (2.77, 95% CI 1.68-4.58) and nipple discharge (2.09, 95% CI 1.1-3.97). Validation of the rule (n = 911) demonstrated that the probability of breast cancer was higher with an increasing number of these independent variables. The Hosmer-Lemeshow goodness of fit showed no overall significant difference between the expected and the observed numbers of breast cancer (χ²_HL: 6.74, p-value: 0.56).

Conclusions

This study derived and validated a CPR for breast cancer in women attending an Irish national SBU. We found that increasing age, presence of a lump, nipple discharge and nipple change are all associated with increased risk of breast cancer. Further validation of the rule is necessary as well as an assessment of its impact on referral practice.

Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist

Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary.

Developing an international register of clinical prediction rules for use in primary care: a descriptive analysis

PURPOSE

We describe the methodology used to create a register of clinical prediction rules relevant to primary care. We also summarize the rules included in the register according to various characteristics.

METHODS

To identify relevant articles, we searched the MEDLINE database (PubMed) for the years 1980 to 2009 and supplemented the results with searches of secondary sources (books on clinical prediction rules) and personal resources (eg, experts in the field). The rules described in relevant articles were classified according to their clinical domain, the stage of development, and the clinical setting in which they were studied.

RESULTS

Our search identified clinical prediction rules reported between 1965 and 2009. The largest share of rules (37.2%) were retrieved from PubMed. The number of published rules increased substantially over the study decades. We included 745 articles in the register; many contained more than 1 clinical prediction rule study (eg, both a derivation study and a validation study), resulting in 989 individual studies. In all, 434 unique rules had gone through derivation; however, only 54.8% had been validated and merely 2.8% had undergone analysis of their impact on either the process or outcome of clinical care. The rules most commonly pertained to cardiovascular disease, respiratory, and musculoskeletal conditions. They had most often been studied in the primary care or emergency department settings.

CONCLUSIONS

Many clinical prediction rules have been derived, but only about half have been validated and few have been assessed for clinical impact. This lack of thorough evaluation for many rules makes it difficult to retrieve and identify those that are ready for use at the point of patient care. We plan to develop an international web-based register of clinical prediction rules and computer-based clinical decision support systems.

Clinical prediction rules in practice: review of clinical guidelines and survey of GPs

BACKGROUND

The publication of clinical prediction rules (CPRs) studies has risen significantly. It is unclear if this reflects increasing usage of these tools in clinical practice or how this may vary across clinical areas.

AIM

To review clinical guidelines in selected areas and survey GPs in order to explore CPR usefulness in the opinion of experts and use at the point of care.

DESIGN AND SETTING

A review of clinical guidelines and survey of UK GPs.

METHOD

Clinical guidelines in eight clinical domains with published CPRs were reviewed for recommendations to use CPRs including primary prevention of cardiovascular disease, transient ischaemic attack (TIA) and stroke, diabetes mellitus, fracture risk assessment in osteoporosis, lower limb fractures, breast cancer, depression, and acute infections in childhood. An online survey of 401 UK GPs was also conducted.

RESULTS

Guideline review: Of 7637 records screened by title and/or abstract, 243 clinical guidelines met inclusion criteria. CPRs were most commonly recommended in guidelines regarding primary prevention of cardiovascular disease (67%) and depression (67%). There was little consensus across various clinical guidelines as to which CPR to use preferentially.

SURVEY

Of 401 responders to the GP survey, most were aware of and applied named CPRs in the clinical areas of cardiovascular disease and depression. The commonest reasons for using CPRs were to guide management and conform to local policy requirements.

CONCLUSION

GPs use CPRs to guide management but also to comply with local policy requirements. Future research could focus on which clinical areas clinicians would most benefit from CPRs and promoting the use of robust, externally validated CPRs.

Prognostic value of the CAPRA clinical prediction rule: a systematic review and meta-analysis

WHAT'S KNOWN ON THE SUBJECT? AND WHAT DOES THE STUDY ADD?: Prostate cancer is a significant cause of mortality among men. A number of prognostic instruments exist to predict the risk of recurrence among patients with localised prostate cancer. This systematic review examines the totality of evidence in relation to the predictive value of the CAPRA clinical predication rule by combining all studies that validate the rule.

OBJECTIVES

To perform a systematic review with meta-analysis that assesses the 3- and 5-year predictive value of the CAPRA rule, a clinical prediction rule derived to predict biochemical-recurrence-free survival in men with localized prostate cancer after radical prostatectomy. To examine the predictive value of the CAPRA rule at 3 and 5 years stratified by risk group (0-2 low risk, 3-5 intermediate risk, 6-10 high risk).

PATIENTS AND METHODS

A systematic literature search was performed to retrieve papers that validated the CAPRA score. The original derivation study was used as a predictive model and applied to all validation studies with observed and predicted biochemical-recurrence-free survival at 3 and 5 years stratified by risk group (0-2 low, 3-5 intermediate, 6-10 high). Pooled results are presented as risk ratios (RRs) with 95% confidence intervals, in terms of over-prediction (RR > 1) or under-prediction (RR < 1) of biochemical-recurrence-free survival at 3 and 5 years. A chi-squared test for trend was computed to determine if there was a decreasing trend in survival across the three CAPRA risk categories.

RESULTS

Seven validation studies (n = 12 693) predict recurrence-free survival at 5 years after radical prostatectomy. The CAPRA score significantly under-predicts recurrence-free survival across all three risk strata (low risk, RR 0.94, 95% CI 0.90-0.98; intermediate risk, RR 0.94, 95% CI 0.89-0.99; high risk, RR 0.72, 95% CI 0.60-0.85). Data on six studies (n = 6082) are pooled to predict 3-year recurrence-free survival. The CAPRA score correctly predicts recurrence-free survival in all three groups (low risk, RR 0.98, 95% CI 0.95-1.00; intermediate risk, RR 1.03, 95% CI 0.99-1.08; high risk, RR 0.87, 95% CI 0.73-1.05). The chi-squared trend analysis indicates that, as the trichotomized CAPRA score increases, the probability of survival decreases (P < 0.001).

CONCLUSIONS

The results of this pooled analysis confirm the ability of the CAPRA rule to correctly predict biochemical-recurrence-free survival at 3 years after radical prostatectomy. The rule under-predicts recurrence-free survival 5 years after radical prostatectomy across all three strata of risk.

The Alvarado score for predicting acute appendicitis: a systematic review

BACKGROUND

The Alvarado score can be used to stratify patients with symptoms of suspected appendicitis; the validity of the score in certain patient groups and at different cut points is still unclear. The aim of this study was to assess the discrimination (diagnostic accuracy) and calibration performance of the Alvarado score.

METHODS

A systematic search of validation studies in Medline, Embase, DARE and The Cochrane library was performed up to April 2011. We assessed the diagnostic accuracy of the score at the two cut-off points: score of 5 (1 to 4 vs. 5 to 10) and score of 7 (1 to 6 vs. 7 to 10). Calibration was analysed across low (1 to 4), intermediate (5 to 6) and high (7 to 10) risk strata. The analysis focused on three sub-groups: men, women and children.

RESULTS

Forty-two studies were included in the review. In terms of diagnostic accuracy, the cut-point of 5 was good at 'ruling out' admission for appendicitis (sensitivity 99% overall, 96% men, 99% woman, 99% children). At the cut-point of 7, recommended for 'ruling in' appendicitis and progression to surgery, the score performed poorly in each subgroup (specificity overall 81%, men 57%, woman 73%, children 76%). The Alvarado score is well calibrated in men across all risk strata (low RR 1.06, 95% CI 0.87 to 1.28; intermediate 1.09, 0.86 to 1.37 and high 1.02, 0.97 to 1.08). The score over-predicts the probability of appendicitis in children in the intermediate and high risk groups and in women across all risk strata.

CONCLUSIONS

The Alvarado score is a useful diagnostic 'rule out' score at a cut point of 5 for all patient groups. The score is well calibrated in men, inconsistent in children and over-predicts the probability of appendicitis in women across all strata of risk.