A simplified approach to the pooled analysis of calibration of clinical prediction rules for systematic reviews of validation studies

Six Externally Validated Prognostic Models Have Potential Clinical Value to Predict Patient Health Outcomes in the Rehabilitation of Musculoskeletal Conditions: A Systematic Review

OBJECTIVE

The purpose of this systematic review was to identify and appraise externally validated prognostic models to predict a patient's health outcomes relevant to physical rehabilitation of musculoskeletal (MSK) conditions.

METHODS

We systematically reviewed 8 databases and reported our findings according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis 2020. An information specialist designed a search strategy to identify externally validated prognostic models for MSK conditions. Paired reviewers independently screened the title, abstract, and full text and conducted data extraction. We extracted characteristics of included studies (eg, country and study design), prognostic models (eg, performance measures and type of model) and predicted clinical outcomes (eg, pain and disability). We assessed the risk of bias and concerns of applicability using the prediction model risk of bias assessment tool. We proposed and used a 5-step method to determine which prognostic models were clinically valuable.

RESULTS

We found 4896 citations, read 300 full-text articles, and included 46 papers (37 distinct models). Prognostic models were externally validated for the spine, upper limb, lower limb conditions, and MSK trauma, injuries, and pain. All studies presented a high risk of bias. Half of the models showed low concerns for applicability. Reporting of calibration and discrimination performance measures was often lacking. We found 6 externally validated models with adequate measures, which could be deemed clinically valuable [ie, (1) STart Back Screening Tool, (2) Wallis Occupational Rehabilitation RisK model, (3) Da Silva model, (4) PICKUP model, (5) Schellingerhout rule, and (6) Keene model]. Despite having a high risk of bias, which is mostly explained by the very conservative properties of the PROBAST tool, the 6 models remain clinically relevant.

CONCLUSION

We found 6 externally validated prognostic models developed to predict patients' health outcomes that were clinically relevant to the physical rehabilitation of MSK conditions.

IMPACT

Our results provide clinicians with externally validated prognostic models to help them better predict patients' clinical outcomes and facilitate personalized treatment plans. Incorporating clinically valuable prognostic models could inherently improve the value of care provided by physical therapists.

Predicting neurological outcome in adult patients with cardiac arrest: systematic review and meta-analysis of prediction model performance

This work aims to assess the performance of two post-arrest (out-of-hospital cardiac arrest, OHCA, and cardiac arrest hospital prognosis, CAHP) and one pre-arrest (good outcome following attempted resuscitation, GO-FAR) prediction model for the prognostication of neurological outcome after cardiac arrest in a systematic review and meta-analysis. A systematic search was conducted in Embase, Medline, and Web of Science Core Collection from November 2006 to December 2021, and by forward citation tracking of key score publications. The search identified 1'021 records, of which 25 studies with a total of 124'168 patients were included in the review. A random-effects meta-analysis of C-statistics and overall calibration (total observed vs. expected [O:E] ratio) was conducted. Discriminatory performance was good for the OHCA (summary C-statistic: 0.83 [95% CI 0.81-0.85], 16 cohorts) and CAHP score (summary C-statistic: 0.84 [95% CI 0.82-0.87], 14 cohorts) and acceptable for the GO-FAR score (summary C-statistic: 0.78 [95% CI 0.72-0.84], five cohorts). Overall calibration was good for the OHCA (total O:E ratio: 0.78 [95% CI 0.67-0.92], nine cohorts) and the CAHP score (total O:E ratio: 0.78 [95% CI 0.72-0.84], nine cohorts) with an overestimation of poor outcome. Overall calibration of the GO-FAR score was poor with an underestimation of good outcome (total O:E ratio: 1.62 [95% CI 1.28-2.04], five cohorts). Two post-arrest scores showed good prognostic accuracy for predicting neurological outcome after cardiac arrest and may support early discussions about goals-of-care and therapeutic planning on the intensive care unit. A pre-arrest score showed acceptable prognostic accuracy and may support code status discussions.

Mastering Prognostic Tools: An Opportunity to Enhance Personalized Care and to Optimize Clinical Outcomes in Physical Therapy

In health care, clinical decision making is typically based on diagnostic findings. Rehabilitation clinicians commonly rely on pathoanatomical diagnoses to guide treatment and define prognosis. Targeting prognostic factors is a promising way for rehabilitation clinicians to enhance treatment decision-making processes, personalize rehabilitation approaches, and ultimately improve patient outcomes. This can be achieved by using prognostic tools that provide accurate estimates of the probability of future outcomes for a patient in clinical practice. Most literature reviews of prognostic tools in rehabilitation have focused on prescriptive clinical prediction rules. These studies highlight notable methodological issues and conclude that these tools are neither valid nor useful for clinical practice. This has raised the need to open the scope of research to understand what makes a quality prognostic tool that can be used in clinical practice. Methodological guidance in prognosis research has emerged in the last decade, encompassing exploratory studies on the development of prognosis and prognostic models. Methodological rigor is essential to develop prognostic tools, because only prognostic models developed and validated through a rigorous methodological process should guide clinical decision making. This Perspective argues that rehabilitation clinicians need to master the identification and use of prognostic tools to enhance their capacity to provide personalized rehabilitation. It is time for prognosis research to look for prognostic models that were developed and validated following a comprehensive process before being simplified into suitable tools for clinical practice. New models, or rigorous validation of current models, are needed. The approach discussed in this Perspective offers a promising way to overcome the limitations of most models and provide clinicians with quality tools for personalized rehabilitation approaches.

IMPACT

Prognostic research can be applied to clinical rehabilitation; this Perspective proposes solutions to develop high-quality prognostic models to optimize patient outcomes.

Novel approach to meta-analysis of tests and clinical prediction rules with three or more risk categories

OBJECTIVE

Multichotomous tests have three or more outcome or risk categories, and can provide richer information and a better fit with clinical decision-making than dichotomous tests. Our objective is to present a fully developed approach to the meta-analysis of multichotomous clinical prediction rules (CPRs) and tests, including meta-analysis of stratum specific likelihood ratios.

STUDY DESIGN

We have developed a novel approach to the meta-analysis of likelihood ratios for multichotomous tests that avoids the need to dichotomise outcome categories, and demonstrate its application to a sample CPR. We also review previously reported approaches to the meta-analysis of the area under the receiver operating characteristic curve (AUROCC) and meta-analysis of a measure of calibration (observed:expected) for multichotomous tests or CPRs.

RESULTS

Using data from 10 studies of the Cancer of the Prostate Risk Assessment (CAPRA) risk score for prostate cancer recurrence, we calculated summary estimates of the likelihood ratios for low, moderate and high risk groups of 0.40 (95% CI 0.32 to 0.49), 1.24 (95% CI 0.99 to 1.55) and 4.47 (95% CI 3.21 to 6.23), respectively. Applying the summary estimates of the likelihood ratios for each risk group to the overall prevalence of cancer recurrence in a population allows one to estimate the likelihood of recurrence for each risk group in that population.

CONCLUSION

An approach to meta-analysis of multichotomous tests or CPRs is presented. A spreadsheet for data preparation and code for R and Stata are provided for other researchers to download and use. Combined with summary estimates of the AUROCC and calibration, this is a comprehensive strategy for meta-analysis of multichotomous tests and CPRs.

Validation of two risk-prediction models for recurrent falls in the first year after stroke: a prospective cohort study

Background

several multivariable models have been derived to predict post-stroke falls. These require validation before integration into clinical practice. The aim of this study was to externally validate two prediction models for recurrent falls in the first year post-stroke using an Irish prospective cohort study.

Methodology

stroke patients with planned home-discharges from five hospitals were recruited. Falls were recorded with monthly diaries and interviews 6 and 12 months post-discharge. Predictors for falls included in two risk-prediction models were assessed at discharge. Participants were classified into risk groups using these models. Model 1, incorporating inpatient falls history and balance, had a 6-month outcome. Model 2, incorporating inpatient near-falls history and upper limb function, had a 12-month outcome. Measures of calibration, discrimination (area under the curve (AUC)) and clinical utility (sensitivity/specificity) were calculated.

Results

128 participants (mean age = 68.6 years, SD = 13.3) were recruited. The fall status of 117 and 110 participants was available at 6 and 12 months, respectively. Seventeen and 28 participants experienced recurrent falls by these respective time points. Model 1 achieved an AUC = 0.56 (95% CI 0.46-0.67), sensitivity = 18.8% and specificity = 93.6%. Model 2 achieved AUC = 0.55 (95% CI 0.44-0.66), sensitivity = 51.9% and specificity = 58.7%. Model 1 showed no significant difference between predicted and observed events (risk ratio (RR) = 0.87, 95% CI 0.16-4.62). In contrast, model 2 significantly over-predicted fall events in the validation cohort (RR = 1.61, 95% CI 1.04-2.48).

Conclusions

both models showed poor discrimination for predicting recurrent falls. A further large prospective cohort study would be required to derive a clinically useful falls-risk prediction model for a similar population.