Understanding the usefulness of prognostic models in clinical decision-making

Enabling knowledge translation: implementation of a web-based tool for independent walking prediction after traumatic spinal cord injury

Introduction

Several clinical prediction rules (CPRs) have been published, but few are easily accessible or convenient for clinicians to use in practice. We aimed to develop, implement, and describe the process of building a web-based CPR for predicting independent walking 1-year after a traumatic spinal cord injury (TSCI).

Methods

Using the published and validated CPR, a front-end web application called "Ambulation" was built using HyperText Markup Language (HTML), Cascading Style Sheets (CSS), and JavaScript. A survey was created using QualtricsXM Software to gather insights on the application's usability and user experience. Website activity was monitored using Google Analytics. Ambulation was developed with a core team of seven clinicians and researchers. To refine the app's content, website design, and utility, 20 professionals from different disciplines, including persons with lived experience, were consulted.

Results

After 11 revisions, Ambulation was uploaded onto a unique web domain and launched (www.ambulation.ca) as a pilot with 30 clinicians (surgeons, physiatrists, and physiotherapists). The website consists of five web pages: Home, Calculation, Team, Contact, and Privacy Policy. Responses from the user survey (n = 6) were positive and provided insight into the usability of the tool and its clinical utility (e.g., helpful in discharge planning and rehabilitation), and the overall face validity of the CPR. Since its public release on February 7, 2022, to February 28, 2023, Ambulation had 594 total users, 565 (95.1%) new users, 26 (4.4%) returning users, 363 (61.1%) engaged sessions (i.e., the number of sessions that lasted 10 seconds/longer, had one/more conversion events e.g., performing the calculation, or two/more page or screen views), and the majority of the users originating from the United States (39.9%) and Canada (38.2%).

Discussion

Ambulation is a CPR for predicting independent walking 1-year after TSCI and it can assist frontline clinicians with clinical decision-making (e.g., time to surgery or rehabilitation plan), patient education and goal setting soon after injury. This tool is an example of adapting a validated CPR for independent walking into an easily accessible and usable web-based tool for use in clinical practice. This study may help inform how other CPRs can be adopted into clinical practice.

Six Externally Validated Prognostic Models Have Potential Clinical Value to Predict Patient Health Outcomes in the Rehabilitation of Musculoskeletal Conditions: A Systematic Review

OBJECTIVE

The purpose of this systematic review was to identify and appraise externally validated prognostic models to predict a patient's health outcomes relevant to physical rehabilitation of musculoskeletal (MSK) conditions.

METHODS

We systematically reviewed 8 databases and reported our findings according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis 2020. An information specialist designed a search strategy to identify externally validated prognostic models for MSK conditions. Paired reviewers independently screened the title, abstract, and full text and conducted data extraction. We extracted characteristics of included studies (eg, country and study design), prognostic models (eg, performance measures and type of model) and predicted clinical outcomes (eg, pain and disability). We assessed the risk of bias and concerns of applicability using the prediction model risk of bias assessment tool. We proposed and used a 5-step method to determine which prognostic models were clinically valuable.

RESULTS

We found 4896 citations, read 300 full-text articles, and included 46 papers (37 distinct models). Prognostic models were externally validated for the spine, upper limb, lower limb conditions, and MSK trauma, injuries, and pain. All studies presented a high risk of bias. Half of the models showed low concerns for applicability. Reporting of calibration and discrimination performance measures was often lacking. We found 6 externally validated models with adequate measures, which could be deemed clinically valuable [ie, (1) STart Back Screening Tool, (2) Wallis Occupational Rehabilitation RisK model, (3) Da Silva model, (4) PICKUP model, (5) Schellingerhout rule, and (6) Keene model]. Despite having a high risk of bias, which is mostly explained by the very conservative properties of the PROBAST tool, the 6 models remain clinically relevant.

CONCLUSION

We found 6 externally validated prognostic models developed to predict patients' health outcomes that were clinically relevant to the physical rehabilitation of MSK conditions.

IMPACT

Our results provide clinicians with externally validated prognostic models to help them better predict patients' clinical outcomes and facilitate personalized treatment plans. Incorporating clinically valuable prognostic models could inherently improve the value of care provided by physical therapists.

Validation of the German version of the STarT-MSK-Tool: A cohort study with patients from physiotherapy clinics

BACKGROUND

The STarT-MSK-Tool is an adaptation of the well established STarT-Back-Tool, used to risk-stratify patients with a wider range of musculoskeletal presentations.

OBJECTIVE

To formally translate and cross-culturally adapt the Keele STarT-MSK risk stratification tool into German (STarT-MSKG) and to establish its reliability and validity.

METHODS

A formal, multi-step, forward and backward translation approach was used. To assess validity patients aged ≥18 years, with acute, subacute or chronic musculoskeletal presentations in the lumbar spine, hip, knee, shoulder, or neck were included. The prospective cohort was used with initial data collected electronically at the point-of-consultation. Retest and 6-month follow-up questionnaires were sent by email. Test-retest reliability, construct validity, discriminative ability, predictive ability and floor or ceiling effects were analysed using intraclass correlation coefficient, and comparisons with a reference standard (Orebro-Musculoskeletal-Pain-Questionnaire: OMPQ) using correlations, ROC-curves and regression models.

RESULTS

The participants' (n = 287) mean age was 47 (SD = 15.8) years, 51% were female, with 48.8% at low, 43.6% at medium, and 7.7% at high risk. With ICC = 0.75 (95% CI 0.69; 0.81) test-retest-reliability was good. Construct validity was good with correlations for the STarT-MSKG-Tool against the OMPQ-Tool of rs = 0.74 (95% CI 0.68, 0.79). The ability of the tool [comparison OMPQ] to predict 6-month pain and disability was acceptable with AUC = 0.77 (95% CI 0.71, 0.83) [OMPQ = 0.74] and 0.76 (95% CI 0.69, 0.82) [OMPQ = 0.72] respectively. However, the explained variance (linear/logistic regression) for predicting 6-month pain (21% [OMPQ = 17%]/logistic = 29%) and disability (linear = 20%:[OMPQ = 19%]/logistic = 26%), whilst being comparable to the existing OMPQ reference standard, fell short of the a priori target of ≥30%.

CONCLUSIONS

The German version of the STarT-MSK-Tool is a valid instrument for use across multiple musculoskeletal conditions and is availabe for use in clinical practice. Comparison with the OMPQ suggests it is a good alternative.

Challenges and solutions in prognostic prediction models in spinal disorders

Methodological shortcomings in prognostic modeling for patients with spinal disorders are highly common. This general commentary discusses methodological challenges related to the specific nature of this field. Five specific methodological challenges in prognostic modeling for patients with spinal disorders are presented with their potential solutions, as related to the choice of study participants, purpose of studies, limitations in measurements of outcomes and predictors, complexity of recovery predictions, and confusion of prognosis and treatment response. Large studies specifically designed for prognostic model research are needed, using standard baseline measurement sets, clearly describing participants' recruitment and accounting and correcting for measurement limitations.

Meta-analysis of predictive models to assess the clinical validity and utility for patient-centered medical decision making: application to the CAncer of the Prostate Risk Assessment (CAPRA)

BACKGROUND

The Cancer of the Prostate Risk Assessment (CAPRA) score was designed and validated several times to predict the biochemical recurrence-free survival after a radical prostatectomy. Our objectives were, first, to study the clinical validity of the CAPRA score, and, second, to assess its clinical utility for stratified medicine from an original patient-centered approach.

METHODS

We proposed a meta-analysis based on a literature search using MEDLINE. Observed and predicted biochemical-recurrence-free survivals were compared to assess the calibration of the CAPRA score. Discriminative capacities were evaluated by estimating the summary time-dependent ROC curve. The clinical utility of the CAPRA score was evaluated according to the following stratified decisions: active monitoring for low-risk patients, prostatectomy for intermediate-risk patients, or radio-hormonal therapy for high risk patients. For this purpose, we assessed CAPRA's clinical utility in terms of its ability to maximize time-dependent utility functions (i.e. Quality-Adjusted Life-Years - QALYs).

RESULTS

We identified 683 manuscripts and finally retained 9 studies. We reported good discriminative capacities with an area under the SROCt curve at 0.73 [95%CI from 0.67 to 0.79], while graphical calibration seemed acceptable. Nevertheless, we also described that the CAPRA score was unable to discriminate between the three medical alternatives, i.e. it did not allow an increase in the number of life years in perfect health (QALYs) of patients with prostate cancer.

CONCLUSIONS

We confirmed the prognostic capacities of the CAPRA score. In contrast, we were not able to demonstrate its clinical usefulness for stratified medicine from a patient-centered perspective. Our results also highlighted the confusion between clinical validity and utility. This distinction should be better considered in order to develop predictive tools useful in practice.

Comparison of Prognostic Models in Acute Liver Failure: Decision is to be Dynamic

Background and aims

Acute liver failure (ALF) is a rare disease entity with a high mortality. Management is dependent on accurate prognostication.

Materials and methods

One hundred consecutive patients presenting with ALF were prospectively evaluated. The King's college criteria (KCC), ALF early dynamic model (ALFED), sequential organ failure assessment (SOFA) score, and acute physiology and health evaluation II (APACHE II) scores were compared to predict mortality.

Results

There were significant differences in means of all the scores between survivors and nonsurvivors. The SOFA 48 hours had the highest area under receiver operating characteristic curve (AUC) (0.857) closely followed by the ALFED score (0.844). The optimal cutoff for the SOFA score at 48 hours to predict subsequent survival outcome is ≥10 and for the ALFED score is ≥5. Sequential organ failure assessment 48 hours had a good sensitivity of 87%, and the ALFED score showed a good specificity of 84%. The decision curve analysis showed that between a threshold probability of 0.13 and 0.6, use of the SOFA score provided the maximum net benefit and at threshold probabilities of >0.6, the use of ALFED score provided the maximum clinical benefit.

Conclusion

Dynamic scoring results in better prognostication in ALF. The SOFA 48 hours and ALFED score have good prognostication value in nonacetaminophen-induced liver failure.

How to cite this article

Saluja V, Sharma A, Pasupuleti SSR, Mitra LG, Kumar G, Agarwal PM. Comparison of Prognostic Models in Acute Liver Failure: Decision is to be Dynamic. Indian J Crit Care Med 2019;23(12):574–581.

A systematic review of neonatal treatment intensity scores and their potential application in low-resource setting hospitals for predicting mortality, morbidity and estimating resource use

BACKGROUND

Treatment intensity scores can predict mortality and estimate resource use. They may therefore be of interest for essential neonatal care in low resource settings where neonatal mortality remains high. We sought to systematically review neonatal treatment intensity scores to (1) assess the level of evidence on predictive performance in predicting clinical outcomes and estimating resource utilisation and (2) assess the applicability of the identified models to decision making for neonatal care in low resource settings.

METHODS

We conducted a systematic search of PubMed, EMBASE (OVID), CINAHL, Global Health Library (Global index, WHO) and Google Scholar to identify studies published up until 21 December 2016. Included were all articles that used treatments as predictors in neonatal models. Individual studies were appraised using the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS). In addition, Grading of Recommendations Assessment, Development, and Evaluation (GRADE) was used as a guiding framework to assess certainty in the evidence for predicting outcomes across studies.

RESULTS

Three thousand two hundred forty-nine articles were screened, of which ten articles were included in the review. All of the studies were conducted in neonatal intensive care units with sample sizes ranging from 22 to 9978, with a median of 163. Two articles reported model development, while eight reported external application of existing models to new populations. Meta-analysis was not possible due heterogeneity in the conduct and reporting of the identified studies. Discrimination as assessed by area under receiver operating characteristic curve was reported for in-hospital mortality, median 0.84 (range 0.75-0.96, three studies), early adverse outcome and late adverse outcome (0.78 and 0.59, respectively, one study).

CONCLUSION

Existing neonatal treatment intensity models show promise in predicting mortality and morbidity. There is however low certainty in the evidence on their performance in essential neonatal care in low resource settings as all studies had methodological limitations and were conducted in intensive care. The approach may however be developed further for low resource settings like Kenya because treatment data may be easier to obtain compared to measures of physiological status.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42016034205.