Prediction models: the right tool for the right problem

A scoping review of complication prediction models in spinal surgery: An analysis of model development, validation and impact

Background

Predictive analytics are being used increasingly in the field of spinal surgery with the development of models to predict post-surgical complications. Predictive models should be valid, generalizable, and clinically useful. The purpose of this review was to identify existing post-surgical complication prediction models for spinal surgery and to determine if these models are being adequately investigated with internal/external validation, model updating and model impact studies.

Methods

This was a scoping review of studies pertaining to models for the prediction of post-surgical complication after spinal surgery published over 10 years (2010-2020). Qualitative data was extracted from the studies to include study classification, adherence to Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines and risk of bias (ROB) assessment using the Prediction model study Risk Of Bias Assessment Tool (PROBAST). Model evaluation was determined using area under the curve (AUC) when available. The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement was used as a basis for the search methodology in four different databases.

Results

Thirty studies were included in the scoping review and 80% (24/30) included model development with or without internal validation. Twenty percent (6/30) were exclusively external validation studies and only one study included an impact analysis in addition to model development and internal validation. Two studies referenced the TRIPOD guidelines and there was a high ROB in 100% of the studies using the PROBAST tool.

Conclusions

The majority of post-surgical complication prediction models in spinal surgery have not undergone standardized model development and internal validation or adequate external validation and impact evaluation. As such there is uncertainty as to their validity, generalizability, and clinical utility. Future efforts should be made to use existing tools to ensure standardization in development and rigorous evaluation of prediction models in spinal surgery.

Identifying the Target Population for Primary Respiratory Syncytial Virus Two-Step Prevention in Infants: Normative Outcome of Hospitalisation Assessment for Newborns (NOHAN)

Background: Respiratory syncytial virus (RSV) is the leading cause of acute respiratory infection- related hospitalisations in infants (RSVh). Most of these infants are younger than 6 months old with no known risk factors. An efficient RSVh prevention program should address both mothers and infants, relying on Non-Pharmaceutical (NPI) and Pharmaceutical Interventions (PI). This study aimed at identifying the target population for these two interventions. Methods: Laboratory-confirmed RSV-infected infants hospitalised during the first 6 months of life were enrolled from the Hospices Civils de Lyon birth cohort (2014 to 2018). Clinical variables related to pregnancy and birth (sex, month of birth, birth weight, gestational age, parity) were used for descriptive epidemiology, multivariate logistic regression, and predictive score development. Results: Overall, 616 cases of RSVh in 45,648 infants were identified. Being born before the epidemic season, prematurity, and multiparity were independent predictors of RSVh. Infants born in January or June to August with prematurity and multiparity, and those born in September or December with only one other risk factor (prematurity or multiparity) were identified as moderate-risk, identifying the mothers as candidates for a first-level NPI prevention program. Infants born in September or December with prematurity and multiparity, and those born in October or November were identified as high-risk, identifying the mothers and infants as candidates for a second-level (NPI and PI) intervention. Conclusions: It is possible to determine predictors of RSVh at birth, allowing early enrollment of the target population in a two-level RSV prevention intervention.

Which Frailty Evaluation Method Can Better Improve the Predictive Ability of the SASA for Postoperative Complications of Patients Undergoing Elective Abdominal Surgery?

Purpose

To determine which frailty method can better improve the predictive ability of the Surgical Apgar Score combined with American Society of Anesthesiologists physical status classification (SASA).

Patients and Methods

A prospective cohort study was conducted. A total of 194 elderly patients undergoing elective abdominal surgery were included. Preoperative frailty using FRAIL questionnaire, frailty index (FI), Clinical Frailty Scale (CFS) and SASA scores was assessed. Primary outcome was in-hospital Clavien-Dindo ≥grade II complications. Multiple logistic regression was used to examine the association between frailty and complications. Receiver operating characteristic curves were used to explore the predictive ability of frailty.

Results

According to the FRAIL, FI and CFS criteria, the prevalence of frailty in the study population was 43.8%, 32.5%, and 36.6%, respectively. After adjusting for all covariates, frailty was significantly associated with postoperative complications in hospital by FRAIL [odds ratio: 5.11, 95% CI: 1.41–18.44, P = 0.013], by FI [OR: 4.25, 95% CI: 1.21–14.90, P = 0.024] and by CFS [OR: 5.10, 95% CI: 1.52–17.17, P = 0.008]. The area under the curve (AUC) for SASA was 0.768 (95% CI: 0.702–0.826). Addition of frailty assessment (FRAIL, FI and CFS) increased the AUC to 0.787 (95% CI: 0.722–0.842), 0.798 (95% CI: 0.734–0.852), and 0.815 (95% CI: 0.753–0.867), respectively. Compared to SASA, only addition of CFS had a significant difference (P = 0.0478).

Conclusion

Frailty is an effective predictor of postoperative complications in elderly Chinese patients undergoing elective abdominal surgery. Frailty assessment of CFS can better improve the predictive ability of SASA.

How to Develop a Risk Prediction Smartphone App

Purpose. Powered by big data, predictive models provide individualized risk stratification to inform clinical decision-making and mitigate long-term morbidity. We describe how to transform a large institutional dataset into a real-time, interactive clinical decision support mobile user interface for risk prediction. Methods. A clinical decision point ideal for risk stratification and modification was identified. Demographics, medical comorbidities, and operative characteristics were abstracted from the electronic medical record (EMR) using ICD-9 codes. Surgery-specific predictive models were generated using regression modeling and corroborated with internal validation. A clinical support interface was designed in partnership with an app developer, followed by subsequent beta testing and clinical implementation of the final tool. Results. Individual, specialty-specific, and preoperatively actionable models incorporating clustered procedural codes were created. Using longitudinal inpatient, outpatient, and office-based data from a large multicenter health system, all patient and operative variables were weighted according to ß-coefficients. The individual risk model parameters were incorporated into specialty-specific modules and implemented into an accessible iOS/Android compatible mobile application. Conclusions. As proof of concept, we provide a framework for developing a clinical decision support mobile user interface, through the use of clinical and administrative longitudinal data. Point-of-care applications, particularly ones designed with implementation and actionability in mind, have the potential to aid clinicians in identifying and optimizing risk factors that impact the outcome of interest's occurrence, thereby enabling clinicians to take targeted risk-reduction actions. In addition, such applications may help facilitate counseling, informed consent, and shared decision-making, leading to improved patient-centered care.

Using the Causal Inference Framework to Support Individualized Drug Treatment Decisions Based on Observational Healthcare Data

When healthcare professionals have the choice between several drug treatments for their patients, they often experience considerable decision uncertainty because many decisions simply have no single “best” choice. The challenges are manifold and include that guideline recommendations focus on randomized controlled trials whose populations do not necessarily correspond to specific patients in everyday treatment. Further reasons may be insufficient evidence on outcomes, lack of direct comparison of distinct options, and the need to individually balance benefits and risks. All these situations will occur in routine care, its outcomes will be mirrored in routine data, and could thus be used to guide decisions. We propose a concept to facilitate decision-making by exploiting this wealth of information. Our working example for illustration assumes that the response to a particular (drug) treatment can substantially differ between individual patients depending on their characteristics (heterogeneous treatment effects, HTE), and that decisions will be more precise if they are based on real-world evidence of HTE considering this information. However, such methods must account for confounding by indication and effect measure modification, eg, by adequately using machine learning methods or parametric regressions to estimate individual responses to pharmacological treatments. The better a model assesses the underlying HTE, the more accurate are predicted probabilities of treatment response. After probabilities for treatment-related benefit and harm have been calculated, decision rules can be applied and patient preferences can be considered to provide individual recommendations. Emulated trials in observational data are a straightforward technique to predict the effects of such decision rules when applied in routine care. Prediction-based decision rules from routine data have the potential to efficiently supplement clinical guidelines and support healthcare professionals in creating personalized treatment plans using decision support tools.

Development and Validation of a Multivariate Prediction Model of Perioperative Mortality in Neurosurgery: The New Zealand Neurosurgical Risk Tool (NZRISK-NEURO)

BACKGROUND

Multivariate risk prediction models individualize prediction of adverse outcomes, assisting perioperative decision-making. There are currently no models specifically designed for the neurosurgical population.

OBJECTIVE

To develop and validate a neurosurgical risk prediction model, with 30-d, 1-yr, and 2-yr mortality endpoints.

METHODS

We accessed information on all adults in New Zealand who underwent neurosurgery or spinal surgery between July 1, 2011, and June 30, 2016, from an administrative database. Our dataset comprised of 18 375 participants, split randomly into derivation (75%) and validation (25%) datasets. Previously established covariates tested included American Society of Anesthesiologists physical status grade (ASA-PS), surgical acuity, operative severity, cancer status, and age. Exploratory covariates included anatomical site, gender, diabetes, trauma, ethnicity, and socioeconomic status. Least absolute shrinkage and selection operator (LASSO) regression analysis was used to construct 30-d, 1-yr, and 2-yr mortality models.

RESULTS

Our final models included 8 covariates: age, ASA-PS grade, surgical acuity, cancer status, anatomical site, diabetes, ethnicity, and trauma. The area under the receiver operating curve for the 30-d, 1-yr, and 2-yr mortality models was 0.90, 0.91, and 0.91 indicating excellent discrimination, respectively. Calibration also showed excellent performance with McFadden's pseudo R2 statistics of 0.28, 0.37, and 0.41 and calibration plot slopes of 0.93, 0.95, and 0.94, respectively. The strongest predictors of mortality were ASA-PS 4 and 5 (30 d) and cancer (1 and 2 yr).

CONCLUSION

NZRISK-NEURO is a robust multivariate calculator created specifically for neurosurgery, enabling physicians to generate data-driven individualized risk estimates, assisting shared decision-making and perioperative planning.

Imprecise Data and Their Impact on Translational Research in Medicine

The medical field expects from big data essentially two main results: the ability to build predictive models and the possibility of applying them to obtain accurate patient risk profiles and/or health trajectories. Note that the paradigm of precision has determined that similar challenges need to be faced in both population and individualized studies, namely the need of assembling, integrating, modeling, and interpreting data from a variety of information sources and scales potentially influencing disease from onset to progression. In many cases, data require computational treatment through solutions for otherwise intractable problems. However, as precision medicine remains subject to a substantial amount of data imprecision and lack of translational impact, a revision of methodological inference approaches is needed. Both the relevance and the usefulness of such revision crucially deal with the assimilation of data features dynamically interconnected.

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

Frailty and postoperative complications in older Chinese adults undergoing major thoracic and abdominal surgery

Purpose: To determine the association between frailty and postoperative complications in elderly Chinese patients and to determine whether addition of frailty assessment improves the predictive ability of the American Society of Anesthesiologists (ASA) score, Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (POSSUM), and Estimation of Physiologic Ability and Surgical Stress (E-PASS) score.

Patients and methods: A prospective cohort study was conducted in a tertiary hospital. Elderly patients undergoing major thoracic or abdominal surgery were included. Frailty phenotype and ASA, POSSUM, and E-PASS scores were assessed. Demographic, preoperative, and surgical variables were extracted from medical records. Primary outcome measure was in-hospital Clavien-Dindo ≥ grade II complications. Multiple logistic regression was used to examine the association between frailty and complications. Receiver operating characteristic curves were used to explore the predictive ability of frailty.

Results: Prevalence of frailty was 26.12%. Significant differences were observed between the frail and non-frail groups with respect to age, Activities of Daily Living, Charlson Comorbidity Index, respiratory function, presence of malignancy, serum albumin, prealbumin, and hemoglobin levels (P<0.05). ASA, POSSUM, and E-PASS scores were higher in the frail group. After adjusting for all covariates, frailty was significantly associated with postoperative complications in hospital [odds ratio: 16.59, 95% CI: 4.56–60.40, P<0.001]. The area under the curve (AUC) for frailty was 0.762 (95% CI: 0.703–0.814). The AUC for ASA, POSSUM, and E-PASS for prediction of complications was 0.751 (95% CI: 0.692–0.804), 0.762 (95% CI: 0.704–0.814), and 0.824 (95% CI: 0.771–0.870), respectively. Addition of frailty assessment increased the AUC to 0.858 (95% CI: 0.808–0.899), 0.842 (95% CI: 0.790–0.885), and 0.854 (95% CI: 0.803–0.896), respectively.

Conclusion: Frailty is an effective predictor of postoperative complications in elderly Chinese patients undergoing major thoracic and abdominal surgery. Frailty assessment can improve the predictive ability of current surgical risk assessment tools. Frailty phenotype should be considered perioperatively. Frailty assessment could also expand the scope for nurses to evaluate patients for safety management.

A Human(e) Factor in Clinical Decision Support Systems

The overwhelming amount, production speed, multidimensionality, and potential value of data currently available—often simplified and referred to as big data —exceed the limits of understanding of the human brain. At the same time, developments in data analytics and computational power provide the opportunity to obtain new insights and transfer data-provided added value to clinical practice in real time. What is the role of the health care professional in collaboration with the data scientist in the changing landscape of modern care? We discuss how health care professionals should provide expert knowledge in each of the stages of clinical decision support design: data level, algorithm level, and decision support level. Including various ethical considerations, we advocate for health care professionals to responsibly initiate and guide interprofessional teams, including patients, and embrace novel analytic technologies to translate big data into patient benefit driven by human(e) values.

Evaluating the impact of prediction models: lessons learned, challenges, and recommendations

An important aim of clinical prediction models is to positively impact clinical decision making and subsequent patient outcomes. The impact on clinical decision making and patient outcome can be quantified in prospective comparative-ideally cluster-randomized-studies, known as 'impact studies'. However, such impact studies often require a lot of time and resources, especially when they are (cluster-)randomized studies. Before envisioning such large-scale randomized impact study, it is important to ensure a reasonable chance that the use of the prediction model by the targeted healthcare professionals and patients will indeed have a positive effect on both decision making and subsequent outcomes. We recently performed two differently designed, prospective impact studies on a clinical prediction model to be used in surgical patients. Both studies taught us new valuable lessons on several aspects of prediction model impact studies, and which considerations may guide researchers in their decision to conduct a prospective comparative impact study. We provide considerations on how to prepare a prediction model for implementation in practice, how to present the model predictions, and how to choose the proper design for a prediction model impact study.