Validation of clinical prediction models: what does the "calibration slope" really measure?

Externally validated treatment algorithm acceptably predicts nonoperative treatment success in patients with anterior cruciate ligament rupture

PURPOSE

This study aims to develop and externally validate a treatment algorithm to predict nonoperative treatment success or failure in patients with anterior cruciate ligament (ACL) rupture.

METHODS

Data were used from two completed studies of adult patients with ACL ruptures: the Conservative versus Operative Methods for Patients with ACL Rupture Evaluation study (development cohort) and the KNee osteoArthritis anterior cruciate Ligament Lesion study (validation cohort). The primary outcome variable is nonoperative treatment success or failure. Potential predictor variables were collected, entered into the univariable logistic regression model and then incorporated into the multivariable logistic regression model for constructing the treatment algorithm. Finally, predictive performance and goodness-of-fit were assessed and externally validated by discrimination and calibration measures.

RESULTS

In the univariable logistic regression model, a stable knee measured with the pivot shift test and a posttrauma International Knee Documentation Committee (IKDC) score <50 were predictive of needing an ACL reconstruction. Age >30 years and a body mass index > 30 kg/m2 were predictive for not needing an ACL reconstruction. Age, pretrauma Tegner score, the outcome of the pivot shift test and the posttrauma IKDC score are entered into the treatment algorithm. The predictability of needing an ACL reconstruction after nonoperative treatment (discrimination) is acceptable in both the development and the validation cohort: area under the curve = resp. 0.69 (95% confidence interval [CI]: 0.58-0.81) and 0.68 (95% CI: 0.58-0.78).

CONCLUSION

This study shows that the treatment algorithm can acceptably predict whether an ACL injury patient will have a(n) (un)successful nonoperative treatment (discrimination). Calibration of the treatment algorithm suggests a systematical underestimation of the need for ACL reconstruction. Given the limitations regarding the sample size of this study, larger data sets must be constructed to improve the treatment algorithm further.

LEVEL OF EVIDENCE

Level II.

Evaluating individualized treatment effect predictions: A model-based perspective on discrimination and calibration assessment

In recent years, there has been a growing interest in the prediction of individualized treatment effects. While there is a rapidly growing literature on the development of such models, there is little literature on the evaluation of their performance. In this paper, we aim to facilitate the validation of prediction models for individualized treatment effects. The estimands of interest are defined based on the potential outcomes framework, which facilitates a comparison of existing and novel measures. In particular, we examine existing measures of discrimination for benefit (variations of the c-for-benefit), and propose model-based extensions to the treatment effect setting for discrimination and calibration metrics that have a strong basis in outcome risk prediction. The main focus is on randomized trial data with binary endpoints and on models that provide individualized treatment effect predictions and potential outcome predictions. We use simulated data to provide insight into the characteristics of the examined discrimination and calibration statistics under consideration, and further illustrate all methods in a trial of acute ischemic stroke treatment. The results show that the proposed model-based statistics had the best characteristics in terms of bias and accuracy. While resampling methods adjusted for the optimism of performance estimates in the development data, they had a high variance across replications that limited their accuracy. Therefore, individualized treatment effect models are best validated in independent data. To aid implementation, a software implementation of the proposed methods was made available in R.

Generalizability of machine learning models predicting 30-day unplanned readmission after primary total knee arthroplasty using a nationally representative database

Unplanned readmission after primary total knee arthroplasty (TKA) costs an average of US $39,000 per episode and negatively impacts patient outcomes. Although predictive machine learning (ML) models show promise for risk stratification in specific populations, existing studies do not address model generalizability. This study aimed to establish the generalizability of previous institutionally developed ML models to predict 30-day readmission following primary TKA using a national database. Data from 424,354 patients from the ACS-NSQIP database was used to develop and validate four ML models to predict 30-day readmission risk after primary TKA. Individual model performance was assessed and compared based on discrimination, accuracy, calibration, and clinical utility. Length of stay (> 2.5 days), body mass index (BMI) (> 33.21 kg/m2), and operation time (> 93 min) were important determinants of 30-day readmission. All ML models demonstrated equally good accuracy, calibration, and discriminatory ability (Brier score, ANN = RF = HGB = NEPLR = 0.03; ANN, slope = 0.90, intercept =  - 0.11; RF, slope = 0.93, intercept =  - 0.12; HGB, slope = 0.90, intercept =  - 0.12; NEPLR, slope = 0.77, intercept = 0.01; AUCANN = AUCRF = AUCHGB = AUCNEPLR = 0.78). This study validates the generalizability of four previously developed ML algorithms in predicting readmission risk in patients undergoing TKA and offers surgeons an opportunity to reduce readmissions by optimizing discharge planning, BMI, and surgical efficiency.

Predicting 30-day unplanned hospital readmission after revision total knee arthroplasty: machine learning model analysis of a national patient cohort

Revision total knee arthroplasty (TKA) is associated with a higher risk of readmission than primary TKA. Identifying individual patients predisposed to readmission can facilitate proactive optimization and increase care efficiency. This study developed machine learning (ML) models to predict unplanned readmission following revision TKA using a national-scale patient dataset. A total of 17,443 revision TKA cases (2013-2020) were acquired from the ACS NSQIP database. Four ML models (artificial neural networks, random forest, histogram-based gradient boosting, and k-nearest neighbor) were developed on relevant patient variables to predict readmission following revision TKA. The length of stay, operation time, body mass index (BMI), and laboratory test results were the strongest predictors of readmission. Histogram-based gradient boosting was the best performer in distinguishing readmission (AUC: 0.95) and estimating the readmission probability for individual patients (calibration slope: 1.13; calibration intercept: -0.00; Brier score: 0.064). All models produced higher net benefit than the default strategies of treating all or no patients, supporting the clinical utility of the models. ML demonstrated excellent performance for the prediction of readmission following revision TKA. Optimization of important predictors highlighted by our model may decrease preventable hospital readmission following surgery, thereby leading to reduced financial burden and improved patient satisfaction.

Associations between BMI and hospital resource use in patients hospitalised for COVID-19 in England: a community-based cohort study

BACKGROUND

Excess weight is a major risk factor for severe disease after infection with SARS-CoV-2. However, the effect of BMI on COVID-19 hospital resource use has not been fully quantified. This study aimed to identify the association between BMI and hospital resource use for COVID-19 admissions with the intention of informing future national hospital resource allocation.

METHODS

In this community-based cohort study, we analysed patient-level data from 57 415 patients admitted to hospital in England with COVID-19 between April 1, 2020, and Dec 31, 2021. Patients who were aged 20-99 years, had been registered with a general practitioner (GP) surgery that contributed to the QResearch database for the whole preceding year (2019) with at least one BMI value measured before April 1, 2020, available in their GP record, and were admitted to hospital for COVID-19 were included. Outcomes of interest were duration of hospital stay, transfer to an intensive care unit (ICU), and duration of ICU stay. Costs of hospitalisation were estimated from these outcomes. Generalised linear and logit models were used to estimate associations between BMI and hospital resource use outcomes.

FINDINGS

Patients living with obesity (BMI >30·0 kg/m2) had longer hospital stays relative to patients in the reference BMI group (18·5-25·0 kg/m2; IRR 1·07, 95% CI 1·03-1·10); the reference group had a mean length of stay of 8·82 days (95% CI 8·62-9·01). Patients living with obesity were more likely to be admitted to ICU than the reference group (OR 2·02, 95% CI 1·86-2·19); the reference group had a mean probability of ICU admission of 5·9% (95% CI 5·5-6·3). No association was found between BMI and duration of ICU stay. The mean cost of COVID-19 hospitalisation was £19 877 (SD 17 918) in the reference BMI group. Hospital costs were estimated to be £2736 (95% CI 2224-3248) higher for patients living with obesity.

INTERPRETATION

Patients admitted to hospital with COVID-19 with a BMI above the healthy range had longer stays, were more likely to be admitted to ICU, and had higher health-care costs associated with hospital treatment of COVID-19 infection as a result. This information can inform national resource allocation to match hospital capacity to areas where BMI profiles indicate higher demand.

FUNDING

National Institute for Health Research.

Prediction of 30-Day Mortality Following Revision Total Hip and Knee Arthroplasty: Machine Learning Algorithms Outperform CARDE-B, 5-Item, and 6-Item Modified Frailty Index Risk Scores

BACKGROUND

Although risk calculators are used to prognosticate postoperative outcomes following revision total hip and knee arthroplasty (total joint arthroplasty [TJA]), machine learning (ML) based predictive tools have emerged as a promising alternative for improved risk stratification. This study aimed to compare the predictive ability of ML models for 30-day mortality following revision TJA to that of traditional risk-assessment indices such as the CARDE-B score (congestive heart failure, albumin (< 3.5 mg/dL), renal failure on dialysis, dependence for daily living, elderly (> 65 years of age), and body mass index (BMI) of < 25 kg/m2), 5-item modified frailty index (5MFI), and 6MFI.

METHODS

Adult patients undergoing revision TJA between 2013 and 2020 were selected from the American College of Surgeons National Surgical Quality Improvement Program database and randomly split 80:20 to compose the training and validation cohorts. There were 3 ML models - extreme gradient boosting, random forest, and elastic-net penalized logistic regression (NEPLR) - that were developed and evaluated using discrimination, calibration metrics, and accuracy. The discrimination of CARDE-B, 5MFI, and 6MFI scores was assessed individually and compared to that of ML models.

RESULTS

All models were equally accurate (Brier score = 0.005) and demonstrated outstanding discrimination with similar areas under the receiver operating characteristic curve (AUCs, extreme gradient boosting = 0.94, random forest = NEPLR = 0.93). The NEPLR was the best-calibrated model overall (slope = 0.54, intercept = -0.004). The CARDE-B had the highest discrimination among the scores (AUC = 0.89), followed by 6MFI (AUC = 0.80), and 5MFI (AUC = 0.68). Albumin < 3.5 mg/dL and BMI (< 30.15) were the most important predictors of 30-day mortality following revision TJA.

CONCLUSIONS

The ML models outperform traditional risk-assessment indices in predicting postoperative 30-day mortality after revision TJA. Our findings highlight the utility of ML for risk stratification in a clinical setting. The identification of hypoalbuminemia and BMI as prognostic markers may allow patient-specific perioperative optimization strategies to improve outcomes following revision TJA.

Prediction of 24-Hour Urinary Sodium Excretion Using Machine-Learning Algorithms

BACKGROUND

Accurate quantification of sodium intake based on self-reported dietary assessments has been a persistent challenge. We aimed to apply machine-learning (ML) algorithms to predict 24-hour urinary sodium excretion from self-reported questionnaire information.

METHODS AND RESULTS

We analyzed 3454 participants from the NHS (Nurses' Health Study), NHS-II (Nurses' Health Study II), and HPFS (Health Professionals Follow-Up Study), with repeated measures of 24-hour urinary sodium excretion over 1 year. We used an ensemble approach to predict averaged 24-hour urinary sodium excretion using 36 characteristics. The TOHP-I (Trial of Hypertension Prevention I) was used for the external validation. The final ML algorithms were applied to 167 920 nonhypertensive adults with 30-year follow-up to estimate confounder-adjusted hazard ratio (HR) of incident hypertension for predicted sodium. Averaged 24-hour urinary sodium excretion was better predicted and calibrated with ML compared with the food frequency questionnaire (Spearman correlation coefficient, 0.51 [95% CI, 0.49-0.54] with ML; 0.19 [95% CI, 0.16-0.23] with the food frequency questionnaire; 0.46 [95% CI, 0.42-0.50] in the TOHP-I). However, the prediction heavily depended on body size, and the prediction of energy-adjusted 24-hour sodium excretion was modestly better using ML. ML-predicted sodium was modestly more strongly associated than food frequency questionnaire-based sodium in the NHS-II (HR comparing Q5 versus Q1, 1.48 [95% CI, 1.40-1.56] with ML; 1.04 [95% CI, 0.99-1.08] with the food frequency questionnaire), but no material differences were observed in the NHS or HPFS.

CONCLUSIONS

The present ML algorithm improved prediction of participants' absolute 24-hour urinary sodium excretion. The present algorithms may be a generalizable approach for predicting absolute sodium intake but do not substantially reduce the bias stemming from measurement error in disease associations.

Visualising and quantifying the usefulness of new predictors stratified by outcome class: The U-smile method

Binary classification methods encompass various algorithms to categorize data points into two distinct classes. Binary prediction, in contrast, estimates the likelihood of a binary event occurring. We introduce a novel graphical and quantitative approach, the U-smile method, for assessing prediction improvement stratified by binary outcome class. The U-smile method utilizes a smile-like plot and novel coefficients to measure the relative and absolute change in prediction compared with the reference method. The likelihood-ratio test was used to assess the significance of the change in prediction. Logistic regression models using the Heart Disease dataset and generated random variables were employed to validate the U-smile method. The receiver operating characteristic (ROC) curve was used to compare the results of the U-smile method. The likelihood-ratio test demonstrated that the proposed coefficients consistently generated smile-shaped U-smile plots for the most informative predictors. The U-smile plot proved more effective than the ROC curve in comparing the effects of adding new predictors to the reference method. It effectively highlighted differences in model performance for both non-events and events. Visual analysis of the U-smile plots provided an immediate impression of the usefulness of different predictors at a glance. The U-smile method can guide the selection of the most valuable predictors. It can also be helpful in applications beyond prediction.

Performance of a Claims-Based Frailty Proxy Using Varying Frailty Ascertainment Lookback Windows

BACKGROUND

Frailty is an aging-related syndrome of reduced physiological reserve to maintain homeostasis. The Faurot frailty index has been validated as a Medicare claims-based proxy for predicting frailty using billing information from a user-specified ascertainment window.

OBJECTIVES

We assessed the validity of the Faurot frailty index as a predictor of the frailty phenotype and 1-year mortality using varying frailty ascertainment windows.

RESEARCH DESIGN

We identified older adults (66+ y) in Round 5 (2015) of the National Health and Aging Trends Study with Medicare claims linkage. Gold standard frailty was assessed using the frailty phenotype. We calculated the Faurot frailty index using 3, 6, 8, and 12 months of claims prior to the survey or all-available lookback. Model performance for each window in predicting the frailty phenotype was assessed by quantifying calibration and discrimination. Predictive performance for 1-year mortality was assessed by estimating risk differences across claims-based frailty strata.

RESULTS

Among 4253 older adults, the 6 and 8-month windows had the best frailty phenotype calibration (calibration slopes: 0.88 and 0.87). All-available lookback had the best discrimination (C-statistic=0.780), but poor calibration. Mortality associations were strongest using a 3-month window and monotonically decreased with longer windows. Subgroup analyses revealed worse performance in Black and Hispanic individuals than counterparts.

CONCLUSIONS

The optimal ascertainment window for the Faurot frailty index may depend on the clinical context, and researchers should consider tradeoffs between discrimination, calibration, and mortality. Sensitivity analyses using different durations can enhance the robustness of inferences. Research is needed to improve prediction across racial and ethnic groups.

Externally validated prediction models for pre-eclampsia: systematic review and meta-analysis

OBJECTIVE

This systematic review and meta-analysis aimed to evaluate the performance of existing externally validated prediction models for pre-eclampsia (PE) (specifically, any-onset, early-onset, late-onset and preterm PE).

METHODS

A systematic search was conducted in five databases (MEDLINE, EMBASE, Emcare, CINAHL and Maternity & Infant Care Database) and using Google Scholar/reference search to identify studies based on the Population, Index prediction model, Comparator, Outcome, Timing and Setting (PICOTS) approach until 20 May 2023. We extracted data using the CHARMS checklist and appraised the risk of bias using the PROBAST tool. A meta-analysis of discrimination and calibration performance was conducted when appropriate.

RESULTS

Twenty-three studies reported 52 externally validated prediction models for PE (one preterm, 20 any-onset, 17 early-onset and 14 late-onset PE models). No model had the same set of predictors. Fifteen any-onset PE models were validated externally once, two were validated twice and three were validated three times, while the Fetal Medicine Foundation (FMF) competing-risks model for preterm PE prediction was validated widely in 16 different settings. The most common predictors were maternal characteristics (prepregnancy body mass index, prior PE, family history of PE, chronic medical conditions and ethnicity) and biomarkers (uterine artery pulsatility index and pregnancy-associated plasma protein-A). The FMF model for preterm PE (triple test plus maternal factors) had the best performance, with a pooled area under the receiver-operating-characteristics curve (AUC) of 0.90 (95% prediction interval (PI), 0.76-0.96), and was well calibrated. The other models generally had poor-to-good discrimination performance (median AUC, 0.66 (range, 0.53-0.77)) and were overfitted on external validation. Apart from the FMF model, only two models that were validated multiple times for any-onset PE prediction, which were based on maternal characteristics only, produced reasonable pooled AUCs of 0.71 (95% PI, 0.66-0.76) and 0.73 (95% PI, 0.55-0.86).

CONCLUSIONS

Existing externally validated prediction models for any-, early- and late-onset PE have limited discrimination and calibration performance, and include inconsistent input variables. The triple-test FMF model had outstanding discrimination performance in predicting preterm PE in numerous settings, but the inclusion of specialized biomarkers may limit feasibility and implementation outside of high-resource settings. © 2023 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Imprinted gene detection effectively improves the diagnostic accuracy for papillary thyroid carcinoma

BACKGROUND

Papillary thyroid carcinoma (PTC) is the most frequent histological type of thyroid carcinoma. Although an increasing number of diagnostic methods have recently been developed, the diagnosis of a few nodules is still unsatisfactory. Therefore, the present study aimed to develop and validate a comprehensive prediction model to optimize the diagnosis of PTC.

METHODS

A total of 152 thyroid nodules that were evaluated by postoperative pathological examination were included in the development and validation cohorts recruited from two centres between August 2019 and February 2022. Patient data, including general information, cytopathology, imprinted gene detection, and ultrasound features, were obtained to establish a prediction model for PTC. Multivariate logistic regression analysis with a bidirectional elimination approach was performed to identify the predictors and develop the model.

RESULTS

A comprehensive prediction model with predictors, such as component, microcalcification, imprinted gene detection, and cytopathology, was developed. The area under the curve (AUC), sensitivity, specificity, and accuracy of the developed model were 0.98, 97.0%, 89.5%, and 94.4%, respectively. The prediction model also showed satisfactory performance in both internal and external validations. Moreover, the novel method (imprinted gene detection) was demonstrated to play a role in improving the diagnosis of PTC.

CONCLUSION

The present study developed and validated a comprehensive prediction model for PTC, and a visualized nomogram based on the prediction model was provided for clinical application. The prediction model with imprinted gene detection effectively improves the diagnosis of PTCs that are undetermined by the current means.

Prediction models in prehospital and emergency medicine research: How to derive and internally validate a clinical prediction model

Clinical prediction models are created to help clinicians with medical decision making, aid in risk stratification, and improve diagnosis and/or prognosis. With growing availability of both prehospital and in-hospital observational registries and electronic health records, there is an opportunity to develop, validate, and incorporate prediction models into clinical practice. However, many prediction models have high risk of bias due to poor methodology. Given that there are no methodological standards aimed at developing prediction models specifically in the prehospital setting, the objective of this paper is to describe the appropriate methodology for the derivation and validation of clinical prediction models in this setting. What follows can also be applied to the emergency medicine (EM) setting. There are eight steps that should be followed when developing and internally validating a prediction model: (1) problem definition, (2) coding of predictors, (3) addressing missing data, (4) ensuring adequate sample size, (5) variable selection, (6) evaluating model performance, (7) internal validation, and (8) model presentation. Subsequent steps include external validation, assessment of impact, and cost-effectiveness. By following these steps, researchers can develop a prediction model with the methodological rigor and quality required for prehospital and EM research.

Validation of Machine Learning Model Performance in Predicting Blood Transfusion After Primary and Revision Total Hip Arthroplasty

BACKGROUND

The rates of blood transfusion following primary and revision total hip arthroplasty (THA) remain as high as 9% and 18%, respectively, contributing to patient morbidity and healthcare costs. Existing predictive tools are limited to specific populations, thereby diminishing their clinical applicability. This study aimed to externally validate our previous institutionally developed machine learning (ML) algorithms to predict the risk of postoperative blood transfusion following primary and revision THA using national inpatient data.

METHODS

Five ML algorithms were trained and validated using data from 101,266 primary THA and 8,594 revision THA patients from a large national database to predict postoperative transfusion risk after primary and revision THA. Models were assessed and compared based on discrimination, calibration, and decision curve analysis.

RESULTS

The most important predictors of transfusion following primary and revision THA were preoperative hematocrit (<39.4%) and operation time (>157 minutes), respectively. All ML models demonstrated excellent discrimination (area under the curve (AUC) >0.8) in primary and revision THA patients, with artificial neural network (AUC = 0.84, slope = 1.11, intercept = -0.04, Brier score = 0.04), and elastic-net-penalized logistic regression (AUC = 0.85, slope = 1.08, intercept = -0.01, and Brier score = 0.12) performing best, respectively. On decision curve analysis, all 5 models demonstrated a higher net benefit than the conventional strategy of intervening for all or no patients in both patient cohorts.

CONCLUSIONS

This study successfully validated our previous institutionally developed ML algorithms for the prediction of blood transfusion following primary and revision THA. Our findings highlight the potential generalizability of predictive ML tools developed using nationally representative data in THA patients.

Cost-Effectiveness of SGLT2 Inhibitors in a Real-World Population: A MICADO Model-Based Analysis Using Routine Data from a GP Registry

OBJECTIVE

Sodium-glucose cotransporter 2 inhibitors (SGLT2i) have been shown to reduce the risk of cardiovascular complications, which largely drive diabetes' health and economic burdens. Trial results indicated that SGLT2i are cost effective. However, these findings may not be generalizable to the real-world target population. This study aims to evaluate the cost effectiveness of SGLT2i in a routine care type 2 diabetes population that meets Dutch reimbursement criteria using the MICADO model.

METHODS

Individuals from the Hoorn Diabetes Care System cohort (N = 15,392) were filtered to satisfy trial inclusion criteria (including EMPA-REG, CANVAS, and DECLARE-TIMI58) or satisfy the current Dutch reimbursement criteria for SGLT2i. We validated a health economic model (MICADO) by comparing simulated and observed outcomes regarding the relative risks of events in the intervention and comparator arm from three trials, and used the validated model to evaluate the long-term health outcomes using the filtered cohorts' baseline characteristics and treatment effects from trials and a review of observational studies. The incremental cost-effectiveness ratio (ICER) of SGLT2i, compared with care-as-usual, was assessed from a third-party payer perspective, measured in euros (2021 price level), using a discount rate of 4% for costs and 1.5% for effects.

RESULTS

From Dutch individuals with diabetes in routine care, 15.8% qualify for the current Dutch reimbursement criteria for SGLT2i. Their characteristics were significantly different (lower HbA1c, higher age, and generally more preexisting complications) than trial populations. After validating the MICADO model, we found that lifetime ICERs of SGLT2i, when compared with usual care, were favorable (< €20,000/QALY) for all filtered cohorts, resulting in an ICER of €5440/QALY using trial-based treatment effect estimates in reimbursed population. Several pragmatic scenarios were tested, the ICERs remained favorable.

CONCLUSIONS

Although the Dutch reimbursement indications led to a target group that deviates from trial populations, SGLT2i are likely to be cost effective when compared with usual care.

Comparison of a multidomain frailty index from routine health data with the hospital frailty risk score in older patients in an Australian hospital

BACKGROUND

Frailty is an important determinant of health-care needs and outcomes for people in hospital.

OBJECTIVES

To compare characteristics and predictive ability of a multidomain frailty index derived from routine health data (electronic frailty index-acute hospital; eFI-AH) with the hospital frailty risk score (HFRS).

METHODS

This retrospective study included 6771 patients aged ≥75 years admitted to an Australian metropolitan tertiary referral hospital between October 2019 and September 2020. The eFI-AH and the HFRS were calculated for each patient and compared with respect to characteristics, agreement, association with age and ability to predict outcomes.

RESULTS

Median eFI-AH was 0.17 (range 0-0.66) whilst median HFRS was 3.2 (range 0-42.9). Moderate agreement was shown between the tools (Pearson's r 0.61). After adjusting for age and gender, both models had associations with long hospital stay, in-hospital mortality, unplanned all-cause readmission and fall-related readmission. Specifically, the eFI-AH had the strongest association with in-hospital mortality (adjusted odds ratio (aOR) 2.81, 95% confidence intervals (CI) 2.49-3.17), whilst the HFRS was most strongly associated with long hospital stay (aOR 1.20, 95% CI 1.18-1.21). Both tools predicted hospital stay >10 days with good discrimination and calibration.

CONCLUSIONS

Although the eFI-AH and the HFRS did not consistently identify the same inpatients as frail, both were associated with adverse outcomes and they had comparable predictive ability for prolonged hospitalisation. These two constructs of frailty may have different implications for clinical practice and health service provision and planning.

Dissecting contributions of individual systemic inflammatory response syndrome criteria from a prospective algorithm to the prediction and diagnosis of sepsis in a polytrauma cohort

Background

Sepsis is the leading cause of death in intensive care units (ICUs), and its timely detection and treatment improve clinical outcome and survival. Systemic inflammatory response syndrome (SIRS) refers to the concurrent fulfillment of at least two out of the following four clinical criteria: tachycardia, tachypnea, abnormal body temperature, and abnormal leukocyte count. While SIRS was controversially abandoned from the current sepsis definition, a dynamic SIRS representation still has potential for sepsis prediction and diagnosis.

Objective

We retrospectively elucidate the individual contributions of the SIRS criteria in a polytrauma cohort from the post-surgical ICU of University Medical Center Mannheim (Germany).

Methods

We used a dynamic and prospective SIRS algorithm tailored to the ICU setting by accounting for catecholamine therapy and mechanical ventilation. Two clinically relevant tasks are considered: (i) sepsis prediction using the first 24 h after admission to our ICU, and (ii) sepsis diagnosis using the last 24 h before sepsis onset and a time point of comparable ICU treatment duration for controls, respectively. We determine the importance of individual SIRS criteria by systematically varying criteria weights when summarizing the SIRS algorithm output with SIRS descriptors and assessing the classification performance of the resulting logistic regression models using a specifically developed ranking score.

Results

Our models perform better for the diagnosis than the prediction task (maximum AUROC 0.816 vs. 0.693). Risk models containing only the SIRS level average mostly show reasonable performance across criteria weights, with prediction and diagnosis AUROCs ranging from 0.455 (weight on leukocyte criterion only) to 0.693 and 0.619 to 0.800, respectively. For sepsis prediction, temperature and tachypnea are the most important SIRS criteria, whereas the leukocytes criterion is least important and potentially even counterproductive. For sepsis diagnosis, all SIRS criteria are relevant, with the temperature criterion being most influential.

Conclusion

SIRS is relevant for sepsis prediction and diagnosis in polytrauma, and no criterion should a priori be omitted. Hence, the original expert-defined SIRS criteria are valid, capturing important sepsis risk determinants. Our prospective SIRS algorithm provides dynamic determination of SIRS criteria and descriptors, allowing their integration in sepsis risk models also in other settings.

Calibration of the PREdiction of DELIRium in ICu Patients (PRE-DELIRIC) Score in a Cohort of Critically Ill Patients: A Retrospective Cohort Study

BACKGROUND

To predict delirium in intensive care unit (ICU) patients, the Prediction of Delirium in ICU Patients (PRE-DELIRIC) score may be used. This model may help nurses to predict delirium in high-risk ICU patients.

OBJECTIVES

The aims of this study were to externally validate the PRE-DELIRIC model and to identify predictive factors and outcomes for ICU delirium.

METHOD

All patients underwent delirium risk assessment by the PRE-DELIRIC model at admission. We used the Intensive Care Delirium Screening Check List to identify patients with delirium. The receiver operating characteristic curve measured discrimination capacity among patients with or without ICU delirium. Calibration ability was determined by slope and intercept.

RESULTS

The prevalence of ICU delirium was 55.8%. Discrimination capacity (Intensive Care Delirium Screening Check List score ≥4) expressed by the area under the receiver operating characteristic curve was 0.81 (95% confidence interval, 0.75-0.88), whereas sensitivity was 91.3% and specificity was 64.4%. The best cut-off was 27%, obtained by the max Youden index. Calibration of the model was adequate, with a slope of 1.03 and intercept of 8.14. The onset of ICU delirium was associated with an increase in ICU length of stay (P < .0001), higher ICU mortality (P = .008), increased duration of mechanical ventilation (P < .0001), and more prolonged respiratory weaning (P < .0001) compared with patients without delirium.

DISCUSSION

The PRE-DELIRIC score is a sensitive measure that may be useful in early detection of patients at high risk for developing delirium. The baseline PRE-DELIRIC score could be useful to trigger use of standardized protocols, including nonpharmacologic interventions.

Development and Validation of Risk Prediction Models for Colorectal Cancer in Patients with Symptoms

We aimed to develop and validate prediction models incorporating demographics, clinical features, and a weighted genetic risk score (wGRS) for individual prediction of colorectal cancer (CRC) risk in patients with gastroenterological symptoms. Prediction models were developed with internal validation [CRC Cases: n = 1686/Controls: n = 963]. Candidate predictors included age, sex, BMI, wGRS, family history, and symptoms (changes in bowel habits, rectal bleeding, weight loss, anaemia, abdominal pain). The baseline model included all the non-genetic predictors. Models A (baseline model + wGRS) and B (baseline model) were developed based on LASSO regression to select predictors. Models C (baseline model + wGRS) and D (baseline model) were built using all variables. Models' calibration and discrimination were evaluated through the Hosmer-Lemeshow test (calibration curves were plotted) and C-statistics (corrected based on 1000 bootstrapping). The models' prediction performance was: model A (corrected C-statistic = 0.765); model B (corrected C-statistic = 0.753); model C (corrected C-statistic = 0.764); and model D (corrected C-statistic = 0.752). Models A and C, that integrated wGRS with demographic and clinical predictors, had a statistically significant improved prediction performance. Our findings suggest that future application of genetic predictors holds significant promise, which could enhance CRC risk prediction. Therefore, further investigation through model external validation and clinical impact is merited.

Multi-domain prognostic models used in middle-aged adults without known cognitive impairment for predicting subsequent dementia

BACKGROUND

Dementia, a global health priority, has no current cure. Around 50 million people worldwide currently live with dementia, and this number is expected to treble by 2050. Some health conditions and lifestyle behaviours can increase or decrease the risk of dementia and are known as 'predictors'. Prognostic models combine such predictors to measure the risk of future dementia. Models that can accurately predict future dementia would help clinicians select high-risk adults in middle age and implement targeted risk reduction.

OBJECTIVES

Our primary objective was to identify multi-domain prognostic models used in middle-aged adults (aged 45 to 65 years) for predicting dementia or cognitive impairment. Eligible multi-domain prognostic models involved two or more of the modifiable dementia predictors identified in a 2020 Lancet Commission report and a 2019 World Health Organization (WHO) report (less education, hearing loss, traumatic brain injury, hypertension, excessive alcohol intake, obesity, smoking, depression, social isolation, physical inactivity, diabetes mellitus, air pollution, poor diet, and cognitive inactivity). Our secondary objectives were to summarise the prognostic models, to appraise their predictive accuracy (discrimination and calibration) as reported in the development and validation studies, and to identify the implications of using dementia prognostic models for the management of people at a higher risk for future dementia.

SEARCH METHODS

We searched MEDLINE, Embase, PsycINFO, CINAHL, and ISI Web of Science Core Collection from inception until 6 June 2022. We performed forwards and backwards citation tracking of included studies using the Web of Science platform.  SELECTION CRITERIA: We included development and validation studies of multi-domain prognostic models. The minimum eligible follow-up was five years. Our primary outcome was an incident clinical diagnosis of dementia based on validated diagnostic criteria, and our secondary outcome was dementia or cognitive impairment determined by any other method.

DATA COLLECTION AND ANALYSIS

Two review authors independently screened the references, extracted data using a template based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS), and assessed risk of bias and applicability of included studies using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We synthesised the C-statistics of models that had been externally validated in at least three comparable studies.  MAIN RESULTS: We identified 20 eligible studies; eight were development studies and 12 were validation studies. There were 14 unique prognostic models: seven models with validation studies and seven models with development-only studies. The models included a median of nine predictors (range 6 to 34); the median number of modifiable predictors was five (range 2 to 11). The most common modifiable predictors in externally validated models were diabetes, hypertension, smoking, physical activity, and obesity. In development-only models, the most common modifiable predictors were obesity, diabetes, hypertension, and smoking. No models included hearing loss or air pollution as predictors. Nineteen studies had a high risk of bias according to the PROBAST assessment, mainly because of inappropriate analysis methods, particularly lack of reported calibration measures. Applicability concerns were low for 12 studies, as their population, predictors, and outcomes were consistent with those of interest for this review. Applicability concerns were high for nine studies, as they lacked baseline cognitive screening or excluded an age group within the range of 45 to 65 years. Only one model, Cardiovascular Risk Factors, Ageing, and Dementia (CAIDE), had been externally validated in multiple studies, allowing for meta-analysis. The CAIDE model included eight predictors (four modifiable predictors): age, education, sex, systolic blood pressure, body mass index (BMI), total cholesterol, physical activity and APOEƐ4 status. Overall, our confidence in the prediction accuracy of CAIDE was very low; our main reasons for downgrading the certainty of the evidence were high risk of bias across all the studies, high concern of applicability, non-overlapping confidence intervals (CIs), and a high degree of heterogeneity. The summary C-statistic was 0.71 (95% CI 0.66 to 0.76; 3 studies; very low-certainty evidence) for the incident clinical diagnosis of dementia, and 0.67 (95% CI 0.61 to 0.73; 3 studies; very low-certainty evidence) for dementia or cognitive impairment based on cognitive scores. Meta-analysis of calibration measures was not possible, as few studies provided these data.

AUTHORS' CONCLUSIONS

We identified 14 unique multi-domain prognostic models used in middle-aged adults for predicting subsequent dementia. Diabetes, hypertension, obesity, and smoking were the most common modifiable risk factors used as predictors in the models. We performed meta-analyses of C-statistics for one model (CAIDE), but the summary values were unreliable. Owing to lack of data, we were unable to meta-analyse the calibration measures of CAIDE. This review highlights the need for further robust external validations of multi-domain prognostic models for predicting future risk of dementia in middle-aged adults.

Prospective multicenter external validation of postoperative mortality prediction tools in patients undergoing emergency laparotomy

BACKGROUND

Accurate preoperative risk assessment in emergency laparotomy (EL) is valuable for informed decision making and rational use of resources. Available risk prediction tools have not been validated adequately across diverse health care settings. Herein, we report a comparative external validation of four widely cited prognostic models.

METHODS

A multicenter cohort was prospectively composed of consecutive patients undergoing EL in 11 Greek hospitals from January 2020 to May 2021 using the National Emergency Laparotomy Audit (NELA) inclusion criteria. Thirty-day mortality risk predictions were calculated using the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP), NELA, Portsmouth Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (P-POSSUM), and Predictive Optimal Trees in Emergency Surgery Risk tools. Surgeons' assessment of postoperative mortality using predefined cutoffs was recorded, and a surgeon-adjusted ACS-NSQIP prediction was calculated when the original model's prediction was relatively low. Predictive performances were compared using scaled Brier scores, discrimination and calibration measures and plots, and decision curve analysis. Heterogeneity across hospitals was assessed by random-effects meta-analysis.

RESULTS

A total of 631 patients were included, and 30-day mortality was 16.3%. The ACS-NSQIP and its surgeon-adjusted version had the highest scaled Brier scores. All models presented high discriminative ability, with concordance statistics ranging from 0.79 for P-POSSUM to 0.85 for NELA. However, except the surgeon-adjusted ACS-NSQIP (Hosmer-Lemeshow test, p = 0.742), all other models were poorly calibrated ( p < 0.001). Decision curve analysis revealed superior clinical utility of the ACS-NSQIP. Following recalibrations, predictive accuracy improved for all models, but ACS-NSQIP retained the lead. Between-hospital heterogeneity was minimum for the ACS-NSQIP model and maximum for P-POSSUM.

CONCLUSION

The ACS-NSQIP tool was most accurate for mortality predictions after EL in a broad external validation cohort, demonstrating utility for facilitating preoperative risk management in the Greek health care system. Subjective surgeon assessments of patient prognosis may optimize ACS-NSQIP predictions.

LEVEL OF EVIDENCE

Diagnostic Test/Criteria; Level II.

Temporal validation of a multivariable surgical mortality prediction model (NZRisk): a New Zealand national cohort study

OBJECTIVES

Clinical risk calculators (CRCs), such as NZRisk, are used daily by clinicians to guide clinical decisions and explain individual risk to patients. The utility and robustness of these tools depends on the methods used to create the underlying mathematical model, as well as the stability of that model in relation to changing clinical practice and patient populations over time. The later should be checked by temporal validation using external data. Few if any of the clinical prediction models in current clinical use have published temporal validation. Here, we use a large external dataset to temporally validate NZRisk; a perioperative risk prediction model used in the New Zealand population.

METHODS

A sample of 1 976 362 adult non-cardiac surgical procedures collected over 15 years from the New Zealand Ministry of Health National Minimum Dataset, was used to temporally validate NZRisk. We divided the dataset into 15 single year cohorts and compared 13 of these to our NZRisk model (2 years used for the model building were excluded). We compared the area under the curve (AUC) value, calibration slope and intercept for each single year cohort, to the same values produced by the data used to create NZRisk, by fitting a random effects meta-regression with each year cohort acting as a separate study point. In addition, we used two-sided t-tests to compare each measure across the cohorts.

RESULTS

The AUC values for the 30-day NZRisk model applied to our single year cohorts ranged from 0.918 to 0.940 (NZRisk AUC was 0.921). There were eight statistically different AUC values for the following years 2007-2009, 2016 and 2018-2021. The intercept values ranged from -0.004 to 0.007 and 7 years had statistically significant different intercepts during leave-one-out t-tests; 2007-2010, 2012, 2018 and 2021. The slope values ranged from 0.72 to 1.12 and 7 years had statistically significant different slopes during leave-one-out t-tests; 2010, 2011, 2017, 2018 and 2019-2021. The random effects meta-regression upheld our results related to AUC (0.54 (95% CI 0.40 to 0.99), I² 67.57 (95% CI 40.67 to 88.50), Cochran's Q<0.001) and slope (τ 0.14 (95% CI 0.01 to 0.23), I² 98.61 (95% CI 97.31 to 99.50), Cochran's Q<0.001) between year difference.

CONCLUSION

The NZRisk model shows differences in AUC and slope but not intercept values over time. The biggest differences were in the calibration slope. The models maintained excellent discrimination over time as shown by the AUC values. These findings suggest we update our model in the next 5 years. To our knowledge, this is the first temporal validation of a CRC in current use.

Validation and Generalizability of Machine Learning Models for the Prediction of Discharge Disposition Following Revision Total Knee Arthroplasty

BACKGROUND

Postoperative discharge to facilities account for over 33% of the $ 2.7 billion revision total knee arthroplasty (TKA)-associated annual expenditures and are associated with increased complications when compared to home discharges. Prior studies predicting discharge disposition using advanced machine learning (ML) have been limited due to a lack of generalizability and validation. This study aimed to establish ML model generalizability by externally validating its prediction for nonhome discharge following revision TKA using national and institutional databases.

METHODS

The national and institutional cohorts comprised 52,533 and 1,628 patients, respectively, with 20.6 and 19.4% nonhome discharge rates. Five ML models were trained and internally validated (five-fold cross-validation) on a large national dataset. Subsequently, external validation was performed on our institutional dataset. Model performance was assessed using discrimination, calibration, and clinical utility. Global predictor importance plots and local surrogate models were used for interpretation.

RESULTS

The strongest predictors of nonhome discharge were patient age, body mass index, and surgical indication. The area under the receiver operating characteristic curve increased from internal to external validation and ranged between 0.77 and 0.79. Artificial neural network was the best predictive model for identifying patients at risk for nonhome discharge (area under the receiver operating characteristic curve = 0.78), and also the most accurate (calibration slope = 0.93, intercept = 0.02, and Brier score = 0.12).

CONCLUSION

All five ML models demonstrated good-to-excellent discrimination, calibration, and clinical utility on external validation, with artificial neural network being the best model for predicting discharge disposition following revision TKA. Our findings establish the generalizability of ML models developed using data from a national database. The integration of these predictive models into clinical workflow may assist in optimizing discharge planning, bed management, and cost containment associated with revision TKA.

Prediction models for cardiovascular disease risk among people living with HIV: A systematic review and meta-analysis

Background

HIV continues to be a major global health issue. The relative risk of cardiovascular disease (CVD) among people living with HIV (PLWH) was 2.16 compared to non-HIV-infections. The prediction of CVD is becoming an important issue in current HIV management. However, there is no consensus on optional CVD risk models for PLWH. Therefore, we aimed to systematically summarize and compare prediction models for CVD risk among PLWH.

Methods

Longitudinal studies that developed or validated prediction models for CVD risk among PLWH were systematically searched. Five databases were searched up to January 2022. The quality of the included articles was evaluated by using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We applied meta-analysis to pool the logit-transformed C-statistics for discrimination performance.

Results

Thirteen articles describing 17 models were included. All the included studies had a high risk of bias. In the meta-analysis, the pooled estimated C-statistic was 0.76 (95% CI: 0.72-0.81, I ² = 84.8%) for the Data collection on Adverse Effects of Anti-HIV Drugs Study risk equation (D:A:D) (2010), 0.75 (95% CI: 0.70-0.79, I ² = 82.4%) for the D:A:D (2010) 10-year risk version, 0.77 (95% CI: 0.74-0.80, I ² = 82.2%) for the full D:A:D (2016) model, 0.74 (95% CI: 0.68-0.79, I ² = 86.2%) for the reduced D:A:D (2016) model, 0.71 (95% CI: 0.61-0.79, I ² = 87.9%) for the Framingham Risk Score (FRS) for coronary heart disease (CHD) (1998), 0.74 (95% CI: 0.70-0.78, I ² = 87.8%) for the FRS CVD model (2008), 0.72 (95% CI: 0.67-0.76, I ² = 75.0%) for the pooled cohort equations of the American Heart Society/ American score (PCE), and 0.67 (95% CI: 0.56-0.77, I ² = 51.3%) for the Systematic COronary Risk Evaluation (SCORE). In the subgroup analysis, the discrimination of PCE was significantly better in the group aged ≤40 years than in the group aged 40-45 years (P = 0.024) and the group aged ≥45 years (P = 0.010). No models were developed or validated in Sub-Saharan Africa and the Asia region.

Conclusions

The full D:A:D (2016) model performed the best in terms of discrimination, followed by the D:A:D (2010) and PCE. However, there were no significant differences between any of the model pairings. Specific CVD risk models for older PLWH and for PLWH in Sub-Saharan Africa and the Asia region should be established.Systematic Review Registration: PROSPERO CRD42022322024.

External validation of a CT-based radiomics signature in oropharyngeal cancer: Assessing sources of variation

BACKGROUND AND PURPOSE

Radiomics is a high-throughput approach that allows for quantitative analysis of imaging data for prognostic applications. Medical images are used in oropharyngeal cancer (OPC) diagnosis and treatment planning and these images may contain prognostic information allowing for treatment personalization. However, the lack of validated models has been a barrier to the translation of radiomic research to the clinic. We hypothesize that a previously developed radiomics model for risk stratification in OPC can be validated in a local dataset.

MATERIALS AND METHODS

The radiomics signature predicting overall survival incorporates features derived from the primary gross tumor volume of OPC patients treated with radiation +/- chemotherapy at a single institution (n = 343). Model fit, calibration, discrimination, and utility were evaluated. The signature was compared with a clinical model using overall stage and a model incorporating both radiomics and clinical data. A model detecting dental artifacts on computed tomography images was also validated.

RESULTS

The radiomics signature had a Concordance index (C-index) of 0.66 comparable to the clinical model's C-index of 0.65. The combined model significantly outperformed (C-index of 0.69, p = 0.024) the clinical model, suggesting that radiomics provides added value. The dental artifact model demonstrated strong ability in detecting dental artifacts with an area under the curve of 0.87.

CONCLUSION

This work demonstrates model performance comparable to previous validation work and provides a framework for future independent and multi-center validation efforts. With sufficient validation, radiomic models have the potential to improve traditional systems of risk stratification, treatment personalization and patient outcomes.

Development of a prediction model to determine responders to conservative treatment in people with symptomatic hand osteoarthritis: A secondary analysis of a single-centre, randomised feasibility trial

BACKGROUND

Conservative treatments are beneficial for people with hand osteoarthritis (OA).

OBJECTIVE

It was the purpose of this study to develop and internally validate both a basic model and a more complex model that could predict responders to conservative treatments in people with hand OA.

DESIGN

This was a secondary analysis of a single-centre, randomised feasibility study.

METHODS

Fifty-nine participants (34 responders) with hand osteoarthritis were recruited from the general population. Participants were randomised to receive either advice alone, or advice in combination with blood flow restriction training (BFRT), or traditional high intensity training (HIT). Participants underwent supervised hand exercises three times per week for six weeks. The OMERACT-OARSI criteria were utilised to determine responders vs non responders to treatment at the end of six weeks. A basic logistic regression model (treatment type, expectations, adherence) and a more complex logistic regression model (basic model variables plus pain catastrophising and neuropathic pain features) were created. Discrimination ability, and calibration were assessed. Internal model validation through bootstrapping (200 repetitions) was utilised to calculate the prediction model optimism.

RESULTS

The results showed that the basic model presented with acceptable discrimination (optimism corrected c-statistic: 0.72, 95% CI 0.71-0.73) and calibration (slope = 1.41; intercept = 0.68). The more complex model had better discrimination but poorer calibration.

CONCLUSION

A prediction tool was created to provide an individualised estimate of treatment response in people with hand OA. Future studies will need to validate this model in other groups of patients.

TRIAL REGISTRATION

https://www.anzctr.org.au/- ACTRN12617001270303.

Mortality Prediction in Hip Fracture Patients: Physician Assessment Versus Prognostic Models

OBJECTIVES

To evaluate 2 prognostic models for mortality after a fracture of the hip, the Nottingham Hip Fracture Score and Hip Fracture Estimator of Mortality Amsterdam and to compare their predictive performance to physician assessment of mortality risk in hip fracture patients.

DESIGN

Prospective cohort study.

SETTING

Two level-2 trauma centers located in the Netherlands.

PATIENTS

Two hundred forty-four patients admitted to the Emergency Departments of both hospitals with a fractured hip.

INTERVENTION

Data used in both prediction models were collected at the time of admission for each individual patient, as well as predictions of mortality by treating physicians.

MAIN OUTCOME MEASURES

Predictive performances were evaluated for 30-day, 1-year, and 5-year mortality. Discrimination was assessed with the area under the curve (AUC); calibration with the Hosmer-Lemeshow goodness-of-fit test and calibration plots; clinical usefulness in terms of accuracy, sensitivity, and specificity.

RESULTS

Mortality was 7.4% after 30 days, 22.1% after 1 year, and 59.4% after 5 years. There were no statistically significant differences in discrimination between the prediction methods (AUC 0.73-0.80). The Nottingham Hip Fracture Score demonstrated underfitting for 30-day mortality and failed to identify the majority of high-risk patients (sensitivity 33%). The Hip fracture Estimator of Mortality Amsterdam showed systematic overestimation and overfitting. Physicians were able to identify most high-risk patients for 30-day mortality (sensitivity 78%) but with some overestimation. Both risk models demonstrated a lack of fit when used for 1-year and 5-year mortality predictions.

CONCLUSIONS

In this study, prognostic models and physicians demonstrated similar discriminating abilities when predicting mortality in hip fracture patients. Although physicians overestimated mortality, they were better at identifying high-risk patients and at predicting long-term mortality.

LEVEL OF EVIDENCE

Prognostic Level II. See Instructions for Authors for a complete description of levels of evidence.

External Validation of an Ensemble Model for Automated Mammography Interpretation by Artificial Intelligence

Importance

With a shortfall in fellowship-trained breast radiologists, mammography screening programs are looking toward artificial intelligence (AI) to increase efficiency and diagnostic accuracy. External validation studies provide an initial assessment of how promising AI algorithms perform in different practice settings.

Objective

To externally validate an ensemble deep-learning model using data from a high-volume, distributed screening program of an academic health system with a diverse patient population.

Design, Setting, and Participants

In this diagnostic study, an ensemble learning method, which reweights outputs of the 11 highest-performing individual AI models from the Digital Mammography Dialogue on Reverse Engineering Assessment and Methods (DREAM) Mammography Challenge, was used to predict the cancer status of an individual using a standard set of screening mammography images. This study was conducted using retrospective patient data collected between 2010 and 2020 from women aged 40 years and older who underwent a routine breast screening examination and participated in the Athena Breast Health Network at the University of California, Los Angeles (UCLA).

Main Outcomes and Measures

Performance of the challenge ensemble method (CEM) and the CEM combined with radiologist assessment (CEM+R) were compared with diagnosed ductal carcinoma in situ and invasive cancers within a year of the screening examination using performance metrics, such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC).

Results

Evaluated on 37 317 examinations from 26 817 women (mean [SD] age, 58.4 [11.5] years), individual model AUROC estimates ranged from 0.77 (95% CI, 0.75-0.79) to 0.83 (95% CI, 0.81-0.85). The CEM model achieved an AUROC of 0.85 (95% CI, 0.84-0.87) in the UCLA cohort, lower than the performance achieved in the Kaiser Permanente Washington (AUROC, 0.90) and Karolinska Institute (AUROC, 0.92) cohorts. The CEM+R model achieved a sensitivity (0.813 [95% CI, 0.781-0.843] vs 0.826 [95% CI, 0.795-0.856]; P = .20) and specificity (0.925 [95% CI, 0.916-0.934] vs 0.930 [95% CI, 0.929-0.932]; P = .18) similar to the radiologist performance. The CEM+R model had significantly lower sensitivity (0.596 [95% CI, 0.466-0.717] vs 0.850 [95% CI, 0.766-0.923]; P < .001) and specificity (0.803 [95% CI, 0.734-0.861] vs 0.945 [95% CI, 0.936-0.954]; P < .001) than the radiologist in women with a prior history of breast cancer and Hispanic women (0.894 [95% CI, 0.873-0.910] vs 0.926 [95% CI, 0.919-0.933]; P = .004).

Conclusions and Relevance

This study found that the high performance of an ensemble deep-learning model for automated screening mammography interpretation did not generalize to a more diverse screening cohort, suggesting that the model experienced underspecification. This study suggests the need for model transparency and fine-tuning of AI models for specific target populations prior to their clinical adoption.

Development and training of a machine learning algorithm to identify patients at risk for recurrence following an arthroscopic Bankart repair (CLEARER): protocol for a retrospective, multicentre, cohort study

INTRODUCTION

Shoulder instability is a common injury, with a reported incidence of 23.9 per 100 000 person-years. There is still an ongoing debate on the most effective treatment strategy. Non-operative treatment has recurrence rates of up to 60%, whereas operative treatments such as the Bankart repair and bone block procedures show lower recurrence rates (16% and 2%, respectively) but higher complication rates (<2% and up to 30%, respectively). Methods to determine risk of recurrence have been developed; however, patient-specific decision-making tools are still lacking. Artificial intelligence and machine learning algorithms use self-learning complex models that can be used to make patient-specific decision-making tools. The aim of the current study is to develop and train a machine learning algorithm to create a prediction model to be used in clinical practice-as an online prediction tool-to estimate recurrence rates following a Bankart repair.

METHODS AND ANALYSIS

This is a multicentre retrospective cohort study. Patients with traumatic anterior shoulder dislocations that were treated with an arthroscopic Bankart repair without remplissage will be included. This study includes two parts. Part 1, collecting all potential factors influencing the recurrence rate following an arthroscopic Bankart repair in patients using multicentre data, aiming to include data from >1000 patients worldwide. Part 2, the multicentre data will be re-evaluated (and where applicable complemented) using machine learning algorithms to predict outcomes. Recurrence will be the primary outcome measure.

ETHICS AND DISSEMINATION

For safe multicentre data exchange and analysis, our Machine Learning Consortium adhered to the WHO regulation 'Policy on Use and Sharing of Data Collected by WHO in Member States Outside the Context of Public Health Emergencies'. The study results will be disseminated through publication in a peer-reviewed journal. No Institutional Review Board is required for this study.

Validation of two IgA nephropathy risk-prediction tools using a cohort with a long follow-up

BACKGROUND

Recently, two immunoglobulin A nephropathy prediction tools were developed that combine clinical and histopathological parameters. The International IgAN Prediction Tool predicts the risk for 50% declines in the estimated glomerular filtration rate or end-stage renal disease up to 80 months after diagnosis. The IgA Nephropathy Clinical Decision Support System uses artificial neural networks to estimate the risk for end-stage renal disease. We aimed to externally validate both prediction tools using a Norwegian cohort with a long-term follow-up.

METHODS

We included 306 patients with biopsy-proven primary immunoglobulin A nephropathy in this study. Histopathologic samples were retrieved from the Norwegian Kidney Biopsy Registry and reclassified according to the Oxford classification. We used discrimination and calibration as principles for externally validating the prognostic models.

RESULTS

The median patient follow-up was 17.1 years. A cumulative dynamic time-dependent receiver operating characteristic analysis showed area under the curve values of ranging from 0.90 at 5 years to 0.83 at 20 years for the International IgAN Prediction Tool, while time-naive analysis showed an area under the curve value at 0.83 for the IgA Nephropathy Clinical Decision Support System. The International IgAN Prediction Tool was well calibrated, while the IgA Nephropathy Clinical Decision Support System tends to underestimate risk for patients with higher risk, and overestimates risk in the lower risk categories.

CONCLUSIONS

We have externally validated two prediction tools for IgA nephropathy. The International IgAN Prediction Tool performed well, while the IgA Nephropathy Clinical Decision Support System has some limitations.

Prediction of Five-Year Cardiovascular Disease Risk in People with Type 2 Diabetes Mellitus: Derivation in Nanjing, China and External Validation in Scotland, UK

Background

To use routinely collected data to develop a five-year cardiovascular disease (CVD) risk prediction model for Chinese adults with type 2 diabetes with validation of its performance in a population of European ancestry.

Methods

People with incident type 2 diabetes and no history of CVD at diagnosis of diabetes between 2008 and 2017 were included in derivation and validation cohorts. The derivation cohort was identified from a pseudonymized research extract of data from the First Affiliated Hospital of Nanjing Medical University (NMU). Five-year risk of CVD was estimated using basic and extended Cox proportional hazards regression models including 6 and 11 predictors respectively. The risk prediction models were internally validated and externally validated in a Scottish population-based cohort with CVD events identified from linked hospital records. Discrimination and calibration were assessed using Harrell's C-statistic and calibration plots, respectively.

Results

Mean age of the derivation and validation cohorts were 58.4 and 59.2 years, respectively, with 53.5% and 56.9% men. During a median follow-up time of 4.75 [2.67, 7.42] years, 18,827 (22.25%) of the 84,630 people in the NMU-Diabetes cohort and 8,763 (7.31%) of the Scottish cohort of 119,891 people developed CVD. The extended model had a C-statistic of 0.723 [0.721-0.724] in internal validation and 0.716 [0.713-0.719] in external validation.

Conclusions

It is possible to generate a risk prediction model with moderate discriminative power in internal and external validation derived from routinely collected Chinese hospital data. The proposed risk score could be used to improve CVD prevention in people with diabetes.

Development and validation of a hypertension risk prediction model and construction of a risk score in a Canadian population

Identifying high-risk individuals for targeted intervention may prevent or delay hypertension onset. We developed a hypertension risk prediction model and subsequent risk sore among the Canadian population using measures readily available in a primary care setting. A Canadian cohort of 18,322 participants aged 35-69 years without hypertension at baseline was followed for hypertension incidence, and 625 new hypertension cases were reported. At a 2:1 ratio, the sample was randomly divided into derivation and validation sets. In the derivation sample, a Cox proportional hazard model was used to develop the model, and the model's performance was evaluated in the validation sample. Finally, a risk score table was created incorporating regression coefficients from the model. The multivariable Cox model identified age, body mass index, systolic blood pressure, diabetes, total physical activity time, and cardiovascular disease as significant risk factors (p < 0.05) of hypertension incidence. The variable sex was forced to enter the final model. Some interaction terms were identified as significant but were excluded due to their lack of incremental predictive capacity. Our model showed good discrimination (Harrel's C-statistic 0.77) and calibration (Grønnesby and Borgan test, [Formula: see text] statistic = 8.75, p = 0.07; calibration slope 1.006). A point-based score for the risks of developing hypertension was presented after 2-, 3-, 5-, and 6 years of observation. This simple, practical prediction score can reliably identify Canadian adults at high risk of developing incident hypertension in the primary care setting and facilitate discussions on modifying this risk most effectively.

Natural language processing in low back pain and spine diseases: A systematic review

Natural Language Processing (NLP) is a discipline at the intersection between Computer Science (CS), Artificial Intelligence (AI), and Linguistics that leverages unstructured human-interpretable (natural) language text. In recent years, it gained momentum also in health-related applications and research. Although preliminary, studies concerning Low Back Pain (LBP) and other related spine disorders with relevant applications of NLP methodologies have been reported in the literature over the last few years. It motivated us to systematically review the literature comprised of two major public databases, PubMed and Scopus. To do so, we first formulated our research question following the PICO guidelines. Then, we followed a PRISMA-like protocol by performing a search query including terminologies of both technical (e.g., natural language and computational linguistics) and clinical (e.g., lumbar and spine surgery) domains. We collected 221 non-duplicated studies, 16 of which were eligible for our analysis. In this work, we present these studies divided into sub-categories, from both tasks and exploited models’ points of view. Furthermore, we report a detailed description of techniques used to extract and process textual features and the several evaluation metrics used to assess the performance of the NLP models. However, what is clear from our analysis is that additional studies on larger datasets are needed to better define the role of NLP in the care of patients with spinal disorders.

Prediction models of diabetes complications: a scoping review

BACKGROUND

Diabetes often places a large burden on people with diabetes (hereafter 'patients') and the society, that is, in part attributable to its complications. However, evidence from models predicting diabetes complications in patients remains unclear. With the collaboration of patient partners, we aimed to describe existing prediction models of physical and mental health complications of diabetes.

METHODS

Building on existing frameworks, we systematically searched for studies in Ovid-Medline and Embase. We included studies describing prognostic prediction models that used data from patients with pre-diabetes or any type of diabetes, published between 2000 and 2020. Independent reviewers screened articles, extracted data and narratively synthesised findings using established reporting standards.

RESULTS

Overall, 78 studies reported 260 risk prediction models of cardiovascular complications (n=42 studies), mortality (n=16), kidney complications (n=14), eye complications (n=10), hypoglycaemia (n=8), nerve complications (n=3), cancer (n=2), fracture (n=2) and dementia (n=1). Prevalent complications deemed important by patients such as amputation and mental health were poorly or not at all represented. Studies primarily analysed data from older people with type 2 diabetes (n=54), with little focus on pre-diabetes (n=0), type 1 diabetes (n=8), younger (n=1) and racialised people (n=10). Per complication, predictors vary substantially between models. Studies with details of calibration and discrimination mostly exhibited good model performance.

CONCLUSION

This rigorous knowledge synthesis provides evidence of gaps in the landscape of diabetes complication prediction models. Future studies should address unmet needs for analyses of complications n> and among patient groups currently under-represented in the literature and should consistently report relevant statistics. SCOPING REVIEW REGISTRATION: https://osf.io/fjubt/.

Predictive models for determining treatment response to nonprescription acute medications in migraine: Results from the American Migraine Prevalence and Prevention Study

OBJECTIVE

To identify predictors of acute treatment response for nonprescription (over-the-counter [OTC]) medications among people with migraine and develop improved models for predicting treatment response.

BACKGROUND

Pain freedom and sustained pain relief are important priorities in the acute treatment of migraine. OTC medications are widely used for migraine; however, it is not clear which treatment works best for each patient without going through the trial and error process.

METHODS

A prediction model development study was completed using the 2006 American Migraine Prevalence and Prevention Study survey, from participants who were aged ≥18, met criteria and headache day frequency for episodic migraine, did not take prescription medication for migraine, and used ≥1 of the following acute migraine medication classes: acetaminophen, aspirin, NSAIDs, or caffeine containing combination products (CCP). Two items from the Migraine Treatment Optimization Questionnaire were used to evaluate treatment response, adequate 2-h pain freedom (2hPF) and 24-h pain relief (24hPR), which were defined by a response to treatment ≥half the time at 2 h and 24 h post treatment, respectively. We identified predictors of adequate treatment response and developed models to predict probability of treatment response to each medication class.

RESULTS

The sample included 3852 participants (3038 [79.0%] females) with an average age of 45.0 years (SD = 12.8). Only 1602/3852 (41.6%) and 1718/3852 (44.6%) of the participants reported adequate 2hPF and 24hPR, respectively. Adequate treatment-response was significantly predicted by lower average headache pain intensity, less cutaneous allodynia, and lower depressive symptom scores. Lower migraine symptom severity was predictive of adequate 2hPF and fewer monthly headache days was predictive of adequate 24hPR. Among participants reporting OTC monotherapy (n = 2168, 56.3%) individuals taking CCP were more likely to have adequate 2hPF (OR = 1.55, 95% CI 1.23-1.95) and 24hPR (OR = 1.79, 95% CI 1.18-1.88) in comparison with those taking acetaminophen. Predictive models were modestly predictive of responders to OTC medications (c-statistics = 0.65; 95% CI 0.62-0.68).

CONCLUSION

These results show that response to acute migraine treatments is not optimized in the majority of people with migraine treating with OTC medications. Predictive models can improve our ability to choose the best therapeutic option for individuals with episodic migraine and increase the proportion of patients with optimized response to treatments.

Clinical prediction models for diagnosis of COVID-19 among adult patients: a validation and agreement study

BACKGROUND

Since the beginning of the pandemic, hospitals have been constantly overcrowded, with several observed waves of infected cases and hospitalisations. To avoid as much as possible this situation, efficient tools to facilitate the diagnosis of COVID-19 are needed.

OBJECTIVE

To evaluate and compare prediction models to diagnose COVID-19 identified in a systematic review published recently using performance indicators such as discrimination and calibration measures.

METHODS

A total of 1618 adult patients present at two Emergency Department triage centers and for whom qRT-PCR tests had been performed were included in this study. Six previously published models were reconstructed and assessed using diagnostic tests as sensitivity (Se) and negative predictive value (NPV), discrimination (Area Under the Roc Curve (AUROC)) and calibration measures. Agreement was also measured between them using Kappa's coefficient and IntraClass Correlation Coefficient (ICC). A sensitivity analysis has been conducted by waves of patients.

RESULTS

Among the 6 selected models, those based only on symptoms and/or risk exposure were found to be less efficient than those based on biological parameters and/or radiological examination with smallest AUROC values (< 0.80). However, all models showed good calibration and values above > 0.75 for Se and NPV but poor agreement (Kappa and ICC < 0.5) between them. The results of the first wave were similar to those of the second wave.

CONCLUSION

Although quite acceptable and similar results were found between all models, the importance of radiological examination was also emphasized, making it difficult to find an appropriate triage system to classify patients at risk for COVID-19.

Performance assessment across different care settings of a heart failure hospitalisation risk-score for type 2 diabetes using administrative claims

Predicting the risk of cardiovascular complications, in particular heart failure hospitalisation (HHF), can improve the management of type 2 diabetes (T2D). Most predictive models proposed so far rely on clinical data not available at the higher Institutional level. Therefore, it is of interest to assess the risk of HHF in people with T2D using administrative claims data only, which are more easily obtainable and could allow public health systems to identify high-risk individuals. In this paper, the administrative claims of > 175,000 patients with T2D were used to develop a new risk score for HHF based on Cox regression. Internal validation on the administrative data cohort yielded satisfactory results in terms of discrimination (max AUROC = 0.792, C-index = 0.786) and calibration (Hosmer-Lemeshow test p value < 0.05). The risk score was then tested on data gathered from two independent centers (one diabetes outpatient clinic and one primary care network) to demonstrate its applicability to different care settings in the medium-long term. Thanks to the large size and broad demographics of the administrative dataset used for training, the proposed model was able to predict HHF without significant performance loss concerning bespoke models developed within each setting using more informative, but harder-to-acquire clinical variables.

Characteristics and Quality of Diagnostic and Risk Prediction Models for Frailty in Older Adults: A Systematic Review

Several prediction models for frailty in older adults have been published, but their characteristics and methodological quality are unclear. This review aims to summarize and critically appraise the prediction models. Studies describing multivariable prediction models for frailty among older adults were included. PubMed, Embase, Web of Science, and PsycINFO were searched from outset to Feb 21, 2021. Methodological and reporting quality of included models were evaluated by PROBAST and TRIPOD, respectively. All results were descriptively summarized. Twenty articles including 39 models were identified. The included models showed good predictive discrimination with C indices ranging from 0.70 to 0.98. However, all studies except one were assessed as high risk of bias mainly due to inappropriate analysis; meanwhile, poor reporting quality was also frequently observed. Few mature prediction models can be used in practice. Researchers should pay more attention to external validation and improving the quality both in methodology and reporting.

Predicting the risk of active pulmonary tuberculosis in people living with HIV: development and validation of a nomogram

BACKGROUND

Diagnosis of pulmonary tuberculosis (PTB) among people living with HIV (PLHIV) was challenging. The study aimed to develop and validated a simple, convenient screening model for prioritizing TB among PLHIV.

METHODS

The study included eligible adult PLHIV participants who attended health care in Yunnan, China, from January 2016 to July 2019. Participants included before June 2018 were in the primary set; others were in the independent validation set. The research applied the least absolute shrinkage and selection operator regression to identify predictors associated with bacteriological confirmed PTB. The TB nomogram was developed by multivariate logistic regression. The C-index, receiver operating characteristic curve (ROC), the Hosmer-Lemeshow goodness of fit test (H-L), and the calibration curves were applied to evaluate and calibrate the nomogram. The developed nomogram was validated in the validation set. The clinical usefulness was assessed by cutoff analysis and decision curve analysis in the primary set.

RESULT

The study enrolled 766 PLHIV, of which 507 were in the primary set and 259 in the validation set, 21.5% and 14.3% individuals were confirmed PTB in two sets, respectively. The final nomogram included 5 predictors: current CD 4 cell count, the number of WHO screen tool, previous TB history, pulmonary cavity, and smoking status (p < 0.05). The C-statistic was 0.72 (95% CI 0.66-0.77) in primary set and 0.68 (95% CI 0.58-0.75) in validation set, ROC performed better than other models. The nomogram calibration was good (H-L χ² = 8.14, p = 0.15). The area under the decision curve (0.025) outperformed the existing models. The optimal cutoff for screening TB among PLHIV was the score of 100 (sensitivity = 0.93, specificity = 0.35).

CONCLUSION

The study developed and validated a discriminative TB nomogram among PLHIV in the moderate prevalence of TB and HIV. The easy-to-use and straightforward nomogram would be beneficial for clinical practice and rapid risk screening in resource-limited settings.

Predicting the earliest deviation in weight gain in the course towards manifest overweight in offspring exposed to obesity in pregnancy: a longitudinal cohort study

BACKGROUND

Obesity in pregnancy and related early-life factors place the offspring at the highest risk of being overweight. Despite convincing evidence on these associations, there is an unmet public health need to identify "high-risk" offspring by predicting very early deviations in weight gain patterns as a subclinical stage towards overweight. However, data and methods for individual risk prediction are lacking. We aimed to identify those infants exposed to obesity in pregnancy at ages 3 months, 1 year, and 2 years who likely will follow a higher-than-normal body mass index (BMI) growth trajectory towards manifest overweight by developing an early-risk quantification system.

METHODS

This study uses data from the prospective mother-child cohort study Programming of Enhanced Adiposity Risk in CHildhood-Early Screening (PEACHES) comprising 1671 mothers with pre-conception obesity and without (controls) and their offspring. Exposures were pre- and postnatal risks documented in patient-held maternal and child health records. The main outcome was a "higher-than-normal BMI growth pattern" preceding overweight, defined as BMI z-score >1 SD (i.e., World Health Organization [WHO] cut-off "at risk of overweight") at least twice during consecutive offspring growth periods between age 6 months and 5 years. The independent cohort PErinatal Prevention of Obesity (PEPO) comprising 11,730 mother-child pairs recruited close to school entry (around age 6 years) was available for data validation. Cluster analysis and sequential prediction modelling were performed.

RESULTS

Data of 1557 PEACHES mother-child pairs and the validation cohort were analyzed comprising more than 50,000 offspring BMI measurements. More than 1-in-5 offspring exposed to obesity in pregnancy belonged to an upper BMI z-score cluster as a distinct pattern of BMI development (above the cut-off of 1 SD) from the first months of life onwards resulting in preschool overweight/obesity (age 5 years: odds ratio [OR] 16.13; 95% confidence interval [CI] 9.98-26.05). Contributing early-life factors including excessive weight gain (OR 2.08; 95% CI 1.25-3.45) and smoking (OR 1.94; 95% CI 1.27-2.95) in pregnancy were instrumental in predicting a "higher-than-normal BMI growth pattern" at age 3 months and re-evaluating the risk at ages 1 year and 2 years (area under the receiver operating characteristic [AUROC] 0.69-0.79, sensitivity 70.7-76.0%, specificity 64.7-78.1%). External validation of prediction models demonstrated adequate predictive performances.

CONCLUSIONS

We devised a novel sequential strategy of individual prediction and re-evaluation of a higher-than-normal weight gain in "high-risk" infants well before developing overweight to guide decision-making. The strategy holds promise to elaborate interventions in an early preventive manner for integration in systems of well-child care.

Head-to-head comparison of 14 prediction models for postoperative delirium in elderly non-ICU patients: an external validation study

OBJECTIVES

Delirium is associated with increased morbidity, mortality, prolonged hospitalisation and increased healthcare costs. The number of clinical prediction models (CPM) to predict postoperative delirium has increased exponentially. Our goal is to perform a head-to-head comparison of CPMs predicting postoperative delirium in non-intensive care unit (non-ICU) elderly patients to identify the best performing models.

SETTING

Single-site university hospital.

DESIGN

Secondary analysis of prospective cohort study.

PARTICIPANTS AND INCLUSION

CPMs published within the timeframe of 1 January 1990 to 1 May 2020 were checked for eligibility (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). For the time period of 1 January 1990 to 1 January 2017, included CPMs were identified in systematic reviews based on prespecified inclusion and exclusion criteria. An extended literature search for original studies was performed independently by two authors, including CPMs published between 1 January 2017 and 1 May 2020. External validation was performed using a surgical cohort consisting of 292 elderly non-ICU patients.

PRIMARY OUTCOME MEASURES

Discrimination, calibration and clinical usefulness.

RESULTS

14 CPMs were eligible for analysis out of 366 full texts reviewed. External validation was previously published for 8/14 (57%) CPMs. C-indices ranged from 0.52 to 0.74, intercepts from -0.02 to 0.34, slopes from -0.74 to 1.96 and scaled Brier from -1.29 to 0.088. Based on predefined criteria, the two best performing models were those of Dai et al (c-index: 0.739; (95% CI: 0.664 to 0.813); intercept: -0.018; slope: 1.96; scaled Brier: 0.049) and Litaker et al (c-index: 0.706 (95% CI: 0.590 to 0.823); intercept: -0.015; slope: 0.995; scaled Brier: 0.088). For the remaining CPMs, model discrimination was considered poor with corresponding c-indices <0.70.

CONCLUSION

Our head-to-head analysis identified 2 out of 14 CPMs as best-performing models with a fair discrimination and acceptable calibration. Based on our findings, these models might assist physicians in postoperative delirium risk estimation and patient selection for preventive measures.

Cohort design and natural language processing to reduce bias in electronic health records research

Electronic health record (EHR) datasets are statistically powerful but are subject to ascertainment bias and missingness. Using the Mass General Brigham multi-institutional EHR, we approximated a community-based cohort by sampling patients receiving longitudinal primary care between 2001-2018 (Community Care Cohort Project [C3PO], n = 520,868). We utilized natural language processing (NLP) to recover vital signs from unstructured notes. We assessed the validity of C3PO by deploying established risk models for myocardial infarction/stroke and atrial fibrillation. We then compared C3PO to Convenience Samples including all individuals from the same EHR with complete data, but without a longitudinal primary care requirement. NLP reduced the missingness of vital signs by 31%. NLP-recovered vital signs were highly correlated with values derived from structured fields (Pearson r range 0.95-0.99). Atrial fibrillation and myocardial infarction/stroke incidence were lower and risk models were better calibrated in C3PO as opposed to the Convenience Samples (calibration error range for myocardial infarction/stroke: 0.012-0.030 in C3PO vs. 0.028-0.046 in Convenience Samples; calibration error for atrial fibrillation 0.028 in C3PO vs. 0.036 in Convenience Samples). Sampling patients receiving regular primary care and using NLP to recover missing data may reduce bias and maximize generalizability of EHR research.

Individualised screening of diabetic foot: creation of a prediction model based on penalised regression and assessment of theoretical efficacy

AIMS/HYPOTHESIS

A large proportion of people with diabetes do not receive proper foot screening due to insufficiencies in healthcare systems. Introducing an effective risk prediction model into the screening protocol would potentially reduce the required screening frequency for those considered at low risk for diabetic foot complications. The main aim of the study was to investigate the value of individualised risk assignment for foot complications for optimisation of screening.

METHODS

From 2015 to 2020, 11,878 routine follow-up foot investigations were performed in the tertiary diabetes clinic. From these, 4282 screening investigations with complete data containing all of 18 designated variables collected at regular clinical and foot screening visits were selected for the study sample. Penalised logistic regression models for the prediction of loss of protective sensation (LOPS) and loss of peripheral pulses (LPP) were developed and evaluated.

RESULTS

Using leave-one-out cross validation (LOOCV), the penalised regression model showed an AUC of 0.84 (95% CI 0.82, 0.85) for prediction of LOPS and 0.80 (95% CI 0.78, 0.83) for prediction of LPP. Calibration analysis (based on LOOCV) presented consistent recall of probabilities, with a Brier score of 0.08 (intercept 0.01 [95% CI -0.09, 0.12], slope 1.00 [95% CI 0.92, 1.09]) for LOPS and a Brier score of 0.05 (intercept 0.01 [95% CI -0.12, 0.14], slope 1.09 [95% CI 0.95, 1.22]) for LPP. In a hypothetical follow-up period of 2 years, the regular screening interval was increased from 1 year to 2 years for individuals at low risk. In individuals with an International Working Group on the Diabetic Foot (IWGDF) risk 0, we could show a 40.5% reduction in the absolute number of screening examinations (3614 instead of 6074 screenings) when a 10% risk cut-off was used and a 26.5% reduction (4463 instead of 6074 screenings) when the risk cut-off was set to 5%.

CONCLUSIONS/INTERPRETATION

Enhancement of the protocol for diabetic foot screening by inclusion of a prediction model allows differentiation of individuals with diabetes based on the likelihood of complications. This could potentially reduce the number of screenings needed in those considered at low risk of diabetic foot complications. The proposed model requires further refinement and external validation, but it shows the potential for improving compliance with screening guidelines.

ECG-Based Deep Learning and Clinical Risk Factors to Predict Atrial Fibrillation

BACKGROUND

Artificial intelligence (AI)-enabled analysis of 12-lead ECGs may facilitate efficient estimation of incident atrial fibrillation (AF) risk. However, it remains unclear whether AI provides meaningful and generalizable improvement in predictive accuracy beyond clinical risk factors for AF.

METHODS

We trained a convolutional neural network (ECG-AI) to infer 5-year incident AF risk using 12-lead ECGs in patients receiving longitudinal primary care at Massachusetts General Hospital (MGH). We then fit 3 Cox proportional hazards models, composed of ECG-AI 5-year AF probability, CHARGE-AF clinical risk score (Cohorts for Heart and Aging in Genomic Epidemiology-Atrial Fibrillation), and terms for both ECG-AI and CHARGE-AF (CH-AI), respectively. We assessed model performance by calculating discrimination (area under the receiver operating characteristic curve) and calibration in an internal test set and 2 external test sets (Brigham and Women's Hospital [BWH] and UK Biobank). Models were recalibrated to estimate 2-year AF risk in the UK Biobank given limited available follow-up. We used saliency mapping to identify ECG features most influential on ECG-AI risk predictions and assessed correlation between ECG-AI and CHARGE-AF linear predictors.

RESULTS

The training set comprised 45 770 individuals (age 55±17 years, 53% women, 2171 AF events) and the test sets comprised 83 162 individuals (age 59±13 years, 56% women, 2424 AF events). Area under the receiver operating characteristic curve was comparable using CHARGE-AF (MGH, 0.802 [95% CI, 0.767-0.836]; BWH, 0.752 [95% CI, 0.741-0.763]; UK Biobank, 0.732 [95% CI, 0.704-0.759]) and ECG-AI (MGH, 0.823 [95% CI, 0.790-0.856]; BWH, 0.747 [95% CI, 0.736-0.759]; UK Biobank, 0.705 [95% CI, 0.673-0.737]). Area under the receiver operating characteristic curve was highest using CH-AI (MGH, 0.838 [95% CI, 0.807 to 0.869]; BWH, 0.777 [95% CI, 0.766 to 0.788]; UK Biobank, 0.746 [95% CI, 0.716 to 0.776]). Calibration error was low using ECG-AI (MGH, 0.0212; BWH, 0.0129; UK Biobank, 0.0035) and CH-AI (MGH, 0.012; BWH, 0.0108; UK Biobank, 0.0001). In saliency analyses, the ECG P-wave had the greatest influence on AI model predictions. ECG-AI and CHARGE-AF linear predictors were correlated (Pearson r: MGH, 0.61; BWH, 0.66; UK Biobank, 0.41).

CONCLUSIONS

AI-based analysis of 12-lead ECGs has similar predictive usefulness to a clinical risk factor model for incident AF and the approaches are complementary. ECG-AI may enable efficient quantification of future AF risk.

A data-adaptive Bayesian regression approach for polygenic risk prediction

MOTIVATION

Polygenic risk score (PRS) has been widely exploited for genetic risk prediction due to its accuracy and conceptual simplicity. We introduce a unified Bayesian regression framework, NeuPred, for PRS construction, which accommodates varying genetic architectures and improves overall prediction accuracy for complex diseases by allowing for a wide class of prior choices. To take full advantage of the framework, we propose a summary-statistics-based cross-validation strategy to automatically select suitable chromosome-level priors, which demonstrates a striking variability of the prior preference of each chromosome, for the same complex disease, and further significantly improves the prediction accuracy.

RESULTS

Simulation studies and real data applications with seven disease datasets from the Wellcome Trust Case Control Consortium cohort and eight groups of large-scale genome-wide association studies demonstrate that NeuPred achieves substantial and consistent improvements in terms of predictive r2 over existing methods. In addition, NeuPred has similar or advantageous computational efficiency compared with the state-of-the-art Bayesian methods.

AVAILABILITY AND IMPLEMENTATION

The R package implementing NeuPred is available at https://github.com/shuangsong0110/NeuPred.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Interpretation and reporting of predictive or diagnostic machine-learning research in Trauma & Orthopaedics

There is increasing popularity in the use of artificial intelligence and machine-learning techniques to provide diagnostic and prognostic models for various aspects of Trauma & Orthopaedic surgery. However, correct interpretation of these models is difficult for those without specific knowledge of computing or health data science methodology. Lack of current reporting standards leads to the potential for significant heterogeneity in the design and quality of published studies. We provide an overview of machine-learning techniques for the lay individual, including key terminology and best practice reporting guidelines. Cite this article: Bone Joint J 2021;103-B(12):1754-1758.

Algorithm for Individual Prediction of COVID-19-Related Hospitalization Based on Symptoms: Development and Implementation Study

BACKGROUND

The COVID-19 pandemic has placed a huge strain on the health care system globally. The metropolitan area of Milan, Italy, was one of the regions most impacted by the COVID-19 pandemic worldwide. Risk prediction models developed by combining administrative databases and basic clinical data are needed to stratify individual patient risk for public health purposes.

OBJECTIVE

This study aims to develop a stratification tool aimed at improving COVID-19 patient management and health care organization.

METHODS

A predictive algorithm was developed and applied to 36,834 patients with COVID-19 in Italy between March 8 and the October 9, 2020, in order to foresee their risk of hospitalization. Exposures considered were age, sex, comorbidities, and symptoms associated with COVID-19 (eg, vomiting, cough, fever, diarrhea, myalgia, asthenia, headache, anosmia, ageusia, and dyspnea). The outcome was hospitalizations and emergency department admissions for COVID-19. Discrimination and calibration of the model were also assessed.

RESULTS

The predictive model showed a good fit for predicting COVID-19 hospitalization (C-index 0.79) and a good overall prediction accuracy (Brier score 0.14). The model was well calibrated (intercept -0.0028, slope 0.9970). Based on these results, 118,804 patients diagnosed with COVID-19 from October 25 to December 11, 2020, were stratified into low, medium, and high risk for COVID-19 severity. Among the overall study population, 67,030 (56.42%) were classified as low-risk patients; 43,886 (36.94%), as medium-risk patients; and 7888 (6.64%), as high-risk patients. In all, 89.37% (106,179/118,804) of the overall study population was being assisted at home, 9% (10,695/118,804) was hospitalized, and 1.62% (1930/118,804) died. Among those assisted at home, most people (63,983/106,179, 60.26%) were classified as low risk, whereas only 3.63% (3858/106,179) were classified at high risk. According to ordinal logistic regression, the odds ratio (OR) of being hospitalized or dead was 5.0 (95% CI 4.6-5.4) among high-risk patients and 2.7 (95% CI 2.6-2.9) among medium-risk patients, as compared to low-risk patients.

CONCLUSIONS

A simple monitoring system, based on primary care data sets linked to COVID-19 testing results, hospital admissions data, and death records may assist in the proper planning and allocation of patients and resources during the ongoing COVID-19 pandemic.

Nijeholt G, van Es A, van der Lugt A, Majoie C, Burke J, Roozenbeek B, Lingsma H, Dippel D. Outcome Prediction Models for Endovascular Treatment of Ischemic Stroke: Systematic Review and External Validation

Supplemental Digital Content is available in the text.

Prediction models for outcome of patients with acute ischemic stroke who will undergo endovascular treatment have been developed to improve patient management. The aim of the current study is to provide an overview of preintervention models for functional outcome after endovascular treatment and to validate these models with data from daily clinical practice.

Prediction of thirty-day mortality risk after thoracic endovascular aortic repair for intact descending thoracic aortic aneurysms: Derivation of risk calculator in the Vascular Quality Initiative

OBJECTIVE

Thoracic endovascular aortic repair (TEVAR) for descending thoracic aortic aneurysm (DTAA) is associated with high perioperative survival, although mortality is a possible outcome. However, no risk score has been developed to predict mortality after TEVAR for intact DTAA to aid in risk discussion and preoperative patient selection. Our objective was to use a multi-institutional database to develop a 30-day mortality risk calculator for TEVAR after DTAA repair.

METHODS

The Vascular Quality Initiative database was queried for patients treated with TEVAR for intact DTAA between August 2014 and August 2020. Univariable and multivariable analyses aided in developing a 30-day mortality risk score. Internal validation was done with K-fold cross-validation and calibration curve analysis.

RESULTS

Of 2141 patients included in the analysis, 90 (4.2%) died within 30 days after the procedure. Clinically relevant variables identified to be independently associated with 30-day mortality and therefore used to derive the predictive model included age 75 years or greater (odds ratio [OR], 2.27; 95% confidence interval [CI], 1.50-3.44; P < .001), coronary artery disease (OR, 1.60; 95% CI, 1.03-2.47; P = .036), American Society of Anesthesiologists class IV/V (OR, 2.39; 95% CI, 1.39-4.10; P = .002), urgent vs elective procedure (OR, 3.47; 95% CI, 1.90-6.33; P < .001), emergent vs elective procedure (OR, 5.27; 95% CI, 2.36-11.75; P < .001), prior carotid revascularization (OR, 3.24; 95% CI, 1.64-6.39; P = .001), and proximal landing zone <3 (OR, 2.51; 95% CI, 1.65-3.81; P < .001). The model showed an area under the receiver operating characteristic curve of 0.75. Internal validation demonstrated a bias-corrected area under the receiver operating characteristic curve of 0.73 (95% CI, 0.66-0.79) and a calibration slope of 1.00 with a corresponding intercept of 0.00.

CONCLUSIONS

This study provides a novel clinically relevant risk prediction model to estimate 30-day mortality risk after TEVAR for DTAA. The TEVAR Mortality Risk Calculator provides useful prognostic information to guide patient selection and facilitate preoperative discussions and shared decision making. An easily accessible online version of the TEVAR Mortality Risk Score is available to facilitate ease of use.

COVID Mortality Prediction with Machine Learning Methods: A Systematic Review and Critical Appraisal

More than a year has passed since the report of the first case of coronavirus disease 2019 (COVID), and increasing deaths continue to occur. Minimizing the time required for resource allocation and clinical decision making, such as triage, choice of ventilation modes and admission to the intensive care unit is important. Machine learning techniques are acquiring an increasingly sought-after role in predicting the outcome of COVID patients. Particularly, the use of baseline machine learning techniques is rapidly developing in COVID mortality prediction, since a mortality prediction model could rapidly and effectively help clinical decision-making for COVID patients at imminent risk of death. Recent studies reviewed predictive models for SARS-CoV-2 diagnosis, severity, length of hospital stay, intensive care unit admission or mechanical ventilation modes outcomes; however, systematic reviews focused on prediction of COVID mortality outcome with machine learning methods are lacking in the literature. The present review looked into the studies that implemented machine learning, including deep learning, methods in COVID mortality prediction thus trying to present the existing published literature and to provide possible explanations of the best results that the studies obtained. The study also discussed challenging aspects of current studies, providing suggestions for future developments.

Accuracy of risk models used for public reporting of heart transplant center performance

BACKGROUND

Heart transplant programs and regulatory entities require highly accurate performance metrics to support internal quality improvement activities and national oversight of transplant programs, respectively. We assessed the accuracy of publicly reported performance measures.

METHODS

We used the United Network for Organ Sharing registry to study patients who underwent heart transplantation between January 1, 2016 and June 30, 2018. We used tests of calibration to compare the observed rate of 1-year graft failure to the expected risk of 1-year graft failure, which was calculated for each recipient using the July 2019 method published by the Scientific Registry of Transplant Recipients (SRTR). The primary study outcome was the joint test of calibration, which accounts for both the total number of events predicted (calibration-in-the-large) and dispersion of risk predictions (calibration slope).

RESULTS

6,528 heart transplants were analyzed. The primary test of calibration failed (p <0.0001), indicating poor accuracy of the SRTR model. The calibration-in-the-large statistic (0.63, 95% confidence interval [CI] 0.58-0.68, p < 0.0001) demonstrated overestimation of event rates while the calibration slope statistic (0.56, 95% CI 0.49-0.62, p <0.0001) indicated over-dispersion of event rates. Pre-specified subgroup analyses demonstrated poor calibration for all subgroups (each p <0.01). After recalibration, program-level observed/expected ratios increased by a median of 0.14 (p <0.0001).

CONCLUSIONS

Risk models employed for publicly-reported graft survival at U.S. heart transplant centers lack accuracy in general and in all subgroups tested. The use of disease-specific models may improve the accuracy of program performance metrics.