Calibration: the Achilles heel of predictive analytics

Developing and externally validating multinomial prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: results from an international collaboration

OBJECTIVES

In rheumatology, there is a clinical need to identify patients at high risk (>50%) of not responding to the first-line therapy methotrexate (MTX) due to lack of disease control or discontinuation due to adverse events (AEs). Despite this need, previous prediction models in this context are at high risk of bias and ignore AEs. Our objectives were to (i) develop a multinomial model for outcomes of low disease activity and discontinuing due to AEs 6 months after starting MTX, (ii) update prognosis 3-month following treatment initiation, and (iii) externally validate these models.

STUDY DESIGN AND SETTING

A multinomial model for low disease activity (submodel 1) and discontinuing due to AEs (submodel 2) was developed using data from the UK Rheumatoid Arthritis Medication Study, updated using landmarking analysis, internally validated using bootstrapping, and externally validated in the Norwegian Disease-Modifying Antirheumatic Drug register. Performance was assessed using calibration (calibration-slope and calibration-in-the-large), and discrimination (concordance-statistic and polytomous discriminatory index).

RESULTS

The internally validated model showed good calibration in the development setting with a calibration-slope of 1.01 (0.87, 1.14) (submodel 1) and 0.83 (0.30, 1.34) (submodel 2), and moderate discrimination with a c-statistic of 0.72 (0.69, 0.74) and 0.53 (0.48, 0.59), respectively. Predictive performance decreased after external validation (calibration-slope 0.78 (0.64, 0.93) (submodel 1) and 0.86 (0.34, 1.38) (submodel 2)), which may be due to differences in disease-specific characteristics and outcome prevalence.

CONCLUSION

We addressed previously identified methodological limitations of prediction models for outcomes of MTX therapy. The multinomial approach predicted outcomes of disease activity more accurately than AEs, which should be addressed in future work to aid implementation into clinical practice.

Differences in adverse outcomes across race and ethnicity among Veterans with similar predicted risks of an overdose or suicide-related event

OBJECTIVE

To evaluate the degree to which differences in incidence of mortality and serious adverse events exist across patient race and ethnicity among Veterans Health Administration (VHA) patients receiving outpatient opioid prescriptions and who have similar predicted risks of adverse outcomes. Patients were assigned scores via the VHA Stratification Tool for Opioid Risk Mitigation (STORM), a model used to predict the risk of experiencing overdose- or suicide-related health care events or death. Individuals with the highest STORM risk scores are targeted for case review.

DESIGN

Retrospective cohort study of high-risk veterans who received an outpatient prescription opioid between 4/2018-3/2019.

SETTING

All VHA medical centers.

PARTICIPANTS

In total, 84 473 patients whose estimated risk scores were between 0.0420 and 0.0609, the risk scores associated with the top 5%-10% of risk in the STORM development sample.

METHODS

We examined the expected probability of mortality and serious adverse events (SAEs; overdose or suicide-related events) given a patient's risk score and race.

RESULTS

Given a similar risk score, Black patients were less likely than White patients to have a recorded SAE within 6 months of risk score calculation. Black, Hispanic, and Asian patients were less likely than White patients with similar risk scores to die within 6 months of risk score calculation. Some of the mortality differences were driven by age differences in the composition of racial and ethnic groups in our sample.

CONCLUSIONS

Our results suggest that relying on the STORM model to identify patients who may benefit from an interdisciplinary case review may identify patients with clinically meaningful differences in outcome risk across race and ethnicity.

Development and validation of a risk model for hospital-acquired venous thrombosis: the Medical Inpatients Thrombosis and Hemostasis study

BACKGROUND

Regulatory organizations recommend assessing hospital-acquired (HA) venous thromboembolism (VTE) risk for medical inpatients.

OBJECTIVES

To develop and validate a risk assessment model (RAM) for HA-VTE in medical inpatients using objective and assessable risk factors knowable at admission.

METHODS

The development cohort included people admitted to medical services at the University of Vermont Medical Center (Burlington, Vermont) between 2010 and 2019, and the validation cohorts included people admitted to Hennepin County Medical Center (Minneapolis, Minnesota), University of Michigan Medical Center (Ann Arbor, Michigan), and Harris Health Systems (Houston, Texas). Individuals with VTE at admission, aged <18 years, and admitted for <1 midnight were excluded. We used a Bayesian penalized regression technique to select candidate HA-VTE risk factors for final inclusion in the RAM.

RESULTS

The development cohort included 60 633 admissions and 227 HA-VTE, and the validation cohorts included 111 269 admissions and 651 HA-VTE. Seven HA-VTE risk factors with t statistics ≥1.5 were included in the RAM: history of VTE, low hemoglobin level, elevated creatinine level, active cancer, hyponatremia, increased red cell distribution width, and malnutrition. The areas under the receiver operating characteristic curve and calibration slope were 0.72 and 1.10, respectively. The areas under the receiver operating characteristic curve and calibration slope were 0.70 and 0.93 at Hennepin County Medical Center, 0.70 and 0.87 at the University of Michigan Medical Center, and 0.71 and 1.00 at Harris Health Systems, respectively. The RAM performed well stratified by age, sex, and race.

CONCLUSION

We developed and validated a RAM for HA-VTE in medical inpatients. By quantifying risk, clinicians can determine the potential benefits of measures to reduce HA-VTE.

Algorithm Versus Expert: Machine Learning Versus Surgeon-Predicted Symptom Improvement After Carpal Tunnel Release

BACKGROUND AND OBJECTIVES

Surgeons rely on clinical experience when making predictions about treatment effects. Incorporating algorithm-based predictions of symptom improvement after carpal tunnel release (CTR) could support medical decision-making. However, these algorithm-based predictions need to outperform predictions made by surgeons to add value. We compared predictions of a validated prediction model for symptom improvement after CTR with predictions made by surgeons.

METHODS

This cohort study included 97 patients scheduled for CTR. Preoperatively, surgeons estimated each patient's probability of improvement 6 months after surgery, defined as reaching the minimally clinically important difference on the Boston Carpal Tunnel Syndrome Symptom Severity Score. We assessed model and surgeon performance using calibration (calibration belts), discrimination (area under the curve [AUC]), sensitivity, and specificity. In addition, we assessed the net benefit of decision-making based on the prediction model's estimates vs the surgeon's judgement.

RESULTS

The surgeon predictions had poor calibration and suboptimal discrimination (AUC 0.62, 95%-CI 0.49-0.74), while the prediction model showed good calibration and appropriate discrimination (AUC 0.77, 95%-CI 0.66-0.89, P = .05). The accuracy of surgeon predictions was 0.65 (95%-CI 0.37-0.78) vs 0.78 (95%-CI 0.67-0.89) for the prediction model ( P = .03). The sensitivity of surgeon predictions and the prediction model was 0.72 (95%-CI 0.15-0.96) and 0.85 (95%-CI 0.62-0.97), respectively ( P = .04). The specificity of the surgeon predictions was similar to the model's specificity ( P = .25). The net benefit analysis showed better decision-making based on the prediction model compared with the surgeons' decision-making (ie, more correctly predicted improvements and/or fewer incorrectly predicted improvements).

CONCLUSION

The prediction model outperformed surgeon predictions of improvement after CTR in terms of calibration, accuracy, and sensitivity. Furthermore, the net benefit analysis indicated that using the prediction model instead of relying solely on surgeon decision-making increases the number of patients who will improve after CTR, without increasing the number of unnecessary surgeries.

The Predictive Abilities of Machine Learning Algorithms in Patients with Thoracolumbar Spinal Cord Injuries

OBJECTIVES

The goal of this study is to implement machine learning (ML) algorithms to predict mortality, non-home discharge, prolonged length of stay (LOS), prolonged length of intensive care unit stay (ICU-LOS), and major complications in patients diagnosed with thoracolumbar spinal cord injury, while creating a publicly accessible online tool.

METHODS

The American College of Surgeons Trauma Quality Program database was used to identify patients with thoracolumbar spinal cord injury. Feature selection was performed with the Least Absolute Shrinkage and Selection Operator algorithm. Five ML algorithms, including TabPFN, TabNet, XGBoost, LightGBM, and Random Forest, were used along with the Optuna optimization library for hyperparameter tuning.

RESULTS

A total of 147,819 patients were included in the analysis. For each outcome, we determined the best model for deployment in our web application based on the area under the receiver operating characteristic (AUROC) values. The top performing algorithms were as follows: LightGBM for mortality with an AUROC of 0.885, TabPFN for non-home discharge with an AUROC of 0.801, LightGBM for prolonged LOS with an AUROC of 0.673, Random Forest for prolonged ICU-LOS with an AUROC of 0.664, and LightGBM for major complications with an AUROC of 0.73.

CONCLUSIONS

ML models demonstrate good predictive ability for in-hospital mortality and non-home discharge, fair predictive ability for major complications and prolonged ICU-LOS, but poor predictive ability for prolonged LOS. We have developed a web application that allows these models to be accessed.

Validation of Systematic Coronary Risk Evaluation 2 (SCORE2) and SCORE2-Older Persons in the EPIC-Norfolk prospective population cohort

AIMS

The European Systematic Coronary Risk Evaluation 2 (SCORE2) and SCORE2-Older Persons (OP) models are recommended to identify individuals at high 10-year risk for cardiovascular disease (CVD). Independent validation and assessment of clinical utility is needed. This study aims to assess discrimination, calibration, and clinical utility of low-risk SCORE2 and SCORE2-OP.

METHODS AND RESULTS

Validation in individuals aged 40-69 years (SCORE2) and 70-79 years (SCORE2-OP) without baseline CVD or diabetes from the European Prospective Investigation of Cancer (EPIC) Norfolk prospective population study. We compared 10-year CVD risk estimates with observed outcomes (cardiovascular mortality, non-fatal myocardial infarction, and stroke). For SCORE2, 19 560 individuals (57% women) had 10-year CVD risk estimates of 3.7% [95% confidence interval (CI) 3.6-3.7] vs. observed 3.8% (95% CI 3.6-4.1) [observed (O)/expected (E) ratio 1.0 (95% CI 1.0-1.1)]. The area under the curve (AUC) was 0.75 (95% CI 0.74-0.77), with underestimation of risk in men [O/E 1.4 (95% CI 1.3-1.6)] and overestimation in women [O/E 0.7 (95% CI 0.6-0.8)]. Decision curve analysis (DCA) showed clinical benefit. Systematic Coronary Risk Evaluation 2-Older Persons in 3113 individuals (58% women) predicted 10-year CVD events in 10.2% (95% CI 10.1-10.3) vs. observed 15.3% (95% CI 14.0-16.5) [O/E ratio 1.6 (95% CI 1.5-1.7)]. The AUC was 0.63 (95% CI 0.60-0.65) with underestimation of risk across sex and risk ranges. Decision curve analysis showed limited clinical benefit.

CONCLUSION

In a UK population cohort, the SCORE2 low-risk model showed fair discrimination and calibration, with clinical benefit for preventive treatment initiation decisions. In contrast, in individuals aged 70-79 years, SCORE2-OP demonstrated poor discrimination, underestimated risk in both sexes, and limited clinical utility.

Retrospective validation study of an artificial neural network-based preoperative decision-support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer (ISRCTN14341750)

BACKGROUND

Surgical sentinel lymph node biopsy (SLNB) is routinely used to reliably stage axillary lymph nodes in early breast cancer (BC). However, SLNB may be associated with postoperative arm morbidities. For most patients with BC undergoing SLNB, the findings are benign, and the procedure is currently questioned. A decision-support tool for the prediction of benign sentinel lymph nodes based on preoperatively available data has been developed using artificial neural network modelling.

METHODS

This was a retrospective geographical and temporal validation study of the noninvasive lymph node staging (NILS) model, based on preoperatively available data from 586 women consecutively diagnosed with primary BC at two sites. Ten preoperative clinicopathological characteristics from each patient were entered into the web-based calculator, and the probability of benign lymph nodes was predicted. The performance of the NILS model was assessed in terms of discrimination with the area under the receiver operating characteristic curve (AUC) and calibration, that is, comparison of the observed and predicted event rates of benign axillary nodal status (N0) using calibration slope and intercept. The primary endpoint was axillary nodal status (discrimination, benign [N0] vs. metastatic axillary nodal status [N+]) determined by the NILS model compared to nodal status by definitive pathology.

RESULTS

The mean age of the women in the cohort was 65 years, and most of them (93%) had luminal cancers. Approximately three-fourths of the patients had no metastases in SLNB (N0 74% and 73%, respectively). The AUC for the predicted probabilities for the whole cohort was 0.6741 (95% confidence interval: 0.6255-0.7227). More than one in four patients (n = 151, 26%) were identified as candidates for SLNB omission when applying the predefined cut-off for lymph node-negative status from the development cohort. The NILS model showed the best calibration in patients with a predicted high probability of healthy axilla.

CONCLUSION

The performance of the NILS model was satisfactory. In approximately every fourth patient, SLNB could potentially be omitted. Considering the shift from postoperatively to preoperatively available predictors in this validation study, we have demonstrated the robustness of the NILS model. The clinical usability of the web interface will be evaluated before its clinical implementation.

TRIAL REGISTRATION

Registered in the ISRCTN registry with study ID ISRCTN14341750. Date of registration 23/11/2018.

Development and validation of a predictive model based on clinical and MpMRI findings to reduce additional systematic prostate biopsy

OBJECTIVES

To develop and validate a predictive model based on clinical features and multiparametric magnetic resonance imaging (mpMRI) to reduce unnecessary systematic biopsies (SBs) in biopsy-naïve patients with suspected prostate cancer (PCa).

METHODS

A total of 274 patients who underwent combined cognitive MRI-targeted biopsy (MRTB) with SB were retrospectively enrolled and temporally split into development (n = 201) and validation (n = 73) cohorts. Multivariable logistic regression analyses were used to determine independent predictors of clinically significant PCa (csPCa) on cognitive MRTB, and the clinical, MRI, and combined models were established respectively. Area under the receiver operating characteristic curve (AUC), calibration plots, and decision curve analyses were assessed.

RESULTS

Prostate imaging data and reporting system (PI-RADS) score, index lesion (IL) on the peripheral zone, age, and prostate-specific antigen density (PSAD) were independent predictors and included in the combined model. The combined model achieved the best discrimination (AUC 0.88) as compared to both the MRI model incorporated by PI-RADS score, IL level, and zone (AUC 0.86) and the clinical model incorporated by age and PSAD (AUC 0.70). The combined model also showed good calibration and enabled great net benefit. Applying the combined model as a reference for performing MRTB alone with a cutoff of 60% would reduce 43.8% of additional SB, while missing 2.9% csPCa.

CONCLUSIONS

The combined model based on clinical and mpMRI findings improved csPCa prediction and might be useful in making a decision about which patient could safely avoid unnecessary SB in addition to MRTB in biopsy-naïve patients.

CRITICAL RELEVANCE STATEMENT

The combined model based on clinical and mpMRI findings improved csPCa prediction and might be useful in making a decision about which patient could safely avoid unnecessary SB in addition to MRTB in biopsy-naïve patients.

KEY POINTS

• Age, PSAD, PI-RADS score, and peripheral index lesion were independent predictors of csPCa. • Risk models were used to predict the probability of detecting csPCa on cognitive MRTB. • The combined model might reduce 43.8% of unnecessary SBs, while missing 2.9% csPCa.

Long-term risk prediction after major lower limb amputation: 1-year results of the PERCEIVE study

BACKGROUND

Decision-making when considering major lower limb amputation is complex and requires individualized outcome estimation. It is unknown how accurate healthcare professionals or relevant outcome prediction tools are at predicting outcomes at 1-year after major lower limb amputation.

METHODS

An international, multicentre prospective observational study evaluating healthcare professional accuracy in predicting outcomes 1 year after major lower limb amputation and evaluation of relevant outcome prediction tools identified in a systematic search of the literature was undertaken. Observed outcomes at 1 year were compared with: healthcare professionals' preoperative predictions of death (surgeons and anaesthetists), major lower limb amputation revision (surgeons) and ambulation (surgeons, specialist physiotherapists and vascular nurse practitioners); and probabilities calculated from relevant outcome prediction tools.

RESULTS

A total of 537 patients and 2244 healthcare professional predictions of outcomes were included. Surgeons and anaesthetists had acceptable discrimination (C-statistic = 0.715), calibration and overall performance (Brier score = 0.200) when predicting 1-year death, but performed worse when predicting major lower limb amputation revision and ambulation (C-statistics = 0.627 and 0.662 respectively). Healthcare professionals overestimated the death and major lower limb amputation revision risks. Consultants outperformed trainees, especially when predicting ambulation. Allied healthcare professionals marginally outperformed surgeons in predicting ambulation. Two outcome prediction tools (C-statistics = 0.755 and 0.717, Brier scores = 0.158 and 0.178) outperformed healthcare professionals' discrimination, calibration and overall performance in predicting death. Two outcome prediction tools for ambulation (C-statistics = 0.688 and 0.667) marginally outperformed healthcare professionals.

CONCLUSION

There is uncertainty in predicting 1-year outcomes following major lower limb amputation. Different professional groups performed comparably in this study. Two outcome prediction tools for death and two for ambulation outperformed healthcare professionals and may support shared decision-making.

Development and external validation of a clinical prediction model for predicting quality of recovery up to 1 week after surgery

The Quality of Recovery Score-40 (QoR-40) has been increasingly used for assessing recovery after patients undergoing surgery. However, a prediction model estimating quality of recovery is lacking. The aim of the present study was to develop and externally validate a clinical prediction model that predicts quality of recovery up to one week after surgery. The modelling procedure consisted of two models of increasing complexity (basic and full model). To assess the internal validity of the developed model, bootstrapping (1000 times) was applied. At external validation, the model performance was evaluated according to measures for overall model performance (explained variance (R2)) and calibration (calibration plot and slope). The full model consisted of age, sex, previous surgery, BMI, ASA classification, duration of surgery, HADS and preoperative QoR-40 score. At model development, the R2 of the full model was 0.24. At external validation the R2 dropped as expected. The calibration analysis showed that the QoR-40 predictions provided by the developed prediction models are reliable. The presented models can be used as a starting point for future updating in prediction studies. When the predictive performance is improved it could be implemented clinically in the future.

Disulfidptosis-related prognostic signature correlates with immunotherapy response in colorectal cancer

Disulfidptosis (DSP), a form of cell death caused by disulphide stress, plays an important role in tumour progression. However, the mechanisms by which DSP regulates the tumour microenvironment remain unclear. Thus, we analysed the transcriptome profiles and clinical data, which were obtained from the TCGA database, of 540 patients with colorectal cancer. Compared with the patients with low DSP expression, those with high DSP expression exhibited significantly better survival outcomes; lower stromal and ESTIMATE scores; significantly higher numbers of CD4+ T cells, M2 macrophages, dendritic cells, and neutrophils; higher expression of immune checkpoint-related genes; and lower Tregs and HLA-DQB2 levels. A prognostic signature established based on DSP-related genes demonstrated an increase in risk score with a higher clinical stage. Risk scores negatively correlated with dendritic cells, eosinophils, and CD4+ T cells and significantly positively correlated with Treg cells. Patients with higher risk scores experienced significantly worse survival outcomes and immunotherapy non-response. Our nomogram model, combining clinicopathological features and risk scores, exhibited robust prognostic and predictive power. In conclusion, DSP-related genes actively participated in regulating the tumour microenvironment. Thus, they can serve as biomarkers to provide targeted treatment for colorectal cancer.

Advancing precision prognostication in neuro-oncology: Machine learning models for data-driven personalized survival predictions in IDH-wildtype glioblastoma

Background

Glioblastoma (GBM) remains associated with a dismal prognoses despite standard therapies. While population-level survival statistics are established, generating individualized prognosis remains challenging. We aim to develop machine learning (ML) models that generate personalized survival predictions for GBM patients to enhance prognostication.

Methods

Adult patients with histologically confirmed IDH-wildtype GBM from the National Cancer Database (NCDB) were analyzed. ML models were developed with TabPFN, TabNet, XGBoost, LightGBM, and Random Forest algorithms to predict mortality at 6, 12, 18, and 24 months postdiagnosis. SHapley Additive exPlanations (SHAP) were employed to enhance the interpretability of the models. Models were primarily evaluated using the area under the receiver operating characteristic (AUROC) values, and the top-performing models indicated by the highest AUROCs for each outcome were deployed in a web application that was created for individualized predictions.

Results

A total of 7537 patients were retrieved from the NCDB. Performance evaluation revealed the top-performing models for each outcome were built using the TabPFN algorithm. The TabPFN models yielded mean AUROCs of 0.836, 0.78, 0.732, and 0.724 in predicting 6, 12, 18, and 24 month mortality, respectively.

Conclusions

This study establishes ML models tailored to individual patients to enhance GBM prognostication. Future work should focus on external validation and dynamic updating as new data emerge.

Multi-institutional Development and External Validation of a Machine Learning Model for the Prediction of Distant Metastasis in Patients Treated by Salvage Radiotherapy for Biochemical Failure After Radical Prostatectomy

BACKGROUND

Up to 40% of patients with prostate cancer may develop biochemical recurrence after surgery, with salvage radiation therapy (SRT) being the only curative option. In 2016, Tendulkar et al. (Contemporary update of a multi-institutional predictive nomogram for salvage radiotherapy after radical prostatectomy. J Clin Oncol 2016;34:3648-54) published a nomogram to predict distant metastasis in a cohort of patients treated with SRT with pre-SRT prostate-specific antigen (PSA) of 0.5 ng/ml after radical prostatectomy. In modern practice, SRT is delivered at lower PSA values.

OBJECTIVE

To train and externally validate a machine learning model to predict the risk of distant metastasis at 5 yr in a contemporary cohort of patients receiving SRT.

DESIGN, SETTING, AND PARTICIPANTS

We trained a machine learning model on data from 2418 patients treated with SRT at one institution, with a median PSA value of 0.27 ng/ml. External validation was done in 475 patients treated at two different institutions. Patients with cM1, pN1, or pT4 disease were excluded, as were patients with PSA >2 ng/ml or PSA 0, and patients with radiation dose <60 or ≥80 Gy.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

Model performance was assessed using calibration and time-dependent area under the receiver operating curve (tAUC).

RESULTS AND LIMITATIONS

Our model had better calibration and showed improved discrimination (tAUC = 0.72) compared with the Tendulkar model (tAUC = 0.60, p < 0.001). The main limitations of this study are its retrospective design and lack of validation on patients who received hormone therapy.

CONCLUSIONS

The updated model can be used to provide more individualized risk assessments to patients treated with SRT at low PSA values, improving decision-making.

PATIENT SUMMARY

Up to 40% of patients with prostate cancer may develop biochemical recurrence after surgery, with salvage radiation therapy as the only potentially curative option. We trained and validated a machine learning model using clinical and surgical data to predict a patient's risk of distant metastasis at 5 yr after treatment. Our model outperformed the reference tool and can improve clinical decision-making by providing more personalized risk assessment.

Clinical Utility of Melanoma Sentinel Lymph Node Biopsy Nomograms

BACKGROUND

For patients with melanoma, the decision to perform sentinel lymph node biopsy (SLNB) is based on the estimated risk of lymph node metastasis. We assessed 3 melanoma SLNB risk-prediction models' statistical performance and their ability to improve clinical decision making (clinical utility) on a cohort of melanoma SLNB cases.

STUDY DESIGN

Melanoma patients undergoing SLNB at a single center from 2003 to 2021 were identified. The predicted probabilities of sentinel lymph node positivity using the Melanoma Institute of Australia, Memorial Sloan Kettering Cancer Center (MSK), and Friedman nomograms were calculated. Receiver operating characteristic and calibration curves were generated. Clinical utility was assessed via decision curve analysis, calculating the net SLNBs that could have been avoided had a given model guided selection at different risk thresholds.

RESULTS

Of 2,464 melanoma cases that underwent SLNB, 567 (23.0%) had a positive sentinel lymph node. The areas under the receiver operating characteristic curves for the Melanoma Institute of Australia, MSK, and Friedman models were 0.726 (95% CI, 0.702 to 0.750), 0.720 (95% CI, 0.697 to 0.744), and 0.721 (95% CI, 0.699 to 0.744), respectively. For all models, calibration was best at predicted positivity rates below 30%. The MSK model underpredicted risk. At a 10% risk threshold, only the Friedman model would correctly avoid a net of 6.2 SLNBs per 100 patients. The other models did not reduce net avoidable SLNBs at risk thresholds of ≤10%.

CONCLUSIONS

The tested nomograms had comparable performance in our cohort. The only model that achieved clinical utility at risk thresholds of ≤10% was the Friedman model.

Updating and recalibrating causal probabilistic models on a new target population

OBJECTIVE

Very often the performance of a Bayesian Network (BN) is affected when applied to a new target population. This is mainly because of differences in population characteristics. External validation of the model performance on different populations is a standard approach to test model's generalisability. However, a good predictive performance is not enough to show that the model represents the unique population characteristics and can be adopted in the new environment.

METHODS

In this paper, we present a methodology for updating and recalibrating developed BN models - both their structure and parameters - to better account for the characteristics of the target population. Attention has been given on incorporating expert knowledge and recalibrating latent variables, which are usually omitted from data-driven models.

RESULTS

The method is successfully applied to a clinical case study about the prediction of trauma-induced coagulopathy, where a BN has already been developed for civilian trauma patients and now it is recalibrated on combat casualties.

CONCLUSION

The methodology proposed in this study is important for developing credible models that can demonstrate a good predictive performance when applied to a target population. Another advantage of the proposed methodology is that it is not limited to data-driven techniques and shows how expert knowledge can also be used when updating and recalibrating the model.

Recalibration of a Non-Laboratory-Based Risk Model to Estimate Pre-Diabetes/Diabetes Mellitus Risk in Primary Care in Hong Kong

INTRODUCTION/OBJECTIVES

A non-laboratory-based pre-diabetes/diabetes mellitus (pre-DM/DM) risk prediction model developed from the Hong Kong Chinese population showed good external discrimination in a primary care (PC) population, but the estimated risk level was significantly lower than the observed incidence, indicating poor calibration. This study explored whether recalibrating/updating methods could improve the model's accuracy in estimating individuals' risks in PC.

METHODS

We performed a secondary analysis on the model's predictors and blood test results of 919 Chinese adults with no prior DM diagnosis recruited from PC clinics from April 2021 to January 2022 in HK. The dataset was randomly split in half into a training set and a test set. The model was recalibrated/updated based on a seven-step methodology, including model recalibrating, revising and extending methods. The primary outcome was the calibration of the recalibrated/updated models, indicated by calibration plots. The models' discrimination, indicated by the area under the receiver operating characteristic curves (AUC-ROC), was also evaluated.

RESULTS

Recalibrating the model's regression constant, with no change to the predictors' coefficients, improved the model's accuracy (calibration plot intercept: -0.01, slope: 0.69). More extensive methods could not improve any further. All recalibrated/updated models had similar AUC-ROCs to the original model.

CONCLUSION

The simple recalibration method can adapt the HK Chinese pre-DM/DM model to PC populations with different pre-test probabilities. The recalibrated model can be used as a first-step screening tool and as a measure to monitor changes in pre-DM/DM risks over time or after interventions.

Five-year outcomes for extremely preterm babies with active perinatal management: A clinical prediction model

OBJECTIVE

To develop and validate a clinical prediction model for outcomes at 5 years of age for children born extremely preterm and receiving active perinatal management.

DESIGN

The EPIPAGE-2 national prospective cohort.

SETTING

France, 2011.

POPULATION

Live-born neonates between 24+0 and 26+6  weeks of gestation who received active perinatal management (i.e. birth in a tertiary-level hospital, with antenatal steroids and resuscitation at birth).

METHODS

A prediction model using logistic modelling, including gestational age, small-for gestational-age (SGA) status and sex, was developed. Model performance was assessed through calibration and discrimination, with bootstrap internal validation.

MAIN OUTCOME MEASURES

Survival without moderate or severe neurodevelopmental disability (NDD) at 5 years.

RESULTS

Among the 557 neonates included, 401 (72%) survived to 5 years, of which 59% survived without NDD (95% CI 54% to 63%). Predicted rates of survival without NDD ranged from 45% (95% CI 33% to 57%), to 56% (95% CI 49% to 64%) to 64% (95% CI 57% to 70%) for neonates born at 24, 25 and 26 weeks of gestation, respectively. Predicted rates of survival without NDD were 47% (95% CI 18% to 76%) and 62% (95% CI 49% to 76%) for SGA and non-SGA children, respectively. The model showed good calibration (calibration slope 0.85, 95% CI 0.54 to 1.16; calibration-in-the-large -0.0123, 95% CI -0.25 to 0.23) and modest discrimination (C-index 0.59, 95% CI 0.53 to 0.65).

CONCLUSIONS

A simple prediction model using three factors easily known antenatally may help doctors and families in their decision-making for extremely preterm neonates receiving active perinatal management.

Radiation-induced long-term dysphagia in survivors of head and neck cancer and association with dose-volume parameters

BACKGROUND

Although dysphagia is a common side effect after radiotherapy (RT) of head and neck cancer (HNC), data on long-term dysphagia is scarce. We aimed to 1) compare radiation dose parameters in HNC survivors with and without dysphagia, 2) investigate factors associated with long-term dysphagia and its possible impact on health-related quality of life (HRQoL), and 3) investigate how our data agree with existing NTCP models.

METHODS

This cross-sectional study conducted in 2018-2020, included HNC survivors treated in 2007-2013. Participants attended a one-day examination in hospital and filled in patient questionnaires. Dysphagia was measured with the EORTC QLQ-H&N35 swallowing scale. Toxicity was scored with CTCAE v.4. We contoured swallowing organs at risk (SWOAR) on RT plans, calculated dose-volume histograms (DVHs), performed logistic regression analyses and tested our data in established NTCP models.

RESULTS

Of the 239 participants, 75 (31%) reported dysphagia. Compared to survivors without dysphagia, this group had reduced HRQoL and the DVHs for infrahyoid SWOAR were significantly shifted to the right. Long-term dysphagia was associated with age (OR 1.07, 95% CI 1.03-1.10), female sex (OR 2.75, 95% CI 1.45-5.21), and mean dose to middle pharyngeal constrictor muscle (MD-MPCM) (OR 1.06, 95% CI 1.03-1.09). NTCP models overall underestimated the risk of long-term dysphagia.

CONCLUSIONS

Long-term dysphagia was associated with higher age, being female, and high MD-MPCM. Doses to distally located SWOAR seemed to be risk factors. Existing NTCP models do not sufficiently predict long-term dysphagia. Further efforts are needed to reduce the prevalence and consequences of this late effect.

Predictive Performance for Hospital Mortality of SAPS 3, SOFA, ISS, and New ISS in Critically Ill Trauma Patients: A Validation Cohort Study

Background: It is not known whether anatomical scores perform better than general critical care scores for trauma patients admitted to the intensive care unit (ICU). We compare the predictive performance for hospital mortality of general critical care scores (SAPS 3 and SOFA) with anatomical injury-based scores (Injury Severity Score [ISS] and New ISS [NISS]). Methods: Retrospective cohort study of patients admitted to a specialized trauma ICU from a tertiary hospital in São Paulo, Brazil between May, 2012 and January, 2016. We retrieved data from the ICU database for critical care scores and calculated ISS and NISS from chart data and whole body computed tomography results. We compared the predictive performance for hospital mortality of each model through discrimination, calibration, and decision-curve analysis. Results: The sample comprised 1053 victims of trauma admitted to the ICU, with 84.2% male patients and mean age of 40 (±18) years. Main injury mechanism was blunt trauma (90.7%). Traumatic brain injury was present in 67.8% of patients; 43.3% with severe TBI. At the time of ICU admission, 846 patients (80.3%) were on mechanical ventilation and 644 (64.3%) on vasoactive drugs. Hospital mortality was 23.8% (251). Median SAPS 3 was 41; median maximum SOFA within 24 h of admission, 7; ISS, 29; and NISS, 41. AUROCs (95% CI) were: SAPS 3 = 0.786 (0.756-0.817), SOFA = 0.807 (0.778-0.837), ISS = 0.616 (0.577-0.656), and NISS = 0.689 (0.649-0.729). In pairwise comparisons, SAPS 3 and SOFA did not differ, while both outperformed the anatomical scores (p < .001). Maximum SOFA within 24 h of admission presented the best calibration and net benefit in decision-curve analysis. Conclusions: Trauma-specific anatomical scores have fair performance in critically ill trauma patients and are outperformed by SAPS 3 and SOFA. Illness severity is best characterized by organ dysfunction and physiological variables than anatomical injuries.

Validation of a Machine Learning Model to Predict Immunotherapy Response in Head and Neck Squamous Cell Carcinoma

Head and neck squamous-cell carcinoma (HNSCC) is a disease with a generally poor prognosis; half of treated patients eventually develop recurrent and/or metastatic (R/M) disease. Patients with R/M HNSCC generally have incurable disease with a median survival of 10 to 15 months. Although immune-checkpoint blockade (ICB) has improved outcomes in patients with R/M HNSCC, identifying patients who are likely to benefit from ICB remains a challenge. Biomarkers in current clinical use include tumor mutational burden and immunohistochemistry for programmed death-ligand 1, both of which have only modest predictive power. Machine learning (ML) has the potential to aid in clinical decision-making as an approach to estimate a tumor's likelihood of response or a patient's likelihood of experiencing clinical benefit from therapies such as ICB. Previously, we described a random forest ML model that had value in predicting ICB response using 11 or 16 clinical, laboratory, and genomic features in a pan-cancer development cohort. However, its applicability to certain cancer types, such as HNSCC, has been unknown, due to a lack of cancer-type-specific validation. Here, we present the first validation of a random forest ML tool to predict the likelihood of ICB response in patients with R/M HNSCC. The tool had adequate predictive power for tumor response (area under the receiver operating characteristic curve = 0.65) and was able to stratify patients by overall (HR = 0.53 [95% CI 0.29-0.99], p = 0.045) and progression-free (HR = 0.49 [95% CI 0.27-0.87], p = 0.016) survival. The overall accuracy was 0.72. Our study validates an ML predictor in HNSCC, demonstrating promising performance in a novel cohort of patients. Further studies are needed to validate the generalizability of this algorithm in larger patient samples from additional multi-institutional contexts.

Transportability of bacterial infection prediction models for critically ill patients

OBJECTIVE

Bacterial infections (BIs) are common, costly, and potentially life-threatening in critically ill patients. Patients with suspected BIs may require empiric multidrug antibiotic regimens and therefore potentially be exposed to prolonged and unnecessary antibiotics. We previously developed a BI risk model to augment practices and help shorten the duration of unnecessary antibiotics to improve patient outcomes. Here, we have performed a transportability assessment of this BI risk model in 2 tertiary intensive care unit (ICU) settings and a community ICU setting. We additionally explored how simple multisite learning techniques impacted model transportability.

METHODS

Patients suspected of having a community-acquired BI were identified in 3 datasets: Medical Information Mart for Intensive Care III (MIMIC), Northwestern Medicine Tertiary (NM-T) ICUs, and NM "community-based" ICUs. ICU encounters from MIMIC and NM-T datasets were split into 70/30 train and test sets. Models developed on training data were evaluated against the NM-T and MIMIC test sets, as well as NM community validation data.

RESULTS

During internal validations, models achieved AUROCs of 0.78 (MIMIC) and 0.81 (NM-T) and were well calibrated. In the external community ICU validation, the NM-T model had robust transportability (AUROC 0.81) while the MIMIC model transported less favorably (AUROC 0.74), likely due to case-mix differences. Multisite learning provided no significant discrimination benefit in internal validation studies but offered more stability during transport across all evaluation datasets.

DISCUSSION

These results suggest that our BI risk models maintain predictive utility when transported to external cohorts.

CONCLUSION

Our findings highlight the importance of performing external model validation on myriad clinically relevant populations prior to implementation.

Predicting outcome in acute stroke with large vessel occlusion-application and validation of MR PREDICTS in the ESCAPE-NA1 population

BACKGROUND

Predicting outcome after endovascular treatment for acute ischemic stroke is challenging. We aim to investigate differences between predicted and observed outcomes in patients with acute ischemic stroke treated with endovascular treatment and to evaluate the performance of a validated outcome prediction score.

PATIENTS AND METHODS

MR PREDICTS is an outcome prediction tool based on a logistic regression model designed to predict the treatment benefit of endovascular treatment based on the MR CLEAN and HERMES populations. ESCAPE-NA1 is a randomized trial of nerinetide vs. placebo in patients with acute stroke and large vessel occlusion. We applied MR PREDICTS to patients in the control arm of ESCAPE-NA1. Model performance was assessed by calculating its discriminative ability and calibration.

RESULTS

Overall, 556/1105 patients (50.3%) in the ESCAPE-NA1-trial were randomized to the control arm, 435/556 (78.2%) were treated within 6 h of symptom onset. Good outcome (modified Rankin scale 0-2) at 3 months was achieved in 275/435 patients (63.2%), the predicted probability of good outcome was 52.5%. Baseline characteristics were similar in the study and model derivation cohort except for age (ESCAPE-NA1: mean: 70 y vs. HERMES: 66 y), hypertension (72% vs. 57%), and collaterals (good collaterals, 15% vs. 44%). Compared to HERMES we observed higher rates of successful reperfusion (TICI 2b-3, ESCAPE-NA1: 87% vs. HERMES: 71%) and faster times from symptom onset to reperfusion (median: 201 min vs. 286 min). Model performance was good, indicated by a c-statistic of 0.76 (95%confidence interval: 0.71-0.81).

CONCLUSION

Outcome-prediction using models created from HERMES data, based on information available in the emergency department underestimated the actual outcome in patients with acute ischemic stroke and large vessel occlusion receiving endovascular treatment despite overall good model performance, which might be explained by differences in quality of and time to reperfusion. These findings underline the importance of timely and successful reperfusion for functional outcomes in acute stroke patients.

Simplified Approach to Predicting Obstructive Coronary Disease With Integration of Coronary Calcium: Development and External Validation

BACKGROUND

The Diamond-Forrester model was used extensively to predict obstructive coronary artery disease (CAD) but overestimates probability in current populations. Coronary artery calcium (CAC) is a useful marker of CAD, which is not routinely integrated with other features. We derived simple likelihood tables, integrating CAC with age, sex, and cardiac chest pain to predict obstructive CAD.

METHODS AND RESULTS

The training population included patients from 3 multinational sites (n=2055), with 2 sites for external testing (n=3321). We determined associations between age, sex, cardiac chest pain, and CAC with the presence of obstructive CAD, defined as any stenosis ≥50% on coronary computed tomography angiography. Prediction performance was assessed using area under the receiver-operating characteristic curves (AUCs) and compared with the CAD Consortium models with and without CAC, which require detailed calculations, and the updated Diamond-Forrester model. In external testing, the proposed likelihood tables had higher AUC (0.875 [95% CI, 0.862-0.889]) than the CAD Consortium clinical+CAC score (AUC, 0.868 [95% CI, 0.855-0.881]; P=0.030) and the updated Diamond-Forrester model (AUC, 0.679 [95% CI, 0.658-0.699]; P<0.001). The calibration for the likelihood tables was better than the CAD Consortium model (Brier score, 0.116 versus 0.121; P=0.005).

CONCLUSIONS

We have developed and externally validated simple likelihood tables to integrate CAC with age, sex, and cardiac chest pain, demonstrating improved prediction performance compared with other risk models. Our tool affords physicians with the opportunity to rapidly and easily integrate a small number of important features to estimate a patient's likelihood of obstructive CAD as an aid to clinical management.

Clinical prediction models and the multiverse of madness

BACKGROUND

Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.

MAIN BODY

We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.

CONCLUSIONS

Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Keeping It Simple: Developing a Prognostic Tool for Spinal Epidural Abscess

STUDY DESIGN

A retrospective study.

OBJECTIVE

To develop a prognostic score for mortality and treatment failure in Spinal epidural abscess (SEA), based on simplicity and multidimensional assessment principles.

METHODS

One-hundred-fifty patients were reviewed. Variables assessed included comorbidities, functional status, clinical presentation, Frankel classification, and biochemical and radiological parameters. The main outcomes were the 90-day mortality and treatment failure, corresponding to any intensification of the initial treatment plan. Variables were sorted out with a factorial analysis. Logistic regressions were performed, and the new score was derived from the coefficients. ROC curves with Area Under Curve, calibration plots, and cross-validation were performed.

RESULTS

Forty-three patients (29%) had treatment failure, and 15 died (10%) by 90 days. Factorization created 3 groups: Comorbidities (C), Severity (S), and Function (F). For 90-day mortality, Odds ratios were 1.20 (P = .0002), 1.15, (P = .03), 1.36, (P < 10-4) for C, S, F, respectively. The new score 'CSF' had 1 point per item, ranging from zero to 3. OR increased by 1.2/point for 90-day mortality (P < 10-4), AUC was .86. For failures OR increased by 1.15/point (P = .014), AUC was .58, and increased to .64 for patients who survived after 90 days, probably due to competing risks.

CONCLUSIONS

Comorbidities, Severity, and Function is a new simplistic tool, easy to use in daily practice; its performances were excellent for 90-day mortality, and acceptable for failures. Simple tools are more likely to be adopted into practice. External validation of this technique is desirable.

Developing non-response weights to account for attrition-related bias in a longitudinal pregnancy cohort

BACKGROUND

Prospective cohorts may be vulnerable to bias due to attrition. Inverse probability weights have been proposed as a method to help mitigate this bias. The current study used the "All Our Families" longitudinal pregnancy cohort of 3351 maternal-infant pairs and aimed to develop inverse probability weights using logistic regression models to predict study continuation versus drop-out from baseline to the three-year data collection wave.

METHODS

Two methods of variable selection took place. One method was a knowledge-based a priori variable selection approach, while the second used Least Absolute Shrinkage and Selection Operator (LASSO). The ability of each model to predict continuing participation through discrimination and calibration for both approaches were evaluated by examining area under the receiver operating curve (AUROC) and calibration plots, respectively. Stabilized inverse probability weights were generated using predicted probabilities. Weight performance was assessed using standardized differences of baseline characteristics for those who continue in study and those that do not, with and without weights (unadjusted estimates).

RESULTS

The a priori and LASSO variable selection method prediction models had good and fair discrimination with AUROC of 0.69 (95% Confidence Interval [CI]: 0.67-0.71) and 0.73 (95% CI: 0.71-0.75), respectively. Calibration plots and non-significant Hosmer-Lemeshow Goodness of Fit Tests indicated that both the a priori (p = 0.329) and LASSO model (p = 0.242) were well-calibrated. Unweighted results indicated large (> 10%) standardized differences in 15 demographic variables (range: 11 - 29%), when comparing those who continued in the study with those that did not. Weights derived from the a priori and LASSO models reduced standardized differences relative to unadjusted estimates, with the largest differences of 13% and 5%, respectively. Additionally, when applying the same LASSO variable selection method to develop weights in future data collection waves, standardized differences remained below 10% for each demographic variable.

CONCLUSION

The LASSO variable selection approach produced robust weights that addressed non-response bias more than the knowledge-driven approach. These weights can be applied to analyses across multiple longitudinal waves of data collection to reduce bias.

Machine learning models for blood pressure phenotypes combining multiple polygenic risk scores

We construct non-linear machine learning (ML) prediction models for systolic and diastolic blood pressure (SBP, DBP) using demographic and clinical variables and polygenic risk scores (PRSs). We developed a two-model ensemble, consisting of a baseline model, where prediction is based on demographic and clinical variables only, and a genetic model, where we also include PRSs. We evaluate the use of a linear versus a non-linear model at both the baseline and the genetic model levels and assess the improvement in performance when incorporating multiple PRSs. We report the ensemble model's performance as percentage variance explained (PVE) on a held-out test dataset. A non-linear baseline model improved the PVEs from 28.1% to 30.1% (SBP) and 14.3% to 17.4% (DBP) compared with a linear baseline model. Including seven PRSs in the genetic model computed based on the largest available GWAS of SBP/DBP improved the genetic model PVE from 4.8% to 5.1% (SBP) and 4.7% to 5% (DBP) compared to using a single PRS. Adding additional 14 PRSs computed based on two independent GWASs further increased the genetic model PVE to 6.3% (SBP) and 5.7% (DBP). PVE differed across self-reported race/ethnicity groups, with primarily all non-White groups benefitting from the inclusion of additional PRSs.

Dynamic updating of clinical survival prediction models in a changing environment

BACKGROUND

Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced.

METHODS

We illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK.

RESULTS

In simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance.

CONCLUSIONS

We found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods.

Artificial Intelligence and Machine Learning in Cancer Related Pain: A Systematic Review

Background/objective

Pain is a challenging multifaceted symptom reported by most cancer patients, resulting in a substantial burden on both patients and healthcare systems. This systematic review aims to explore applications of artificial intelligence/machine learning (AI/ML) in predicting pain-related outcomes and supporting decision-making processes in pain management in cancer.

Methods

A comprehensive search of Ovid MEDLINE, EMBASE and Web of Science databases was conducted using terms including "Cancer", "Pain", "Pain Management", "Analgesics", "Opioids", "Artificial Intelligence", "Machine Learning", "Deep Learning", and "Neural Networks" published up to September 7, 2023. The screening process was performed using the Covidence screening tool. Only original studies conducted in human cohorts were included. AI/ML models, their validation and performance and adherence to TRIPOD guidelines were summarized from the final included studies.

Results

This systematic review included 44 studies from 2006-2023. Most studies were prospective and uni-institutional. There was an increase in the trend of AI/ML studies in cancer pain in the last 4 years. Nineteen studies used AI/ML for classifying cancer patients' pain development after cancer therapy, with median AUC 0.80 (range 0.76-0.94). Eighteen studies focused on cancer pain research with median AUC 0.86 (range 0.50-0.99), and 7 focused on applying AI/ML for cancer pain management decisions with median AUC 0.71 (range 0.47-0.89). Multiple ML models were investigated with. median AUC across all models in all studies (0.77). Random forest models demonstrated the highest performance (median AUC 0.81), lasso models had the highest median sensitivity (1), while Support Vector Machine had the highest median specificity (0.74). Overall adherence of included studies to TRIPOD guidelines was 70.7%. Lack of external validation (14%) and clinical application (23%) of most included studies was detected. Reporting of model calibration was also missing in the majority of studies (5%).

Conclusion

Implementation of various novel AI/ML tools promises significant advances in the classification, risk stratification, and management decisions for cancer pain. These advanced tools will integrate big health-related data for personalized pain management in cancer patients. Further research focusing on model calibration and rigorous external clinical validation in real healthcare settings is imperative for ensuring its practical and reliable application in clinical practice.

Attention-based neural networks for clinical prediction modelling on electronic health records

BACKGROUND

Deep learning models have had a lot of success in various fields. However, on structured data they have struggled. Here we apply four state-of-the-art supervised deep learning models using the attention mechanism and compare against logistic regression and XGBoost using discrimination, calibration and clinical utility.

METHODS

We develop the models using a general practitioners database. We implement a recurrent neural network, a transformer with and without reverse distillation and a graph neural network. We measure discrimination using the area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve (AUPRC). We assess smooth calibration using restricted cubic splines and clinical utility with decision curve analysis.

RESULTS

Our results show that deep learning approaches can improve discrimination up to 2.5% points AUC and 7.4% points AUPRC. However, on average the baselines are competitive. Most models are similarly calibrated as the baselines except for the graph neural network. The transformer using reverse distillation shows the best performance in clinical utility on two out of three prediction problems over most of the prediction thresholds.

CONCLUSION

In this study, we evaluated various approaches in supervised learning using neural networks and attention. Here we do a rigorous comparison, not only looking at discrimination but also calibration and clinical utility. There is value in using deep learning models on electronic health record data since it can improve discrimination and clinical utility while providing good calibration. However, good baseline methods are still competitive.

Five critical quality criteria for artificial intelligence-based prediction models

To raise the quality of clinical artificial intelligence (AI) prediction modelling studies in the cardiovascular health domain and thereby improve their impact and relevancy, the editors for digital health, innovation, and quality standards of the European Heart Journal propose five minimal quality criteria for AI-based prediction model development and validation studies: complete reporting, carefully defined intended use of the model, rigorous validation, large enough sample size, and openness of code and software.

External validation of prediction models for vaginal delivery after the trial of labour among women with previous one caesarean section - A cohort study

OBJECTIVE

To externally validate three predictive models (the Grobman model (2007), the Zhang model (2020), and the Grobman model (2021)) for identifying women with increased chances of a successful trial of labour after caesarean section (TOLAC).

METHODS

This retrospective observational cohort study was conducted in a tertiary teaching hospital from 2018 to 2021. Individual probabilities were calculated for women with previous one caesarean section who underwent TOLAC at term, using the predicted probabilities from the logistic regression models. The primary outcome of this study was vaginal delivery following attempted TOLAC. The predictive ability of the models was assessed using the area under the receiver operative characteristics curves (AUC) and a calibration graph.

RESULTS

Of 1515 eligible women who underwent TOLAC, we found an overall rate of successful TOLAC of 60.3 %. No significant difference was noticed in adverse scar outcome and neonatal morbidity while comparing successful and failed TOLAC. The discriminative ability of Grobman-2007 and Grobman-2021 and the Zhang model were fair to poor with the AUC of 0.54(95 % CI 0.51-0.57), 0.62(95 % CI 0.59-0.65) and 0.66(95 % CI 0.63-0.69) respectively. The agreement between the observed rates of TOLAC success and the predicted probabilities for all three models was poor.

CONCLUSION

The performance of all three models predicting success after TOLAC was poor in the study population. A population-specific model may be needed, with the addition of factors influencing the labour, such as the methods of induction, which may aid in predicting the outcome.

Artificial intelligence in the prediction of venous thromboembolism: A systematic review and pooled analysis

BACKGROUND

Accurate diagnostic and prognostic predictions of venous thromboembolism (VTE) are crucial for VTE management. Artificial intelligence (AI) enables autonomous identification of the most predictive patterns from large complex data. Although evidence regarding its performance in VTE prediction is emerging, a comprehensive analysis of performance is lacking.

AIMS

To systematically review the performance of AI in the diagnosis and prediction of VTE and compare it to clinical risk assessment models (RAMs) or logistic regression models.

METHODS

A systematic literature search was performed using PubMed, MEDLINE, EMBASE, and Web of Science from inception to April 20, 2021. Search terms included "artificial intelligence" and "venous thromboembolism." Eligible criteria were original studies evaluating AI in the prediction of VTE in adults and reporting one of the following outcomes: sensitivity, specificity, positive predictive value, negative predictive value, or area under receiver operating curve (AUC). Risks of bias were assessed using the PROBAST tool. Unpaired t-test was performed to compare the mean AUC from AI versus conventional methods (RAMs or logistic regression models).

RESULTS

A total of 20 studies were included. Number of participants ranged from 31 to 111 888. The AI-based models included artificial neural network (six studies), support vector machines (four studies), Bayesian methods (one study), super learner ensemble (one study), genetic programming (one study), unspecified machine learning models (two studies), and multiple machine learning models (five studies). Twelve studies (60%) had both training and testing cohorts. Among 14 studies (70%) where AUCs were reported, the mean AUC for AI versus conventional methods were 0.79 (95% CI: 0.74-0.85) versus 0.61 (95% CI: 0.54-0.68), respectively (p < .001). However, the good to excellent discriminative performance of AI methods is unlikely to be replicated when used in clinical practice, because most studies had high risk of bias due to missing data handling and outcome determination.

CONCLUSION

The use of AI appears to improve the accuracy of diagnostic and prognostic prediction of VTE over conventional risk models; however, there was a high risk of bias observed across studies. Future studies should focus on transparent reporting, external validation, and clinical application of these models.

Development and validation of a digital biomarker predicting acute kidney injury following cardiac surgery on an hourly basis

Objectives

To develop and validate a digital biomarker for predicting the onset of acute kidney injury (AKI) on an hourly basis up to 24 hours in advance in the intensive care unit after cardiac surgery.

Methods

The study analyzed data from 6056 adult patients undergoing coronary artery bypass graft and/or valve surgery between April 1, 2012, and December 31, 2018 (development phase, training, and testing) and 3572 patients between January 1, 2019, and June 30, 2022 (validation phase). The study used 2 dynamic predictive modeling approaches, namely logistic regression and bootstrap aggregated regression trees machine (BARTm), to predict AKI. The mean area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and positive and negative predictive values across all lead times before the occurrence of AKI were reported. The clinical practicality was assessed using calibration.

Results

Of all included patients, 8.45% and 16.66% had AKI in the development and validation phases, respectively. When applied to testing data, AKI was predicted with the mean AUC of 0.850 and 0.802 by BARTm and logistic regression, respectively. When applied to validation data, BARTm and LR resulted in a mean AUC of 0.844 and 0.786, respectively.

Conclusions

This study demonstrated the successful prediction of AKI on an hourly basis up to 24 hours in advance. The digital biomarkers developed and validated in this study have the potential to assist clinicians in optimizing treatment and implementing preventive strategies for patients at risk of developing AKI after cardiac surgery in the intensive care unit.

Stability of clinical prediction models developed using statistical or machine learning methods

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

Prediction of clinical outcomes after percutaneous coronary intervention: Machine-learning analysis of the National Inpatient Sample

BACKGROUND

This study aimed to develop a multiclass machine-learning (ML) model to predict all-cause mortality, ischemic and hemorrhagic events in unselected hospitalized patients undergoing percutaneous coronary intervention (PCI).

METHODS

This retrospective study included 1,815,595 unselected weighted hospitalizations undergoing PCI from the National Inpatient Sample (2016-2019). Five most common ML algorithms (logistic regression, support vector machine (SVM), naive Bayes, random forest (RF), and extreme gradient boosting (XGBoost)) were trained and tested with 101 input features. The study endpoints were different combinations of all-cause mortality, ischemic cerebrovascular events (CVE) and major bleeding. An area under the curve (AUC) with 95% confidence interval (95% CI) was selected as a performance metric.

RESULTS

The study population was split to a training cohort of 1,186,880 PCI discharges, validation cohort (for calibration) of 296,725 hospitalizations and a test cohort of 331,990 PCI discharges. A total of 98,180 (5.4%) hospital entries included study outcomes. Logistic regression, SVM, naive Bayes, and RF model demonstrated AUCs of 0.83 (95% CI 0.82-0.84), 0.84 (95% CI 0.83-0.86), 0.81 (95% CI 0.80-0.82), and 0.83 (95% CI 0.81-0.84), retrospectively. The XGBoost classifier performed the best with an AUC of 0.86 (95% CI 0.85-0.87) with excellent calibration. We then built a web-based application that provides predictions based on the XGBoost model.

CONCLUSION

We derived the multi-task XGBoost classifier based on 101 features to predict different combinations of all-cause death, ischemic CVE and major bleeding. Such models may be useful in benchmarking and risk prediction using routinely collected administrative data.

Survival benefit from liver transplantation for patients with and without hepatocellular carcinoma

Background & Aims

In the USA, inequal liver transplantation (LT) access exists between patients with and without hepatocellular carcinoma (HCC). Survival benefit considers survival without and with LT and could equalise LT access. We calculated bias-corrected LT survival benefit for patients with(out) HCC who underwent a transplant, based on longitudinal data in a recent United States cohort.

Methods

Adult LT candidates with(out) HCC between 2010 and 2019 were included. Waitlist survival over time was contrasted to post-transplant survival, to estimate 5-year survival benefit from the moment of LT. Waitlist survival was modelled with a bias-corrected Cox regression, and post-transplant survival was estimated through Cox proportional hazards regression.

Results

Mean HCC survival without LT was always lower than non-HCC waitlist survival. Below model for end-stage liver disease (sodium) (MELD(-Na)) 30, patients with HCC gained more life-years from LT than patients without HCC at the same MELD(-Na) score. Only patients without HCC below MELD(-Na) 9 had negative benefit. Most patients with HCC underwent a transplant below MELD(-Na) 14, and most patients without HCC underwent a transplant above MELD(-Na) 26. Liver function [MELD(-Na), albumin] was the main predictor of 5-year benefit. Therefore, during 5 years, most patients with HCC gained 0.12 to 1.96 years from LT, whereas most patients without HCC gained 2.48 to 3.45 years.

Conclusions

On an individual level, performing a transplant in patients with HCC resulted in survival benefit. However, on a population level, benefit was indirectly decreased, as patients without HCC were likely to gain more survival owing to decreased liver function. For patients who underwent a transplant, a constructed online calculator estimates 5-year survival benefit given specific patient characteristics. Survival benefit scores could serve to equalise LT access.

Impact and implications

Benefit is a comparison of the survival with and without liver transplantation, and it is important when deciding who should undergo a transplant. Liver function is most important when predicting possible benefit from transplantation. Patients with liver cancer die sooner on the waiting list than similar patients without liver cancer. However, patients with liver cancer more often have better liver function. Most patients without liver cancer derive more benefit from transplantation than patients with liver cancer.

External validation of a nomogram predicting conditional survival after tri-modality treatment of esophageal cancer

BACKGROUND

A conditional survival nomogram was developed at a single high-volume center to predict 5-year overall survival for esophageal cancer patients after neoadjuvant chemoradiation and esophagectomy. The aim of this study was to externally validate the nomogram in a cohort of patients with esophageal adeno- or squamous cell carcinoma from another high-volume center.

METHODS

Consecutive patients with an esophageal adeno- or squamous cell carcinoma who had undergone esophagectomy after being treated with preoperative chemoradiation between 2004 and 2016 were selected from a prospectively maintained institutional database. The level of discrimination for prediction of 5-year overall survival was quantified by Harrell's C statistic. Calibration of the conditional survival nomogram was visualized by plotting predicted 5-year survival and observed 5-year survival for comparison.

RESULTS

Of the 296 patients examined, the probability of 5-year overall survival directly after surgery was 45% and increased to 51%, 68%, 78%, and 89% for each additional year survived. The predicted 5-year overall survival differed from the observed survival, with a calibration slope of 0.54, 0.55, 0.59, 0.73, and 1.09 directly after surgery and 1, 2, 3, and 4 years of survival after surgery, respectively. The nomogram's discrimination level for 5-year survival was moderate, with a C statistic of 0.65 compared to the 0.70 reported in the original study.

CONCLUSION

The nomogram model has moderate predictive discrimination and accuracy, supporting its applicability to external cohorts to predict conditional survival. Further validation studies should empirically assess the model for predictive performance.

External validation and updating of prognostic prediction models for nonrecovery among older adults seeking primary care for back pain

ABSTRACT

Prognostic prediction models for 3 different definitions of nonrecovery were developed in the Back Complaints in the Elders study in the Netherlands. The models' performance was good (optimism-adjusted area under receiver operating characteristics [AUC] curve ≥0.77, R2 ≥0.3). This study aimed to assess the external validity of the 3 prognostic prediction models in the Norwegian Back Complaints in the Elders study. We conducted a prospective cohort study, including 452 patients aged ≥55 years, seeking primary care for a new episode of back pain. Nonrecovery was defined for 2 outcomes, combining 6- and 12-month follow-up data: Persistent back pain (≥3/10 on numeric rating scale) and persistent disability (≥4/24 on Roland-Morris Disability Questionnaire). We could not assess the third model (self-reported nonrecovery) because of substantial missing data (>50%). The models consisted of biopsychosocial prognostic factors. First, we assessed Nagelkerke R2 , discrimination (AUC) and calibration (calibration-in-the-large [CITL], slope, and calibration plot). Step 2 was to recalibrate the models based on CITL and slope. Step 3 was to reestimate the model coefficients and assess if this improved performance. The back pain model demonstrated acceptable discrimination (AUC 0.74, 95% confidence interval: 0.69-0.79), and R2 was 0.23. The disability model demonstrated excellent discrimination (AUC 0.81, 95% confidence interval: 0.76-0.85), and R2 was 0.35. Both models had poor calibration (CITL <0, slope <1). Recalibration yielded acceptable calibration for both models, according to the calibration plots. Step 3 did not improve performance substantially. The recalibrated models may need further external validation, and the models' clinical impact should be assessed.

Development and Validation of the CANHEART Population-Based Laboratory Prediction Models for Atherosclerotic Cardiovascular Disease

BACKGROUND

Prediction of atherosclerotic cardiovascular disease (ASCVD) in primary prevention assessments exclusively with laboratory results may facilitate automated risk reporting and improve uptake of preventive therapies.

OBJECTIVE

To develop and validate sex-specific prediction models for ASCVD using age and routine laboratory tests and compare their performance with that of the pooled cohort equations (PCEs).

DESIGN

Derivation and validation of the CANHEART (Cardiovascular Health in Ambulatory Care Research Team) Lab Models.

SETTING

Population-based cohort study in Ontario, Canada.

PARTICIPANTS

A derivation and internal validation cohort of adults aged 40 to 75 years without cardiovascular disease from April 2009 to December 2015; an external validation cohort of primary care patients from January 2010 to December 2014.

MEASUREMENTS

Age and laboratory predictors measured in the outpatient setting included serum total cholesterol, high-density lipoprotein cholesterol, triglycerides, hemoglobin, mean corpuscular volume, platelets, leukocytes, estimated glomerular filtration rate, and glucose. The ASCVD outcomes were defined as myocardial infarction, stroke, and death from ischemic heart or cerebrovascular disease within 5 years.

RESULTS

Sex-specific models were developed and internally validated in 2 160 497 women and 1 833 147 men. They were well calibrated, with relative differences less than 1% between mean predicted and observed risk for both sexes. The c-statistic was 0.77 in women and 0.71 in men. External validation in 31 697 primary care patients showed a relative difference less than 14% and an absolute difference less than 0.3 percentage points in mean predicted and observed risks for both sexes. The c-statistics for the laboratory models were 0.72 for both sexes and were not statistically significantly different from those for the PCEs in women (change in c-statistic, -0.01 [95% CI, -0.03 to 0.01]) or men (change in c-statistic, -0.01 [CI, -0.04 to 0.02]).

LIMITATION

Medication use was not available at the population level.

CONCLUSION

The CANHEART Lab Models predict ASCVD with similar accuracy to more complex models, such as the PCEs.

PRIMARY FUNDING SOURCE

Canadian Institutes of Health Research.

Risk Stratification Model for Hepatocellular Cancer in Patients With Cirrhosis

BACKGROUND & AIMS

The available risk stratification indices for hepatocellular cancer (HCC) have limited applicability. We developed and externally validated an HCC risk stratification index in U.S. cohorts of patients with cirrhosis.

METHODS

We used data from 2 prospective U.S. cohorts to develop the risk index. Patients with cirrhosis were enrolled from 8 centers and followed until development of HCC, death, or December 31, 2021. We identified an optimal set of predictors with the highest discriminatory ability (C-index) for HCC. The predictors were refit using competing risk regression and its predictive performance was evaluated using the area under the receiver-operating characteristic curve (AUROC). External validation was performed in a cohort of 21,550 patients with cirrhosis seen in the U.S Veterans Affairs system between 2018 and 2019 with follow-up through 2021.

RESULTS

We developed the model in 2431 patients (mean age 60 years, 31% women, 24% cured hepatitis C, 16% alcoholic liver disease, and 29% nonalcoholic fatty liver disease). The selected model had a C-index of 0.77 (95% confidence interval [CI], 0.73-0.81), and the predictors were age, sex, smoking, alcohol use, body mass index, etiology, α-fetoprotein, albumin, alanine aminotransferase, and platelet levels. The AUROCs were 0.75 (95% CI, 0.65-0.85) at 1 year and 0.77 (95% CI, 0.71-0.83) at 2 years, and the model was well calibrated. In the external validation cohort, the AUROC at 2 years was 0.70 with excellent calibration.

CONCLUSION

The risk index, including objective and routinely available risk factors, can differentiate patients with cirrhosis who will develop HCC and help guide discussions regarding HCC surveillance and prevention. Future studies are needed for additional external validation and refinement of risk stratification.

Diagnostic validation of vertebral heart score machine learning algorithm for canine lateral chest radiographs

OBJECTIVES

The vertebral heart score is a measurement used to index heart size relative to thoracic vertebra. Vertebral heart score can be a useful tool for identifying and staging heart disease and providing prognostic information. The purpose of this study is to validate the use of a vertebral heart score algorithm compared to manual vertebral heart scoring by three board-certified veterinary cardiologists.

MATERIALS AND METHODS

A convolutional neural network centred around semantic segmentation of relevant anatomical features was developed to predict heart size and vertebral bodies. These predictions were used to calculate the vertebral heart score. An external validation study consisting of 1200 canine lateral radiographs was randomly selected to match the underlying distribution of vertebral heart scores. Three American College of Veterinary Internal Medicine board-certified cardiologists were enrolled to manually score 400 images each using the traditional Buchanan method. Post-scoring, the cardiologists evaluated the algorithm for misaligned anatomic landmarks and overall image quality.

RESULTS

The 95th percentile absolute difference between the cardiologist vertebral heart score and the algorithm vertebral heart score was 1.05 vertebrae (95% confidence interval: 0.97 to 1.20 vertebrae) with a mean bias of -0.09 vertebrae (95% confidence interval: -0.12 to -0.05 vertebrae). In addition, the model was observed to be well calibrated across the predictive range.

CLINICAL SIGNIFICANCE

We have found the performance of the vertebral heart score algorithm comparable to three board-certified cardiologists. While validation of this vertebral heart score algorithm has shown strong performance compared to veterinarians, further external validation in other clinical settings is warranted before use in those settings.

Exploring the impact of missingness on racial disparities in predictive performance of a machine learning model for emergency department triage

Objective

To investigate how missing data in the patient problem list may impact racial disparities in the predictive performance of a machine learning (ML) model for emergency department (ED) triage.

Materials and Methods

Racial disparities may exist in the missingness of EHR data (eg, systematic differences in access, testing, and/or treatment) that can impact model predictions across racialized patient groups. We use an ML model that predicts patients' risk for adverse events to produce triage-level recommendations, patterned after a clinical decision support tool deployed at multiple EDs. We compared the model's predictive performance on sets of observed (problem list data at the point of triage) versus manipulated (updated to the more complete problem list at the end of the encounter) test data. These differences were compared between Black and non-Hispanic White patient groups using multiple performance measures relevant to health equity.

Results

There were modest, but significant, changes in predictive performance comparing the observed to manipulated models across both Black and non-Hispanic White patient groups; c-statistic improvement ranged between 0.027 and 0.058. The manipulation produced no between-group differences in c-statistic by race. However, there were small between-group differences in other performance measures, with greater change for non-Hispanic White patients.

Discussion

Problem list missingness impacted model performance for both patient groups, with marginal differences detected by race.

Conclusion

Further exploration is needed to examine how missingness may contribute to racial disparities in clinical model predictions across settings. The novel manipulation method demonstrated may aid future research.

Deep Learning-Based Computer-Aided Detection System for Preoperative Chest Radiographs to Predict Postoperative Pneumonia

RATIONALE AND OBJECTIVES

The role of preoperative chest radiography (CR) for prediction of postoperative pneumonia remains uncertain. We aimed to develop and validate a prediction model for postoperative pneumonia incorporating findings of preoperative CRs evaluated by a deep learning-based computer-aided detection (DL-CAD) system MATERIALS AND METHODS: This retrospective study included consecutive patients who underwent surgery between January 2019 and March 2020 and divided into development (surgery in 2019) and validation (surgery between January and March 2020) cohorts. Preoperative CRs obtained within 1-month before surgery were analyzed with a commercialized DL-CAD that provided probability values for the presence of 10 different abnormalities in CRs. Logistic regression models to predict postoperative pneumonia were built using clinical variables (clinical model), and both clinical variables and DL-CAD results for preoperative CRs (DL-CAD model). The discriminative performances of the models were evaluated by area under the receiver operating characteristic curves.

RESULTS

In development cohort (n = 19,349; mean age, 57 years; 11,392 men), DL-CAD results for pulmonary nodules (odds ratio [OR, for 1% increase in probability value], 1.007; p = 0.021), consolidation (OR, 1.019; p < 0.001), and cardiomegaly (OR, 1.013; p < 0.001) were independent predictors of postoperative pneumonia and were included in the DL-CAD model. In validation cohort (n = 4957; mean age, 56 years; 2848 men), the DL-CAD model exhibited a higher AUROC than the clinical model (0.843 vs. 0.815; p = 0.012).

CONCLUSION

Abnormalities in preoperative CRs evaluated by a DL-CAD were independent risk factors for postoperative pneumonia. Using DL-CAD results for preoperative CRs led to an improved prediction of postoperative pneumonia.