Use of Brier score to assess binary predictions

A predictive machine-learning model for clinical decision-making in washed microbiota transplantation on ulcerative colitis

Machine learning based on clinical data and treatment protocols for better clinical decision-making is a current research hotspot. This study aimed to build a machine learning model on washed microbiota transplantation (WMT) for ulcerative colitis (UC), providing patients and clinicians with a new evaluation system to optimize clinical decision-making. Methods Patients with UC who underwent WMT via mid-gut or colonic delivery route at an affiliated hospital of Nanjing Medical University from April 2013 to June 2022 were recruited. Model ensembles based on the clinical indicators were constructed by machine-learning to predict the clinical response of WMT after one month. Results A total of 366 patients were enrolled in this study, with 210 patients allocated for training and internal validation, and 156 patients for external validation. The low level of indirect bilirubin, activated antithrombin III, defecation frequency and cholinesterase and the elderly and high level of creatine kinase, HCO3 - and thrombin time were related to the clinical response of WMT at one month. Besides, the voting ensembles exhibited an area under curve (AUC) of 0.769 ± 0.019 [accuracy, 0.754; F1-score, 0.845] in the internal validation; the AUC of the external validation was 0.614 ± 0.017 [accuracy, 0.801; F1-score, 0.887]. Additionally, the model was available at https://wmtpredict.streamlit.app. Conclusions This study pioneered the development of a machine learning model to predict the one-month clinical response of WMT on UC. The findings demonstrate the potential value of machine learning applications in the field of WMT, opening new avenues for personalized treatment strategies in gastrointestinal disorders. Trial registration clinical trials, NCT01790061. Registered 09 February 2013 - Retrospectively registered, https://clinicaltrials.gov/study/NCT01790061.

Derivation and Internal Validation of a Novel Risk Assessment Tool to Identify Infants and Young Children at Risk for Post-Discharge Mortality in Dar es Salaam, Tanzania and Monrovia, Liberia

OBJECTIVE

To derive and validate internally a novel risk assessment tool to identify young children at risk for all-cause mortality ≤60 days of discharge from hospitals in sub-Saharan Africa.

STUDY DESIGN

We performed a prospective observational cohort study of children aged 1-59 months discharged from Muhimbili National Hospital in Dar es Salaam, Tanzania and John F. Kennedy Medical Center in Monrovia, Liberia (2019-2022). Caregivers received telephone calls up to 60 days after discharge to ascertain participant vital status. We collected socioeconomic, demographic, clinical, and anthropometric data during hospitalization. Candidate variables with P < .20 in bivariate analyses were included in a multivariable logistic regression model with best subset selection to identify risk factors for the outcome. We internally validated our tool using bootstrapping with 500 repetitions.

RESULTS

There were 1933 young children enrolled in the study. The median (IQR) age was 11 (4, 23) months and 58.7% were males. In total, 67 (3.5%) died during follow-up. Ten variables contributed to our tool (total possible score 82). Cancer (aOR 10.6, 95% CI 2.58, 34.6), pedal edema (aOR 6.94, 95% CI 1.69, 22.6), and leaving against medical advice (aOR 6.46, 95% CI 2.46, 15.3) were most predictive of post-discharge mortality. Our risk assessment tool demonstrated good discriminatory value (optimism corrected area under the receiver operating characteristic curve 0.77), high precision, and sufficient calibration.

CONCLUSIONS

After validation, this tool may be used to identify young children at risk for post-discharge mortality to direct resources for follow-up of high-risk children.

Machine learning-based delirium prediction in surgical in-patients: a prospective validation study

Objective

Delirium is a syndrome that leads to severe complications in hospitalized patients, but is considered preventable in many cases. One of the biggest challenges is to identify patients at risk in a hectic clinical routine, as most screening tools cause additional workload. The aim of this study was to validate a machine learning (ML)-based delirium prediction tool on surgical in-patients undergoing a systematic assessment of delirium.

Materials and Methods

738 in-patients of a vascular surgery, a trauma surgery and an orthopedic surgery department were screened for delirium using the DOS scale twice a day over their hospital stay. Concurrently, delirium risk was predicted by the ML algorithm in real-time for all patients at admission and evening of admission. The prediction was performed automatically based on existing EHR data and without any additional documentation needed.

Results

103 patients (14.0%) were screened positive for delirium using the DOS scale. Out of them, 85 (82.5%) were correctly identified by the ML algorithm. Specificity was slightly lower, detecting 463 (72.9%) out of 635 patients without delirium. The AUROC of the algorithm was 0.883 (95% CI, 0.8523-0.9147).

Discussion

In this prospective validation study, the implemented machine-learning algorithm was able to detect patients with delirium in surgical departments with high discriminative performance.

Conclusion

In future, this tool or similar decision support systems may help to replace time-intensive screening tools and enable efficient prevention of delirium.

Coronary CTA-based vascular radiomics predicts atherosclerosis development proximal to LAD myocardial bridging

AIMS

Cardiac cycle morphological changes can accelerate plaque growth proximal to myocardial bridging (MB) in the left anterior descending artery (LAD). To assess coronary computed tomography angiography (CCTA)-based vascular radiomics for predicting proximal plaque development in LAD MB.

METHODS AND RESULTS

Patients with repeated CCTA scans showing LAD MB without proximal plaque in index CCTA were included from Jinling Hospital as a development set. They were divided into training and internal testing in an 8:2 ratio. Patients from four other tertiary hospitals were set as external validation set. The endpoint was proximal plaque development of LAD MB in follow-up CCTA. Four vascular radiomics models were built: MB centreline (MB CL), proximal MB CL (pMB CL), MB cross-section (MB CS), and proximal MB CS (pMB CS), whose performances were evaluated using area under the receiver operating characteristic curve (AUC), integrated discrimination improvement (IDI), and net reclassification improvement (NRI). In total, 295 patients were included in the development (n = 192; median age, 54 ± 11 years; 137 men) and external validation sets (n = 103; median age, 57 ± 9 years; 57 men). The pMB CS vascular radiomics model exhibited higher AUCs in training, internal test, and external sets (AUC = 0.78, 0.75, 0.75) than the clinical and anatomical model (all P < 0.05). Integration of the pMB CS vascular radiomics model significantly raised the AUC of the clinical and anatomical model from 0.56 to 0.75 (P = 0.002), along with enhanced NRI [0.76 (0.37-1.14), P < 0.001] and IDI [0.17 (0.07-0.26), P < 0.001] in the external validation set.

CONCLUSION

The CCTA-based pMB CS vascular radiomics model can predict plaque development in LAD MB.

Acute cholecystitis diagnosis in the emergency department: an artificial intelligence-based approach

OBJECTIVES

This study aimed to assess the diagnostic performance of a support vector machine (SVM) algorithm for acute cholecystitis and evaluate its effectiveness in accurately diagnosing this condition.

METHODS

Using a retrospective analysis of patient data from a single center, individuals with abdominal pain lasting one week or less were included. The SVM model was trained and optimized using standard procedures. Model performance was assessed through sensitivity, specificity, accuracy, and AUC-ROC, with probability calibration evaluated using the Brier score.

RESULTS

Among 534 patients, 198 (37.07%) were diagnosed with acute cholecystitis. The SVM model showed balanced performance, with a sensitivity of 83.08% (95% CI: 71.73-91.24%), a specificity of 80.21% (95% CI: 70.83-87.64%), and an accuracy of 81.37% (95% CI: 74.48-87.06%). The positive predictive value (PPV) was 73.97% (95% CI: 65.18-81.18%), the negative predictive value (NPV) was 87.50% (95% CI: 80.19-92.37%), and the AUC-ROC was 0.89 (95% CI: 0.85 to 0.93). The Brier score indicated well-calibrated probability estimates.

CONCLUSION

The SVM algorithm demonstrated promising potential for accurately diagnosing acute cholecystitis. Further refinement and validation are needed to enhance its reliability in clinical practice.

Center of Pressure- and Machine Learning-based Gait Score and Clinical Risk Factors for Predicting Functional Outcome in Acute Ischemic Stroke

OBJECTIVES

To investigate whether machine learning (ML)-based center of pressure (COP) analysis for gait assessment, when used in conjunction with clinical information, offers additive benefits in predicting functional outcomes in patients with acute ischemic stroke.

DESIGN

A prospective, single-center cohort study.

SETTING

A tertiary hospital setting.

PARTICIPANTS

A total of 185 patients with acute ischemic stroke, capable of walking 10 m with or without a gait aid by day 7 postadmission. From these patients, 10,804 pairs of consecutive footfalls were included for analysis.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

The dependent variable was a 3-month poor functional outcome, defined as modified Rankin scale score ≥2. For independent variables, 65 clinical variables including demographics, anthropometrics, comorbidities, laboratory data, questionnaires, and drug history were included. Gait function was evaluated using a pressure-sensitive mat. Time-series COP data were parameterized into spatial and temporal variables and analyzed with logistic regression and 2 ML models (light gradient-boosting machine and multilayer perceptron [MLP]). We derived GAIT-AI output scores from the best-performing model analyzed COP data and constructed multivariable logistic regression models using clinical variables and the GAIT scores.

RESULTS

Among the included patients, 70 (37.8%) experienced unfavorable outcomes. The MLP model demonstrated the highest predictive performance with an area under the receiver operating characteristic curve (AUROC) of 0.799. Multivariable logistic regression identified age, initial National Institutes of Health Stroke Scale, and initial Fall Efficacy Scale-International as associated factors with unfavorable outcomes. The combined multivariable logistic regression incorporating COP-derived output scores improved the AUROC to 0.812.

CONCLUSIONS

Gait function, assessed through COP analysis, serves as a significant predictor of functional outcome in patients with acute ischemic stroke. ML-based COP analysis, when combined with clinical data, enhances the prediction of poor functional outcomes.

Development and validation of a novel nomogram to avoid unnecessary biopsy in patients with PI-RADS category ≥ 4 lesions and PSA ≤ 20 ng/ml

OBJECTIVES

To develop and validate a prediction model for identifying non-prostate cancer (non-PCa) in biopsy-naive patients with PI-RADS category ≥ 4 lesions and PSA ≤ 20 ng/ml to avoid unnecessary biopsy.

PATIENTS AND METHODS

Eligible patients who underwent transperineal biopsies at West China Hospital between 2018 and 2022 were included. The patients were randomly divided into training cohort (70%) and validation cohort (30%). Logistic regression was used to screen for independent predictors of non-PCa, and a nomogram was constructed based on the regression coefficients. The discrimination and calibration were assessed by the C-index and calibration plots, respectively. Decision curve analysis (DCA) and clinical impact curves (CIC) were applied to measure the clinical net benefit.

RESULTS

A total of 1580 patients were included, with 634 non-PCa. Age, prostate volume, prostate-specific antigen density (PSAD), apparent diffusion coefficient (ADC) and lesion zone were independent predictors incorporated into the optimal prediction model, and a corresponding nomogram was constructed ( https://nomogramscu.shinyapps.io/PI-RADS-4-5/ ). The model achieved a C-index of 0.931 (95% CI, 0.910-0.953) in the validation cohort. The DCA and CIC demonstrated an increased net benefit over a wide range of threshold probabilities. At biopsy-free thresholds of 60%, 70%, and 80%, the nomogram was able to avoid 74.0%, 65.8%, and 55.6% of unnecessary biopsies against 9.0%, 5.0%, and 3.6% of missed PCa (or 35.9%, 30.2% and 25.1% of foregone biopsies, respectively).

CONCLUSION

The developed nomogram has favorable predictive capability and clinical utility can help identify non-PCa to support clinical decision-making and reduce unnecessary prostate biopsies.

Thy-DAMP: deep artificial neural network model for prediction of thyroid cancer mortality

PURPOSE

Despite the rising incidence of differentiated thyroid cancer (DTC), mortality rates have remained relatively low yet crucial for effective patient management. This study aims to develop a deep neural network capable of predicting mortality in patients with differentiated thyroid cancer.

METHODS

Leveraging data from the Surveillance, Epidemiology, and End Results (SEER) database, we developed Thy-DAMP (Deep Artificial Neural Network Model for Prediction of Thyroid Cancer) to forecast mortality in DTC patients. The dataset comprised demographic, histologic, and staging information. Following data normalization and feature encoding, the dataset was partitioned into training, testing, and validation subsets, with model hyperparameters fine-tuned via cross-validation.

RESULTS

Among the 63,513 patients, the mean age was 48.22 years (SD = 14.8), with 77.32% being female. Papillary carcinoma emerged as the predominant subtype, representing 62.94% of cases. The majority presented with stage I disease (73.96%). Thy-DAMP demonstrated acceptable performance metrics on both the test and validation sets. Sensitivity was 83.24% (95% CI 76.95-88.40%), specificity was 93.53% (95% CI 93.01-94.02%), and accuracy stood at 93.33% (95% CI 92.82-93.83%). The model exhibited a positive predictive value of 19.76% (95% CI 18.20-21.42%) and a negative predictive value of 99.66% (95% CI 99.53-99.75%). Additionally, Thy-DAMP demonstrated a positive likelihood ratio of 12.86 (95% CI 11.62-14.23), a negative likelihood ratio of 0.18 (95% CI 0.13-0.25), and an area under the receiver operating characteristic curve (AUROC) of 0.95. The model was externally validated on a separate dataset with nearly identical performance.

CONCLUSION

Thy-DAMP showcases considerable promise in accurately predicting mortality in DTC patients, leveraging limited set of patient data.

Accurate diagnosis of acute appendicitis in the emergency department: an artificial intelligence-based approach

The diagnosis of abdominal pain in emergency departments is challenging, and appendicitis is a common concern. Atypical symptoms often delay diagnosis. Although the Alvarado score aids in decision-making, its low specificity can lead to unnecessary surgeries. By leveraging machine learning, we aim to enhance diagnostic accuracy by predicting appendicitis and distinguishing it from other causes of abdominal pain in the emergency department. Data were collected from 534 patients who presented with acute abdominal pain. Patient characteristics, laboratory results, and causes of pain were recorded. Machine learning algorithms (support vector classifier, random forest classifier, gradient boosting classifier, and Gaussian naive Bayes) were used to predict the cause of pain. Model calibration was assessed using the Brier score. The mean age was 46.89 (20.3) years, with an almost equal sex distribution (49% male, 51% female). Cholecystitis was the most prevalent outcome (37.07%), followed by appendicitis (25.84%). The Gaussian naive Bayes model exhibited superior performance in terms of accuracy (95.03% 95% CI 90.44-97.83%), sensitivity (87.18% 95% CI 72.57-95.70%), and specificity (97.54% 95% CI 92.98-99.49%), while the random forest model showed a sensitivity of 79.49%, specificity of 96.72%, and accuracy of 92.55%. The gradient boosting algorithm achieved a sensitivity, specificity, and accuracy of 89.74%, 95.90%, and 94.41%, respectively. The support vector classifier demonstrated a sensitivity of 89.74%, specificity of 92.62%, and accuracy of 91.93%. The use of modern machine learning methods aids in the accurate diagnosis of appendicitis.

American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator as a Predictor of Postoperative Outcomes After Adult Spinal Deformity Surgery: A Retrospective Cohort Analysis

BACKGROUND AND OBJECTIVES

In recent years, there has been an outpouring of scoring systems that were built to predict outcomes after various surgical procedures; however, research validating these studies in spinal surgery is quite limited. In this study, we evaluated the predictability of the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator (ACS NSQIP SRC) for various postoperative outcomes after spinal deformity surgery.

METHODS

A retrospective chart review was conducted to identify patients who underwent spinal deformity surgery at our hospital between January 1, 2014, and December 31, 2022. Demographic and clinical data necessary to use the ACS NSQIP SRC and postoperative outcomes were collected for these patients. Predictability was analyzed using the area under the curve (AUC) of receiver operating characteristic curves and Brier scores.

RESULTS

Among the 159 study patients, the mean age was 64.5 ± 9.5 years, mean body mass index was 31.9 ± 6.6, and 95 (59.7%) patients were women. The outcome most accurately predicted by the ACS NSQIP SRC was postoperative pneumonia (observed = 5.0% vs predicted = 3.2%, AUC = 0.75, Brier score = 0.05), but its predictability still fell below the acceptable threshold. Other outcomes that were underpredicted by the ACS NSQIP SRC were readmission within 30 days (observed = 13.8% vs predicted = 9.0%, AUC = 0.63, Brier score = 0.12), rate of discharge to nursing home or rehabilitation facilities (observed = 56.0% vs predicted = 46.6%, AUC = 0.59, Brier = 0.26), reoperation (observed 11.9% vs predicted 5.4%, AUC = 0.60, Brier = 0.11), surgical site infection (observed 9.4% vs predicted 3.5%, AUC = 0.61, Brier = 0.05), and any complication (observed 33.3% vs 19%, AUC = 0.65, Brier = 0.23). Predicted and observed length of stay were not significantly associated (β = 0.132, P = .47).

CONCLUSION

The ACS NSQIP SRC is a poor predictor of outcomes after spinal deformity surgery.

A joint model for the estimation of species distributions and environmental characteristics from point-referenced data

BACKGROUND

Predicting and explaining species occurrence using environmental characteristics is essential for nature conservation and management. Species distribution models consider species occurrence as the dependent variable and environmental conditions as the independent variables. Suitable conditions are estimated based on a sample of species observations, where one assumes that the underlying environmental conditions are known. This is not always the case, as environmental variables at broad spatial scales are regularly extrapolated from point-referenced data. However, treating the predicted environmental conditions as accurate surveys of independent variables at a specific point does not take into account their uncertainty.

METHODS

We present a joint hierarchical Bayesian model where models for the environmental variables, rather than a set of predicted values, are input to the species distribution model. All models are fitted together based only on point-referenced observations, which results in a correct propagation of uncertainty. We use 50 plant species representative of the Dutch flora in natural areas with 8 soil condition predictors taken during field visits in the Netherlands as a case study. We compare the proposed model to the standard approach by studying the difference in associations, predicted maps, and cross-validated accuracy.

FINDINGS

We find that there are differences between the two approaches in the estimated association between soil conditions and species occurrence (correlation 0.64-0.84), but the predicted maps are quite similar (correlation 0.82-1.00). The differences are more pronounced in the rarer species. The cross-validated accuracy is substantially better for 5 species out of the 50, and the species can also help to predict the soil characteristics. The estimated associations tend to have a smaller magnitude with more certainty.

CONCLUSION

These findings suggests that the standard model is often sufficient for prediction, but effort should be taken to develop models which take the uncertainty in the independent variables into account for interpretation.

Prostate Cancer Diagnosis via Visual Representation of Tabular Data and Deep Transfer Learning

Prostate cancer (PC) is a prevalent and potentially fatal form of cancer that affects men globally. However, the existing diagnostic methods, such as biopsies or digital rectal examination (DRE), have limitations in terms of invasiveness, cost, and accuracy. This study proposes a novel machine learning approach for the diagnosis of PC by leveraging clinical biomarkers and personalized questionnaires. In our research, we explore various machine learning methods, including traditional, tree-based, and advanced tabular deep learning methods, to analyze tabular data related to PC. Additionally, we introduce the novel utilization of convolutional neural networks (CNNs) and transfer learning, which have been predominantly applied in image-related tasks, for handling tabular data after being transformed to proper graphical representations via our proposed Tab2Visual modeling framework. Furthermore, we investigate leveraging the prediction accuracy further by constructing ensemble models. An experimental evaluation of our proposed approach demonstrates its effectiveness in achieving superior performance attaining an F1-score of 0.907 and an AUC of 0.911. This offers promising potential for the accurate detection of PC without the reliance on invasive and high-cost procedures.

Transformer-Based Multilabel Deep Learning Model Is Efficient for Detecting Ankle Lateral and Medial Ligament Injuries on Magnetic Resonance Imaging and Improving Clinicians' Diagnostic Accuracy for Rotational Chronic Ankle Instability

PURPOSE

To develop a deep learning (DL) model that can simultaneously detect lateral and medial collateral ligament injuries of the ankle, aiding in the diagnosis of chronic ankle instability (CAI), and assess its impact on clinicians' diagnostic performance.

METHODS

DL models were developed and externally validated on retrospectively collected ankle magnetic resonance imaging (MRI) between April 2016 and March 2022 respectively at 3 centers. Included patients had confirmed diagnoses of CAI through arthroscopy, as well as individuals who had undergone MRI and physical examinations that ruled out ligament injuries. DL models were constructed based on a multilabel paradigm. A transformer-based multilabel DL model (AnkleNet) was developed and compared with 4 convolution neural network (CNN) models. Subsequently, a reader study was conducted to evaluate the impact of model assistance on clinicians when diagnosing challenging cases: identifying rotational CAI (RCAI). Diagnostic performance was assessed using area under the receiver operating characteristic curve (AUC).

RESULTS

Our transformer-based model achieved AUCs of 0.910 and 0.892 for detecting lateral and medial collateral ligament injury, respectively, both of which were significantly higher than those of CNN-based models (all P < .001). In terms of further CAI diagnosis, there was a macro-average AUC of 0.870 and a balanced accuracy of 0.805. The reader study indicated that incorporation with our model significantly enhanced the diagnostic accuracy of clinicians (P = .042), particularly junior clinicians, and led to a reduction in diagnostic variability. The code of the model can be accessed at https://github.com/ChiariRay/AnkleNet.

CONCLUSIONS

Our transformer-based model was able to detect lateral and medial collateral ligament injuries based on MRI and outperformed CNN-based models, demonstrating a promising performance in diagnosing CAI, especially patients with RCAI.

CLINICAL RELEVANCE

Developing such an algorithm can improve the diagnostic performance of clinicians, aiding in identifying patients who would benefit from arthroscopy, such as patients with RCAI.

ResDeepSurv: A Survival Model for Deep Neural Networks Based on Residual Blocks and Self-attention Mechanism

Survival analysis, as a widely used method for analyzing and predicting the timing of event occurrence, plays a crucial role in the medicine field. Medical professionals utilize survival models to gain insight into the effects of patient covariates on the disease, and the correlation with the effectiveness of different treatment strategies. This knowledge is essential for the development of treatment plans and the enhancement of treatment approaches. Conventional survival models, such as the Cox proportional hazards model, require a significant amount of feature engineering or prior knowledge to facilitate personalized modeling. To address these limitations, we propose a novel residual-based self-attention deep neural network for survival modeling, called ResDeepSurv, which combines the benefits of neural networks and the Cox proportional hazards regression model. The model proposed in our study simulates the distribution of survival time and the correlation between covariates and outcomes, but does not impose strict assumptions on the basic distribution of survival data. This approach effectively accounts for both linear and nonlinear risk functions in survival data analysis. The performance of our model in analyzing survival data with various risk functions is on par with or even superior to that of other existing survival analysis methods. Furthermore, we validate the superior performance of our model in comparison to currently existing methods by evaluating multiple publicly available clinical datasets. Through this study, we prove the effectiveness of our proposed model in survival analysis, providing a promising alternative to traditional approaches. The application of deep learning techniques and the ability to capture complex relationships between covariates and survival outcomes without relying on extensive feature engineering make our model a valuable tool for personalized medicine and decision-making in clinical practice.

AB2CO risk score for in-hospital mortality of COVID-19 patients admitted to intensive care units

PURPOSE

To develop a mortality risk score for COVID-19 patients admitted to intensive care units (ICU), and to compare it with other existing scores.

MATERIALS AND METHODS

This retrospective observational study included consecutive adult patients with laboratory-confirmed COVID-19 admitted to ICUs of 18 hospitals from nine Brazilian cities, from September 2021 to July 2022. Potential predictors were selected based on the literature review. Generalized Additive Models were used to examine outcomes and predictors. LASSO regression was used to derive the mortality score.

RESULTS

From 558 patients, median age was 69 years (IQR 58-78), 56.3 % were men, 19.7 % required mechanical ventilation (MV), and 44.8 % died. The final model comprised six variables: age, pO2/FiO2, respiratory function (respiratory rate or if in MV), chronic obstructive pulmonary disease, and obesity. The AB2CO had an AUROC of 0.781 (95 % CI 0.744 to 0.819), good overall performance (Brier score = 0.191) and an excellent calibration (slope = 1.063, intercept = 0.015, p-value = 0.834). The model was compared with other scores and displayed better discrimination ability than the majority of them.

CONCLUSIONS

The AB2CO score is a fast and easy tool to be used upon ICU admission.

Clinical use of polygenic risk scores for detection of peripheral artery disease and cardiovascular events

We have previously shown that polygenic risk scores (PRS) can improve risk stratification of peripheral artery disease (PAD) in a large, retrospective cohort. Here, we evaluate the potential of PRS in improving the detection of PAD and prediction of major adverse cardiovascular and cerebrovascular events (MACCE) and adverse events (AE) in an institutional patient cohort. We created a cohort of 278 patients (52 cases and 226 controls) and fit a PAD-specific PRS based on the weighted sum of risk alleles. We built traditional clinical risk models and machine learning (ML) models using clinical and genetic variables to detect PAD, MACCE, and AE. The models' performances were measured using the area under the curve (AUC), net reclassification index (NRI), integrated discrimination improvement (IDI), and Brier score. We also evaluated the clinical utility of our PAD model using decision curve analysis (DCA). We found a modest, but not statistically significant improvement in the PAD detection model's performance with the inclusion of PRS from 0.902 (95% CI: 0.846-0.957) (clinical variables only) to 0.909 (95% CI: 0.856-0.961) (clinical variables with PRS). The PRS inclusion significantly improved risk re-classification of PAD with an NRI of 0.07 (95% CI: 0.002-0.137), p = 0.04. For our ML model predicting MACCE, the addition of PRS did not significantly improve the AUC, however, NRI analysis demonstrated significant improvement in risk re-classification (p = 2e-05). Decision curve analysis showed higher net benefit of our combined PRS-clinical model across all thresholds of PAD detection. Including PRS to a clinical PAD-risk model was associated with improvement in risk stratification and clinical utility, although we did not see a significant change in AUC. This result underscores the potential clinical utility of incorporating PRS data into clinical risk models for prevalent PAD and the need for use of evaluation metrics that can discern the clinical impact of using new biomarkers in smaller populations.

Using Machine Learning to Evaluate the Value of Genetic Liabilities in the Classification of Hypertension within the UK Biobank

Background and Objective: Hypertension increases the risk of cardiovascular diseases (CVD) such as stroke, heart attack, heart failure, and kidney disease, contributing to global disease burden and premature mortality. Previous studies have utilized statistical and machine learning techniques to develop hypertension prediction models. Only a few have included genetic liabilities and evaluated their predictive values. This study aimed to develop an effective hypertension classification model and investigate the potential influence of genetic liability for multiple risk factors linked to CVD on hypertension risk using the random forest and the neural network. Materials and Methods: The study involved 244,718 European participants, who were divided into training and testing sets. Genetic liabilities were constructed using genetic variants associated with CVD risk factors obtained from genome-wide association studies (GWAS). Various combinations of machine learning models before and after feature selection were tested to develop the best classification model. The models were evaluated using area under the curve (AUC), calibration, and net reclassification improvement in the testing set. Results: The models without genetic liabilities achieved AUCs of 0.70 and 0.72 using the random forest and the neural network methods, respectively. Adding genetic liabilities improved the AUC for the random forest but not for the neural network. The best classification model was achieved when feature selection and classification were performed using random forest (AUC = 0.71, Spiegelhalter z score = 0.10, p-value = 0.92, calibration slope = 0.99). This model included genetic liabilities for total cholesterol and low-density lipoprotein (LDL). Conclusions: The study highlighted that incorporating genetic liabilities for lipids in a machine learning model may provide incremental value for hypertension classification beyond baseline characteristics.

Development of an MRI-Based Prediction Model for Anti-TNF Treatment Failure in Perianal Crohn's Disease: A Multicenter Study

BACKGROUND & AIMS

Clinical and radiologic variables associated with perianal fistula (PAF) outcomes are poorly understood. We developed prediction models for anti-tumor necrosis factor (TNF) treatment failure in patients with Crohn's disease-related PAF.

METHODS

In a multicenter retrospective study between 2005 and 2022 we included biologic-naive adults (>17 years) who initiated their first anti-TNF therapy for PAF after pelvic magnetic resonance imaging (MRI). Pretreatment MRI studies were prospectively reread centrally by blinded radiologists. We developed and internally validated a prediction model based on clinical and radiologic parameters to predict the likelihood of anti-TNF treatment failure, clinically, at 6 months. We compared our model and a simplified version of MRI parameters alone with existing imaging-based PAF activity indices (MAGNIFI-CD and modified Van Assche MRI scores) by De Long statistical test.

RESULTS

We included 221 patients: 32 ± 14 years, 60% males, 76% complex fistulas; 68% treated with infliximab and 32% treated with adalimumab. Treatment failure occurred in 102 (46%) patients. Our prediction model included age at PAF diagnosis, time to initiate anti-TNF treatment, and smoking and 8 MRI characteristics (supra/extrasphincteric anatomy, fistula length >4.3 cm, primary tracts >1, secondary tracts >1, external openings >1, tract hyperintensity on T1-weighted imaging, horseshoe anatomy, and collections >1.3 cm). Our full and simplified MRI models had fair discriminatory capacity for anti-TNF treatment failure (concordance statistic, 0.67 and 0.65, respectively) and outperformed MAGNIFI-CD (P = .002 and < .0005) and modified Van Assche MRI scores (P < .0001 and < .0001), respectively.

CONCLUSIONS

Our risk prediction models consisting of clinical and/or radiologic variables accurately predict treatment failure in patients with PAF.

Deep learning model for predicting postoperative survival of patients with gastric cancer

Background

Prognostic prediction for surgical treatment of gastric cancer remains valuable in clinical practice. This study aimed to develop survival models for postoperative gastric cancer patients.

Methods

Eleven thousand seventy-five patients from the Surveillance, Epidemiology, and End Results (SEER) database were included, and 122 patients from the Chinese database were used for external validation. The training cohort was created to create three separate models, including Cox regression, RSF, and DeepSurv, using data from the SEER database split into training and test cohorts with a 7:3 ratio. Test cohort was used to evaluate model performance using c-index, Brier scores, calibration, and the area under the curve (AUC). The new risk stratification based on the best model will be compared with the AJCC stage on the test and Chinese cohorts using decision curve analysis (DCA), the net reclassification index (NRI), and integrated discrimination improvement (IDI).

Results

It was discovered that the DeepSurv model predicted postoperative gastric cancer patients' overall survival (OS) with a c-index of 0.787; the area under the curve reached 0.781, 0.798, 0.868 at 1-, 3- and 5- years, respectively; the Brier score was below 0.25 at different time points; showing an advantage over the Cox and RSF models. The results are also validated in the China cohort. The calibration plots demonstrated good agreement between the DeepSurv model's forecast and actual results. The NRI values (test cohort: 0.399, 0.288, 0.267 for 1-, 3- and 5-year OS prediction; China cohort:0.399, 0.288 for 1- and 3-year OS prediction) and IDI (test cohort: 0.188, 0.169, 0.157 for 1-, 3- and 5-year OS prediction; China cohort: 0.189, 0.169 for 1- and 3-year OS prediction) indicated that the risk score stratification performed significantly better than the AJCC staging alone (P < 0.05). DCA showed that the risk score stratification was clinically useful and had better discriminative ability than the AJCC staging. Finally, an interactive native web-based prediction tool was constructed for the survival prediction of patients with postoperative gastric cancer.

Conclusion

In this study, a high-performance prediction model for the postoperative prognosis of gastric cancer was developed using DeepSurv, which offers essential benefits for risk stratification and prognosis prediction for each patient.

Derivation and validation of a risk-stratification model for patients with probable or proven COVID-19 in EDs: the revised HOME-CoV score

BACKGROUND

The HOME-CoV (Hospitalisation or Outpatient ManagEment of patients with SARS-CoV-2 infection) score is a validated list of uniquely clinical criteria indicating which patients with probable or proven COVID-19 can be treated at home. The aim of this study was to optimise the score to improve its ability to discriminate between patients who do and do not need admission.

METHODS

A revised HOME-CoV score was derived using data from a previous prospective multicentre study which evaluated the original Home-CoV score. Patients with proven or probable COVID-19 attending 34 EDs in France, Monaco and Belgium between April and May 2020 were included. The population was split into a derivation and validation sample corresponding to the observational and interventional phases of the original study. The main outcome was non-invasive or invasive ventilation or all-cause death within 7 days following inclusion. Two threshold values were defined using a sensitivity of >0.9 and a specificity of >0.9 to identify low-risk and high-risk patients, respectively. The revised HOME-CoV score was then validated by retrospectively applying it to patients in the same EDs with proven or probable COVID-19 during the interventional phase. The revised HOME-CoV score was also tested against original HOME-CoV, qCSI, qSOFA, CRB65 and SMART-COP in this validation cohort.

RESULTS

There were 1696 patients in the derivation cohort, of whom 65 (3.8%) required non-invasive ventilation or mechanical ventilation or died within 7 days and 1304 patients in the validation cohort, of whom 22 (1.7%) had a progression of illness. The revised score included seven clinical criteria. The area under the curve (AUC) was 87.6 (95% CI 84.7 to 90.6). The cut-offs to define low-risk and high-risk patients were <2 and >3, respectively. In the validation cohort, the AUC was 85.8 (95% CI 80.6 to 91.0). A score of <2 qualified 73% of patients as low risk with a sensitivity of 0.77 (0.55-0.92) and a negative predictive value of 0.99 (0.99-1.00).

CONCLUSION

The revised HOME-CoV score, which does not require laboratory testing, may allow accurate risk stratification and safely qualify a significant proportion of patients with probable or proven COVID-19 for home treatment.

External validation and application of the Diabetes Population Risk Tool (DPoRT) for prediction of type 2 diabetes onset in the US population

INTRODUCTION

Characterizing diabetes risk in the population is important for population health assessment and diabetes prevention planning. We aimed to externally validate an existing 10-year population risk model for type 2 diabetes in the USA and model the population benefit of diabetes prevention approaches using population survey data.

RESEARCH DESIGN AND METHODS

The Diabetes Population Risk Tool (DPoRT), originally derived and validated in Canada, was applied to an external validation cohort of 23 477 adults from the 2009 National Health Interview Survey (NHIS). We assessed predictive performance for discrimination (C-statistic) and calibration plots against observed incident diabetes cases identified from the NHIS 2009-2018 cycles. We applied DPoRT to the 2018 NHIS cohort (n=21 187) to generate 10-year risk prediction estimates and characterize the preventive benefit of three diabetes prevention scenarios: (1) community-wide strategy; (2) high-risk strategy and (3) combined approach.

RESULTS

DPoRT demonstrated good discrimination (C-statistic=0.778 (males); 0.787 (females)) and good calibration across the range of risk. We predicted a baseline risk of 10.2% and 21 076 000 new cases of diabetes in the USA from 2018 to 2028. The community-wide strategy and high-risk strategy estimated diabetes risk reductions of 0.2% and 0.3%, respectively. The combined approach estimated a 0.4% risk reduction and 843 000 diabetes cases averted in 10 years.

CONCLUSIONS

DPoRT has transportability for predicting population-level diabetes risk in the USA using routinely collected survey data. We demonstrate the model's applicability for population health assessment and diabetes prevention planning. Our modeling predicted that the combination of community-wide and targeted prevention approaches for those at highest risk are needed to reduce diabetes burden in the USA.

The sound of Parkinson's disease: A model of audible bradykinesia

INTRODUCTION

Evaluation of bradykinesia is based on five motor tasks from the MDS-UPDRS. Visually scoring these motor tasks is subjective, resulting in significant interrater variability. Recent observations suggest that it may be easier to hear the characteristic features of bradykinesia, such as the decrement in sound intensity or force of repetitive movements. The objective is to evaluate whether audio signals derived during four MDS-UPDRS tasks can be used to detect and grade bradykinesia, using two machine learning models.

METHODS

54 patients with Parkinson's disease and 28 healthy controls were filmed while executing the bradykinesia motor tasks. Several features were extracted from the audio signal, including number of taps, speed, sound intensity, decrement and freezes. For each motor task, two supervised machine learning models were trained, Logistic Regression (LR) and Support Vector Machine (SVM).

RESULTS

Both classifiers were able to separate patients from controls reasonably well for the leg agility task, area under the receiver operating characteristic curve (AUC): 0.92 (95%CI: 0.78-0.99) for LR and 0.93 (0.81-1.00) for SVM. Also, models were able to differentiate less severe bradykinesia from severe bradykinesia, particularly for the pronation-supination motor task, with AUC: 0.90 (0.62-1.00) for LR and 0.82 (0.45-0.97) for SVM.

CONCLUSION

This audio-based approach discriminates PD from healthy controls with moderate-high accuracy and separated individuals with less severe bradykinesia from those with severe bradykinesia. Sound analysis may contribute to the identification and monitoring of bradykinesia.

Assessment of Sepsis Risk at Admission to the Emergency Department: Clinical Interpretable Prediction Model

This study aims to develop an interpretable prediction model based on explainable artificial intelligence to predict bacterial sepsis and discover important biomarkers. A total of 1572 adult patients, 560 of whom were sepsis positive and 1012 of whom were negative, who were admitted to the emergency department with suspicion of sepsis, were examined. We investigated the performance characteristics of sepsis biomarkers alone and in combination for confirmed sepsis diagnosis using Sepsis-3 criteria. Three different tree-based algorithms-Extreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM), Adaptive Boosting (AdaBoost)-were used for sepsis prediction, and after examining comprehensive performance metrics, descriptions of the optimal model were obtained with the SHAP method. The XGBoost model achieved accuracy of 0.898 (0.868-0.929) and area under the ROC curve (AUC) of 0.940 (0.898-0.980) with a 95% confidence interval. The five biomarkers for predicting sepsis were age, respiratory rate, oxygen saturation, procalcitonin, and positive blood culture. SHAP results revealed that older age, higher respiratory rate, procalcitonin, neutrophil-lymphocyte count ratio, C-reactive protein, plaque, leukocyte particle concentration, as well as lower oxygen saturation, systolic blood pressure, and hemoglobin levels increased the risk of sepsis. As a result, the Explainable Artificial Intelligence (XAI)-based prediction model can guide clinicians in the early diagnosis and treatment of sepsis, providing more effective sepsis management and potentially reducing mortality rates and medical costs.

A comparative study of explainable ensemble learning and logistic regression for predicting in-hospital mortality in the emergency department

This study addresses the challenges associated with emergency department (ED) overcrowding and emphasizes the need for efficient risk stratification tools to identify high-risk patients for early intervention. While several scoring systems, often based on logistic regression (LR) models, have been proposed to indicate patient illness severity, this study aims to compare the predictive performance of ensemble learning (EL) models with LR for in-hospital mortality in the ED. A cross-sectional single-center study was conducted at the ED of Imam Reza Hospital in northeast Iran from March 2016 to March 2017. The study included adult patients with one to three levels of emergency severity index. EL models using Bagging, AdaBoost, random forests (RF), Stacking and extreme gradient boosting (XGB) algorithms, along with an LR model, were constructed. The training and validation visits from the ED were randomly divided into 80% and 20%, respectively. After training the proposed models using tenfold cross-validation, their predictive performance was evaluated. Model performance was compared using the Brier score (BS), The area under the receiver operating characteristics curve (AUROC), The area and precision-recall curve (AUCPR), Hosmer-Lemeshow (H-L) goodness-of-fit test, precision, sensitivity, accuracy, F1-score, and Matthews correlation coefficient (MCC). The study included 2025 unique patients admitted to the hospital's ED, with a total percentage of hospital deaths at approximately 19%. In the training group and the validation group, 274 of 1476 (18.6%) and 152 of 728 (20.8%) patients died during hospitalization, respectively. According to the evaluation of the presented framework, EL models, particularly Bagging, predicted in-hospital mortality with the highest AUROC (0.839, CI (0.802-0.875)) and AUCPR = 0.64 comparable in terms of discrimination power with LR (AUROC (0.826, CI (0.787-0.864)) and AUCPR = 0.61). XGB achieved the highest precision (0.83), sensitivity (0.831), accuracy (0.842), F1-score (0.833), and the highest MCC (0.48). Additionally, the most accurate models in the unbalanced dataset belonged to RF with the lowest BS (0.128). Although all studied models overestimate mortality risk and have insufficient calibration (P > 0.05), stacking demonstrated relatively good agreement between predicted and actual mortality. EL models are not superior to LR in predicting in-hospital mortality in the ED. Both EL and LR models can be considered as screening tools to identify patients at risk of mortality.

Predicting postoperative outcomes in lumbar spinal fusion: development of a machine learning model

BACKGROUND CONTEXT

Degenerative lumbar spondylolisthesis (DLS) is a prevalent spinal disorder, often requiring surgical intervention. Accurately predicting surgical outcomes is crucial to guide clinical decision-making, but this is challenging due to the multifactorial nature of postoperative results. Traditional risk assessment tools have limitations, and with the advent of machine learning, there is potential to enhance the precision and comprehensiveness of preoperative evaluations.

PURPOSE

We aimed to develop a machine-learning algorithm to predict surgical outcomes in patients with degenerative lumbar spondylolisthesis (DLS) undergoing spinal fusion surgery, only using preoperative data.

STUDY DESIGN

Retrospective cross-sectional study.

PATIENT SAMPLE

Patients with DLS undergoing lumbar spinal fusion surgery.

OUTCOME MEASURES

This study aimed to predict the occurrence of lower back pain (LBP) ≥4 on the numeric analogue scale (NAS) 2 years after surgery. LBP was evaluated as the average pain patients experienced at rest in the week before questioning. NAS ranges from 0 to 10, 0 representing no pain and 10 representing the worst pain imaginable.

METHODS

We conducted a retrospective analysis of prospectively enrolled patients who underwent spinal fusion surgery for degenerative lumbar spondylolistheses at our institution in the United States between January 2016 and December 2018. The initial patient characteristics to be included in the training of the model were chosen by clinical expertise and through a literature review and included demographic characteristics, comorbidities, and radiologic features. The data was split into a training and validation datasets using a 60/40 split. Four different machine learning models were trained, including the modern XGBoost model, logistic regression, random-forest, and support vector machine (SVM). The models were evaluated according to the area under the curve (AUC) of the receiver operating characteristics (ROC) curve. An AUC of 0.7 to 0.8 was considered fair, 0.8 to 0.9 good, and ≥ 0.9 excellent. Additionally, a calibration plot and the Brier score were calculated for each model.

RESULTS

A total of 135 patients (66% female) were included. A total of 38 (28%) patients reported LBP ≥ 4 after 2 years, representing the positive class. The XGBoost model demonstrated the best performance in the validation set with an AUC of 0.81 (95% CI 0.67-0.95). The other machine learning models performed significantly worse: with an AUC of 0.52 (95% CI 0.37-0.68) for the SVM, 0.56 (95% CI 0.37-0.76) for the logistic regression and an AUC of 0.56 (95% CI 0.37-0.78) for the random forest. In the XGBoost model age, composition of the erector spinae, and severity of lumbar spinal stenosis as were identified as the most important features.

CONCLUSIONS

This study represents a novel approach to predicting surgical outcomes in spinal fusion patients. The XGBoost demonstrated a better performance compared with classical models and highlighted the potential contributions of age and paraspinal musculature atrophy as significant factors. These findings have important implications for enhancing patient care through the identification of high-risk individuals and modifiable risk factors. As the incorporation of machine learning algorithms into clinical decision-making continues to gain traction in research and clinical practice, our insights reinforce this trajectory by showcasing the potential of these techniques in forecasting surgical results.

Development and validation of machine learning-augmented algorithm for insulin sensitivity assessment in the community and primary care settings: a population-based study in China

Objective

Insulin plays a central role in the regulation of energy and glucose homeostasis, and insulin resistance (IR) is widely considered as the "common soil" of a cluster of cardiometabolic disorders. Assessment of insulin sensitivity is very important in preventing and treating IR-related disease. This study aims to develop and validate machine learning (ML)-augmented algorithms for insulin sensitivity assessment in the community and primary care settings.

Methods

We analyzed the data of 9358 participants over 40 years old who participated in the population-based cohort of the Hubei center of the REACTION study (Risk Evaluation of Cancers in Chinese Diabetic Individuals). Three non-ensemble algorithms and four ensemble algorithms were used to develop the models with 70 non-laboratory variables for the community and 87 (70 non-laboratory and 17 laboratory) variables for the primary care settings to screen the classifier of the state-of-the-art. The models with the best performance were further streamlined using top-ranked 5, 8, 10, 13, 15, and 20 features. Performances of these ML models were evaluated using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPR), and the Brier score. The Shapley additive explanation (SHAP) analysis was employed to evaluate the importance of features and interpret the models.

Results

The LightGBM models developed for the community (AUROC 0.794, AUPR 0.575, Brier score 0.145) and primary care settings (AUROC 0.867, AUPR 0.705, Brier score 0.119) achieved higher performance than the models constructed by the other six algorithms. The streamlined LightGBM models for the community (AUROC 0.791, AUPR 0.563, Brier score 0.146) and primary care settings (AUROC 0.863, AUPR 0.692, Brier score 0.124) using the 20 top-ranked variables also showed excellent performance. SHAP analysis indicated that the top-ranked features included fasting plasma glucose (FPG), waist circumference (WC), body mass index (BMI), triglycerides (TG), gender, waist-to-height ratio (WHtR), the number of daughters born, resting pulse rate (RPR), etc.

Conclusion

The ML models using the LightGBM algorithm are efficient to predict insulin sensitivity in the community and primary care settings accurately and might potentially become an efficient and practical tool for insulin sensitivity assessment in these settings.

Preoperative prediction of early recurrence in resectable pancreatic cancer integrating clinical, radiologic, and CT radiomics features

OBJECTIVES

To use clinical, radiographic, and CT radiomics features to develop and validate a preoperative prediction model for the early recurrence of pancreatic cancer.

METHODS

We retrospectively analyzed 190 patients (150 and 40 in the development and test cohort from different centers) with pancreatic cancer who underwent pancreatectomy between January 2018 and June 2021. Radiomics, clinical-radiologic (CR), and clinical-radiologic-radiomics (CRR) models were developed for the prediction of recurrence within 12 months after surgery. Performance was evaluated using the area under the curve (AUC), Brier score, sensitivity, and specificity.

RESULTS

Early recurrence occurred in 36.7% and 42.5% of the development and test cohorts, respectively (P = 0.62). The features for the CR model included carbohydrate antigen 19-9 > 500 U/mL (odds ratio [OR], 3.60; P = 0.01), abutment to the portal and/or superior mesenteric vein (OR, 2.54; P = 0.054), and adjacent organ invasion (OR, 2.91; P = 0.03). The CRR model demonstrated significantly higher AUCs than the radiomics model in the internal (0.77 vs. 0.73; P = 0.048) and external (0.83 vs. 0.69; P = 0.038) validations. Although we found no significant difference between AUCs of the CR and CRR models (0.83 vs. 0.76; P = 0.17), CRR models showed more balanced sensitivity and specificity (0.65 and 0.87) than CR model (0.41 and 0.91) in the test cohort.

CONCLUSIONS

The CRR model outperformed the radiomics and CR models in predicting the early recurrence of pancreatic cancer, providing valuable information for risk stratification and treatment guidance.

The effect of feature normalization methods in radiomics

OBJECTIVES

In radiomics, different feature normalization methods, such as z-Score or Min-Max, are currently utilized, but their specific impact on the model is unclear. We aimed to measure their effect on the predictive performance and the feature selection.

METHODS

We employed fifteen publicly available radiomics datasets to compare seven normalization methods. Using four feature selection and classifier methods, we used cross-validation to measure the area under the curve (AUC) of the resulting models, the agreement of selected features, and the model calibration. In addition, we assessed whether normalization before cross-validation introduces bias.

RESULTS

On average, the difference between the normalization methods was relatively small, with a gain of at most + 0.012 in AUC when comparing the z-Score (mean AUC: 0.707 ± 0.102) to no normalization (mean AUC: 0.719 ± 0.107). However, on some datasets, the difference reached + 0.051. The z-Score performed best, while the tanh transformation showed the worst performance and even decreased the overall predictive performance. While quantile transformation performed, on average, slightly worse than the z-Score, it outperformed all other methods on one out of three datasets. The agreement between the features selected by different normalization methods was only mild, reaching at most 62%. Applying the normalization before cross-validation did not introduce significant bias.

CONCLUSION

The choice of the feature normalization method influenced the predictive performance but depended strongly on the dataset. It strongly impacted the set of selected features.

CRITICAL RELEVANCE STATEMENT

Feature normalization plays a crucial role in the preprocessing and influences the predictive performance and the selected features, complicating feature interpretation.

KEY POINTS

• The impact of feature normalization methods on radiomic models was measured. • Normalization methods performed similarly on average, but differed more strongly on some datasets. • Different methods led to different sets of selected features, impeding feature interpretation. • Model calibration was not largely affected by the normalization method.

A human-interpretable machine learning pipeline based on ultrasound to support leiomyosarcoma diagnosis

The preoperative evaluation of myometrial tumors is essential to avoid delayed treatment and to establish the appropriate surgical approach. Specifically, the differential diagnosis of leiomyosarcoma (LMS) is particularly challenging due to the overlapping of clinical, laboratory and ultrasound features between fibroids and LMS. In this work, we present a human-interpretable machine learning (ML) pipeline to support the preoperative differential diagnosis of LMS from leiomyomas, based on both clinical data and gynecological ultrasound assessment of 68 patients (8 with LMS diagnosis). The pipeline provides the following novel contributions: (i) end-users have been involved both in the definition of the ML tasks and in the evaluation of the overall approach; (ii) clinical specialists get a full understanding of both the decision-making mechanisms of the ML algorithms and the impact of the features on each automatic decision. Moreover, the proposed pipeline addresses some of the problems concerning both the imbalance of the two classes by analyzing and selecting the best combination of the synthetic oversampling strategy of the minority class and the classification algorithm among different choices, and the explainability of the features at global and local levels. The results show very high performance of the best strategy (AUC = 0.99, F1 = 0.87) and the strong and stable impact of two ultrasound-based features (i.e., tumor borders and consistency of the lesions). Furthermore, the SHAP algorithm was exploited to quantify the impact of the features at the local level and a specific module was developed to provide a template-based natural language (NL) translation of the explanations for enhancing their interpretability and fostering the use of ML in the clinical setting.

Can machine learning models predict prolonged length of hospital stay following primary total knee arthroplasty based on a national patient cohort data?

INTRODUCTION

The total length of stay (LOS) is one of the biggest determinators of overall care costs associated with total knee arthroplasty (TKA). An accurate prediction of LOS could aid in optimizing discharge strategy for patients in need and diminishing healthcare expenditure. The aim of this study was to predict LOS following TKA using machine learning models developed on a national-scale patient cohort.

METHODS

The ACS-NSQIP database was queried to acquire 267,966 TKA cases from 2013 to 2020. Four machine learning models-artificial neural network (ANN), random forest, histogram-based gradient boosting, and k-nearest neighbor were trained and tested on the dataset for the prediction of prolonged LOS (LOS exceeded the 75th of all values in the cohort). The model performance was assessed by discrimination (area under the receiver operating characteristic curve [AUC]), calibration, and clinical utility.

RESULTS

ANN delivered the best performance among the four models. ANN distinguished prolonged LOS in the study cohort with an AUC of 0.71 and accurately predicted the probability of prolonged LOS for individual patients (calibration slope: 0.82; calibration intercept: 0.03; Brier score: 0.089). All models demonstrated clinical utility by generating positive net benefits in decision curve analyses. Operation time, pre-operative transfusion, pre-operative laboratory tests (hematocrit, platelet count, and white blood cell count), and BMI were the strongest predictors of prolonged LOS.

CONCLUSION

ANN demonstrated modest discrimination capacity and excellent performance in calibration and clinical utility for the prediction of prolonged LOS following TKA. Clinical application of the machine learning models has the potential to improve care coordination and discharge planning for patients at high risk of extended hospitalization after surgery. Incorporating more relevant patient factors may further increase the models' prediction strength.

Prediction of Hearing Prognosis After Intact Canal Wall Mastoidectomy With Tympanoplasty Using Artificial Intelligence

OBJECTIVE

To evaluate the performance of a machine learning model and the effects of major prognostic factors on hearing outcomes following intact canal wall (ICW) mastoidectomy with tympanoplasty.

STUDY DESIGN

Retrospective cross-sectional study.

SETTING

Tertiary hospital.

METHODS

A total of 484 patients with chronic otitis media who underwent ICW tympanomastoidectomy between January 2007 and December 2020 were included in this study. Successful hearing outcomes were defined by a postoperative air-bone gap (ABG) of ≤20 dB and preoperative air conduction (AC)-postoperative AC value of ≥15 dB according to the Korean Otological Society guidelines for outcome reporting after chronic otitis media surgery. The light gradient boosting machine (LightGBM) and multilayer perceptron (MLP) models were tested as artificial intelligence models and compared using logistic regression. The main outcome assessed was the successful hearing outcome after surgery, measured using the area under the receiver operating characteristic curve (AUROC).

RESULTS

In the analysis using the postoperative ABG criterion, the LightGBM exhibited a significantly higher AUROC compared to those of the baseline model (mean, 0.811). According to the difference between preoperative and postoperative AC, the MLP showed a significantly higher AUROC than those of the baseline model (mean, 0.795).

CONCLUSION

This study analyzed multiple factors that could affect the hearing outcome using different artificial intelligence models and found that preoperative hearing status was the most important factor. Our findings provide additional information regarding postoperative hearing for clinicians.

Predicting functional dependency using machine learning among a middle-aged and older Chinese population

OBJECTIVE

To develop prediction models for assessing functional dependency in a middle-aged and older Chinese population.

METHOD

Adults ≥45 years old from the China Health and Retirement Longitudinal Study (CHARLS) and without functional dependency at baseline were included. Functional dependency was defined as needing any help in any basic activities of daily living (ADL) or instrumental activities of daily living (IADL). The outcomes were overall functional dependency, ADL and IADL dependency. Stacked ensemble models were constructed based on five selected machine learning models. Models were trained and tested in the 2011-2015 cohort, and were externally validated in the 2015-2018 cohort. SHapley Additive exPlanations (SHAP) was utilized to quantify the significance of predictors.

RESULT

In the training cohort, a total of 6,297 participants were included at baseline, 1,893 developed functional dependency during the follow-up period. The stacked ensemble model achieved the best performance in terms of discrimination ability for predicting overall functional dependency, ADL and IADL dependency, with AUCs of 0.750, 0.690 and 0.748, respectively; in external validation cohort, the corresponding AUCs were 0.725, 0.719 and 0.727, respectively. A compact model was further developed and maintained similar predictive performance.

CONCLUSION

The stacked ensemble approach can serve as a useful tool for identifying the risk of functional dependency in a large Chinese population. For ADL dependency, arthritis, age, self-report health, and waist circumference were identified as highly significant predictors. Conversely, cognitive function, age, living in rural areas, and performance in chair stand test emerged as highly ranked predictors for IADL dependency.

Validation and Comparison of Four Mortality Prediction Models in a Geriatric Ward in China

Purpose

The efficacy of mortality risk prediction models among older patients in China remains uncertain. We aimed to validate and compare the performances of the Walter Index, Geriatric Prognostic Index (GPI), Charlson Comorbidity Index (CCI), and FRAIL Scale in predicting 1-year all-cause mortality post-discharge in geriatric inpatients in China.

Patients and Methods

This study was conducted at a geriatric ward of a tertiary Hospital in Beijing, including patients aged 70 years or older with a documented comprehensive geriatric assessment, discharged between January 1, 2016, and December 31, 2021. Patients with a hospital stay ≤24 h or >60 days were excluded. All-cause mortality data within one year of discharge were collected from medical files and telephone interviews between August 2022 and February 2023. Multiple imputation, Logistic regression analysis, Brier scores, C-statistics, Hosmer-Lemeshow goodness-of-fit-test, and calibration plots were employed for statistical analysis.

Results

We included 832 patients with a median (interquartile range) age of 77 (74-82) years. One-hundred patients (12.0%) died within one year. After adjusting for covariates-marital status, social support, cigarette use, length of stay, number of medications, hemoglobin levels, handgrip strength, and Short Physical Performance Battery-CCI scores of 3-4 and >4, and increased Walter Index, GPI, and FRAIL Scale scores were significantly associated with 1-year mortality risk. The Brier scores varied from 0.07 (Walter Index) to 0.10 (FRAIL Scale). The C-statistic ranged from 0.74 (95% confidence interval, 0.69-0.78) for FRAIL Scale to 0.88 (95% confidence interval, 0.84-0.91) for the Walter Index. Calibration curves showed that the Walter Index, GPI, and FRAIL Scale were well calibrated, while the CCI was poor.

Conclusion

Combining the Brier score, discrimination and calibration, the Walter Index was confirmed for the first time to be the best model to predict the 1-year mortality risk of geriatric inpatients in China among the four models.

Assessment of the ABC2-SPH risk score to predict invasive mechanical ventilation in COVID-19 patients and comparison to other scores

Background

Predicting the need for invasive mechanical ventilation (IMV) is important for the allocation of human and technological resources, improvement of surveillance, and use of effective therapeutic measures. This study aimed (i) to assess whether the ABC2-SPH score is able to predict the receipt of IMV in COVID-19 patients; (ii) to compare its performance with other existing scores; (iii) to perform score recalibration, and to assess whether recalibration improved prediction.

Methods

Retrospective observational cohort, which included adult laboratory-confirmed COVID-19 patients admitted in 32 hospitals, from 14 Brazilian cities. This study was conducted in two stages: (i) for the assessment of the ABC2-SPH score and comparison with other available scores, patients hospitalized from July 31, 2020, to March 31, 2022, were included; (ii) for ABC2-SPH score recalibration and also comparison with other existing scores, patients admitted from January 1, 2021, to March 31, 2022, were enrolled. For both steps, the area under the receiving operator characteristic score (AUROC) was calculated for all scores, while a calibration plot was assessed only for the ABC2-SPH score. Comparisons between ABC2-SPH and the other scores followed the Delong Test recommendations. Logistic recalibration methods were used to improve results and adapt to the studied sample.

Results

Overall, 9,350 patients were included in the study, the median age was 58.5 (IQR 47.0-69.0) years old, and 45.4% were women. Of those, 33.5% were admitted to the ICU, 25.2% received IMV, and 17.8% died. The ABC2-SPH score showed a significantly greater discriminatory capacity, than the CURB-65, STSS, and SUM scores, with potentialized results when we consider only patients younger than 80 years old (AUROC 0.714 [95% CI 0.698-0.731]). Thus, after the ABC2-SPH score recalibration, we observed improvements in calibration (slope = 1.135, intercept = 0.242) and overall performance (Brier score = 0.127).

Conclusion

The ABC2-SPHr risk score demonstrated a good performance to predict the need for mechanical ventilation in COVID-19 hospitalized patients under 80 years of age.

Predicting short-term outcomes in atrial-fibrillation-related stroke using machine learning

Background

Prognostic prediction and the identification of prognostic factors are critical during the early period of atrial-fibrillation (AF)-related strokes as AF is associated with poor outcomes in stroke patients.

Methods

Two independent datasets, namely, the Korean Atrial Fibrillation Evaluation Registry in Ischemic Stroke Patients (K-ATTENTION) and the Korea University Stroke Registry (KUSR), were used for internal and external validation, respectively. These datasets include common variables such as demographic, laboratory, and imaging findings during early hospitalization. Outcomes were unfavorable functional status with modified Rankin scores of 3 or higher and mortality at 3 months. We developed two machine learning models, namely, a tree-based model and a multi-layer perceptron (MLP), along with a baseline logistic regression model. The area under the receiver operating characteristic curve (AUROC) was used as the outcome metric. The Shapley additive explanation (SHAP) method was used to evaluate the contributions of variables.

Results

Machine learning models outperformed logistic regression in predicting both outcomes. For 3-month unfavorable outcomes, MLP exhibited significantly higher AUROC values of 0.890 and 0.859 in internal and external validation sets, respectively, than those of logistic regression. For 3-month mortality, both machine learning models exhibited significantly higher AUROC values than the logistic regression for internal validation but not for external validation. The most significant predictor for both outcomes was the initial National Institute of Health and Stroke Scale.

Conclusion

The explainable machine learning model can reliably predict short-term outcomes and identify high-risk patients with AF-related strokes.

Clinical Usefulness of Echocardiographic Measurement of Proximal Aortic Diameter in Early Differentiation Between Type A Acute Aortic Dissection and ST-Segment-Elevation Myocardial Infarction

Background Contradictions between management modalities of type A acute aortic dissection (TAAAD) and ST-elevation-myocardial infarction (STEMI) may result in clinical catastrophe. Therefore, we aimed to explore which 2-dimensional echocardiography (2DE) findings are optimal for differentiating TAAAD from STEMI. Methods and Results This study included 340 patients with STEMI and 340 patients with TAAAD who underwent 2DE in the emergency department between 2012 and 2021. The proximal ascending aorta (PAA) diameter and other echocardiographic parameters were analyzed. PAA diameters were measured at 4 levels in the parasternal view: Valsalva, the sinotubular junction (STJ), the PAA at 1 cm above the STJ, and the PAA at 2 cm above the STJ. Receiver-operating characteristic curve analysis showed that Valsalva, STJ, PAA at 1 cm above the STJ, and PAA at 2 cm above the STJ were significant predictors of TAAAD (areas under the curve: 0.777, 0.924, 0.965, and 0.975, respectively; P<0.001) with the respective cutoff values of 39.4, 38.5, 39.8, and 41.2 mm. Multivariable analysis suggested that all 2DE parameters were significant predictors of TAAAD. Among the 2DE parameters examined, the incorporation of PAA at 2 cm above the STJ to clinical indicators exhibited the most significant diagnostic capability (C-statistics, 0.97; net reclassification improvement, 1.81; integrated discrimination improvement, 0.61). When only TAAAD with coronary malperfusion and STEMI were analyzed, the diagnostic utility of PAA at 1 cm above the STJ was evident (C-statistics, 0.99; net reclassification improvement, 1.79; integrated discrimination improvement, 0.67), with PAA at 2 cm above the STJ ranking second in diagnostic significance (C-statistics, 0.99; net reclassification improvement, 1.12; integrated discrimination improvement, 0.66). Conclusions PAA measurements were the most beneficial for diagnosing TAAAD in all 2DE findings and TAAAD from STEMI.

Personalized Antiviral Drug Selection in Patients With Chronic Hepatitis B Using a Machine Learning Model: A Multinational Study

INTRODUCTION

Tenofovir disoproxil fumarate (TDF) is reportedly superior or at least comparable to entecavir (ETV) for the prevention of hepatocellular carcinoma (HCC) in patients with chronic hepatitis B; however, it has distinct long-term renal and bone toxicities. This study aimed to develop and validate a machine learning model (designated as Prediction of Liver cancer using Artificial intelligence-driven model for Network-antiviral Selection for hepatitis B [PLAN-S]) to predict an individualized risk of HCC during ETV or TDF therapy.

METHODS

This multinational study included 13,970 patients with chronic hepatitis B. The derivation (n = 6,790), Korean validation (n = 4,543), and Hong Kong-Taiwan validation cohorts (n = 2,637) were established. Patients were classified as the TDF-superior group when a PLAN-S-predicted HCC risk under ETV treatment is greater than under TDF treatment, and the others were defined as the TDF-nonsuperior group.

RESULTS

The PLAN-S model was derived using 8 variables and generated a c-index between 0.67 and 0.78 for each cohort. The TDF-superior group included a higher proportion of male patients and patients with cirrhosis than the TDF-nonsuperior group. In the derivation, Korean validation, and Hong Kong-Taiwan validation cohorts, 65.3%, 63.5%, and 76.4% of patients were classified as the TDF-superior group, respectively. In the TDF-superior group of each cohort, TDF was associated with a significantly lower risk of HCC than ETV (hazard ratio = 0.60-0.73, all P < 0.05). In the TDF-nonsuperior group, however, there was no significant difference between the 2 drugs (hazard ratio = 1.16-1.29, all P > 0.1).

DISCUSSION

Considering the individual HCC risk predicted by PLAN-S and the potential TDF-related toxicities, TDF and ETV treatment may be recommended for the TDF-superior and TDF-nonsuperior groups, respectively.

Low dose tacrolimus exposure and early steroid withdrawal with strict body weight control can improve post kidney transplant glucose tolerance in Japanese patients

The development of diabetes mellitus (DM) after living donor kidney transplantation (KT) is a risk factor for worsening transplant kidney function, cardiac disease, and cerebrovascular disease, which may affect prognosis after KT. At our institution, all patients' glucose tolerance is evaluated perioperatively by oral glucose tolerance tests (OGTTs) at pre-KT, and 3, 6, and 12 month (mo.) after KT. We analyzed the insulinogenic index (ISI) and homeostasis model assessment beta cell (HOMA-β) based on the immunoreactive insulin (IRI) levels to determine how glucose tolerance changed after KT in 214 patients who had not been diagnosed with DM before KT. In addition, we analyzed the body mass index (BMI) which may also influence glucose tolerance after KT. The concentration of tacrolimus (TAC) in blood was also measured as the area under the curve (AUC) to examine its effects at each sampling point. The preoperative-OGTTs showed that DM was newly diagnosed in 22 of 214 patients (10.3%) who had not been given a diagnosis of DM by the pre-KT fasting blood sugar (FBS) tests. The glucose tolerance was improved in 15 of 22 DM patients at 12 mo. after KT. ISI and IRI deteriorated only at 3 mo. after KT but improved over time. There was a trend of an inverse correlation between HOMA-β and TAC-AUC. We also found inverse correlations between IRI and an increase in BMI from 3 to 12 mo. after KT. Early corticosteroid withdrawal or the steroid minimization protocol with tacrolimus to maintain a low level of diabetogenic tacrolimus and BMI decrease after KT used by our hospital individualizes lifestyle interventions for each patient might contribute to an improvement in post-KT glucose tolerance.

Combining clinical and imaging data for predicting functional outcomes after acute ischemic stroke: an automated machine learning approach

This study aimed to develop and validate an automated machine learning (ML) system that predicts 3-month functional outcomes in acute ischemic stroke (AIS) patients by combining clinical and neuroimaging features. Functional outcomes were categorized as unfavorable (modified Rankin Scale ≥ 3) or not. A clinical model employing optimal clinical features (Model_A), a convolutional neural network model incorporating imaging data (Model_B), and an integrated model combining both imaging and clinical features (Model_C) were developed and tested to predict unfavorable outcomes. The developed models were compared with each other and with traditional risk-scoring models. The dataset comprised 4147 patients from a multicenter stroke registry, with 1268 (30.6%) experiencing unfavorable outcomes. Age, initial NIHSS, and early neurologic deterioration were identified as the most important clinical features. The ML model prediction achieved an area under the curves of 0.757 (95% CI 0.726-0.789) for Model_A, 0.725 (95% CI 0.693-0.755) for Model_B, and 0.786 (95% CI 0.757-0.814) for Model_C in the test set. The integrated models outperformed traditional risk-scoring models by 0.21 (95% CI 0.16-0.25) for HIAT and 0.15 (95% CI 0.11-0.19) for THRIVE. In conclusion, the integrated ML system enhanced stroke outcome prediction by combining imaging data and clinical features, outperforming traditional risk-scoring models.

Temporal validation of the MMCD score to predict kidney replacement therapy and in-hospital mortality in COVID-19 patients

BACKGROUND

Acute kidney injury has been described as a common complication in patients hospitalized with COVID-19, which may lead to the need for kidney replacement therapy (KRT) in its most severe forms. Our group developed and validated the MMCD score in Brazilian COVID-19 patients to predict KRT, which showed excellent performance using data from 2020. This study aimed to validate the MMCD score in a large cohort of patients hospitalized with COVID-19 in a different pandemic phase and assess its performance to predict in-hospital mortality.

METHODS

This study is part of the "Brazilian COVID-19 Registry", a retrospective observational cohort of consecutive patients hospitalized for laboratory-confirmed COVID-19 in 25 Brazilian hospitals between March 2021 and August 2022. The primary outcome was KRT during hospitalization and the secondary was in-hospital mortality. We also searched literature for other prediction models for KRT, to assess the results in our database. Performance was assessed using area under the receiving operator characteristic curve (AUROC) and the Brier score.

RESULTS

A total of 9422 patients were included, 53.8% were men, with a median age of 59 (IQR 48-70) years old. The incidence of KRT was 8.8% and in-hospital mortality was 18.1%. The MMCD score had excellent discrimination and overall performance to predict KRT (AUROC: 0.916 [95% CI 0.909-0.924]; Brier score = 0.057). Despite the excellent discrimination and overall performance (AUROC: 0.922 [95% CI 0.914-0.929]; Brier score = 0.100), the calibration was not satisfactory concerning in-hospital mortality. A random forest model was applied in the database, with inferior performance to predict KRT requirement (AUROC: 0.71 [95% CI 0.69-0.73]).

CONCLUSION

The MMCD score is not appropriate for in-hospital mortality but demonstrates an excellent predictive ability to predict KRT in COVID-19 patients. The instrument is low cost, objective, fast and accurate, and can contribute to supporting clinical decisions in the efficient allocation of assistance resources in patients with COVID-19.

Health and Socioeconomic Risk Factors for Unplanned Hospitalization Following Ambulatory Unicompartmental Knee Arthroplasty: Development of a Patient Selection Tool Using Machine Learning

BACKGROUND

Identifying ambulatory surgical candidates at risk for adverse surgical outcomes can optimize outcomes. The purpose of this study was to develop and internally validate a machine learning (ML) algorithm to predict contributors to unexpected hospitalizations after ambulatory unicompartmental knee arthroplasty (UKA).

METHODS

A total of 2,521 patients undergoing UKA from 2006 to 2018 were retrospectively evaluated. Patients admitted overnight postoperatively were identified as those who had a length of stay ≥ 1 day were analyzed with four individual ML models (ie, random forest, extreme gradient boosting, adaptive boosting, and elastic net penalized logistic regression). An additional model was produced as a weighted ensemble of the four individual algorithms. Area under the receiver operating characteristics (AUROC) compared predictive capacity of these models to conventional logistic regression techniques.

RESULTS

Of the 2,521 patients identified, 103 (4.1%) required at least one overnight stay following ambulatory UKA. The ML ensemble model achieved the best performance based on discrimination assessed via internal validation (AUROC = 87.3), outperforming individual models and conventional logistic regression (AUROC = 81.9-85.7). The variables determined most important by the ensemble model were cumulative time in the operating room, utilization of general anesthesia, increasing age, and patient residency in more urban areas. The model was integrated into a web-based open-access application.

CONCLUSION

The ensemble gradient-boosted ML algorithm demonstrated the highest performance in identifying factors contributing to unexpected hospitalizations in patients receiving UKA. This tool allows physicians and healthcare systems to identify patients at a higher risk of needing inpatient care after UKA.

Evaluation of the ACS-NSQIP Surgical Risk Calculator in Patients with Hepatic Metastases from Colorectal Cancer Undergoing Liver Resection

BACKGROUND

The American College of Surgeons National Surgical Quality Improvement Program surgical risk calculator (ACS-NSQIP SRC) has been designed to predict morbidity and mortality and help stratify surgical patients. This study evaluates the performance of the SRC for patients undergoing surgery for colorectal liver metastases (CRLM).

METHODS

SRC was retrospectively computed for patients undergoing liver or simultaneous colon and liver surgery for colorectal cancer (CRC) in two high tertiary referral centres from 2011 to 2020. C-statistics and Brier score were calculated as a mean of discrimination and calibration respectively, for both group and for every level of surgeon adjustment score (SAS) for liver resections in case of simultaneous liver-colon surgery. An AUC ≥ 0.7 shows acceptable discrimination; a Brier score next to 0 means the prediction tool has good calibration.

RESULTS

Four hundred ten patients were included, 153 underwent simultaneous resection, and 257 underwent liver-only resections. For simultaneous surgery, the ACS-NSQIP SRC showed good calibration and discrimination only for cardiac complication (AUC = 0.720, 0.740, and 0.702 for liver resection unadjusted, SAS-2, and SAS-3 respectively; 0.714 for colon resection; and Brier score = 0.04 in every case). For liver-only surgery, it only showed good calibration for cardiac complications (Brier score = 0.03). The SRC underestimated the incidence of overall complications, pneumonia, cardiac complications, and the length of hospital stay.

CONCLUSIONS

ACS-NSQIP SRC showed good predicting capabilities only for 1 out of 5 evaluated outcomes; therefore, it is not a reliable tool for patients undergoing liver surgery for CRLM, both in the simultaneous and staged resections.

Predicting Survival for Veno-Arterial ECMO Using Conditional Inference Trees-A Multicenter Study

BACKGROUND

Despite increasing use and understanding of the process, veno-arterial extracorporeal membrane oxygenation (VA-ECMO) therapy is still associated with considerable mortality. Personalized and quick survival predictions using machine learning methods can assist in clinical decision making before ECMO insertion.

METHODS

This is a multicenter study to develop and validate an easy-to-use prognostic model to predict in-hospital mortality of VA-ECMO therapy, using unbiased recursive partitioning with conditional inference trees. We compared two sets with different numbers of variables (small and comprehensive), all of which were available just before ECMO initiation. The area under the curve (AUC), the cross-validated Brier score, and the error rate were applied to assess model performance. Data were collected retrospectively between 2007 and 2019.

RESULTS

837 patients were eligible for this study; 679 patients in the derivation cohort (median (IQR) age 60 (49 to 69) years; 187 (28%) female patients) and a total of 158 patients in two external validation cohorts (median (IQR) age 57 (49 to 65) and 70 (63 to 76) years). For the small data set, the model showed a cross-validated error rate of 35.79% and an AUC of 0.70 (95% confidence interval from 0.66 to 0.74). In the comprehensive data set, the error rate was the same with a value of 35.35%, with an AUC of 0.71 (95% confidence interval from 0.67 to 0.75). The mean Brier scores of the two models were 0.210 (small data set) and 0.211 (comprehensive data set). External validation showed an error rate of 43% and AUC of 0.60 (95% confidence interval from 0.52 to 0.69) using the small tree and an error rate of 35% with an AUC of 0.63 (95% confidence interval from 0.54 to 0.72) using the comprehensive tree. There were large differences between the two validation sets.

CONCLUSIONS

Conditional inference trees are able to augment prognostic clinical decision making for patients undergoing ECMO treatment. They may provide a degree of accuracy in mortality prediction and prognostic stratification using readily available variables.

Prediction of Maxillary Bone Invasion in Hard Palate/Upper Alveolus Cancer: A Multi-Center Retrospective Study

BACKGROUND

maxillary bone invasion (MBI) is not uncommon in hard palate or upper alveolus (HP/UA) cancer; however, there have been relatively few reports about the MBI of HP/UA cancer.

PATIENTS AND METHODS

this was a multi-center retrospective study, enrolling 144 cases of HP/UA cancer. MBI was defined by surgical pathology or radiology follow-up. The multiple prediction models for MBI were developed in total cases and in cases having primary bone resection, using clinical and radiological variables.

RESULTS

computerized tomography (CT) alone predicted MBI, with an area under receiver operating curve (AUC) of 0.779 (95% confidence interval (CI) = 0.712-0.847). The AUC was increased in a model that combined tumor dimensions and clinical factors (male sex and nodal metastasis) (0.854 (95%CI = 0.790-0.918)). In patients who underwent 18fluorodeoxyglucose positron emission tomography/CT (PET/CT), the discrimination performance of a model including the maximal standardized uptake value (SUVmax) had an AUC of 0.911 (95%CI = 0.847-0.975). The scoring system using CT finding, tumor dimension, and clinical factors, with/without PET/CT SUVmax clearly distinguished low-, intermediate-, and high-risk groups for MBI.

CONCLUSION

using information from CT, tumor dimension, clinical factors, and the SUVmax value, the MBI of HP/UA cancer can be predicted with a relatively high discrimination performance.

An Artificial Intelligence Approach to Predicting Unplanned Intubation Following Anterior Cervical Discectomy and Fusion

STUDY DESIGN

Level III retrospective database study.

OBJECTIVES

The purpose of this study is to determine if machine learning algorithms are effective in predicting unplanned intubation following anterior cervical discectomy and fusion (ACDF).

METHODS

The National Surgical Quality Initiative Program (NSQIP) was queried to select patients who had undergone ACDF. Machine learning analysis was conducted in Python and multivariate regression analysis was conducted in R. C-Statistics area under the curve (AUC) and prediction accuracy were used to measure the classifier's effectiveness in distinguishing cases.

RESULTS

In total, 54 502 patients met the study criteria. Of these patients, .51% underwent an unplanned re-intubation. Machine learning algorithms accurately classified between 72%-100% of the test cases with AUC values of between .52-.77. Multivariable regression indicated that the number of levels fused, male sex, COPD, American Society of Anesthesiologists (ASA) > 2, increased operating time, Age > 65, pre-operative weight loss, dialysis, and disseminated cancer were associated with increased risk of unplanned intubation.

CONCLUSIONS

The models presented here achieved high accuracy in predicting risk factors for re-intubation following ACDF surgery. Machine learning analysis may be useful in identifying patients who are at a higher risk of unplanned post-operative re-intubation and their treatment plans can be modified to prophylactically prevent respiratory compromise and consequently unplanned re-intubation.

Developing a Machine Learning Risk-adjustment Method for Hospitalizations and Emergency Department Visits of Nursing Home Residents With Dementia

BACKGROUND

Long-stay nursing home (NH) residents with Alzheimer disease and related dementias (ADRD) are at high risk of hospital transfers. Machine learning might improve risk-adjustment methods for NHs.

OBJECTIVES

The objective of this study was to develop and compare NH risk-adjusted rates of hospitalizations and emergency department (ED) visits among long-stay residents with ADRD using Extreme Gradient Boosting (XGBoost) and logistic regression.

RESEARCH DESIGN

Secondary analysis of national Medicare claims and NH assessment data in 2012 Q3. Data were equally split into the training and test sets. Both XGBoost and logistic regression predicted any hospitalization and ED visit using 58 predictors. NH-level risk-adjusted rates from XGBoost and logistic regression were constructed and compared. Multivariate regressions examined NH and market factors associated with rates of hospitalization and ED visits.

SUBJECTS

Long-stay Medicare residents with ADRD (N=413,557) from 14,057 NHs.

RESULTS

A total of 8.1% and 8.9% residents experienced any hospitalization and ED visit in a quarter, respectively. XGBoost slightly outperformed logistic regression in area under the curve (0.88 vs. 0.86 for hospitalization; 0.85 vs. 0.83 for ED visit). NH-level risk-adjusted rates from XGBoost were slightly lower than logistic regression (hospitalization=8.3% and 8.4%; ED=8.9% and 9.0%, respectively), but were highly correlated. Facility and market factors associated with the XGBoost and logistic regression-adjusted hospitalization and ED rates were similar. NHs serving more residents with ADRD and having a higher registered nurse-to-total nursing staff ratio had lower rates.

CONCLUSIONS

XGBoost and logistic regression provide comparable estimates of risk-adjusted hospitalization and ED rates.

An open competition involving thousands of competitors failed to construct useful abstract classifiers for new diagnostic test accuracy systematic reviews

There are currently no abstract classifiers, which can be used for new diagnostic test accuracy (DTA) systematic reviews to select primary DTA study abstracts from database searches. Our goal was to develop machine-learning-based abstract classifiers for new DTA systematic reviews through an open competition. We prepared a dataset of abstracts obtained through database searches from 11 reviews in different clinical areas. As the reference standard, we used the abstract lists that required manual full-text review. We randomly splitted the datasets into a train set, a public test set, and a private test set. Competition participants used the training set to develop classifiers and validated their classifiers using the public test set. The classifiers were refined based on the performance of the public test set. They could submit as many times as they wanted during the competition. Finally, we used the private test set to rank the submitted classifiers. To reduce false exclusions, we used the Fbeta measure with a beta set to seven for evaluating classifiers. After the competition, we conducted the external validation using a dataset from a cardiology DTA review. We received 13,774 submissions from 1429 teams or persons over 4 months. The top-honored classifier achieved a Fbeta score of 0.4036 and a recall of 0.2352 in the external validation. In conclusion, we were unable to develop an abstract classifier with sufficient recall for immediate application to new DTA systematic reviews. Further studies are needed to update and validate classifiers with datasets from other clinical areas.

Prehospital prediction of hospital admission for emergent acuity patients transported by paramedics: A population-based cohort study using machine learning

INTRODUCTION

The closest emergency department (ED) may not always be the optimal hospital for certain stable high acuity patients if further distanced ED's can provide specialized care or are less overcrowded. Machine learning (ML) predictions may support paramedic decision-making to transport a subgroup of emergent patients to a more suitable, albeit more distanced, ED if hospital admission is unlikely. We examined whether characteristics known to paramedics in the prehospital setting were predictive of hospital admission in emergent acuity patients.

MATERIALS AND METHODS

We conducted a population-level cohort study using four ML algorithms to analyze ED visits of the National Ambulatory Care Reporting System from January 1, 2018 to December 31, 2019 in Ontario, Canada. We included all adult patients (≥18 years) transported to the ED by paramedics with an emergent Canadian Triage Acuity Scale score. We included eight characteristic classes as model predictors that are recorded at ED triage. All ML algorithms were trained and assessed using 10-fold cross-validation to predict hospital admission from the ED. Predictive model performance was determined using the area under curve (AUC) with 95% confidence intervals and probabilistic accuracy using the Brier Scaled score. Variable importance scores were computed to determine the top 10 predictors of hospital admission.

RESULTS

All machine learning algorithms demonstrated acceptable accuracy in predicting hospital admission (AUC 0.77-0.78, Brier Scaled 0.22-0.24). The characteristics most predictive of admission were age between 65 to 105 years, referral source from a residential care facility, presenting with a respiratory complaint, and receiving home care.

DISCUSSION

Hospital admission was accurately predicted based on patient characteristics known prehospital to paramedics prior to arrival. Our results support consideration of policy modification to permit certain emergent acuity patients to be transported to a further distanced ED. Additionally, this study demonstrates the utility of ML in paramedic and prehospital research.

Prediction of nonsentinel lymph node metastasis in breast cancer patients based on machine learning

BACKGROUND

Develop the best machine learning (ML) model to predict nonsentinel lymph node metastases (NSLNM) in breast cancer patients.

METHODS

From June 2016 to August 2022, 1005 breast cancer patients were included in this retrospective study. Univariate and multivariate analyses were performed using logistic regression. Six ML models were introduced, and their performance was compared.

RESULTS

NSLNM occurred in 338 (33.6%) of 1005 patients. The best ML model was XGBoost, whose average area under the curve (AUC) based on 10-fold cross-verification was 0.722. It performed better than the nomogram, which was based on logistic regression (AUC: 0.764 vs. 0.706).

CONCLUSIONS

The ML model XGBoost can well predict NSLNM in breast cancer patients.

Normal tissue complication probability of acute eyelids erythema following radiotherapy of head and neck cancers and skull-base tumors

PURPOSE

Radiation therapy is broadly used as one of the main treatment methods for patients with head and neck cancers and skull base tumors. However, it can lead to normal tissue complications. Therefore, this study aimed to model normal tissue complication probability (NTCP) of eyelid skin erythema after radiation therapy.

METHODS

The dataset of 45 patients with head and neck and skull base tumors was prospectively collected from their dose-volume histograms (DVHs). Grade 1 + eyelid skin erythema based on the Common Terminology Criteria for Adverse Events (CTCAE 4.0) was evaluated as the endpoint after a three-month follow-up. The Lyman-Kutcher-Burman (LKB) radiobiological model was developed based on generalized equivalent uniform dose (gEUD). Model parameters were calculated by maximum likelihood estimation. Model performance was evaluated by ROC-AUC, Brier score and Hosmer-Lemeshow test.

RESULTS

After three months of follow-up, 13.33% of patients experienced eyelids skin erythema grade 1 or more. The parameters of the LKB model were: TD50 = 30 Gy, m = 0.14, and n = 0.10. The model showed good predictive performance with ROC-AUC = 0.80 (CI:0.66-0.94) and a Brier score of 0.20.

CONCLUSIONS

In this study, NTCP of eyelid skin erythema was modeled based on the LKB radiobiological model with good predictive performance.