Development and internal validation of a machine-learning-developed model for predicting 1-year mortality after fragility hip fracture

Establishment and validation of an artificial intelligence web application for predicting postoperative in-hospital mortality in patients with hip fracture: a national cohort study of 52 707 cases

BACKGROUND

In-hospital mortality following hip fractures is a significant concern, and accurate prediction of this outcome is crucial for appropriate clinical management. Nonetheless, there is a lack of effective prediction tools in clinical practice. By utilizing artificial intelligence (AI) and machine learning techniques, this study aims to develop a predictive model that can assist clinicians in identifying geriatric hip fracture patients at a higher risk of in-hospital mortality.

METHODS

A total of 52 707 geriatric hip fracture patients treated with surgery from 90 hospitals were included in this study. The primary outcome was postoperative in-hospital mortality. The patients were randomly divided into two groups, with a ratio of 7:3. The majority of patients, assigned to the training cohort, were used to develop the AI models. The remaining patients, assigned to the validation cohort, were used to validate the models. Various machine learning algorithms, including logistic regression (LR), decision tree (DT), naïve bayesian (NB), neural network (NN), eXGBoosting machine (eXGBM), and random forest (RF), were employed for model development. A comprehensive scoring system, incorporating 10 evaluation metrics, was developed to assess the prediction performance, with higher scores indicating superior predictive capability. Based on the best machine learning-based model, an AI application was developed on the Internet. In addition, a comparative testing of prediction performance between doctors and the AI application.

FINDINGS

The eXGBM model exhibited the best prediction performance, with an area under the curve (AUC) of 0.908 (95% CI: 0.881-0.932), as well as the highest accuracy (0.820), precision (0.817), specificity (0.814), and F1 score (0.822), and the lowest Brier score (0.120) and log loss (0.374). Additionally, the model showed favorable calibration, with a slope of 0.999 and an intercept of 0.028. According to the scoring system incorporating 10 evaluation metrics, the eXGBM model achieved the highest score (56), followed by the RF model (48) and NN model (41). The LR, DT, and NB models had total scores of 27, 30, and 13, respectively. The AI application has been deployed online at https://in-hospitaldeathinhipfracture-l9vhqo3l55fy8dkdvuskvu.streamlit.app/ , based on the eXGBM model. The comparative testing revealed that the AI application's predictive capabilities significantly outperformed those of the doctors in terms of AUC values (0.908 vs. 0.682, P <0.001).

CONCLUSIONS

The eXGBM model demonstrates promising predictive performance in assessing the risk of postoperative in-hospital mortality among geriatric hip fracture patients. The developed AI model serves as a valuable tool to enhance clinical decision-making.

Artificial Intelligence Models Are Limited in Predicting Clinical Outcomes Following Hip Arthroscopy: A Systematic Review

BACKGROUND

Hip arthroscopy has seen a significant surge in utilization, but complications remain, and optimal functional outcomes are not guaranteed. Artificial intelligence (AI) has emerged as an effective supportive decision-making tool for surgeons. The purpose of this systematic review was to characterize the outcomes, performance, and validity (generalizability) of AI-based prediction models for hip arthroscopy in current literature.

METHODS

Two reviewers independently completed structured searches using PubMed/MEDLINE and Embase databases on August 10, 2022. The search query used the terms as follows: (artificial intelligence OR machine learning OR deep learning) AND (hip arthroscopy). Studies that investigated AI-based risk prediction models in hip arthroscopy were included. The primary outcomes of interest were the variable(s) predicted by the models, best model performance achieved (primarily based on area under the curve, but also accuracy, etc), and whether the model(s) had been externally validated (generalizable).

RESULTS

Seventy-seven studies were identified from the primary search. Thirteen studies were included in the final analysis. Six studies (n = 6,568) applied AI for predicting the achievement of minimal clinically important difference for various patient-reported outcome measures such as the visual analog scale and the International Hip Outcome Tool 12-Item Questionnaire, with area under a receiver-operating characteristic curve (AUC) values ranging from 0.572 to 0.94. Three studies used AI for predicting repeat hip surgery with AUC values between 0.67 and 0.848. Four studies focused on predicting other risks, such as prolonged postoperative opioid use, with AUC values ranging from 0.71 to 0.76. None of the 13 studies assessed the generalizability of their models through external validation.

CONCLUSION

AI is being investigated for predicting clinical outcomes after hip arthroscopy. However, the performance of AI models varies widely, with AUC values ranging from 0.572 to 0.94. Critically, none of the models have undergone external validation, limiting their clinical applicability. Further research is needed to improve model performance and ensure generalizability before these tools can be reliably integrated into patient care.

LEVEL OF EVIDENCE

Level IV. See Instructions for Authors for a complete description of levels of evidence.

Systematic review of multivariable prognostic models for outcomes at least 30 days after hip fracture finds 18 mortality models but no nonmortality models warranting validation

OBJECTIVES

Prognostic models have the potential to aid clinical decision-making after hip fracture. This systematic review aimed to identify, critically appraise, and summarize multivariable prediction models for mortality or other long-term recovery outcomes occurring at least 30 days after hip fracture.

STUDY DESIGN AND SETTING

MEDLINE, Embase, Scopus, Web of Science, and CINAHL databases were searched up to May 2023. Studies were included that aimed to develop multivariable models to make predictions for individuals at least 30 days after hip fracture. Risk of bias (ROB) was dual-assessed using the Prediction model Risk Of Bias ASsessment Tool. Study and model details were extracted and summarized.

RESULTS

From 5571 records, 80 eligible studies were identified. They predicted mortality in n = 55 studies/81 models and nonmortality outcomes (mobility, function, residence, medical, and surgical complications) in n = 30 studies/45 models. Most (n = 46; 58%) studies were published since 2020. A quarter of studies (n = 19; 24%) reported using 'machine-learning methods', while the remainder used logistic regression (n = 54; 68%) and other statistical methods (n = 11; 14%) to build models. Overall, 15 studies (19%) presented 18 low ROB models, all predicting mortality. Common concerns were sample size, missing data handling, inadequate internal validation, and calibration assessment. Many studies with nonmortality outcomes (n = 11; 37%) had clear data complexities that were not correctly modeled.

CONCLUSION

This review has comprehensively summarized and appraised multivariable prediction models for long-term outcomes after hip fracture. Only 15 studies of 55 predicting mortality were rated as low ROB, warranting further development of their models. All studies predicting nonmortality outcomes were high or unclear ROB. Careful consideration is required for both the methods used and justification for developing further nonmortality prediction models for this clinical population.

Application of Machine Learning in Predicting Perioperative Outcomes in Patients with Cancer: A Narrative Review for Clinicians

This narrative review explores the utilization of machine learning (ML) and artificial intelligence (AI) models to enhance perioperative cancer care. ML and AI models offer significant potential to improve perioperative cancer care by predicting outcomes and supporting clinical decision-making. Tailored for perioperative professionals including anesthesiologists, surgeons, critical care physicians, nurse anesthetists, and perioperative nurses, this review provides a comprehensive framework for the integration of ML and AI models to enhance patient care delivery throughout the perioperative continuum.

Development and validation of an artificial intelligence mobile application for predicting 30-day mortality in critically ill patients with orthopaedic trauma

BACKGROUND

Given the intricate and grave nature of trauma-related injuries in ICU settings, it is imperative to develop and deploy reliable predictive tools that can aid in the early identification of high-risk patients who are at risk of early death. The objective of this study is to create and validate an artificial intelligence (AI) model that can accurately predict early mortality among critical fracture patients.

METHODS

A total of 2662 critically ill patients with orthopaedic trauma were included from the MIMIC III database. Early mortality was defined as death within 30 days in this study. The patients were randomly divided into a model training cohort and a model validation cohort. Various algorithms, including logistic regression (LR), extreme gradient boosting machine (eXGBM), decision tree (DT), support vector machine (SVM), random forest (RF), and neural network (NN), were employed. Evaluation metrics, including discrimination and calibration, were used to develop a comprehensive scoring system ranging from 0 to 60, with higher scores indicating better prediction performance. Furthermore, external validation was carried out using 131 patients. The optimal model was deployed as an internet-based AI tool.

RESULTS

Among all models, the eXGBM demonstrated the highest area under the curve (AUC) value (0.974, 95%CI: 0.959-0.983), followed by the RF model (0.951, 95%CI: 0.935-0.967) and the NN model (0.922, 95%CI: 0.905-0.941). Additionally, the eXGBM model outperformed other models in terms of accuracy (0.915), precision (0.906), recall (0.926), F1 score (0.916), Brier score (0.062), log loss (0.210), and discrimination slope (0.767). Based on the scoring system, the eXGBM model achieved the highest score (53), followed by RF (42) and NN (39). The LR, DT, and SVM models obtained scores of 28, 18, and 32, respectively. Decision curve analysis further confirmed the superior clinical net benefits of the eXGBM model. External validation of the model achieved an AUC value of 0.913 (95%CI: 0.878-0.948). Consequently, the model was deployed on the Internet at https://30-daymortalityincriticallyillpatients-fnfsynbpbp6rgineaspuim.streamlit.app/, allowing users to input patient features and obtain predicted risks of early mortality among critical fracture patients. Furthermore, the AI model successfully stratified patients into low or high risk of early mortality based on a predefined threshold and provided recommendations for appropriate therapeutic interventions.

CONCLUSION

This study successfully develops and validates an AI model, with the eXGBM algorithm demonstrating the highest predictive performance for early mortality in critical fracture patients. By deploying the model as a web-based AI application, healthcare professionals can easily access the tool, enabling them to predict 30-day mortality and aiding in the identification and management of high-risk patients among those critically ill with orthopedic trauma.

Using machine-learning to decode postoperative hip mortality Trends: Actionable insights from an extensive clinical dataset

BACKGROUND

Hip fractures are one of the most common injuries experienced by the general population. Despite advances in surgical techniques, postoperative mortality rates remain high. identifying relevant clinical factors associated with mortality is essential to preoperative risk stratification and tailored post-surgical interventions to improve patient outcomes. The purpose of this study aimed to identify preoperative risk factors and develop predictive models for increased hip fracture-related mortality within 30 days post-surgery, using one of the largest patient cohorts to date.

METHODS

Data from the American College of Surgeons National Surgical Quality Improvement Program database, comprising 107,660 hip fracture patients treated with surgical fixation was used. A penalized regression approach, least absolute shrinkage and selection operator was employed to develop two predictive models: one using preoperative factors and the second incorporating both preoperative and postoperative factors.

RESULTS

The analysis identified 68 preoperative factor outcomes associated with 30-day mortality. The combined model revealed 84 relevant factors, showing strong predictive power for determining postoperative mortality, with an AUC of 0.83.

CONCLUSIONS

The study's comprehensive methodology provides risk assessment tools for clinicians to identify high-risk patients and optimize patient-specific care.

Machine learning in the prediction of in-hospital mortality in patients with first acute myocardial infarction

BACKGROUND

Persistent efforts are required to further reduce the in-hospital mortality of patients suffering from acute myocardial infarction (AMI), even in the face of a global trend of declining AMI-related fatalities. We investigated deep machine learning models for in-hospital death prediction in patients on their first AMI.

METHOD

In this 2-center retrospective analysis, first AMI patients from Hospital I and Hospital II were included; 4783 patients from Hospital 1 were split in a 7:3 ratio between the training and testing sets. Data from 1053 AMI patients in Hospital II was used for further validation. 70 clinical information and laboratory examination parameters as predictive indicators were included. Logistic Regression Classifier (LR), Random Forests Classifier (RF), eXtreme Gradient Boosting (XGB), Support Vector Machine Classifier (SVM), Multilayer Perceptron (MLP), Gradient Boosting Machine (GBM), Bootstrapped Aggregation (Bagging) models with AMI patients were developed. The importance of selected variables was obtained through the Shapley Additive exPlanations (SHAP) method. The performance of each model was shown using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (Average Precision; AP).

RESULT

The in-hospital mortality for AMI in the training, testing, and validation sets were 5.7 %, 5.6 %, and 6.0 %, respectively. The top 8 predictors were D-dimer, brain natriuretic peptide, cardiogenic shock, neutrophil, prothrombin time, blood urea nitrogen, cardiac arrest, and phosphorus. In the testing cohort, the models of LR, RF, XGB, SVM, MLP, GBM, and Bagging yielded AUROC values of 0.929, 0.931, 0.907, 0.868, 0.907, 0.923, and 0.932, respectively. Bagging has good predictive value and certain clinical value in external validation with AUROC 0.893.

CONCLUSION

In order to improve the forecasting accuracy of the risk of AMI patients, guide clinical nursing practice, and lower AMI inpatient mortality, this study looked into significant indicators and the optimal models for predicting AMI inpatient mortality.

Prognostic Factors for Functional Recovery at 1-Year Following Fragility Hip Fractures

Background

The main objective of treating fragility hip fractures is to maximize the patients' ability to return to their basic activities of daily living (ADL) levels. This study explored prognostic factors associated with the ability to recover pre-fracture ADL levels at 1 year after fragility hip fractures.

Methods

We retrospectively recruited patients admitted with fragility hip fractures between July 2016 and September 2018. Details of the following were extracted from electronic medical records: age, sex, body mass index; pre-fracture Charlson Comorbidity Index (CCI), Barthel index, and EuroQol-Visual Analog Scale (EQ-VAS) scores; pre-fracture ambulatory status; and fracture type and treatment. The primary endpoint was the ability to return to the pre-fracture ADL status at 1 year. Multivariable logistic regression analysis assessed the prognostic ability of predictors.

Results

Of 405 patients, 284 (70.1%) managed to return to their pre-fracture ADL status. Multivariable logistic regression analysis demonstrated that the predictor with the most apparent effect size was pre-fracture EQ-VAS scores ≥ 65 (multivariable odds ratio [mOR], 12.90; p = 0.03). Other influential predictors were CCI scores < 5 (mOR, 1.96; p = 0.01) and surgical treatment for the hip fracture.

Conclusions

Three prognostic factors can predict a hip fracture patient's ability to return to the pre-fracture ambulatory status at 1 year. They are the patient's CCI score, operative treatment for the hip fracture, and the pre-fracture EQ-VAS score. This information could be used to develop a clinical prediction model based on the prognostic factors.

Systematic review of machine-learning models in orthopaedic trauma

Aims

Machine-learning (ML) prediction models in orthopaedic trauma hold great promise in assisting clinicians in various tasks, such as personalized risk stratification. However, an overview of current applications and critical appraisal to peer-reviewed guidelines is lacking. The objectives of this study are to 1) provide an overview of current ML prediction models in orthopaedic trauma; 2) evaluate the completeness of reporting following the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement; and 3) assess the risk of bias following the Prediction model Risk Of Bias Assessment Tool (PROBAST) tool.

Methods

A systematic search screening 3,252 studies identified 45 ML-based prediction models in orthopaedic trauma up to January 2023. The TRIPOD statement assessed transparent reporting and the PROBAST tool the risk of bias.

Results

A total of 40 studies reported on training and internal validation; four studies performed both development and external validation, and one study performed only external validation. The most commonly reported outcomes were mortality (33%, 15/45) and length of hospital stay (9%, 4/45), and the majority of prediction models were developed in the hip fracture population (60%, 27/45). The overall median completeness for the TRIPOD statement was 62% (interquartile range 30 to 81%). The overall risk of bias in the PROBAST tool was low in 24% (11/45), high in 69% (31/45), and unclear in 7% (3/45) of the studies. High risk of bias was mainly due to analysis domain concerns including small datasets with low number of outcomes, complete-case analysis in case of missing data, and no reporting of performance measures.

Conclusion

The results of this study showed that despite a myriad of potential clinically useful applications, a substantial part of ML studies in orthopaedic trauma lack transparent reporting, and are at high risk of bias. These problems must be resolved by following established guidelines to instil confidence in ML models among patients and clinicians. Otherwise, there will remain a sizeable gap between the development of ML prediction models and their clinical application in our day-to-day orthopaedic trauma practice.

Prediction of Complications and Prognostication in Perioperative Medicine: A Systematic Review and PROBAST Assessment of Machine Learning Tools

BACKGROUND

The utilization of artificial intelligence and machine learning as diagnostic and predictive tools in perioperative medicine holds great promise. Indeed, many studies have been performed in recent years to explore the potential. The purpose of this systematic review is to assess the current state of machine learning in perioperative medicine, its utility in prediction of complications and prognostication, and limitations related to bias and validation.

METHODS

A multidisciplinary team of clinicians and engineers conducted a systematic review using the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) protocol. Multiple databases were searched, including Scopus, Cumulative Index to Nursing and Allied Health Literature (CINAHL), the Cochrane Library, PubMed, Medline, Embase, and Web of Science. The systematic review focused on study design, type of machine learning model used, validation techniques applied, and reported model performance on prediction of complications and prognostication. This review further classified outcomes and machine learning applications using an ad hoc classification system. The Prediction model Risk Of Bias Assessment Tool (PROBAST) was used to assess risk of bias and applicability of the studies.

RESULTS

A total of 103 studies were identified. The models reported in the literature were primarily based on single-center validations (75%), with only 13% being externally validated across multiple centers. Most of the mortality models demonstrated a limited ability to discriminate and classify effectively. The PROBAST assessment indicated a high risk of systematic errors in predicted outcomes and artificial intelligence or machine learning applications.

CONCLUSIONS

The findings indicate that the development of this field is still in its early stages. This systematic review indicates that application of machine learning in perioperative medicine is still at an early stage. While many studies suggest potential utility, several key challenges must be first overcome before their introduction into clinical practice.

EDITOR’S PERSPECTIVE

Predicting unmet activities of daily living needs among the oldest old with disabilities in China: a machine learning approach

Background

The ageing population in China has led to a significant increase in the number of older persons with disabilities. These individuals face substantial challenges in accessing adequate activities of daily living (ADL) assistance. Unmet ADL needs among this population can result in severe health consequences and strain an already burdened care system. This study aims to identify the factors influencing unmet ADL needs of the oldest old (those aged 80 and above) with disabilities using six machine learning methods.

Methods

Drawing from the Chinese Longitudinal Healthy Longevity Survey (CLHLS) 2017-2018 data, we employed six machine learning methods to predict unmet ADL needs among the oldest old with disabilities. The predictive effects of various factors on unmet ADL needs were explored using Shapley Additive exPlanations (SHAP).

Results

The Random Forest model showed the highest prediction accuracy among the six machine learning methods tested. SHAP analysis based on the Random Forest model revealed that factors such as household registration, disability class, economic rank, self-rated health, caregiver willingness, perceived control, economic satisfaction, pension, educational attainment, financial support given to children, living arrangement, number of children, and primary caregiver played significant roles in the unmet ADL needs of the oldest old with disabilities.

Conclusion

Our study highlights the importance of socioeconomic factors (e.g., household registration and economic rank), health status (e.g., disability class and self-rated health), and caregiving relationship factors (e.g., caregiver willingness and perceived control) in reducing unmet ADL needs among the oldest old with disabilities in China. Government interventions aimed at bridging the urban-rural divide, targeting groups with deteriorating health status, and enhancing caregiver skills are essential for ensuring the well-being of this vulnerable population. These findings can inform policy decisions and interventions to better address the unmet ADL needs among the oldest old with disabilities.

Artificial Intelligence for Hip Fracture Detection and Outcome Prediction: A Systematic Review and Meta-analysis

IMPORTANCE

Artificial intelligence (AI) enables powerful models for establishment of clinical diagnostic and prognostic tools for hip fractures; however the performance and potential impact of these newly developed algorithms are currently unknown.

OBJECTIVE

To evaluate the performance of AI algorithms designed to diagnose hip fractures on radiographs and predict postoperative clinical outcomes following hip fracture surgery relative to current practices.

DATA SOURCES

A systematic review of the literature was performed using the MEDLINE, Embase, and Cochrane Library databases for all articles published from database inception to January 23, 2023. A manual reference search of included articles was also undertaken to identify any additional relevant articles.

STUDY SELECTION

Studies developing machine learning (ML) models for the diagnosis of hip fractures from hip or pelvic radiographs or to predict any postoperative patient outcome following hip fracture surgery were included.

DATA EXTRACTION AND SYNTHESIS

This study followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses and was registered with PROSPERO. Eligible full-text articles were evaluated and relevant data extracted independently using a template data extraction form. For studies that predicted postoperative outcomes, the performance of traditional predictive statistical models, either multivariable logistic or linear regression, was recorded and compared with the performance of the best ML model on the same out-of-sample data set.

MAIN OUTCOMES AND MEASURES

Diagnostic accuracy of AI models was compared with the diagnostic accuracy of expert clinicians using odds ratios (ORs) with 95% CIs. Areas under the curve for postoperative outcome prediction between traditional statistical models (multivariable linear or logistic regression) and ML models were compared.

RESULTS

Of 39 studies that met all criteria and were included in this analysis, 18 (46.2%) used AI models to diagnose hip fractures on plain radiographs and 21 (53.8%) used AI models to predict patient outcomes following hip fracture surgery. A total of 39 598 plain radiographs and 714 939 hip fractures were used for training, validating, and testing ML models specific to diagnosis and postoperative outcome prediction, respectively. Mortality and length of hospital stay were the most predicted outcomes. On pooled data analysis, compared with clinicians, the OR for diagnostic error of ML models was 0.79 (95% CI, 0.48-1.31; P = .36; I2 = 60%) for hip fracture radiographs. For the ML models, the mean (SD) sensitivity was 89.3% (8.5%), specificity was 87.5% (9.9%), and F1 score was 0.90 (0.06). The mean area under the curve for mortality prediction was 0.84 with ML models compared with 0.79 for alternative controls (P = .09).

CONCLUSIONS AND RELEVANCE

The findings of this systematic review and meta-analysis suggest that the potential applications of AI to aid with diagnosis from hip radiographs are promising. The performance of AI in diagnosing hip fractures was comparable with that of expert radiologists and surgeons. However, current implementations of AI for outcome prediction do not seem to provide substantial benefit over traditional multivariable predictive statistics.