Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review

Use of natural language processing techniques to predict patient selection for total hip and knee arthroplasty from radiology reports

Aims

To examine whether natural language processing (NLP) using a clinically based large language model (LLM) could be used to predict patient selection for total hip or total knee arthroplasty (THA/TKA) from routinely available free-text radiology reports.

Methods

Data pre-processing and analyses were conducted according to the Artificial intelligence to Revolutionize the patient Care pathway in Hip and knEe aRthroplastY (ARCHERY) project protocol. This included use of de-identified Scottish regional clinical data of patients referred for consideration of THA/TKA, held in a secure data environment designed for artificial intelligence (AI) inference. Only preoperative radiology reports were included. NLP algorithms were based on the freely available GatorTron model, a LLM trained on over 82 billion words of de-identified clinical text. Two inference tasks were performed: assessment after model-fine tuning (50 Epochs and three cycles of k-fold cross validation), and external validation.

Results

For THA, there were 5,558 patient radiology reports included, of which 4,137 were used for model training and testing, and 1,421 for external validation. Following training, model performance demonstrated average (mean across three folds) accuracy, F1 score, and area under the receiver operating curve (AUROC) values of 0.850 (95% confidence interval (CI) 0.833 to 0.867), 0.813 (95% CI 0.785 to 0.841), and 0.847 (95% CI 0.822 to 0.872), respectively. For TKA, 7,457 patient radiology reports were included, with 3,478 used for model training and testing, and 3,152 for external validation. Performance metrics included accuracy, F1 score, and AUROC values of 0.757 (95% CI 0.702 to 0.811), 0.543 (95% CI 0.479 to 0.607), and 0.717 (95% CI 0.657 to 0.778) respectively. There was a notable deterioration in performance on external validation in both cohorts.

Conclusion

The use of routinely available preoperative radiology reports provides promising potential to help screen suitable candidates for THA, but not for TKA. The external validation results demonstrate the importance of further model testing and training when confronted with new clinical cohorts.

Don't be misled: 3 misconceptions about external validation of clinical prediction models

Clinical prediction models provide risks of health outcomes that can inform patients and support medical decisions. However, most models never make it to actual implementation in practice. A commonly heard reason for this lack of implementation is that prediction models are often not externally validated. While we generally encourage external validation, we argue that an external validation is often neither sufficient nor required as an essential step before implementation. As such, any available external validation should not be perceived as a license for model implementation. We clarify this argument by discussing 3 common misconceptions about external validation. We argue that there is not one type of recommended validation design, not always a necessity for external validation, and sometimes a need for multiple external validations. The insights from this paper can help readers to consider, design, interpret, and appreciate external validation studies.

Accuracy of machine learning to predict the outcomes of shoulder arthroplasty: a systematic review

BACKGROUND

Artificial intelligence (AI) uses computer systems to simulate cognitive capacities to accomplish goals like problem-solving and decision-making. Machine learning (ML), a branch of AI, makes algorithms find connections between preset variables, thereby producing prediction models. ML can aid shoulder surgeons in determining which patients may be susceptible to worse outcomes and complications following shoulder arthroplasty (SA) and align patient expectations following SA. However, limited literature is available on ML utilization in total shoulder arthroplasty (TSA) and reverse TSA.

METHODS

A systematic literature review in accordance with PRISMA guidelines was performed to identify primary research articles evaluating ML's ability to predict SA outcomes. With duplicates removed, the initial query yielded 327 articles, and after applying inclusion and exclusion criteria, 12 articles that had at least 1 month follow-up time were included.

RESULTS

ML can predict 30-day postoperative complications with a 90% accuracy, postoperative range of motion with a higher-than-85% accuracy, and clinical improvement in patient-reported outcome measures above minimal clinically important differences with a 93%-99% accuracy. ML can predict length of stay, operative time, discharge disposition, and hospitalization costs.

CONCLUSION

ML can accurately predict outcomes and complications following SA and healthcare utilization. Outcomes are highly dependent on the type of algorithms used, data input, and features selected for the model.

LEVEL OF EVIDENCE

III.

Age and medial compartmental OA were important predictors of the lateral compartmental OA in the discoid lateral meniscus: Analysis using machine learning approach

PURPOSE

The objective of this study was to develop a machine learning model that would predict lateral compartment osteoarthritis (OA) in the discoid lateral meniscus (DLM), from which to then identify factors contributing to lateral compartment OA, with a key focus on the patient's age.

METHODS

Data were collected from 611 patients with symptomatic DLM diagnosed using magnetic resonance imaging between April 2003 and May 2022. Twenty features, including demographic, clinical and radiological data and six algorithms were used to develop the predictive machine learning models. Shapley additive explanation (SHAP) analysis was performed on the best model, in addition to subgroup analyses according to age.

RESULTS

Extreme gradient boosting classifier was identified as the best prediction model, with an area under the receiver operating characteristic curve (AUROC) of 0.968, the highest among all the models, regardless of age (AUROC of 0.977 in young age and AUROC of 0.937 in old age). In the SHAP analysis, the most predictive feature was age, followed by the presence of medial compartment OA. In the subgroup analysis, the most predictive feature was age in young age, whereas the most predictive feature was the presence of medial compartment OA in old age.

CONCLUSION

The machine learning model developed in this study showed a high predictive performance with regard to predicting lateral compartment OA of the DLM. Age was identified as the most important factor, followed by medial compartment OA. In subgroup analysis, medial compartmental OA was found to be the most important factor in the older age group, whereas age remained the most important factor in the younger age group. These findings provide insights that may prove useful for the establishment of strategies for the treatment of patients with symptomatic DLM.

LEVEL OF EVIDENCE

Level III.

An Integrated Voice Recognition and Natural Language Processing Platform to Automatically Extract Thoracolumbar Injury Classification Score Features From Radiology Reports

BACKGROUND

Many predictive models for estimating clinical outcomes after spine surgery have been reported in the literature. However, implementation of predictive scores in practice is limited by the time-intensive nature of manually abstracting relevant predictors. In this study, we designed natural language processing (NLP) algorithms to automate data abstraction for the thoracolumbar injury classification score (TLICS).

METHODS

We retrieved the radiology reports of all Mayo Clinic patients with an International Classification of Diseases, 9th or 10th revision, code corresponding to a fracture of the thoracolumbar spine between January 2005 and October 2020. Annotated data were used to train an N-gram NLP model using machine learning methods, including random forest, stepwise linear discriminant analysis, k-nearest neighbors, and penalized logistic regression models.

RESULTS

A total of 1085 spine radiology reports were included in our analysis. Our dataset included 483 compression, 401 burst, 103 translational/rotational, and 98 distraction fractures. A total of 103 reports had documented an injury of the posterior ligamentous complex. The overall accuracy of the random forest model for fracture morphology feature detection was 76.96% versus 65.90% in the stepwise linear discriminant analysis, 50.69% in the k-nearest neighbors, and 62.67% in the penalized logistic regression. The overall accuracy to detect posterior ligamentous complex integrity was highest in the random forest model at 83.41%. Our random forest model was implemented in the backend of a web application in which users can dictate reports and have TLICS features automatically extracted.

CONCLUSIONS

We have developed a machine learning NLP model for extracting TLICS features from radiology reports, which we deployed in a web application that can be integrated into clinical practice.

Psoas muscle area is an independent survival prognosticator in patients undergoing surgery for long-bone metastases

BACKGROUND

Predictive analytics is gaining popularity as an aid to treatment planning for patients with bone metastases, whose expected survival should be considered. Decreased psoas muscle area (PMA), a morphometric indicator of suboptimal nutritional status, has been associated with mortality in various cancers, but never been integrated into current survival prediction algorithms (SPA) for patients with skeletal metastases. This study investigates whether decreased PMA predicts worse survival in patients with extremity metastases and whether incorporating PMA into three modern SPAs (PATHFx, SORG-NG, and SORG-MLA) improves their performance.

METHODS

One hundred eighty-five patients surgically treated for long-bone metastases between 2014 and 2019 were divided into three PMA tertiles (small, medium, and large) based on their psoas size on CT. Kaplan-Meier, multivariable regression, and Cox proportional hazards analyses were employed to compare survival between tertiles and examine factors associated with mortality. Logistic regression analysis was used to assess whether incorporating adjusted PMA values enhanced the three SPAs' discriminatory abilities. The clinical utility of incorporating PMA into these SPAs was evaluated by decision curve analysis (DCA).

RESULTS

Patients with small PMA had worse 90-day and 1-year survival after surgery (log-rank test p < 0.001). Patients in the large PMA group had a higher chance of surviving 90 days (odds ratio, OR, 3.72, p = 0.02) and 1 year than those in the small PMA group (OR 3.28, p = 0.004). All three SPAs had increased AUC after incorporation of adjusted PMA. DCA indicated increased net benefits at threshold probabilities >0.5 after the addition of adjusted PMA to these SPAs.

CONCLUSIONS

Decreased PMA on CT is associated with worse survival in surgically treated patients with extremity metastases, even after controlling for three contemporary SPAs. Physicians should consider the additional prognostic value of PMA on survival in patients undergoing consideration for operative management due to extremity metastases.

Systematic review of machine-learning models in orthopaedic trauma

Aims

Machine-learning (ML) prediction models in orthopaedic trauma hold great promise in assisting clinicians in various tasks, such as personalized risk stratification. However, an overview of current applications and critical appraisal to peer-reviewed guidelines is lacking. The objectives of this study are to 1) provide an overview of current ML prediction models in orthopaedic trauma; 2) evaluate the completeness of reporting following the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement; and 3) assess the risk of bias following the Prediction model Risk Of Bias Assessment Tool (PROBAST) tool.

Methods

A systematic search screening 3,252 studies identified 45 ML-based prediction models in orthopaedic trauma up to January 2023. The TRIPOD statement assessed transparent reporting and the PROBAST tool the risk of bias.

Results

A total of 40 studies reported on training and internal validation; four studies performed both development and external validation, and one study performed only external validation. The most commonly reported outcomes were mortality (33%, 15/45) and length of hospital stay (9%, 4/45), and the majority of prediction models were developed in the hip fracture population (60%, 27/45). The overall median completeness for the TRIPOD statement was 62% (interquartile range 30 to 81%). The overall risk of bias in the PROBAST tool was low in 24% (11/45), high in 69% (31/45), and unclear in 7% (3/45) of the studies. High risk of bias was mainly due to analysis domain concerns including small datasets with low number of outcomes, complete-case analysis in case of missing data, and no reporting of performance measures.

Conclusion

The results of this study showed that despite a myriad of potential clinically useful applications, a substantial part of ML studies in orthopaedic trauma lack transparent reporting, and are at high risk of bias. These problems must be resolved by following established guidelines to instil confidence in ML models among patients and clinicians. Otherwise, there will remain a sizeable gap between the development of ML prediction models and their clinical application in our day-to-day orthopaedic trauma practice.

External validation of machine learning algorithm predicting prolonged opioid prescriptions in opioid-naïve lumbar spine surgery patients using a Taiwanese cohort

BACKGROUND/PURPOSE

Identifying patients at risk of prolonged opioid use after surgery prompts appropriate prescription and personalized treatment plans. The Skeletal Oncology Research Group machine learning algorithm (SORG-MLA) was developed to predict the risk of prolonged opioid use in opioid-naive patients after lumbar spine surgery. However, its utility in a distinct country remains unknown.

METHODS

A Taiwanese cohort containing 2795 patients who were 20 years or older undergoing primary surgery for lumbar decompression from 2010 to 2018 were used to validate the SORG-MLA. Discrimination (area under receiver operating characteristic curve [AUROC] and area under precision-recall curve [AUPRC]), calibration, overall performance (Brier score), and decision curve analysis were applied.

RESULTS

Among 2795 patients, the prolonged opioid prescription rate was 5.2%. The validation cohort were older, more inpatient disposition, and more common pharmaceutical history of NSAIDs. Despite the differences, the SORG-MLA provided a good discriminative ability (AUROC of 0.71 and AURPC of 0.36), a good overall performance (Brier score of 0.044 compared to that of 0.039 in the developmental cohort). However, the probability of prolonged opioid prescription tended to be overestimated (calibration intercept of -0.07 and calibration slope of 1.45). Decision curve analysis suggested greater clinical net benefit in a wide range of clinical scenarios.

CONCLUSION

The SORG-MLA retained good discriminative abilities and overall performances in a geologically and medicolegally different region. It was suitable for predicting patients in risk of prolonged postoperative opioid use in Taiwan.

Artificial Intelligence for Automated Implant Identification in Knee Arthroplasty: A Multicenter External Validation Study Exceeding 3.5 Million Plain Radiographs

BACKGROUND

Surgical management of complications following knee arthroplasty demands accurate and timely identification of implant manufacturer and model. Automated image processing using deep machine learning has been previously developed and internally validated; however, external validation is essential prior to scaling clinical implementation for generalizability.

METHODS

We trained, validated, and externally tested a deep learning system to classify knee arthroplasty systems as one of the 9 models from 4 manufacturers derived from 4,724 original, retrospectively collected anteroposterior plain knee radiographs across 3 academic referral centers. From these radiographs, 3,568 were used for training, 412 for validation, and 744 for external testing. Augmentation was applied to the training set (n = 3,568,000) to increase model robustness. Performance was determined by the area under the receiver operating characteristic curve, sensitivity, specificity, and accuracy. Implant identification processing speed was calculated. The training and testing sets were drawn from statistically different populations of implants (P < .001).

RESULTS

After 1,000 training epochs by the deep learning system, the system discriminated 9 implant models with a mean area under the receiver operating characteristic curve of 0.989, accuracy of 97.4%, sensitivity of 89.2%, and specificity of 99.0% in the external testing dataset of 744 anteroposterior radiographs. The software classified implants at a mean speed of 0.02 seconds per image.

CONCLUSION

An artificial intelligence-based software for identifying knee arthroplasty implants demonstrated excellent internal and external validation. Although continued surveillance is necessary with implant library expansion, this software represents a responsible and meaningful clinical application of artificial intelligence with immediate potential to globally scale and assist in preoperative planning prior to revision knee arthroplasty.

An Overview of Machine Learning in Orthopedic Surgery: An Educational Paper

The growth of artificial intelligence combined with the collection and storage of large amounts of data in the electronic medical record collection has created an opportunity for orthopedic research and translation into the clinical environment. Machine learning (ML) is a type of artificial intelligence tool well suited for processing the large amount of available data. Specific areas of ML frequently used by orthopedic surgeons performing total joint arthroplasty include tabular data analysis (spreadsheets), medical imaging processing, and natural language processing (extracting concepts from text). Previous studies have discussed models able to identify fractures in radiographs, identify implant type in radiographs, and determine the stage of osteoarthritis based on walking analysis. Despite the growing popularity of ML, there are limitations including its reliance on "good" data, potential for overfitting, long life cycle for creation, and ability to only perform one narrow task. This educational article will further discuss a general overview of ML, discussing these challenges and including examples of successfully published models.

Internal and External Validation of the Generalizability of Machine Learning Algorithms in Predicting Non-home Discharge Disposition Following Primary Total Knee Joint Arthroplasty

BACKGROUND

Nonhome discharge disposition following primary total knee arthroplasty (TKA) is associated with a higher rate of complications and constitutes a socioeconomic burden on the health care system. While existing algorithms predicting nonhome discharge disposition varied in degrees of mathematical complexity and prediction power, their capacity to generalize predictions beyond the development dataset remains limited. Therefore, this study aimed to establish the machine learning model generalizability by performing internal and external validations using nation-scale and institutional cohorts, respectively.

METHODS

Four machine learning models were trained using the national cohort. Recursive feature elimination and hyper-parameter tuning were applied. Internal validation was achieved through five-fold cross-validation during model training. The trained models' performance was externally validated using the institutional cohort and assessed by discrimination, calibration, and clinical utility.

RESULTS

The national (424,354 patients) and institutional (10,196 patients) cohorts had non-home discharge rates of 19.4 and 36.4%, respectively. The areas under the receiver operating curve of the model predictions were 0.83 to 0.84 during internal validation and increased to 0.88 to 0.89 during external validation. Artificial neural network and histogram-based gradient boosting elicited the best performance with a mean area under the receiver operating curve of 0.89, calibration slope of 1.39, and Brier score of 0.14, which indicated that the two models were robust in distinguishing non-home discharge and well-calibrated with accurate predictions of the probabilities. The low inter-dataset similarity indicated reliable external validation. Length of stay, age, body mass index, and sex were the strongest predictors of discharge destination after primary TKA.

CONCLUSION

The machine learning models demonstrated excellent predictive performance during both internal and external validations, supporting their generalizability across different patient cohorts and potential applicability in the clinical workflow.

Artificial Intelligence for Automated Implant Identification in Total Hip Arthroplasty: A Multicenter External Validation Study Exceeding Two Million Plain Radiographs

BACKGROUND

The surgical management of complications after total hip arthroplasty (THA) necessitates accurate identification of the femoral implant manufacturer and model. Automated image processing using deep learning has been previously developed and internally validated; however, external validation is necessary prior to responsible application of artificial intelligence (AI)-based technologies.

METHODS

We trained, validated, and externally tested a deep learning system to classify femoral-sided THA implants as one of the 8 models from 2 manufacturers derived from 2,954 original, deidentified, retrospectively collected anteroposterior plain radiographs across 3 academic referral centers and 13 surgeons. From these radiographs, 2,117 were used for training, 249 for validation, and 588 for external testing. Augmentation was applied to the training set (n = 2,117,000) to increase model robustness. Performance was evaluated by area under the receiver operating characteristic curve, sensitivity, specificity, and accuracy. Implant identification processing speed was calculated.

RESULTS

The training and testing sets were drawn from statistically different populations of implants (P < .001). After 1,000 training epochs by the deep learning system, the system discriminated 8 implant models with a mean area under the receiver operating characteristic curve of 0.991, accuracy of 97.9%, sensitivity of 88.6%, and specificity of 98.9% in the external testing dataset of 588 anteroposterior radiographs. The software classified implants at a mean speed of 0.02 seconds per image.

CONCLUSION

An AI-based software demonstrated excellent internal and external validation. Although continued surveillance is necessary with implant library expansion, this software represents responsible and meaningful clinical application of AI with immediate potential to globally scale and assist in preoperative planning prior to revision THA.

Determining patients with spinal metastases suitable for surgical intervention: A cost-effective analysis

BACKGROUND

Both nonoperative and operative treatments for spinal metastasis are expensive interventions. Patients' expected 3-month survival is believed to be a key factor to determine the most suitable treatment. However, to the best of our knowledge, no previous study lends support to the hypothesis. We sought to determine the cost-effectiveness of operative and nonoperative interventions, stratified by patients' predicted probability of 3-month survival.

METHODS

A Markov model with four defined health states was used to estimate the quality-adjusted life years (QALYs) and costs for operative intervention with postoperative radiotherapy and radiotherapy alone (palliative low-dose external beam radiotherapy) of spine metastases. Transition probabilities for the model, including the risks of mortality and functional deterioration, were obtained from secondary and our institutional data. Willingness to pay thresholds were prespecified at $100,000 and $150,000. The analyses were censored after 5-year simulation from a health system perspective and discounted outcomes at 3% per year. Sensitivity analyses were conducted to test the robustness of the study design.

RESULTS

The incremental cost-effectiveness ratios were $140,907 per QALY for patients with a 3-month survival probability >50%, $3,178,510 per QALY for patients with a 3-month survival probability <50%, and $168,385 per QALY for patients with independent ambulatory and 3-month survival probability >50%.

CONCLUSIONS

This study emphasizes the need to choose patients carefully and estimate preoperative survival for those with spinal metastases. In addition to reaffirming previous research regarding the influence of ambulatory status on cost-effectiveness, our study goes a step further by highlighting that operative intervention with postoperative radiotherapy could be more cost-effective than radiotherapy alone for patients with a better survival outlook. Accurate survival prediction tools and larger future studies could offer more detailed insights for clinical decisions.

Prospective multicenter external validation of postoperative mortality prediction tools in patients undergoing emergency laparotomy

BACKGROUND

Accurate preoperative risk assessment in emergency laparotomy (EL) is valuable for informed decision making and rational use of resources. Available risk prediction tools have not been validated adequately across diverse health care settings. Herein, we report a comparative external validation of four widely cited prognostic models.

METHODS

A multicenter cohort was prospectively composed of consecutive patients undergoing EL in 11 Greek hospitals from January 2020 to May 2021 using the National Emergency Laparotomy Audit (NELA) inclusion criteria. Thirty-day mortality risk predictions were calculated using the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP), NELA, Portsmouth Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (P-POSSUM), and Predictive Optimal Trees in Emergency Surgery Risk tools. Surgeons' assessment of postoperative mortality using predefined cutoffs was recorded, and a surgeon-adjusted ACS-NSQIP prediction was calculated when the original model's prediction was relatively low. Predictive performances were compared using scaled Brier scores, discrimination and calibration measures and plots, and decision curve analysis. Heterogeneity across hospitals was assessed by random-effects meta-analysis.

RESULTS

A total of 631 patients were included, and 30-day mortality was 16.3%. The ACS-NSQIP and its surgeon-adjusted version had the highest scaled Brier scores. All models presented high discriminative ability, with concordance statistics ranging from 0.79 for P-POSSUM to 0.85 for NELA. However, except the surgeon-adjusted ACS-NSQIP (Hosmer-Lemeshow test, p = 0.742), all other models were poorly calibrated ( p < 0.001). Decision curve analysis revealed superior clinical utility of the ACS-NSQIP. Following recalibrations, predictive accuracy improved for all models, but ACS-NSQIP retained the lead. Between-hospital heterogeneity was minimum for the ACS-NSQIP model and maximum for P-POSSUM.

CONCLUSION

The ACS-NSQIP tool was most accurate for mortality predictions after EL in a broad external validation cohort, demonstrating utility for facilitating preoperative risk management in the Greek health care system. Subjective surgeon assessments of patient prognosis may optimize ACS-NSQIP predictions.

LEVEL OF EVIDENCE

Diagnostic Test/Criteria; Level II.

Social determinants of health in prognostic machine learning models for orthopaedic outcomes: A systematic review

RATIONAL

Social determinants of health (SDOH) are being considered more frequently when providing orthopaedic care due to their impact on treatment outcomes. Simultaneously, prognostic machine learning (ML) models that facilitate clinical decision making have become popular tools in the field of orthopaedic surgery. When ML-driven tools are developed, it is important that the perpetuation of potential disparities is minimized. One approach is to consider SDOH during model development. To date, it remains unclear whether and how existing prognostic ML models for orthopaedic outcomes consider SDOH variables.

OBJECTIVE

To investigate whether prognostic ML models for orthopaedic surgery outcomes account for SDOH, and to what extent SDOH variables are included in the final models.

METHODS

A systematic search was conducted in PubMed, Embase and Cochrane for studies published up to 17 November 2020. Two reviewers independently extracted SDOH features using the PROGRESS+ framework (place of residence, race/ethnicity, Occupation, gender/sex, religion, education, social capital, socioeconomic status, 'Plus+' age, disability, and sexual orientation).

RESULTS

The search yielded 7138 studies, of which 59 met the inclusion criteria. Across all studies, 96% (57/59) considered at least one PROGRESS+ factor during development. The most common factors were age (95%; 56/59) and gender/sex (96%; 57/59). Differential effect analyses, such as subgroup analysis, covariate adjustment, and baseline comparison, were rarely reported (10%; 6/59). The majority of models included age (92%; 54/59) and gender/sex (69%; 41/59) as final input variables. However, factors such as insurance status (7%; 4/59), marital status (7%; 4/59) and income (3%; 2/59) were seldom included.

CONCLUSION

The current level of reporting and consideration of SDOH during the development of prognostic ML models for orthopaedic outcomes is limited. Healthcare providers should be critical of the models they consider using and knowledgeable regarding the quality of model development, such as adherence to recognized methodological standards. Future efforts should aim to avoid bias and disparities when developing ML-driven applications for orthopaedics.

Validation and Generalizability of Machine Learning Models for the Prediction of Discharge Disposition Following Revision Total Knee Arthroplasty

BACKGROUND

Postoperative discharge to facilities account for over 33% of the $ 2.7 billion revision total knee arthroplasty (TKA)-associated annual expenditures and are associated with increased complications when compared to home discharges. Prior studies predicting discharge disposition using advanced machine learning (ML) have been limited due to a lack of generalizability and validation. This study aimed to establish ML model generalizability by externally validating its prediction for nonhome discharge following revision TKA using national and institutional databases.

METHODS

The national and institutional cohorts comprised 52,533 and 1,628 patients, respectively, with 20.6 and 19.4% nonhome discharge rates. Five ML models were trained and internally validated (five-fold cross-validation) on a large national dataset. Subsequently, external validation was performed on our institutional dataset. Model performance was assessed using discrimination, calibration, and clinical utility. Global predictor importance plots and local surrogate models were used for interpretation.

RESULTS

The strongest predictors of nonhome discharge were patient age, body mass index, and surgical indication. The area under the receiver operating characteristic curve increased from internal to external validation and ranged between 0.77 and 0.79. Artificial neural network was the best predictive model for identifying patients at risk for nonhome discharge (area under the receiver operating characteristic curve = 0.78), and also the most accurate (calibration slope = 0.93, intercept = 0.02, and Brier score = 0.12).

CONCLUSION

All five ML models demonstrated good-to-excellent discrimination, calibration, and clinical utility on external validation, with artificial neural network being the best model for predicting discharge disposition following revision TKA. Our findings establish the generalizability of ML models developed using data from a national database. The integration of these predictive models into clinical workflow may assist in optimizing discharge planning, bed management, and cost containment associated with revision TKA.

Intercontinental validation of a clinical prediction model for predicting 90-day and 2-year mortality in an Israeli cohort of 2033 patients with a femoral neck fracture aged 65 or above

PURPOSE

Mortality prediction in elderly femoral neck fracture patients is valuable in treatment decision-making. A previously developed and internally validated clinical prediction model shows promise in identifying patients at risk of 90-day and 2-year mortality. Validation in an independent cohort is required to assess the generalizability; especially in geographically distinct regions. Therefore we questioned, is the SORG Orthopaedic Research Group (SORG) femoral neck fracture mortality algorithm externally valid in an Israeli cohort to predict 90-day and 2-year mortality?

METHODS

We previously developed a prediction model in 2022 for estimating the risk of mortality in femoral neck fracture patients using a multicenter institutional cohort of 2,478 patients from the USA. The model included the following input variables that are available on clinical admission: age, male gender, creatinine level, absolute neutrophil, hemoglobin level, international normalized ratio (INR), congestive heart failure (CHF), displaced fracture, hemiplegia, chronic obstructive pulmonary disease (COPD), history of cerebrovascular accident (CVA) and beta-blocker use. To assess the generalizability, we used an intercontinental institutional cohort from the Sheba Medical Center in Israel (level I trauma center), queried between June 2008 and February 2022. Generalizability of the model was assessed using discrimination, calibration, Brier score, and decision curve analysis.

RESULTS

The validation cohort included 2,033 patients, aged 65 years or above, that underwent femoral neck fracture surgery. Most patients were female 64.8% (n = 1317), the median age was 81 years (interquartile range = 75-86), and 80.4% (n = 1635) patients sustained a displaced fracture (Garden III/IV). The 90-day mortality was 9.4% (n = 190) and 2-year mortality was 30.0% (n = 610). Despite numerous baseline differences, the model performed acceptably to the validation cohort on discrimination (c-statistic 0.67 for 90-day, 0.67 for 2-year), calibration, Brier score, and decision curve analysis.

CONCLUSIONS

The previously developed SORG femoral neck fracture mortality algorithm demonstrated good performance in an independent intercontinental population. Current iteration should not be relied on for patient care, though suggesting potential utility in assessing patients at low risk for 90-day or 2-year mortality. Further studies should evaluate this tool in a prospective setting and evaluate its feasibility and efficacy in clinical practice. The algorithm can be freely accessed: https://sorg-apps.shinyapps.io/hipfracturemortality/ .

LEVEL OF EVIDENCE

Level III, Prognostic study.

Predicting complications of spine surgery: external validation of three models

BACKGROUND CONTEXT

Numerous prediction tools are available for estimating postoperative risk following spine surgery. External validation and comparison of these tools is critical prior to clinical use. No model for adverse events after spine surgery has undergone decision curve analysis.

PURPOSE

External validation, comparison, and decision curve analysis of 3 previously described models [SpineSage, Risk Assessment Tool (RAT), National Surgical Quality Improvement Program Risk Calculator (NSQIP)] for predicting 30-day postoperative complications after spine surgery STUDY DESIGN: Retrospective cohort study.

PATIENT SAMPLE

Three hundred fifteen patients who underwent spine surgery at a tertiary academic surgical center in New Zealand between January 2019 and April 2020.

OUTCOME MEASURES

As defined by each risk prediction tool and objectively using the Comprehensive Complication Index.

METHODS

We retrospectively reviewed risk of postoperative complication was calculated for each patient according to the 3 models. Overall model fit, calibration, discrimination, and decision curve analysis for each model were assessed in line with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines.

RESULTS

100 (35%) patients experienced complications. SpineSage and RAT were well calibrated, NSQIP systematically underestimated risk. Area under the curve was greatest for SpineSage (0.75) compared with the NSQIP (0.72) and the RAT (0.69). Decision curve analysis showed SpineSage resulted in greatest net benefit across all risk thresholds.

CONCLUSIONS

Of the models studied, SpineSage most accurately predicted risk and can be expected to perform better than a strategy of treating all patients if patient or surgeon deem complication risk >10% significant. NSQIP may not be suitable for the clinical use in our local population.

Does the SORG Orthopaedic Research Group Hip Fracture Delirium Algorithm Perform Well on an Independent Intercontinental Cohort of Patients With Hip Fractures Who Are 60 Years or Older?

BACKGROUND

Postoperative delirium in patients aged 60 years or older with hip fractures adversely affects clinical and functional outcomes. The economic cost of delirium is estimated to be as high as USD 25,000 per patient, with a total budgetary impact between USD 6.6 to USD 82.4 billion annually in the United States alone. Forty percent of delirium episodes are preventable, and accurate risk stratification can decrease the incidence and improve clinical outcomes in patients. A previously developed clinical prediction model (the SORG Orthopaedic Research Group hip fracture delirium machine-learning algorithm) is highly accurate on internal validation (in 28,207 patients with hip fractures aged 60 years or older in a US cohort) in identifying at-risk patients, and it can facilitate the best use of preventive interventions; however, it has not been tested in an independent population. For an algorithm to be useful in real life, it must be valid externally, meaning that it must perform well in a patient cohort different from the cohort used to "train" it. With many promising machine-learning prediction models and many promising delirium models, only few have also been externally validated, and even fewer are international validation studies.

QUESTION/PURPOSE

Does the SORG hip fracture delirium algorithm, initially trained on a database from the United States, perform well on external validation in patients aged 60 years or older in Australia and New Zealand?

METHODS

We previously developed a model in 2021 for assessing risk of delirium in hip fracture patients using records of 28,207 patients obtained from the American College of Surgeons National Surgical Quality Improvement Program. Variables included in the original model included age, American Society of Anesthesiologists (ASA) class, functional status (independent or partially or totally dependent for any activities of daily living), preoperative dementia, preoperative delirium, and preoperative need for a mobility aid. To assess whether this model could be applied elsewhere, we used records from an international hip fracture registry. Between June 2017 and December 2018, 6672 patients older than 60 years of age in Australia and New Zealand were treated surgically for a femoral neck, intertrochanteric hip, or subtrochanteric hip fracture and entered into the Australian & New Zealand Hip Fracture Registry. Patients were excluded if they had a pathological hip fracture or septic shock. Of all patients, 6% (402 of 6672) did not meet the inclusion criteria, leaving 94% (6270 of 6672) of patients available for inclusion in this retrospective analysis. Seventy-one percent (4249 of 5986) of patients were aged 80 years or older, after accounting for 5% (284 of 6270) of missing values; 68% (4292 of 6266) were female, after accounting for 0.06% (4 of 6270) of missing values, and 83% (4690 of 5661) of patients were classified as ASA III/IV, after accounting for 10% (609 of 6270) of missing values. Missing data were imputed using the missForest methodology. In total, 39% (2467 of 6270) of patients developed postoperative delirium. The performance of the SORG hip fracture delirium algorithm on the validation cohort was assessed by discrimination, calibration, Brier score, and a decision curve analysis. Discrimination, known as the area under the receiver operating characteristic curves (c-statistic), measures the model's ability to distinguish patients who achieved the outcomes from those who did not and ranges from 0.5 to 1.0, with 1.0 indicating the highest discrimination score and 0.50 the lowest. Calibration plots the predicted versus the observed probabilities, a perfect plot has an intercept of 0 and a slope of 1. The Brier score calculates a composite of discrimination and calibration, with 0 indicating perfect prediction and 1 the poorest.

RESULTS

The SORG hip fracture algorithm, when applied to an external patient cohort, distinguished between patients at low risk and patients at moderate to high risk of developing postoperative delirium. The SORG hip fracture algorithm performed with a c-statistic of 0.74 (95% confidence interval 0.73 to 0.76). The calibration plot showed high accuracy in the lower predicted probabilities (intercept -0.28, slope 0.52) and a Brier score of 0.22 (the null model Brier score was 0.24). The decision curve analysis showed that the model can be beneficial compared with no model or compared with characterizing all patients as at risk for developing delirium.

CONCLUSION

Algorithms developed with machine learning are a potential tool for refining treatment of at-risk patients. If high-risk patients can be reliably identified, resources can be appropriately directed toward their care. Although the current iteration of SORG should not be relied on for patient care, it suggests potential utility in assessing risk. Further assessment in different populations, made easier by international collaborations and standardization of registries, would be useful in the development of universally valid prediction models. The model can be freely accessed at: https://sorg-apps.shinyapps.io/hipfxdelirium/ .

LEVEL OF EVIDENCE

Level III, therapeutic study.

Can a Bayesian belief network for survival prediction in patients with extremity metastases (PATHFx) be externally validated in an Asian cohort of 356 surgically treated patients?

BACKGROUND AND PURPOSE

Predicted survival may influence the treatment decision for patients with skeletal extremity metastasis, and PATHFx was designed to predict the likelihood of a patient dying in the next 24 months. However, the performance of prediction models could have ethnogeographical variations. We asked if PATHFx generalized well to our Taiwanese cohort consisting of 356 surgically treated patients with extremity metastasis.

PATIENTS AND METHODS

We included 356 patients who underwent surgery for skeletal extremity metastasis in a tertiary center in Taiwan between 2014 and 2019 to validate PATHFx's survival predictions at 6 different time points. Model performance was assessed by concordance index (c-index), calibration analysis, decision curve analysis (DCA), Brier score, and model consistency (MC).

RESULTS

The c-indexes for the 1-, 3-, 6-, 12-, 18-, and 24-month survival estimations were 0.71, 0.66, 0.65, 0.69, 0.68, and 0.67, respectively. The calibration analysis demonstrated positive calibration intercepts for survival predictions at all 6 timepoints, indicating PATHFx tended to underestimate the actual survival. The Brier scores for the 6 models were all less than their respective null model's. DCA demonstrated that only the 6-, 12-, 18-, and 24-month predictions appeared useful for clinical decision-making across a wide range of threshold probabilities. The MC was < 0.9 when the 6- and 12-month models were compared with the 12-month and 18-month models, respectively.

INTERPRETATION

In this Asian cohort, PATHFx's performance was not as encouraging as those of prior validation studies. Clinicians should be cognizant of the potential decline in validity of any tools designed using data outside their particular patient population. Developers of survival prediction tools such as PATHFx might refine their algorithms using data from diverse, contemporary patients that is more reflective of the world's population.

Development of a machine learning model to predict lateral hinge fractures by analyzing patient factors before open wedge high tibial osteotomy

PURPOSE

Several methods have been developed to prevent lateral hinge fractures (LHFs), using only classic statistical models. Machine learning is under the spotlight because of its ability to analyze various weights and model nonlinear relationships. The purpose of this study was to create a machine learning model that predicts LHF with high predictive performance.

METHODS

Data were collected from a total of 439 knees with medial osteoarthritis (OA) treated with Medial open wedge high tibial osteotomy (MOW-HTO) from March 2014 to February 2020. The patient data included age, sex, height, and weight. Preoperative, determined, and modifiable factors were categorized using X-ray and CT data to create ensemble models with better predictive performance. Among the 57 ensemble models, which is the total number of possible combinations with six models, the model with the highest area under curve (AUC) or F1-score was selected as the final ensemble model. Gain feature importance analysis and the Shapley additive explanations (SHAP) feature explanation were performed on the best models.

RESULTS

The ensemble model with the highest AUC was a combination of a light gradient boosting machine (LGBM) and multilayer perceptron (MLP) (AUC = 0.992). The ensemble model with the highest F1-score was the model that combined logistic regression (LR) and MLP (F1-score = 0.765). Distance X was the most predictive feature in the results of both model interpretation analyses.

CONCLUSION

Two types of ensemble models, LGBM with MLP and LR with MLP, were developed as machine learning models to predict LHF with high predictive performance. Using these models, surgeons can identify important features to prevent LHF and establish strategies by adjusting modifiable factors.

STUDY DESIGN

Retrospective cohort study.

A machine learning algorithm for predicting prolonged postoperative opioid prescription after lumbar disc herniation surgery. An external validation study using 1,316 patients from a Taiwanese cohort

BACKGROUND CONTEXT

Preoperative prediction of prolonged postoperative opioid prescription helps identify patients for increased surveillance after surgery. The SORG machine learning model has been developed and successfully tested using 5,413 patients from the United States (US) to predict the risk of prolonged opioid prescription after surgery for lumbar disc herniation. However, external validation is an often-overlooked element in the process of incorporating prediction models in current clinical practice. This cannot be stressed enough in prediction models where medicolegal and cultural differences may play a major role.

PURPOSE

The authors aimed to investigate the generalizability of the US citizens prediction model SORG to a Taiwanese patient cohort.

STUDY DESIGN

Retrospective study at a large academic medical center in Taiwan.

PATIENT SAMPLE

Of 1,316 patients who were 20 years or older undergoing initial operative management for lumbar disc herniation between 2010 and 2018.

OUTCOME MEASURES

The primary outcome of interest was prolonged opioid prescription defined as continuing opioid prescription to at least 90 to 180 days after the first surgery for lumbar disc herniation at our institution.

METHODS

Baseline characteristics were compared between the external validation cohort and the original developmental cohorts. Discrimination (area under the receiver operating characteristic curve and the area under the precision-recall curve), calibration, overall performance (Brier score), and decision curve analysis were used to assess the performance of the SORG ML algorithm in the validation cohort. This study had no funding source or conflict of interests.

RESULTS

Overall, 1,316 patients were identified with sustained postoperative opioid prescription in 41 (3.1%) patients. The validation cohort differed from the development cohort on several variables including 93% of Taiwanese patients receiving NSAIDS preoperatively compared with 22% of US citizens patients, while 30% of Taiwanese patients received opioids versus 25% in the US. Despite these differences, the SORG prediction model retained good discrimination (area under the receiver operating characteristic curve of 0.76 and the area under the precision-recall curve of 0.33) and good overall performance (Brier score of 0.028 compared with null model Brier score of 0.030) while somewhat overestimating the chance of prolonged opioid use (calibration slope of 1.07 and calibration intercept of -0.87). Decision-curve analysis showed the SORG model was suitable for clinical use.

CONCLUSIONS

Despite differences at baseline and a very strict opioid policy, the SORG algorithm for prolonged opioid use after surgery for lumbar disc herniation has good discriminative abilities and good overall performance in a Han Chinese patient group in Taiwan. This freely available digital application can be used to identify high-risk patients and tailor prevention policies for these patients that may mitigate the long-term adverse consequence of opioid dependence: https://sorg-apps.shinyapps.io/lumbardiscopioid/.

Predictive Algorithms in the Diagnosis and Management of Pediatric Hip and Periarticular Infection

➤ Although the criteria of Kocher et al. were an important advancement in our ability to diagnose septic arthritis of the hip early, the changing microbial landscape and availability of advanced imaging have rendered it insufficient for contemporary clinical use.➤ Routine use of magnetic resonance imaging and recognition of disseminated disease have prompted the development of algorithms to predict concurrent osteoarticular infection in cases of septic arthritis and osteomyelitis that were previously assumed to be "isolated."➤ Recent research has attempted to stratify childhood bone and joint infection (BJI) by severity to guide treatment planning. This is valuable, as patients with multifocal disease, more virulent pathogens, and immunocompromise can have longer hospital stays and require multiple surgeries.➤ The increasing prevalence of clinical prediction algorithms in childhood BJI is not completely matched by quality in methodology. Clinicians need to be wary of adopting predictive algorithms prior to robust external validation.

An Evolution Gaining Momentum—The Growing Role of Artificial Intelligence in the Diagnosis and Treatment of Spinal Diseases

In recent years, applications using artificial intelligence have been gaining importance in the diagnosis and treatment of spinal diseases. In our review, we describe the basic features of artificial intelligence which are currently applied in the field of spine diagnosis and treatment, and we provide an orientation of the recent technical developments and their applications. Furthermore, we point out the possible limitations and challenges in dealing with such technological advances. Despite the momentary limitations in practical application, artificial intelligence is gaining ground in the field of spine treatment. As an applying physician, it is therefore necessary to engage with it in order to benefit from those advances in the interest of the patient and to prevent these applications being misused by non-medical partners.

Body Composition Predictors of Adverse Postoperative Events in Patients Undergoing Surgery for Long Bone Metastases

Body composition assessed using opportunistic CT has been recently identified as a predictor of outcome in patients with cancer. The purpose of this study was to determine whether the cross-sectional area (CSA) and the attenuation of abdominal subcutaneous adipose tissue, visceral adipose tissue (VAT), and paraspinous and abdominal muscles are the predictors of length of hospital stay, 30-day postoperative complications, and revision surgery in patients treated for long bone metastases.