Application of Machine Learning Algorithms to Predict Clinically Meaningful Improvement After Arthroscopic Anterior Cruciate Ligament Reconstruction

Several factors predict the achievement of the patient acceptable symptom state and minimal clinically important difference for patient-reported outcome measures following anterior cruciate ligament reconstruction: A systematic review

PURPOSE

To summarize the predictors of the patient acceptable symptom state (PASS), minimal clinically important difference (MCID) and minimal important change (MIC) for patient-reported outcome measures (PROMs) following anterior cruciate ligament reconstruction (ACLR).

METHODS

MEDLINE, PubMed and Embase were searched from inception to 5 January 2024. The authors adhered to PRISMA/R-AMSTAR guidelines, and the Cochrane Handbook for Systematic Reviews of Interventions. Data on statistical associations between predictive factors and PROMs were extracted. Inverse odds ratios (ORs) and confidence intervals (reverse group comparison) were calculated when appropriate to ensure comparative consistency.

RESULTS

Thirteen studies comprising 21,235 patients (48.1% female) were included (mean age 29.3 years). Eight studies comprising 3857 patients identified predictors of PASS, including lateral extra-articular tenodesis (LET) (OR = 11.08, p = 0.01), hamstring tendon (HT) autografts (OR range: 2.02-2.63, p ≤ 0.011), age over 30 (OR range: 1.37-2.28, p ≤ 0.02), male sex (OR range: 1.03-1.32, p ≤ 0.01) and higher pre-operative PROMs (OR range: 1.04-1.21). Eight studies comprising 18,069 patients identified negative predictors of MCID or MIC, including female sex (OR = 0.93, p = 0.034), absence of HT autografts (OR = 0.70, p < 0.0001), higher pre-operative PROMs (OR = 0.76-0.84, p ≤ 0.01), meniscectomy (OR = 0.67, p = 0.014) and collision sports (OR = 0.02-0.60, p ≤ 0.05).

CONCLUSION

Higher pre-operative PROMs, age over 30, male sex, LETs and HT autografts predicted PASS achievement. Lower pre-operative PROMs, male sex, non-collision sports, and lack of meniscectomies predicted MCID/MIC achievement. This review provides a comprehensive understanding of the predictors of clinically significant post-ACLR outcomes, thus improving clinical decision-making and the management of patient expectations.

LEVEL OF EVIDENCE

Level IV.

Artificial Intelligence Models Are Limited in Predicting Clinical Outcomes Following Hip Arthroscopy: A Systematic Review

BACKGROUND

Hip arthroscopy has seen a significant surge in utilization, but complications remain, and optimal functional outcomes are not guaranteed. Artificial intelligence (AI) has emerged as an effective supportive decision-making tool for surgeons. The purpose of this systematic review was to characterize the outcomes, performance, and validity (generalizability) of AI-based prediction models for hip arthroscopy in current literature.

METHODS

Two reviewers independently completed structured searches using PubMed/MEDLINE and Embase databases on August 10, 2022. The search query used the terms as follows: (artificial intelligence OR machine learning OR deep learning) AND (hip arthroscopy). Studies that investigated AI-based risk prediction models in hip arthroscopy were included. The primary outcomes of interest were the variable(s) predicted by the models, best model performance achieved (primarily based on area under the curve, but also accuracy, etc), and whether the model(s) had been externally validated (generalizable).

RESULTS

Seventy-seven studies were identified from the primary search. Thirteen studies were included in the final analysis. Six studies (n = 6,568) applied AI for predicting the achievement of minimal clinically important difference for various patient-reported outcome measures such as the visual analog scale and the International Hip Outcome Tool 12-Item Questionnaire, with area under a receiver-operating characteristic curve (AUC) values ranging from 0.572 to 0.94. Three studies used AI for predicting repeat hip surgery with AUC values between 0.67 and 0.848. Four studies focused on predicting other risks, such as prolonged postoperative opioid use, with AUC values ranging from 0.71 to 0.76. None of the 13 studies assessed the generalizability of their models through external validation.

CONCLUSION

AI is being investigated for predicting clinical outcomes after hip arthroscopy. However, the performance of AI models varies widely, with AUC values ranging from 0.572 to 0.94. Critically, none of the models have undergone external validation, limiting their clinical applicability. Further research is needed to improve model performance and ensure generalizability before these tools can be reliably integrated into patient care.

LEVEL OF EVIDENCE

Level IV. See Instructions for Authors for a complete description of levels of evidence.

Ensemble Algorithm for Risk Prediction of Clinical Failure After Anterior Cruciate Ligament Reconstruction

Background

Patient-specific risk profiles of clinical failure after anterior cruciate ligament reconstruction (ACLR) are meaningful for preoperative surgical planning and postoperative rehabilitation guidance.

Purpose

To create an ensemble algorithm machine learning (ML) model and ML-based web-based tool that can predict the patient-specific risk of clinical failure after ACLR.

Study Design

Cohort study; Level of evidence, 3.

Methods

Included were 432 patients (mean age, 26.8 ± 8.4 years; 74.1% male) who underwent anatomic double-bundle ACLR with hamstring tendon autograft between January 2010 and February 2019. The primary outcome was the probability of clinical failure at a minimum 2-year follow-up. The authors included 24 independent variables for feature selection and model development. The data set was split randomly into training sets (75%) and test sets (25%). Models were built using 4 ML algorithms: extreme gradient boosting, random forest, light gradient boosting machine, and adaptive boosting. In addition, a weighted-average voting (WAV) ensemble model was constructed using the ensemble-voting technique to predict clinical failure after ACLR. Concordance (area under the receiver operating characteristic curve [AUC]), calibration, and decision curve analysis were used to evaluate predictive performances of the 5 models.

Results

Clinical failure occurred in 73 of the 432 patients (16.9%). The 8 most important predictors for clinical failure were follow-up period, high-grade preoperative knee laxity, time from injury to ACLR, participation in competitive sports, posterior tibial slope, graft diameter, age at surgery, and medial meniscus resection. The WAV ensemble algorithm achieved the best predictive performance based on concordance (AUC, 0.9139), calibration (calibration intercept, -0.1806; calibration slope, 1.2794; Brier score, 0.0888), and decision curve analysis (greatest net benefits) and was used to develop an web-based application to predict a patient's clinical failure risk of ACLR.

Conclusion

The WAV ensemble algorithm was able to accurately predict patient-specific risk of clinical failure after ACLR. Clinicians and patients can use the web-based application during preoperative consultation to understand individual prediction outcomes.

Mobile App for Enhanced Anterior Cruciate Ligament (ACL) Assessment in Conscious Subjects: "Pivot-Shift Meter"

Anterior cruciate ligament (ACL) instability poses a considerable challenge in traumatology and orthopedic medicine, demanding precise diagnostics for optimal treatment. The pivot-shift test, a pivotal assessment tool, relies on subjective interpretation, emphasizing the need for supplementary imaging. This study addresses this limitation by introducing a machine learning classification algorithm integrated into a mobile application, leveraging smartphones' built-in inertial sensors for dynamic rotational stability assessment during knee examinations. Orthopedic specialists conducted knee evaluations on a cohort of 52 subjects, yielding valuable insights. Quantitative analyses, employing the Intraclass Correlation Coefficient (ICC), demonstrated robust agreement in both intraobserver and interobserver assessments. Specifically, ICC values of 0.94 reflected strong concordance in the timing between maneuvers, while signal amplitude exhibited consistency, with the ICC ranging from 0.71 to 0.66. The introduced machine learning algorithms proved effective, accurately classifying 90% of cases exhibiting joint hypermobility. These quantifiable results underscore the algorithm's reliability in assessing knee stability. This study emphasizes the practicality and effectiveness of implementing machine learning algorithms within a mobile application, showcasing its potential as a valuable tool for categorizing signals captured by smartphone inertial sensors during the pivot-shift test.

Artificial Intelligence for Clinically Meaningful Outcome Prediction in Orthopedic Research: Current Applications and Limitations

PURPOSE OF REVIEW

Patient-reported outcome measures (PROM) play a critical role in evaluating the success of treatment interventions for musculoskeletal conditions. However, predicting which patients will benefit from treatment interventions is complex and influenced by a multitude of factors. Artificial intelligence (AI) may better anticipate the propensity to achieve clinically meaningful outcomes through leveraging complex predictive analytics that allow for personalized medicine. This article provides a contemporary review of current applications of AI developed to predict clinically significant outcome (CSO) achievement after musculoskeletal treatment interventions.

RECENT FINDINGS

The highest volume of literature exists in the subspecialties of total joint arthroplasty, spine, and sports medicine, with only three studies identified in the remaining orthopedic subspecialties combined. Performance is widely variable across models, with most studies only reporting discrimination as a performance metric. Given the complexity inherent in predictive modeling for this task, including data availability, data handling, model architecture, and outcome selection, studies vary widely in their methodology and results. Importantly, the majority of studies have not been externally validated or demonstrate important methodological limitations, precluding their implementation into clinical settings. A substantial body of literature has accumulated demonstrating variable internal validity, limited scope, and low potential for clinical deployment. The majority of studies attempt to predict the MCID-the lowest bar of clinical achievement. Though a small proportion of models demonstrate promise and highlight the utility of AI, important methodological limitations need to be addressed moving forward to leverage AI-based applications for clinical deployment.

Estimating the Thumb Rotation Angle by Using a Tablet Device With a Posture Estimation Artificial Intelligence Model

MediaPipe Hand (MediaPipe) is an artificial intelligence (AI)-based pose estimation library. In this study, MediaPipe was combined with four machine learning (ML) models to estimate the rotation angle of the thumb. Videos of the right hands of 15 healthy volunteers were recorded and processed into 9000 images. The rotation angle of the thumb (defined as angle θ from the palmar plane, which is defined as 0°) was measured using an angle measuring device, expressed in a radian system. Angle θ was then estimated by the ML model by using parameters calculated from the hand coordinates detected by MediaPipe. The linear regression model showed a root mean square error (RMSE) of 12.23, a mean absolute error (MAE) of 9.9, and a correlation coefficient of 0.91. The ElasticNet model showed an RMSE of 12.23, an MAE of 9.95, and a correlation coefficient of 0.91; the support vector machine (SVM) model showed an RMSE of 4.7, an MAE of 2.5, and a correlation coefficient of 0.99. The LightGBM model achieved high values: an RMSE of 4.58, an MAE of 2.62, and a correlation coefficient of 0.99. Based on these findings, we concluded that the thumb rotation angle can be estimated with high accuracy by combining MediaPipe and ML.

Challenges and Opportunities for the Use of Patient-Reported Outcome Measures in Orthopaedic Pediatric and Sports Medicine Surgery

Patient-reported outcome measures (PROMs) are essential tools in assessing treatment response, informing clinical decision making, driving healthcare policy, and providing important prognostic data regarding patient health status change. These tools become essential in orthopaedic disciplines, such as pediatrics and sports medicine, given the diversity of patient populations and procedures. However, the creation and routine administration of standard PROMs alone do not suffice to appropriately facilitate the aforementioned functions. Indeed, both the interpretation and optimal application of PROMs are essential to provide to achieve greatest clinical benefit. Contemporary developments and technologies surrounding PROMs may help augment this benefit, including the application of artificial intelligence, novel PROM structure with improved interpretability and validity, and PROM delivery methods that provide increased access to patients resulting in greater compliance and data acquisition yields. Despite these exciting innovations, several challenges remain in this realm that must be addressed to continue to advance the clinical usefulness and subsequent benefit of PROMs. This review will highlight the opportunities and challenges surrounding contemporary PROM use in the orthopaedic subspecialties of pediatrics and sports medicine.

Machine learning can accurately predict risk factors for all-cause reoperation after ACLR: creating a clinical tool to improve patient counseling and outcomes

PURPOSE

Identifying predictive factors for all-cause reoperation after anterior cruciate ligament reconstruction could inform clinical decision making and improve risk mitigation. The primary purposes of this study are to (1) determine the incidence of all-cause reoperation after anterior cruciate ligament reconstruction, (2) identify predictors of reoperation after anterior cruciate ligament reconstruction using machine learning methodology, and (3) compare the predictive capacity of the machine learning methods to that of traditional logistic regression.

METHODS

A longitudinal geographical database was utilized to identify patients with a diagnosis of new anterior cruciate ligament injury. Eight machine learning models were appraised on their ability to predict all-cause reoperation after anterior cruciate ligament reconstruction. Model performance was evaluated via area under the receiver operating characteristics curve. To explore modeling interpretability and radiomic feature influence on the predictions, we utilized a game-theory-based method through SHapley Additive exPlanations.

RESULTS

A total of 1400 patients underwent anterior cruciate ligament reconstruction with a mean postoperative follow-up of 9 years. Two-hundred and eighteen (16%) patients experienced a reoperation after anterior cruciate ligament reconstruction, of which 6% of these were revision ACL reconstruction. SHapley Additive exPlanations plots identified the following risk factors as predictive for all-cause reoperation: diagnosis of systemic inflammatory disease, distal tear location, concomitant medial collateral ligament repair, higher visual analog scale pain score prior to surgery, hamstring autograft, tibial fixation via radial expansion device, younger age at initial injury, and concomitant meniscal repair. Pertinent negatives, when compared to previous studies, included sex and timing of surgery. XGBoost was the best-performing model (area under the receiver operating characteristics curve of 0.77) and outperformed logistic regression in this regard.

CONCLUSIONS

All-cause reoperation after anterior cruciate ligament reconstruction occurred at a rate of 16%. Machine learning models outperformed traditional statistics and identified diagnosis of systemic inflammatory disease, distal tear location, concomitant medial collateral ligament repair, higher visual analog scale pain score prior to surgery, hamstring autograft, tibial fixation via radial expansion device, younger age at initial injury, and concomitant meniscal repair as predictive risk factors for reoperation. Pertinent negatives, when compared to previous studies, included sex and timing of surgery. These models will allow surgeons to tabulate individualized risk for future reoperation for patients undergoing anterior cruciate ligament reconstruction.

LEVEL OF EVIDENCE

III.

Technology-assisted anterior cruciate ligament reconstruction improves tunnel placement but leads to no change in clinical outcomes: a systematic review and meta-analysis

PURPOSE

To investigate the effect of technology-assisted Anterior Cruciate Ligament Reconstruction (ACLR) on post-operative clinical outcomes and tunnel placement compared to conventional arthroscopic ACLR.

METHODS

CENTRAL, MEDLINE, and Embase were searched from January 2000 to November 17, 2022. Articles were included if there was intraoperative use of computer-assisted navigation, robotics, diagnostic imaging, computer simulations, or 3D printing (3DP). Two reviewers searched, screened, and evaluated the included studies for data quality. Data were abstracted using descriptive statistics and pooled using relative risk ratios (RR) or mean differences (MD), both with 95% confidence intervals (CI), where appropriate.

RESULTS

Eleven studies were included with total 775 patients and majority male participants (70.7%). Ages ranged from 14 to 54 years (391 patients) and follow-up ranged from 12 to 60 months (775 patients). Subjective International Knee Documentation Committee (IKDC) scores increased in the technology-assisted surgery group (473 patients; P = 0.02; MD 1.97, 95% CI 0.27 to 3.66). There was no difference in objective IKDC scores (447 patients; RR 1.02, 95% CI 0.98 to 1.06), Lysholm scores (199 patients; MD 1.14, 95% CI - 1.03 to 3.30) or negative pivot-shift tests (278 patients; RR 1.07, 95% CI 0.97 to 1.18) between the two groups. When using technology-assisted surgery, 6 (351 patients) of 8 (451 patients) studies reported more accurate femoral tunnel placement and 6 (321 patients) of 10 (561 patients) studies reported more accurate tibial tunnel placement in at least one measure. One study (209 patients) demonstrated a significant increase in cost associated with use of computer-assisted navigation (mean 1158€) versus conventional surgery (mean 704€). Of the two studies using 3DP templates, production costs ranging from $10 to $42 USD were cited. There was no difference in adverse events between the two groups.

CONCLUSION

Clinical outcomes do not differ between technology-assisted surgery and conventional surgery. Computer-assisted navigation is more expensive and time consuming while 3DP is inexpensive and does not lead to greater operating times. ACLR tunnels can be more accurately located in radiologically ideal places by using technology, but anatomic placement is still undetermined because of variability and inaccuracy of the evaluation systems utilized.

LEVEL OF EVIDENCE

Level III.

Propensity for Clinically Meaningful Improvement and Surgical Failure After Anterior Cruciate Ligament Repair

Background

Primary repair of the anterior cruciate ligament (ACL) confers an alternative to ACL reconstruction in appropriately selected patients.

Purpose

To prospectively assess survivorship and to define the clinically meaningful outcomes after ACL repair.

Study Design

Case series; Level of evidence, 4.

Methods

Included were consecutive patients with Sherman grade 1-2 tears who underwent primary ACL repair with or without suture augmentation between 2017 and 2019. Patient-reported outcomes (Lysholm, Tegner, International Knee Documentation Committee, Western Ontario and McMaster Universities Osteoarthritis Index, and Knee injury and Osteoarthritis Outcome Score [KOOS] subscales) were collected preoperatively and at 6 months, 1 year, and 2 years postoperatively. The minimal clinically important difference (MCID) was calculated using a distribution-based method, whereas the Patient Acceptable Symptom State (PASS) and substantial clinical benefit (SCB) were calculated using an anchor-based method. Plain radiographs and magnetic resonance imaging (MRI) were obtained at 6 months, 1 year, and 2 years postoperatively.

Results

A total of 120 patients were included. The overall failure rate was 11.3% at 2 years postoperatively. Changes in outcome scores required to achieve the MCID ranged between 5.1 and 14.3 at 6 months, 4.6 and 8.4 at 1 year, and 4.7 and 11.9 at 2 years postoperatively. Thresholds for PASS achievement ranged between 62.5 and 89 at 6 months, 75 and 89 at 1 year, and 78.6 and 93.2 at 2 years postoperatively. Threshold scores (absolute/change based) for achieving the SCB ranged between 82.8 and 96.4/17.7 and 40.1 at 6 months, between 94.7 and 100/23 and 45 at 1 year, and between 95.3 and 100/29.4 and 45 at 2 years. More patients achieved the MCID and PASS at 1 year compared with 6 months and 2 years. For SCB, this trend was also observed for non-KOOS outcomes, while for KOOS subdomains, more patients achieved the SCB at 2 years. High-intensity signal of the ACL repair (odds ratio [OR], 31.7 [95% CI, 1.5-73.4]; P = .030) and bone contusions on MRI (OR, 4.2 [95% CI, 1.7-25.2]; P = .041) at 1 year postoperatively were independently associated with increased risk of ACL repair failure.

Conclusion

The rate of clinically meaningful outcome improvement was high early after ACL repair, with the greatest proportion of patients achieving the MCID, PASS, and SCB at 1 year postoperatively. Bone contusions involving the posterolateral tibia and lateral femoral condyle as well as high repair signal intensity at 1 year postoperatively were independent predictors of failure at 2 years postoperatively.

Predicting the Objective and Subjective Clinical Outcomes of Anterior Cruciate Ligament Reconstruction: A Machine Learning Analysis of 432 Patients

BACKGROUND

Sports levels, baseline patient-reported outcome measures (PROMs), and surgical procedures are correlated with the outcomes of anterior cruciate ligament reconstruction (ACLR). Machine learning may be superior to conventional statistical methods in making repeatable and accurate predictions.

PURPOSE

To identify the best-performing machine learning models for predicting the objective and subjective clinical outcomes of ACLR and to determine the most important predictors.

STUDY DESIGN

Case-control study; Level of evidence, 3.

METHODS

A total of 432 patients who underwent anatomic double-bundle ACLR with hamstring tendon autograft between January 2010 and February 2019 were included in the machine learning analysis. A total of 15 predictive variables and 6 outcome variables were selected to validate the logistic regression, Gaussian naïve Bayes machine, random forest, Extreme Gradient Boosting (XGBoost), isotonically calibrated XGBoost, and sigmoid calibrated XGBoost models. For each clinical outcome, the best-performing model was determined using the area under the receiver operating characteristic curve (AUC), whereas the importance and direction of each predictive variable were demonstrated in a Shapley Additive Explanations summary plot.

RESULTS

The AUC and accuracy of the best-performing model, respectively, were 0.944 (excellent) and 98.6% for graft failure; 0.920 (excellent) and 91.4% for residual laxity; 0.930 (excellent) and 91.0% for failure to achieve the minimal clinically important difference (MCID) of the Lysholm score; 0.942 (excellent) and 95.1% for failure to achieve the MCID of the International Knee Documentation Committee (IKDC) score; 0.773 (fair) and 70.5% for return to preinjury sports; and 0.777 (fair) and 69.2% for return to pivoting sports. Medial meniscal resection, participation in competitive sports, and steep posterior tibial slope were top predictors of graft failure, whereas high-grade preoperative knee laxity, long follow-up period, and participation in competitive sports were top predictors of residual laxity. High preoperative Lysholm and IKDC scores were highly predictive of not achieving the MCIDs of PROMs. Young age, male sex, high preoperative IKDC score, and large graft diameter were important predictors of return to preinjury or pivoting sports.

CONCLUSION

Machine learning analysis can provide reliable predictions for the objective and subjective clinical outcomes (graft failure, residual laxity, PROMs, and return to sports) of ACLR. Patient-specific evaluation and decision making are recommended before and after surgery.

Machine Learning Algorithms Predict Achievement of Clinically Significant Outcomes After Orthopaedic Surgery: A Systematic Review

PURPOSE

To determine what subspecialties have applied machine learning (ML) to predict clinically significant outcomes (CSOs) within orthopaedic surgery and to determine whether the performance of these models was acceptable through assessing discrimination and other ML metrics where reported.

METHODS

The PubMed, EMBASE, and Cochrane Central Register of Controlled Trials databases were queried for articles that used ML to predict achievement of the minimal clinically important difference (MCID), patient acceptable symptomatic state (PASS), or substantial clinical benefit (SCB) after orthopaedic surgical procedures. Data pertaining to demographic characteristics, subspecialty, specific ML algorithms, and algorithm performance were analyzed.

RESULTS

Eighteen articles met the inclusion criteria. Seventeen studies developed novel algorithms, whereas one study externally validated an established algorithm. All studies used ML to predict MCID achievement, whereas 3 (16.7%) predicted SCB achievement and none predicted PASS achievement. Of the studies, 7 (38.9%) concerned outcomes after spine surgery; 6 (33.3%), after sports medicine surgery; 3 (16.7%), after total joint arthroplasty (TJA); and 2 (11.1%), after shoulder arthroplasty. No studies were found regarding trauma, hand, elbow, pediatric, or foot and ankle surgery. In spine surgery, concordance statistics (C-statistics) ranged from 0.65 to 0.92; in hip arthroscopy, 0.51 to 0.94; in TJA, 0.63 to 0.89; and in shoulder arthroplasty, 0.70 to 0.95. Most studies reported C-statistics at the upper end of these ranges, although populations were heterogeneous.

CONCLUSIONS

Currently available ML algorithms can discriminate the propensity to achieve CSOs using the MCID after spine, TJA, sports medicine, and shoulder surgery with a fair to good performance as evidenced by C-statistics ranging from 0.6 to 0.95 in most analyses. Less evidence is available on the ability of ML to predict achievement of SCB, and no evidence is available for achievement of the PASS. Such algorithms may augment shared decision-making practices and allow clinicians to provide more appropriate patient expectations using individualized risk assessments. However, these studies remain limited by variable reporting of performance metrics, CSO quantification methods, and adherence to predictive modeling guidelines, as well as limited external validation.

LEVEL OF EVIDENCE

Level III, systematic review of Level III studies.