Interpretation and reporting of predictive or diagnostic machine-learning research in Trauma & Orthopaedics

The Clinical Practice Integration of Artificial Intelligence (CPI-AI) framework

Despite the vast quantities of published artificial intelligence (AI) algorithms that target trauma and orthopaedic applications, very few progress to inform clinical practice. One key reason for this is the lack of a clear pathway from development to deployment. In order to assist with this process, we have developed the Clinical Practice Integration of Artificial Intelligence (CPI-AI) framework - a five-stage approach to the clinical practice adoption of AI in the setting of trauma and orthopaedics, based on the IDEAL principles (https://www.ideal-collaboration.net/). Adherence to the framework would provide a robust evidence-based mechanism for developing trust in AI applications, where the underlying algorithms are unlikely to be fully understood by clinical teams.

Use of natural language processing techniques to predict patient selection for total hip and knee arthroplasty from radiology reports

Aims

To examine whether natural language processing (NLP) using a clinically based large language model (LLM) could be used to predict patient selection for total hip or total knee arthroplasty (THA/TKA) from routinely available free-text radiology reports.

Methods

Data pre-processing and analyses were conducted according to the Artificial intelligence to Revolutionize the patient Care pathway in Hip and knEe aRthroplastY (ARCHERY) project protocol. This included use of de-identified Scottish regional clinical data of patients referred for consideration of THA/TKA, held in a secure data environment designed for artificial intelligence (AI) inference. Only preoperative radiology reports were included. NLP algorithms were based on the freely available GatorTron model, a LLM trained on over 82 billion words of de-identified clinical text. Two inference tasks were performed: assessment after model-fine tuning (50 Epochs and three cycles of k-fold cross validation), and external validation.

Results

For THA, there were 5,558 patient radiology reports included, of which 4,137 were used for model training and testing, and 1,421 for external validation. Following training, model performance demonstrated average (mean across three folds) accuracy, F1 score, and area under the receiver operating curve (AUROC) values of 0.850 (95% confidence interval (CI) 0.833 to 0.867), 0.813 (95% CI 0.785 to 0.841), and 0.847 (95% CI 0.822 to 0.872), respectively. For TKA, 7,457 patient radiology reports were included, with 3,478 used for model training and testing, and 3,152 for external validation. Performance metrics included accuracy, F1 score, and AUROC values of 0.757 (95% CI 0.702 to 0.811), 0.543 (95% CI 0.479 to 0.607), and 0.717 (95% CI 0.657 to 0.778) respectively. There was a notable deterioration in performance on external validation in both cohorts.

Conclusion

The use of routinely available preoperative radiology reports provides promising potential to help screen suitable candidates for THA, but not for TKA. The external validation results demonstrate the importance of further model testing and training when confronted with new clinical cohorts.

Predicting Functional Outcomes of Total Hip Arthroplasty Using Machine Learning: A Systematic Review

The aim of this review was to assess the reliability of machine learning (ML) techniques to predict the functional outcome of total hip arthroplasty. The literature search was performed up to October 2023, using MEDLINE/PubMed, Embase, Web of Science, and NIH Clinical Trials. Level I to IV evidence was included. Seven studies were identified that included 44,121 patients. The time to follow-up varied from 3 months to more than 2 years. Each study employed one to six ML techniques. The best-performing models were for health-related quality of life (HRQoL) outcomes, with an area under the curve (AUC) of more than 84%. In contrast, predicting the outcome of hip-specific measures was less reliable, with an AUC of between 71% to 87%. Random forest and neural networks were generally the best-performing models. Three studies compared the reliability of ML with traditional regression analysis: one found in favour of ML, one was not clear and stated regression closely followed the best-performing ML model, and one showed a similar AUC for HRQoL outcomes but did show a greater reliability for ML to predict a clinically significant change in the hip-specific function. ML offers acceptable-to-excellent discrimination of predicting functional outcomes and may have a marginal advantage over traditional regression analysis, especially in relation to hip-specific hip functional outcomes.

Unveiling the potential of artificial intelligence in orthopaedic surgery

Artificial intelligence is paving the way in contemporary medical advances, with the potential to revolutionise orthopaedic surgical care. By harnessing the power of complex algorithms, artificial intelligence yields outputs that have diverse applications including, but not limited to, identifying implants, diagnostic imaging for fracture and tumour recognition, prognostic tools through the use of electronic medical records, assessing arthroplasty outcomes, length of hospital stay and economic costs, monitoring the progress of functional rehabilitation, and innovative surgical training via simulation. However, amid the promising potential and enthusiasm surrounding artificial intelligence, clinicians should understand its limitations, and caution is needed before artificial intelligence-driven tools are introduced to clinical practice.

Understanding the use of artificial intelligence for implant analysis in total joint arthroplasty: a systematic review

INTRODUCTION

In recent years, there has been a significant increase in the development of artificial intelligence (AI) algorithms aimed at reviewing radiographs after total joint arthroplasty (TJA). This disruptive technology is particularly promising in the context of preoperative planning for revision TJA. Yet, the efficacy of AI algorithms regarding TJA implant analysis has not been examined comprehensively.

METHODS

PubMed, EBSCO, and Google Scholar electronic databases were utilized to identify all studies evaluating AI algorithms related to TJA implant analysis between 1 January 2000, and 27 February 2023 (PROSPERO study protocol registration: CRD42023403497). The mean methodological index for non-randomized studies score was 20.4 ± 0.6. We reported the accuracy, sensitivity, specificity, positive predictive value, and area under the curve (AUC) for the performance of each outcome measure.

RESULTS

Our initial search yielded 374 articles, and a total of 20 studies with three main use cases were included. Sixteen studies analyzed implant identification, two addressed implant failure, and two addressed implant measurements. Each use case had a median AUC and accuracy above 0.90 and 90%, respectively, indicative of a well-performing AI algorithm. Most studies failed to include explainability methods and conduct external validity testing.

CONCLUSION

These findings highlight the promising role of AI in recognizing implants in TJA. Preliminary studies have shown strong performance in implant identification, implant failure, and accurately measuring implant dimensions. Future research should follow a standardized guideline to develop and train models and place a strong emphasis on transparency and clarity in reporting results.

LEVEL OF EVIDENCE

Level III.

Machine learning models to predict surgical case duration compared to current industry standards: scoping review

BACKGROUND

Surgical waiting lists have risen dramatically across the UK as a result of the COVID-19 pandemic. The effective use of operating theatres by optimal scheduling could help mitigate this, but this requires accurate case duration predictions. Current standards for predicting the duration of surgery are inaccurate. Artificial intelligence (AI) offers the potential for greater accuracy in predicting surgical case duration. This study aimed to investigate whether there is evidence to support that AI is more accurate than current industry standards at predicting surgical case duration, with a secondary aim of analysing whether the implementation of the models used produced efficiency savings.

METHOD

PubMed, Embase, and MEDLINE libraries were searched through to July 2023 to identify appropriate articles. PRISMA extension for scoping reviews and the Arksey and O'Malley framework were followed. Study quality was assessed using a modified version of the reporting guidelines for surgical AI papers by Farrow et al. Algorithm performance was reported using evaluation metrics.

RESULTS

The search identified 2593 articles: 14 were suitable for inclusion and 13 reported on the accuracy of AI algorithms against industry standards, with seven demonstrating a statistically significant improvement in prediction accuracy (P < 0.05). The larger studies demonstrated the superiority of neural networks over other machine learning techniques. Efficiency savings were identified in a RCT. Significant methodological limitations were identified across most studies.

CONCLUSION

The studies suggest that machine learning and deep learning models are more accurate at predicting the duration of surgery; however, further research is required to determine the best way to implement this technology.

Acceptance and understanding of artificial intelligence in medical research among orthopaedic surgeons

Aims

The principles of evidence-based medicine (EBM) are the foundation of modern medical practice. Surgeons are familiar with the commonly used statistical techniques to test hypotheses, summarize findings, and provide answers within a specified range of probability. Based on this knowledge, they are able to critically evaluate research before deciding whether or not to adopt the findings into practice. Recently, there has been an increased use of artificial intelligence (AI) to analyze information and derive findings in orthopaedic research. These techniques use a set of statistical tools that are increasingly complex and may be unfamiliar to the orthopaedic surgeon. It is unclear if this shift towards less familiar techniques is widely accepted in the orthopaedic community. This study aimed to provide an exploration of understanding and acceptance of AI use in research among orthopaedic surgeons.

Methods

Semi-structured in-depth interviews were carried out on a sample of 12 orthopaedic surgeons. Inductive thematic analysis was used to identify key themes.

Results

The four intersecting themes identified were: 1) validity in traditional research, 2) confusion around the definition of AI, 3) an inability to validate AI research, and 4) cautious optimism about AI research. Underpinning these themes is the notion of a validity heuristic that is strongly rooted in traditional research teaching and embedded in medical and surgical training.

Conclusion

Research involving AI sometimes challenges the accepted traditional evidence-based framework. This can give rise to confusion among orthopaedic surgeons, who may be unable to confidently validate findings. In our study, the impact of this was mediated by cautious optimism based on an ingrained validity heuristic that orthopaedic surgeons develop through their medical training. Adding to this, the integration of AI into everyday life works to reduce suspicion and aid acceptance.

Automatic detection of early osteonecrosis of the femoral head from various hip pathologies using deep convolutional neural network: a multi-centre study

PURPOSE

The aim of this study was to develop a deep convolutional neural network (DCNN) for detecting early osteonecrosis of the femoral head (ONFH) from various hip pathologies and evaluate the feasibility of its application.

METHODS

We retrospectively reviewed and annotated hip magnetic resonance imaging (MRI) of ONFH patients from four participated institutions and constructed a multi-centre dataset to develop the DCNN system. The diagnostic performance of the DCNN in the internal and external test datasets was calculated, including area under the receiver operating characteristic curve (AUROC), accuracy, precision, recall, and F1 score, and gradient-weighted class activation mapping (Grad-CAM) technique was used to visualize its decision-making process. In addition, a human-machine comparison trial was performed.

RESULTS

Overall, 11,730 hip MRI segments from 794 participants were used to develop and optimize the DCNN system. The AUROC, accuracy, and precision of the DCNN in internal test dataset were 0.97 (95% CI, 0.93-1.00), 96.6% (95% CI: 93.0-100%), and 97.6% (95% CI: 94.6-100%), and in external test dataset, they were 0.95 (95% CI, 0.91- 0.99), 95.2% (95% CI, 91.1-99.4%), and 95.7% (95% CI, 91.7-99.7%). Compared with attending orthopaedic surgeons, the DCNN showed superior diagnostic performance. The Grad-CAM demonstrated that the DCNN placed focus on the necrotic region.

CONCLUSION

Compared with clinician-led diagnoses, the developed DCNN system is more accurate in diagnosing early ONFH, avoiding empirical dependence and inter-reader variability. Our findings support the integration of deep learning systems into real clinical settings to assist orthopaedic surgeons in diagnosing early ONFH.

Artificial intelligence in orthopaedic surgery

The use of artificial intelligence (AI) is rapidly growing across many domains, of which the medical field is no exception. AI is an umbrella term defining the practical application of algorithms to generate useful output, without the need of human cognition. Owing to the expanding volume of patient information collected, known as 'big data', AI is showing promise as a useful tool in healthcare research and across all aspects of patient care pathways. Practical applications in orthopaedic surgery include: diagnostics, such as fracture recognition and tumour detection; predictive models of clinical and patient-reported outcome measures, such as calculating mortality rates and length of hospital stay; and real-time rehabilitation monitoring and surgical training. However, clinicians should remain cognizant of AI's limitations, as the development of robust reporting and validation frameworks is of paramount importance to prevent avoidable errors and biases. The aim of this review article is to provide a comprehensive understanding of AI and its subfields, as well as to delineate its existing clinical applications in trauma and orthopaedic surgery. Furthermore, this narrative review expands upon the limitations of AI and future direction.

A role for artificial intelligence applications inside and outside of the operating theatre: a review of contemporary use associated with total knee arthroplasty

BACKGROUND

Artificial intelligence (AI) has become involved in many aspects of everyday life, from voice-activated virtual assistants built into smartphones to global online search engines. Similarly, many areas of modern medicine have found ways to incorporate such technologies into mainstream practice. Despite the enthusiasm, robust evidence to support the utility of AI in contemporary total knee arthroplasty (TKA) remains limited. The purpose of this review was to provide an up-to-date summary of the use of AI in TKA and to explore its current and future value.

METHODS

Initially, a structured systematic review of the literature was carried out, following PRISMA search principles, with the aim of summarising the understanding of the field and identifying clinical and knowledge gaps.

RESULTS

A limited body of published work exists in this area. Much of the available literature is of poor methodological quality and many published studies could be best described as "demonstration of concepts" rather than "proof of concepts". There exists almost no independent validation of reported findings away from designer/host sites, and the extrapolation of key results to general orthopaedic sites is limited.

CONCLUSION

While AI has certainly shown value in a small number of specific TKA-associated applications, the majority to date have focused on risk, cost and outcome prediction, rather than surgical care, per se. Extensive future work is needed to demonstrate external validity and reliability in non-designer settings. Well-performed studies are warranted to ensure that the scientific evidence base supporting the use of AI in knee arthroplasty matches the global hype.

Can an artificial intelligence powered software reliably assess pelvic radiographs?

PURPOSE

Despite advances of three-dimensional imaging pelvic radiographs remain the cornerstone in the evaluation of the hip joint. However, large inter- and intra-rater variabilities were reported due to subjective landmark setting. Artificial intelligence (AI)-powered software applications could improve the reproducibility of pelvic radiograph evaluation by providing standardized measurements. The aim of this study was to evaluate the reliability and agreement of a newly developed AI algorithm for the evaluation of pelvic radiographs.

METHODS

Three-hundred pelvic radiographs from 280 patients with different degrees of acetabular coverage and osteoarthritis (Tönnis Grade 0 to 3) were evaluated. Reliability and agreement between manual measurements and the outputs of the AI software were assessed for the lateral-center-edge (LCE) angle, neck-shaft angle, sharp angle, acetabular index, as well as the femoral head extrusion index.

RESULTS

The AI software provided reliable results in 94.3% (283/300). The ICC values ranged between 0.73 for the Acetabular Index to 0.80 for the LCE Angle. Agreement between readers and AI outputs, given by the standard error of measurement (SEM), was good for hips with normal coverage (LCE-SEM: 3.4°) and no osteoarthritis (LCE-SEM: 3.3°) and worse for hips with undercoverage (LCE-SEM: 5.2°) or severe osteoarthritis (LCE-SEM: 5.1°).

CONCLUSION

AI-powered applications are a reliable alternative to manual evaluation of pelvic radiographs. While being accurate for patients with normal acetabular coverage and mild signs of osteoarthritis, it needs improvement in the evaluation of patients with hip dysplasia and severe osteoarthritis.

Camouflage uncovered: what should happen next?

Recent publications have drawn attention to the fact that some brands of joint replacement may contain variants which perform significantly worse (or better) than their 'siblings'. As a result, the National Joint Registry has performed much more detailed analysis on the larger families of knee arthroplasties in order to identify exactly where these differences may be present and may hitherto have remained hidden. The analysis of the Nexgen knee arthroplasty brand identified that some posterior-stabilized combinations have particularly high revision rates for aseptic loosening of the tibia, and consequently a medical device recall has been issued for the Nexgen 'option' tibial component which was implicated. More elaborate signal detection is required in order to identify such variation in results in a routine fashion if patients are to be protected from such variation in outcomes between closely related implant types.

SMART choice (knee) tool: a patient-focused predictive model to predict improvement in health-related quality of life after total knee arthroplasty

BACKGROUND

Current predictive tools for TKA focus on clinicians rather than patients as the intended user. The purpose of this study was to develop a patient-focused model to predict health-related quality of life outcomes at 1-year post-TKA.

METHODS

Patients who underwent primary TKA for osteoarthritis from a tertiary institutional registry after January 2006 were analysed. The primary outcome was improvement after TKA defined by the minimal clinically important difference in utility score at 1-year post-surgery. Potential predictors included demographic information, comorbidities, lifestyle factors, and patient-reported outcome measures. Four models were developed, including both conventional statistics and machine learning (artificial intelligence) methods: logistic regression, classification tree, extreme gradient boosted trees, and random forest models. Models were evaluated using discrimination and calibration metrics.

RESULTS

A total of 3755 patients were included in the study. The logistic regression model performed the best with respect to both discrimination (AUC = 0.712) and calibration (intercept = -0.083, slope = 1.123, Brier score = 0.202). Less than 2% (n = 52) of the data were missing and therefore removed for complete case analysis. The final model used age (categorical), sex, baseline utility score, and baseline Veterans-RAND 12 responses as predictors.

CONCLUSION

The logistic regression model performed better than machine learning algorithms with respect to AUC and calibration plot. The logistic regression model was well calibrated enough to stratify patients into risk deciles based on their likelihood of improvement after surgery. Further research is required to evaluate the performance of predictive tools through pragmatic clinical trials.

LEVEL OF EVIDENCE

Level II, decision analysis.

Artificial intelligence for image analysis in total hip and total knee arthroplasty : a scoping review

AIMS

Total hip arthroplasty (THA) and total knee arthroplasty (TKA) are common orthopaedic procedures requiring postoperative radiographs to confirm implant positioning and identify complications. Artificial intelligence (AI)-based image analysis has the potential to automate this postoperative surveillance. The aim of this study was to prepare a scoping review to investigate how AI is being used in the analysis of radiographs following THA and TKA, and how accurate these tools are.

METHODS

The Embase, MEDLINE, and PubMed libraries were systematically searched to identify relevant articles. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews and Arksey and O'Malley framework were followed. Study quality was assessed using a modified Methodological Index for Non-Randomized Studies tool. AI performance was reported using either the area under the curve (AUC) or accuracy.

RESULTS

Of the 455 studies identified, only 12 were suitable for inclusion. Nine reported implant identification and three described predicting risk of implant failure. Of the 12, three studies compared AI performance with orthopaedic surgeons. AI-based implant identification achieved AUC 0.992 to 1, and most algorithms reported an accuracy > 90%, using 550 to 320,000 training radiographs. AI prediction of dislocation risk post-THA, determined after five-year follow-up, was satisfactory (AUC 76.67; 8,500 training radiographs). Diagnosis of hip implant loosening was good (accuracy 88.3%; 420 training radiographs) and measurement of postoperative acetabular angles was comparable to humans (mean absolute difference 1.35° to 1.39°). However, 11 of the 12 studies had several methodological limitations introducing a high risk of bias. None of the studies were externally validated.

CONCLUSION

These studies show that AI is promising. While it already has the ability to analyze images with significant precision, there is currently insufficient high-level evidence to support its widespread clinical use. Further research to design robust studies that follow standard reporting guidelines should be encouraged to develop AI models that could be easily translated into real-world conditions. Cite this article: Bone Joint J 2022;104-B(8):929-937.

Artificial intelligence and computer vision in orthopaedic trauma : the why, what, and how

Artificial intelligence (AI) is, in essence, the concept of 'computer thinking', encompassing methods that train computers to perform and learn from executing certain tasks, called machine learning, and methods to build intricate computer models that both learn and adapt, called complex neural networks. Computer vision is a function of AI by which machine learning and complex neural networks can be applied to enable computers to capture, analyze, and interpret information from clinical images and visual inputs. This annotation summarizes key considerations and future perspectives concerning computer vision, questioning the need for this technology (the 'why'), the current applications (the 'what'), and the approach to unlocking its full potential (the 'how'). Cite this article: Bone Joint J 2022;104-B(8):911-914.