Artificial intelligence in orthopaedic surgery: transforming technological innovation in patient care and surgical training

Identification of ChatGPT-Generated Abstracts Within Shoulder and Elbow Surgery Poses a Challenge for Reviewers

PURPOSE

To evaluate the extent to which experienced reviewers can accurately discern between AI-generated and original research abstracts published in the field of shoulder and elbow surgery and compare this to the performance of an AI-detection tool.

METHODS

Twenty-five shoulder and elbow-related articles published in high-impact journals in 2023 were randomly selected. ChatGPT was prompted with only the abstract title to create an AI-generated version of each abstract. The resulting 50 abstracts were randomly distributed to and evaluated by 8 blinded peer reviewers with at least 5 years of experience. Reviewers were tasked with distinguishing between original and AI-generated text. A Likert scale assessed reviewer confidence for each interpretation and the primary reason guiding assessment of generated text was collected. AI output detector (0-100%) and plagiarism (0-100%) scores were evaluated using GPTZero.

RESULTS

Reviewers correctly identified 62% of AI-generated abstracts and misclassified 38% of original abstracts as being AI-generated. GPTZero reported a significantly higher probability of AI output among generated abstracts (median 56%, IQR 51-77%) compared to original abstracts (median 10%, IQR 4-37%; p < 0.01). Generated abstracts scored significantly lower on the plagiarism detector (median 7%, IQR 5-14%) relative to original abstracts (median 82%, IQR 72-92%; p < 0.01). Correct identification of AI-generated abstracts was predominately attributed to the presence of unrealistic data/values. The primary reason for misidentifying original abstracts as AI was attributed to writing style.

CONCLUSIONS

Experienced reviewers faced difficulties in distinguishing between human and AI-generated research content within shoulder and elbow surgery. The presence of unrealistic data facilitated correct identification of AI abstracts, whereas misidentification of original abstracts was often ascribed to writing style.

A Scientometric Worldview of Artificial Intelligence in Musculoskeletal Diseases Since the 21st Century

Purpose

Over the past 24 years, significant advancements have been made in applying artificial intelligence (AI) to musculoskeletal (MSK) diseases. However, there is a lack of analytical and descriptive investigations on the trajectory, essential research directions, current research scenario, pivotal focuses, and future perspectives. This research aims to provide a thorough update on the progress in AI for MSK diseases over the last 24 years.

Methods

Data from the Web of Science database, covering January 1, 2000, to March 1, 2024, was analyzed. Using advanced analytical tools, we conducted comprehensive scientometric and visual analyses.

Results

The findings highlight the predominant influence of the USA, which accounts for 28.53% of the total publications and plays a key role in shaping research in this field. Notable productivity was seen at institutions such as the University of California, San Francisco, Harvard Medical School, and Seoul National University. Valentina Pedoia is identified as the most prolific contributor. Scientific Reports had the highest number of publications in this area. The five most significant diseases are joint diseases, bone fractures, bone tumors, cartilage diseases, and spondylitis.

Conclusion

This comprehensive scientometric assessment benefits both experienced researchers and newcomers, providing quick access to essential information and fostering the development of innovative concepts in this field.

Postoperative Communication Volume Following Total Joint Arthroplasty Can Be a Precursor for Emergency Department Visits

Background

Unplanned calls, messages, and visits to the clinic can occur at a higher rate as newer technologies allow patients more accessibility and connectivity to clinicians. By reviewing postoperative patient phone calls and electronic portal messages, we compared the methods and frequency of communications between conventional and robotic joint arthroplasty cases.

Methods

A retrospective review of total hip, total knee, and unicompartmental knee arthroplasty procedures by fellowship-trained adult reconstruction surgeons at our hospitals between 2017 and 2022 was performed. Any unplanned postoperative communication within 30 days of the postoperative period and unplanned emergency department visits were collected.

Results

There were 12,300 robotic and manual consecutive primary total hip, total knee, and unicompartmental knee arthroplasty procedures performed on 10,908 patients over the study period. A total of 905 (40.4%) patients and 2012 (23.2%) patients sent an electronic text message (ETM) in the robotic and manual arthroplasty cohorts (P < .0001), respectively. Overall, 1942 (86.6%) patients in the robotic arthroplasty group and 6417 (74%) patients in the manual arthroplasty group had at least one phone call within the first month after their joint arthroplasty.

Conclusions

Robotic arthroplasty patients place an increased demand on the orthopaedic surgery department in terms of unplanned patient contacts. Robotic arthroplasty patients had a significantly increased rate of unplanned postoperative ETMs and phone calls when compared to manual arthroplasty patients. An increased number of postoperative phone calls, but not ETMs, can also be indicative of an emergency department visit. These findings can be used in the perioperative setting to counsel and educate patients about expectations.

ChatGPT performance on the American Shoulder and Elbow Surgeons maintenance of certification exam

BACKGROUND

While multiple studies have tested the ability of large language models (LLMs), such as ChatGPT, to pass standardized medical exams at different levels of training, LLMs have never been tested on surgical sub-specialty examinations, such as the American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC). The purpose of this study was to compare results of ChatGPT 3.5, GPT-4, and fellowship-trained surgeons on the 2023 ASES MOC self-assessment exam.

METHODS

ChatGPT 3.5 and GPT-4 were subjected to the same set of text-only questions from the ASES MOC exam, and GPT-4 was additionally subjected to image-based MOC exam questions. Question responses from both models were compared against the correct answers. Performance of both models was compared to corresponding average human performance on the same question subsets. One sided proportional z-test were utilized to analyze data.

RESULTS

Humans performed significantly better than Chat GPT 3.5 on exclusively text-based questions (76.4% vs. 60.8%, P = .044). Humans also performed significantly better than GPT 4 on image-based questions (73.9% vs. 53.2%, P = .019). There was no significant difference between humans and GPT 4 in text-based questions (76.4% vs. 66.7%, P = .136). Accounting for all questions, humans significantly outperformed GPT-4 (75.3% vs. 60.2%, P = .012). GPT-4 did not perform statistically significantly betterer than ChatGPT 3.5 on text-only questions (66.7% vs. 60.8%, P = .268).

DISCUSSION

Although human performance was overall superior, ChatGPT demonstrated the capacity to analyze orthopedic information and answer specialty-specific questions on the ASES MOC exam for both text and image-based questions. With continued advancements in deep learning, LLMs may someday rival exam performance of fellowship-trained surgeons.

Evaluating ChatGPT Performance on the Orthopaedic In-Training Examination

Background

Artificial intelligence (AI) holds potential in improving medical education and healthcare delivery. ChatGPT is a state-of-the-art natural language processing AI model which has shown impressive capabilities, scoring in the top percentiles on numerous standardized examinations, including the Uniform Bar Exam and Scholastic Aptitude Test. The goal of this study was to evaluate ChatGPT performance on the Orthopaedic In-Training Examination (OITE), an assessment of medical knowledge for orthopedic residents.

Methods

OITE 2020, 2021, and 2022 questions without images were inputted into ChatGPT version 3.5 and version 4 (GPT-4) with zero prompting. The performance of ChatGPT was evaluated as a percentage of correct responses and compared with the national average of orthopedic surgery residents at each postgraduate year (PGY) level. ChatGPT was asked to provide a source for its answer, which was categorized as being a journal article, book, or website, and if the source could be verified. Impact factor for the journal cited was also recorded.

Results

ChatGPT answered 196 of 360 answers correctly (54.3%), corresponding to a PGY-1 level. ChatGPT cited a verifiable source in 47.2% of questions, with an average median journal impact factor of 5.4. GPT-4 answered 265 of 360 questions correctly (73.6%), corresponding to the average performance of a PGY-5 and exceeding the corresponding passing score for the American Board of Orthopaedic Surgery Part I Examination of 67%. GPT-4 cited a verifiable source in 87.9% of questions, with an average median journal impact factor of 5.2.

Conclusions

ChatGPT performed above the average PGY-1 level and GPT-4 performed better than the average PGY-5 level, showing major improvement. Further investigation is needed to determine how successive versions of ChatGPT would perform and how to optimize this technology to improve medical education.

Clinical Relevance

AI has the potential to aid in medical education and healthcare delivery.

Challenges of Integrating New Technologies for Orthopedic Doctors to Face up to Difficulties during the Pandemic Era

In the field of orthopedics, competitive progress is growing faster because new technologies used to facilitate the work of physicians are continuously developing. Based on the issues generated in the pandemic era in this field, a research study was developed to identify the intention of orthopedic doctors to integrate new medical technologies. The survey was based on a questionnaire that was used for data collection. The quantitative study registered a sample of 145 orthopedic doctors. The data analysis was performed based on the IBM SPSS program. A multiple linear regression model was applied, which analyzed how the independent variables can influence the dependent variables. After analyzing the data, it was observed that the intention of orthopedic doctors to use new medical technologies is influenced by the advantages and disadvantages perceived by them, the perceived risks, the quality of the medical technologies, the experience of physicians in their use, and their receptivity to other digital tools. The obtained results are highly important both for hospital managers and authorities, illustrating the main factors that influence doctors to use emergent technologies in their clinical work.

The Changing Environment in Postgraduate Education in Orthopedic Surgery and Neurosurgery and Its Impact on Technology-Driven Targeted Interventional and Surgical Pain Management: Perspectives from Europe, Latin America, Asia, and The United States

Personalized care models are dominating modern medicine. These models are rooted in teaching future physicians the skill set to keep up with innovation. In orthopedic surgery and neurosurgery, education is increasingly influenced by augmented reality, simulation, navigation, robotics, and in some cases, artificial intelligence. The postpandemic learning environment has also changed, emphasizing online learning and skill- and competency-based teaching models incorporating clinical and bench-top research. Attempts to improve work-life balance and minimize physician burnout have led to work-hour restrictions in postgraduate training programs. These restrictions have made it particularly challenging for orthopedic and neurosurgery residents to acquire the knowledge and skill set to meet the requirements for certification. The fast-paced flow of information and the rapid implementation of innovation require higher efficiencies in the modern postgraduate training environment. However, what is taught typically lags several years behind. Examples include minimally invasive tissue-sparing techniques through tubular small-bladed retractor systems, robotic and navigation, endoscopic, patient-specific implants made possible by advances in imaging technology and 3D printing, and regenerative strategies. Currently, the traditional roles of mentee and mentor are being redefined. The future orthopedic surgeons and neurosurgeons involved in personalized surgical pain management will need to be versed in several disciplines ranging from bioengineering, basic research, computer, social and health sciences, clinical study, trial design, public health policy development, and economic accountability. Solutions to the fast-paced innovation cycle in orthopedic surgery and neurosurgery include adaptive learning skills to seize opportunities for innovation with execution and implementation by facilitating translational research and clinical program development across traditional boundaries between clinical and nonclinical specialties. Preparing the future generation of surgeons to have the aptitude to keep up with the rapid technological advances is challenging for postgraduate residency programs and accreditation agencies. However, implementing clinical protocol change when the entrepreneur-investigator surgeon substantiates it with high-grade clinical evidence is at the heart of personalized surgical pain management.