Levels of Evidence for Hand Questions on the Orthopaedic In-Training Examination

Can generative artificial intelligence pass the orthopaedic board examination?

Background

Resident training programs in the US use the Orthopaedic In-Training Examination (OITE) developed by the American Academy of Orthopaedic Surgeons (AAOS) to assess the current knowledge of their residents and to identify the residents at risk of failing the Amerian Board of Orthopaedic Surgery (ABOS) examination. Optimal strategies for OITE preparation are constantly being explored. There may be a role for Large Language Models (LLMs) in orthopaedic resident education. ChatGPT, an LLM launched in late 2022 has demonstrated the ability to produce accurate, detailed answers, potentially enabling it to aid in medical education and clinical decision-making. The purpose of this study is to evaluate the performance of ChatGPT on Orthopaedic In-Training Examinations using Self-Assessment Exams from the AAOS database and approved literature as a proxy for the Orthopaedic Board Examination.

Methods

301 SAE questions from the AAOS database and associated AAOS literature were input into ChatGPT's interface in a question and multiple-choice format and the answers were then analyzed to determine which answer choice was selected. A new chat was used for every question. All answers were recorded, categorized, and compared to the answer given by the OITE and SAE exams, noting whether the answer was right or wrong.

Results

Of the 301 questions asked, ChatGPT was able to correctly answer 183 (60.8%) of them. The subjects with the highest percentage of correct questions were basic science (81%), oncology (72.7%, shoulder and elbow (71.9%), and sports (71.4%). The questions were further subdivided into 3 groups: those about management, diagnosis, or knowledge recall. There were 86 management questions and 47 were correct (54.7%), 45 diagnosis questions with 32 correct (71.7%), and 168 knowledge recall questions with 102 correct (60.7%).

Conclusions

ChatGPT has the potential to provide orthopedic educators and trainees with accurate clinical conclusions for the majority of board-style questions, although its reasoning should be carefully analyzed for accuracy and clinical validity. As such, its usefulness in a clinical educational context is currently limited but rapidly evolving.

Clinical relevance

ChatGPT can access a multitude of medical data and may help provide accurate answers to clinical questions.

Comparison of Artificial Intelligence to Resident Performance on Upper-Extremity Orthopaedic In-Training Examination Questions

Purpose

Currently, there is a paucity of prior investigations and studies examining applications for artificial intelligence (AI) in upper-extremity (UE) surgical education. The purpose of this investigation was to assess the performance of a novel AI tool (ChatGPT) on UE questions on the Orthopaedic In-Training Examination (OITE). We aimed to compare the performance of ChatGPT to the examination performance of hand surgery residents.

Methods

We selected questions from the 2020-2022 OITEs that focused on both the hand and UE as well as the shoulder and elbow content domains. These questions were divided into two categories: those with text-only prompts (text-only questions) and those that included supplementary images or videos (media questions). Two authors (B.K.F. and G.S.M.) converted the accompanying media into text-based descriptions. Included questions were inputted into ChatGPT (version 3.5) to generate responses. Each OITE question was entered into ChatGPT three times: (1) open-ended response, which requested a free-text response; (2) multiple-choice responses without asking for justification; and (3) multiple-choice response with justification. We referred to the OITE scoring guide for each year in order to compare the percentage of correct AI responses to correct resident responses.

Results

A total of 102 UE OITE questions were included; 59 were text-only questions, and 43 were media-based. ChatGPT correctly answered 46 (45%) of 102 questions using the Multiple Choice No Justification prompt requirement (42% for text-based and 44% for media questions). Compared to ChatGPT, postgraduate year 1 orthopaedic residents achieved an average score of 51% correct. Postgraduate year 5 residents answered 76% of the same questions correctly.

Conclusions

ChatGPT answered fewer UE OITE questions correctly compared to hand surgery residents of all training levels.

Clinical relevance

Further development of novel AI tools may be necessary if this technology is going to have a role in UE education.

Fifty Most-Cited Research Articles in Elbow Surgery: A Modern Reading List

Purpose

Bibliometric analysis is a common method for evaluating current trends within a scientific field. The primary aim of this study was to define and analyze the 50 most frequently cited articles in the field of elbow surgery, both of all time and those published during the 21st century.

Methods

We searched the Journal Citation Report to identify articles related to elbow surgery within academic journals. Articles were sorted by total citations. The overall top 50 articles and those published since 2000 were identified, and data were collected, including title, journal of publication, publication year, country of publication, citation density, level of evidence, article type, institution, and sex of the lead and senior authors, and inclusion on the reference list for the Orthopaedic In-Training Examination within the last 5 years. Descriptive statistics were reported, and correlation analysis was performed using Spearman test.

Results

For the most-cited elbow surgery articles, "fracture" was overall the most reported topic, whereas "lateral epicondylosis" and "fracture" were equal for those published since 2000. The United States was the most represented overall and for articles published since 2000. Women comprised 1/50 (2%) of lead authors overall, increasing to 8/50 (16%) for articles published during the 21st century. Most articles in during both periods contained level IV evidence, with level I evidence appearing infrequently (4%). Six percent of the most-cited articles of all time had appeared on the reference list of the Orthopaedic In-Training Examination within the past 5 years.

Conclusions

The top 50 most-cited elbow surgery articles often assess fracture and lateral epicondylosis, most commonly originating from the United States. Level IV retrospective series comprises over half of the articles on this list. Women remain underrepresented as authors.

Clinical Relevance

This study provides a modern reading list for upper-extremity surgeons about impactful elbow surgery articles.

Analysis of Questions in Sections of the Orthopaedic In-Training Examination: A Scoping Review

PURPOSE

The purpose of this review was to assess all available studies that analyzed the types of questions in individual sections of the Orthopaedic In-Training Examination, which may be used as a reference for residents studying for their examination.

METHODS

Following the Providing Innovative Service Models and Assessment extension for Scoping Reviews guidelines, a systematic review was conducted on studies that report on sections or question categories of the Orthopaedic In-Training Examination using PubMed, MEDLINE, and Web of Science databases. Two reviewers and an arbitrator reviewed and extracted relevant data from 20 included studies which made up the systematic review.

RESULTS

All 20 studies in the review reported the mean number of questions per section, with the highest coming from musculoskeletal trauma (18.9% to 19.0%). 18 studies reported the Buckwalter taxonomic classification; 42.0% of questions were T1, 18.2% were T2, and 39.5% were T3 with a wide range from section to section. Primary sources were nearly three times more likely to be cited when compared with textbook sources. There were 12 journals that were commonly cited with the most being the Journal of Bone and Joint Surgery: American Volume (17/18).

DISCUSSION

This study accurately portrays the characteristics of each section of the Orthopaedic In-Training Examination over the past 10 years. These data suggest that orthopaedic residents may be inclined to focus on musculoskeletal trauma, topics related to clinical management, and primary journal sources for studying. In addition, residency programs may choose to focus on higher yield sources or material to prepare their residents for the examination.

Analysis of Hand Surgery Questions on Orthopedic In-training Examination From 2014 to 2019

BACKGROUND

The orthopedic in-training examination (OITE) continues to evolve over time. It is important for orthopedic residents and residency programs to have an up-to-date understanding of the content and resources being used on the OITE to study and tailor curricula accordingly. This study presents an updated analysis of the OITE hand domain from 2014 to 2019.

METHODS

All OITE questions related to hand surgery from 2014 to 2019 were analyzed for topic, subtopic, taxonomy, imaging modalities, and bibliometric factors related to cited references.

RESULTS

Of the 1600 OITE questions, there were 113 hand surgery questions (7.1%) over a 6-year period. The most commonly tested topics were nerve (n = 22; 19%), fracture/dislocation (n = 21; 19%), and tendon/ligament (n = 19; 17%). Complex clinical management questions were the most common taxonomic category (n = 66; 58%). Two hundred fifty-two references were cited, the most common of which were from the Journal of Hand Surgery (American Volume) (n = 76; 30%), Journal of the American Academy of Orthopaedic Surgeons (n = 27; 11%), and Hand Clinics (n = 21; 8%). Publication lag decreased over the study period (P = .009). Twenty-five questions (22%) used imaging modalities, and 21 (19%) used clinical photos. Compared with a prior analysis from 2002 to 2006, there were more questions related to nerves (19.5% vs 9.8%, P = .041).

CONCLUSIONS

Residents and residency programs can benefit from an updated understanding of OITE hand surgery content and resources. The current analysis identifies high-yield topics and resources that can guide resident preparation for the OITE.

Orthopaedic In-Training Examination: History, Perspective, and Tips for Residents

Introduced in 1963, the orthopaedic in-training examination (OITE) is a standardized, national test administered annually to orthopaedic residents by the American Academy of Orthopaedic Surgeons. The examination consists of 275 multiple-choice questions that cover 11 domains of orthopaedic knowledge, including basic science, foot and ankle, hand, hip and knee, oncology, pediatrics, shoulder and elbow, spine, sports medicine, trauma, and practice management. The OITE has been validated and is considered predictive of success in both orthopaedic surgery residency and on the American Board of Orthopaedic Surgery part I examination. This article provides a historical overview of the OITE, details its current structure and scoring system, and reviews currently available study materials. For examination preparation, the residents are encouraged to (1) start the examination preparation early, (2) practice on old OITE or self-assessment examination questions, (3) focus on the questions answered incorrectly, (4) focus on comprehension over memorization, and (5) recognize and avoid burnout. Finally, the residents should have a systemic way of approaching each multiple-choice question, both during practice and on the actual examination.

Updated Analysis of the Oncology Section of the OITE from 2007 to 2019

The Orthopaedic In-Training Exam (OITE) is administered annually to orthopedic surgery residents to assess their medical knowledge. The authors provide a comprehensive review of the orthopedic oncology portion of the exam in order to aid residents in preparation for future in-training and licensing examinations as well as to help guide oncology residency education curriculum. All of the orthopedic oncology questions on the OITE from 2007 to 2019 were reviewed. Analysis included (1) the number of oncology questions each year, (2) question topic, (3) question taxonomy (knowledge versus interpretation), (4) the type of imaging modalities (radiological, histological), (5) most commonly cited references, and (6) level of evidence. Descriptive statistics were utilized to compare means between variables. From 2007 to 2019, there was a total of 292 tumor-related questions with a mean of 22.5 tumor-related questions (range 19-28) per year. Of the questions, 54.8% pertained to malignant tumors and 45.2% to benign tumors. Assessment of question taxonomy showed that 79.8% of questions required interpretation of imaging and analysis of the information provided versus 20.2% of questions being knowledge recall type. Of the questions, 76.7% required interpretation of radiological images, pathological images, or both. Orthopaedic Knowledge Update, Journal of the American Academy of Orthopaedic Surgeons, and Journal of Bone and Joint Surgery were the three most commonly cited question sources. Only 29 (9.84%) oncology questions over the past 13 years have been supported by level I or II sources of evidence. Better understanding of the OITE make-up, question distribution, and number and style of question, reference sources can improve an orthopedic residents' performance as well as better guide educational curriculum to prepare residents in their orthopedic oncology education.

Analysis of the Sports Medicine Section of the Orthopaedic In-Training Examination: Improvements in Levels of Evidence and Question Taxonomy Over a 12-Year Period

There has been a recent shift within the orthopedic literature to publish articles with higher levels of evidence. In this investigation, the trends in question taxonomy and the levels of evidence of the references for sports medicine questions on the Orthopaedic In-Training Examination (OITE) during a 12-year period were evaluated. Sports medicine questions were obtained from the OITEs administered between 2005 and 2007 and between 2014 and 2016. The taxonomy of each question was characterized, and levels of evidence for all references were assigned using American Academy of Orthopaedic Surgeons guidelines. Question taxonomy and article levels of evidence from 2005 to 2007 were compared with those from 2014 to 2016. Sports medicine questions comprised 8% of the OITEs in both examination groups. The questions from 2014 to 2016 had a higher mean taxonomic level (2.26 vs 1.52, P=.0001) and a greater proportion of studies with high levels of evidence (levels 1 and 2) (21% vs 10%, P=.027). However, references with low levels of evidence or nonprimary resources made up 82.2% and 68.5% of the total references on the older and more recent examinations, respectively. References from 2014 to 2016 were, on average, 2 years older than those from earlier examinations. This study indicated that sports medicine questions on recent OITEs cite references of higher levels of evidence and contain higher taxonomic question structure than examinations 10 to 12 years ago. However, the majority of questions still cite articles with low levels of evidence or nonprimary sources. These findings can be used to guide resident education and continue improvements in the selection of references for questions on the Sports Medicine section of the OITE. [Orthopedics. 2020;43(5):e460-e464.].

Evidence hierarchies relating to hand surgery: current status and improvement. A bibliometric analysis study

BACKGROUND

Hierarchy of evidence is an important measurement for assessing quality of literature. Information regarding quality of evidence within the Brazilian hand surgery setting is sparse, especially regarding whether research has improved in either quality or quantity. This study aimed to identify and classify hand surgery studies published in the two most important Brazilian orthopedics journals based on hierarchy of evidence, with comparisons with previously published data.

DESIGN AND SETTING

Bibliometric analysis study performed in a federal university.

METHODS

Two independent researchers conducted an electronic database search for hand surgery studies published between 2010 and 2016 in Acta Ortopédica Brasileira and Revista Brasileira de Ortopedia. Eligible studies were subsequently classified according to methodological design, based on the Haynes pyramid model (HP) and the JBJS/AAOS levels of evidence and grades of recommendations (LOR). Qualitative and quantitative data were gathered regarding all studies. Previous data were considered to assess whether the proportion of high-quality studies had improved over time (2000-2009 versus 2010-2016).

RESULTS

The final analysis included 123 studies, mostly originating from the southeastern region (78.8%) and private institutions (65%), with self-funding (91.8%). Methodological assessment showed that 15.4% were classified as level I/II using HP and 16.4% using LOR. No significant difference in proportions of high-quality studies was found between the two periods of time assessed (5% versus 12%; P = 0.13).

CONCLUSION

Approximately 15% of hand surgery studies published in two major Brazilian journals were likely to be classified as high-quality through two different systems. Moreover, no trend towards quality-of-evidence improvement was found over the last 15 years.