Analysis of References on the Plastic Surgery In-Service Training Exam

Examining the Most Impactful Strategies for In-service Preparation: A Systemic Review

INTRODUCTION

Resident performance on the Plastic Surgery In-Service Examination (PSITE) is used as a predictor of success on the American Board of Plastic Surgery Written Examination, as well as resident progression and fellowship applications. However, information specifically addressing strategies on optimal PSITE preparation is lacking in the plastic surgery literature when compared to general surgery. For this reason, we aim to understand if the topic is well-studied and denote effective study strategies and curricular interventions in both fields that can help residents and programs optimize PSITE performance.

METHODS

A literature search including studies from 2012 to 2022 was conducted following PRISMA guidelines in PubMed and EMBASE to identify articles on strategies to improve in-service exam scores for general surgery and plastic surgery. Only studies that reported measurable outcomes in raw score, percentile score, or percent correct were included.

RESULTS

Qualitative analysis of 30 articles revealed 2 categories of interventions: individual study habits and institutional curricular interventions. In general surgery literature, 27 articles examined interventions positively impacting resident ABSITE scores, with 21 studies classified as institutional curricular interventions and 6 articles addressing individual study habits. Themes associated with improved ABSITE performance included mandatory remediation programs, dedicated study time, and problem-based learning interventions. In contrast, only 3 articles in plastic surgery literature discussed interventions associated with improved PSITE scores, all falling under curricular interventions.

CONCLUSION

Unfortunately, the plastic surgery literature lacks concrete evidence on how residents can improve performance. Future research in plastic surgery should replicate successful strategies from general surgery and further investigate optimal preparation strategies for the PSITE. Such endeavors can contribute to improving resident performance and advancing plastic surgery education and patient care.

ChatGPT Is Equivalent to First-Year Plastic Surgery Residents: Evaluation of ChatGPT on the Plastic Surgery In-Service Examination

BACKGROUND

ChatGPT is an artificial intelligence language model developed and released by OpenAI (San Francisco, CA) in late 2022.

OBJECTIVES

The aim of this study was to evaluate the performance of ChatGPT on the Plastic Surgery In-Service Examination and to compare it to residents' performance nationally.

METHODS

The Plastic Surgery In-Service Examinations from 2018 to 2022 were used as a question source. For each question, the stem and all multiple-choice options were imported into ChatGPT. The 2022 examination was used to compare the performance of ChatGPT to plastic surgery residents nationally.

RESULTS

In total, 1129 questions were included in the final analysis and ChatGPT answered 630 (55.8%) of these correctly. ChatGPT scored the highest on the 2021 exam (60.1%) and on the comprehensive section (58.7%). There were no significant differences regarding questions answered correctly among exam years or among the different exam sections. ChatGPT answered 57% of questions correctly on the 2022 exam. When compared to the performance of plastic surgery residents in 2022, ChatGPT would rank in the 49th percentile for first-year integrated plastic surgery residents, 13th percentile for second-year residents, 5th percentile for third- and fourth-year residents, and 0th percentile for fifth- and sixth-year residents.

CONCLUSIONS

ChatGPT performs at the level of a first-year resident on the Plastic Surgery In-Service Examination. However, it performed poorly when compared with residents in more advanced years of training. Although ChatGPT has many undeniable benefits and potential uses in the field of healthcare and medical education, it will require additional research to assess its efficacy.

ChatGPT Performs at the Level of a Third-Year Orthopaedic Surgery Resident on the Orthopaedic In-Training Examination

Introduction

Publicly available AI language models such as ChatGPT have demonstrated utility in text generation and even problem-solving when provided with clear instructions. Amidst this transformative shift, the aim of this study is to assess ChatGPT's performance on the orthopaedic surgery in-training examination (OITE).

Methods

All 213 OITE 2021 web-based questions were retrieved from the AAOS-ResStudy website (https://www.aaos.org/education/examinations/ResStudy). Two independent reviewers copied and pasted the questions and response options into ChatGPT Plus (version 4.0) and recorded the generated answers. All media-containing questions were flagged and carefully examined. Twelve OITE media-containing questions that relied purely on images (clinical pictures, radiographs, MRIs, CT scans) and could not be rationalized from the clinical presentation were excluded. Cohen's Kappa coefficient was used to examine the agreement of ChatGPT-generated responses between reviewers. Descriptive statistics were used to summarize the performance (% correct) of ChatGPT Plus. The 2021 norm table was used to compare ChatGPT Plus' performance on the OITE to national orthopaedic surgery residents in that same year.

Results

A total of 201 questions were evaluated by ChatGPT Plus. Excellent agreement was observed between raters for the 201 ChatGPT-generated responses, with a Cohen's Kappa coefficient of 0.947. 45.8% (92/201) were media-containing questions. ChatGPT had an average overall score of 61.2% (123/201). Its score was 64.2% (70/109) on non-media questions. When compared to the performance of all national orthopaedic surgery residents in 2021, ChatGPT Plus performed at the level of an average PGY3.

Discussion

ChatGPT Plus is able to pass the OITE with an overall score of 61.2%, ranking at the level of a third-year orthopaedic surgery resident. It provided logical reasoning and justifications that may help residents improve their understanding of OITE cases and general orthopaedic principles. Further studies are still needed to examine their efficacy and impact on long-term learning and OITE/ABOS performance.

The Plastic Surgery In-service Training Examination: An In-depth Reference Analysis

The goal of this study was to characterize the references provided as supporting evidence of the Plastic Surgery In-service Training Examination (PSITE) syllabi, including those on the novel “core surgical principles” section.

The Benefit of a Formal Plastic Surgery In-Service Conference

INTRODUCTION

Annually, residents are expected to take an in-service examination to gauge their understanding of plastic surgery knowledge and prepare them for the American Board of Plastic Surgery written examination. In addition, in-service score are now being used as an assessment tool for fellowship applicants. Because of the breadth of Plastic Surgery material, it is difficult to prepare a resident for such a comprehensive examination. At the University of Utah, a weekly conference was instituted to help prepare residents for the in-service and board examination with the goal of improving scores.

MATERIALS AND METHODS

A weekly 90min review conference was initiated at the University of Utah in an effort to improve in-service scores. Residents along with a member of the faculty reviewed old in-service examination questions and discussed the selected topics in depth. The residents' examination score averages per PGY level were compared from years before and after initiation of the conference. In addition, examination scores for each individual were compared before and after initiation of the conference. Paired t-test comparisons were performed to analyze the results.

RESULTS

Statistically significant improvement in residents examination scores averages were observed from years before and after initiation of the conference after the second year of training (42% vs 62%, p = 0.03). Furthermore, examination scores for each individual obtained the years before and the year after initiation of the conference significantly improved (31% vs 71%, p = 0.01). When comparing individuals in years prior to implementation of the conference there was no statistically significant improvement from year to year.

CONCLUSION

Implementation of a formal weekly in-service conference significantly improved performance on the in-service examination. Improvement was found when comparing between PGY training level after the second year of training and individually for residents. These results advocate for a focused educational conference for preparation for the in-service examination.