The Plastic Surgery In-service Training Examination: An In-depth Reference Analysis

Comparing Graduating Plastic Surgery Residents' Case Logs With Accreditation Council for Graduate Medical Education Requirements, Content at National Meetings, and In-Service Examination Test Items

BACKGROUND

The importance of adaptable and up-to-date plastic surgery graduate medical education (GME) has taken on new meaning amidst accelerating surgical innovation and increasing calls for competency-based training standards. We aimed to examine the extent to which the procedures plastic surgery residents perform, as represented in case log data, align with 2 core standardized components of plastic surgery GME: ACGME (Accreditation Council for Graduate Medical Education) minimum procedure count requirements and the PSITE (Plastic Surgery In-Service Training Examination). We also examined their alignment with procedural representation at 2 major plastic surgery meetings.

METHODS

Nine categories of reconstructive and aesthetic procedures were identified. Three-year averages for the number of procedures completed in each category by residents graduating in 2019-2021 were calculated from ACGME national case log data reports. The ACGME procedure count minimum requirements were also ascertained. The titles and durations of medical programming sessions scheduled for Plastic Surgery The Meeting (PSTM) 2022 and the Plastic Surgery Research Council (PSRC) Annual Meeting 2022 were retrieved from online data. Finally, test items from the 2020 to 2022 administrations of the PSITE were retrieved. Conference sessions and test items were assigned to a single procedure category when possible. Percent differences were calculated for comparison.

RESULTS

The distribution of procedures on plastic surgery resident case logs differs from those of the major mechanisms of standardization in plastic surgery GME, in-service examination content more so than ACGME requirements. Meeting content at PSTM and PSRC had the largest percent differences with case log data, with PSTM being skewed toward aesthetics and PSRC toward reconstructive head and neck surgery.

DISCUSSION

The criteria and standards by which plastic surgery residents are evaluated and content at national meetings differ from the procedures they actually complete during their training. Although largely reflecting heterogeneity of the specialty, following these comparisons will likely prove useful in the continual evaluation of plastic surgery residency training, especially in the preparation of residents for the variety of training and practice settings they pursue.

Performance of ChatGPT on the Plastic Surgery Inservice Training Examination

BACKGROUND

Developed originally as a tool for resident self-evaluation, the Plastic Surgery Inservice Training Examination (PSITE) has become a standardized tool adopted by Plastic Surgery residency programs. The introduction of large language models (LLMs), such as ChatGPT (OpenAI, San Francisco, CA), has demonstrated the potential to help propel the field of Plastic Surgery.

OBJECTIVES

The authors of this study wanted to assess whether or not ChatGPT could be utilized as a tool in resident education by assessing its accuracy on the PSITE.

METHODS

Questions were obtained from the 2022 PSITE, which was present on the American Council of Academic Plastic Surgeons (ACAPS) website. Questions containing images or tables were carefully inspected and flagged before being inputted into ChatGPT. All responses by ChatGPT were qualified utilizing the properties of natural coherence. Responses that were found to be incorrect were divided into the following categories: logical, informational, or explicit fallacy.

RESULTS

ChatGPT answered a total of 242 questions with an accuracy of 54.96%. The software incorporated logical reasoning in 88.8% of questions, internal information in 95.5% of questions, and external information in 92.1% of questions. When stratified by correct and incorrect responses, we determined that there was a statistically significant difference in ChatGPT's use of external information (P < .05).

CONCLUSIONS

ChatGPT is a versatile tool that has the potential to impact resident education by providing general knowledge, clarifying information, providing case-based learning, and promoting evidence-based medicine. With advancements in LLM and artificial intelligence (AI), it is possible that ChatGPT may be an impactful tool for resident education within Plastic Surgery.

Bard Versus the 2022 American Society of Plastic Surgeons In-Service Examination: Performance on the Examination in Its Intern Year

Background

Bard is a conversational generative artificial intelligence (AI) platform released by Google (Mountain View, CA) to the public in May 2023.

Objectives

This study investigates the performance of Bard on the American Society of Plastic Surgeons (ASPS) In-Service Examination to compare it to residents' performance nationally. We hypothesized that Bard would perform best on the comprehensive and core surgical principles portions of the examination.

Methods

Google's 2023 Bard was used to answer questions from the 2022 ASPS In-Service Examination. Each question was asked as written with the stem and multiple-choice options. The 2022 ASPS Norm Table was utilized to compare Bard's performance to that of subgroups of plastic surgery residents.

Results

A total of 231 questions were included. Bard answered 143 questions correctly corresponding to an accuracy of 62%. The highest-performing section was the comprehensive portion (73%). When compared with integrated residents nationally, Bard scored in the 74th percentile for post-graduate year (PGY)-1, 34th percentile for PGY-2, 20th percentile for PGY-3, 8th percentile for PGY-4, 1st percentile for PGY-5, and 2nd percentile for PGY-6.

Conclusions

Bard outperformed more than half of the first-year integrated residents (74th percentile). Its best sections were the comprehensive and core surgical principle portions of the examination. Further analysis of the chatbot's incorrect questions might help improve the overall quality of the examination's questions.