Do scores of the USMLE Step 1 and OITE correlate with the ABOS Part I certifying examination?: a multicenter study

Evaluating Urology Program Directors' Perception on Resident Application Parameters Following the Transition of USMLE Step 1 to Pass/Fail

OBJECTIVE

To investigate how the shift of the United States Medical Licensing Examination (USMLE) Step 1 to a Pass/Fail (P/F) scoring system impacts the perceptions of Urology Program Directors (PDs) on evaluating urology residency applicants.

METHODS AND MATERIALS

A cross-sectional survey was sent to 117 PDs, including questions about program characteristics, perceptions of shelf scores and medical school rank post-transition, beliefs about the predictive value of Step 1 and Step 2 Clinical Knowledge (CK) scores for board success and residency performance, and changes in applicant parameter ranking.

RESULTS

Forty-five PDs (38% response rate) participated. Notably, 49% favored releasing quantitative clerkship grades, and 71% valued medical school rank more. Opinions on Step 1 scores' correlation with board success were split (49% agreed), and 44% endorsed Step 2 CK scores' connection to board performance. As predictors of good residents, only 9% and 22% considered Step 1 and Step 2 CK scores, respectively, indicative. Clerkship grades and Urology rotation recommendation letters maintained significance, while research experience gained importance. Step 2 CK scores' importance rose but did not match Step 1 scores' previous significance.

CONCLUSION

The transition to P/F for USMLE Step 1 adds intricacies to urology residency selection, exposing PDs' uncertainties regarding clerkship grades and the relevance of medical school rank. This research underscores the dynamic nature of urology residency admissions, emphasizing the increasing importance of research in evaluating applicants and a diminishing emphasis on volunteering and leadership.

Performance of Two Artificial Intelligence Generative Language Models on the Orthopaedic In-Training Examination

BACKGROUND

Artificial intelligence (AI) generative large language models are powerful and increasingly accessible tools with potential applications in health care education and training. The annual Orthopaedic In-Training Examination (OITE) is widely used to assess resident academic progress and preparation for the American Board of Orthopaedic Surgery Part 1 Examination.

MATERIALS AND METHODS

Open AI's ChatGPT and Google's Bard generative language models were administered the 2022 OITE. Question stems that contained images were input without and then with a text-based description of the imaging findings.

RESULTS

ChatGPT answered 69.1% of questions correctly. When provided with text describing accompanying media, this increased to 77.8% correct. In contrast, Bard answered 49.8% of questions correctly. This increased to 58% correct when text describing imaging in question stems was provided (P<.0001). ChatGPT was most accurate in questions within the shoulder category, with 90.9% correct. Bard performed best in the sports category, with 65.4% correct. ChatGPT performed above the published mean of Accreditation Council for Graduate Medical Education orthopedic resident test-takers (66%).

CONCLUSION

There is significant variability in the accuracy of publicly available AI models on the OITE. AI generative language software may play numerous potential roles in the future in orthopedic education, including simulating patient presentations and clinical scenarios, customizing individual learning plans, and driving evidence-based case discussion. Further research and collaboration within the orthopedic community is required to safely adopt these tools and minimize risks associated with their use. [Orthopedics. 2024;47(3):e146-e150.].

Comparison of Artificial Intelligence to Resident Performance on Upper-Extremity Orthopaedic In-Training Examination Questions

Purpose

Currently, there is a paucity of prior investigations and studies examining applications for artificial intelligence (AI) in upper-extremity (UE) surgical education. The purpose of this investigation was to assess the performance of a novel AI tool (ChatGPT) on UE questions on the Orthopaedic In-Training Examination (OITE). We aimed to compare the performance of ChatGPT to the examination performance of hand surgery residents.

Methods

We selected questions from the 2020-2022 OITEs that focused on both the hand and UE as well as the shoulder and elbow content domains. These questions were divided into two categories: those with text-only prompts (text-only questions) and those that included supplementary images or videos (media questions). Two authors (B.K.F. and G.S.M.) converted the accompanying media into text-based descriptions. Included questions were inputted into ChatGPT (version 3.5) to generate responses. Each OITE question was entered into ChatGPT three times: (1) open-ended response, which requested a free-text response; (2) multiple-choice responses without asking for justification; and (3) multiple-choice response with justification. We referred to the OITE scoring guide for each year in order to compare the percentage of correct AI responses to correct resident responses.

Results

A total of 102 UE OITE questions were included; 59 were text-only questions, and 43 were media-based. ChatGPT correctly answered 46 (45%) of 102 questions using the Multiple Choice No Justification prompt requirement (42% for text-based and 44% for media questions). Compared to ChatGPT, postgraduate year 1 orthopaedic residents achieved an average score of 51% correct. Postgraduate year 5 residents answered 76% of the same questions correctly.

Conclusions

ChatGPT answered fewer UE OITE questions correctly compared to hand surgery residents of all training levels.

Clinical relevance

Further development of novel AI tools may be necessary if this technology is going to have a role in UE education.

A Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States

Background Aligning resident and training program attributes is critical. Many programs screen and select residents using assessment tools not grounded in available evidence. This can introduce bias and inappropriate trainee recruitment. Prior reviews of this literature did not include the important lens of diversity, equity, and inclusion (DEI). Objective This study's objective is to summarize the evidence linking elements in the Electronic Residency Application Service (ERAS) application with selection and training outcomes, including DEI factors. Methods A systematic review was conducted on March 30, 2022, concordant with PRISMA guidelines, to identify the data supporting the use of elements contained in ERAS and interviews for residency training programs in the United States. Studies were coded into the topics of research, awards, United States Medical Licensing Examination (USMLE) scores, personal statement, letters of recommendation, medical school transcripts, work and volunteer experiences, medical school demographics, DEI, and presence of additional degrees, as well as the interview. Results The 2599 identified unique studies were reviewed by 2 authors with conflicts adjudicated by a third. Ultimately, 231 meeting inclusion criteria were included (kappa=0.53). Conclusions Based on the studies reviewed, low-quality research supports use of the interview, Medical Student Performance Evaluation, personal statement, research productivity, prior experience, and letters of recommendation in resident selection, while USMLE scores, grades, national ranking, attainment of additional degrees, and receipt of awards should have a limited role in this process.

Acquisition of Medical Knowledge Varies During Allopathic and Osteopathic Orthopedic Surgery Residency Training

Currently, little is known about the differences in medical knowledge acquisition between osteopathic and allopathic orthopedic surgery residents. The purpose of this study was to determine the relationship between Orthopaedic In-Training Examination (OITE) performance and training pathway in orthopedic surgery. This was a retrospective cohort study of all orthopedic surgery residents taking the OITE during the 2019-2020 academic year. Comparisons in OITE performance were made with parametric tests. A total of 4407 orthopedic surgery residents were in allopathic (86%) and osteopathic (14%) training programs. There was significant improvement in OITE performance between subsequent postgraduate year (PGY) levels among allopathic residents (P<.001). Among osteopathic residents, OITE performance increased between PGY1 and PGY4 (P<.001) but plateaued between PGY4 and PGY5 (P>.05). At the PGY1 level, osteopathic residents had higher OITE performance than allopathic residents (P<.001), but scores were equivalent at the PGY2 to PGY4 levels (P>.05). At the PGY5 level, allopathic residents had higher OITE performance than osteopathic residents (P<.001). Allopathic medical students scored higher on the Step 1 (248±19 vs 242±17, P<.001) and Step 2 (255±16 vs 250±15, P<.001) board examinations. Medical knowledge increases during orthopedic surgery residency. Disparities exist by training pathway, with osteopathic residents outperforming allopathic residents at the PGY1 level but then underperforming at the PGY5 level. Ultimately, this study provides insights into how resident promotion and training pathway impacts the acquisition of medical knowledge during orthopedic surgery residency. [Orthopedics. 2023;46(6):379-383.].

Academic Faculty Demonstrate Weak Agreement in Evaluating Orthopaedic Surgery Residents

Background

As objective metrics fade, subjective elements of orthopaedic surgery applicants carry increasing importance during recruitment. Academic orthopaedic surgeons believe they can select for high-performing orthopaedic residents. However, can they agree? The purpose of this study was (1) to analyze an academic orthopaedic surgery department to determine whether they agree on which residents performed best during residency and; (2) to correlate preresidency and intraresidency factors with postresidency evaluations of resident performance.

Methods

With Institutional Review Board [IRB] approval, an orthopaedic surgery department completed surveys to evaluate residency performance for 10 years of graduated residents (2012-2022). Faculty determined (1) Post-Residency Class Rank (PRCR)-ranked from the highest (1) to lowest performing resident (5) relative to their class based on faculty perspective of performance-and (2) Rank List Score (RLS)-ranked based off the 5-point AOA SLOR used during recruitment interviews. RLS assessed how likely the department would have graduates in the program again based on residency performance. Free marginal Cohen's kappa statistics assessed faculty inter-rater agreement. Preresidency metrics (United States Medical Licensing Exam [USMLE] 1 and 2 scores, research publications, etc) were correlated with Orthopaedic In-Training Exam (OITE) scores, research productivity, American Board of Orthopaedic Surgery (ABOS) scores, and faculty-derived rankings. Linear regressions with forward variable entry (p < 0.05) were used to determine factors associated with excellent resident performance.

Results

Eighteen of 25 faculty members (72%) provided survey responses evaluating 46 residents. Faculty agreed 37% and 38% of the time for PRCR (kappa 0.26) and RLS (kappa 0.23), respectively. Step 2 score was the only preresidency factor significantly associated with PRCR (p = 0.03, r2 = 0.15) and RLS (p = 0.02, r2 = 0.3). The only intraresidency factor significantly correlated with PRCR (p = 0.002, r2 = 0.50) and RLS (p = 0.01, r2 = 0.39) was PGY-4 OITE score.

Conclusions

An academic orthopaedic surgery department is able to come to a consensus on evaluations of residency performance relative to peers in the same year of training (PRCR) and an objective standard (RLS). Step 2 and Post-Graduate Year (PGY)-4 OITE scores were the only preresidency and intraresidency factors with significant association to higher postresidency, faculty-derived performance scores.

Level of Evidence

III.

Update on Educational Resources and Evaluation Tools for Orthopaedic Surgery Residents

Innovations in orthopaedic resident educational resources and evaluation tools are essential to ensuring appropriate training and ultimately the graduation of competent orthopaedic surgeons. In recent years, there have been several advancements in comprehensive educational platforms within orthopaedic surgery. Orthobullets PASS, Journal of Bone and Joint Surgery Clinical Classroom, and American Academy of Orthopaedic Surgery Resident Orthopaedic Core Knowledge each have their own unique advantages in preparation for the Orthopaedic In-Training Examination and American Board of Orthopaedic Surgery board certification examinations. In addition, the Accreditation Council for Graduate Medical Education Milestones 2.0 and the American Board of Orthopaedic Surgery Knowledge Skills Behavior program each provide objective assessment of resident core competencies. Understanding and using these new platforms will help orthopaedic residents, faculty, residency programs, and program leadership to best train and evaluate their residents.

The Impact of Post-graduate Year and Program Accreditation Status on In-Training Examination Performance in Orthopaedic Surgery

Introduction  The progression of medical knowledge competency during surgical residency training is poorly understood. This study measures the acquisition of medical knowledge as orthopedic surgery residents advance during training and the impact of accreditation status on orthopedic in-training examination (OITE) performance. Methods  Orthopedic surgery residents taking the OITE during 2020 and 2021 were included. Residents were grouped into cohorts by post-graduate year (PGY) and Accreditation Council for Graduate Medical Education (ACGME) accreditation status. Comparisons were made with parametric tests. Results  Eight thousand eight hundred and seventy-one ACGME-accredited residents (89%) and 1,057 non-ACGME-accredited residents (11%) were evenly distributed by the PGY level (range, 19-21%). Residents in both ACGME- and non-ACGME-accredited residency programs had a significant increase in OITE performance at each PGY level (P<0.001). At ACGME-accredited programs, OITE performance increased from PGY1 (51%), PGY2 (59%), PGY3 (65%), PGY4 (68%), and PGY5 (70%) (P<0.001). There were progressively smaller percentage increases in OITE performance during accredited residency training (range, 2-8%), but this increase was linear in non-accredited residency training (range, 4%). At each PGY level, residents at accredited programs outperformed their counterparts at non-accredited programs (P<0.001). Conclusion OITE performance increases during residency training. Among ACGME-accredited residents, performance on the OITE progresses rapidly during junior years and plateaus during senior years. Residents in ACGME-accredited residency programs outperform their counterparts in non-accredited residency programs. More research is needed to understand optimal training environments that promote medical knowledge acquisition during orthopedic surgery residency.

The Evolution of the Musculoskeletal Trauma Section of the Orthopaedic In-Training Examination

INTRODUCTION

In 1963, the American Academy of Orthopaedic Surgeons administered the Orthopaedic In-Training Examination (OITE), the first and longest running yearly medical specialty examination. There have been no recent studies to evaluate the content of the musculoskeletal trauma section of the OITE.

METHODS

We analyzed all questions that were classified by the American Academy of Orthopaedic Surgeons as musculoskeletal trauma from 2012 to 2019. We recorded the number of musculoskeletal trauma questions in each examination, the topics and imaging modalities tested, the references cited, and the taxonomy classification of each question. We extrapolated from a similar musculoskeletal trauma study published in 2011 to create the previous examination cohort for comparison.

RESULTS

For the current cohort, the average number of musculoskeletal trauma questions was 43.5 questions per examination (18.4%). The most frequently tested topics were proximal tibia fractures, pediatric trauma, hip fractures, and diaphyseal femur fractures, respectively. In previous examinations, questions from T1 and 2 were tested significantly more frequently compared with the current examinations (P < 0.001 and P = 0.02, respectively). In the current cohort, T3 questions were tested significantly more frequently than previous examinations (P = 0.001). Previous examinations had significantly more questions without an image (36 questions per year versus 25 questions per year, P < 0.001). In current versions of the examination, radiographs are tested significantly more frequently than other imaging modalities (P < 0.001).

DISCUSSION

The musculoskeletal trauma section of the OITE has evolved. To improve and focus study efforts, residents may use this study as a guide when preparing for the examination.

Which Application Factors Are Associated With Outstanding Performance in Orthopaedic Surgery Residency?

BACKGROUND

Identifying ideal candidates for orthopaedic surgery residency is difficult. Data available for applicant selection are evolving; preclinical grades and the Alpha Omega Alpha (AOA) honors society are being phased out at some medical schools. Similarly, three-digit United States Medical Licensing Examination (USMLE) Step 1 scores have been eliminated. There is renewed interest in improving resident selection to provide a diverse, comprehensive educational opportunity that produces orthopaedic surgeons who are prepared for practice.

QUESTIONS/PURPOSES

We sought to identify whether (1) academic achievements, (2) letters of recommendation, (3) research activity, and (4) miscellaneous factors available on Electronic Residency Application Service (ERAS) applications were associated with outstanding residency performance.

METHODS

Ten faculty members (22% of all full-time faculty) with extensive educational involvement for at least 7 years, whose expertise covered all subspecialty departments at an urban, academic orthopaedic surgery residency program, were given an anonymous survey on the performance of the four most recent classes of residency graduates (24 residents). This survey was developed due to the lack of a validated residency outcomes tool or objective metrics for residency performance. The evaluated criteria were decided upon after discussion by a relatively large group of academic orthopaedic surgeons considering the factors most important for graduating orthopaedic residents. The faculty were selected based on their long-term knowledge of the residency, along with their diversity of specialty and backgrounds; there were no nonresponders. Faculty graded each resident on a scale from 1 to 10 (higher is better) on six criteria: surgical technical skills, research productivity, clinical knowledge, professionalism, personality, and fellowship match. The mean of the faculty ratings made by all faculty for all six criteria was calculated, producing the overall residency performance score. Factors available on each resident's ERAS application were then correlated with their overall residency performance score. Categorical ERAS factors, including AOA status, five or more honors in core clerkships, at least three exceptional letters of recommendation, collegiate athletics participation, expertise with a musical instrument, and research (6-year) track residents, were correlated with overall residency performance score via point biserial analysis. Continuous ERAS factors including USMLE Step 1 and Step 2 scores, number of publications before residency, number of research years before residency, medical school ranking, and number of volunteer experiences were correlated with overall residency performance score via Pearson correlation. USMLE Step 1 three-digit scores were evaluated despite their recent elimination because of their historic importance as a screening tool for residency interviews and for comparison to USMLE Step 2, which retains a three-digit score. Application factors with a p < 0.2 on univariate analysis (five or more honors in core clerkships, at least three exceptional letters of recommendation, research track residents) were included in a stepwise linear regression model with "overall residency performance score" as the outcome variable. All p values < 0.05 were considered significant.

RESULTS

The mean overall residency performance score was 7.9 ± 1.2. Applicants with at least five honors grades in core clerkships had overall residency performance scores 1.2 points greater than those of their peers (95% confidence interval (CI) 0.3 to 2.0; p = 0.01, Cohen ƒ 2 = 0.2, representing a small effect size). ERAS applications including at least three exceptional letters of recommendation were associated with a 0.9-point increase in residency performance (95% CI 0.02 to 1.7; p = 0.046, Cohen ƒ 2 = 0.1, representing a small effect size). Participation in the residency research (6-year) track was associated with a 1-point improvement in residency performance (95% CI 0.1 to 1.9; p = 0.03, Cohen ƒ 2 = 0.2, again, representing a small effect size). Together, these three factors accounted for 53% of the variance in overall residency performance score observed in this study.

CONCLUSION

Past clinical excellence, measured by core clerkship grades and exceptional letters of recommendation, is associated with slightly improved overall orthopaedic residency performance scores. Applicants meeting both criteria who also complete a research track residency may perform substantially better in residency than their counterparts, as these three factors accounted for half of all the variance observed in the current study. Although minimum requirements are necessary, traditionally used screening factors (such as USMLE scores, AOA status, medical school rank, and number of publications) may be of less utility in identifying successful future residents than previously thought.

LEVEL OF EVIDENCE

Level III, therapeutic study.

Comparison of Sports Medicine Questions on the Orthopaedic In-Training Examination Between 2009 and 2012 and 2017 and 2020 Reveals an Increasing Number of References

Purpose

To provide an updated analysis of the sports medicine section of the Orthopedic In-Training Examination (OITE).

Methods

A cross-sectional review of OITE sports medicine questions from 2009 to 2012 and 2017-2020 was performed. Subtopics, taxonomy, references, and use of imaging modalities were recorded and changes between the time periods were analyzed.

Results

The most tested sports medicine subtopics included ACL (12.6%), rotator cuff (10.5%), and throwing injuries to the shoulder (7.4%) in the early subset, while ACL (10%), rotator cuff (6.25%), shoulder instability (6.25%), and throwing injuries to the elbow (6.25%) were the most common in the later subset. The American Journal of Sports Medicine (28.3%) was the most cited journal referenced from 2009 to 2012, while The Journal of the American Academy of Orthopaedic Surgeons (17.5%) was most referenced in questions from 2017 to 2020. The number of references per question increased from the early to the late subset (P < .001). There was a trend toward an increased taxonomy type one questions (P = .114), while type 2 questions had a decreased trend (P = .263) when comparing the new subset to the early group.

Conclusion

When comparing sports medicine OITE questions from 2009 to 2012 and 2017 to 2020, there was an increase in the number of references per question. Subtopics, taxonomy, lag time, and use of imaging modalities did not show statistically significant changes.

Clinical Relevance

This study provides a detailed analysis of the sports medicine section of the OITE, which can be used by residents and program directors to direct their preparation for the annual examination. The results of this study may help examining boards align their examinations and provide a benchmark for future studies.

Advocacy for Diversity in Hand Surgery

Diversity in the Hand Surgery workforce improves the quality of care delivered, advances a wider variety of innovation within the field and leads to higher patient satisfaction, greater access to care and patient adherence to advice. An understanding of the data makes a compelling argument for change. Advocacy is necessary to stop the "leaky pipeline" of the loss of diversity in more senior and leadership roles. Hand surgeons who are both women and from underrepresented minority groups are especially vulnerable to bias from the health-care system, with focused support and mentoring required throughout their training and career.

The US Residency Selection Process After the United States Medical Licensing Examination Step 1 Pass/Fail Change: Overview for Applicants and Educators

The United States Medical Licensing Examination (USMLE) Step 1, arguably the most significant assessment in the USMLE examination series, changed from a 3-digit score to a pass/fail outcome in January 2022. Given the rapidly evolving body of literature on this subject, this paper aims to provide a comprehensive review of the historical context and impact of this change on various stakeholders involved in residency selection. For this, relevant keyword-based searches were performed in PubMed, Google Scholar, and Scopus to identify relevant literature. Given the unique history of USMLE Step 1 in the US residency selection process and the score's correlation with future performance in board-certifying examinations in different specialties, this scoring change is predicted to significantly impact US Doctor of Medicine students, US Doctor of Osteopathic Medicine students, international medical graduates, and residency program directors, among others. The significance and the rationale of the pass/fail change along with the implications for both residency applicants and educators are also summarized in this paper. Although medical programs, academic institutions, and residency organizing bodies across the United States have swiftly stepped up to ensure a seamless transition and have attempted to ensure equity for all, the conversion process carries considerable uncertainty for residency applicants. For educators, the increasing number of applications conflicts with holistic application screening, leading to the expected greater use of objective measures, with USMLE Step 2 Clinical Knowledge likely becoming the preferred screening tool in lieu of Step 1.

On the Predictive Power of the In-Training Examination for the Certification of Anesthesiologist Assistants

OBJECTIVES

In-training examinations (ITEs) are a popular teaching tool for certification programs. This study examines the relationship between examinees' performance on the National Commission for Certification of Anesthesiologist Assistants (NCCAA) ITE and the high-stakes NCCAA Certification Examination.

METHODS

We utilized a mixed-methods approach in our study. Before estimating the models for the predictive validity study, a series of interviews with program directors were conducted to discuss the role of the ITE in students' education. Multiple linear regression analysis was then used to assess the strength of the relationship between the ITE and Certification Examination scores, while considering the percentage of program examinees completed in their anesthesiologist assistant program between their ITE and Certification Examination attempts. Logistic regression analysis was used to estimate the probability of passing the Certification Examination as a function of ITE score.

RESULTS

Interviews with program directors confirmed that the ITE provided a valuable testing experience for students and highlighted the areas where students need to focus. Moreover, both the ITE score and the percentage of the program between exams were deemed statistically significant predictors for Certification Examination scores. The logistic regression model indicated that higher scores on the ITE implied a higher probability of passing the Certification Examination.

CONCLUSION

This research demonstrated the high predictive validity of the ITE examination scores in predicting the Certification Examination outcomes. Together with the proportion of the program covered between exams, the variables explain a significant amount of variability in Certification Examination scores. The ITE feedback helped students assess their preparedness and better focus their studies for the high-stakes certification examination for the profession.

Perspectives of Orthopedic Surgery Program Directors on the USMLE Step 1 Scoring Change

The orthopedic surgery residency application process is highly competitive, and residency programs rely on objective measures, such as the United States Medical Licensing Examination (USMLE) Step 1 score, to assist in applicant selection. To deemphasize standardized test scores and improve student well-being, governing bodies have elected to change the Step 1 examination to a pass/fail grading system beginning in 2022. Given the utility of Step 1 in the orthopedic surgery residency application process, this change will significantly affect how applicants are assessed. To determine how this change will affect the process, we developed and validated a 19-item anonymous electronic survey and invited program directors (PDs) from orthopedic surgery residency programs accredited by the Accreditation Council for Graduate Medical Education to respond. The 86 surveys that were completed represented a 43.2% response rate (86 of 199). Only 4.7% of orthopedic surgery PDs believe that this change is a good idea, and 91.7% of PDs believe that the change will make it more difficult to compare applicants objectively. In addition, 90.7% of PDs report that they will increase emphasis on the Step 2 CK (clinical knowledge) examination in comparing residency applicants for their program, with 90.4% of PDs reporting that they will require students to submit their Step 2 scores via the Electronic Residency Application Service. These results indicate that changing the Step 1 scoring to pass/fail is unpopular among orthopedic surgery PDs. Further, losing the numeric Step 1 score will increase reliance on Step 2 scores and emphasis on less objective measures, such as where an applicant attended medical school. [Orthopedics. 2022;45(5):e257-e262.].

Analysis of Current Orthopedic Surgery Residents and Their Prior Medical Education: Does Medical School Ranking Matter in Orthopedic Surgery Match?

OBJECTIVE

The purpose of this study was to determine the strength of the association between medical school ranking and orthopedic surgery residency ranking using the current cohort of orthopedic surgery residents.

DESIGN

We obtained a list of accredited programs from Doximity for orthopedic surgery residency programs and U.S. News & World Report for medical schools. Each orthopedic surgery residency program webpage was evaluated for the presence of an orthopedic surgery residency roster. For each resident, the medical school attended, allopathic or osteopathic degree, and year of post-graduate training was recorded. Orthopedic surgery residency programs and medical schools were assigned to one of four tiers for each based on their respective ranking. Descriptive statistics, Chi squared tests and Pearson residuals were used to analyze the association of orthopedic surgery residency tier and medical school tier. Post-hoc pairwise comparisons were performed utilizing the Bonferroni correction to account for 16 tests, correcting the significance level to p = 0.003.

SETTING

187 orthopedic surgery residency program webpages.

PARTICIPANTS

4123 orthopedic surgery residents.

RESULTS

There was a significant association between medical school tier and orthopedic surgery residency tier (X² [9] = 1214.78, p < 0.001). The post-hoc residual values were statistically significant for 75% (12/16) of tests performed. The majority of Tier 1 orthopedic surgery residents 50.5% (800/1585) attended a Tier 1 medical school. The strongest positive association exists between Tier 1 medical students attending Tier 1 residencies (residual = 23.978, p < 0.001). The strongest negative association with Tier 4 residencies was with Tier 1 medical schools (residual= -15.656, p< 0.001).

CONCLUSIONS

Medical school ranking is an important consideration for prospective orthopedic surgery applicants and may become more important with less objective measures of academic performance such as United States Medical Licensing Examination Step 1.

LEVEL OF EVIDENCE

Observational.

Cross-Sectional Analysis of Foot and Ankle Questions on the Orthopaedic In-Training Examination: A Guide for Resident Preparation

Background:

The Orthopaedic In-Training Examination (OITE) is a standardized examination administered annually to orthopaedic surgery residents. The examination is designed to evaluate resident knowledge and academic performance of residency programs.

Methods:

All OITE foot and ankle questions from 2009 through 2012 and 2017 through 2020 were analyzed. Subtopics, taxonomy, references, and use of imaging modalities were recorded.

Results:

There were a total of 167 foot and ankle (F&A)–related questions across 8 years of OITE examinations. Trauma remained the most commonly tested subtopic of F&A across both subsets, followed by rehabilitation, tendon disorders, and arthritis. We found an increase in questions related to arthritis (P = .05) and a decrease of questions related to the diabetic foot (P = .02). Taxonomy 3 questions constituted 49.5% of F&A questions from 2009 through 2012 compared with 44.7% of questions from 2017 to 2020 (P = .54). Radiography was the most commonly used imaging modality in both subsets. From 2009 to 2012, 63.6% of questions included a radiograph compared with 76.5% in 2017 through 2020 (P = .13). FAI (Foot & Ankle International), JAAOS (Journal of the American Academy of Orthopaedic Surgeons), and JBJS (The Journal of Bone and Joint Surgery) were the most commonly cited journals, making up more than 50% of total citations. Citations per question increased from 2.20 to 2.42 from 2009-2012 to 2017-2020 (P = .01). The average lag time in the early subset was 8.2 years and 8.9 years in the later subset.

Conclusion:

This study provides a detailed analysis of the F&A section of the OITE. Use of this analysis can provide residents with a guide on how to better prepare for the OITE examination.

Level of Evidence:

Level IV, cross-sectional review of Orthopaedic In-Training Examination questions

Innovation in Resident Selection: Life Without Step 1

INTRODUCTION

The announcement of Step 1 shifting to a Pass/Fail metric has prompted resident selection committees (RSCs) to pursue objective methods of evaluating prospective residents. Regardless of the program's specialty or affiliated hospital/school, RSCs universally aim to recognize and choose applicants who are an "optimal fit" to their programs.¹ An optimal fit can be defined as a candidate who thrives in the clinical and academic setting, both contributing to and benefiting from their respective training environments.

OBJECTIVE

The objective of this scoping review is to evaluate alternative, innovative methods by which RSCs can evaluate applicants and predict success during residency. Objective methods include: Step 2 scores, Traditionally Used Metrics (core clerkship scores), interview performance, musical talent, sports involvement, AOA membership, research publications, unprofessional behavior, Dean's letters, Rank list, judgement testing, and specialty-specific shelf exams.^13-15.

METHODS

A scoping review was performed in compliance with the guidelines indicated by the PRISMA Protocol for scoping review.¹⁸ 9308 results were identified in the original PubMed search for articles with the key words "Resident Success". Abstract screening and application of inclusion and exclusion criteria yielded 97 articles that were critically appraised via review of full manuscript.

RESULTS

Of the articles that focused on personality traits, situational judgement testing, and specialty specific pre-assessment, all of them demonstrated some level of predictability for resident success. Standardized Letter of Recommendations, Traditionally Used Metrics, and STEP 2 did not show a unanimous consensus in demonstrating predictability of a resident's success, this is because some articles suggested predictability and some articles disputed predictability.

CONCLUSION

The authors found personality traits, situational judgement testing, and specialty specific assessments to be predictive in selecting successful residents. Further research should aim to analyze exactly how RSCs utilize these assessment tools to aid in screening their large and competitive applicant pools to find residents that will be successful in their program.

Do Orthopaedic In-Training Examination Scores Predict the Likelihood of Passing the American Board of Orthopaedic Surgery Part I Examination? An Update With 2014 to 2018 Data

INTRODUCTION

Previous research shows a correlation in performance between the Orthopaedic In-Training Examination (OITE) and the American Board of Orthopaedic Surgery (ABOS) Certifying Examination Part I; however, these studies are not current, with the most recent data from 2009. The purpose of this study was to update the relationship between the OITE and ABOS Part I scores with the five most recent nationwide cohorts of Part I scores and their corresponding OITE scores.

METHODS

The American Academy of Orthopaedic Surgeons provided the results for each resident taking the OITE from 2013 to 2017. The ABOS provided the results for each resident taking the Part I examination from 2014 to 2018. These two datasets were matched at the individual level and analyzed.

RESULTS

Between 2014 and 2018, 3,502 first-time test-takers were present for the ABOS Part I Examination. A 96.6% pass rate was noted (3,383 of 3,502). A statistically significant correlation between the OITE score and ABOS Part I score was observed at all levels of training: postgraduate year (PGY) 1 r = 0.380, PGY2 r = 0.463, PGY3 r = 0.498, PGY4 r = 0.504, and PGY5 r = 0.504 (P < 0.001 for all correlations).

CONCLUSION

The OITE scores continue to correlate with the ABOS scores and the pass rate for the ABOS with moderate strength correlation. However, although a correlation of the examinations exists when evaluated as a group, the predictive value of the OITE for passing the Part I examination for any specific individual is far from perfect. Individuals with PGY5 OITE percentile scores less than 10 pass the Part I examination, and individuals with PGY5 OITE percentile scores greater than 90 fail the Part I examination.

Resident Selection: The Role of Assessment of Emotional Intelligence and Grit

SUMMARY

The ability of our resident selection process to identify individuals who will ultimately become competent plastic surgeons is crucial to the specialty's future. Current criteria in use are not productive of that outcome. The presence of emotional intelligence and the element of grit have been incorporated in business and the military as factors to be evaluated in potential candidates. Plastic surgery should initiate an investigation of inclusion of a similar assessment of resident applicants.

Analysis of the OITE Oncology Section: An Updated Review of Years 2013 to 2019

OBJECTIVES

The importance of the Orthopaedic In-Training Examination (OITE) is well documented. The purpose of this study is to provide an updated analysis of the oncology section compared to the review by Frassica et al. from 2002 to 2006, as a means to provide insight into more focused resident study.

DESIGN

This study is a retrospective database review using published OITE exams from years 2013 to 2019. Methods were based off a previous review by Frassica et al. where oncology-related questions were analyzed for underlying diagnosis, benign versus malignant condition, imaging and histology provided, and genes/translocations tested. Questions were classified by category and taxonomy.

RESULTS

Oncology-related questions per exam ranged from 19 (7%) to 23 (8.4%) of total questions, which is fewer than years 2002 to 2006. Twenty malignant and 27 benign entities were tested with malignant conditions tested at a higher rate of 1.3:1 versus benign. Eighteen combinations of imaging modalities were provided for analysis versus 11 from 2002 to 2006. With regard to taxonomy, the average number of questions per classification ranged from 2.4 to 5.4. The least common classification tested was treatment modality and the most commonly tested classification was treatment from diagnosis (taxonomy 3). Previously, tumor knowledge (taxonomy 1) was most tested with an average of 8.8 questions per exam. 12 questions were directly related to genes and translocations across all years.

CONCLUSIONS

Oncology questions made up a smaller percentage of the OITE exam than previous years, although more conditions were tested using more imaging combinations, necessitating a wider range of knowledge. However, malignant conditions continue to be tested more commonly. There has been a shift toward Taxonomy 3 level questions, indicating a higher level of thought processing required from residents as opposed to recall. Additionally, genes and translocations became more commonly tested throughout the most recent analysis, indicating a focus for future years of study.

Orthopaedic In-Training Examination: History, Perspective, and Tips for Residents

Introduced in 1963, the orthopaedic in-training examination (OITE) is a standardized, national test administered annually to orthopaedic residents by the American Academy of Orthopaedic Surgeons. The examination consists of 275 multiple-choice questions that cover 11 domains of orthopaedic knowledge, including basic science, foot and ankle, hand, hip and knee, oncology, pediatrics, shoulder and elbow, spine, sports medicine, trauma, and practice management. The OITE has been validated and is considered predictive of success in both orthopaedic surgery residency and on the American Board of Orthopaedic Surgery part I examination. This article provides a historical overview of the OITE, details its current structure and scoring system, and reviews currently available study materials. For examination preparation, the residents are encouraged to (1) start the examination preparation early, (2) practice on old OITE or self-assessment examination questions, (3) focus on the questions answered incorrectly, (4) focus on comprehension over memorization, and (5) recognize and avoid burnout. Finally, the residents should have a systemic way of approaching each multiple-choice question, both during practice and on the actual examination.

Updated Analysis of the Oncology Section of the OITE from 2007 to 2019

The Orthopaedic In-Training Exam (OITE) is administered annually to orthopedic surgery residents to assess their medical knowledge. The authors provide a comprehensive review of the orthopedic oncology portion of the exam in order to aid residents in preparation for future in-training and licensing examinations as well as to help guide oncology residency education curriculum. All of the orthopedic oncology questions on the OITE from 2007 to 2019 were reviewed. Analysis included (1) the number of oncology questions each year, (2) question topic, (3) question taxonomy (knowledge versus interpretation), (4) the type of imaging modalities (radiological, histological), (5) most commonly cited references, and (6) level of evidence. Descriptive statistics were utilized to compare means between variables. From 2007 to 2019, there was a total of 292 tumor-related questions with a mean of 22.5 tumor-related questions (range 19-28) per year. Of the questions, 54.8% pertained to malignant tumors and 45.2% to benign tumors. Assessment of question taxonomy showed that 79.8% of questions required interpretation of imaging and analysis of the information provided versus 20.2% of questions being knowledge recall type. Of the questions, 76.7% required interpretation of radiological images, pathological images, or both. Orthopaedic Knowledge Update, Journal of the American Academy of Orthopaedic Surgeons, and Journal of Bone and Joint Surgery were the three most commonly cited question sources. Only 29 (9.84%) oncology questions over the past 13 years have been supported by level I or II sources of evidence. Better understanding of the OITE make-up, question distribution, and number and style of question, reference sources can improve an orthopedic residents' performance as well as better guide educational curriculum to prepare residents in their orthopedic oncology education.

A Systematic Review of the Relationship Between In-Training Examination Scores and Specialty Board Examination Scores

BACKGROUND

In-training examinations (ITEs) are intended for low-stakes, formative assessment of residents' knowledge, but are increasingly used for high-stake purposes, such as to predict board examination failures.

OBJECTIVE

The aim of this review was to investigate the relationship between performance on ITEs and board examination performance across medical specialties.

METHODS

A search of the literature for studies assessing the strength of the relationship between ITE and board examination performance from January 2000 to March 2019 was completed. Results were categorized based on the type of statistical analysis used to determine the relationship between ITE performance and board examination performance.

RESULTS

Of 1407 articles initially identified, 89 articles underwent full-text review, and 32 articles were included in this review. There was a moderate-strong relationship between ITE and board examination performance, and ITE scores significantly predict board examination scores for the majority of studies. Performing well on an ITE predicts a passing outcome for the board examination, but there is less evidence that performing poorly on an ITE will result in failing the associated specialty board examination.

CONCLUSIONS

There is a moderate to strong correlation between ITE performance and subsequent performance on board examinations. That the predictive value for passing the board examination is stronger than the predictive value for failing calls into question the "common wisdom" that ITE scores can be used to identify "at risk" residents. The graduate medical education community should continue to exercise caution and restraint in using ITE scores for moderate to high-stakes decisions.

The Change of USMLE Step 1 to Pass/Fail: Perspectives of the Surgery Program Director

OBJECTIVE

This study sought to evaluate the perspectives of surgical program directors regarding the change of USMLE Step 1 to pass/fail grading.

DESIGN

Validated electronic survey.

SETTING

Vanderbilt University Medical Center Department of Plastic Surgery.

PARTICIPANTS

Program directors of all ACMGE-accredited General Surgery, Integrated Vascular Surgery, Integrated Thoracic Surgery, and Integrated Plastic Surgery residency programs.

RESULTS

The overall response rate was 55.5%. Most PDs (78.1%) disagreed with the scoring change. Only 19.6% believe this change will improve medical student well-being. For 63.5% of PDs, medical school pedigree will become more important, and 52.7% believe it will place international medical graduates at a disadvantage. Only 6.2% believe Step 2 CK should also be pass/fail, while 88.7% will increase the weight of Step 2 CK and 88.4% will now require Step 2 CK score submission with the electronic residency application service.

CONCLUSIONS

While well-intentioned, changing USMLE Step 1 to pass/fail may have unintended consequences and may disadvantage certain groups of applicants. The emphasis on Step 1, and resulting test-taking apprehension, will likely shift to Step 2 CK. Proponents of equitable evaluation should direct their efforts toward increasing, not decreasing, the number of objective measures available for student assessment.

Three Decades Later: A Scoping Review of the Literature Related to the United States Medical Licensing Examination

PURPOSE

To conduct a scoping review of the timing, scope, and purpose of literature related to the United States Medical Licensing Examination (USMLE) given the recent impetus to revise USMLE scoring.

METHOD

The authors searched PubMed, PsycInfo, and ERIC for relevant articles published from 1990 to 2019. Articles selected for review were labeled as research or commentaries and coded by USMLE Step level, sample characteristics (e.g., year(s), single/multiple institutions), how scores were used (e.g., predictor/outcome/descriptor), and purpose (e.g., clarification/justification/description).

RESULTS

Of the 741 articles meeting inclusion criteria, 636 were research and 105 were commentaries. Publication totals in the past 5 years exceeded those of the first 20 years.Step 1 was the sole focus of 38%, and included in 84%, of all publications. Approximately half of all research articles used scores as a predictor or outcome measure related to other curricular/assessment efforts, with a marked increase in the use of scores as predictors in the past 10 years. The overwhelming majority of studies were classified as descriptive in purpose.

CONCLUSIONS

Nearly 30 years after the inception of the USMLE, aspirations for its predictive utility are rising faster than evidence supporting the manner in which the scores are used. A closer look is warranted to systematically review and analyze the contexts and purposes for which USMLE scores can productively be used. Future research should explore cognitive and noncognitive factors that can be used in conjunction with constrained use of USMLE results to inform evaluation of medical students and schools and to support the residency selection process.

Associations Between Medical Education Assessments and American Board of Family Medicine Certification Examination Score and Failure to Obtain Certification

PURPOSE

Family medicine residency programs can be cited for low pass or take rates on the American Board of Family Medicine (ABFM) certification examination, and the relationships among standardized medical education assessments and performance on board certification examinations and eventual board certification have not been comprehensively studied. The objective of this study was to evaluate the associations of all required standardized examinations in medical education with ABFM certification examination scores and eventual ABFM certification.

METHOD

All graduates of U.S. MD-granting family medicine residency programs from 2008 to 2012 were included. Data on ABFM certification examination score, ABFM certification status (as of December 31, 2014), Medical College Admission Test (MCAT) section scores, undergraduate grade point average, all United States Medical Licensing Examination (USMLE) Step scores, and all ABFM in-training examination scores were linked. Nested logistic and linear regression models, controlling for clustering by residency program, determined associations between assessments and both certification examination scores and board certification status. As many international medical graduates (IMGs) do not take the MCAT, separate models for U.S. medical graduates (USMG) and IMGs were run.

RESULTS

The study sample was 15,902 family medicine graduates, of whom 92.1% (14,648/15,902) obtained board certification. In models for both IMGs and USMGs, the addition of more recent assessments weakened the associations of earlier assessments. USMLE Step 2 Clinical Knowledge was predictive of certification examination scores and certification status in all models in which it was included.

CONCLUSIONS

For family medicine residents, more recent assessments generally have stronger associations with board certification score and status than earlier assessments. Solely using medical school admissions (grade point average and MCAT) and licensure (USMLE) scores for resident selection may not adequately predict ultimate board certification.

Analysis of the Sports Medicine Section of the Orthopaedic In-Training Examination: Improvements in Levels of Evidence and Question Taxonomy Over a 12-Year Period

There has been a recent shift within the orthopedic literature to publish articles with higher levels of evidence. In this investigation, the trends in question taxonomy and the levels of evidence of the references for sports medicine questions on the Orthopaedic In-Training Examination (OITE) during a 12-year period were evaluated. Sports medicine questions were obtained from the OITEs administered between 2005 and 2007 and between 2014 and 2016. The taxonomy of each question was characterized, and levels of evidence for all references were assigned using American Academy of Orthopaedic Surgeons guidelines. Question taxonomy and article levels of evidence from 2005 to 2007 were compared with those from 2014 to 2016. Sports medicine questions comprised 8% of the OITEs in both examination groups. The questions from 2014 to 2016 had a higher mean taxonomic level (2.26 vs 1.52, P=.0001) and a greater proportion of studies with high levels of evidence (levels 1 and 2) (21% vs 10%, P=.027). However, references with low levels of evidence or nonprimary resources made up 82.2% and 68.5% of the total references on the older and more recent examinations, respectively. References from 2014 to 2016 were, on average, 2 years older than those from earlier examinations. This study indicated that sports medicine questions on recent OITEs cite references of higher levels of evidence and contain higher taxonomic question structure than examinations 10 to 12 years ago. However, the majority of questions still cite articles with low levels of evidence or nonprimary sources. These findings can be used to guide resident education and continue improvements in the selection of references for questions on the Sports Medicine section of the OITE. [Orthopedics. 2020;43(5):e460-e464.].

Evaluating Urology Residency Applications: What Matters Most and What Comes Next?

PURPOSE OF REVIEW

In light of the announcement that the United States Medical Licensing Examination Step 1 exam will transition to pass/fail reporting, we reviewed recent literature on evaluating residency applicants with a focus on identifying objective measurements of applicant potential.

RECENT FINDINGS

References from attending urologists, Step 1 scores, overall academic performance, and research publications are among the most important criteria used to assess applicants. There has been a substantial increase in the average number of applications submitted per applicant, with both applicants and residency directors indicating support for a cap on the number of applications that may be submitted. Additionally, there are increasing efforts to promote diversity with the goal of improving care and representation in urology. Despite progress in standardizing interview protocols, inappropriate questioning remains an issue. Opportunities to improve residency application include promoting diversity, enforcing prohibitions of illegal practices, limiting application numbers, and finding more transparent and equitable screening measures to replace Step 1.

The Educational Impact of a Fellowship-trained Orthopaedic Oncologist

INTRODUCTION

Musculoskeletal oncology is a subspecialty of orthopaedics with few fellowship-training locations. Although orthopaedic oncologists comprise a minority within the field of orthopaedic surgery, most work at academic centers and serve in leadership roles with notable impact on patients and the training of residents. This article investigates the objective impact orthopaedic oncologists have regarding resident operative case volume and performance on in-training examinations.

METHODS

The William Beaumont Army Medical Center and Texas Tech University Health Sciences Center of El Paso combined orthopaedic residency program's case logs and Orthopaedic In-Training Examination (OITE) scores between 2013 and 2018 were reviewed. This provided 3 academic years of data before and after an orthopaedic oncology faculty member arrived in 2016. The case volume and OITE examination performance before and after the addition of the orthopaedic oncology faculty member were compared.

RESULTS

After the addition of an orthopaedic oncology faculty member, a significant increase was observed in the program's OITE overall correctly answered questions (171.30 versus 181.03, P = 0.004) and oncology subsection percentile (56th to the 66th percentile, P = 0.038). An increase was also observed in resident oncology case volume from 29 oncology cases per year to 138 cases on average (P = 0.022).

DISCUSSION

The addition of a fellowship-trained orthopaedic oncologist results in increased exposure to orthopaedic oncology cases and improved resident performance on the OITE. This may correlate to improved American Board of Orthopaedic Surgeons Part I pass rates and improved overall resident satisfaction.

The Impact of COVID-19 on the Orthopaedic Surgery Residency Application Process

Over recent months, coronavirus disease 2019 (COVID-19) has swept the world as a global pandemic, largely changing the practice of medicine as it was previously known. Physician trainees have not been immune to these changes-uncertainty during this time is undeniable for medical students at all levels of training. Of particular importance is the potential impact of COVID-19 on the upcoming residency application process for rising fourth-year students; a further source of added complexity in light of the newly integrated allopathic and osteopathic match in the 2020 to 2021 cycle. Owing to the impact COVID-19 could have on the residency match, insight regarding inevitable alterations to the application process and how medical students can adapt is in high demand. Furthermore, it is very possible that programs will inquire about how applicants spent their time while not in the hospital because of COVID-19, and applicants should be prepared to provide a meaningful answer. Although competitive at a basal level, the complexity of COVID-19 now presents an unforeseen, superimposed development in the quest to match. In this article, we aim to discuss and provide potential strategies for navigating the impact of COVID-19 on the residency application process for orthopaedic surgery.

Do USMLE steps, and ITE score predict the American Board of Internal Medicine Certifying Exam results?

BACKGROUND

To evaluate if United States Medical Licensing Examination (USMLE) Step 1, USMLE Step 2 CK, USMLE Step 3, and residency third-year in-service training exam (ITE) scores predict the results of American Board of Internal Medicine Certifying Exam (ABIM-CE).

METHODS

We performed a retrospective review of USMLE Step 1, USMLE Step 2 CK, USMLE Step 3, third-year residency ITE scores and ABIM-CE results of IM residents at our residency program from 2004 through 2017. Statistical analysis was perfrormed using Pearson correlation coefficient, and logistic regression to assess the relationship between USMLE Step 1, USMLE Step 2CK, USMLE Step 3, 3rd year ITE scores and ABIM-CE results. We used Multivariate logistic regression to predict pass or fail results in ABIM-CE based on USMLE and third-year ITE test scores controlling for other covariates.

RESULTS

Among 114 Internal Medicine MD residents included in the study, 92% (n = 105) passed the ABIM-CE. The OR of passing ABIM-CE was 2.70 (95% CI = 1.38-5.29), 2.31 (95% CI = 1.33-4.01), and 1.63 (95% CI = 0.81-3.29) with a ten-point increase in USMLE Step 1, USMLE Step 2 CK and USMLE Step 3 scores respectively. The OR of ABIM-CE passing chance was 2.96 (95% CI = 0.95-9.20), with a ten-point increase in the average score of the above three exams. A 5 % increase in ITE percentage raised the likelihood of passing ABIM-CE (OR 2.92, 95% CI 1.15-7.38). All residents who failed ABIM-CE had Step 1 scores < 220. Among 31 residents with Step 2 CK score < 220, 20% (n = 6) failed ABIM. Similarly, 9% of residents with USMLE Step 3 score < 220 failed ABIM-CE; all residents who failed had scored < 220. The probability curve predicted that the chance of passing ABIM- CE was around 80% with USMLE scores greater than 200 and increased to almost 100% with USMLE scores of 250 or more.

CONCLUSION

USMLE Step 1, USMLE Step 2 CK, and third-year ITE scores can predict the chances of passing ABIM-CE. The third-year ITE score has a higher preditive value compared to USMLE Step 1 and USMLE Step 2 scores. USMLE Step 1 scores more predictive of ABIM-CE results compared to USMLE Step 2CK scores. Thus, residency programs can identify internal medicine residents at risk of failing ABIM-CE and formulate interventions at an early stage during residency training. Measures such as enrolling them in question banks or board review courses can be helpful in improving their chances of passing ABIM-CE.

Matching in Orthopaedic Surgery

In 2016, 1,137 fourth year medical students submitted applications for orthopedic surgery residency positions. Students applied to an average of 79 programs, resulting in in a total of 89,846 applications being submitted for 727 first year residency positions. This ratio of 124 applications per position is two SDs above the mean relative to other medical specialties. The average applicant for orthopaedic surgery residency attends 2.4 away rotations, as attending 2 away rotations increases an applicant's odds of matching, and submits 83 applications. This excessive number of applications overburdens programs, subjects applicants to considerable costs, and diminishes the quality of fit between interviewees and programs. Eighty-three percent of program directors use step 1 United States Medical Licensing Examination scores as a screening tool to decrease the number of applications necessary for review. The average matched applicant attended 11.5 interviews, and Step 1 scores, research productivity, and Alpha Omega Alpha (AOA) status can be used to predict the number of applications necessary to obtain 12 interviews. AOA membership has the strongest influence on interview yield. Applicants report spending an average of approximately $7,000 on the interview process, and 72% borrow money to cover these costs. Post-interview contact, although forbidden by the National Resident Matching Program , has been reported by 60% to 64% of applicants.

Orthopaedic Resident Burnout Is Associated with Poor In-Training Examination Performance

BACKGROUND

Resident burnout-the state of exhaustion, maladaptive detachment, and low sense of accomplishment-is a widely documented phenomenon that affects between 27% and 75% of residents in the United States. To our knowledge, no previous study has examined the relationship between resident burnout and performance on the Orthopaedic In-Training Examination (OITE). The current investigation sought to evaluate whether an association exists between indices of orthopaedic surgery resident burnout as assessed by the Maslach Burnout Inventory (MBI) and performance on the OITE.

METHODS

In a cross-sectional study of the orthopaedic surgery residents at a single large academic institution, the MBI was completed by all trainees in May 2016. The results of the 2016 OITE were documented for each resident, including the percentage of correctly answered questions and OITE percentile ranking. To control for individual test-taking skills, United States Medical Licensing Examination (USMLE) Step-1 and Step-2 scores also were documented for each resident. The relationship between the MBI subscale scores and OITE performance was evaluated.

RESULTS

The analysis included 100% of the 62 orthopaedic surgery residents in training at our institution. Sixteen (25.8%) of the residents experienced at least moderate emotional exhaustion, while 32 (51.6%) of the residents experienced at least moderate depersonalization and 8 (12.9%) of the residents experienced a moderate sense of impaired personal accomplishment. Postgraduate year (PGY)-2 residents had the highest emotional exhaustion and depersonalization scores compared with residents in other years of training. Each of the 3 MBI indices of burnout was associated with worse OITE performance when controlling for general test-taking ability.

CONCLUSIONS

In this study of orthopaedic surgery residents at a large academic training program, burnout was present among residents in all PGYs of training; it was most prevalent during the second year of training. Increased levels of the 3 components of burnout were associated with worse performance on the OITE. While there is a lack of consensus in the existing literature, this study provides additional evidence that burnout is negatively associated with 1 aspect of overall resident performance.

CLINICAL RELEVANCE

Orthopaedic surgery residency training is challenging; residents are tasked to acquire a considerable amount of knowledge, develop complex surgical skills, and hone critical clinical thinking in a relatively short period of time. Identifying modifiable contributors to resident burnout and the development of strategies to promote resident wellness during training are important as we strive toward developing the next generation of capable, competent, and well-balanced orthopaedic surgeons.

Beyond the United States Medical Licensing Examination Score: Assessing Competence for Entering Residency

Assessments of physician learners during the transition from undergraduate to graduate medical education generate information that may inform their learning and improvement needs, determine readiness to move along the medical education continuum, and predict success in their residency programs. To achieve a constructive transition for the learner, residency program, and patients, high-quality assessments should provide meaningful information regarding applicant characteristics, academic achievement, and competence that lead to a suitable match between the learner and the residency program's culture and focus.The authors discuss alternative assessment models that may correlate with resident physician clinical performance and patient care outcomes. Currently, passing the United States Medical Licensing Examination Step examinations provides one element of reliable assessment data that could inform judgments about a learner's likelihood for success in residency. Yet, learner capabilities in areas beyond those traditionally valued in future physicians, such as life experiences, community engagement, language skills, and leadership attributes, are not afforded the same level of influence when candidate selections are made.While promising new methods of screening and assessment-such as objective structured clinical examinations, holistic assessments, and competency-based assessments-have attracted increased attention in the medical education community, currently they may be expensive, be less psychometrically sound, lack a national comparison group, or be complicated to administer. Future research and experimentation are needed to establish measures that can best meet the needs of programs, faculty, staff, students, and, more importantly, patients.

Factors Predictive of Orthopaedic In-training Examination Performance and Research Productivity Among Orthopaedic Residents

INTRODUCTION

Selection of qualified candidates for orthopaedic residency is necessary for growth and innovation. The purpose of this study was to determine predictors of Orthopaedic In-training Exam (OITE) performance and research productivity.

METHODS

A survey was distributed to 13 residency programs collecting demographics, United States Medical Licensing Examination (USMLE) and OITE scores, and authored publications. Associations between preresidency qualifications and OITE scores and publications were determined.

RESULTS

A total of 274 of 294 surveys were returned (93.2%). We found a positive correlation between USMLE step 1 and 2 scores with recent OITE percentile (P < 0.001). Preresidency authorship (P < 0.001) and postgraduate training year (P < 0.001) were independent predictors of authorship during residency, whereas USMLE step 1 score was not (P = 0.094).

CONCLUSION

Candidates who perform well on the USMLE are likely to perform well on the OITE, whereas those with greater authored publications are likely to continue research during residency.

Objective Predictors of Grit, Self-Control, and Conscientiousness in Orthopaedic Surgery Residency Applicants

INTRODUCTION

The purpose of this study was to identify objective predictors of grit, self-control, and conscientiousness in orthopaedic surgery residency applicants.

METHODS

The following attributes were assessed in 455 applicants: grit, self-control, conscientiousness, consistency of interest, perseverance of effort, and ambition. These measures were correlated with standard, objective demographics obtained during the application process.

RESULTS

Alpha Omega Alpha status, additional degrees, and number of publications did not predict any of the studied attributes. Grit increased with age (P < 0.001) but decreased with increasing board scores (P = 0.004). Former collegiate athletes demonstrated greater grit (P < 0.001), consistency of interest (P = 0.007), perseverance (P = 0.006), and self-control (P = 0.019). Female applicants demonstrated more grit (P = 0.044), consistency of interest (P = 0.003), and conscientiousness (P = 0.029) than males. Applicants with military experience had increased ambition (P = 0.033) and conscientiousness (P = 0.001).

CONCLUSION

Overall, orthopaedics applicants possess increased grit compared with the general public, and a number of objective variables reliably predicted the studied attributes.

LEVEL OF EVIDENCE

Level III, Cross-sectional study.

Performance on the Plastic Surgery In-Service Examination Can Predict Success on the American Board of Plastic Surgery Written Examination

BACKGROUND

Originally developed for resident self-assessment, the Plastic Surgery In-Service Examination has been administered for over 45 years. The Accreditation Council for Graduate Medical Education requires that at least 70 percent of graduates pass the American Board of Plastic Surgery Written Examination on their first attempt. This study evaluates the role of In-Service Exam scores in predicting Written Exam success.

METHODS

In-Service Exam scores from 2009 to 2015 were collected from the National Board of Medical Examiners. Data included residency training track, training year, and examination year. Written Exam data were gathered from the American Board of Plastic Surgery. Multivariate analysis was performed and receiver operating characteristic curves were used to identify optimal In-Service Exam score cut-points for Written Exam success.

RESULTS

Data from 1364 residents were included. Residents who failed the Written Exam had significantly lower In-Service Exam scores than those who passed (p < 0.001). Independent residents were 7.0 times more likely to fail compared with integrated/combined residents (p < 0.001). Residents who scored above the optimal cut-points were significantly more likely to pass the Written Exam. The optimal cut-point score for independent residents was the thirty-sixth percentile and the twenty-second percentile for integrated/combined residents.

CONCLUSIONS

Plastic Surgery In-Service Exam scores can predict success on the American Board of Plastic Surgery Written Exam. Residents who score below the cut-points are at an increased risk of failing. These data can help identify residents at risk for early intervention.

Predicting American Board of Emergency Medicine Qualifying Examination Passage Using United States Medical Licensing Examination Step Scores

Background

The objective of the current study was to determine whether emergency medicine residents' United States Medical Licensing Examination (USMLE) scores are significantly associated with first-attempt passage of the American Board of Emergency Medicine (ABEM) qualifying (written) examination. We hypothesized that USMLE Step 2 Clinical Knowledge (CK) scores would be useful in predicting students who passed the ABEM qualifying examination on their first attempt.

Methods

For this retrospective cohort study, we examined the data of residents who successfully completed training at two emergency medicine residency programs between the years 2002-2013. Because scores on the USMLE Step examinations varied greatly across years, we obtained means and standard deviations from the National Board of Medical Examiners. We subtracted the mean score for the year each resident took the examination from the resident's examination score, creating centered Step 1 and centered Step 2 CK scores.

Results

A multivariate logistic regression analysis indicated that centered Step 2 CK scores could be used to predict the odds of passing the ABEM qualifying examination (odds ratio = 1.05 [95% confidence interval 1.02 to 1.08, P < 0.001]). Using a Step 2 CK score cutoff of 7 points lower than the mean yielded 64% sensitivity and 81% specificity for predicting passage of the ABEM written examination on the first attempt.

Conclusion

Program directors and selection committees may wish to consider whether applicants' Step 2 CK scores are near the national average when making ranking decisions, as this variable is highly predictive of passing the ABEM qualifying examination on the initial attempt.

Non-cognitive factors predicting success in orthopedic surgery residency

Admissions to orthopedic surgery is a highly competitive process. Traditionally measures such as United States Medical Licensing Examination (USMLE) Step 1, class rank, AOA status have been major determinants in the ranking process. However, these traditional objective measures show mixed correlation to clinical success in orthopedic surgery residency. There have been several studies on the cognitive factors and their correlation with success in residency. However, it is clear that residency requires more than objective cognition, emphasizing complex social interactions that are influenced by non-cognitive variables including personality, work ethic, etc. This review aims to summarize the current understanding of non-cognitive factors influencing performance in orthopaedic surgical residency.

Orthopaedic Surgery Residency Rotations and Correlation With Orthopaedic In-Training Examination Performance

OBJECTIVE

The Orthopaedic In-Training Examination (OITE) is administered annually and is used to assess medical knowledge of orthopedic surgery residents. Beginning in the 2013 to 2014 academic year, the ACGME expanded the postgraduate year (PGY)-1 curriculum from 3 to 6 months of orthopedic surgery rotations. The purpose of this study is to evaluate the effect of increased PGY-1 orthopedic surgery exposure on medical knowledge as measured by the OITE.

DESIGN

From 2011 to 2013, 24 PGY-2 residents completed 3 months of PGY-1 orthopedic training (Group 1). From 2014 to 2016, 24 PGY-2 residents completed 6 months of PGY-1 orthopedic training (Group 2). The effect of an initial PGY-2 pediatrics rotation (Sub-group A), compared to a trauma rotation (Sub-group B) was also analyzed. The hypothesis of this study is that Group 2 scores higher on the OITE than Group 1. Raw percentage and overall percentile scores for all PGY-2 residents from 2011 to 2016 for the pediatrics subsection, the trauma subsection, and for the overall OITE test in our program were recorded. Group 1 versus Group 2, and Sub-group A versus Sub-group B were compared (Student's t-test).

SETTING

University of Minnesota (Institutional, Tertiary); Gillette Children's Hospital (Institutional, Tertiary); Regions Hospital (Institutional, Tertiary).

PARTICIPANTS

48 PGY-2 residents from 2011 to 2016 were included in the study.

RESULTS

Group 2 achieved higher raw and percentile scores on the OITE during their PGY-2 year than Group 1. Sub-group B scored higher than Sub-group A on all OITE subsections and overall.

CONCLUSIONS

This study suggests that raw percentage and percentile OITE scores improve with an additional 3 months of orthopedic training in the PGY-1 year. Clinical exposure, specifically in orthopedic trauma, correlates with higher OITE performance in our residency program.

United States Medical Licensing Exam Step I Score as a Predictor of Neurosurgical Career Beyond Residency

BACKGROUND

United States Medical Licensing Exam (USMLE) Step I score is cited as one of the most important factors when for applying to neurosurgery residencies. No studies have documented a correlation between USMLE Step I score and metrics of neurosurgical career trajectory beyond residency.

OBJECTIVE

To determine whether USMLE Step I exam scores are predictive of neurosurgical career beyond residency, as defined by American Board of Neurological Surgery (ABNS) certification status, practice type, academic rank, and research productivity.

METHODS

A database of neurosurgery residency applicants who matched into neurosurgery from 1997 to 2007 was utilized that included USMLE Step I score. Online databases were used to determine h-index, National Institutes of Health (NIH) grant funding, academic rank, practice type, and ABNS certification status of each applicant. Linear regression and nonparametric testing determined associations between USMLE Step I scores and these variables.

RESULTS

USMLE Step I scores were higher for neurosurgeons in academic positions (237) when compared to community practice (234) and non-neurosurgeons (233, P < .01). USMLE Step I score was not different between neurosurgeons of different academic rank (P = .21) or ABNS certification status (P = .78). USMLE Step I score was not correlated with h-index for academic neurosurgeons (R2 = 0.002, P = .36).

CONCLUSION

USMLE Step I score has little utility in predicting the future careers of neurosurgery resident applicants. A career in academic neurosurgery is associated with a slightly higher USMLE Step I score. However, USMLE Step I score does not predict academic rank or productivity (h-index or NIH funding) nor does USMLE Step I score predict ABNS certification status.

Analysis of the 1990-2007 neurosurgery residency match: does applicant gender affect neurosurgery match outcome?

OBJECTIVE With nearly half of graduating US medical students being female, it is imperative to understand why females typically make up less than 20% of the neurosurgery applicant pool, a number that has changed very slowly over the past several decades. Organized neurosurgery has strongly indicated the desire to overcome the underrepresentation of women, and it is critical to explore whether females are at a disadvantage during the residency application process, one of the first steps in a neurosurgical career. To date, there are no published studies on specific applicant characteristics, including gender, that are associated with match outcome among neurosurgery resident applicants. The purpose of this study is to determine which characteristics of neurosurgery residency applicants, including gender, are associated with a successful match outcome. METHODS De-identified neurosurgical resident applicant data obtained from the San Francisco Fellowship and Residency Matching Service for the years 1990-2007 were analyzed. Applicant characteristics including gender, medical school attended, year of application, United States Medical Licensing Exam (USMLE) Step 1 score, Alpha Omega Alpha (AOA) status, and match outcome were available for study. RESULTS Of the total 3426 applicants studied, 473 (13.8%) applicants were female and 2953 (86.2%) were male. Two thousand four hundred forty-eight (71.5%) applicants successfully matched. USMLE Step 1 score was the strongest predictor of match outcome with scores > 245 having an OR of 20.84 (95% CI 10.31-42.12) compared with those scoring < 215. The mean USMLE Step 1 score for applicants who successfully matched was 233.2 and was 210.8 for those applicants who did not match (p < 0.001). Medical school rank was also associated with match outcome (p < 0.001). AOA status was not significantly associated with match outcome. Female gender was associated with significantly lower odds of matching in both simple (OR 0.59, 95% CI 0.48-0.72) and multivariate analyses (OR 0.57, 95% CI 0.34-0.94 CI). USMLE Step 1 scores were significantly lower for females compared to males with a mean score of 230.1 for males and 221.5 for females (p < 0.001). There was no significant difference in medical school ranking or AOA status when stratified by applicant gender. CONCLUSIONS The limited historical applicant data from 1990-2007 suggests that USMLE Step 1 score is the best predictor of match outcome, although applicant gender may also play a role.