Orthopedic Surgery Postgraduate Year 1 Intern Curriculum Improves Initial Orthopedic In-Training Examination Performance:

Association Between USMLE Step 1 Scores and In-Training Examination Performance: A Meta-Analysis

PURPOSE

On February 12, 2020, the sponsors of the United States Medical Licensing Examination announced that Step 1 will transition to pass/fail scoring in 2022. Step 1 performance has historically carried substantial weight in the evaluation of residency applicants and as a predictor of subsequent subject-specific medical knowledge. Using a systematic review and meta-analysis, the authors sought to determine the association between Step 1 scores and in-training examination (ITE) performance, which is often used to assess knowledge acquisition during residency.

METHOD

The authors systematically searched Medline, EMBASE, and Web of Science for observational studies published from 1992 through May 10, 2020. Observational studies reporting associations between Step 1 and ITE scores, regardless of medical or surgical specialty, were eligible for inclusion. Pairs of researchers screened all studies, evaluated quality assessment using a modified Newcastle-Ottawa Scale, and extracted data in a standardized fashion. The primary endpoint was the correlation of Step 1 and ITE scores.

RESULTS

Of 1,432 observational studies identified, 49 were systematically reviewed and 37 were included in the meta-analysis. Overall study quality was low to moderate. The pooled estimate of the correlation coefficient was 0.42 (95% confidence interval [CI]: 0.36, 0.48; P < .001), suggesting a weak-to-moderate positive correlation between Step 1 and ITE scores. The random-effects meta-regression found the association between Step 1 and ITE scores was weaker for surgical (versus medical) specialties (beta -0.25 [95% CI: -0.41, -0.09; P = .003]) and fellowship (versus residency) training programs (beta -0.25 [95% CI: -0.47, -0.03; P = .030]).

CONCLUSIONS

The authors identified a weak-to-moderate positive correlation between Step 1 and ITE scores based on a meta-analysis of low-to-moderate quality observational data. With Step 1 scoring transitioning to pass/fail, the undergraduate and graduate medical education communities should continue to develop better tools for evaluating medical students.

The New PGY-1 Year: Lessons Learned

PURPOSE OF REVIEW

Describe the ACGME's changes to the PGY-1 year in urology and discuss the benefits and challenges faced by training programs.

RECENT FINDINGS

There are no publications detailing the integration of the PGY-1 year in urology; however, response of other surgical subspecialties to their own integration has been studied. Benefits of integration include earlier exposure to techniques and knowledge specific to urology, potentially leading to increased preparedness for next steps in training and exams. Program directors have more flexibility to select rotations relevant to urology. Resident wellness may be improved as interns are incorporated into the department earlier and can help distribute the workload for senior residents. Challenges include decreased exposure to basic surgical knowledge and skills, decreased camaraderie with general surgery colleagues, and difficulties associated with evaluating interns who are spending limited time with urology departments. Overall, the change seems to have a positive impact on urological training.

Orthopaedic In-Training Examination Resources and Residency Training for the Foot and Ankle Domain

BACKGROUND

Residency programs use the annual Orthopaedic In-Training Examination (OITE) prepared by the American Academy of Orthopaedic Surgeons (AAOS) to monitor resident progress and prepare them for the part 1 of the American Board of Orthopaedic Surgeons (ABOS) Certifying Examination. The purpose of this study was to determine resources residents currently use to prepare for the OITE and also to learn about their perception of training they receive in the foot and ankle subspecialty in their program and their interest in foot and ankle fellowship after residency.

METHODS

An anonymous survey was sent to both allopathic programs and osteopathic residents to learn what resources residents used to study for the OITE, preparatory question sets, on-call resources, their perception on training received in foot and ankle surgery, and their intent to pursue fellowship training.

RESULTS

A total of 130 residents participated in the survey. The majority of residents in allopathic and osteopathic residencies used Orthobullets (OB) to prepare for the OITE and use this resource while on-call. Most residents also used OB question sets to study along with the AAOS self-assessment examinations. In total, 43.2% of osteopathic residents felt they did not get enough exposure to foot and ankle subspecialty while in training, in contrast to 31.2% of allopathic residents. A total of 35% of all orthopaedic surgery residents felt they lacked enough exposure to foot and ankle orthopaedic surgery. Only 7 residents (6%, 6 allopathic, 1 osteopathic) intended to pursue a foot and ankle fellowship following graduation.

CONCLUSION

Online resources such as OB continue to be frequently used by residents for preparation for the OITE. Greater than one-third of orthopaedic residents feel they do not get enough exposure to foot and ankle orthopaedic surgery. Improvement in this area could be helped by continued endeavors from the American Orthopaedic Foot and Ankle Society such as the Visiting Professor Program and Resident Scholarship Program.

LEVELS OF EVIDENCE

Level V: Single Cross-Sectional Study.

Residents and Program Director Perspectives Often Differ on Optimal Preparation Strategies and the Value of the Orthopedic In-Training Examination

OBJECTIVES

The purpose of this work was to compare resident and program director (PD) perspectives on the value of the Orthopaedic In-Training Examination (OITE), resident study habits, and best resources for optimal performance.

DESIGN

A national survey of orthopedic surgery residents and PDs.

SETTING

Mayo Clinic, Rochester, MN PARTICIPANTS: The survey was completed by 445 (41.5%) eligible orthopedic surgery residents and 37 (77.1%) PDs.

RESULTS

Although residents and PDs agreed on when (p = 0.896) and how much (p = 0.171) residents currently study, residents felt that the OITE was not as valuable of an assessment of their knowledge, and also felt their individual scores were less likely to remain confidential compared to PDs (p < 0.001). The mean OITE score below which residents were concerned about their ability to pass American Board of Orthopaedic Surgeons Part 1 was 9.7 percentile points higher than PDs threshold (42.3% vs. 32.6%, respectively, p = 0.003). Both groups agreed that it is important to dedicate focused study time to the OITE (p = 0.680) and to perform well (p = 0.099). Regarding the best resources and preparation strategies, both residents and PDs tended to agree on the value of most (6 of 10) study methods. Residents ranked practice question websites (mean ranking of 2.6 vs. 3.8 of 10, respectively; p < 0.001) and formal rotations in a subspecialty (6.0 vs. 7.7 respectively, p < 0.001) higher than PDs. In contrast, PDs tended to value their program's formal OITE prep program (4.1. vs. 5.3, respectively, p = 0.012) and reading primary literature (5.6 vs. 6.6, respectively, p = 0.012) more than residents.

CONCLUSION

Residents and PDs agreed on many critical components of this process; however, a number of key differences in perspectives exist.

The orthopaedic in-training examination: perspectives of program directors and residents from the United States and Canada

OBJECTIVE

Objective assessment of resident performance continues to task program directors (PDs) with a formidable challenge. This study evaluated attitudes toward the Orthopaedic In-Training Examination (OITE), compared its value between countries, assessed its value against other metrics of resident performance, and examined program and resident factors predictive of high achievement.

DESIGN

Survey.

SETTING

Orthopedic surgery residency programs across the United States and Canada.

PARTICIPANTS

One hundred sixty-six PDs and 945 residents.

RESULTS

Eighty-eight PDs and 331 residents completed the surveys (response rates, 54% and 35%, respectively). PDs and residents in the United States assigned greater importance to the OITE than did those in Canada and reported OITE scores from the United States were significantly higher. PDs in the United States reported greater consequences for residents with poor scores than did PDs from Canada, including remediation and reprimand. Observed structured clinical examinations, internal examinations, and in-training evaluation reports were assigned greater importance by PDs and residents in Canada, but low or no importance by those in the United States. In preparation for the OITE, residents strongly favored prior OITE and American Academy of Orthopaedic Surgeons self-assessment questions, the 'AAOS Comprehensive Orthopaedic Review' textbook, the Journal of the American Academy of Orthopaedic Surgeons, and an OITE-based multiple-choice question website. Regression analysis identified resident and program emphasis on OITE studying and higher level of training as positive predictors for higher OITE scores.

CONCLUSIONS

The OITE is more important to PDs and residents in the United States than it is in Canada, and the reported OITE scores reflect these attitudes. PDs in Canada also employ a greater diversity of evaluative tools, a practice in keeping with recent advances toward competency-based medical education. The findings of this report may help PDs be aware of alternative methods of formative resident evaluation and ultimately improve the training of future independent surgeons.