Which Application Factors Are Associated With Outstanding Performance in Orthopaedic Surgery Residency?

Factors Related to Clinical Performance in Spine Surgery Fellowship: Can We Predict Success

INTRODUCTION

The factors most important in the spine fellowship match may not ultimately correlate with quality of performance during fellowship. This study examined the spine fellow applicant metrics correlated with high application rank compared with the metrics associated with the strongest clinical performance during fellowship.

METHODS

Spine fellow applications at three academic institutions were retrieved from the San Francisco Match database (first available to 2021) and deidentified for application review. Application metrics pertaining to research, academics, education, extracurriculars, leadership, examinations, career interests, and letter of recommendations were extracted. Attending spine surgeons involved in spine fellow selection at their institutions were sent a survey to rank (1) fellow applicants based on their perceived candidacy and (2) the strength of performance of their previous fellows. Pearson correlation assessed the associations of application metrics with theoretical fellow rank and actual performance.

RESULTS

A total of 37 spine fellow applications were included (Institution A: 15, Institution B: 12, Institution C: 10), rated by 14 spine surgeons (Institution A: 6, Institution B: 4, Institution C: 4). Theoretical fellow rank demonstrated a moderate positive association with overall research, residency program rank, recommendation writer H-index, US Medical Licensing Examination (USMLE) scores, and journal reviewer positions. Actual fellow performance demonstrated a moderate positive association with residency program rank, recommendation writer H-index, USMLE scores, and journal reviewer positions. Linear regressions identified journal reviewer positions (ß = 1.73, P = 0.002), Step 1 (ß = 0.09, P = 0.010) and Step 3 (ß = 0.10, P = 0.002) scores, recommendation writer H-index (ß = 0.06, P = 0.029, and ß = 0.07, P = 0.006), and overall research (ß = 0.01, P = 0.005) as predictors of theoretical rank. Recommendation writer H-index (ß = 0.21, P = 0.030) and Alpha Omega Alpha achievement (ß = 6.88, P = 0.021) predicted actual performance.

CONCLUSION

Residency program reputation, USMLE scores, and a recommendation from an established spine surgeon were important in application review and performance during fellowship. Research productivity, although important during application review, was not predictive of fellow performance.

LEVEL OF EVIDENCE

III.

STUDY DESIGN

Cohort Study.

Analysis of variability and trends in medical school clerkship grades

Background

Medical school clerkship grades are used to evaluate orthopedic surgery residency applicants, however, high interinstitutional variability in grade distribution calls into question the utility of clerkship grades when evaluating applicants from different medical schools. This study aims to evaluate the variability in grade distribution among medical schools and look for trends in grade distribution over recent years.

Methods

Applications submitted to Rush University's orthopedic surgery residency program from 2015, 2019, and 2022 were collected from the Electronic Residency Application Service. Applications from the top 100 schools according to the 2023-2024 U.S. News and World Report Research Rankings were reviewed. The percentage of "honors" grades awarded by medical schools for the surgery and internal medicine clerkships were extracted from applicants' Medical Student Performance Evaluation letters.

Results

The median percentage of honors given in 2022 was 36.0 % (range 10.0-82.0) for the surgery clerkship and 33.0 % (range 6.7-80.0) for the internal medicine clerkship. Honors were given 6.6 % more in the surgery clerkship in 2022 compared to 2015. There was a negative correlation between a higher (worse) U.S. News and World Report research ranking and the percentage of honors awarded in 2022 for the surgery and internal medicine clerkships.

Conclusion

There is substantial interinstitutional variability in the rate that medical schools award an "honors" grade with evidence of grade inflation in the surgery clerkship. Residency programs using clerkship grades to compare applicants should do so cautiously provided the variability demonstrated in this study.

Identification, Characterization, and Ranking of Candidate Metrics for Selection to Anesthesiology Residency: An Iterative Survey of Program Directors

Background

The primary aim of this study was to identify and stratify candidate metrics used by anesthesiology residency program directors (PDs) to develop their residency rank lists through the National Resident Matching Program.

Methods

Sixteen PDs comprised the participants, selected for diversity in geography and program size. We used a 3-round iterative survey to identify and stratify candidate metrics. In the first round, participants listed metrics they planned to use to evaluate candidates. In the second round, metrics from the first round were ranked by importance, and criteria were solicited to define an exceptional, strong, average, marginal, and uncompetitive candidate for each metric. In the third round, aggregated results were presented and participants refined their rankings.

Results

Of the 16 PDs selected, 15 participated in the first and second survey rounds, and 10 in the third. Eighteen candidate metrics were indicated by 8 or more PDs for residency selection. All 10 PDs from the final round identified passing Step 1 of the United States Medical Licensing Exam (USMLE) and the absence of "red flags" like a failed rotation as key selection metrics, both averaging an importance score of 4.9 out of 5. Other metrics identified by all PDs included clerkship evaluation comments, USMLE Step 2 scores, class rank, letters of recommendation, personal statement, and program and geographical signals.

Conclusions

The study reveals key metrics anesthesiology residency PDs use for candidate ranking, which may offer candidates insights into their competitiveness for anesthesiology residency.

An analysis of osteopathic medical students applying to surgical residencies following transition to a single graduate medical education accreditation system

CONTEXT

Upon requests from osteopathic medical schools, the National Resident Matching Program (NRMP) Charting Outcomes were redesigned to include osteopathic medical school seniors beginning in 2018 and one joint graduate medical education (GME) accreditation system, the Accreditation Council for Graduate Medical Education (ACGME), formed in 2020.

OBJECTIVES

The goal of this study is to analyze the match outcomes and characteristics of osteopathic applicants applying to surgical specialties following the ACGME transition.

METHODS

A retrospective analysis of osteopathic senior match outcomes in surgical specialties from the NRMP Main Residency Match data from 2020 to 2022 and the NRMP Charting Outcomes data from 2020 to 2022 was performed.

RESULTS

For surgical specialties, results show matching increased as United States Medical Licensing Examination (USMLE) Step 2 CK (clinical knowledge) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 2 CE (cognitive evaluation) scores increased along with the number of contiguous rankings (p<0.001). The greatest indication for matching looking at scores alone were those who scored greater than 230 on Step 2 CK compared to below (p<0.001) and above 650 on Level 2 CE (p<0.001). However, those who scored 240 (p=0.025) on Step 2 CK were just as likely to match as those who scored 250 (p=0.022) when compared to those who scored below those scores. Increasing research involvement had little to no significance with the likelihood of matching across most surgical subspecialties.

CONCLUSIONS

Our study demonstrates that there are unique thresholds for Step 2 CK scores, Level 2 CE scores, and the number of contiguous ranks for each surgical specialty that, when reached, are significantly associated with match success. Although certain board score delineations are linked with higher match success rates, the rates level off after this point for most surgical specialties and do not significantly increase further with higher scores. In addition, thresholds within contiguous ranks for increasing match likelihood exist and vary across surgical specialties. Overall, this study highlights that the quantitative metrics utilized to assess applicants lack the correlation reported historically, and the data presently available need to be more substantiated.

Socioeconomic Disparities in Research Participation: Bias in Plastic Surgery Residency Match

Background

Integrated plastic surgery residency applicants have increased at a rate disproportionate to available positions. Research productivity has become a surrogate marker for competitiveness, and many applicants pursue it to distinguish themselves. To date, no study has investigated socioeconomic disparities in extended research experience (ERE) participation.

Methods

A 35-question cross-sectional survey was distributed to applicants to United States-based integrated plastic surgery residency programs during the 2019-2022 application cycles. Summary tables, student t test, and chi-square tests were used for statistical analysis.

Results

A total of 161 responses (response rate: 20.9%) were recorded. Fifty-nine (40.7%) respondents participated in an ERE. The most common reason for ERE participation was strengthening one's application. The most common reason against participation was avoiding delays in career progression. A greater percentage of respondents from Northeastern medical schools participated in EREs (P = 0.019). There were no significant differences in debt burden between those who did or did not participate in an ERE. A greater percentage of applicants whose parents had advanced degrees participated in EREs (P = 0.053).

Conclusions

There may be geographic and socioeconomic biases present in access to ERE for students interested in plastic surgery. The growing popularity of EREs may have unintended consequences for applicant diversity. As most plastic surgeons ultimately practice in nonacademic settings, applicants and plastic surgeons may consider the financial hardships and possible socioeconomic disparities in research opportunities before participating in or recommending them.

The Utility of the Standardized Letter of Recommendation for Orthopedic Surgery Residency Applicants: A Systematic Review

Despite widespread adoption for evaluating residency candidates, few studies have evaluated the orthopedic standardized letter of recommendation (SLOR). A systematic review using PubMed, Embase, and Web of Science was performed in June 2022. Study design and results from SLOR investigations were compiled. Common outcomes studied were summative rank statement scores and SLOR individual domains. Applicants were rated ranked to match or in the top one-third of rank lists in non-normally distributed frequencies. The association of summative rank statement score with match outcome was rarely studied. Applicants' ratings skew positively, the utility is reportedly limited, and influence on match outcome has been inadequately studied. [Orthopedics. 2024;47(1):e1-e5.].

Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School

Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.

Academic Faculty Demonstrate Weak Agreement in Evaluating Orthopaedic Surgery Residents

Background

As objective metrics fade, subjective elements of orthopaedic surgery applicants carry increasing importance during recruitment. Academic orthopaedic surgeons believe they can select for high-performing orthopaedic residents. However, can they agree? The purpose of this study was (1) to analyze an academic orthopaedic surgery department to determine whether they agree on which residents performed best during residency and; (2) to correlate preresidency and intraresidency factors with postresidency evaluations of resident performance.

Methods

With Institutional Review Board [IRB] approval, an orthopaedic surgery department completed surveys to evaluate residency performance for 10 years of graduated residents (2012-2022). Faculty determined (1) Post-Residency Class Rank (PRCR)-ranked from the highest (1) to lowest performing resident (5) relative to their class based on faculty perspective of performance-and (2) Rank List Score (RLS)-ranked based off the 5-point AOA SLOR used during recruitment interviews. RLS assessed how likely the department would have graduates in the program again based on residency performance. Free marginal Cohen's kappa statistics assessed faculty inter-rater agreement. Preresidency metrics (United States Medical Licensing Exam [USMLE] 1 and 2 scores, research publications, etc) were correlated with Orthopaedic In-Training Exam (OITE) scores, research productivity, American Board of Orthopaedic Surgery (ABOS) scores, and faculty-derived rankings. Linear regressions with forward variable entry (p < 0.05) were used to determine factors associated with excellent resident performance.

Results

Eighteen of 25 faculty members (72%) provided survey responses evaluating 46 residents. Faculty agreed 37% and 38% of the time for PRCR (kappa 0.26) and RLS (kappa 0.23), respectively. Step 2 score was the only preresidency factor significantly associated with PRCR (p = 0.03, r2 = 0.15) and RLS (p = 0.02, r2 = 0.3). The only intraresidency factor significantly correlated with PRCR (p = 0.002, r2 = 0.50) and RLS (p = 0.01, r2 = 0.39) was PGY-4 OITE score.

Conclusions

An academic orthopaedic surgery department is able to come to a consensus on evaluations of residency performance relative to peers in the same year of training (PRCR) and an objective standard (RLS). Step 2 and Post-Graduate Year (PGY)-4 OITE scores were the only preresidency and intraresidency factors with significant association to higher postresidency, faculty-derived performance scores.

Level of Evidence

III.