A Narrative Review of the Evidence Supporting Factors Used by Residency Program Directors to Select Applicants for Interviews

Resident Recruitment in a New Era

ABSTRACT

This chapter focuses on resident recruitment and recent US National Resident Matching Program changes and the impact in the evaluation and ranking of applicants within the specialty of anesthesiology. Recruitment challenges are examined as well as program strategies and potential future directions. Also discussed are DEI initiatives within the recruitment process.

Measuring and Predicting Faculty Consensus Rankings of Standardized Letters of Evaluation

Background Standardized letters of evaluation (SLOE) are becoming more widely incorporated into the residency application process to make the letter of recommendation, an already critical component in a residency application packet, more objective. However, it is not currently known if the reviewers of these letters share consensus regarding the strength of an applicant determined by their SLOE. Objective We measured the level of faculty agreement regarding applicant competitiveness as determined by SLOEs and the ability of 2 algorithms to predict faculty consensus rankings. Methods Using data from the 2021-2022 Match cycle from the Council of Residency Directors in Emergency Medicine SLOE Database as a blueprint, authors created 50 fictional SLOEs representative of the national data. Seven faculty then rated these SLOEs in order of applicant competitiveness, defined as suggested rank position. Consensus was evaluated using cutoffs established a priori, and 2 prediction models, a point-based system and a linear regression model, were tested to determine their ability to predict consensus rankings. Results There was strong faculty consensus regarding the interpretation of SLOEs. Within narrow windows of agreement, faculty demonstrated similar ranking patterns with 83% and 93% agreement for "close" and "loose" agreement, respectively. Predictive models yielded a strong correlation with the consensus ranking (point-based system r=0.97, linear regression r=0.97). Conclusions Faculty displayed strong consensus regarding the competitiveness of applicants via SLOEs, adding further support to the use of SLOEs for selection and advising. Two models predicted consensus competitiveness rankings with a high degree of accuracy.

Components of effective letters of recommendation: A cross-sectional survey of academic faculty

INTRODUCTION

Conventional merit-based criteria, including standardized test scores and grade point averages, have become less available to residency programs to help distinguish applicants, making other components of the application, including letters of recommendation (LORs), important surrogate markers for performance. Despite their impact on applications, there is limited published data on LORs in the international setting.

METHODS

A cross-sectional survey of academic faculty was conducted between 9 January 2023 and 12 March 2023 at two large academic medical centers in the United Arab Emirates. Descriptive statistics were used to tabulate variable frequencies.

RESULTS

Of the 98 respondents, the majority were male (n = 67; 68.4%), Western-trained (n = 66; 67.3%), mid-career physicians (n = 46; 46.9%). Most respondents (n = 77; 78.6%) believed that the purpose of an LOR was to help an applicant match into their desired program. Letters rarely included important skills, such as leadership (n = 37; 37.8%), applicant involvement in research (n = 43; 43.9%), education (n = 38; 38.8%), or patient advocacy (n = 30; 30.6%). Most faculty (n = 81; 82.7%) were not familiar with standardized letters of recommendation. Only 7.3% (n = 7) of respondents previously received training in writing LORs, but 87.7% (n = 86) expressed an interest in this professional development opportunity.

CONCLUSION

There is variability in perceptions and practices related to LOR writing in our international setting, with several areas for improvement. Given the increasing importance of LORs to a candidate's application, faculty development is necessary.

A Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States

Background Aligning resident and training program attributes is critical. Many programs screen and select residents using assessment tools not grounded in available evidence. This can introduce bias and inappropriate trainee recruitment. Prior reviews of this literature did not include the important lens of diversity, equity, and inclusion (DEI). Objective This study's objective is to summarize the evidence linking elements in the Electronic Residency Application Service (ERAS) application with selection and training outcomes, including DEI factors. Methods A systematic review was conducted on March 30, 2022, concordant with PRISMA guidelines, to identify the data supporting the use of elements contained in ERAS and interviews for residency training programs in the United States. Studies were coded into the topics of research, awards, United States Medical Licensing Examination (USMLE) scores, personal statement, letters of recommendation, medical school transcripts, work and volunteer experiences, medical school demographics, DEI, and presence of additional degrees, as well as the interview. Results The 2599 identified unique studies were reviewed by 2 authors with conflicts adjudicated by a third. Ultimately, 231 meeting inclusion criteria were included (kappa=0.53). Conclusions Based on the studies reviewed, low-quality research supports use of the interview, Medical Student Performance Evaluation, personal statement, research productivity, prior experience, and letters of recommendation in resident selection, while USMLE scores, grades, national ranking, attainment of additional degrees, and receipt of awards should have a limited role in this process.

A Practical Guide to Implementing Holistic Review during Surgery Resident Selection

Provider workforce diversity is a key component of improving healthcare quality and addressing healthcare disparities. Furthermore, the traditional approach of "score-centered" application metrics do not consistently correlate with meeting milestones in surgery, nor do they adequately predict a surgical resident's clinical strength and operative abilities. We present here an adaptable process by which surgical residency programs can identify their values and incorporate holistic review into their resident selection process to improve resident selection and physician workforce diversity.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

PURPOSE

The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.

METHOD

Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.

RESULTS

Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.

CONCLUSIONS

This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Obstetrics and gynecology clerkship directors' experiences advising residency applicants

BACKGROUND

The evolving landscape of application processes for obstetrics and gynecology residency applicants poses many challenges for applicants and advisors. The lack of data coordination among national groups creates crucial gaps in information for stakeholder groups.

OBJECTIVE

This study aimed to identify the current state of the advising milieu for obstetrics and gynecology residency applicants and their career advisors, the annual Association of Professors of Gynecology and Obstetrics survey focused on US clerkship directors' experiences advising students through these processes.

STUDY DESIGN

A 23-item anonymous survey was developed that asked respondents about demographics and outcomes for the students that they advised through the 2021 application process and their experiences with dual applicants and students not matching. The survey was sent electronically to all obstetrics and gynecology clerkship directors with active Association of Professors of Gynecology and Obstetrics memberships in April 2021.

RESULTS

Of 224 total clerkship directors, 143 (63.8%) responded to the survey, Of the 143 respondents, almost all (136 [95.1%]) served as career advisors, and 50 (35.0%) were aware of students dual applying. Furthermore, obstetrics and gynecology was rarely the backup to a more competitive specialty. For the 2021 application cycle, 79 of 143 respondents (55.2%) reported having students not successfully match into obstetrics and gynecology, with "academic concerns" followed by "poor communication skills" as the primary reasons cited for students not matching.

CONCLUSION

This snapshot of clerkship directors' experiences advising students in the residency application process reveals notably high rates of dual applicants and students not matching into obstetrics and gynecology. This work fills key gaps in our knowledge of current processes and highlights the importance of career advising at multiple points during the application process.

Key factors for residency interview selection from the National Resident Matching Program: analysis of residency Program Director surveys, 2016-2020

CONTEXT

As the number of medical school graduates continues to outpace the available residency training positions, applying for residency in the United States has become a highly competitive process, often associated with a low rate of selection and invitation for interview. The National Resident Matching Program (NRMP) Program Director survey provides data assessing factors considered by Program Directors (PD) in selecting and inviting candidates for interview. Assessing the evolution of these factors over time is efficacious to inform and guide prospective applicants toward improving preparation for residency application.

OBJECTIVES

We aim to synthesize NRMP data showing factors that PDs reported and rated as important in their decision to select and invite applicants for interview.

METHODS

Data from residency PD surveys from 2008 to 2021 were accessed, but after applying inclusion/exclusion criteria, only the data from 2016 to 2020 were reviewed and analyzed. The NRMP survey reports provided two metrics that characterized PDs' evaluation of the residency factors for interview, namely, "percent citing factor" and "average rating" on a 0 to 5 Likert-type scale. These two metrics were combined into an aggregate measure of importance (AI), and another measure of relative importance (RI) was constructed from normalizing the AI of each individual factor to the sum of the AI within each survey year.

RESULTS

The top ranked factors were United States Medical Licensing Examination (USMLE) Step 1/Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1, Letter of Recommendation (LOR) in the specialty, Medical Student Performance Evaluation (MSPE/Dean's Letter), and USMLE Step 2 Clinical Knowledge (CK)/COMLEX Level 2 Cognitive Exam (CE) score, any failed attempt in USMLE/COMLEX, and perceived commitment to specialty. Factors rising in importance were Audition Elective/Rotation Within Your Department, Personal Statement (PS), Perceived Commitment to Specialty, Perceived Interest in Program, LOR in the Specialty, Other Life Experience, and Personal Prior Knowledge of the Applicant. Factors with declining importance were Interest in Academic Career, Awards or Special Honors in Basic Sciences, Graduate of Highly Regarded US Medical School, Awards or Special Honors in Clinical Clerkships, Lack of Gaps in Medical Education, Awards or Special Honors in Clerkship in Desired Specialty, and Consistency of Grades. Compared to the 2021 PD survey, our findings show continued predictive consistency, particularly related to specialty and program commitment.

CONCLUSIONS

The factors identified for the selection of medical school graduates for interview into a residency program reveal that PDs move toward a more integrated approach. Specifically, PDs are placing increasing emphasis on factors that border on subjective qualities more so than the more traditional, quantitative, and objective metrics. Medical students and educators need to continually apprise themselves of the NRMP data to inform students' preparation endeavors throughout medical school to strengthen their application portfolios and enhance their competitiveness for the matching process.

Residency Program Directors' Views on Research Conducted During Medical School: A National Survey

PURPOSE

With the United States Medical Licensing Examination Step 1 transition to pass/fail in 2022, uncertainty exists regarding how other residency application components, including research conducted during medical school, will inform interview and ranking decisions. The authors explore program director (PD) views on medical student research, the importance of disseminating that work, and the translatable skill set of research participation.

METHOD

Surveys were distributed to all U.S. residency PDs and remained open from August to November 2021 to query the importance of research participation in assessing applicants, whether certain types of research were more valued, productivity measures that reflect meaningful research participation, and traits for which research serves as a proxy. The survey also queried whether research would be more important without a numeric Step 1 score and the importance of research vs other application components.

RESULTS

A total of 885 responses from 393 institutions were received. Ten PDs indicated that research is not considered when reviewing applicants, leaving 875 responses for analysis. Among 873 PDs (2 nonrespondents), 358 (41.0%) replied that meaningful research participation will be more important in offering interviews. A total of 164 of 304 most competitive specialties (53.9%) reported increased research importance compared with 99 of 282 competitive (35.1%) and 95 of 287 least competitive (33.1%) specialties. PDs reported that meaningful research participation demonstrated intellectual curiosity (545 [62.3%]), critical and analytical thinking skills (482 [55.1%]), and self-directed learning skills (455 [52.0%]). PDs from the most competitive specialties were significantly more likely to indicate that they value basic science research vs PDs from the least competitive specialties.

CONCLUSIONS

This study demonstrates how PDs value research in their review of applicants, what they perceive research represents in an applicant, and how these views are shifting as the Step 1 exam transitions to pass/fail.

Efforts to Recruit Diverse Resident Complements in Family Medicine

BACKGROUND AND OBJECTIVES

Family medicine is the most demographically diverse specialty in medicine today. Specialty associations and the Accreditation Council for Graduate Medical Education (ACGME) urge residency programs to engage in systematic efforts to recruit diverse resident complements. Using responses from program directors to the ACGME's mandatory annual update, we enumerate the efforts in resident recruiting. This allows us to compare these statements to the recommendations of two highly respected commissions: the Sullivan Commission on Diversity in the Healthcare Workforce and the Institute of Medicine's In the Nation's Compelling Interest: Ensuring Diversity of the Healthcare Workforce.

METHODS

We compiled the annual updates from 689 family medicine programs and analyzed them using a qualitative method called template analysis. We then classified the efforts and compared them to the recommendations of the Sullivan Commission and Institute of Medicine (IOM).

RESULTS

Nearly all (98%) of the programs completed the portion of the annual update inquiring about recruiting residents. The Sullivan Commission and IOM recommended 23 steps to diversify workforce recruiting. We found that programs engaged in all but one of these recommendations. Among the most frequently employed recommendations were doing holistic reviews and using data for planning. None mentioned engaging in public awareness campaigns. Programs also implemented eight strategies not suggested in either report, with staff training in nondiscrimination policies being among the most frequently mentioned. Among program efforts not included in the Sullivan Commission or IOM recommendations were extracurricular activities; appointing diversity, equity, and inclusion (DEI) committees or advocates; subinternship (Sub-I) experiences; recruiting at conferences; blind reviews; legal compliance; and merit criteria. In total, we found 31 interventions in use.

CONCLUSIONS

The Sullivan Commission's guidance, IOM recommendations, and program-developed initiatives can be combined to create a comprehensive roster of diversity recruiting initiatives. Programs may use this authoritative resource for identifying their next steps in advancing their recruiting efforts.

USMLE step 1 and step 2 CK as indicators of resident performance

BACKGROUND

The purpose of this systematic review was to (1) determine the scope of literature measuring USMLE Step 1 and Step 2 CK as predictors or indicators of quality resident performance across all medical specialties and (2) summarize the ability of Step 1 and Step 2 CK to predict quality resident performance, stratified by ACGME specialties, based on available literature.

METHODS

This systematic review was designed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [16]. The original search strategy surveyed MEDLINE and was adapted to survey Cochrane Library and Embase. A study was deemed eligible if it provided all three of the following relevant information: (a) Step 1 or Step 2 CK as indicators for (b) resident outcomes in (c) any ACGME accredited specialty training program.

RESULTS

A total of 1803 articles were screened from three separate databases. The 92 included studies were stratified by specialty, with Surgery (21.7% [20/92]), Emergency Medicine (13.0% [12/92]), Internal Medicine (10.9% [10/92]), and Orthopedic Surgery (8.7% [8/92]) being the most common. Common resident performance measures included ITE scores, board certification, ACGME milestone ratings, and program director evaluations.

CONCLUSIONS

Further studies are imperative to discern the utility of Step 1 and Step 2 CK as predictors of resident performance and as tools for resident recruitment and selection. The results of this systematic review suggest that a scored Step 1 dated prior to January 2022 can be useful as a tool in a holistic review of future resident performance, and that Step 2 CK score performance may be an effective tool in the holistic review process. Given its inherent complexity, multiple tools across many assessment modalities are necessary to assess resident performance comprehensively and effectively.

A Simulation of the Effect of Interview Caps on the National Resident Matching Program Match in Otolaryngology

Concerns regarding interview hoarding in the Match process have led to calls for a cap on the number of interviews an applicant can accept. However, no study has examined the effects of interview caps on applicants or a specialty. The authors created a simulation otolaryngology Match to analyze the effects of interview caps. The simulation included 120 residency programs, 360 positions, and 570 applicants (each assigned a competitiveness weighted value). Programs created interview lists (subject to 4 different interview cap conditions: no, 16, 12, and 8 cap) using weighted random selection and the competitiveness weighted values. Programs then selected 3 interviewees to fill their positions via weighted random selection using the competitiveness weighted values. Data analysis explored the effects of interview caps on individual applicants' probability of matching, the probability of matching with a certain number of interviews, and the average matched applicant competitiveness rank using one-way analysis of variance. The introduction of interview caps caused interviews to be distributed more equally among applicants. The 12-interview cap caused the applicants in the 100th competitiveness percentile to see their probability of matching fall from 99.3% to 67.2% ( P < .001). Applicants with ≥ 12 interviews had a probability of matching of 94.4% when there was no cap, compared to 76.9% with a 12-interview cap ( P < .001). The average competitiveness rank of matched applicants fell from 191 with no cap to 245 with a 12-interview cap ( P < .001). Interview caps in the otolaryngology Match may create major distortions in the probability of matching, leading to much lower chances of matching for more competitive applicants, lower probabilities of matching with certain numbers of interviews, and a decreased average competitiveness of matched applicants.

Perspectives of Dermatology Program Directors on the Impact of Step 1 Pass/Fail

INTRODUCTION

The shift of Step 1 to Pass/Fail has generated several questions and concerns about obtaining residency positions among allopathic and osteopathic students alike. Determining the perspectives of Dermatology Program Directors in regards to post-Step 1 Pass/Fail is critical for students to better prepare for matching into dermatology.

METHODS

After receiving Institutional Review Board (IRB) exemption status, the program directors were chosen from 144 Accreditation Council for Graduate Medical Education (ACGME) and 27 American Osteopathic Association (AOA) Dermatology programs using contact information from their respective online website databases. An eight-item survey was constructed on a three-point Likert scale, one free text response, and four demographic questions. The anonymous survey was sent out over the course of three weeks with weekly individualized reminder requests for participation.

RESULTS

A total of 54.54% of responders had "Letters of Recommendation" in their top 3. Forty-five percent of responders had "Completed Audition Rotation at Program" in their top 3. And, 38.09% of responders had "USMLE Step 2 CK Scores" in their top 3.

CONCLUSION

Approximately 50% of responders agreed that all medical students will have more difficulty matching dermatology. Based on the survey study, Dermatology program directors want to focus more on letters of recommendation, audition rotations, and Step 2 CK scores. Because each field seems to prioritize different aspects of an application, students should attempt to gain as much exposure to different fields such as through research and shadowing to narrow down their ideal specialties. Consequently, the student will have more time to tailor their applications to what residency admissions are looking for.

Effect of Holistic Review, Interview Blinding, and Structured Questions in Resident Selection: Can we Predict Who Will Do Well in a Residency Interview?

OBJECTIVE

To examine the Urology residency application process, particularly the interview. Historically, the residency interview has been vulnerable to bias and not determined to be a predictor of future residency performance. Our goal is to determine the relationship between pre-interview metrics and post-interview ranking using best practices for Urology resident selection including holistic review, blinded interviews, and structured behaviorally anchored questions.

METHODS

Applications were assessed on cognitive (Alpha Omega Alpha, class rank, junior year clinical clerkship grades) and non-cognitive attributes (letters of recommendation [LOR], personal statement [PS], demographics, research, personal characteristics) by reviewers blinded to USMLE scores and photograph. Interviewers were blinded to the application other than PS and LORs. Interviews consisted of a structured behaviorally anchored question (SBI) and an unstructured interview (UI). Odds ratios were determined comparing pre-interview and interview impressions.

RESULTS

Fifty-one applicants were included in the analysis. USMLE step 1 score (average 245) was associated with Alpha Omega Alpha, class rank, junior year clinical clerkship, and PS. The UI score was associated with the LOR (P = .04) whereas SBI scores were not (P = .5). Faculty rank was associated with SBI, UI, and overall interview (OI) scores (P < .001). Faculty rank was also associated with LOR. Resident impression of interviewees were associated with faculty interview scores (P = .001) and faculty rank (P < .001).

CONCLUSION

Traditional interviews may be biased toward application materials and may be balanced with behavioral questions. While Step 1 score does not offer additional information over other PI metrics, blinded interviews may offer discriminant validity over a PI rubric.

Hawks and Doves: Perceptions and Reality of Faculty Evaluations

OBJECTIVES

Internal medicine clerkship grades are important for residency selection, but inconsistencies between evaluator ratings threaten their ability to accurately represent student performance and perceived fairness. Clerkship grading committees are recommended as best practice, but the mechanisms by which they promote accuracy and fairness are not certain. The ability of a committee to reliably assess and account for grading stringency of individual evaluators has not been previously studied.

METHODS

This is a retrospective analysis of evaluations completed by faculty considered to be stringent, lenient, or neutral graders by members of a grading committee of a single medical college. Faculty evaluations were assessed for differences in ratings on individual skills and recommendations for final grade between perceived stringency categories. Logistic regression was used to determine if actual assigned ratings varied based on perceived faculty's grading stringency category.

RESULTS

"Easy graders" consistently had the highest probability of awarding an above-average rating, and "hard graders" consistently had the lowest probability of awarding an above-average rating, though this finding only reached statistical significance only for 2 of 8 questions on the evaluation form (P = .033 and P = .001). Odds ratios of assigning a higher final suggested grade followed the expected pattern (higher for "easy" and "neutral" compared to "hard," higher for "easy" compared to "neutral") but did not reach statistical significance.

CONCLUSIONS

Perceived differences in faculty grading stringency have basis in reality for clerkship evaluation elements. However, final grades recommended by faculty perceived as "stringent" or "lenient" did not differ. Perceptions of "hawks" and "doves" are not just lore but may not have implications for students' final grades. Continued research to describe the "hawk and dove effect" will be crucial to enable assessment of local grading variation and empower local educational leadership to correct, but not overcorrect, for this effect to maintain fairness in student evaluations.

Deconstructing the roadmap to surgical residency: a national survey of residents illuminates factors associated with recruitment success as well as applicants' needs and beliefs

Purpose

As applications increase and residency becomes more competitive, applicants and programs will be challenged by increased demands on recruitment, metric assessment, and rank determination. Studies have investigated program opinions; however, this survey sought to illuminate the process from an applicant's perspective.

Methods

An anonymous survey was distributed to past or current surgery residents nationwide using social media and program director emails. Regression analyses were performed to assess factors correlating with percentage of programs which offered the applicant an interview.

Results

There were 223 respondents who applied to an average of 61 programs (± 40) with 16 (± 11) interviews offered. Applicants believed that programs were most interested in (1) personality, (2) letter of recommendation (LOR) writers, and (3) medical school reputation. Top factors considered by applicants in ranking were resident culture, location, program reputation, and autonomy. Bivariate analysis found factors that decreased percent of interview invites to be Asian race, whereas factors that increased interview invites included age, year of match, surgery clerkship grade, medicine clerkship grade, AOA status, honor surgery rotation, gold humanism (GHHS) status, phone call for interview made, and step scores (all p < 0.05). AOA status, step scores, honor surgery rotation, year of match, and Asian race remained significant after multivariate analysis.

Conclusions

National surveys illuminate how applicants approach the application process and what programs and applicants appear to value. This information provides insight and guidance to candidates and programs as the process of matching becomes more challenging with surging application numbers, changes in testing parameters and virtual interviews.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-022-00070-9.

Program Directors' Selection Criteria for General Surgery and Internal Medicine Residency Programs: A Cross-Sectional Study

Background and aims General surgery is a specialty that calls for a variety of abilities such as strong hand-eye coordination, the ability to function well under pressure, and the ability to make quick, informed decisions. On the other hand, internal medicine focuses on diagnosing, treating, and preventing adult non-surgical disorders. The present study aims to investigate the commonly used selection criteria employed by program directors (PDs) for general surgery and internal medicine residency programs. It also identifies how PDs value those criteria in the western and central regions of Saudi Arabia. Methods A retrospective cross-sectional study was conducted using a questionnaire adopted from the literature and modified following expert opinion. It was distributed to PDs in both the central and the western regions of Saudi Arabia. Results In total, 34 PDs completed the questionnaire, of which 32 (94.1%) were men. The mean age of participants was 42.53 ± 5.05 years; 21 (61.8%) PDs were general surgeons, and 47.1% were from the Jeddah region. Conclusion The study showed that the most selected criterion for both specialties was communication skills followed by clinical rotation in the same hospital; leadership skills were also highly considered by PDs.

Simulation as Soothsayer: Simulated Surgical Skills MMIs During Residency Interviews are Associated With First Year Residency Performance

OBJECTIVE

The main consideration during residency recruitment is identifying applicants who will succeed during residency. However, few studies have identified applicant characteristics that are associated with competency development during residency, such as the Accreditation Council for Graduate Medical Education milestones. As mini multiple interviews (MMIs) can be used to assess various competencies, we aimed to determine if simulated surgical skills MMI scores during a general surgery residency interview were associated with Accreditation Council for Graduate Medical Education milestone ratings at the conclusion of intern year.

DESIGN

Retrospective cohort study. Interns' Step 1 and 2 clinical knowledge (CK) scores, interview day simulated surgical skills MMI overall score, traditional faculty interview scores, average overall milestone ratings in the spring of residency, and intern American Board of Surgery In-Training Examination (ABSITE) percentile scores were gathered. Two multiple linear regression were performed analyzing the association between Step 1, Step 2 CK, MMI, and traditional faculty interview scores with (1) average overall milestone rating and (2) ABSITE percentile scores, controlling for categorical/preliminary intern classification.

SETTING

One academic medical center PARTICIPANTS: General surgery interns matriculating in 2020-2021 RESULTS: Nineteen interns were included. Multiple linear regression revealed that higher overall simulated surgical skills MMI score was associated with higher average milestone ratings (β = .45, p = 0.03) and higher ABSITE score (β = .43, p = 0.02) while neither Step 1, Step 2 CK, nor faculty interview scores were significantly associated with average milestone ratings.

CONCLUSIONS

Surgical residency programs invest a tremendous amount of effort into training residents, thus metrics for predicting applicants that will succeed are needed. Higher scores on a simulated surgical skills MMIs are associated with higher milestone ratings 1 year into residency and higher intern ABSITE percentiles. These results indicate a noteworthy method, simulated surgical skills MMIs, as an additional metric that may select residents that will have early success in residency.

A Retrospective Analysis of Medical Student Performance Evaluations, 2014-2020: Recommend with Reservations

BACKGROUND

The Medical Student Performance Evaluations (MSPE) is a cornerstone of residency applications. Little is known regarding adherence to Association of American Medical Colleges (AAMC) MSPE recommendations and longitudinal changes in MSPE content.

OBJECTIVES

Evaluate current MSPE quality and longitudinal changes in MSPE and grading practices.

DESIGN

Retrospective analysis.

PARTICIPANTS

Students from all Liaison Committee on Medical Education (LCME)-accredited medical schools from which the Stanford University Internal Medicine residency program received applications between 2014-2015 and 2019-2020.

MAIN MEASURES

Inclusion of key words to describe applicant performance and metrics thereof, including distribution among students and key word assignment explanation; inclusion of clerkship grades, grade distributions, and grade composition; and evidence of grade inflation over time.

KEY RESULTS

MSPE comprehensiveness varied substantially among the 149 schools analyzed. In total, 25% of schools provided complete information consistent with AAMC recommendations regarding key word/categorization of medical students and clerkship grades in 2019-2020. Seventy-seven distinct key word terms appeared across the 139 schools examined in 2019-2020. Grading practices markedly varied, with 2-83% of students receiving the top internal medicine clerkship grade depending on the year and school. Individual schools frequently changed key word and grading practices, with 33% and 18% of schools starting and/or stopping use of key words and grades, respectively. Significant grade inflation occurred over the 6-year study period, with an average 14% relative increase in the proportion of students receiving top clerkship grades.

CONCLUSIONS

A minority of schools complies with AAMC MSPE guidelines, and MSPEs are inconsistent across time and schools. These practices may impair evaluation of students within and between schools.

Medical Schools as Racialized Organizations: How Race-Neutral Structures Sustain Racial Inequality in Medical Education-a Narrative Review

In 2021, The American Association of Medical Colleges released a framework addressing structural racism in academic medicine, following the significant, nationwide Movement for Black Lives. The first step of this framework is to "begin self-reflection and educating ourselves." Indeed, ample evidence shows that medical schools have a long history of racially exclusionary practices. Drawing on racialized organizations theory from the field of sociology, we compile and examine scholarship on the role of race and racism in medical training, focusing on disparities in educational and career outcomes, experiences along racial lines in medical training, and long-term implications. From the entrance into medical school through the residency application process, organizational factors such as reliance on standardized tests to predict future success, a hostile learning climate, and racially biased performance metrics negatively impact the careers of trainees of color, particularly those underrepresented in medicine (URiM). Indeed, in addition to structural biases associated with otherwise "objective" metrics, there are racial disparities across subjective outcomes such as the language used in medical trainees' performance evaluations, even when adjusting for grades and board exam scores. These disadvantages contribute to URIM trainees' lower odds of matching, steering into less competitive and lucrative specialties, and burnout and attrition from academic careers. Additionally, hostile racial climates and less diverse medical schools negatively influence White trainees' interest in practicing in underserved communities, disproportionally racial and ethnic minorities. Trainees' mental health suffers along the way, as do medical schools' recruitment, retention, diversity, and inclusion efforts. Evidence shows that seemingly race-neutral processes and structures within medical education, in conjunction with individuals' biases and interpersonal discrimination, may reproduce and sustain racial inequality among medical trainees. Medical schools whose goals include training a more diverse physician workforce towards addressing racial health disparities require a new playbook.

Making the Personal Statement "Truly Personal": Recommendations From a Qualitative Case Study of Internal Medicine Program and Associate Program Directors

BACKGROUND

The personal statement is an integral part of a residency application but little guidance exists for medical students about what content to include.

OBJECTIVE

We use the framework of isomorphism, the process by which institutions model themselves after one another, to understand what internal medicine program directors (PDs) and associate program directors (APDs) recommend be included in the personal statement and how programs use personal statements in their selection of applicants to interview and rank.

METHODS

Semi-structured phone interviews were conducted between August and October 2020 with 13 academic PDs and APDs, who were selected for participation based on program size and geographic location. Interviews were recorded, transcribed, and coded using content analysis.

RESULTS

Effective personal statements should be well-written, present unique information, and demonstrate fit with a residency program. PDs and APDs recommended against expressing lack of interest in a program or highlighting negative personal characteristics. PDs and APDs used personal statements to distinguish between applicants and noted that personal statements help programs form an impression of the applicant. Consensus among PDs and APDs about what personal statements should include and how they are used indicates that isomorphism influences the match process.

CONCLUSIONS

Our study found that the personal statement is a valued part of the residency application when it includes unique attributes and reveals personal values that align with that of the program. Additionally, PDs and APDs noted that when applicants highlight their unique characteristics, it can help distinguish themselves from others.

What Does Honors Mean? National Analysis of Medical School Clinical Clerkship Grading

OBJECTIVE

Medical Student Performance Evaluation (MSPE) letters provide critical comparative information about clerkship performance, and are a crucial part of the surgical residency application. The elimination of USMLE Step 1 numeric reporting increases the importance of transparency, standardization, and accessibility of comparative information reported on the MSPE. The objective of our study was to measure the variability in clerkship grade reporting on the MSPE from US medical schools, particularly focusing on the highest (honors) grades.

DESIGN, SETTING, AND PARTICIPANTS

We identified representative MSPE letters from US medical schools and recorded the percentage of honors for 5 core clerkships. We grouped medical schools according to medical school rankings, geographic region, and number of grading categories RESULTS: Of 122 medical schools, 106 schools (87%) reported their grading scheme and percent honors. The most commonly used grading scheme was a 4-tier system (51/122; 42%). The percentage of honors was highly variable (from 1-91%) and did not vary by region. However, schools in the top 20 research ranking were less likely to report grade comparisons (30% vs. 10%), and more likely to award more students honors in 4 of the 5 clerkships. Schools in the top 20 primary care ranking were more likely to award more honors in the medicine clerkship.

CONCLUSIONS

There is significant variability in the number of grading tiers used and the percentage of students awarded honors across US medical schools. Factors that correlated to higher grades included schools with higher rankings, and higher ranked schools were less likely to report comparative information at all.

Changing Assessment Scenarios: Lessons for Changing Practice

Assessment is a process that includes ascertainment of improvement in the performance of students over time, motivation of students to study, evaluation of teaching methods, and ranking of student capabilities. It is an important component of the educational process influencing student learning. Although we have embarked on a new curricular model, assessment has remained largely ignored despite being the hallmark of competency-based education. During the earlier stages, the assessment was considered akin to "measurement," believing that competence is "generic, fixed and transferable across content," could be measured quantitatively and can be expressed as a single score. The objective assessment was the norm and subjective tools were considered unreliable and biased. It was soon realized that "competence is specific and nontransferable," mandating the use of multiple assessment tools across multiple content areas using multiple assessors. A paradigm change through "programmatic assessment" only occurred with the understanding that competence is "dynamic, incremental and contextual." Here, information about the students' competence and progress is gathered continually over time, analysed and supplemented with purposefully collected additional information when needed, using carefully selected combination of tools and assessor expertise, leading to an authentic, observation-driven, institutional assessment system. In the conduct of any performance assessment, the assessor remains an important part of the process, therefore making assessor training indispensable. In this paper, we look at the changing paradigms of our understanding of clinical competence, corresponding global changes in assessment and then try to make out a case for adopting the prevailing trends in the assessment of clinical competence.

Analyzing the Landscape of the 2021 Virtual Match: A Nationwide Survey of Radiology Programs' Stakeholders

RATIONALE AND OBJECTIVES

We aimed to provide insights into the adaptive strategies, benefits, and challenges faced by the radiology programs during the 2021 residency virtual Match. Furthermore, we explored the potential impacts of related topics, such as diversity and social media use on the Match process and outcomes.

MATERIALS AND METHODS

A cross-sectional survey of 31 questions was designed and distributed via e-mails to individuals involved radiology programs match process during the 2021 Match. Descriptive statistics were used to analyze the results of most questions. Two questions comparing the changes in factors influencing the selection of applicants on a Likert scale of 1-5 were analyzed using paired t-test and Wilcoxon signed-rank test where p-value <0.05 was considered statistically significant.

RESULTS

Responses from 125 participants were analyzed. The following factors carried less weight in evaluating applicants during 2021 Match: away rotations (p < 0.01), no failed attempts in USMLE Step 1/CK (p < 0.01), grades in radiology clerkship (p < 0.04), and class rank/quartile (p < 0.04), while personal statements were more important (p < 0.03). Out of the 125 respondents, 80 (64%) and 58 (47%) strongly or somewhat agree on the effectiveness of virtual interviews in gauging applicants' candidacy and showing their programs' advantages, respectively. Advantages of virtual interviews included decreased cost, time flexibility, less faculty burden, and an increased number of offered interviews according to 81% (101/125), 46% (58/125), 40%, (50/125), and 34% (43/125), respectively. The most helpful platforms that showcased program advantages were program websites followed by Twitter and Instagram.

CONCLUSION

Most radiology programs were able to adjust to the virtual interview process, and the majority agree on their effectiveness citing many benefits. However, there were mixed opinions if it could be sustained in future cycles.

Pass/Fail USMLE Step 1 Scoring-A Radiology Program Director Survey

Rationale and Objectives

In February 2020, administrators for the US medical licensing examination (USMLE) announced that Step 1 reporting would change to pass/fail in hopes of reducing the overemphasis of USMLE performance on the residency selection system and improving medical student well-being. Our objective was to determine the perspectives of diagnostic radiology (DR), interventional radiology (IR), and nuclear medicine (NM) program directors (PDs) regarding pass/fail USMLE Step 1 scoring.

Materials and Methods

A survey composed of thirteen questions on a three-point Likert scale, five demographic questions, and a free-text question was distributed to 179 DR, 84 IR, and 34 NM PDs from ACGME-accredited residency programs.

Results

In total, 140 unique responses were obtained (response rate = 47.1%). The PD respondents had a male predominance of 79.1%, average age of 46 ± 7.2 years, and average tenure of 5.9 ± 5.2 years. A majority of PDs (69.6%) disagreed that the change is a good idea, and a minority (21.6%) believe the change will improve medical student well-being. Further, 90.7% of PDs believe a pass/fail format will make it more difficult to objectively compare applicants and most will place more emphasis on USMLE Step 2 scores and medical school reputation (89.3% and 72.7%, respectively).

Conclusion

The lasting impact of pass/fail Step 1 scoring are uncertain and many radiology PDs do not support this change. While the central motivations to reduce the overemphasis on USMLE Step 1 performance and improve medical student well-being are admirable, it remains to be seen if pass/fail scoring will accomplish these goals.

Don't Judge a Letter by its Title: Linguistic Analysis of Letters of Recommendation by Author's Academic Rank

OBJECTIVE

This study analyzed the linguistic differences in letters of recommendation (LORs) for general surgery residency applicants written by authors of various academic ranks. Given that many general surgery residency programs require a LOR from the Chair of surgery, this study also examined whether LORs written by the Chair demonstrate linguistic differences to support this practice.

DESIGN

A single institution, retrospective review analyzed LORs from two application cycles of general surgery residency applicants who were selected for interview at a large academic institution. Word count (WC) and linguistic characteristics of LORs were analyzed with a previously developed institution-specific dictionary using the Linguistic Inquiry and Word Count software (LIWC2015; Pennebaker Conglomerates, Inc., Austin, Texas). WC and linguistic characteristics of LORs reported as frequencies of terms within twenty-four categories were examined based on the letter authors' academic rank. Further examination compared LORs written by a Chair of surgery with those written by non-Chairs.

SETTING

A single large, Midwestern academic general surgery residency program PARTICIPANTS: Four hundred and sixty-five letters of recommendation received during two interview cycles were included for analysis.

RESULTS

A total of 465 LORs written by assistant (n = 82), associate (n = 94), and full professors (n = 289) were included in the study. No statistically significant difference was noted in the WC of LORs based on the letter writers' academic ranks (p = 0.95). Assistant professors utilized grindstone, communal, and technical skill terms with higher frequencies compared to associate professors and full professors. LORs written by assistant professors demonstrated the highest authentic variable score followed by associate professors then full professors (4.94, 3.92, 3.28, p < 0.01). LORs written by Chairs (n = 128) had lower authentic variable scores compared to LORs written by non-Chairs (n = 337; 2.71 vs. 3.91, p = 0.001). Only 50 (39%) LORs written by Chairs indicated working directly with the applicant, and sub-group analysis demonstrated a higher authentic variable score in this group compared with LORs written by Chairs who did not indicate having worked directly with the applicant (3.51 vs. 2.5, p = 0.01).

CONCLUSIONS

Linguistic analysis of LORs for general surgery residency applicants demonstrated minor yet statistically significant differences based on the author's academic rank. If applicants can obtain linguistically similar LORs from surgeons of any academic rank, but less authentic LORs from writers with higher academic ranks, these LORs may be less valuable for the residency programs when evaluating applicants. Based on the subgroup analysis, less than 40% of Chair LORs indicated that the Chair worked directly with the applicant, calling into question the utility of the Chair LORs as meaningful evaluation of applicants. Further study to compare LORs of applicants selected and not selected for interview may add additional insight into linguistic differences in LORs written by authors of different academic ranks.

Crowd-Sourced Assessment of Surgical Skills of Urology Resident Applicants: Four-Year Experience

OBJECTIVE

To determine a) if surgical skills among urology resident applicants could be reliably assessed via crowdsourcing and b) to what extent surgical skills testing impacts resident selection.

DESIGN

Interviewees completed the following surgical skills tasks during their interview day: open knot tying (OKT), laparoscopic peg transfer (LPT), and robotic suturing (RS). Urology faculty and crowd-workers evaluated each applicant's video-recorded performance using validated scoring and were assessed for agreement using Cronbach's alpha. Applicants' USMLE scores, interview scores, and Jefferson Scale of Physician Empathy (JSPE-S) scores were assessed for correlation with skills testing scores and match rank. Additionally, a survey was distributed to interviewees assessing match outcomes.

SETTING

University of California Irvine Department of Urology, Surgical Skills Laboratory PARTICIPANTS: All 94 urology residency interviewees at the University of California Irvine Department of Urology from 2015-2018 were invited to complete the three surgical skills tasks on their interview day.

RESULTS

Survey responses were received from all 94 interviewees (100%). Crowd and expert agreement was good (α=0.88), fair (α=0.67), and poor (α=0.32) for LPT, RS, and OKT scores, respectively. The skills testing scores did not correlate with match rank, USMLE score, or JSPE-S score. On multivariate analysis, only interview score (r= -0.723; p<0.001) and faculty LPT score (r=-0.262; p=0.001) were significant predictors of match rank. Interviewees who reported matching into a top 3 residency choice had significantly higher faculty LPT scores than those who did not (11.9 vs. 9.7, p=0.03).

CONCLUSIONS

Surgical skills overall did not significantly impact match rank. Expert assessment of laparoscopic peg transfer skills and interview performance among urology resident applicants correlated with match rank.

A call to assess the performance of international medical graduates in Obstetrics and Gynecology residency training programs in the United States

International medical graduates contribute to an important section of the practicing physicians in the United States. However, there is not enough data to understand their performance in Obstetrics and Gynecology residency training programs of the United States. International medical graduates who apply to residency programs, including those with previous clinical experience, are often disfavored by most Obstetrics and Gynecology residency training programs, based on the theoretical inferior performance of these applicants during their residency years. The previous experience obtained by the international medical graduate-applicants are often valued by other developed countries that were reported to have better healthcare system in women health compared to that of the United States. Studying the performance of international medical graduate-residents in Obstetrics and Gynecology training programs could help set a standard that aid the program leaders choosing their potential residents and could also protect the rights of these international medical graduate-applicants. This is especially important amid recent changes in certifying exams, volatile immigration system of the United States, and the increasing number of medical graduates.

Reducing Over-Interviewing in the Anesthesiology Residency Match

Background The U.S. residency recruitment process is expensive and time-consuming because of application inflation and over-invitation. Objective Using interview and match data, we quantify the predicted effects if anesthesiology residency programs excluded interviews for applicants who are very unlikely to match. Methods We previously published the validity and accuracy of the logistic regression model based on data from interview scheduling software used by 32 U.S. anesthesiology residency programs and 1300 applicants from 2015-18. Data used were program region, applicant address, numbers of interviews of the interviewee, medical school US News and World Report (USNWR) rank, the difference between United States Medical Licensing Exam (USMLE) Step 1 and 2 Clinical Knowledge (CK) scores, and the historical average of USMLE scores of program residents. In the current study completed in 2020, the predicted probabilities and their variances were summed among interviewees for 30 deidentified programs. Results For anesthesiology, the median residency program could reduce their interviews by 16.9% (97.5% confidence interval 8.5%-24.1%) supposing they would not invite applicants if the 99% upper prediction limit for the probability of matching was less than 10.0%. The corresponding median savings would be 0.80 interviews per matched spot (0.34-1.33). In doing so, the median program would sustain a risk of 5.3% (97.5% confidence interval 2.3%-7.9%) of having at least one interviewee removed from their final rank-to-match list. Conclusion Using novel interview data and analyses, we demonstrate that residency programs can substantively reduce interviews with less effect on rank-to-match lists. The data-driven approach to manage marginal interviews allows program leadership to better weigh costs and benefits when composing their annual list of interviewees.

Medical School to Residency: How Can We Trust the Process?

Background

To say that the transition from undergraduate medical education (UME) to graduate medical education (GME) is under scrutiny would be an understatement. Findings from a panel discussion at the 2018 Association of American Medical Colleges Annual meeting entitled, “Pass-Fail in Medical School and the Residency Application Process and Graduate Medical Education Transition” addressed what and when information should be shared with residency programs, and how and when that information should be shared.

Materials and Methods

Over 250 participants representing UME and GME (e.g. leadership, faculty, medical students) completed worksheets addressing these questions. During report-back times, verbal comments were transcribed in real time, and written comments on worksheets were later transcribed. All comments were anonymous. Thematic analysis was conducted manually by the research team to analyze the worksheet responses and report back comments.

Results

Themes based on suggestions of what information should be shared included the following: 1) developmental/assessment benchmarks such as demonstrating the ability/competencies to do clinical work; 2) performance on examinations; 3) grades and class ranking; 4) 360 evaluations; 5) narrative evaluations; 6) failures/remediation/gaps in training; 7) professionalism lapses; 8) characteristics of students such as resiliency/reliability; and 9) service/leadership/participation. In terms of how this information should be shared, the participants suggested enhancements to the current process of transmitting documents rather than alternative methods (e.g., video, telephonic, face-to-face discussions) and information sharing at both the time of the match and again near/at graduation to include information about post-match rotations.

Discussion

Considerations to address concerns with the transition from medical school to residency include further enhancements to the Medical Student Performance Evaluation, viewing departmental letters as ones of evaluation and not recommendation, a more meaningful educational handoff, and limits on the number of residency applications allowed for each student. The current medical education environment is ready for meaningful change in the UME to GME transition.

Association Between Internal Medicine Residency Applicant Characteristics and Performance on ACGME Milestones During Intern Year

BACKGROUND

Residency programs apply varying criteria to the resident selection process. However, it is unclear which applicant characteristics reflect preparedness for residency.

OBJECTIVE

We determined the applicant characteristics associated with first-year performance in internal medicine residency as assessed by performance on Accreditation Council for Graduate Medical Education (ACGME) Milestones.

METHODS

We examined the association between applicant characteristics and performance on ACGME Milestones during intern year for individuals entering Northwestern University's internal medicine residency between 2013 and 2018. We used bivariate analysis and a multivariable linear regression model to determine the association between individual factors and Milestone performance.

RESULTS

Of 203 eligible residents, 198 (98%) were included in the final sample. One hundred fourteen residents (58%) were female, and 116 residents (59%) were White. Mean Step 1 and Step 2 CK scores were 245.5 (SD 12.0) and 258 (SD 10.8) respectively. Step 1 scores, Alpha Omega Alpha membership, medicine clerkship grades, and interview scores were not associated with Milestone performance in the bivariate analysis and were not included in the multivariable model. In the multivariable model, overall clerkship grades, ranking of the medical school, and year entering residency were significantly associated with Milestone performance (P ≤ .04).

CONCLUSIONS

Most traditional metrics used in residency selection were not associated with early performance on ACGME Milestones during internal medicine residency.

Curated collection for clinician educators: Six key papers on residency recruitment

INTRODUCTION

All emergency medicine (EM) residency programs must recruit new medical school graduates each year. The process is often overwhelming, with each program receiving far more applicants than available positions. We searched for evidence-based best practices to guide residency programs in screening, interviewing, and ranking applicants to ensure a high-performing and diverse residency class.

METHODS

A literature search was conducted on the topic of residency recruitment, utilizing a call on social media as well as multiple databases. After identifying relevant articles, we performed a modified Delphi process in three rounds, utilizing junior educators as well as more senior faculty.

RESULTS

We identified 51 relevant articles on the topic of residency recruitment. The Delphi process yielded six articles that were deemed most highly relevant over the three rounds. Transparency with selection criteria, holistic application review, standardized letters of evaluation, and blinding applicant files for interviewers were among noted best practices.

CONCLUSIONS

Well-supported evidence-based practices exist for residency recruitment, and programs may benefit from understanding which common recruitment practices offer the most value. The articles discussed here provide a foundation for faculty looking to improve their program's recruiting practices.

One Institution's evaluation of family medicine residency applicant data for academic predictors of success

BACKGROUND

Family Medicine residencies are navigating recruitment in a changing environment. The consolidation of accreditation for allopathic and osteopathic programs, the high volume of applicants, and the forthcoming transition of the United States Medical Licensing Exam (USMLE) Step 1 to pass/fail reporting all contribute. This retrospective cohort study evaluated which components of a student's academic history best predict readiness for residency.

METHODS

In 2020, we analyzed applicant data and initial residency data for program graduates at a single residency program between 2013 and 2020. This included undergraduate education characteristics, medical school academic performance, medical school academic problems (including professionalism), STEP exams, location of medical school, and assessments during the first 6 months of residency. Of 110 matriculating residents, assessment data was available for 97 (88%).

RESULTS

Pre-matriculation USMLE data had a positive correlation with initial American Board of Family Medicine (ABFM) in-training exams. Pre-matriculation exam data did not have a positive correlation with resident assessment across any of the six Accreditation Council for Graduate Medical Education (ACGME) competency domains. A defined cohort of residents with a history of academic struggles during medical school or failure on a USMLE exam performed statistically similarly to residents with no such history on assessments across the six ACGME competency domains.

CONCLUSIONS

Applicants with a history of academic problems perform similarly in the clinical environment to those without. While a positive correlation between pre-matriculation exams and the ABFM in-training exam was found, this did not extend to clinical assessments across the ACGME competency domains.

Interview Data Highlight Importance of "Same-State" on Anesthesiology Residency Match

BACKGROUND

The US residency application, interview, and match processes are costly and time-intensive. We sought to quantify the importance of an applicant being from the same-state as a residency program in terms of how this impacted the number of interviews needed to match.

METHODS

We examined data from interview scheduling software used by 32 programs located in 31 US states and 1300 applicants for the US anesthesiology recruitment cycles from 2015 to 2018. Interviewee data (distance from program, region, numbers of interviews, and program at which interview occurred) were analyzed to quantify the effect of the interviewee being from the same state as the residency program on the odds of matching to that program. Other variables of interest (medical school, current address, US Medical Licensing Exam [USMLE] Step 1 and 2 clinical knowledge [CK] scores, Alpha Omega Alpha [AOA] status, medical school ranking) were also examined as controls. Confidence intervals (CI) were calculated for the ratios of odds ratios.

RESULTS

An interviewee living in the same state as the interviewing program could have 5.42 fewer total interviews (97.5% CI, 3.02-7.81) while having the same odds of matching. The same state effect had an equivalent value as an approximately 4.14 USMLE points-difference from the program's mean (97.5% CI was 2.34-5.94 USMLE points). Addition of whether the interviewee belonged to an affiliated medical school did not significantly improve the model; same-state remained significant (P < .0001) while affiliated medical school was not (P = .40).

CONCLUSIONS

Our analysis of anesthesiology residency recruitment using previously unstudied interview data shows that same-state locality is a viable predictor of residency matching and should be strongly considered when evaluating whether to interview an applicant.

A novel faculty development tool for writing a letter of recommendation

Objective

Based on a national survey of program directors we developed a letter of recommendation (LOR) scoring rubric (SR) to assess LORs submitted to a pediatric residency program. The objective was to use the SR to analyze: the consistency of LOR ratings across raters and LOR components that contributed to impression of the LOR and candidate.

Methods

We graded 30 LORs submitted to a pediatric residency program that were evenly distributed based on final rank by our program. The SR contained 3 sections (letter features, phrases, and applicant abilities) and 2 questions about the quality of the LOR (LORQ) and impression of the candidate (IC) after reading the LOR on a 5-point Likert scale. Inter-rater reliability was calculated with intraclass correlation coefficients (ICC(2,1)). Pearson (r) correlations and stepwise multivariate linear regression modeling predicted LORQ and IC. Mean scores of phrases, features, and applicant abilities were analyzed with ANOVA and Bonferroni correction.

Results

Phrases (ICC(2,1) = 0.82, p<0.001)) and features (ICC(2,1) = 0.60, p<0.001)) were rated consistently, while applicant abilities were not (ICC(2,1) = 0.28, p<0.001)). For features, LORQ (R² = 0.75, p<0.001) and IC (R² = 0.58, p<0.001) were best predicated by: writing about candidates’ abilities, strength of recommendation, and depth of interaction with the applicant. For abilities, LORQ (R² = 0.47, p<0.001) and IC (R² = 0.51, p<0.001) were best predicted by: clinical reasoning, leadership, and communication skills (0.2). There were significant differences for phrases and features (p<0.05).

Conclusions

The SR was consistent across raters and correlates with impression of LORQ and IC. This rubric has potential as a faculty development tool for writing LORS.

Reimagining Residency Selection: Part 3-A Practical Guide to Ranking Applicants in the Post-COVID-19 Era

Collapse

Identifying and Supporting Students to Prevent USMLE Step 1 Failures When Testing Follows Clerkships: Insights From 9 Schools

Several schools have moved the United States Medical Licensing Examination Step 1 exam after core clerkships, and others are considering this change. Delaying Step 1 may improve Step 1 performance and lower Step 1 failure rates. Schools considering moving Step 1 are particularly concerned about late identification of struggling students and late Step failures, which can be particularly problematic due to reduced time to remediate and accumulated debt if remediation is ultimately unsuccessful. In the literature published to date, little attention has been given to these students. In this article, authors from 9 medical schools with a postclerkship Step 1 exam share their experiences. The authors describe curricular policies, early warning and identification strategies, and interventions to enhance success for all students and struggling students in particular. Such learners can be identified by understanding challenges that place them "at risk" and by tracking performance outcomes, particularly on other standardized assessments. All learners can benefit from early coaching and advising, mechanisms to ensure early feedback on performance, commercial study tools, learning specialists or resources to enhance learning skills, and wellness programs. Some students may need intensive tutoring, neuropsychological testing and exam accommodations, board preparation courses, deceleration pathways, and options to postpone Step 1. In rare instances, a student may need a compassionate off-ramp from medical school. With the National Board of Medical Examiner's announcement that Step 1 scoring will change to pass/fail as early as January 2022, residency program directors might use failing Step 1 scores to screen out candidates. Institutions altering the timing of Step 1 can benefit from practical guidance by those who have made the change, to both prevent Step 1 failures and minimize adverse effects on those who fail.

A Stepping Stone Toward Necessary Change: How the New USMLE Step 1 Scoring System Could Affect the Residency Application Process

Score reporting for the United States Medical Licensing Examination Step 1 will change from a 3-digit number to pass/fail as soon as January 1, 2022. The shift is meant to prevent residency program directors from using Step 1 scores to select applicants for interviews, a purpose for which the exam was not designed. Using Step 1 scores in this way also has put undue stress on medical students applying to residency. However, the score reporting change represents only one stepping stone toward an improved transition from undergraduate to graduate medical education. To enable a more reliable and holistic review of applicants, residency program directors and medical school administrators must promote other standardized evaluation tools and address the hypercompetitive and frenzied state of the residency application process. For example, medical schools should provide program directors with assessments of students' fit and readiness for residency that are not burdensome to understand and compare. In addition, residency programs should implement "traffic rules" to improve the interview process for applicants. These changes will significantly mitigate the burden on all stakeholders. As residents who recently experienced this transition, the authors of this Invited Commentary argue that now is the opportune time to redefine selection criteria and reemphasize the characteristics that truly matter in training competent future physicians.

Analysis of Electronic Residency Application Service (ERAS) Data Can Improve House Staff Diversity

BACKGROUND

Training diverse house staff, including those who are underrepresented in medicine, is vital to provide high-quality patient care for the communities that we serve. In 2018, the Accreditation Council for Graduate Medical Education announced new common program requirements for systematic efforts to recruit and retain a diverse workforce. However, questions remain about how to implement such efforts.

MATERIALS AND METHODS

Electronic Residency Application Service (ERAS) data from eight residency programs spanning two recruitment cycles (2017-2018, 2018-2019) was reviewed. The number of candidates at each stage in the process (applicant, invited to interview, interviewed, and matched) was examined by self-identified race or ethnicity. These data were presented to residency program directors at our Graduate Medical Education committee meeting before the next recruitment cycle. Data were analyzed following the 2019-20 residency match. Odds ratios and Pearson's chi-squared test were used to assess statistical significance.

RESULTS

A total of 10,445 and 10,982 medical students applied to our 8 core residency programs in 2017 and 2018, respectively. Medical students who applied and self-identified as Asian, Black or African American, and Hispanic or Latino or Spanish origin had lower odds of being invited to interview than those who self-identified as White. After data presentation, the odds of inviting Black or African American applicants to interview increased significantly. The odds of attending an interview once invited were the same across groups.

CONCLUSIONS

Sharing ERAS data patterns with residency program directors was associated with a significant year over year change in interviewee diversity. Structured analysis of institutional ERAS data can provide insight into the resident selection process and may be a useful tool to improve house staff diversity.

Resident selection for emergency medicine specialty training in Canada: A survey of existing practice with recommendations for programs, applicants, and references

OBJECTIVE

Little is known about how the Royal College of Emergency Medicine (RCEM) residency programs are selecting their residents. This creates uncertainty regarding alignment between current selection processes and known best practices. We seek to describe the current selection processes of Canadian RCEM programs.

METHODS

An online survey was distributed to all RCEM program directors and assistant directors. The survey instrument included 22 questions and sought both qualitative and quantitative data from the following six domains: application file, letters of reference, elective selection, interview, rank order, and selection process evaluation.

RESULTS

We received responses from 13 of 14 programs for an aggregate response rate of 92.9%. A candidate's letters of reference were identified as the most important criterion from the paper application (38.5%). Having a high level of familiarity with the applicant was the most important characteristic of a reference letter author (46.2%). In determining rank order, 53.8% of programs weighed the interview more heavily than the paper application. Once final candidate scores are established following the interview stage, all program respondents indicated that further adjustment is made to the final rank order list. Only 1 of 13 program respondents reported ever having completed a formal evaluation of their selection process.

CONCLUSION

We have identified elements of the selection process that will inform recommendations for programs, students, and referees. We encourage programs to conduct regular reviews of their selection process going forward to be in alignment with best practices.

Which Applicant Factors Predict Success in Emergency Medicine Training Programs? A Scoping Review

BACKGROUND

Program directors (PDs) in emergency medicine (EM) receive an abundance of applications for very few residency training spots. It is unclear which selection strategies will yield the most successful residents. Many authors have attempted to determine which items in an applicant's file predict future performance in EM.

OBJECTIVES

The purpose of this scoping review is to examine the breadth of evidence related to the predictive value of selection factors for performance in EM residency.

METHODS

The authors systematically searched four databases and websites for peer-reviewed and gray literature related to EM admissions published between 1992 and February 2019. Two reviewers screened titles and abstracts for articles that met the inclusion criteria, according to the scoping review study protocol. The authors included studies if they specifically examined selection factors and whether those factors predicted performance in EM residency training in the United States.

RESULTS

After screening 23,243 records, the authors selected 60 for full review. From these, the authors selected 15 published manuscripts, one unpublished manuscript, and 11 abstracts for inclusion in the review. These studies examined the United States Medical Licensing Examination (USMLE), Standardized Letters of Evaluation, Medical Student Performance Evaluation, medical school attended, clerkship grades, membership in honor societies, and other less common factors and their association with future EM residency training performance.

CONCLUSIONS

The USMLE was the most common factor studied. It unreliably predicts clinical performance, but more reliably predicts performance on licensing examinations. All other factors were less commonly studied and, similar to the USMLE, yielded mixed results.

Japanese International Medical Graduates and entrance into US clinical training: Challenges and methods to overcome them

INTRODUCTION

Entering US clinical training requires completing requirements and navigating an application process differing from the Japanese system. Additionally, increases to the number of US medical school graduates have increased competition for US residency positions. We examined profiles of Japanese International Medical Graduates (IMGs) who completed US clinical training, the timelines to securing US clinical positions, and the greatest challenges during this process and methods to overcome them.

METHODS

Individual semistructured interviews were conducted with 35 purposively sampled Japanese IMGs. We performed exploratory thematic analysis using iterative data collection and constant comparison.

RESULTS

Twenty percent of participants lived in a native English-speaking country during childhood. The United States Medical Licensing Examinations were completed at ages 25-40 years. Challenges were categorized as: (1) English communication, (2) understanding the application process, (3) motivation to persevere through the process, (4) time management to complete Educational Commission for Foreign Medical Graduates requirements, (5) receiving letters of recommendation and overcoming competition for US residency positions, (6) financial cost of the process. Pragmatic generally self-dependent methods helped overcome challenges 1-4 and 6. Participants detailed personal or, more commonly, institutional connections to US training programs required to overcome challenge 5.

CONCLUSIONS

Japanese IMGs pursue US clinical training from diverse backgrounds commonly without the advantage of prior English fluency. Amidst increased competition internationally to enter US residency coupled with cultural and linguistic differences making this challenge often greater for Japanese IMGs, the competition to participate in institutionalized connections to US training programs is anticipated to increase.

Beyond High-Stakes Testing: Learner Trust, Educational Commodification, and the Loss of Medical School Professionalism

With ever-growing emphasis on high-stakes testing in medical education, such as the Medical College Admission Test and the United States Medical Licensing Examination Step 1, there has been a recent surge of concerns on the rise of a "Step 1 climate" within U.S. medical schools. The authors propose an alternative source of the "climate problem" in current institutions of medical education. Drawing on the intertwined concepts of trust and professionalism as organizational constructs, the authors propose that the core problem is not hijacking-by-exam but rather a hijackable learning environment weakened by a pernicious and under-recognized tide of commodification within the U.S. medical education system. The authors discuss several factors contributing to this weakening of medicine's control over its learning environments, including erosion of trust in medical school curricula as adequate preparation for entry into the profession, increasing reliance on external profit-driven sources of medical education, and the emergence of an internal medical education marketplace. They call attention to breaches in the core tenets of a profession-namely a logic that differentiates its work from market and managerial forces, along with related slippages in discretionary decision making. The authors suggest reducing reliance on external performance metrics (high-stakes exams and corporate rankings), identifying and investing in alternative metrics that matter, abandoning the marketization of medical education "products," and attending to the language of educational praxis and its potential corruption by market and managerial lexicons. These steps might salvage some self-governing independence implied in the term "profession" and make possible (if not probable) a recovery of a public trust becoming of the term and its training institutions.

The Impact of COVID-19 on Medical Education

In the wake of the novel coronavirus (COVID-19) pandemic, it is abundantly clear to all the necessity of studying the pathology and widespread health consequences associated with the virus. However, what is much less clear is the impact of COVID-19 on medical education. Already, faculty and medical students are grappling with the changes that have been made and attempting to consolidate these with their plan of career development. Changes that may seem relatively minor in comparison to the global pandemic have the potential to be drastic turning points in the career progression of many. As not much is known regarding the long-lasting impact of COVID-19 on medical education, it is therefore also necessary to record and study the full impact of the changes being made.

The path to entering a successful residency has been predictable for the last few years - do well on Step 1, give conference presentations, go the extra mile in clerkships and shadowing opportunities, and have meaningful non-academic extracurricular activities - all of which designed to best demonstrate a student's knowledge, persistence, collaborative spirit, and dedication to medicine. This trajectory has been changed with COVID-19 disrupting routines in hospitals, medical schools and beyond. The replacement of in-person classes with online equivalents is an obvious necessity at this time but creates a loss of collaborative experiences that has the potential to be a significant detriment to education. Likewise, the cancellation of clerkships, which are necessary for both skill acquisition as well as for relationship building, is a serious issue which students and medical schools must now resolve. Many medical students have also lost the opportunity for personal development through conference presentations. These presentations play a large role in distinguishing applicants during the residency application process, and therefore these lost opportunities have the potential to be a serious detriment to medical students’ career trajectory. While implementing technology to help resolve these issues is a unique way to help students to develop these skills, it is now necessary for medical students to demonstrate the same set of skills which they would have previously in a completely new and innovative manner.

Persistence and adaptability during this time of challenge are attributes that medical students can demonstrate more readily. While every student has a personal story of how COVID-19 has impacted their education, there is no question that the impacts of COVID-19 will be felt on an extensive level. The panic in the community is palpable, and many are confused by how to proceed in the wake of COVID-19. This is no different for medical students and faculty and the questions that arise regarding medical education and their future careers.