Interview Data Highlight Importance of "Same-State" on Anesthesiology Residency Match

Do Program Directors of Anesthesiology Residency Programs Interpret Narrative Letters of Recommendation as Intended?

Background Letters of recommendation (LORs) are an important part of the application process for medical residency programs with most specialties preferring a narrative format. Given the inherent subjectivity of narrative LORs, the current study sought to determine whether the intended messages of narrative LORs written for applicants to anesthesiology residency programs are accurately interpreted by readers. Methodology Anonymous online surveys were sent via the Qualtrics platform to program directors (PDs) of the Accreditation Council for Graduate Medical Education-accredited anesthesiology residency programs in the Mid-Atlantic region as designated by the Electronic Residency Application Service, which consists of the states of New York, Pennsylvania, and New Jersey. Each PD participant received five surveys, each of which was attached to a de-identified LOR that was written by another PD located at an institution in the same region. Both the letter writer and study participants were asked to score LORs on a Likert-like scale. Participants were additionally asked whether the LORs, if received, would influence their decision to either offer an interview to the applicant or to rank the applicant. Finally, participants were asked to note any specific words or phrases within the LORs that they found to be particularly impactful. Results Overall, 10 of 34, 29.41%, PDs responded to the survey. There was a high correlation between the LOR intent and the respondents' assigned rating (Spearman's rho = 0.7973, p < 0.001). Responses were more accurate for "outstanding and excellent" LORs compared to the lower three categories. Results were unaffected after adjusting for respondents' years of experience as PDs. Additionally, 71.6% indicated that the LORs would influence the decision about offering an interview, and 56.5% stated that the LORs would influence a ranking decision. Conclusions Our results indicate that respondents' perception of LORs correlated strongly with the intent of the writer. Additionally, respondents seemed to value LORs for interview and ranking decisions.

Does "Hoarding" Impact Interview Distribution Among Orthopaedic Surgery Residency Applicants? A Modeling Study Based on a Large Database

BACKGROUND

"Interview hoarding" is commonly used to refer to the concentration of interview offers among a small number of high-performing residency applicants. Theoretically, if the same applicants interview at every program, fewer rank lists will be submitted than open residency positions, leading to a "match crisis" with unfilled positions after the match. There are no published studies we are aware of that describe the observed distribution of residency interview offers among orthopaedic surgery applicants or the potential impact of "hoarding" on that distribution.

QUESTIONS/PURPOSES

We examined the distribution of interview invitations extended to orthopaedic surgery residency applicants in the 2020 to 2021 and 2021 to 2022 application cycles. The change in the shape of the interview invitation distribution was the primary outcome for two central questions: (1) Does the interview offer distribution curve among orthopaedic surgery applicants change meaningfully from baseline with implementation of an interview cap (Model 1)? (2) What is the impact on the distribution of invitations with a reduction in the number of applicants in the field (Model 2)?

METHODS

This was a retrospective study of orthopaedic surgery interview invitations extended to applicants via the Thalamus interview management program during the 2020 to 2021 and 2021 to 2022 residency application cycles. The Thalamus database was chosen because it contains data on interview invitations for orthopaedic surgery residency positions and has the largest market share in orthopaedics compared with similar databases. Thalamus data represent 1565 applicants and 53 residency programs (90% and 25% of the national total, respectively) in 2021 to 2022 and 993 applicants and 46 programs (77% and 23%, respectively) in 2020 to 2021. It has been shown to contain a representative sample of orthopaedic residency programs. An interview cap (Model 1) was simulated by removing excess interviews held by applicants above the 75th and 95th percentiles, which were chosen to represent a formal cap and an informal cap, respectively. A reduction in the size of the applicant pool was similarly modeled by randomly removing 5% and 25% of applicants, chosen to simulate informal and formal application requirements, respectively. In both models, the excess interviews were redistributed among the remaining applicants.

RESULTS

Applicants received a mean of 1.8 ± 2.2 Thalamus interview invitations in 2020 to 2021 and 1.7 ± 2.4 invitations in 2021 to 2022, with no change to the overall distribution curve. A total of 39% (606 of 1565) of applicants received no Thalamus interview invitations in 2021 to 2022, 75% (1176 of 1565) received two or fewer, and < 1% (14 of 1565) of applicants received 10 or more invitations. Redistributing excess interviews held by the top 5% of applicants resulted in 2% (61 of 2651) of interviews being redistributed (Model 1). Removing 5% of the total applicant pool resulted in a redistribution of 3% (87 of 2651) of the interview invitations (Model 2).

CONCLUSION

Orthopaedic surgery interview data demonstrated an expected uneven distribution of interview invitations, with a small proportion of highly competitive applicants receiving a higher number of interview offers as well as a large group of applicants receiving no interview invitations in Thalamus. Concerns that "hoarding" would lead to a crisis resulting in many unmatched residency positions seemed unfounded, given the excess of applicants relative to positions and the minimal change in the distribution of interviews in the cap model.

CLINICAL RELEVANCE

Medical students applying to orthopaedic residency should seek individual advising to improve their individual odds of matching, while understanding that interview hoarding does not seem to alter the distribution of interviews. Program directors and medical students' advisors should be cognizant that a small proportion of applicants are broadly interviewed and may benefit from steps taken to ensure applicants have genuine interest in the program.

Over-Application and Interviewing in the 2021 United States Primary Care Virtual Recruitment Season

Importance Over-application and interviewing are believed to be widespread in residency recruitment. These may have increased during the 2021 virtual recruitment season. The increase does not correspond to an increase in available residency positions and likely results in more interviews with low probabilities of yielding matches. Prior work demonstrates that such marginal interviews are identifiable ‑ from key explanatory factors like same-state for interviewee and program ‑ in sufficient volume to allow programs to substantially decrease interviews. Objective To evaluate the importance of same-state relationships in primary care and to determine the extent of over-interviewing in the 2021 virtual recruitment season. Design The National Resident Matching Program and Thalamus merged match (outcomes) and interview (explanatory variables) data from primary care specialties (family medicine, internal medicine, pediatrics). Data were analyzed by logistic regression, trained on the 2017-2020 seasons, and projected on the 2021 season for testing. Setting The setting was the 2017-2021 main residency matches. Participants This comprised 4,442 interviewees applying to 167 residency programs in primary care. Intervention This included the transition to virtual recruitment from in-person recruitment in the 2021 residency recruitment season. Measurements A total of 20,415 interviews and 20,791 preferred programs with program and interviewee characteristics and match outcomes were included. Results Same-state geographic relations predicted match probability in primary care residency interviews better than medical school/residency affiliation, with 86.0% of interviewees matching consistently with their preferences for the same state. Same-state was more effective than medical school affiliations with programs in predicting matching. Eliminating interviews with less than a 5% probability of matching (upper 95% prediction limit) removed 31.5% of interviews. Conclusions and relevance The large number of low-match probability interviews demonstrates over-interviewing in primary care. We suggest that programs eliminate interview offers to applications falling below their chosen match probability threshold.

Orthopaedic Surgery Residency Program Adherence to Universal Interview Offer Day Guidelines: A Retrospective Analysis

INTRODUCTION

The 2020 to 2021 application cycle was marked by structural changes due to the COVID-19 pandemic. Adaptations included the American Orthopaedic Association Council of Residency Program Directors recommendations for a universal interview offer day (UIOD), synchronizing applicant interview offer release. As a novel process within orthopaedics, there are limited data on the execution and effect of a UIOD. Study goals include determining (1) residency program adherence to the Council of Residency Program Directors UIOD guidelines during the 2020 to 2021 cycle, (2) how quickly programs filled available interviews, and (3) any difference in time from release to booking between applicants using edu and com e-mail domains.

METHODS

Orthopaedic residency programs and applicants using the Thalamus interview management software platform during the 2020 to 2021 residency application cycle were analyzed, representing 46 residency programs and 993 interviewees (22% and 77% of the national total, respectively).

RESULTS

Of the programs included in this study, 19 (41%) were strictly adherent to the UIOD and time, 14 (30%) were weakly adherent and sent out offers outside of the assigned time, and 13 (28%) were nonadherent and sent out offers on dates other than the UIOD. The average time to fill to 80% capacity was 26 ± 14 minutes (range 3 to 77 minutes) for the 33 programs that released on the UIOD. Applicants with edu e-mail domains scheduled their first interview an average of 1.8 minutes after those with com e-mail domains (14.8 versus 13.0 minutes, P < 0.05).

CONCLUSION

Despite more than 60% of the residency programs committing to participate, less than half of the programs that initially agreed to participate were strictly adherent to guidelines during the first UIOD in orthopaedic surgery. Although additional research is needed to analyze the unique, fully virtual 2020 to 2021 recruitment season, a phased or waved approach to the UIOD may improve the process for all stakeholders in future cycles.

LEVEL OF EVIDENCE

Level V.

Geographic mobility in the emergency medicine residency match and the influence of gender

OBJECTIVES

Women are underrepresented in emergency medicine (EM) leadership. Some evidence suggests that geographic mobility improves career advancement. We compared movement between medical school and residency by gender. Our hypothesis was that women move a shorter distance than men.

METHODS

We collected National Residency Matching Program (NRMP) lists of ranked applicants from eight EM residency programs from the 2020 Main Residency Match. We added the gender expressed in interviews and left the Association of American Medical Colleges (AAMC) number as the unique identifier. Applicant data for matched osteopathic and allopathic seniors in the continental United States was included. We obtained street addresses for medical schools from an AAMC database and residency program addresses from the ACGME website. We performed geospatial analysis using ArcGIS Pro and compared results by gender. NRMP approved the data use and our institutional review board granted exempt status.

RESULTS

A total of 881 of 944 unique applicants met inclusion criteria and included 48.5% (830/1,713) of matched allopaths and 37% of all matched seniors; 48% (420) were female. There was no significant difference between genders for distance moved (p = 0.31). Women moved a mean (±SD) 619 (±698) miles (median = 341 miles, range = 0-2,679 miles); and men, a mean (±SD) 641 (±717) miles (median = 315 miles, range = 0-2,671 miles). Further analysis of applicants traveling less than 50 miles (49 women, 51 men) and by census division showed no significant frequency differences.

CONCLUSION

Women and men travel similar distances for EM residency with the majority staying within geographic proximity to their medical school. This suggests that professional mobility at this stage is not a constraint. Our study findings are limited because we do not know which personal and professional factors inform relocation decisions. Gender is not associated with a difference in distance moved by students for residency. This finding may have implications for resident selection and career development.

Reducing Over-Interviewing in the Anesthesiology Residency Match

Background The U.S. residency recruitment process is expensive and time-consuming because of application inflation and over-invitation. Objective Using interview and match data, we quantify the predicted effects if anesthesiology residency programs excluded interviews for applicants who are very unlikely to match. Methods We previously published the validity and accuracy of the logistic regression model based on data from interview scheduling software used by 32 U.S. anesthesiology residency programs and 1300 applicants from 2015-18. Data used were program region, applicant address, numbers of interviews of the interviewee, medical school US News and World Report (USNWR) rank, the difference between United States Medical Licensing Exam (USMLE) Step 1 and 2 Clinical Knowledge (CK) scores, and the historical average of USMLE scores of program residents. In the current study completed in 2020, the predicted probabilities and their variances were summed among interviewees for 30 deidentified programs. Results For anesthesiology, the median residency program could reduce their interviews by 16.9% (97.5% confidence interval 8.5%-24.1%) supposing they would not invite applicants if the 99% upper prediction limit for the probability of matching was less than 10.0%. The corresponding median savings would be 0.80 interviews per matched spot (0.34-1.33). In doing so, the median program would sustain a risk of 5.3% (97.5% confidence interval 2.3%-7.9%) of having at least one interviewee removed from their final rank-to-match list. Conclusion Using novel interview data and analyses, we demonstrate that residency programs can substantively reduce interviews with less effect on rank-to-match lists. The data-driven approach to manage marginal interviews allows program leadership to better weigh costs and benefits when composing their annual list of interviewees.

Red Flags, Geography, Exam Scores, and Other Factors Used by Program Directors in Determining Which Applicants Are Offered an Interview for Anesthesiology Residency

Objective

The goal of this study was to measure the most important factors in candidate applications that anesthesiology program directors (PDs) use to decide who to invite for an interview, and how that might change once the United States Medical Licensing Examination (USMLE) Step 1 is only reported as pass/fail.

Design

Based on a literature review, a comprehensive list of 27 factors used by PDs to select candidates for the interview was developed. An anonymous survey link was emailed to PDs of all Accreditation Council for Graduate Medical Education (ACGME) accredited Anesthesiology residencies. The survey asked PDs to rank order the top 10 factors they currently consider for making interview invitation, and then to repeat the rank ordering as if the USMLE Step 1 score was instead reported as pass/fail as will be done beginning in 2022.

Results

Forty-five of 159 (28%) PDs responded, with 82% disagreeing with changing the Step 1 score to pass/fail. 84% consider the Step 1 score (77% for Step 2) moderately or very important for selecting an applicant for an interview. The most frequently mentioned “red flags” were failure of a licensing exam, failure of a medical school course, gaps in education without explanation, and criminal history. 69% of PDs agreed that applicants coming from the medical school affiliated with their program would have an advantage over other applicants. Although, the three factors most commonly ranked in the top 10 in importance were the Step 1 score, followed by letters of recommendation, and then the Medical School Performance Evaluation, variability exists in how PDs ranked factors. For example, of the PDs that had Step 1 in the top 10, 27% had it ranked between the 6th and 10th most important. 9% of PDs did not have Step 1 score in the top 10. Core clinical clerkship grades were one of the top 5 factors by 49% of PDs, yet overall was the 6th most common top 10 factor as 36% of PDs did not have core clerkship grades at all in the top 10. Once Step 1 is reported only as pass/fail, PDs had letters of recommendation, Step 2, and the Medical School Performance Evaluation as the most frequently ranked factors in the top 10. 64% of the PDs supported restricting the number of programs a candidate can apply to, with the majority suggesting a limit of 15 to 20 programs per applicant.

Conclusion

Variability exists among anesthesiology PDs in the key criteria for offering an applicant an interview. Once Step 1 is reported as pass/fail, there will be an increased emphasis on Step 2 scores.