Use of the Interview in Resident Candidate Selection: A Review of the Literature

Resident Recruitment in a New Era

ABSTRACT

This chapter focuses on resident recruitment and recent US National Resident Matching Program changes and the impact in the evaluation and ranking of applicants within the specialty of anesthesiology. Recruitment challenges are examined as well as program strategies and potential future directions. Also discussed are DEI initiatives within the recruitment process.

Reimagining general surgery resident selection: Collaborative innovation through design thinking

Introduction

The process by which surgery residency programs select applicants is complex, opaque, and susceptible to bias. Despite attempts by program directors and educational researchers to address these issues, residents have limited ability to affect change within the process at present. Here, we present the results of a design thinking brainstorm to improve resident selection and propose this technique as a framework for surgical residents to creatively solve problems and generate actionable changes.

Methods

Members of the Collaboration of Surgical Education Fellows (CoSEF) used the design thinking framework to brainstorm ways to improve the resident selection process. Members participated in one virtual focus group focused on identifying pain points and developing divergent solutions to those pain points. Pain points and solutions were subsequently organized into themes. Finally, members participated in a second virtual focus group to design prototypes to test the proposed solutions.

Results

Sixteen CoSEF members participated in one or both focus groups. Participants identified twelve pain points and 57 potential solutions. Pain points and solutions were grouped into the three themes of transparency, fairness, and applicant experience. Members subsequently developed five prototype ideas that could be rapidly developed and tested to improve resident selection.

Conclusions

The design thinking framework can help surgical residents come up with creative ideas to improve pain points within surgical training. Furthermore, this framework can supplement existing quantitative and qualitative methods within surgical education research. Future work will be needed to implement the prototypes devised during our sessions and turn them into complete interventions.

Key message

In this paper, we demonstrate the results of a resident-led design thinking brainstorm on improving resident selection in which our team identified twelve pain points in resident selection, ideated 57 solutions, and developed five prototypes for further testing. In addition to sharing our results, we believe design thinking can be a useful framework for creative problem solving within surgical education.

Does Masked Interviewing Encourage Holistic Review in Residency Selection? A Mixed-Methods Study

Problem: Medical educators increasingly champion holistic review. However, in U.S. residency selection, holistic review has been difficult to implement, hindered by a reliance on standardized academic criteria such as board scores. Masking faculty interviewers to applicants' academic files is a potential means of promoting holistic residency selection by increasing the interview's ability to make a discrete contribution to evaluation. However, little research has directly analyzed the effects of masking on how residency selection committees evaluate applicants. This mixed-methods study examined how masking interviews altered residency selection in an anesthesiology program at a large U.S. academic medical center. Intervention: During the 2019-2020 residency selection season in the University of Pennsylvania's Department of Anesthesiology & Critical Care, we masked interviewers to the major academic components of candidates' application files (board scores, transcripts, letters) on approximately half of interview days. The intent of the masking intervention was to mitigate the tendency of interviewers to form predispositions about candidates based on standardized academic criteria and thereby allow the interview to make a more independent contribution to candidate evaluation. Context: Our examination of the masking intervention used a concurrent, partially mixed, equal-status mixed-methods design guided by a pragmatist approach. We audio-recorded selection committee meetings and qualitatively analyzed them to explore how masking affected the process of candidate evaluation. We also collected independent candidate ratings from interviewers and consensus committee ratings and statistically compared ratings of candidates interviewed on masked days to ratings from conventional days. Impact: In conventional committee meetings, interviewers focused on how to reconcile academic metrics and interviews, and their evaluations of interviews were framed according to predispositions about candidates formed through perusal of application files. In masked meetings, members instead spent considerable effort evaluating candidates' "fit" and whether they came off as tactful. Masked interviewers gave halting opinions of candidates and sometimes pushed for committee leaders to reveal academic information, leading to masking breaches. Higher USMLE Step 1 score and higher medical school ranking were statistically associated with more favorable consensus rating. We found no significant differences in rating outcomes between masked and conventional interview days. Lessons learned: Elimination of academic metrics during the residency interview phase does not straightforwardly promote holistic review. While critical reflection among medical educators about the fairness and utility of such metrics has been productive, research and intervention should focus on the more proximate topic of how programs apply academic and other criteria to evaluate applicants.

Evidence-Based Practices for Interviewing Graduate Medical Education Applicants: A Systematic Review

Background Although the selection interview is a standard admission practice for graduate medical education (GME) programs in the United States, there is a dearth of recent reviews on optimizing the trainee interview process, which has low reliability, high cost, and major risk of bias. Objective To investigate the evidence base for different selection interview practices in GME. Methods We searched 4 literature databases from inception through September 2022. Two investigators independently conducted title/abstract screening, full-text review, data extraction, and quality assessment. Disagreements were mediated by discussion. We used backward reference searching of included articles to identify additional studies. We included studies of different interview methods and excluded literature reviews, non-GME related publications, and studies comparing different applicant populations. We examined study characteristics, applicant and interviewer preferences, and interview format. We evaluated study quality using the Medical Education Research Study Quality Instrument (MERSQI). Results Of 2192 studies, 39 (2%) met our inclusion criteria. The evidence base was rated as moderately low quality using MERSQI criteria. Applicants reported preferences for several one-on-one interviews lasting 15 to 20 minutes, interviews by current trainees, and interviews including social events with only trainees. Applicants had mixed perceptions of virtual versus in-person interviews and reported that virtual interviews saved costs. The multiple mini interview (MMI) required more applicant and interviewer time than individual interviews but demonstrated construct and predictive validity and was preferred by applicants and interviewers. Conclusions Based on moderately low-quality evidence, using the MMI, training interviewers, and providing applicants with basic program information in advance should be considered for GME selection interviews.

Otolaryngology Applicant Perspectives on In-person and Virtual Residency Interviews

OBJECTIVE

The objective of this study was to understand applicant perspectives on in-person and virtual otolaryngology residency interviews.

STUDY DESIGN

Survey study.

SETTING

Otolaryngology residency applicants who were interviewed during 2022-2023.

METHODS

Survey sent to all otolaryngology residency applicants who interviewed during the 2022-2023 interview season.

RESULTS

A total of 499 applicants were surveyed with 150 responses (30%). Approximately 48.3% of respondents were offered an in-person interview with 78.9% accepting the offer. Of those who did not accept, reasons included not wanting to travel (21.1%) and time conflicts (15.5%). When comparing virtual versus in-person interviews, those who attended virtual interviews were more likely to disagree that they connected with residents (P = .02) and that they had an improved perspective of the program (P = .002). The majority of applicants agreed that virtual interviews are more inclusive and equitable than in-person interviews (70.4%). When asked which interview style applicants would prefer, 63.1% of applicants preferred an in-person interview when compared to virtual with a second look option (29.5%) and virtual (7.4%). Respondents who self-identified as being underrepresented in medicine were less likely to choose in-person as their preferred interview format (P = .01) and were more likely to decline an in-person interview offer due to monetary limitations (P = .04).

CONCLUSIONS

Applicants indicated dissatisfaction with connecting with residents and improving their perspective of the program when in a virtual setting. Applicants felt that virtual interviews were more equitable, but that if the barriers to equity were lessened then they would prefer in-person interviews.

Development and validation of an objective virtual reality tool for assessing technical aptitude among potential candidates for surgical training

BACKGROUND

Good technical skills are crucial for surgeons. Yet although surgical training programs strive to assess technical aptitude when selecting surgical residents, valid assessments of such aptitude are still lacking. Surgical simulators have been proposed as a potentially effective tool for this purpose. The current study aims to develop a technical aptitude test using a virtual reality surgical simulator, and to validate its use for the selection of surgical residents.

METHODS

The study had three phases. In Phase 1, we developed an initial version of the technical aptitude test using the Lap-X-VR laparoscopic simulator. In Phases 2 and 3 we refined the test and collected empirical data to evaluate four main sources of validity evidence (content, response process, internal structure, and relationships with other variables), and to evaluate the feasibility and acceptability of the test. Specifically, Phase 2 comprised a review of the test by 30 senior surgeons, and in Phase 3 a revised version of the test was administered to 152 interns to determine its psychometric properties.

RESULTS

Both the surgeons and interns rated the test as highly relevant for selecting surgical residents. Analyses of the data obtained from the trial administration of the test supported the appropriateness of the score calculation process and showed good psychometric properties, including reliability (α = 0.83) and task discrimination (mean discrimination = 0.5, SD = 0.1). The correlations between test scores and background variables revealed significant correlations with gender, surgical simulator experience, and video game experience (ps < 0.001). These variables, however, explained together only 10% of the variance in test scores.

CONCLUSIONS

We describe the systematic development of an innovative virtual reality test for assessing technical aptitude in candidates for surgical training, and present evidence for its validity, feasibility and acceptability. Further validation is required to support the application of the test for selection, as well as to discern the impact of gender, surgical simulator experience, and video game experience on the fairness of test results. However, the test appears to be a promising tool that may help training programs assess the suitability of candidates for surgical training.

The Chief Residency in U.S. and Canadian Graduate Medical Education: A Scoping Review

PHENOMENON

Despite the nearly universal presence of chief residents within U.S. and Canadian residency programs and their critical importance in graduate medical education, to our knowledge, a comprehensive synthesis of publications about chief residency does not exist. An understanding of the current state of the literature can be helpful to program leadership to make evidence-based improvements to the chief residency and for medical education researchers to recognize and fill gaps in the literature.

APPROACH

We performed a scoping review of the literature about chief residency. We searched OVID Medline, PsycINFO, ERIC, and Web of Science databases through January 2023 for publications about chief residency. We included publications addressing chief residency in ACGME specialties in the U.S. and Canada and only those using the term "chief resident" to refer to additional responsibilities beyond the typical residency training. We excluded publications using chief residents as a convenience sample. We performed a topic analysis to identify common topics among studies.

FINDINGS

We identified 2,064 publications. We performed title and abstract screening on 1,306 and full text review on 208, resulting in 146 included studies. Roughly half of the publications represented the specialties of Internal Medicine (n = 37, 25.3%) and Psychiatry (n = 30, 20.5%). Topic analysis revealed six major topics: (1) selection of chief residents (2) qualities of chief residents (3) training of chief residents (4) roles of chief residents (5) benefits/challenges of chief residency (6) outcomes after chief residency.

INSIGHTS

After reviewing our topic analysis, we identified three key areas warranting increased attention with opportunity for future study: (1) addressing equity and bias in chief resident selection (2) establishment of structured expectations, mentorship, and training of chief residents and (3) increased attention to chief resident experience and career development, including potential downsides of the role.

Predictors of Ophthalmology Resident Performance From Medical Student Application Materials

OBJECTIVE

To determine whether elements in ophthalmology residency applications are predictors of future resident performance.

DESIGN

This multi-institutional, cross-sectional, observational study retrospectively reviewed the residency application materials of ophthalmology residents who graduated from residency from 2006 through 2018. Resident performance was scored by 2 faculty reviewers in 4 domains (clinical, surgical, academic, and global performance). Correlation between specific elements of the residency application and resident performance was assessed by Spearman correlation coefficients (univariate) and linear regression (multivariate) for continuous variables and logistic regression (multivariate) for categorical variables.

SETTING

Seven ophthalmology residency programs in the US.

PARTICIPANTS

Ophthalmology residents who graduated from their residency program.

RESULTS

High-performing residents were a diverse group, in terms of sex, ethnicity, visa status, and educational background. Residents with United States Medical Licensing Examination Step 1 scores higher than the national average for that year had significantly higher scores in all 4 performance domains than those who scored at or below the mean (all domains P < 0.05). Residents who had honors in at least 4 core clerkships and who were members of Alpha Omega Alpha Medical Honor Society also had higher scores in all 4 performance domains (all domains P ≤ 0.04). Step 1 score (ρ=0.26, P < 0.001) and the difference between Step 1 score and the national average for that year (ρ=0.19, P = 0.009) positively correlated with total resident performance scores. Residents who passed the American Board of Ophthalmology Written Qualifying Examination or Oral Examination on their first attempt had significantly higher Step 1/2 scores (P ≤ 0.005), Ophthalmology Knowledge Assessment Program scores (P = 0.001), and resident performance scores (P ≤ 0.004).

CONCLUSIONS

In this new landscape of increasing numbers of applicants to residency programs and changing of the Step 1 score to pass/fail, our findings may help guide selection committees as they holistically review applicants to select exceptional future residents in ophthalmology.

'Selection into training will always be an inexact process': A survey of Directors of Physician Education on selection into Basic Physician Training in Australia and New Zealand

BACKGROUND

Despite being one of the largest medical specialty training programmes in Australasia, there is no standardised method for selection into Basic Physician Training (BPT), and limited data exist regarding current practices.

AIMS

To address existing knowledge gaps, we aimed to create a 'snapshot' of current BPT selection practices and explore the perspectives of Directors of Physician Education (DPEs) regarding trainee selection.

METHODS

An electronic survey of DPEs from adult and paediatric medicine BPT sites in Australia and New Zealand was undertaken in January-February 2022. A combination of free text, multiple-choice and yes/no answers were analysed using descriptive statistics and qualitative content analysis.

RESULTS

A total of 70 responses were received, achieving a response rate of 35% (70/198). Selection practices were found to be heterogenous across BPT sites. Respondents had varying opinions regarding the utility of selection tools and desirable candidate attributes. A heavy reliance upon interviews and the reported use of subjective assessments raise concerns for selection process bias.

CONCLUSION

BPT sites should critically evaluate their selection methods, and more research in this field is needed to establish best practice.

A Systematic Review of Metrics Utilized in the Selection and Prediction of Future Performance of Residents in the United States

Background Aligning resident and training program attributes is critical. Many programs screen and select residents using assessment tools not grounded in available evidence. This can introduce bias and inappropriate trainee recruitment. Prior reviews of this literature did not include the important lens of diversity, equity, and inclusion (DEI). Objective This study's objective is to summarize the evidence linking elements in the Electronic Residency Application Service (ERAS) application with selection and training outcomes, including DEI factors. Methods A systematic review was conducted on March 30, 2022, concordant with PRISMA guidelines, to identify the data supporting the use of elements contained in ERAS and interviews for residency training programs in the United States. Studies were coded into the topics of research, awards, United States Medical Licensing Examination (USMLE) scores, personal statement, letters of recommendation, medical school transcripts, work and volunteer experiences, medical school demographics, DEI, and presence of additional degrees, as well as the interview. Results The 2599 identified unique studies were reviewed by 2 authors with conflicts adjudicated by a third. Ultimately, 231 meeting inclusion criteria were included (kappa=0.53). Conclusions Based on the studies reviewed, low-quality research supports use of the interview, Medical Student Performance Evaluation, personal statement, research productivity, prior experience, and letters of recommendation in resident selection, while USMLE scores, grades, national ranking, attainment of additional degrees, and receipt of awards should have a limited role in this process.

Successful Integration of Blinded Interviews for Resident Selection: Applicant and Faculty Perspective

OBJECTIVE

To evaluate interviewer and interviewee perceptions of semiblinded interviews performed during 2021-2022 and 2022-2023 urology matches at our institution. Traditional interviews, where interviewers have access to the entire application, are open to significant bias. Blinded interviews are common in industry but under explored in resident selection.

METHODS

Interviewers had access to a limited portion of the application (personal statement and letters of recommendation). Applicants were ranked by faculty based solely on their interview and these documents. Following the interview, a survey was given to applicants and faculty regarding their experience.

RESULTS

A total of 67 applicants and 10 faculty responded to the questionnaire. Among applicants, 51% felt that blinding of interviewers offered a better assessment of fit into our program (39% neutral), while 37% felt they had improved eye contact with the interviewer (51% neutral) and that interviewers had improved (66%) or similar (19%) knowledge of their application in relation to nonblinded interviews. All but one faculty member felt able to accurately assess a candidate with the information provided, and 80% felt that the blinded interview allowed them to focus more on the applicant during the interview (20% neutral).

CONCLUSION

Semiblinded interviews allow for accurate assessment of applicants and decrease bias in the interview process. Overall applicants and faculty were highly receptive to blinded interviews. Reducing the amount of information available to the interviewer allowed them to spend more time on the materials provided, leading to both improved eye contact and improved engagement between participants.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

PURPOSE

The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.

METHOD

Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.

RESULTS

Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.

CONCLUSIONS

This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Association of Interview and Holistic Review Metrics With Resident Performance-Related Difficulties in an Internal Medicine Residency Program

Background The utility of traditional academic factors to predict residency candidates' performance is unclear. Many programs utilize holistic review processes assessing applicants on an expanded range of application and interview characteristics. Determining which characteristics might predict performance-related difficulty in residency is needed. Objective We aim to elucidate factors associated with residency performance-related difficulty in a large academic internal medicine residency program. Methods In 2022, we conducted a retrospective cohort study of Electronic Residency Application Service and interview data for residents matriculating between 2018 and 2020. The primary outcome was a composite of performance-related difficulty during residency (referral to the Clinical Competency Committee; any rotation evaluation score of 2 out of 5 or lower; and/or a confidential "comment of concern" to the program director). Logistic regression models were fit to assess associations between resident characteristics and the composite outcome. Results Thirty-eight of 117 residents met the composite outcome. Gold Humanism Honor Society (odds ratio [OR] 0.24, 95% confidence interval [CI] 0.16-0.87) or Alpha Omega Alpha (OR 0.36, 95% CI 0.14-0.99) members were less likely to have performance-related difficulty, as were residents with higher United States Medical Licensing Examination Step 2 Clinical Knowledge scores (OR 0.97, 95% CI 0.47-1.00). One-point increases in general faculty overall interview score, leadership competency score, and leadership overall score were associated with 41% to 63% lower odds of meeting the composite outcome. Interview or file review "flags" had an OR of 2.82 (95% CI 1.37-5.80) for the composite outcome. Conclusions Seven metrics were associated with the composite outcome of resident performance-related difficulty.

RESIDENTS AS INTERVIEWERS: A COMPARISON OF RESIDENT AND FACULTY EVALUATIONS OF EMERGENCY MEDICINE RESIDENCY APPLICANTS

BACKGROUND

Interviews are an integral component of the residency selection process. Many programs use current residents as interviewers in addition to faculty. Although the reliability of interview scores between faculty members has been examined, little is known about the reliability between resident and faculty interviewers.

OBJECTIVE

This study evaluates the reliability of residents as interviewers compared with faculty.

METHODS

A retrospective analysis of interview scores from the 2020-2021 application cycle was performed at an emergency medicine (EM) residency program. Each applicant participated in five separate one-on-one interviews led by four faculty members and one senior resident. Interviewers assigned applicants a score from 0 to 10. Consistency between interviewers was measured using the intraclass correlation coefficient (ICC). Generalizability theory was used to measure variance components including applicant, interviewer, and rater type (resident vs. faculty) and their impact on scoring.

RESULTS

There were 250 applicants interviewed by 16 faculty members and 7 senior residents during the application cycle. The mean (SD) interview score given by resident interviewers was 7.10 (1.53) and the mean (SD) score given by faculty was 7.07 (1.69). There was no statistically significant difference between the pooled scores (p = 0.97). Reliability between interviewers was good to excellent (ICC = 0.90; 95% CI 0.88-0.92). The generalizability study showed most score variance was attributable to applicant characteristics and only 0.6% was attributable to interviewer or rater type (resident vs. faculty).

CONCLUSIONS

There was strong concordance between faculty and resident interview scores indicating reliability of EM resident scoring compared to faculty.

Effect of Holistic Review, Interview Blinding, and Structured Questions in Resident Selection: Can we Predict Who Will Do Well in a Residency Interview?

OBJECTIVE

To examine the Urology residency application process, particularly the interview. Historically, the residency interview has been vulnerable to bias and not determined to be a predictor of future residency performance. Our goal is to determine the relationship between pre-interview metrics and post-interview ranking using best practices for Urology resident selection including holistic review, blinded interviews, and structured behaviorally anchored questions.

METHODS

Applications were assessed on cognitive (Alpha Omega Alpha, class rank, junior year clinical clerkship grades) and non-cognitive attributes (letters of recommendation [LOR], personal statement [PS], demographics, research, personal characteristics) by reviewers blinded to USMLE scores and photograph. Interviewers were blinded to the application other than PS and LORs. Interviews consisted of a structured behaviorally anchored question (SBI) and an unstructured interview (UI). Odds ratios were determined comparing pre-interview and interview impressions.

RESULTS

Fifty-one applicants were included in the analysis. USMLE step 1 score (average 245) was associated with Alpha Omega Alpha, class rank, junior year clinical clerkship, and PS. The UI score was associated with the LOR (P = .04) whereas SBI scores were not (P = .5). Faculty rank was associated with SBI, UI, and overall interview (OI) scores (P < .001). Faculty rank was also associated with LOR. Resident impression of interviewees were associated with faculty interview scores (P = .001) and faculty rank (P < .001).

CONCLUSION

Traditional interviews may be biased toward application materials and may be balanced with behavioral questions. While Step 1 score does not offer additional information over other PI metrics, blinded interviews may offer discriminant validity over a PI rubric.

Implementation of the AAMC's Holistic Review Model for Psychiatry Resident Recruitment

Introduction

In psychiatry, several converging factors are impacting the recruitment of residents: the increased competitiveness of the specialty, the national trend to take active steps to improve diversity and inclusion, and the decision from USMLE to change Step 1 to a pass/fail result.

Methods

We developed a workshop for psychiatry residency program directors to meet these challenges and transition into using a holistic review model during recruitment. The workshop included (1) a didactic session providing background on the AAMC holistic review model; (2) a small-group exercise to determine and prioritize experiences, attributes, competencies, and metrics (EACMs) aligned with the program's mission and aims; (3) a review of the rankings from the previous exercise, selection of two "very important" criteria for each of the four domains of the EACM model, and operationalization of these criteria based on the recruitment process; and (4) a discussion focused on application of program criteria with example applicants.

Results

The holistic review workshop was conducted at the American Association of Psychiatry Residency Directors conference in 2021 with 48 self-selected attendees. Following the workshop, 74% of attendees reported a likelihood of implementing holistic applications during their next application cycle, 78% were able to leave with at least one actionable item, 100% thought that the session was interactive, and 78% felt that the session met their expectations.

Discussion

Implementing a holistic review for psychiatry residency recruitment can assist programs in responding to the rapidly changing landscape and achieve aims for improving diversity and inclusion.

Selection of Neurosurgical Applicants in the High-Income Developing Country Lacking Local Residency Program: A Cross-Sectional Study

Background  Neurosurgery residency became one of the most competitive specialties in the medical field worldwide, which is increasing with time in contrast to the limited positions. Therefore, the requirements for the program have increased. There are different criteria for each program, which are determined by specific factors. It has become increasingly important for medical students to be aware of the factors that affect their acceptance into the program. There was a lack of data regarding the factors that contribute to the selection of neurosurgery residents in Oman Methods  A questionnaire composed of 14 questions was conducted, using the SurveyMonkey Web site, among neurosurgeons in Oman which was distributed to the five hospitals that have neurosurgery departments in Oman. SPSS software was used in the analysis of the collected data. Results  Forty-four participants responded to the survey. Ninety-five percent of them answered all the questions. Out of all participants, only two were female participants. Standardized international exam scores, such as the United State Medical Licensing Examination and Medical Council of Canada Qualifying Examination, ranked as the most important factor with a percentage of 44, followed by interview performance with a percentage of 33. While the least important factor was the age of applicants, which 46% of the participants ranked 8. Conclusion  Most of the participants agreed that standardized exams are the most important factor in the selection of neurosurgery residents followed by interview performance, although there was no significant statistical difference between the two.

The effect of assessor professional background on interview evaluation during residency selection: A mixed-methods study

BACKGROUND

Residency interviewer scores are greatly variable and seems to be influenced by personal characteristics of assessors, although factors contributing to variability remain unclear. The study sought to determine how different professional backgrounds influence assessors' scores.

METHODS

Fifty-five general surgery applicants rotated through an interview station assessing teamwork. They were scored by surgeons, human-resource managers, pilots, athletes. Pearson's correlation and a repeated-measures ANOVA were used to determine correlations between professions. Structured interviews were used to probe for scoring rationale.

RESULTS

Interview scores differed significantly between professions (F (3, 159) = 11.12, p < 0.001. Qualitative analysis revealed that due to the challenge of distinguishing between similarly performing candidates, assessors rely on global impressions informed by personal values.

CONCLUSION

Assessor variability is ubiquitous, in part due to the subjective nature of interviews and is associated with personal values. When selecting assessors, programs should choose diverse assessors to assess to ensure a reliable selection process.

Using Job Analysis for Identifying the Desired Competencies of 21st-Century Surgeons for Improving Trainees Selection

OBJECTIVE

The current selection for surgical training is based on ineffective methods. In order to identify or to develop more valid selection tools to improve the selection, it is necessary to first define what are the competencies that are most important for success in contemporary surgery. Therefore, the current study aims to identify what competencies are required for success as a surgeon in the 21st-century and to evaluate their relative importance for selection for surgical training.

METHODS

Job analysis was conducted using a mixed-methods design. First, 104 senior surgeons from all surgical fields from various hospitals in Israel were interviewed in order to query their perceptions of competencies associated with success as a surgeon. Their answers were coded and analyzed to create a list of important competencies. Next, a larger sample of 1,102 surgeons and residents from all surgical fields completed a questionnaire in which they rated the importance of each competency in the list for success as a surgeon and for selection for surgical training in the 21st-century.

RESULTS

Twenty-four competencies (five technical skills, six cognitive abilities, 13 personality characteristics) were identified in the interview analysis. Analysis of the questionnaire's data revealed that all 24 competencies were perceived as important for success as a surgeon in the 21st-century as well as for selection for surgical training. The perceived importance of personality characteristics was higher than both cognitive abilities (p < 0.001) and technical skills (p < 0.001). The results did not differ between different surgical fields.

CONCLUSIONS

Twenty-four competencies were identified as important for 21st-century surgeons and for selection for surgical training. Although all competencies were perceived as important, personality characteristics were perceived as more important than technical skills and cognitive abilities. This updated definition of required competencies may aid in developing more valid selection methods of candidates for surgical training.

Virtual interviewing for graduate medical education recruitment and selection: A BEME systematic review: BEME Guide No. 80

BACKGROUND

The COVID-19 pandemic caused graduate medical education (GME) programs to pivot to virtual interviews (VIs) for recruitment and selection. This systematic review synthesizes the rapidly expanding evidence base on VIs, providing insights into preferred formats, strengths, and weaknesses.

METHODS

PubMed/MEDLINE, Scopus, ERIC, PsycINFO, MedEdPublish, and Google Scholar were searched from 1 January 2012 to 21 February 2022. Two authors independently screened titles, abstracts, full texts, performed data extraction, and assessed risk of bias using the Medical Education Research Quality Instrument. Findings were reported according to Best Evidence in Medical Education guidance.

RESULTS

One hundred ten studies were included. The majority (97%) were from North America. Fourteen were conducted before COVID-19 and 96 during the pandemic. Studies involved both medical students applying to residencies (61%) and residents applying to fellowships (39%). Surgical specialties were more represented than other specialties. Applicants preferred VI days that lasted 4-6 h, with three to five individual interviews (15-20 min each), with virtual tours and opportunities to connect with current faculty and trainees. Satisfaction with VIs was high, though both applicants and programs found VIs inferior to in-person interviews for assessing 'fit.' Confidence in ranking applicants and programs was decreased. Stakeholders universally noted significant cost and time savings with VIs, as well as equity gains and reduced carbon footprint due to eliminating travel.

CONCLUSIONS

The use of VIs for GME recruitment and selection has accelerated rapidly. The findings of this review offer early insights that can guide future practice, policy, and research.

The Virtual Interview Experience: Advantages, Disadvantages, and Trends in Applicant Behavior

Residency programs and applicants were forced to hold virtual interviews during the 2020-2021 application cycle. Inability to evaluate a program and/or applicant in person has intangible drawbacks. However, there are obvious advantages: cost, convenience, and comfort. Do the advantages outweigh the disadvantages? How have applicant behaviors changed to learn about programs in a virtual-only interview process?

METHODS

A survey was distributed to 302 applicants to a single plastic surgery residency program during the 2020 application cycle. Demographics, social media presence and utilization, and experience with the virtual application and interview process were analyzed. A 2018 survey from our institution was compared with a subset of questions for longitudinal analysis.

RESULTS

Seventy-six respondents (25.2%) completed the survey. Most applicants (88.2%) spent less than $1000 during the interview and application cycle. Over half (56.6%) did not receive letters of recommendation from outside their home program. A significant minority (27.6%) of applicants attended more than one interview in a single day. Compared to 2018, applicants in 2021 were significantly more likely to access alternative digital resources (forums/discussion boards, social media, and podcasts) when learning about programs. Average number of interviews remains in the range of pre-COVID studies, but the percentage of interviews attended increased.

CONCLUSIONS

Applicants spent substantially less money on interviews and relied on alternative digital sources to learn about residency programs. This study objectively quantifies the advantages of virtual interviews. Disadvantages include inability to assess "fit" and lack of nonverbal communication.

Recommendations to improve recruitment and retention of underrepresented in medicine trainees in neonatal-perinatal medicine

Physicians who identify as Black, Latinx, American Indian, Pacific Islander, and certain Asian subgroups represent racial and ethnic populations that are underrepresented in medicine (URM). While the proportion of URM pediatric trainees has remained unchanged, that of Neonatal-Perinatal Medicine (NPM) fellows has decreased. Informed by the medical literature and our lived experiences, we compiled and developed a list of recommendations to support NPM fellowship programs in the recruitment, retention, and promotion of URM trainees. We describe ten recommendations that address 1) creating a culture of inclusivity and psychological safety, 2) the critical appraisal of recruitment practices and climate, and 3) an inclusive and holistic fellowship application process. The first two themes lay the foundation, while the final theme spotlights our recommendations for URM recruitment. Each recommendation is a step towards improvement in recruitment and inclusion at a program.

Faculty Scoring of General Surgery Residency Interviewees: A Comparison of In-Person and Virtual Interview Formats

OBJECTIVE

With new rules regarding social distancing and non-essential travel bans, we sought to determine if faculty scoring of general surgery applicants would differ between the in-person interview (IPI) and virtual interview (VI) platforms.

DESIGN

A single institution, retrospective review comparing faculty evaluation scores of applicant interviewees in the 2019 and 2020 MATCH® application cycles (IPIs) and the 2021 and 2022 application cycle (VIs) was conducted. Faculty scored applicants using a 5-point Likert scale in 7 areas of assessment and assigned each student to 1 of 4 tiers (tier 1 highest). A composite score for the 7 assessments (maximum score 35) was calculated. Mean and composite scores and tiers were compared between VI and IPI cycles and adjusted for within-interviewer correlations. The variance of the 2 groups were also compared.

SETTING

Harbor-UCLA Medical Center, an academic, tertiary care hospital.

PARTICIPANTS

General Surgery applicants for the 2019 to 2022 MATCH® application cycles.

RESULTS

Four hundred forty-one faculty IPI ratings of General Surgery applicants were compared to 531VI ratings. No difference in mean composite scores, individual assessments, or tier ranking. Less variance was identified in the VI group for academic credentials (0.6 vs 0.6, p = 0.01), strength of letters (0.7 vs 0.4, p = 0.005), communication skills (0.4 vs 0.6, p = 0.01), personal qualities (0.2 vs 0.5, p = 0.02), overall sense of fit for program (0.6 vs 0.9, p = 0.01), and tier ranking (0.3 vs 0.4, p = 0.004).

CONCLUSIONS

Faculty ratings of General Surgery applicants in the VI format appear to be similar to IPI. However, faculty ratings of VI applicants demonstrated less variability in scores in most assessments. This finding is potentially concerning, as it may suggest an inability of VI to detect subtle differences between applicants as comparted to IPI.

A Semistructured Interview for Surgical Residency Targeting Nontechnical Skills

OBJECTIVE

We review the development, implementation, and initial outcomes of a semistructured interview process to assess the nontechnical skills of surgical residency applicants.

DESIGN

In 2018, we restructured our residency selection interview process. Through semistructured faculty interviews, we sought to evaluate candidates along seven nontechnical skills (grit, ownership, rigor, teamwork, presence, impact, and organizational alignment). We plotted each candidate's scores on a radar plot for graphical representation and calculated the plot area of each candidate. We retrospectively evaluated 3 years of data, comparing the nontechnical skill scores of matriculants into the training program to those of nonmatriculants.

SETTING

Tenwek Hospital is a 361-bed tertiary teaching and referral hospital in rural western Kenya with a 5-year general surgery residency program.

PARTICIPANTS

Thirty-one applicants were interviewed over 3 years. Thirteen matriculated into the program.

RESULTS

Scores for grit, (4.8 vs 3.9; p = 0.0004), impact (4.2 vs 3.5; p = 0.014), ownership (4.2 vs 3.6; p = 0.01), and organizational alignment (4.3 vs 3.8; p = 0.008) were significantly higher in matriculants.

CONCLUSIONS

This semistructured interview process provides a robust and beneficial mechanism for assessing applicants' nontechnical skills, which may allow for the matriculation of more well-rounded candidates into surgical residency and, ultimately, surgical practice.

Simulation as Soothsayer: Simulated Surgical Skills MMIs During Residency Interviews are Associated With First Year Residency Performance

OBJECTIVE

The main consideration during residency recruitment is identifying applicants who will succeed during residency. However, few studies have identified applicant characteristics that are associated with competency development during residency, such as the Accreditation Council for Graduate Medical Education milestones. As mini multiple interviews (MMIs) can be used to assess various competencies, we aimed to determine if simulated surgical skills MMI scores during a general surgery residency interview were associated with Accreditation Council for Graduate Medical Education milestone ratings at the conclusion of intern year.

DESIGN

Retrospective cohort study. Interns' Step 1 and 2 clinical knowledge (CK) scores, interview day simulated surgical skills MMI overall score, traditional faculty interview scores, average overall milestone ratings in the spring of residency, and intern American Board of Surgery In-Training Examination (ABSITE) percentile scores were gathered. Two multiple linear regression were performed analyzing the association between Step 1, Step 2 CK, MMI, and traditional faculty interview scores with (1) average overall milestone rating and (2) ABSITE percentile scores, controlling for categorical/preliminary intern classification.

SETTING

One academic medical center PARTICIPANTS: General surgery interns matriculating in 2020-2021 RESULTS: Nineteen interns were included. Multiple linear regression revealed that higher overall simulated surgical skills MMI score was associated with higher average milestone ratings (β = .45, p = 0.03) and higher ABSITE score (β = .43, p = 0.02) while neither Step 1, Step 2 CK, nor faculty interview scores were significantly associated with average milestone ratings.

CONCLUSIONS

Surgical residency programs invest a tremendous amount of effort into training residents, thus metrics for predicting applicants that will succeed are needed. Higher scores on a simulated surgical skills MMIs are associated with higher milestone ratings 1 year into residency and higher intern ABSITE percentiles. These results indicate a noteworthy method, simulated surgical skills MMIs, as an additional metric that may select residents that will have early success in residency.

Best Practices for Reducing Bias in the Interview Process

PURPOSE OF REVIEW

Objective measures of residency applicants do not correlate to success within residency. While industry and business utilize standardized interviews with blinding and structured questions, residency programs have yet to uniformly incorporate these techniques. This review focuses on an in-depth evaluation of these practices and how they impact interview formatting and resident selection.

RECENT FINDINGS

Structured interviews use standardized questions that are behaviorally or situationally anchored. This requires careful creation of a scoring rubric and interviewer training, ultimately leading to improved interrater agreements and biases as compared to traditional interviews. Blinded interviews eliminate even further biases, such as halo, horn, and affinity bias. This has also been seen in using multiple interviewers, such as in the multiple mini-interview format, which also contributes to increased diversity in programs. These structured formats can be adopted to the virtual interviews as well. There is growing literature that using structured interviews reduces bias, increases diversity, and recruits successful residents. Further research to measure the extent of incorporating this method into residency interviews will be needed in the future.

Evaluation of house staff candidates for program fit: a cohort-based controlled study

BACKGROUND

Medical school academic achievements do not necessarily predict house staff job performance. This study explores a selection mechanism that improves house staff-program fit that enhances the Accreditation Council for Graduate Medical Education Milestones performance ratings.

OBJECTIVE

Traditionally, house staff were selected primarily on medical school academic performance. To improve residency performance outcomes, the Program designed a theory-driven selection tool to assess house staff candidates on their personal values and goals fit with Program values and goals. It was hypothesized cohort performance ratings will improve because of the intervention.

METHODS

Prospective quasi-experimental cohort design with data from two house staff cohorts at a university-based categorical Internal Medicine Residency Program. The intervention cohort, comprising 45 house staff from 2016 to 2017, was selected using a Behaviorally Anchored Rating Scales (BARS) tool for program fit. The control cohort, comprising 44 house staff from the prior year, was selected using medical school academic achievement scores. House staff performance was evaluated using ACGME Milestones indicators. The mean scores for each category were compared between the intervention and control cohorts using Student's t-tests with Bonferroni correction and Cohen's d for effect size.

RESULTS

The cohorts were no different in academic performance scores at time of Program entry. The intervention cohort outperformed the control cohort on all 6 dimensions of Milestones by end-PGY1 and 3 of 6 dimensions by mid-PGY3.

CONCLUSION

Selecting house staff based on compatibility with Residency Program values and objectives may yield higher job performance because trainees benefit more from a better fit with the training program.

Effect of teleconferencing variables on faculty impression of mock residency applicants

Purpose

The objective of this study was to assess how teleconferencing variables influence faculty impressions of mock residency applicants.

Methods

In October 2020, we conducted an online experiment studying five teleconferencing variables: background, lighting, eye contact, internet connectivity, and audio quality. We created interview videos of three mock residency applicants and systematically modified variables in control and intervention conditions. Faculty viewed the videos and rated their immediate impression on a 1-10 scale. The effect of each variable was measured as the mean difference between the intervention and control impression ratings. One-way analysis of variance (ANOVA) was performed to assess whether ratings varied across applicants. Paired-samples Wilcoxon signed-rank tests were conducted to assess the significance of the effect of each variable.

Results

Of 711 faculty members who were emailed a link to the experiment, 97 participated (13.6%). The mean ratings for control videos were 8.1, 7.2, and 7.6 (P < .01). Videos with backlighting, off-center eye contact, choppy internet connectivity, or muffled audio quality had lower ratings when compared with control videos (P < .01). There was no rating difference between home and conference room backgrounds (P = .77). Many faculty participants reported that their immediate impressions were very much or extremely influenced by audio quality (60%), eye contact (57%), and internet connectivity (49%).

Conclusions

Teleconferencing variables may serve as a source of assessment bias during residency interviews. Mock residency applicants received significantly lower ratings when they had off-center eye contact, muffled audio, or choppy internet connectivity, compared to optimal teleconferencing conditions.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-022-00053-w.

Medical school grades may predict future clinical competence

BACKGROUND

In real-world medical education, there is a lack of reliable predictors of future clinical competencies. Hence, we aim to identify the factors associated with clinical competencies and construct a prediction model to identify "improvement required" trainees.

METHODS

We analyzed data from medical students who graduated from National Yang-Ming University with clerkship training and participated in the postgraduate year (PGY) interview at Taipei Veterans General Hospital. Clinical competencies were evaluated using grades of national objective structured clinical examination (OSCEs). This study used data from medical students who graduated in July 2018 as the derivation cohort (N = 50) and those who graduated in July 2020 (n = 56) for validation.

RESULTS

Medical school grades were associated with the performance of national OSCEs (Pearson r = 0.34, p = 0.017), but the grades of the structured PGY interviews were marginally associated with the national OSCE (Pearson r = 0.268, p = 0.06). A prediction model was constructed to identify "improvement required" trainees, defined: trainees with the lowest 25% of scores in the national OSCEs. According to this model, trainees with the lowest 25% medical school grades predicted a higher risk of the "improvement required" clinical performance (Q1-Q3 vs Q4 = 15% vs 60%, odds ratio = 8.5 [95% confidence interval = 1.8-39.4], p = 0.029). In the validation cohort, our prediction model could accurately classify 76.7% "improvement required" and "nonimprovement required" students.

CONCLUSION

Our study suggests that interventions for students with unsatisfactory medical school grades are warranted to improve their clinical competencies.

Perceptions of a virtual interview process for pharmacy residents during the COVID-19 pandemic: A multisite survey of residency candidates, preceptors, and residency program directors

PURPOSE

To describe the perceptions of residency candidates, residency practitioners (current residents and preceptors), and residency program directors (RPDs) regarding a virtual interview process for pharmacy residency programs across multiple institutions.

METHODS

In May 2021, an anonymous web-based questionnaire characterizing perceptions of the virtual interview process used during the coronavirus disease 2019 (COVID-19) pandemic was distributed to residency candidates, residency practitioners, and RPDs across 13 institutions. Quantitative responses measured on a 5-point Likert scale were summarized with descriptive statistics, and open-ended questions were analyzed using thematic qualitative methods.

RESULTS

236 residency candidates and 253 residency practitioners/RPDs completed the questionnaire, yielding response rates of 27.8% (236 of 848), and 38.1% (253 of 663), respectively. Overall, both groups perceived the virtual interview format positively. When asked whether virtual interviews should replace in-person interviews moving forward, 60.0% (18 of 30) of RPDs indicated they agreed or strongly agreed, whereas only 30.5% (61 of 200) of current preceptors/residents and 28.7% (66 of 230) of residency candidates agreed or strongly agreed. Thematic analysis of qualitative responses revealed that while virtual interviews were easier logistically, the lack of in-person interactions was a common concern for many stakeholders. Lastly, the majority (65.0%) of residency candidates reported greater than $1,000 in savings with virtual interviews.

CONCLUSION

Virtual interviews offered logistical and financial benefits. The majority of RPDs were in favor of offering virtual interviews to replace in-person interviews, whereas the majority of residency candidates and practitioners preferred on-site interviews. As restrictions persist with the ongoing pandemic, our results provide insight into best practices for virtual pharmacy residency interviews.

Comparison of student and faculty interviewers using ratings data for admissions decisions

INTRODUCTION

The aim of this study was to evaluate the correlations of cognitive and noncognitive dental school admission factors with interview ratings provided by faculty and student interviewers.

METHODS

Interviewees for the 2019-2021 admissions cycles underwent a 90-minute period consisting of two 45-min interviews and received two interview scores ranging from 1.0 to 2.3. Regression models were utilised to evaluate factors associated with interviewer type and admission status, The correlation strength between faculty and student interview scores was assessed.

RESULTS

A total of 177 applicants were interviewed, and 69 students were admitted over two admission cycles. Admitted students received higher interview ratings compared with nonadmitted applicants after adjusting for cognitive and noncognitive admission factors (average difference = -0.068; 95% CI = -0.123, -0.014). No statistically significant relationship was found between any cognitive admissions factor and interview score. However, having prior leadership role experiences was associated with better faculty score, after adjusting for student interviewee score. There was a strong linear correlation (r = .92) between faculty and student interview scores, with 0.809 change in faculty score with each additional student interview score (95% CI = 0.735, 0.883).

CONCLUSION

The relationship between faculty and student scores was linear and strongly correlated, suggesting that faculty and student interviewers were comparable in their interview scoring. Leadership experience and potential could be advantageous qualities that improve faculty interviewer ratings. However, having a mix of faculty and Student interviewers may promote evaluation of candidates from different aspects, as students are familiar with the learning environment in dental school and may provide a unique perspective on an applicant's background and suitability for the program.

Perceived effectiveness of video interviews for orthopaedic surgery residency during COVID-19

BACKGROUND

During the 2020-21 residency interview season, interviews were conducted through virtual platforms due to the COVID-19 pandemic. The purpose of this study is to assess the general perceptions of applicants, residents and attendings at a single, large, metropolitan orthopaedic residency with regards to the video interview process before and after the interview season.

METHODS

Surveys were sent to all orthopaedic applicants, residents, and attendings before the interview season. Applicants who received interviews and responded to the first survey (46) and faculty who responded to the first survey (28) were sent a second survey after interviews to assess how their perceptions of video interviews changed.

RESULTS

Initially, 50% of applicants (360/722) and 50% of faculty and residents (28/56) responded before interview season. After interviews, 55% of interviewees (25/46) and 64% of faculty and residents (18/28) responded. Before interviews, 91% of applicants stated they would prefer in-person interviews and 71% were worried that video interviews would prevent them from finding the best program fit. Before interviews, 100% of faculty and residents stated they would rather conduct in-person interviews and 86% felt that residencies would be less likely to find applicants who best fit the program. Comparing responses before and after interviews, 16% fewer applicants (p = 0.01) perceived that in-person interviews provide a better sense of a residency program and faculty and residents' perceived ability to build rapport with interviewees improved in 11% of respondents (p = 0.01). However, in-person interviews were still heavily favored by interviewees (84%) and faculty and residents (88%) after the interview season.

CONCLUSIONS

In-person interviews for Orthopaedic Surgery Residency are perceived as superior and are preferred among the overwhelming majority of applicants, residents, and interviewers. Nevertheless, perceptions toward video interviews improved in certain domains after interview season, identifying potential areas of improvement and alternative interview options for future applicants.

A Retrospective Analysis of Medical Student Performance Evaluations, 2014-2020: Recommend with Reservations

BACKGROUND

The Medical Student Performance Evaluations (MSPE) is a cornerstone of residency applications. Little is known regarding adherence to Association of American Medical Colleges (AAMC) MSPE recommendations and longitudinal changes in MSPE content.

OBJECTIVES

Evaluate current MSPE quality and longitudinal changes in MSPE and grading practices.

DESIGN

Retrospective analysis.

PARTICIPANTS

Students from all Liaison Committee on Medical Education (LCME)-accredited medical schools from which the Stanford University Internal Medicine residency program received applications between 2014-2015 and 2019-2020.

MAIN MEASURES

Inclusion of key words to describe applicant performance and metrics thereof, including distribution among students and key word assignment explanation; inclusion of clerkship grades, grade distributions, and grade composition; and evidence of grade inflation over time.

KEY RESULTS

MSPE comprehensiveness varied substantially among the 149 schools analyzed. In total, 25% of schools provided complete information consistent with AAMC recommendations regarding key word/categorization of medical students and clerkship grades in 2019-2020. Seventy-seven distinct key word terms appeared across the 139 schools examined in 2019-2020. Grading practices markedly varied, with 2-83% of students receiving the top internal medicine clerkship grade depending on the year and school. Individual schools frequently changed key word and grading practices, with 33% and 18% of schools starting and/or stopping use of key words and grades, respectively. Significant grade inflation occurred over the 6-year study period, with an average 14% relative increase in the proportion of students receiving top clerkship grades.

CONCLUSIONS

A minority of schools complies with AAMC MSPE guidelines, and MSPEs are inconsistent across time and schools. These practices may impair evaluation of students within and between schools.

Exploring Potential Schedule-Related and Gender Biases in Ophthalmology Residency Interview Scores

Purpose Prior studies have revealed grading discrepancies in evaluation of personal statements and letters of recommendation based on candidate's race and gender. Fatigue and the end-of-day phenomenon can negatively impact task performance but have not been studied in the residency selection process. Our primary objective is to determine whether factors related to interview time and day as well as candidate's and interviewer's gender have a significant effect on residency interview scores.

Methods Seven years of ophthalmology residency candidate evaluation scores from 2013 to 2019 were collected at a single academic institution, standardized by interviewer into relative percentiles (0–100 point grading scale), and grouped into the following categories for comparisons: different interview days (Day 1 vs. Day 2), morning versus afternoon (AM vs. PM), interview session (Day 1 AM/PM vs. Day 2 AM/PM), before and after breaks (morning break, lunch break, and afternoon break), residency candidate's gender, and interviewer's gender.

Results Candidates in the morning sessions were found to have higher scores than afternoon sessions (52.75 vs. 49.28, p < 0.001). Interview scores in the early morning, late morning, and early afternoon were higher than late afternoon scores (54.47, 53.01, 52.15 vs. 46.74, p < 0.001). Across all interview years, there were no differences in scores received before and after morning breaks (51.71 vs. 52.83, p = 0.49), lunch breaks (53.01 vs. 52.15, p = 0.58), and afternoon breaks (50.35 vs. 48.30, p = 0.21). No differences were found in scores received by female versus male candidates (51.55 vs. 50.49, p = 0.21) or scores given by female versus male interviewers (51.31 vs. 50.84, p = 0.58).

Conclusion Afternoon residency candidate interview scores, especially late afternoon, were significantly lower than morning scores, suggesting the need to further study the effects of interviewer's fatigue in the residency interview process. The interview day, presence of break times, candidate's gender, and interviewer's gender had no significant effects on interview score.

LEGO®-based communication assessment in virtual general surgery residency interviews

Purpose

Effective communication skills are a critical quality and skill that is highly sought after for surgeons which largely impacts patient outcomes. Residency programs design their interview processes to select the best candidates. LEGO®-based activities have been frequently used to enhance communication skills and team building. This study investigates the effectiveness and reliability of a novel LEGO®-based communication assessment in interviews for surgical residencies and the feasibility of implementing it in a virtual setting.

Methods

This study conducted a retrospective analysis of a LEGO®-based communication assessment at the program's 2020/2021 residency interviews. Each applicant was assessed on a different model. The total scores were analyzed for consistency among raters and correlated to faculty interviews. Furthermore, the impact of the assessment structure, scoring criteria, and range of models' difficulties on the total scores were explored.

Results

A total of 54 categorical and 55 preliminary applicants interviewed on 2 days. The assessment on different models and had no impact on applicants' total scores for either categorical and preliminary groups (p = 0.791 and 0.709, respectively). The communication components of the assessment showed high consistency between the raters. The two applicant groups displayed a statistically significant difference (p = 0.004) in the communication evaluation and model accuracy components. Total scores did not correlate with the faculty interviews of standardized questions in either group.

Conclusion

This novel LEGO®-based communication assessment showed high reliability and promising results as a tool to assess communication and problem solving for residency interviews that can be readily implemented in a virtual setting.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-022-00021-4.

Trust Me, I Know Them: Assessing Interpersonal Bias in Surgery Residency Interviews

BACKGROUND

Residency selection integrates objective and subjective data sources. Interviews help assess characteristics like insight and communication but have the potential for bias. Structured multiple mini-interviews may mitigate some elements of bias; however, a halo effect is described in assessments of medical trainees, and degree of familiarity with applicants may remain a source of bias in interviews.

OBJECTIVE

To investigate the extent of interviewer bias that results from pre-interview knowledge of the applicant by comparing file review and interview scores for known versus unknown applicants.

METHODS

File review and interview scores of applicants to the University of Ottawa General Surgery Residency Training Program from 2019 to 2021 were gathered retrospectively. Applicants were categorized as "home" if from the institution, "known" if they completed an elective at the institution, or "unknown." The Kruskal-Wallis H test was used to compare median interview scores between groups and Spearman's rank-order correlation (r_s) to determine the correlation between file review and interview scores.

RESULTS

Over a 3-year period, 169 applicants were interviewed; 62% were unknown, 31% were known, and 6% were home applicants. There was a statistically significant difference (P=.01) between the median interview scores of home, known, and unknown applicants. Comparison of groups demonstrated higher positive correlations between file review and interview scores (r_s=0.15 vs 0.36 vs 0.55 in unknown, known, and home applicants) with increasing applicant familiarity.

CONCLUSIONS

There is an increased positive correlation between file review and interview scores with applicant familiarity. The interview process may carry inherent bias insufficiently mitigated by the current structure.

Impact of virtual vs. in-person interviews among neurosurgery residency applicants

BACKGROUND

The interview is considered a key factor in selecting residents in various medical and surgical specialties. However, the reliability of the interview process in selecting neurosurgery training program applicants remains largely under-investigated.

OBJECTIVE

To investigate the reliability of the interview process for neurosurgery residency applicants and to evaluate the impact of virtual interviews on this process.

METHODS

We analyzed the records of neurosurgery residency applicant interviews at our institution between 2016 and 2021. An average of 20 neurosurgery faculty members (clinical and research) interviewed each applicant and graded them 1 (best) to 4 (worst). Intraclass correlation coefficient (ICC) and Levene's test were used to assess the inter-rater and intra-rater reliability, respectively.

RESULTS

214 neurosurgery residency applicants were interviewed at a single institution between 2016 and 2021. The mean applicant rating each year ranged from 1.77 to 1.92. Inter-rater agreement was relatively poor in each year, (ICC < 0.5, P < 0.05). Among 60% of the raters, variability of scores significantly changed from year to year, (p < 0.05). When comparing the scores submitted during the virtual interview process (2021) with the scores submitted in the previous years (2016-2020), 2 interviewers (10%) had less variability using the virtual process.

CONCLUSION

Our analysis found that the current interview process for neurosurgery residency applicants' selection suffers from poor inter- and intra-rater reliability. Virtual interviews may be part of a cost-effective strategy to improve the reliability of the interview process. Further validation is needed, as well as identification of novel strategies to maximize the reliability of the selection process.

Holistic Review, Mitigating Bias, and Other Strategies in Residency Recruitment for Diversity, Equity, and Inclusion: An Evidence-based Guide to Best Practices from the Council of Residency Directors in Emergency Medicine

Advancement of diversity, equity, and inclusion (DEI) in emergency medicine can only occur with intentional recruitment of residency applicants underrepresented in medicine (UIM). Shared experiences from undergraduate and graduate medical education highlight considerations and practices that can contribute to improved diversity in the resident pool, such as holistic review and mitigating bias in the recruitment process. This review, written by members of the Council of Residency Directors in Emergency Medicine (CORD) Best Practices Subcommittee, offers best practice recommendations for the recruitment of UIM applicants. Recommendations address pre-interview readiness, interview approach, and post-interview strategies that residency leadership may use to implement holistic review and mitigate bias for recruitment of a diverse class.

Analyzing the effect of interview time and day on emergency medicine residency interview scores

BACKGROUND

When it comes to scheduling interviews, medical students may wonder if they need a strategy to increase their likelihood of matching. Previous studies examined the temporal effects of the residency interview on overall match rate; however, there are additional factors that affect the match process, including board examination scores and letters of recommendation. Only few studies have examined the effect interview time of day has on match success. The current study examines the impact date and time of interview during the interview season have on candidates' respective interview scores.

METHODS

Interview data over a three-year period (i.e., three interview cycles) was examined at a PGY-1-3, ACGME-accredited EM residency program in Philadelphia. Date of interview and time of day of interview (i.e., morning versus afternoon) was examined. A linear regression analysis was performed to determine if there is a statistically-significant difference in overall interview scores based on date during the interview season and time of day.

RESULTS

There is no statistically-significant effect of time of day or date on residency interview scores.

CONCLUSIONS

Our findings are congruent with other studies on the temporal effects of residency interviews on overall match rate. Findings should provide reassurance to students scheduling interviews, as time slots have not been found to have a significant relationship with overall interview score. Future studies should more holistically analyze the residency application process.

Can Behavior-Based Interviews Reduce Bias in Fellowship Applicant Assessment?

PURPOSE

Components of trainee applications may introduce bias based on race or gender. Behavior-based interviews (BBIs) rely on structured questions to elicit applicants' past experiences to predict future behavior. Our objective was to implement BBIs in one fellowship program and compare applicant assessment by race and gender when using a standardized assessment tool versus a BBI-based tool.

METHODS

In 2019 and 2020, we developed BBIs and BBI-specific assessments; 6 of 15 faculty were trained in this interview method. Applicants completed 6 interviews with either a BBI or unstructured format. All faculty completed a standardized assessment on applicants. BBI faculty also completed a BBI-specific assessment. Normalized average scores were calculated and used to rank applicants into quartiles. Race was categorized into White, underrepresented minorities (URMs; Black and Hispanics), and non-URMs (all others). Faculty and applicants were surveyed about BBIs.

RESULTS

Seventy-five applicants were interviewed. Significant differences were found in standardized assessment scores (White 1.01 +/- 0.09, non-URM 1.02 +/- 0.08, URM .94 +/- 0.07; P = .02) and quartiles by race (P = .05), but not for BBI scores (White 0.98 +/- 0.09, non-URM 1.03 +/- 0.09, URM 1.02 +/- 0.1; P = .18) or quartiles by race (P = .17). There were no significant differences in score or quartile by gender for either tool. The majority of faculty and applicant survey respondents commented positively about BBIs.

CONCLUSION

BBIs were successfully implemented and generally reviewed positively by faculty and applicants. BBIs reduced racial differences in applicant assessments. Applicant assessment may benefit from structured tools to mitigate potential biases.

The Impact of Behavioral Anchors in the Assessment of Fellowship Applicants: Reducing Rater Biases

INTRODUCTION

No standardized evaluation tool for fellowship applicant assessment exists. Assessment tools are subject to biases and scoring tendencies which can skew scores and impact rankings. We aimed to develop and evaluate an objective assessment tool for fellowship applicants.

METHODS

We detected rater effects in our numerically scaled assessment tool (NST), which consisted of 10 domains rated from 0 to 9. We evaluated each domain, consolidated redundant categories, and removed subjective categories. For 7 remaining domains, we described each quality and developed a question with a behaviorally-anchored rating scale (BARS). Applicants were rated by 6 attendings. Ratings from the NST in 2018 were compared with the BARS from 2020 for distribution of data, skewness, and inter-rater reliability.

RESULTS

Thirty-four applicants were evaluated with the NST and 38 with the BARS. Demographics were similar between groups. The median score on the NST was 8 out of 9; scores <5 were used in less than 1% of all evaluations. Distribution of data was improved in the BARS tool. In the NST, scores from 6 of 10 domains demonstrated moderate skewness and 3 high skewness. Three of the 7 domains in the BARS showed moderate skewness and none had high skewness. Two of 10 domains in the NST vs 5 of 7 domains in the BARS achieved good inter-rater reliability.

CONCLUSION

Replacing a standard numeric scale with a BARS normalized the distribution of data, reduced skewness, and enhanced inter-rater reliability in our evaluation tool. This provides some validity evidence for improved applicant assessment and ranking.

Anesthesiology Residency Recruitment: A Prospective Study Comparing In-Person and Virtual Interviews

BACKGROUND

Residency recruitment requires significant resources for both applicants and residency programs. Virtual interviews offer a way to reduce the time and costs required during the residency interview process. This prospective study investigated how virtual interviews affected scoring of anesthesiology residency applicants and whether this effect differed from in-person interview historical controls.

METHODS

Between November 2020 and January 2021, recruitment members at the University of Chicago scored applicants before their interview based upon written application materials alone (preinterview score). Applicants received a second score after their virtual interview (postinterview score). Recruitment members were queried regarding the most important factor affecting the preinterview score as well as the effect of certain specified applicant interview characteristics on the postinterview score. Previously published historical controls were used for comparison to in-person recruitment the year prior from the same institution.

RESULTS

Eight hundred and sixteen virtual interviews involving 272 applicants and 19 faculty members were conducted. The postinterview score was higher than the preinterview score (4.06 versus 3.98, P value of <.0001). The change in scores after virtual interviews did not differ from that after in-person interviews conducted the previous year (P = .378). The effect of each characteristic on score change due to the interview did not differ between in-person and virtual interviews (all P values >.05). The factor identified by faculty as the most important in the preinterview score was academic achievements (64%), and faculty identified the most important interview characteristic to be personality (72%).

CONCLUSIONS

Virtual interviews led to a significant change in scoring of residency applicants, and the magnitude of this change was similar compared with in-person interviews. Further studies should elaborate on the effect of virtual recruitment on residency programs and applicants.

The Virtual Recruitment Onion: Peeling Back the Layers of the Interview Season During the COVID-era

OBJECTIVE

We sought to characterize General Surgery residency program directors' (PDs) baseline perspective on how the COVID-19 mandated changes to the recruitment and interview processes impacted how the PDs evaluated and recruited the applicants.

DESIGN

An anonymous cross-sectional questionnaire survey.

SETTING

A large, mid-western academic general surgery residency program.

PARTICIPANTS

47 PDs of Accreditation Council for Graduate Medical Education (ACGME) accredited General Surgery residency programs.

RESULTS

During the virtual-only interviews during the COVID-19 pandemic-era 2020-21 General Surgery residency application cycle, PDs shifted their focus to virtual outreach efforts and bolstered social media presences to recruit strong applicants. Also, our study found statistically significant changes to the increased value of letters of recommendation (LORs) for the PDs when assessing an applicant's commitment to surgery. These findings suggest that the necessity of adapting to the virtual-only interview format significantly altered how the PDs recruited and evaluated applicants for the General Surgery residency match.

CONCLUSIONS

A complete replacement of the in-person interviews with virtual-only interviews may be challenging unless buy-in exists from key stakeholders in the surgical community. Our study highlights the PDs' hesitation in assessing candidates' commitment to surgery from virtual interviews alone. Incorporating virtual interviews as a part of the screening process for applicants may serve as an avenue to maximize the benefits of the virtual interview format. Furthermore, COVID-19 pandemic has normalized the growing social media presence of residency programs, adding to the changing landscape of recruiting and interviewing applicants for General Surgery residency match.

Faculty Members’ Perspective on Virtual Interviews for Medical Residency Matching during the COVID-19 Crisis: A National Survey

During the COVID-19 pandemic, conducting face-to-face medical residency interviews was challenging due to infection prevention precautions, social distancing, and travel restrictions. Virtual interviews were implemented by the Saudi Commission for Health Specialties (SCFHS) as an alternative process for residency matching while striving to maintain the same quality standards. This national survey was conducted to assess the satisfaction and perceptions of faculty members’ virtual interview performance in the assessment for the medical training residency programs. Among the participating 173 faculty members, 34.1% did not have previous experience with video-conferencing. The Zoom application was the most commonly used platform (65.9%). Most (89.6%) of the faculty perceived virtual interviews as “adequate” platforms on which the candidates could express themselves, while almost half of the faculty (53.8%) agreed that virtual interviews allowed them to accurately reach an impression about the candidates. Overall, 73.4% of faculty felt comfortable ranking the virtually interviewed candidates. We conclude that the acceptance of participating faculty members in the first Saudi medical residency training matching cycle virtual interviewing event was well-perceived. This study provides evidence for future application and research of virtual interviews in residency candidates’ assessment, especially after the pandemic crisis resolves.

Otolaryngology residency selection: are we doing it right?

PURPOSE OF REVIEW

This article will provide an overview of recent disruptions to the otolaryngology residency match process and conclude with questions and resources that can guide future selection system design.

RECENT FINDINGS

During the implementation of the single accreditation system, the loss of osteopathic programs, reduction of osteopathic leadership positions, and lack of interest in Osteopathic Recognition represent serious threats to the profession; this has implications for the distribution of the otolaryngology workforce, plausibly decreasing healthcare access in less-populated communities. Next, the impacts of COVID-19 reverberated throughout the application process, including the reduction/elimination of away rotations, modification of application requirements, conversion to virtual interviews, and initiation of preference signaling. Soon, the transition to pass/fail scoring for the United States Medical Licensing Exam Step 1 could stimulate a paradigm shift, with a heightened emphasis on holistic review.

SUMMARY

The last two match cycles have been the most dynamic and unpredictable in decades. Out of the commotion, the otolaryngology community has an opportunity for a fresh start, combining insights from past literature with recent articles compiled for this review. Moving forward, it will be advantageous to approach residency selection as a well-executed quality improvement project, requiring continuous assessment and adjustment.

The LEGO™ Exercise: An Assessment of Core Competencies in Urology Residency Interviews

BACKGROUND

In competitive residency specialties such as Urology, it has become increasingly challenging to differentiate similarly qualified applicants. Residency interviews are utilized to rank applicants, yet they are often biased and do not explicitly address ACGME core competencies.

OBJECTIVE

We hypothesized a team-based exercise in the urology residency interview centered on building LEGOs assesses core competences.

DESIGN

From 2014-2017, students interviewing for urology residency at two institutions participated in a LEGO™ building activity. Applicants were assigned to "architect"- describing how to construct a structure - or "builder" - constructing the same structure with pieces-using only verbal cues to assemble the structure. Participants were graded using a rubric assessing competencies of interpersonal communication, problem-based learning, professionalism, and manual dexterity (indicator of procedural skill). The total minimum score was 16 and maximum was 80.

SETTING

The study took place at two tertiary referral centers: University of Michigan Medical School in Ann Arbor, MI, and University of Utah School of Medicine in Salt Lake City, UT.

PARTICIPANTS

A total of 176 applicants participated, comprised of applicants interviewing for urology residency at two institutions during the study timeframe.

RESULTS

For architects and builders, there was a maximum score of 80, and minimum of 34 and 32, respectively. Both distributions show a right shift with mean scores of 64.3 and 65.9, and median scores of 69 and 65.5. Successful pairs excelled with consistent nomenclature and clear directionality. Ineffective pairs miscommunicated with false affirmations, inconsistent nomenclature, and lack of patience.

CONCLUSIONS

The LEGO™ exercise allowed for standardized assessment of applicants based on ACGME core competencies. The rubric identified poor performers who do not rise to the challenge of a team-based task.

A carbon footprint study of the Canadian medical residency interview tour

BACKGROUND

Each spring, thousands of Canadian medical students travel across the country to interview for residency positions, a process known as the CaRMS tour. Despite the large scale of travel, the CaRMS tour has received little environmental scrutiny.

PURPOSE

To estimate the national carbon footprint of flights associated with the CaRMS tour, as well as reductions in emissions achievable by transitioning to alternative models.

METHODS

We developed a three-question online commuter survey to collect the unique travel itineraries of applicants in the 2020 CaRMS tour. We calculated the emissions associated with all flights and modelled expected emissions for two alternative in-person interview models, and two virtual interview models.

RESULTS

We collected 960 responses out of 2943 applicants across all 17 Canadian medical schools. We calculated the carbon footprint of flights for the 2020 CaRMS as 4239 tCO₂e (tonnes of carbon dioxide equivalents), averaging 1.44 tCO₂e per applicant. The average applicant's tour emissions represent 35.1% of the average Canadian's annual household carbon footprint, and the emissions of 26.7% of respondents exceeded their entire annual '2050 carbon budget.' Centralized in-person interviews could reduce emissions by 13.7% to 74.7%, and virtual interviews by at least 98.4% to 99.9%.

CONCLUSIONS

Mandatory in-person residency interviews in Canada contribute significant emissions and reflect a culture of emissions-intensive practices. Considerable decarbonization of the CaRMS tour is possible, and transitioning to virtual interviews could eliminate the footprint almost entirely.