Reimagining Residency Selection: Part 2-A Practical Guide to Interviewing in the Post-COVID-19 Era

Evaluation and Assessment of the ABATE Framework to Enhance Implicit Bias Training for Virtual Interviews in Medical Schools

Introduction

The influence of implicit biases in virtual interviews must be addressed to ensure equity within the admissions process. ABATE is a mnemonic framework of five specific categories of implicit bias (affinity-based, backdrop-based, appearance-based, technology and media-based, and enunciation-based biases) that should be anticipated and mitigated for faculty, staff, health professionals, and medical students who conduct virtual interviews at medical schools.

Methods

A 60-minute workshop was developed to educate medical school admissions interviewers about the ABATE model and strategies to mitigate implicit bias during virtual interviews. Four workshops were held over 1 year totaling 217 individual attendees. The workshops were evaluated using a single-group, pre-post questionnaire designed with the Kirkpatrick evaluation model.

Results

Attendees reported that they found the ABATE workshop useful and relevant to improving their ability to minimize implicit bias during virtual interviews. Significant improvements were found in attendee reactions to the utility of implicit bias training (M pre = 2.6, M post = 3.1, p = .002). Significant changes were also reported in attendees' attitudes about interviewing confidence (M pre = 3.0, M post = 3.2, p = .04), bias awareness (M pre = 3.0, M post = 3.4, p = .002), and identifying and applying bias mitigation solutions (M pre = 2.5, M post = 3.0, p = .003). Knowledge specific to backdrop-based biases also significantly increased (M pre = 3.2, M post = 3.4, p = .04).

Discussion

The ABATE workshop demonstrates promise in mitigating implicit bias in virtual medical school interviews.

Intern Experience Reflects Information Gathered During the Virtual Interview Process

OBJECTIVE

The onset of the coronavirus 2019 (COVID-19) pandemic brought many changes to the residency application process including transitioning to a virtual interview platform, which continues today. The transition brought many concerns from general surgery applicants about their ability to obtain adequate information about a program virtually. We sought to characterize how information presented by programs during the first ever virtual interview cycle matched the experience of general surgery interns after training at a program for 1 year.

DESIGN, SETTING, AND PARTICIPANTS

In May of 2022, a survey was distributed to 243 program directors who were asked to forward it to their general surgery categorical interns who matched during the 2021 virtual match cycle. Demographics, resources used to determine an impression of a program, and correlations between information presented virtually and what was subsequently experienced as an intern were collected.

RESULTS

Forty-six program directors confirmed forwarding the survey to their categorical interns. A total of 102 general surgery interns completed the survey. Most interns (88.2%) agreed that their experience matched expectations based on information received through the virtual interview process and 98% of interns were satisfied with their experience at their training program. Interviews with faculty (40.0%), residents (68.0%) and the program web site (29.0%) were the top 3 resources used to create the most accurate impression of a program. Interns felt they were well informed during the virtual interview experience about support from fellow residents (84.3%), culture (73.0%), surgical volume (72.5%), and intern operative experience (71.6%). In addition, 65.7% of participants thought they were able to obtain a good understanding of the program's culture from the virtual process. However, 16.7% thought that their program unintentionally misrepresented aspects of the training program.

CONCLUSIONS

The faculty and residency interviews were the most important factors in program ranking and most participants agreed that their virtual interview experience matched their expectations during their intern year. Most interns felt they were able to obtain a good understanding of the program's culture from the virtual process. In addition, a majority of interns felt well informed during the interview on aspects ranging from surgical volume, autonomy, and work hours to support from faculty and residents. If virtual interviews are to continue, residents can be satisfied that information gathered virtually will match the reality of their training. Programs should continue to make every effort to present their program realistically.

Video Interviews and Surgical Applicants' Ability to Assess Fit to Residency Programs

INTRODUCTION

Due to the COVID-19 pandemic, the recruitment cycle for the 2021 Match was performed virtually. This Association for Surgical Education (ASE)-sponsored survey set out to study applicants' ability to assess the factors contributing to fit through video interviews.

METHODS

An IRB-approved, online, anonymous survey was distributed to surgical applicants at a single academic institution and through the ASE clerkship director distribution list between the rank order list certification deadline and Match Day. Applicants used 5-point Likert-type scales to rate factors for importance to fit and their ease of assessment through video interviewing. A variety of recruitment activities were also rated by applicants for their perceived helpfulness in assessment of fit.

RESULTS

One hundred and eighty-three applicants responded to the survey. The three most important factors for applicant fit were how much the program cared, how satisfied residents seem with their program, and how well residents get along. Resident rapport, diversity of the patient population, and quality of the facilities were hardest to assess through video interviews. In general, diversity-related factors were more important to female and non-White applicants, but not more difficult to assess. Interview day and resident-only virtual panels were the most helpful recruitment activities, while virtual campus tours, faculty-only panels, and a program's social media were the least helpful.

CONCLUSIONS

This study provides valuable insight into the limitations of virtual recruitment for surgical applicants' perception of fit. These findings and the recommendations herein should be taken into consideration by residency program leadership to ensure successful recruitment of diverse residency classes.

Using Kotter's Change Management Framework to Redesign Departmental GME Recruitment

BACKGROUND

In light of the COVID-19 pandemic, dramatic change in the graduate medical education (GME) trainee recruitment process was required. Kotter's 8-Step Change Model is a change management framework that has been successfully applied to a variety of GME initiatives but not for recruitment redesign.

OBJECTIVE

To implement major change in program recruitment during the COVID-19 pandemic while maintaining Match outcomes and a high-quality candidate experience.

METHODS

In 2020, we applied Kotter's 8 steps to implement major changes to program recruitment for a department of internal medicine including 15 GME programs (1 internal medicine residency and 14 subspecialty fellowships). We collected each program's Match fill rates and used Google Analytics to collect monthly website traffic for the year prior to our change process and the subsequent 2 years. Standardized post-interview survey questions were created, and these results were reviewed for descriptive analysis.

RESULTS

We successfully used Kotter's 8 steps to change recruitment to a virtual format. Program fill rates remained high after implementation. Website engagement improved with peak monthly page rates doubling over previous values. During the highest traffic month, the average time on site increased for 7 programs, while the bounce rate decreased by more than half for 10 programs. Candidate descriptive feedback was positive.

CONCLUSIONS

The application of Kotter's 8 steps guided major changes to GME recruitment for 15 programs and was associated with maintained Match fill rates and increased website engagement.

The effect of assessor professional background on interview evaluation during residency selection: A mixed-methods study

BACKGROUND

Residency interviewer scores are greatly variable and seems to be influenced by personal characteristics of assessors, although factors contributing to variability remain unclear. The study sought to determine how different professional backgrounds influence assessors' scores.

METHODS

Fifty-five general surgery applicants rotated through an interview station assessing teamwork. They were scored by surgeons, human-resource managers, pilots, athletes. Pearson's correlation and a repeated-measures ANOVA were used to determine correlations between professions. Structured interviews were used to probe for scoring rationale.

RESULTS

Interview scores differed significantly between professions (F (3, 159) = 11.12, p < 0.001. Qualitative analysis revealed that due to the challenge of distinguishing between similarly performing candidates, assessors rely on global impressions informed by personal values.

CONCLUSION

Assessor variability is ubiquitous, in part due to the subjective nature of interviews and is associated with personal values. When selecting assessors, programs should choose diverse assessors to assess to ensure a reliable selection process.

Virtual Interviews: Assessing How Expectations Meet Reality

OBJECTIVE

This study aimed to determine the post-matriculation perceptions of interns and faculty who participated in the 2020-2021 virtual interview process and how their expectations of the program and the applicants, respectively, aligned with reality.

DESIGN, SETTING, AND PARTICIPANTS

Published surveys on virtual interviewing were reviewed and modified to design two surveys, for interns and for faculty who interviewed. Interns and faculty members from the Departments of Surgery and Medicine at one institution who participated in the 2020-2021 virtual interview process completed the surveys four to six months after the start of the academic year. Following survey completion, surgical interns from the same application cycle participated in one in-person focus group nine months after the start of the academic year to clarify points raised in the surveys.

RESULTS

Forty-six interns and faculty members responded to the survey (subgroup response rates ranging from 13-30%) and ten interns participated in the focus group (participation rate 100%). Most faculty and intern participants found that expectations formed during virtual interviews were accurate. However, our respondents noted limitations to virtual interviews that reduced their usefulness, including challenges with unnatural social interactions, understanding city fit, and getting a sense of resident and program culture. Participants provided possible solutions to address these challenges.

CONCLUSIONS

Overall, this mixed-methods study at a single institution found that perceptions from virtual interviews were generally accurate but with some limitations. We describe several opportunities to improve the virtual interview process and optimize the application experience.

Virtual Interviewing for Radiology Residency: Pluses, Minuses, and A Review of the Literature

This article reviews current medical literature to assess the benefits and drawbacks of virtual interviews for radiology residencies as well as the downstream effects of these changes, best practices, and potential future recruitment methods. Topics covered include the effects of remote recruitment in promoting accessibility and applicant diversity and equality as well as fiscal, environmental, and time savings in combination with technical challenges, the complications of over application, challenges in assessment of program culture and location, impact on morale, and hidden financial and emotional costs. Learnings from other medical specialties are highlighted in addition to the process of signaling, guidelines for conducting and participating in virtual interviews, and matters for future consideration.

Virtual interviewing for graduate medical education recruitment and selection: A BEME systematic review: BEME Guide No. 80

BACKGROUND

The COVID-19 pandemic caused graduate medical education (GME) programs to pivot to virtual interviews (VIs) for recruitment and selection. This systematic review synthesizes the rapidly expanding evidence base on VIs, providing insights into preferred formats, strengths, and weaknesses.

METHODS

PubMed/MEDLINE, Scopus, ERIC, PsycINFO, MedEdPublish, and Google Scholar were searched from 1 January 2012 to 21 February 2022. Two authors independently screened titles, abstracts, full texts, performed data extraction, and assessed risk of bias using the Medical Education Research Quality Instrument. Findings were reported according to Best Evidence in Medical Education guidance.

RESULTS

One hundred ten studies were included. The majority (97%) were from North America. Fourteen were conducted before COVID-19 and 96 during the pandemic. Studies involved both medical students applying to residencies (61%) and residents applying to fellowships (39%). Surgical specialties were more represented than other specialties. Applicants preferred VI days that lasted 4-6 h, with three to five individual interviews (15-20 min each), with virtual tours and opportunities to connect with current faculty and trainees. Satisfaction with VIs was high, though both applicants and programs found VIs inferior to in-person interviews for assessing 'fit.' Confidence in ranking applicants and programs was decreased. Stakeholders universally noted significant cost and time savings with VIs, as well as equity gains and reduced carbon footprint due to eliminating travel.

CONCLUSIONS

The use of VIs for GME recruitment and selection has accelerated rapidly. The findings of this review offer early insights that can guide future practice, policy, and research.

The Impact of Virtual Interviewing During the COVID-19 Pandemic on the Residency Application Process: One Institution's Experience

PURPOSE

To assess the impact of virtual interviewing during the COVID-19 pandemic on the residency application process and to compare applicant costs and time spent interviewing during the 2020-2021 application cycle with prior years.

METHOD

Fourth-year medical students at the University of Kansas School of Medicine applying for first-year residency positions via the National Resident Matching Program Match completed an electronic 46-item survey after submitting their rank lists during each application cycle from 2015-2016 to 2020-2021. The authors used descriptive statistics and t tests to analyze and compare responses to demographics questions and questions regarding number of submitted applications, offered and completed interviews, ranked programs, costs, and time spent interviewing. They used thematic analysis to code respondents' narrative comments about the virtual interviewing experience.

RESULTS

From 2015-2016 to 2020-2021, 994 (of 1,190; 83.5%) respondents completed the survey. From 2019-2020 to 2020-2021, the average total cost of applying to residency per applicant dropped by $3,566 (P &lt; .001) and the average time spent interviewing dropped by 13.3 days (P &lt; .001). At the same time, the average number of applications per applicant dropped by 3.4, and applicants completed the same number of interviews and ranked 2.3 fewer programs, none of which were statistically significant differences. Narrative comments from 113 (79%) respondents in 2020-2021 revealed 4 themes related to virtual interviewing: convenience of time and cost, positive aspects of the process, negative aspects of the process, and overall impressions of the program.

CONCLUSIONS

Virtual interviewing during the 2020-2021 application cycle resulted in an approximately 80% reduction in cost for applicants and an approximately 50% decrease in time spent interviewing compared with previous years but was not associated with large increases in number of submitted applications, completed interviews, or ranked programs. Applicants generally perceived virtual interviewing as positive although they raised notable concerns.

Trust Me, I Know Them: Assessing Interpersonal Bias in Surgery Residency Interviews

BACKGROUND

Residency selection integrates objective and subjective data sources. Interviews help assess characteristics like insight and communication but have the potential for bias. Structured multiple mini-interviews may mitigate some elements of bias; however, a halo effect is described in assessments of medical trainees, and degree of familiarity with applicants may remain a source of bias in interviews.

OBJECTIVE

To investigate the extent of interviewer bias that results from pre-interview knowledge of the applicant by comparing file review and interview scores for known versus unknown applicants.

METHODS

File review and interview scores of applicants to the University of Ottawa General Surgery Residency Training Program from 2019 to 2021 were gathered retrospectively. Applicants were categorized as "home" if from the institution, "known" if they completed an elective at the institution, or "unknown." The Kruskal-Wallis H test was used to compare median interview scores between groups and Spearman's rank-order correlation (r_s) to determine the correlation between file review and interview scores.

RESULTS

Over a 3-year period, 169 applicants were interviewed; 62% were unknown, 31% were known, and 6% were home applicants. There was a statistically significant difference (P=.01) between the median interview scores of home, known, and unknown applicants. Comparison of groups demonstrated higher positive correlations between file review and interview scores (r_s=0.15 vs 0.36 vs 0.55 in unknown, known, and home applicants) with increasing applicant familiarity.

CONCLUSIONS

There is an increased positive correlation between file review and interview scores with applicant familiarity. The interview process may carry inherent bias insufficiently mitigated by the current structure.

Applicants' perception of fit to residency programmes in the video-interview era: A large multidisciplinary survey

INTRODUCTION

'Fit' refers to an applicants' perceived compatibility to a residency programme. A variety of structural, identity-related and relational factors contribute to self-assessments of fit. The 2021 residency recruitment cycle in the USA was performed virtually due to the COVID-19 pandemic. Little is known about how video-interviewing may affect residency applicants' ability to gauge fit.

METHODS

A multidisciplinary, anonymous survey was distributed to applicants at a large academic institution between rank order list (ROL) certification deadline and Match Day 2021. Using Likert-type scales, applicants rated factors for importance to 'fit' and their ease of assessment through video-interviewing. Applicants also self-assigned fit scores to the top-ranked programme in their ROL using Likert-type scales with pairs of anchoring statements.

RESULTS

Four hundred seventy-three applicants responded to the survey (25.7% response rate). The three most important factors to applicants for assessment of fit (how much the programme seemed to care, how satisfied residents seem with their programme and how well the residents get along) were also the factors with the greatest discrepancy between importance and ease of assessment through video-interviewing. Diversity-related factors were more important to female applicants compared with males and to non-White applicants compared with White applicants. Furthermore, White male applicants self-assigned higher fit scores compared with other demographic groups.

CONCLUSION

There is a marked discrepancy between the most important factors to applicants for fit and their ability to assess those factors virtually. Minoritised trainees self-assigned lower fit scores to their top-ranked programme, which should raise concern amongst medical educators and highlights the importance of expanding current diversity, equity and inclusion efforts in academic medicine.

Experiences, perspectives, and advice for using virtual interviews in post-graduate trainee selection: a national survey of CCFP (EM) program directors

Objective

Due to the COVID pandemic, restrictions were put in place mandating that all residency interviews be transitioned to a virtual format. Canadian CCFP(EM) programs were among the first to embark on this universal virtual interview process for resident selection. Although there have been several recent publications suggesting best practice guidelines for virtual interviews in trainee selection, pragmatic experiences and opinions from Program Directors (PDs) are lacking. This study aimed to elicit the experiences and perspectives of CCFP(EM) PDs after being amongst the first to conduct universal virtual interviews in Canada.

Methods

A 17-item online survey was created and distributed to all CCFP(EM) PDs (n = 17). It explored the virtual interview format employed, perceived advantages and disadvantages of a virtual configuration, confidence in determining a candidate’s rank order, and PD preference for employing a virtual interview format in the future. It also elicited practical advice to conduct a smooth and successful virtual interview day.

Results

The survey response rate was 76.5% (13/17). Nine respondents (69.2%) agreed that the virtual interview format enabled them to confidently determine a candidate’s rank order. With respect to preference for future use of virtual interviews, 23.1% agreed, 38.5% disagreed and 38.5% neither agreed nor disagreed. Inductive thematic analysis of free text responses revealed themes related to virtual interview advantages (time, financial, and resource costs), disadvantages (difficulty promoting smaller programs, getting a ‘feel’ for candidates and assessing their interpersonal skills), and practical tips to facilitate virtual interview processes.

Conclusion

Once restrictions are lifted, cost-saving advantages must be weighed against suggested disadvantages such as showcasing program strengths and assessing interpersonal skills in choosing between traditional and virtual formats. Should virtual interviews become a routine part of resident selection, the advice suggested in this study may be considered to help optimize a successful virtual interview process.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43678-022-00312-1.

Systems-Level Reforms to the US Resident Selection Process: A Scoping Review

BACKGROUND

Calls to reform the US resident selection process are growing, given increasing competition and inefficiencies of the current system. Though numerous reforms have been proposed, they have not been comprehensively cataloged.

OBJECTIVE

This scoping review was conducted to characterize and categorize literature proposing systems-level reforms to the resident selection process.

METHODS

Following Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches of Embase, MEDLINE, Scopus, and Web of Science databases were performed for references published from January 2005 to February 2020. Articles were included if they proposed reforms that were applicable or generalizable to all applicants, medical schools, or residency programs. An inductive approach to qualitative content analysis was used to generate codes and higher-order categories.

RESULTS

Of 10 407 unique references screened, 116 met our inclusion criteria. Qualitative analysis generated 34 codes that were grouped into 14 categories according to the broad stages of resident selection: application submission, application review, interviews, and the Match. The most commonly proposed reforms were implementation of an application cap (n = 28), creation of a standardized program database (n = 21), utilization of standardized letters of evaluation (n = 20), and pre-interview screening (n = 13).

CONCLUSIONS

This scoping review collated and categorized proposed reforms to the resident selection process, developing a common language and framework to facilitate national conversations and change.

Reimagining Residency Selection: Part 3-A Practical Guide to Ranking Applicants in the Post-COVID-19 Era

Collapse