The Experience of Virtual Interviews in Resident Selection: A Survey of Program Directors in Surgery

Reimagining general surgery resident selection: Collaborative innovation through design thinking

Introduction

The process by which surgery residency programs select applicants is complex, opaque, and susceptible to bias. Despite attempts by program directors and educational researchers to address these issues, residents have limited ability to affect change within the process at present. Here, we present the results of a design thinking brainstorm to improve resident selection and propose this technique as a framework for surgical residents to creatively solve problems and generate actionable changes.

Methods

Members of the Collaboration of Surgical Education Fellows (CoSEF) used the design thinking framework to brainstorm ways to improve the resident selection process. Members participated in one virtual focus group focused on identifying pain points and developing divergent solutions to those pain points. Pain points and solutions were subsequently organized into themes. Finally, members participated in a second virtual focus group to design prototypes to test the proposed solutions.

Results

Sixteen CoSEF members participated in one or both focus groups. Participants identified twelve pain points and 57 potential solutions. Pain points and solutions were grouped into the three themes of transparency, fairness, and applicant experience. Members subsequently developed five prototype ideas that could be rapidly developed and tested to improve resident selection.

Conclusions

The design thinking framework can help surgical residents come up with creative ideas to improve pain points within surgical training. Furthermore, this framework can supplement existing quantitative and qualitative methods within surgical education research. Future work will be needed to implement the prototypes devised during our sessions and turn them into complete interventions.

Key message

In this paper, we demonstrate the results of a resident-led design thinking brainstorm on improving resident selection in which our team identified twelve pain points in resident selection, ideated 57 solutions, and developed five prototypes for further testing. In addition to sharing our results, we believe design thinking can be a useful framework for creative problem solving within surgical education.

Successful Applicant and Program Director Perspectives on the Virtual Residency Selection Process for Canadian Surgical Subspecialties

Introduction: The COVID-19 pandemic imparted an important shift in strategies postgraduate surgical programs use to recruit, interact with, and select medical students applying through the Canadian Resident Matching Service (CaRMS). With this unprecedented shift toward virtual applicant selection, this study sought to explore and analyze perspectives of the first cohort of program directors (PDs) and applicants who participated in this process. Methods: A cross-sectional survey study was designed using Google Forms for both PDs and applicants participating in the 2021 CaRMS surgical subspecialty selection process. Questions pertained to format and content of virtual engagement methods, the interview itself, as well as advice for future applicants. Results: Thirty-five PDs and 40 successful applicants (n = 75) participated in the study. Cost reduction was the most commonly reported benefit of online interviewing by PDs (85%), followed by efficiency (71%), enhanced resource management (49%), and ability to conduct more interviews (23%). Strong letters of reference (80%) and interview performance (74%) remained the most significant factors in virtual applicant selection. Attendance to virtual recruitment events did not increase the likelihood of offering interviews (n = 24, 69% of PDs), although the ability to perform in-person electives held tremendous value. Most applicants (90%) reported on virtual information sessions as the best method for learning about programs; work culture and environment were topics most valued as discussion points (90%). Successful applicants provided an average confidence of 76% regarding their suitability with their matched programs. Seventy-three percent of applicants (n = 29) had either a preference for virtual interviews or were equivocal, while 51.4% of PDs (n = 18) preferred interviews to be conducted virtually for future cohorts. Conclusion: Trainees are entering residency with confidence following a virtual selection process, and PDs feel confident in their selections. Although no clear consensus exists regarding preference for virtual or in-person interviews, several advantages for virtual resident selection exist. The influence of an in-person elective was found difficult to replace, regardless of interview format. The importance of applicant engagement with programs prior to interviews is highlighted and discussed with recommendations provided for best practices.

Influence of Applicant Interview Format Choice on Demographics and Outcomes from the Residency Match

BACKGROUND

Studies of virtual interviews (VI) for residency lack contemporaneous In-person Interview (IPI) comparators, leaving uncertain the impact of interview mode on the match process. The study aims to evaluate the effect of allowing candidates to choose interview format, the VI versus IPI, on demographic characteristics of candidates and on rank order list (ROL) position.

STUDY DESIGN

A Cohort study of residency applicants (2022-2023 recruiting season) to a general surgery training program. 105 applicants were invited for interview, of whom 84 candidates were interviewed. Invited candidates were allowed to choose between the following interview options: 1) In person only, 2) Virtual only, 3) In person, but would accept virtual, 4) Virtual, but would accept in-person, and 5) No preference/either. The main outcomes were the differences in demographics of candidates and relative ROL position based on interview format.

RESULTS

Most candidates preferred VI (63%), while 26% preferred IPI and 11% had no preference. 43 VI and 41 IPI were conducted. VI candidates were more likely female (62.8% vs. 31.7%, p = 0.004), attended more distant medical schools (609 [207.5, 831] miles vs. 161 [51, 228] miles, p < 0.001), had higher USMLE scores, and better letters of recommendation. IPI candidates were more likely to have signaled interest (19.5% vs. 4.7%, p = 0.037) and were scored higher for interest in the program/area (4.34 ± 0.48 vs. 4.00 ± 0.62, p = 0.007). The format of interview was not associated with ultimate rank position by either univariate or multivariable analysis.

CONCLUSION

Among applicants for residency training positions, allowing a choice of interview format was associated with significant demographic and academic differences between those interviewing virtually versus in-person but had little ultimate effect on ROL position.

Evidence-Based Practices for Interviewing Graduate Medical Education Applicants: A Systematic Review

Background Although the selection interview is a standard admission practice for graduate medical education (GME) programs in the United States, there is a dearth of recent reviews on optimizing the trainee interview process, which has low reliability, high cost, and major risk of bias. Objective To investigate the evidence base for different selection interview practices in GME. Methods We searched 4 literature databases from inception through September 2022. Two investigators independently conducted title/abstract screening, full-text review, data extraction, and quality assessment. Disagreements were mediated by discussion. We used backward reference searching of included articles to identify additional studies. We included studies of different interview methods and excluded literature reviews, non-GME related publications, and studies comparing different applicant populations. We examined study characteristics, applicant and interviewer preferences, and interview format. We evaluated study quality using the Medical Education Research Study Quality Instrument (MERSQI). Results Of 2192 studies, 39 (2%) met our inclusion criteria. The evidence base was rated as moderately low quality using MERSQI criteria. Applicants reported preferences for several one-on-one interviews lasting 15 to 20 minutes, interviews by current trainees, and interviews including social events with only trainees. Applicants had mixed perceptions of virtual versus in-person interviews and reported that virtual interviews saved costs. The multiple mini interview (MMI) required more applicant and interviewer time than individual interviews but demonstrated construct and predictive validity and was preferred by applicants and interviewers. Conclusions Based on moderately low-quality evidence, using the MMI, training interviewers, and providing applicants with basic program information in advance should be considered for GME selection interviews.

Applicants Performance in Interview for a General Surgery Residency Pre- and During Coronavirus Disease-19 Pandemic

INTRODUCTION

The coronavirus disease-19 (COVID-19) pandemic has created a shift from traditional in-person interviews for a residency position to a virtual platform over the last 2 y (2020-2021). With this significant shift, there is controversy about how this may affect applicant performance. We hypothesized that applicants scored higher during virtual interviews than in-person interviews.

METHODS

Applicants interviewed for postgraduate year 1 positions at a single university-based residency program over six interview cycles were divided into group A (4 cycles, pre-COVID-19) and group B (2 cycles, during COVID-19). All candidates were interviewed by faculty and senior surgical residents and scored on letters of recommendation (LOR), personal statement (PS), research (RS), the way the candidate represented themselves (RP), interest in the geographic area (IN), and degree of connection between candidate and interviewer (CN). Differences in mean scores between candidates in these two groups were assessed using an independent-sample t-test.

RESULTS

Of 332 total applicants, 187 (56.3%) were in group A (cycles 2016-2019) and 145 (43.7%) were in group B (cycles 2020-2022). Group-B had higher United States Medical Licensing Examination Step 2 and Step 1 mean (standard deviation) scores (251.1 ± 10.6 versus 247.6 ± 11.8, P = 0.006 and 241.1 ± 10.3 versus 238.4 ± 10.4, P = 0.02). Group A scored higher median [interquartile range] in CN (4.4 [4.1, 4.6] versus 4.2 [3.9, 4.6], P = 0.005) and IN (4.3 [4.0, 4.5] versus 4.1 [3.8, 4.4], P = 0.002), while group B scored higher in LOR (4.3 [4.1, 4.6] versus 4.2 [3.9, 4.5], P = 0.011) and PS (4.3 [4.1, 4.4] versus 4.1 [3.8, 4.4], P = 0.001). No differences were noted between groups A and B in RS (3.9 [3.4, 4.5] versus 4.1 [3.5, 4.5], P = 0.108) or RP (4.4 [4.1, 4.7] versus 4.4 [4.2, 4.8], P = 0.317). No differences in the median [interquartile range] total scores of personal characteristic categories (25.1 [23.7, 26.5] versus 25.1 [23.8, 26.7], P = 0.703) were observed.

CONCLUSIONS

From a program perspective, there were no differences between groups; however, candidates in the pre-COVID era scored higher in CN and IN, while those who interviewed virtually had higher scores in LOR and PS. Applicants perceptions may differ regarding in-person versus virtual interviews.

Did the COVID-19 Pandemic Impact the Caliber of Trainees Taken During Match 2021? A Survey of General Surgery Residency Program Directors

BACKGROUND

The cancellation of clinical rotations (CRs) and implementation of virtual interviews (VIs) profoundly affected the residency selection process leading up to the 2021 NRMP Match. The authors investigated how these changes influenced the caliber of applicants taken by general surgery (GS) residency programs from the perspectives of program directors (PDs).

METHODS

A 14 question, web-based electronic survey was emailed to PDs of ACGME-accredited GS residency programs. Questions sought program characteristics and PDs' perspectives regarding potential differences in subjective characteristics and clinical skills demonstrated by their 2021 Match class relative to previous resident classes.

RESULTS

A total of 75 PDs (27.2%) responded to our survey. Most respondents observed no changes in residents' fit with their program (72.0%), communication skills (68.0%), responsiveness to clinical instruction and feedback (73.3%), work ethic (73.3%), and rotation evaluations (68.0%). Only 21.3% of PDs believed that VIs negatively impacted their ability to accurately assess applicant intangibles. Conversely, 56.0% of PDs reported that the cancellation of CRs in 2020 negatively affected residents' clinical competency at the start of residency. At 1-year following the 2021 NRMP Match, 30.7% of PDs reported that the clinical skills exhibited by their 2021 Match class were poorer than previous resident classes.

DISCUSSION

Our findings suggest that VIs limited selection committees' ability to accurately assess applicant's subjective characteristics to a lesser degree than previously described in the literature. Canceled CRs adversely affected the 2021 Match Class's clinical skills at the start of residency and at 1 year following the 2021 NRMP Match.

The USMLE® STEP 1 Pass or Fail Era of the Vascular Surgery Residency Application Process: Implications for Structural Bias and Recommendations

MOTIVES BEHIND THE SHIFT

USMLE® STEP 1 score reporting has been changed to a binary pass/fail format since January 26, 2022. The motives behind this change were (1) the questionable validity of using USMLE® STEP 1 as a screening tool during the candidate selection process and (2) the negative impact of using standardized examination scores as an initial gatekeeping threshold for the underrepresented in medicine (URiM) candidates applying to GME programs, given their generally lower mean standardized exams scores compared to non-URiM students. The USMLE® administrators justified this change as a tactic to enhance the overall educational experience for all students and to increase the representation of URiM groups. Moreover, they advised the program directors (PDs) to give more attention to other important qualities and components such as the applicant's personality traits, leadership roles and other extracurricular accomplishments, as part of a holistic evaluation strategy. IMPACT OF USMLE® STEP 1 PASS/FAIL: At this early stage, it is unclear how this change will impact Vascular Surgery Integrated residency (VSIR) programs. Several questions are outstanding, most importantly, how VSIR PDs will evaluate applicants absent the variable which heretofore was the primary screening tool. Our previously published survey showed that VSIR PDs will move their attention to other measures such as USMLE® STEP 2CK and letters of recommendation during the VSIR selection process. Furthermore, more emphasis on subjective measures such as the applicant's medical school rank and extracurricular student activities is expected. Given the expected higher weight of USMLE® STEP 2CK in the selection process than ever, many anticipate that medical students will dedicate more of their limited time to its preparation at the expense of both clinical and non-clinical activities. Potentially leaving less time to explore specialty pathways and to determine whether VS is the appropriate career for them.

A FRAMEWORK FOR CANDIDATE EVALUATION

The critical juncture in the VSIR candidate evaluation paradigm presents an opportunity to thoughtfully transform the process via current (Standardized Letter of Recommendation, USMLE® STEP 2CK, and clinical research) and future (Emotional Intelligence, Structure Interview and Personality Assessment) measures which constitute a framework to follow in the USMLE® STEP 1 pass/fail era.

Becoming a Sustainable Academic Surgeon

INTRODUCTION

Health care facilities represent a significant source of pollution, contributing to the growing problems associated with global warming. The resulting climate change impacts our health through worsening air and water quality, diminished access to nutritious food, and safe shelter.

METHODS

We outline here the not only the role of the surgeon in contributing to climate change, but also ways in which to minimize one's carbon footprint.

RESULTS

Surgeons are leaders within healthcare systems. Adopting environmentally conscious practices can reduce solid waste, energy usage, and carbon emissions. Practices outside of the clinical setting can also incorporate sustainability, including the use of virtual recruitment and educational programs, as well as thoughtful and conscientious travel practices.

CONCLUSIONS

Academic surgery combines clinical practice with an element of leadership, at all levels. Our recognition and action to reduce wasteful practices can help leave a better earth for generations to come.

Virtual interviewing for graduate medical education recruitment and selection: A BEME systematic review: BEME Guide No. 80

BACKGROUND

The COVID-19 pandemic caused graduate medical education (GME) programs to pivot to virtual interviews (VIs) for recruitment and selection. This systematic review synthesizes the rapidly expanding evidence base on VIs, providing insights into preferred formats, strengths, and weaknesses.

METHODS

PubMed/MEDLINE, Scopus, ERIC, PsycINFO, MedEdPublish, and Google Scholar were searched from 1 January 2012 to 21 February 2022. Two authors independently screened titles, abstracts, full texts, performed data extraction, and assessed risk of bias using the Medical Education Research Quality Instrument. Findings were reported according to Best Evidence in Medical Education guidance.

RESULTS

One hundred ten studies were included. The majority (97%) were from North America. Fourteen were conducted before COVID-19 and 96 during the pandemic. Studies involved both medical students applying to residencies (61%) and residents applying to fellowships (39%). Surgical specialties were more represented than other specialties. Applicants preferred VI days that lasted 4-6 h, with three to five individual interviews (15-20 min each), with virtual tours and opportunities to connect with current faculty and trainees. Satisfaction with VIs was high, though both applicants and programs found VIs inferior to in-person interviews for assessing 'fit.' Confidence in ranking applicants and programs was decreased. Stakeholders universally noted significant cost and time savings with VIs, as well as equity gains and reduced carbon footprint due to eliminating travel.

CONCLUSIONS

The use of VIs for GME recruitment and selection has accelerated rapidly. The findings of this review offer early insights that can guide future practice, policy, and research.

Faculty Scoring of General Surgery Residency Interviewees: A Comparison of In-Person and Virtual Interview Formats

OBJECTIVE

With new rules regarding social distancing and non-essential travel bans, we sought to determine if faculty scoring of general surgery applicants would differ between the in-person interview (IPI) and virtual interview (VI) platforms.

DESIGN

A single institution, retrospective review comparing faculty evaluation scores of applicant interviewees in the 2019 and 2020 MATCH® application cycles (IPIs) and the 2021 and 2022 application cycle (VIs) was conducted. Faculty scored applicants using a 5-point Likert scale in 7 areas of assessment and assigned each student to 1 of 4 tiers (tier 1 highest). A composite score for the 7 assessments (maximum score 35) was calculated. Mean and composite scores and tiers were compared between VI and IPI cycles and adjusted for within-interviewer correlations. The variance of the 2 groups were also compared.

SETTING

Harbor-UCLA Medical Center, an academic, tertiary care hospital.

PARTICIPANTS

General Surgery applicants for the 2019 to 2022 MATCH® application cycles.

RESULTS

Four hundred forty-one faculty IPI ratings of General Surgery applicants were compared to 531VI ratings. No difference in mean composite scores, individual assessments, or tier ranking. Less variance was identified in the VI group for academic credentials (0.6 vs 0.6, p = 0.01), strength of letters (0.7 vs 0.4, p = 0.005), communication skills (0.4 vs 0.6, p = 0.01), personal qualities (0.2 vs 0.5, p = 0.02), overall sense of fit for program (0.6 vs 0.9, p = 0.01), and tier ranking (0.3 vs 0.4, p = 0.004).

CONCLUSIONS

Faculty ratings of General Surgery applicants in the VI format appear to be similar to IPI. However, faculty ratings of VI applicants demonstrated less variability in scores in most assessments. This finding is potentially concerning, as it may suggest an inability of VI to detect subtle differences between applicants as comparted to IPI.

Interviewing Amidst a Pandemic: Perspectives of US Residency Program Directors on the Virtual Format

COVID-19 imposed significant limitations upon the 2021 U.S. National Resident Matching Program (NRMP), most important of which is the replacement of traditional in-person interviews with a virtual format. To determine the strengths, limitations, and overall utility of virtual interviews (VIs) for residency applicant selection, a 14-question electronic survey was administered to programme directors (PDs) of all American Council for Graduate Medical Education (ACGME)-accredited residency programmes, from December 2020 through March 2021. PDs were asked about their experience with VIs and the ability to assess residency applicants using the virtual format. A total of 1123 PDs (30% response rate) representing 30 different specialities responded to our survey. Compared to in-person interviews, VIs made it more challenging to assess applicants' fit with the programme, emotional intelligence, commitment to speciality, and ability to function as a resident physician. Overall, only 15% of PDs believed that VIs were better than in-person interviews. Once travel restrictions are lifted and in-person interviews are possible, 67% of PDs plan on hosting dual-format residency interviews, while 26% and 5% of PDs will exclusively host in-person interviews and VIs, respectively. This result was significantly different between surgical and non-surgical programmes [35% of surgical PDs suggested they would offer in-person interviews exclusively, compared to 21% of non-surgical PDs, p < 0.0001]. Although proven to be cost and time-efficient, VIs were challenging in evaluating certain qualities of residency applicants. While this study was focused on U.S. residency matching, it provides important insights about the future of VIs in medical recruitment as a whole.