Virtual Multiple Mini-Interviews for Veterinary Admissions

Virtual Multiple Mini Interviews: Implementation and evaluation for undergraduate medical, dental, and oral health degrees

INTRODUCTION

The Multiple Mini Interview (MMI) is a widely used method for evaluating non-cognitive skills of applicants to health professions training programs. The COVID-19 pandemic prompted a swift transition to virtual MMIs (vMMIs), with initial indications of their feasibility and acceptability. This novel study describes the implementation and evaluation of a large-scale vMMI model for selection into undergraduate medical, dental, and oral health degrees within an Australian university.

METHODS

A vMMI process was implemented in November 2022, comprising onsite interviewers (n = 450) and remote applicants (n = 1001). Using the online platform Zoom, applicants rotated through six to eight 10-min virtual stations. Acceptability of the virtual process was evaluated using online surveys, completed by interviewers (n = 396) and applicants (n = 255).

RESULTS

Overall, interviewers (98%) and applicants (96%) reported the vMMI as a positive experience, citing clear information, suitable interview questions, Zoom use, diversity of interviewers, and lack of gender and cultural bias as strengths of the process. Challenges included difficulties with rapport building and the disturbance of background noise.

CONCLUSIONS

A large-scale vMMI for three health programs was successfully implemented and positively evaluated. As the uptake of virtual interviews are likely to increase, this study can provide guidance for institutions planning to implement comparable vMMI processes.

A Comparison of Machine-Graded (ChatGPT) and Human-Graded Essay Scores in Veterinary Admissions

Admissions committees have historically emphasized cognitive measures, but a paradigm shift toward holistic reviews now places greater importance on non-cognitive skills. These holistic reviews may include personal statements, experiences, references, interviews, multiple mini-interviews, and situational judgment tests, often requiring substantial faculty resources. Leveraging advances in artificial intelligence, particularly in natural language processing, this study was conducted to assess the agreement of essay scores graded by both humans and machines (OpenAI's ChatGPT). Correlations were calculated among these scores and cognitive and non-cognitive measures in the admissions process. Human-derived scores from 778 applicants in 2021 and 552 in 2022 had item-specific inter-rater reliabilities ranging from 0.07 to 0.41, while machine-derived inter-replicate reliabilities ranged from 0.41 to 0.61. Pairwise correlations between human- and machine-derived essay scores and other admissions criteria revealed moderate correlations between the two scoring methods (0.41) and fair correlations between the essays and the multiple mini-interview (0.20 and 0.22 for human and machine scores, respectively). Despite having very low correlations, machine-graded scores exhibited slightly stronger correlations with cognitive measures (0.10 to 0.15) compared to human-graded scores (0.01 to 0.02). Importantly, machine scores demonstrated higher precision, approximately two to three times greater than human scores in both years. This study emphasizes the importance of careful item design, rubric development, and prompt formulation when using machine-based essay grading. It also underscores the importance of employing replicates and robust statistical analyses to ensure equitable applicant ranking when integrating machine grading into the admissions process.

'Click, I Guess I'm Done': Applicants' and Assessors' Experiences Transitioning to a Virtual Multiple Mini Interview Format

Introduction

During the COVID-19 pandemic, medical schools were forced to suspend in-person interviews and transition to a virtual Multiple Mini Interview (vMMI) format. MMIs typically comprise multiple short assessments overseen by assessors, with the aim of measuring a wide range of non-cognitive competencies. The adaptation to vMMI required medical schools to make swift changes to their MMI structure and delivery. In this paper, we focus on two specific groups greatly impacted by the decision to transition to vMMIs: medical school applicants and MMI assessors.

Methods

We conducted an interpretive qualitative study to explore medical school applicants' and assessors' experiences transitioning to an asynchronous vMMI format. Ten assessors and five medical students from one Canadian medical school participated in semi-structured interviews. Data was analyzed using a thematic analysis framework.

Results

Both applicants and assessors shared a mutual feeling of longing and nostalgia for an interview experience that, due to the pandemic, was understandably adapted. The most obvious forms of loss experienced - albeit in different ways - were: 1) human connection and 2) missed opportunity. Applicants and assessors described several factors that amplified their grief/loss response. These were: 1) resource availability, 2) technological concerns, and 3) the virtual interview environment.

Discussion

While virtual interviewing has obvious advantages, we cannot overlook that asynchronous vMMIs do not lend themselves to the same caliber of interaction and camaraderie as experienced in in-person interviews. We outline several recommendations medical schools can implement to enhance the vMMI experience for applicants and assessors.

Effects of candidates' demographics and evaluation of the virtual Multiple Mini Interview (vMMI) as a tool for selection into paediatric training in Queensland

INTRODUCTION

The Queensland Basic Paediatric Training Network (QBPTN) is responsible for the selection of candidates into paediatric training in Queensland. The COVID-19 pandemic necessitated interviews to be conducted 'virtually' as virtual Multiple-Mini-Interviews (vMMI). The study aimed to describe the demographic characteristics of candidates applying for selection into paediatric training in Queensland, and to explore their perspectives and experiences with the vMMI selection tool.

METHODOLOGY

The demographic characteristics of candidates and their vMMI outcomes were collected and analysed with a mixed methods approach. The qualitative component was comprised of seven semi-structured interviews with consenting candidates.

RESULTS

Seventy-one shortlisted candidates took part in vMMI and 41 were offered training positions. The demographic characteristics of candidates at various stages of selection were similar. The mean vMMI scores were not statistically different between candidates from the Modified Monash Model 1 (MMM1) location and others [mean (SD): 43.5 (5.1) versus 41.7 (6.7), respectively, p = 0.26]. However, there was a statistically significant difference (p value 0.03) between being offered and not offered a training position for candidates from MMM2 and above. The analysis of the semi-structured interviews suggested that candidate experiences of the vMMI were influenced by the quality of the management of the technology used. Flexibility, convenience, and reduced stress were the main factors that influenced candidates' acceptance of vMMI. Perceptions of the vMMI process focused on the need to build rapport and facilitate communication with the interviewers.

DISCUSSION

vMMI is a viable alternative to face-to-face (FTF) MMI. The vMMI experience can be improved by facilitating enhanced interviewer training, by making provision for adequate candidate preparation and by having contingency plans in place for unexpected technical challenges. Given government priorities in Australia, the impact of candidates' geographical location on the vMMI outcome for candidates from MMM >1 location needs to be further explored.

Validity evidence for a virtual multiple mini interview at a pharmacy program

BACKGROUND

Numerous health professions schools have transitioned to virtual admissions interviews in recent years. While some research suggests that virtual multiple mini-interviews (vMMIs) are feasible, acceptable, and more affordable, there is a paucity of research concerning the validity of this approach. The purpose of this study was to examine the validity and reliability of vMMIs and explore differences in performance between vMMI and in-person MMIs.

METHODS

Data were collected for two years of in-person MMIs and two years of vMMIs at a pharmacy program/school in the United States. An exploratory factor analysis (principal components analysis) with varimax rotation and Kaiser rule (i.e. retaining factors with eigenvalue > 1.0) was used to explore the construct validity of the vMMI data. Pearson correlation was used to examine correlations between vMMI stations and Cronbach alpha was used to determine the internal consistency of each station. Independent t-tests were used to examine differences between in-person MMI and vMMI scores. Cohen's d was used to determine effect sizes.

RESULTS

Four hundred and thirty-eight (42.69%) candidates completed an in-person MMI and 588 (57.31%) completed a vMMI. Factor analysis indicated that each vMMI station formed a single factor with loads ranging from 0.86 to 0.96. The vMMI stations accounted for most of the total variance, demonstrated weak to negligible intercorrelations, and high internal consistency. Significant differences between in-person and vMMI scores were found for the teamwork-giving, teamwork-receiving, and integrity stations. Medium effect sizes were found for teamwork-giving and teamwork-receiving and a small effect size was found for integrity.

CONCLUSIONS

Initial evidence suggests that the vMMI is a valid and reliable alternative to in-person MMIs. Additional research is needed to examine sources of differences in rating patterns between the two approaches and identify strategies that align with institutional priorities for recruitment and admissions.

Part 2: Assessment of a virtual vs. onsite interview experience from the interviewer perspective

INTRODUCTION

The University of Texas at Austin College of Pharmacy transitioned from onsite interviews to virtual interviews in fall 2020. There is limited literature on whether the virtual format impacts an interviewer's assessment of a candidate. This study examined interviewer ability to assess candidates and barriers to participation.

METHODS

During the virtual interview process, interviewers utilized a modified multiple mini interview (mMMI) format to evaluate prospective college of pharmacy students. An 18-item survey was emailed to 62 interviewers from the 2020-2021 cycle. Virtual mMMI scores were compared to the previous year's onsite MMI scores. Descriptive statistics and thematic analysis were used to assess the data.

RESULTS

The response rate to the survey was 53% (33/62), and 59% of interviewers preferred virtual interviews to in-person. Interviewers stated that there were fewer barriers to participation, increased comfort, and more time with applicants during virtual interviews. For six of the nine attributes evaluated, ≥ 90% of interviewers reported that they were able to assess applicants as well as they did in person. When comparing virtual and onsite MMI scores, seven of nine attributes were statistically significantly higher in the virtual cohort than onsite.

CONCLUSIONS

From the perspective of interviewers, the virtual interview decreased barriers to participation and still allowed the ability to assess the candidates. While offering a choice of interview setting to interviewers could increase accessibility, the statistically significant difference in MMI scores between virtual and onsite formats suggests that additional standardization is required to offer both formats simultaneously.