Internet-based multiple mini-interviews for candidate selection for graduate entry programmes

Evidence-Based Practices for Interviewing Graduate Medical Education Applicants: A Systematic Review

Background Although the selection interview is a standard admission practice for graduate medical education (GME) programs in the United States, there is a dearth of recent reviews on optimizing the trainee interview process, which has low reliability, high cost, and major risk of bias. Objective To investigate the evidence base for different selection interview practices in GME. Methods We searched 4 literature databases from inception through September 2022. Two investigators independently conducted title/abstract screening, full-text review, data extraction, and quality assessment. Disagreements were mediated by discussion. We used backward reference searching of included articles to identify additional studies. We included studies of different interview methods and excluded literature reviews, non-GME related publications, and studies comparing different applicant populations. We examined study characteristics, applicant and interviewer preferences, and interview format. We evaluated study quality using the Medical Education Research Study Quality Instrument (MERSQI). Results Of 2192 studies, 39 (2%) met our inclusion criteria. The evidence base was rated as moderately low quality using MERSQI criteria. Applicants reported preferences for several one-on-one interviews lasting 15 to 20 minutes, interviews by current trainees, and interviews including social events with only trainees. Applicants had mixed perceptions of virtual versus in-person interviews and reported that virtual interviews saved costs. The multiple mini interview (MMI) required more applicant and interviewer time than individual interviews but demonstrated construct and predictive validity and was preferred by applicants and interviewers. Conclusions Based on moderately low-quality evidence, using the MMI, training interviewers, and providing applicants with basic program information in advance should be considered for GME selection interviews.

'Click, I Guess I'm Done': Applicants' and Assessors' Experiences Transitioning to a Virtual Multiple Mini Interview Format

Introduction

During the COVID-19 pandemic, medical schools were forced to suspend in-person interviews and transition to a virtual Multiple Mini Interview (vMMI) format. MMIs typically comprise multiple short assessments overseen by assessors, with the aim of measuring a wide range of non-cognitive competencies. The adaptation to vMMI required medical schools to make swift changes to their MMI structure and delivery. In this paper, we focus on two specific groups greatly impacted by the decision to transition to vMMIs: medical school applicants and MMI assessors.

Methods

We conducted an interpretive qualitative study to explore medical school applicants' and assessors' experiences transitioning to an asynchronous vMMI format. Ten assessors and five medical students from one Canadian medical school participated in semi-structured interviews. Data was analyzed using a thematic analysis framework.

Results

Both applicants and assessors shared a mutual feeling of longing and nostalgia for an interview experience that, due to the pandemic, was understandably adapted. The most obvious forms of loss experienced - albeit in different ways - were: 1) human connection and 2) missed opportunity. Applicants and assessors described several factors that amplified their grief/loss response. These were: 1) resource availability, 2) technological concerns, and 3) the virtual interview environment.

Discussion

While virtual interviewing has obvious advantages, we cannot overlook that asynchronous vMMIs do not lend themselves to the same caliber of interaction and camaraderie as experienced in in-person interviews. We outline several recommendations medical schools can implement to enhance the vMMI experience for applicants and assessors.

The more things change the more they stay the same: Factors influencing emergency medicine residency selection in the virtual era

Background

Interviews for emergency medicine (EM) residency positions largely transitioned to a virtual-only format in 2020-2021. The impact of virtual interview factors on applicants' rank of programs is unknown.

Objective

We sought to assess the impact of modifiable factors in virtual interviews on applicants' rank of EM residency programs.

Methods

We conducted a cross-sectional mixed-methods survey of students applying to at least one of seven study authors' EM residency programs in the United States during the 2020-2021 application cycle. The survey was developed using an interactive Delphi process and piloted prior to implementation. The survey was administered from May to June 2021 with up to four email reminders. Quantitative analysis included descriptive statistics. Three authors performed a thematic qualitative analysis of free-text responses.

Results

A total of 664 of 2281 (29.1%) students completed the survey, including 335 (50.5%) male, 316 (47.7%) female, and six (0.9%) nonbinary. A total of 143 (21.6%) respondents identified as underrepresented in medicine and 84 (12.7%) identified as LGBTQIA+. Respondents participated in a median of 14 interviews and ranked a median of 14 programs. Most respondents (335, 50.6%) preferred a choice of in-person or virtual, while 183 (27.6%) preferred all in-person, and 144 (21.8%) preferred all virtual. The program website and interview social were the most important factors influencing respondent ranking. Qualitative analysis revealed several positive aspects of virtual interviews including logistical ease and comfort. Negative aspects include technical issues, perceived interview hoarding, and barriers to applicant assessment and performance. Demonstrated effort by the program, effective information delivery, communication of resident culture, and a well-implemented interview day positively influenced respondents' rank of programs.

Conclusions

This study identified characteristics of the virtual interview format that impact applicants' rank of programs. These results can inform future recruitment practices.

Effects of candidates' demographics and evaluation of the virtual Multiple Mini Interview (vMMI) as a tool for selection into paediatric training in Queensland

INTRODUCTION

The Queensland Basic Paediatric Training Network (QBPTN) is responsible for the selection of candidates into paediatric training in Queensland. The COVID-19 pandemic necessitated interviews to be conducted 'virtually' as virtual Multiple-Mini-Interviews (vMMI). The study aimed to describe the demographic characteristics of candidates applying for selection into paediatric training in Queensland, and to explore their perspectives and experiences with the vMMI selection tool.

METHODOLOGY

The demographic characteristics of candidates and their vMMI outcomes were collected and analysed with a mixed methods approach. The qualitative component was comprised of seven semi-structured interviews with consenting candidates.

RESULTS

Seventy-one shortlisted candidates took part in vMMI and 41 were offered training positions. The demographic characteristics of candidates at various stages of selection were similar. The mean vMMI scores were not statistically different between candidates from the Modified Monash Model 1 (MMM1) location and others [mean (SD): 43.5 (5.1) versus 41.7 (6.7), respectively, p = 0.26]. However, there was a statistically significant difference (p value 0.03) between being offered and not offered a training position for candidates from MMM2 and above. The analysis of the semi-structured interviews suggested that candidate experiences of the vMMI were influenced by the quality of the management of the technology used. Flexibility, convenience, and reduced stress were the main factors that influenced candidates' acceptance of vMMI. Perceptions of the vMMI process focused on the need to build rapport and facilitate communication with the interviewers.

DISCUSSION

vMMI is a viable alternative to face-to-face (FTF) MMI. The vMMI experience can be improved by facilitating enhanced interviewer training, by making provision for adequate candidate preparation and by having contingency plans in place for unexpected technical challenges. Given government priorities in Australia, the impact of candidates' geographical location on the vMMI outcome for candidates from MMM >1 location needs to be further explored.

Validity evidence for a virtual multiple mini interview at a pharmacy program

BACKGROUND

Numerous health professions schools have transitioned to virtual admissions interviews in recent years. While some research suggests that virtual multiple mini-interviews (vMMIs) are feasible, acceptable, and more affordable, there is a paucity of research concerning the validity of this approach. The purpose of this study was to examine the validity and reliability of vMMIs and explore differences in performance between vMMI and in-person MMIs.

METHODS

Data were collected for two years of in-person MMIs and two years of vMMIs at a pharmacy program/school in the United States. An exploratory factor analysis (principal components analysis) with varimax rotation and Kaiser rule (i.e. retaining factors with eigenvalue > 1.0) was used to explore the construct validity of the vMMI data. Pearson correlation was used to examine correlations between vMMI stations and Cronbach alpha was used to determine the internal consistency of each station. Independent t-tests were used to examine differences between in-person MMI and vMMI scores. Cohen's d was used to determine effect sizes.

RESULTS

Four hundred and thirty-eight (42.69%) candidates completed an in-person MMI and 588 (57.31%) completed a vMMI. Factor analysis indicated that each vMMI station formed a single factor with loads ranging from 0.86 to 0.96. The vMMI stations accounted for most of the total variance, demonstrated weak to negligible intercorrelations, and high internal consistency. Significant differences between in-person and vMMI scores were found for the teamwork-giving, teamwork-receiving, and integrity stations. Medium effect sizes were found for teamwork-giving and teamwork-receiving and a small effect size was found for integrity.

CONCLUSIONS

Initial evidence suggests that the vMMI is a valid and reliable alternative to in-person MMIs. Additional research is needed to examine sources of differences in rating patterns between the two approaches and identify strategies that align with institutional priorities for recruitment and admissions.

Multiple mini-interviews is a predictor of students' academic achievements in early undergraduate medical years: a retrospective study

BACKGROUND

Our study determined Multiple Mini-Interview (MMI) effectiveness in evaluating specific skill sets based on medical students' performances during the undergraduate years and compared the academic performances of medical students who appeared for onsite/online MMI.

METHODS

A retrospective study of 140 undergraduate medical students between 2016 and 2020 included data on age, gender, pre-university results, MMI scores, and examination results. Appropriate non-parametric tests were applied to compare the students' MMI and academic performances.

RESULTS

Ninety-eight students from cohorts 12 to 15 had an overall MMI score of 69.0(IQR: 65.0-73.2)/100 and an overall Cumulative Grade Point Average(GPA) of 3.64 (3.42-3.78)/5.0. Spearman's correlation revealed a significantly positive relationship between MMI and cGPA (rho = 0.23) and GPA from the first 2 semesters (GPA1 rho = 0.25, GPA2 rho = 0.27). This observation was similar to that for station A in the first year (cGPA rho = 0.28, GPA1 rho = 0.34, GPA2 rho = 0.24), and in station B (GPA4 rho = 0.25) and D (GPA3 rho = 0.28, GPA4 rho = 0.24) in the second year. Of twenty-nine cohort16 students, 17(58.6%) underwent online and 12(41.4%) offline modes of MMI assessment, respectively. The overall median MMI score was 66.6(IQR: 58.6-71.6)/100, and the overall median cGPA was 3.45 (3.23-3.58)/5.0. When comparing the median marks of cohort16 groups, the online group scored significantly higher marks for station D than the offline group (p = 0.040).

CONCLUSION

Correspondence between MMI scores and cGPA predicted MMI scoring during student selection and entry process might ensure the success of their academic performance in medical school.

Part 2: Assessment of a virtual vs. onsite interview experience from the interviewer perspective

INTRODUCTION

The University of Texas at Austin College of Pharmacy transitioned from onsite interviews to virtual interviews in fall 2020. There is limited literature on whether the virtual format impacts an interviewer's assessment of a candidate. This study examined interviewer ability to assess candidates and barriers to participation.

METHODS

During the virtual interview process, interviewers utilized a modified multiple mini interview (mMMI) format to evaluate prospective college of pharmacy students. An 18-item survey was emailed to 62 interviewers from the 2020-2021 cycle. Virtual mMMI scores were compared to the previous year's onsite MMI scores. Descriptive statistics and thematic analysis were used to assess the data.

RESULTS

The response rate to the survey was 53% (33/62), and 59% of interviewers preferred virtual interviews to in-person. Interviewers stated that there were fewer barriers to participation, increased comfort, and more time with applicants during virtual interviews. For six of the nine attributes evaluated, ≥ 90% of interviewers reported that they were able to assess applicants as well as they did in person. When comparing virtual and onsite MMI scores, seven of nine attributes were statistically significantly higher in the virtual cohort than onsite.

CONCLUSIONS

From the perspective of interviewers, the virtual interview decreased barriers to participation and still allowed the ability to assess the candidates. While offering a choice of interview setting to interviewers could increase accessibility, the statistically significant difference in MMI scores between virtual and onsite formats suggests that additional standardization is required to offer both formats simultaneously.

Best practices for interviewing applicants for medical school admissions: a systematic review

INTRODUCTION

Interviews are commonly used to select applicants for medical school, residency, and fellowship. However, interview techniques vary in acceptability, feasibility, reliability, and validity. This systematic review investigated the effectiveness of different interview methods in selecting the best qualified applicants for admission to medical school and developed a logic model to implement best practices for interviewing.

METHODS

Five electronic literature databases were searched for comparative studies related to interviewing in medical schools from inception through February 1, 2021. Inclusion criteria included publications in English that compared different methods of conducting a selection interview in medical schools with a controlled trial design. General study characteristics, measurement methodologies, and outcomes were reviewed. Quality appraisal was performed using the Medical Education Research Study Quality Instrument (MERSQI) and the Oxford Risk of Bias Scale. Based on these findings, a logic model was constructed using content analysis.

RESULTS

Thirteen studies were included. The multiple mini-interview (MMI) was reliable, unbiased, and predicted clinical and academic performance; the virtual MMI increased reliability and lowered costs. For unstructured interviews, blinding interviewers to academic scores reduced bias towards higher scorers; student and faculty interviewers rated applicants similarly. Applicants preferred structured over unstructured interviews. Study quality was above average per the MERSQI, risk of bias was high per the Oxford scale, and between-study heterogeneity was substantial.

DISCUSSION

There were few high-quality studies on interviewing applicants for admission to medical school; the MMI appears to offer a reliable method of interviewing. A logic model can provide a conceptual framework for conducting evidence-based admissions interviews.

Virtual Multiple Mini-Interviews for Veterinary Admissions

Admissions teams are challenged to select the best applicants for their college. There is a growing emphasis on selecting applicants with personal attributes important for success in a variety of veterinary careers, but there is no clear consensus on how to best identify these individuals. A number of veterinary colleges are utilizing multiple mini-interviews (MMIs), a highly structured type of interview in this selection process. However, due to travel restrictions currently associated with COVID-19, many are now considering virtual MMIs. Long Island University (LIU) took the step to conduct MMIs virtually for its inaugural class before the pandemic restrictions occurred, largely because it hoped to reduce the cost of admission by eliminating travel costs. In this process, we encountered a unique set of challenges, the resolution of which we believe constitutes best practices for virtual MMIs. This report describes the design and execution of an MMI for LIU. We were able to interview 340 applicants in 7 days. Based on feedback from applicants as well as raters, most considered it an acceptable means of interviewing students. Both raters and applicants expressed a high degree of satisfaction with the process, and we were able to separate applicants based on MMI scores with 88% reliability.

Video-based interviewing in medicine: a scoping review

BACKGROUND

The Coronavirus 2019 pandemic necessitated a rapid uptake of video-based interviewing within the personnel selection process in healthcare. While video-based interviews have been evaluated previously, we identified a gap in the literature on the implementation of video-based interviews and how they compare to their face-to-face counterparts.

METHODS

A scoping review was conducted to consolidate the available literature on the benefits and limitations of video-based interviews and to understand the perceived barriers associated with transitioning away from face-to-face interviews. A search strategy, developed in concert with an academic health sciences librarian, was run on Ovid MEDLINE, Embase, PsycInfo, and Cochrane Central. The search was performed on March 31, 2020, and updated on February 21, 2021. Studies that implemented and evaluated the impact of video-based interviewing in healthcare were included in our study. Review articles and editorials were excluded.

RESULTS

Forty-three studies were included in our scoping review, of which 17 were conference abstracts and 26 were peer-reviewed manuscripts. The risk of bias was moderate or high in most studies, with only four studies having a low risk of bias. Both financial costs and opportunity costs associated with the selection process were reported to be improved with video-based interviewing, while no studies explored the impact on environmental costs. Technical limitations, which were not prevalent, were easily managed during the interview process. Overall, video-based interviews were well received by both applicants and interviewers, although most participants still reported a preference for face-to-face interviews.

CONCLUSIONS

While video-based interviewing has become necessary during the Coronavirus 2019 era, there are benefits from a financial, opportunistic, and environmental point of view that argue for its continued use even after the pandemic. Despite its successful implementation with minimal technical issues, a preference still remains for face-to-face interviews. Reasons for this preference are not clear from the available literature. Future studies on the role of nonverbal communication during the video-based interview process are important to better understand how video-based interviewing can be optimized.

SYSTEMATIC REVIEW REGISTRATION

This scoping review was registered with Open Science Framework.

Feasibility of an automated interview grounded in multiple mini interview (MMI) methodology for selection into the health professions: an international multimethod evaluation

OBJECTIVES

Global, COVID-driven restrictions around face-to-face interviews for healthcare student selection have forced admission staff to rapidly adopt adapted online systems before supporting evidence is available. We have developed, what we believe is, the first automated interview grounded in multiple mini-interview (MMI) methodology. This study aimed to explore test-retest reliability, acceptability and usability of the system.

DESIGN, SETTING AND PARTICIPANTS

Multimethod feasibility study in Physician Associate programmes from two UK and one US university during 2019-2020.

PRIMARY, SECONDARY OUTCOMES

Feasibility measures (test-retest reliability, acceptability and usability) were assessed using intraclass correlation (ICC), descriptive statistics, thematic and content analysis.

METHODS

Volunteers took (T1), then repeated (T2), the automated MMI, with a 7-day interval (±2) then completed an evaluation questionnaire. Admission staff participated in focus group discussions.

RESULTS

Sixty-two students and seven admission staff participated; 34 students and 4 staff from UK and 28 students and 3 staff from US universities. Good-excellent test-retest reliability was observed at two sites (US and UK2) with T1 and T2 ICC between 0.65 and 0.81 (p<0.001) when assessed by individual total scores (range 80.6-119), station total scores 0.6-0.91, p<0.005 and individual site (≥0.79 p<0.001). Mean test re-test ICC across all three sites was 0.82 p<0.001 (95% CI 0.7 to 0.9). Admission staff reported potential to reduce resource costs and bias through a more objective screening tool for preselection or to replace some MMI stations in a 'hybrid model'. Maintaining human interaction through 'touch points' was considered essential. Users positively evaluated the system, stating it was intuitive with an accessible interface. Concepts chosen for dynamic probing needed to be appropriately tailored.

CONCLUSION

These preliminary findings suggest that the system is reliable, generating consistent scores for candidates and is acceptable to end users provided human touchpoints are maintained. Thus, there is evidence for the potential of such an automated system to augment healthcare student selection.

Adapting the Admissions Interview During COVID-19: A Comparison of In-Person and Video-Based Interview Validity Evidence

COVID-19 physical distancing limited many medical schools' abilities to conduct in-person interviews for the 2020 admissions cycle. The University of Toronto (U of T) Temerty Faculty of Medicine was already in the midst of its interview process, with two-thirds of applicants having completed the in-person modified personal interview (MPI). As the university and surrounding region were shut down, the shift was made in the middle of the application cycle to a semisynchronous video-based MPI interview (vMPI) approach. U of T undertook the development, deployment, and evaluation of the 2 approaches mid-admissions cycle. Existing resources and tools were used to create a tailored interview process with the assistance of applicants. The vMPI was similar in content and process to the MPI: a 4-station interview with each station mapped to attributes relevant to medical school success. Instead of live interviews, applicants recorded 5-minute responses to questions for each station using their own hardware. These responses were later assessed by raters asynchronously. Out of 627 applicants, 232 applicants completed the vMPI. Validity evidence was generated for the vMPI and compared with the MPI on the internal structure, relationship to other variables, and consequential validity, including applicant and interviewer acceptability. Overall, the vMPI demonstrated similar reliability and factor structure to the MPI. As with the MPI, applicant performance was predicted by nonacademic screening tools but not academic measures. Applicants' acceptance of the vMPI was positive. Most interviewers found the vMPI to be acceptable and reported confidence in their ratings. Continuing physical distancing concerns will require multiple options for admissions committees to select medical students. The vMPI is an example of a customized approach that schools can implement and may have advantages for selection beyond the COVID-19 pandemic. Future evaluation will examine additional validity evidence for the tool.

Implementation and evaluation of virtual multiple mini interviews as a selection tool for entry into paediatric postgraduate training: A Queensland experience

INTRODUCTION

The Queensland Basic Paediatric Training Network (QBPTN) is the centralised pathway for entry into paediatric training in Queensland, Australia. In response to COVID-19 travel and social distancing restrictions imposed in 2020, QBPTN successfully adopted a Virtual Multiple Mini Interviews (vMMIs) model for the selection of candidates for entry into paediatric training. The authors describe the planning, implementation, challenges, and evaluation of candidates' and interviewers' experiences of vMMIs, including the differences between candidates from two geographical areas.

METHODS

The contents of six vMMI stations were similar to face-to-face MMI. Implementation required the identification of ZOOM^TM as a preferred online platform, securing venues, communication, development of contingency plans and central coordination by the network. Candidates' experiences with vMMI were explored through thematic analysis of the qualitative data from focus groups and free text responses, and descriptive analysis of SurveyMonkey© questionnaire responses. Experiences between 'metropolitan' and 'regional and interstate' candidates were compared.

RESULTS

5-minute stations with 2-minute pre-reading were used. 78 candidates and 14 interviewers participated in the selection process. All candidates attended the focus group. 58.7% of candidates responded to post vMMI questionnaire. 93% of survey responders were happy to undertake vMMI in the future, with 23% feeling they would have performed better in face-to-face. Experiences between 'metropolitan' and 'other' groups were similar. Positive experiences of participants were related to the user-friendly IT platform, successful pre-interview communications, preparation, convenience, time, and cost savings. Stress related IT failures and difficulties establishing rapport with interviewers were reported as the main negative experiences.

CONCLUSION

'vMMI' is a feasible and acceptable method of selection into paediatric training. vMMI has many benefits and can be implemented relatively quickly by addressing key logistical requirements. The model under discussion could be adapted by other centres based on local needs.

Conducting multiple mini-interviews in the midst of COVID-19 pandemic

Multiple mini-interview (MMI) is a 'multiple sample-based' approach comprising multiple focused encounters intended to access and assess a range of attributes in order to gain more objectively multiple impressions of an applicant's interpersonal skills, thoughtfulness and general demeanour. It is designed to focus on four domains that are not considered to be comprehensive, but are considered to be vital for a successful career in the health sciences: critical thinking, ethical decision making, communication and knowledge of the healthcare system. Traditionally, the MMI is conducted face-to-face, but with COVID-19 pandemic and the implementation of social distancing measures, no onsite or campus teaching, banning of mass gatherings and cancellation of face-to-face interviews, Pengiran Anak Puteri Rashidah Sa'adatul Bolkiah Institute of Health Sciences at Universiti Brunei Darussalam explored the feasibility of conducting MMI through virtual means. This report provides an account of our experience in conducting internet-MMI for the selection of new applicants into the August 2020 cohort of the Medicine programme. We also aimed to determine whether the scores derived from internet-MMI were reliable and equivalent to the scores derived from traditional MMI.

Rapid Transition to a Virtual Multiple Mini-Interview Admissions Process: A New Medical School's Experience During the COVID-19 Pandemic

PROBLEM

The University of Houston College of Medicine (UH COM) began its first admissions cycle after receiving preliminary accreditation in February 2020. With the advent of remote learning in response to the COVID-19 pandemic, the school moved its admissions process, including multiple mini-interview (MMI), from an in-person to online format in mid-March 2020.

APPROACH

The UH COM selected Zoom as the video conferencing platform for its virtual admissions process, including MMI. On each interview day (3-4 hours), 14-16 applicants joined administrators, faculty, and staff in a virtual meeting room. Applicants were divided into 2 groups: one viewed short presentations about the school, curriculum, and departments, while the other participated in 7 MMI stations (one-on-one interactions with interviewers) via virtual breakout rooms; the groups then switched. The MMI stations were the same as those used in-person in early March. Applicants were able to ask questions at multiple points during the day. Technical support was provided for participants with connectivity issues or unfamiliar with Zoom.

OUTCOMES

Of the 180 applicants interviewed in March-April 2020, 134 (74%) participated in the virtual process and 46 (26%) in the on-site process. Twenty-five (83%) of the 30 members of the inaugural class of 2024 interviewed virtually. Advantages of the virtual format included ease of access for faculty and more flexibility and less expense for applicants. Challenges included the need for applicants to decide whether to accept an offer of admission from a new school without visiting and missed opportunities for faculty to have relatively unstructured interactions with applicants.

NEXT STEPS

This virtual admissions process was a feasible alternative for the inaugural class but is not sustainable. UH COM plans to leverage lessons learned to refine the virtual format for use in future admissions cycles, even when in-person interviews are possible.

Drastic times need drastic measures: COVID-19 and widening access to medicine

In response to COVID-19, schools, colleges and universities across the world have closed or shifted to online/remote or blended teaching, learning and assessment. These changed ways of working pose challenges to students and will likely exacerbate existing educational attainment gaps between different societal groups. Our focus is the potential impact of COVID-19 on widening access to medicine. We provide an account of the process, in the form of comparative cases, of applying for medical school for two applicants from differing backgrounds. Three challenges were identified: family circumstances and support (financial security and parental educational support); staying connected (access to educational material, technology and Wifi); getting the grades and meeting other entry criteria (predicting grades and work experience). We propose that medical schools adopt drastic measures to protect widening access including increasing the use of aptitude tests, contextualised admissions, online multiple mini interviews (MMIs), creative outreach and promotion of alternative means of gaining relevant experience.

Preparing for the interviewing process during Coronavirus disease-19 pandemic: Virtual interviewing experiences of applicants and interviewers, a systematic review

Purpose

Coronavirus disease-19 (COVID-19) has forced upon all academic institutions to conduct virtual interviewing (VI) instead of face-to-face interviewing (FTFI) this interviewing cycle. The purpose of this systematic review was to understand the process of VI, its effectiveness as an alternative to FTFI, and the experiences of applicants and institutions with VI. We also share best practice strategies for applicants and institutions in VI preparation.

Method

PubMed/MEDLINE, Cochrane Library of Systematic Reviews, Web of Science Core Collection, Scopus and CINAHL databases were searched through May 2020. Articles in English evaluating the effectiveness of VI were included, without applying any date limits. Two reviewers selected articles and extracted data.

Results

Of the 934 articles screened, 22 articles underwent full-text article analysis to include 15 studies. There were 4 studies that reported the use of VI as a screening tool. 11 studies completely replaced FTFI with VI. Most applicants could appropriately convey themselves through VI. Most applicants and interviewing programs expressed reservations about VI’s use as an alternative to FTFI.

Conclusion

There is dearth of evidence supporting the efficacy of VI. There is an opportunity for potential research at multi-institutional level to gain better understanding of the efficacy of VI. The knowledge obtained from this systematic review has the potential of helping applicants and institutions in preparing for VI process. Additionally, authors propose supportive strategies to help prepare applicants and institutions for VI.

"You can't always get what you want…": economic thinking, constrained optimization and health professions education

Every choice we make in health professions education has a cost, whether it be financial or otherwise; by choosing one action (e.g., integrating more simulation, studying more for a summative examination) we lose the opportunity to take an alternative action (e.g., freeing up time for other teaching, leisure time). Economics significantly shapes the way we behave and think as educators and learners and so there is increasing interest in using economic ways of thinking and approaches to examine and understand how choices are made, the influence of constraints and boundaries in educational decision making, and how costs are felt. Thus, in this article, we provide a brief historical overview of modern economics, to illustrate how the core concepts of economics-scarcity (and desirability), rationality, and optimization-developed over time. We explain the important concept of bounded rationality, which explains how individual, meso-factors and contextual factors influence decision making. We then consider the opportunities that these concepts afford for health professions education and research. We conclude by proposing that embracing economic thinking opens up new questions and new ways of approaching old questions which can add knowledge about how choice is enacted in contemporary health professions education.

Adaptation of internet-based multiple mini-interviews in a limited-resource medical school during the coronavirus disease 2019 pandemic

The coronavirus disease 2019 pandemic has affected many aspects of life, including medical education. The selection of students using multiple mini-interviews (MMI) should be adapted promptly as internet-based MMI (iMMI) in the current setting. The current transition from MMI to iMMI is supported by experience in conducting conventional MMI in previous years; hence, the station and assessor preparation could be completed without significant issues. The greater challenges to the use of iMMI are due to technical issues, including poor internet connections and adaptations necessary for assessing candidates' nonverbal expressions. This paper aims to describe the experience of conducting iMMI in the current pandemic situation, with further highlights on contextual challenges in a limited-resource setting.

Video-based interviewing in medicine: protocol for a scoping review

BACKGROUND

Careers in healthcare involve an extensive interview process as transitions are made from one level of training to the next. For physicians, interviews mark the gateway from entrance into medical school, acceptance into residency, fellowships, and subsequent job opportunities. Previous literature outlining the costs associated with face-to-face interviews and concerns regarding the climate crisis has triggered an interest in video-based interviews. Barriers to transitioning away from in-person interviews include concerns regarding lack of rapport between applicants and interviewers, and applicants being less able to represent themselves. In a new era ushered in by COVID where many of us have utilized virtual meetings more than any prior time both personally and for work, we wanted to consolidate the current literature on the use of video-based interviews in healthcare and summarize the findings.

METHODS

A scoping review will be conducted to explore the benefits and limitations of video-based interviews for both applicants and interviewers within healthcare fields, as well as the perceived barriers associated with transitioning away from face-to-face interviews. The scoping review methodology outlined by Arksey and O'Malley will be implemented. The search strategy developed by the authors in collaboration with an academic health sciences librarian will be conducted across four electronic databases (Embase, MEDLINE, Cochrane Central, and PsycInfo) and supplemented by a review of the grey literature and reference lists of included studies. The study selection process will be documented using the PRISMA flow diagram, and reasons for exclusion following full-text review will be recorded. The extracted data will be analyzed using quantitative and qualitative analysis.

DISCUSSION

Despite previous literature on the costs associated with face-to-face interviews, there has been hesitancy with transitioning to video-based interviews due to concerns of lack of rapport between applicants and interviewers, and applicants being less able to represent themselves. While these limitations have been explored in previous studies, a succinct review of the current literature to guide the effective restructuring of the interview process is lacking. With our scoping review, we hope to fill this gap in the literature to better understand barriers to transitioning from face-to-face interviews and directions for future research.

The economic costs of selecting medical students: An Australian case study

CONTEXT

The design of selection methods must balance, amongst a range of factors, the desire to select the best possible future doctors with the reality of our resource-constrained environment. Examining the cost of selection processes enables us to identify areas in which efficiencies may be gained.

METHODS

A cost description study was conducted based on selection for 2018 entry into medical school directly from secondary school. The perspectives of applicants, volunteer interviewers and the admitting institution were considered. Costs were modelled based on the Monash University (Australia) selection process, which uses a combination of secondary school matriculation score, aptitude test score (Undergraduate Medicine and Health Sciences Admission Test) and multiple mini-interview score. A variety of data sources were utilised, including bespoke surveys, audit data and existing literature. All costs are expressed in 2018 Australian dollars (AU$). Applicant behaviours in preparing for selection tests were also evaluated.

RESULTS

A total of 381 of 383 applicants returned the survey. Over 70% of applicants had utilised commercial preparation materials. The median total cost to applicants was AU$2586 (interquartile range [IQR] AU$1574-3999), including costs to both prepare for and attend selection tests. Of 217 volunteer interviewers, 108 returned the survey. These were primarily health professional clinicians at a mid-career stage. The median total cost to interviewers was AU$452 (IQR AU$252-715) for participation in a half-day interview session, largely due to the loss of income. The cost to the admitting institution was AU$269 per applicant, accounted for by the costs of equipment and consumables (52%), personnel (34%) and facilities (14%).

CONCLUSIONS

The costs of student selection for medical school are substantial. Understanding costs facilitates achievement of the objective of selecting the desired future medical workforce within the constraints of the resources available. Opportunities for change may arise from changes in applicant preparation behaviours, opportunities for economies of scale, and efficiencies driven by technological solutions.

Opening the black box of selection

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

Multiple Speed Assessments

Abstract. This paper presents Multiple Speed Assessments as an umbrella term to encompass a variety of approaches that include multiple (e.g., 20), short (e.g., 3 min), and often integrated interpersonal simulations to elicit overt behavior in a standardized way across participants. Multiple Speed Assessments can be used to get insight into the behavioral repertoire of a target person in situations sampled from a predefined target domain and their intraindividual variability across these situations. This paper outlines the characteristics and theoretical basis of Multiple Speed Assessments. We also discuss various already existing examples of Multiple Speed Assessments (Objective Structured Clinical Examinations, Multiple Mini-Interviews, and constructed response multimedia tests) and provide an overview of design variations. Finally, we present current research evidence and future research directions related to Multiple Speed Assessments. Although we present Multiple Speed Assessments in the context of personnel selection, it can also be used for assessment in the educational, personality, or clinical psychology field.

The Potential Impact of COVID-19 on the Medical School Application

The coronavirus disease 2019 (COVID-19) pandemic has had a significant impact on the medical community. It is suspected that the pandemic will impact the medical school application process due to effects on standardized testing, performance measures, financial burdens, and interview strategies. It is important to consider these issues early to optimize success of future strategies and mitigate the impact of COVID-19 on the application cycle.

Multiple Mini-Interviews: Current Perspectives on Utility and Limitations

The growing role of healthcare professionals urged admissions committees to restructure their selection process and assess key personal attributes rather than academic achievements only. Multiple mini interviews (MMIs) were designed in 2002 to assess such domains in prospective healthcare professions. Being a high-stake assessment, the utility and limitations of MMI need to be explored. The purpose of this article is to review the available evidence to establish its utility. The claim of the reliability is verified by the studies assessing the effect of number of stations, duration of stations, format and scoring systems of stations and number of raters assessing the applicants. Similarly, by gathering evidence concerning its content validity, convergent/divergent correlation and predictive ability, validity is ensured. Finally, its acceptability and feasibility along with limitations is discussed. This article concludes by providing recommendations for further work required to deal with the limitations and enhance its utility.

Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review

Objectives

Multiple Mini Interviews (MMI) have been conducted across the globe in the student selection process, particularly in health profession education. This paper reported the validity evidence of MMI in various educational settings.

Methods

A literature search was carried out through Scopus, Science Direct, Google Scholar, PubMed, and EBSCOhost databases based on specific search terms. Each article was appraised based on title, abstract, and full text. The selected articles were critically appraised, and relevant information to support the validity of MMI in various educational settings was synthesized. This paper followed the PRISMA guideline to ensure consistency in reporting systematic review results.

Results

A majority of the studies were from Canada, with 41.54%, followed by the United Kingdom (25.39%), the United States (13.85%), and Australia (9.23%). The rest (9.24%) were from Germany, Ireland, the United Arab Emirates, Japan, Pakistan, Taiwan, and Malaysia. Moreover, most MMI stations ranged from seven to 12 with a duration of 10 min per station (including a 2-min gap between stations).

Conclusion

The results suggest that the content, response process, and internal structure of MMI were well supported by evidence; however, the relation and consequences of MMI to important outcome variables were inconsistently supported. The evidence shows that MMI is a non-biased, practical, feasible, reliable, and content-valid admission tool. However, further research on its impact on non-cognitive outcomes is required.

An equivalence study of interview platform: Does videoconference technology impact medical school acceptance rates of different groups?

Web-based interviewing may be an effective element of a medical school's larger approach to promotion of holistic review, as recommended by the Association of American Medical Colleges, by facilitating the feasibility of including rural and community physicians in the interview process. Only 10% of medical schools offer videoconference interviews to applicants and little is known about the impact of this interview modality on the admissions process. This study investigated the impact of overall acceptance rates using videoconference interviews and face-to-face interviews in the medical school selection process using an equivalence trial design. The University of New Mexico School of Medicine integrated a videoconferencing interview option for community and rural physician interviewers in a pseudo-random fashion during the 2014-2016 admissions cycles. Logistic regression was conducted to examine whether videoconference interviews impacted acceptance rates or the characteristics of accepted students. Demographic, admissions and diversity factors were analyzed that included applicant age, MCAT score, cumulative GPA, gender, underrepresented in medicine, socioeconomic status and geographic residency. Data from 752 interviews were analyzed. Adjusted rates of acceptance for face-to-face (37.0%; 95% CI 28.2, 46.7%) and videoconference (36.1%; 95% CI 17.8, 59.5%) interviews were within an a priori ± 5% margin of equivalence. Both interview conditions yielded highly diverse groups of admitted students. Having a higher medical college admission test score, grade point average, and self-identifying as disadvantaged increased odds of admission in both interview modalities. Integration of the videoconference interview did not impact the overall acceptance of a highly diverse and qualified group of applicants, and allowed rural and community physicians to participate in the medical school interview process as well as allowed campus faculty and medical student committee members to interview remotely.

A systematic review of stakeholder views of selection methods for medical schools admission

BACKGROUND

The purpose of this paper is to systematically review the literature with respect to stakeholder views of selection methods for medical school admissions.

METHODS

An electronic search of nine databases was conducted between January 2000-July 2014. Two reviewers independently assessed all titles (n = 1017) and retained abstracts (n = 233) for relevance. Methodological quality of quantitative papers was assessed using the MERSQI instrument. The overall quality of evidence in this field was low. Evidence was synthesised in a narrative review.

RESULTS

Applicants support interviews, and multiple mini interviews (MMIs). There is emerging evidence that situational judgement tests (SJTs) and selection centres (SCs) are also well regarded, but aptitude tests less so. Selectors endorse the use of interviews in general and in particular MMIs judging them to be fair, relevant and appropriate, with emerging evidence of similarly positive reactions to SCs. Aptitude tests and academic records were valued in decisions of whom to call to interview. Medical students prefer interviews based selection to cognitive aptitude tests. They are unconvinced about the transparency and veracity of written applications. Perceptions of organisational justice, which describe views of fairness in organisational processes, appear to be highly influential on stakeholders' views of the acceptability of selection methods. In particular procedural justice (perceived fairness of selection tools in terms of job relevance and characteristics of the test) and distributive justice (perceived fairness of selection outcomes in terms of equal opportunity and equity), appear to be important considerations when deciding on acceptability of selection methods. There were significant gaps with respect to both key stakeholder groups and the range of selection tools assessed.

CONCLUSIONS

Notwithstanding the observed limitations in the quality of research in this field, there appears to be broad concordance of views on the various selection methods, across the diverse stakeholders groups. This review highlights the need for better standards, more appropriate methodologies and for broadening the scope of stakeholder research.

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

BACKGROUND

Selection into specialty training is a high-stakes and resource-intensive process. While substantial literature exists on selection into medical schools, and there are individual studies in postgraduate settings, there seems to be paucity of evidence concerning selection systems and the utility of selection tools in postgraduate training environments.

AIM

To explore, analyze and synthesize the evidence related to selection into postgraduate medical specialty training.

METHOD

Core bibliographic databases including PubMed; Ovid Medline; Embase, CINAHL; ERIC and PsycINFO were searched, and a total of 2640 abstracts were retrieved. After removing duplicates and screening against the inclusion criteria, 202 full papers were coded, of which 116 were included.

RESULTS

Gaps in underlying selection frameworks were illuminated. Frameworks defined by locally derived selection criteria, and heavily weighed on academic parameters seem to be giving way to the evidencing of competency-based selection approaches in some settings. Regarding selection tools, we found favorable psychometric evidence for multiple mini-interviews, situational judgment tests and clinical problem-solving tests, although the bulk of evidence was mostly limited to the United Kingdom. The evidence around the robustness of curriculum vitae, letters of recommendation and personal statements was equivocal. The findings on the predictors of past performance were limited to academic criteria with paucity of long-term evaluations. The evidence around nonacademic criteria was inadequate to make an informed judgment.

CONCLUSIONS

While much has been gained in understanding the utility of individual selection methods, though the evidence around many of them is equivocal, the underlying theoretical and conceptual frameworks for designing holistic and equitable selection systems are yet to be developed.

CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scores

Typically, only a minority of applicants to health professional training are invited to interview. However, pre-interview measures of cognitive skills predict for national licensure scores (Gauer et al. in Med Educ Online 21 2016) and subsequently licensure scores predict for performance in practice (Tamblyn et al. in JAMA 288(23): 3019-3026, 2002; Tamblyn et al. in JAMA 298(9):993-1001, 2007). Assessment of personal and professional characteristics, with the same psychometric rigour of measures of cognitive abilities, are needed upstream in the selection to health profession training programs. To fill that need, Computer-based Assessment for Sampling Personal characteristics (CASPer)-an on-line, video-based screening test-was created. In this paper, we examine the correlation between CASPer and Canadian national licensure examination outcomes in 109 doctors who took CASPer at the time of selection to medical school. Specifically, CASPer scores were correlated against performance on cognitive and 'non-cognitive' subsections of both the Medical Council of Canada Qualifying Examination (MCCQE) Parts I (end of medical school) and Part II (18 months into specialty training). Unlike most national licensure exams, MCCQE has specific subcomponents examining personal/professional qualities, providing a unique opportunity for comparison. The results demonstrated moderate predictive validity of CASPer to national licensure outcomes of personal/professional characteristics three to six years after admission to medical school. These types of disattenuated correlations (r = 0.3-0.5) are not otherwise predicted by traditional screening measures. These data support the ability of a computer-based strategy to screen applicants in a feasible, reliable test, which has now demonstrated predictive validity, lending evidence of its validation for medical school applicant selection.

Multiple independent sampling within medical school admission interviewing: an "intermediate approach"

INTRODUCTION

Balancing reliability and resource limitations as well as recruitment activities during admission interviews is a challenge for many medical schools. The Modified Personal Interview (MPI) has been shown to have good psychometric properties while being resource efficient for specialized admission interviews. We describe implementation of an MPI adaptation integrating psychometric rigour alongside resourcing and recruitment goals for larger-scale medical school admission interviewing at the University of Toronto.

METHODS

The MPI was implemented during the 2013-2014 admission cycle. The MPI uses multiple independent sampling by having applicants interviewed in a circuit of four brief semi-structured interviews. Recruitment is reflected in a longer MPI interviewing time to foster a 'human touch'. Psychometric evaluation includes generalizability studies to examine inter-interview reliability and other major sources of error variance. We evaluated MPI impact upon applicant recruitment yield and resourcing.

RESULTS

MPI reliability is 0.56. MPI implementation maintained recruitment compared with previous year. MPI implementation required 160 interviewers for 600 applicants whereas for pre-MPI implementation 290 interviewers were required to interview 587 applicants. MPI score correlated with first year OSCE performance at 0.30 (p < 0.05).

DISCUSSION

MPI reliability is measured at 0.56 alongside enhanced resource utilization and maintenance of recruitment yield. This 'intermediate approach' may enable broader institutional uptake of integrated multiple independent sampling-based admission interviewing within institution-specific resourcing and recruitment goals.

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37

BACKGROUND

In the 11 years since its development at McMaster University Medical School, the multiple mini-interview (MMI) has become a popular selection tool. We aimed to systematically explore, analyze and synthesize the evidence regarding MMIs for selection to undergraduate health programs.

METHODS

The review protocol was peer-reviewed and prospectively registered with the Best Evidence Medical Education (BEME) collaboration. Thirteen databases were searched through 34 terms and their Boolean combinations. Seven key journals were hand-searched since 2004. The reference sections of all included studies were screened. Studies meeting the inclusion criteria were coded independently by two reviewers using a modified BEME coding sheet. Extracted data were synthesized through narrative synthesis.

RESULTS

A total of 4338 citations were identified and screened, resulting in 41 papers that met inclusion criteria. Thirty-two studies report data for selection to medicine, six for dentistry, three for veterinary medicine, one for pharmacy, one for nursing, one for rehabilitation, and one for health science. Five studies investigated selection to more than one profession. MMIs used for selection to undergraduate health programs appear to have reasonable feasibility, acceptability, validity, and reliability. Reliability is optimized by including 7-12 stations, each with one examiner. The evidence is stronger for face validity, with more research needed to explore content validity and predictive validity. In published studies, MMIs do not appear biased against applicants on the basis of age, gender, or socio-economic status. However, applicants of certain ethnic and social backgrounds did less well in a very small number of published studies. Performance on MMIs does not correlate strongly with other measures of noncognitive attributes, such as personality inventories and measures of emotional intelligence.

DISCUSSION

MMI does not automatically mean a more reliable selection process but it can do, if carefully designed. Effective MMIs require careful identification of the noncognitive attributes sought by the program and institution. Attention needs to be given to the number of stations, the blueprint and examiner training.

CONCLUSION

More work is required on MMIs as they may disadvantage groups of certain ethnic or social backgrounds. There is a compelling argument for multi-institutional studies to investigate areas such as the relationship of MMI content to curriculum domains, graduate outcomes, and social missions; relationships of applicants' performance on different MMIs; bias in selecting applicants of minority groups; and the long-term outcomes appropriate for studies of predictive validity.

How effective are selection methods in medical education? A systematic review

CONTEXT

Selection methods used by medical schools should reliably identify whether candidates are likely to be successful in medical training and ultimately become competent clinicians. However, there is little consensus regarding methods that reliably evaluate non-academic attributes, and longitudinal studies examining predictors of success after qualification are insufficient. This systematic review synthesises the extant research evidence on the relative strengths of various selection methods. We offer a research agenda and identify key considerations to inform policy and practice in the next 50 years.

METHODS

A formalised literature search was conducted for studies published between 1997 and 2015. A total of 194 articles met the inclusion criteria and were appraised in relation to: (i) selection method used; (ii) research question(s) addressed, and (iii) type of study design.

RESULTS

Eight selection methods were identified: (i) aptitude tests; (ii) academic records; (iii) personal statements; (iv) references; (v) situational judgement tests (SJTs); (vi) personality and emotional intelligence assessments; (vii) interviews and multiple mini-interviews (MMIs), and (viii) selection centres (SCs). The evidence relating to each method was reviewed against four evaluation criteria: effectiveness (reliability and validity); procedural issues; acceptability, and cost-effectiveness.

CONCLUSIONS

Evidence shows clearly that academic records, MMIs, aptitude tests, SJTs and SCs are more effective selection methods and are generally fairer than traditional interviews, references and personal statements. However, achievement in different selection methods may differentially predict performance at the various stages of medical education and clinical practice. Research into selection has been over-reliant on cross-sectional study designs and has tended to focus on reliability estimates rather than validity as an indicator of quality. A comprehensive framework of outcome criteria should be developed to allow researchers to interpret empirical evidence and compare selection methods fairly. This review highlights gaps in evidence for the combination of selection tools that is most effective and the weighting to be given to each tool.

Multiple mini-interviews: same concept, different approaches

OBJECTIVES

Increasing numbers of educational institutions in the medical field choose to replace their conventional admissions interviews with a multiple mini-interview (MMI) format because the latter has superior reliability values and reduces interviewer bias. As the MMI format can be adapted to the conditions of each institution, the question of under which circumstances an MMI is most expedient remains unresolved. This article systematically reviews the existing MMI literature to identify the aspects of MMI design that have impact on the reliability, validity and cost-efficiency of the format.

METHODS

Three electronic databases (OVID, PubMed, Web of Science) were searched for any publications in which MMIs and related approaches were discussed. Sixty-six publications were included in the analysis.

RESULTS

Forty studies reported reliability values. Generally, raising the number of stations has more impact on reliability than raising the number of raters per station. Other factors with positive influence include the exclusion of stations that are too easy, and the use of normative anchored rating scales or skills-based rater training. Data on criterion-related validities and analyses of dimensionality were found in 31 studies. Irrespective of design differences, the relationship between MMI results and academic measures is small to zero. The McMaster University MMI predicts in-programme and licensing examination performance. Construct validity analyses are mostly exploratory and their results are inconclusive. Seven publications gave information on required resources or provided suggestions on how to save costs. The most relevant cost factors that are additional to those of conventional interviews are the costs of station development and actor payments.

CONCLUSIONS

The MMI literature provides useful recommendations for reliable and cost-efficient MMI designs, but some important aspects have not yet been fully explored. More theory-driven research is needed concerning dimensionality and construct validity, the predictive validity of MMIs other than those of McMaster University, the comparison of station types, and a cost-efficient station development process.

Cutting costs of multiple mini-interviews - changes in reliability and efficiency of the Hamburg medical school admission test between two applications

BACKGROUND

Multiple mini-interviews (MMIs) are a valuable tool in medical school selection due to their broad acceptance and promising psychometric properties. With respect to the high expenses associated with this procedure, the discussion about its feasibility should be extended to cost-effectiveness issues.

METHODS

Following a pilot test of MMIs for medical school admission at Hamburg University in 2009 (HAM-Int), we took several actions to improve reliability and to reduce costs of the subsequent procedure in 2010. For both years, we assessed overall and inter-rater reliabilities based on multilevel analyses. Moreover, we provide a detailed specification of costs, as well as an extrapolation of the interrelation of costs, reliability, and the setup of the procedure.

RESULTS

The overall reliability of the initial 2009 HAM-Int procedure with twelve stations and an average of 2.33 raters per station was ICC=0.75. Following the improvement actions, in 2010 the ICC remained stable at 0.76, despite the reduction of the process to nine stations and 2.17 raters per station. Moreover, costs were cut down from $915 to $495 per candidate. With the 2010 modalities, we could have reached an ICC of 0.80 with 16 single rater stations ($570 per candidate).

CONCLUSIONS

With respect to reliability and cost-efficiency, it is generally worthwhile to invest in scoring, rater training and scenario development. Moreover, it is more beneficial to increase the number of stations instead of raters within stations. However, if we want to achieve more than 80 % reliability, a minor improvement is paid with skyrocketing costs.