The Multiple Mini-Interview (MMI) for student selection in health professions training - a systematic review

Construct validity of multiple mini interviews - Investigating the role of stations, skills, and raters using Bayesian G-theory

Background: One popular procedure in the medical student selection process are multiple mini-interviews (MMIs), which are designed to assess social skills (e.g., empathy) by means of brief interview and role-play stations. However, it remains unclear whether MMIs reliably measure desired social skills or rather general performance differences that do not depend on specific social skills. Here, we provide a detailed investigation into the construct validity of MMIs, including the identification and quantification of performance facets (social skill-specific performance, station-specific performance, general performance) and their relations with other selection measures.Methods: We used data from three MMI samples (N = 376 applicants, 144 raters) that included six interview and role-play stations and multiple assessed social skills.Results: Bayesian generalizability analyses show that, the largest amount of reliable MMI variance was accounted for by station-specific and general performance differences between applicants. Furthermore, there were low or no correlations with other selection measures.Discussion: Our findings suggest that MMI ratings are less social skill-specific than originally conceptualized and are due more to general performance differences (across and within-stations). Future research should focus on the development of skill-specific MMI stations and on behavioral analyses on the extents to which performance differences are based on desirable skills versus undesired aspects.

Rethinking the Admissions Interview: Piloting Multiple Mini-Interviews in a Graduate Psychology Program

Health profession programs routinely utilize traditional interviews in admissions as a means of assessing important non-academic characteristics (e.g., critical thinking, interpersonal skills, judgment) of candidates. However, the reliability and validity of traditional interviews is highly questionable. Given this, multiple health profession programs (e.g., medicine, nursing, pharmacy, physical therapy) have implemented multiple mini-interviews as an alternative for assessing non-academic characteristics. This paper describes the development and implementation of multiple mini-interviews in the admissions process for a doctoral clinical psychology program, one of the health professions yet to use multiple mini-interviews. This paper also examines the feasibility and acceptability of the multiple mini-interviews in this program. Results of a mixed-method survey of all 120 candidates who participated in admissions days are presented along with discussion of factors associated with satisfaction and dissatisfaction. Recommendations for program refinement and application to other graduate psychology programs for improved admissions processes are discussed.

Effect of Applicant Gender on Multiple Mini-Interview Admissions Score

PURPOSE

Admissions criteria for physician assistant (PA) schools vary considerably, but they often involve a combination of academic measures, noncognitive "humanistic" qualities, and mission-related items. To address noncognitive attributes, some PA schools have implemented the multiple mini-interview (MMI) format developed at McMaster University in Canada. This research project looked at differences in interview scores by gender over 3 consecutive admissions cycles at one PA program.

METHODS

Three years of pre-existing de-identified data gathered as part of the routine admissions process were analyzed retrospectively using SPSS-v25.

RESULTS

Data were available for all interviewees for 2015, 2016, and 2017 (N = 350 total). Between-group differences were not statistically significant by gender. Reliability (Cronbach's α) was 0.865 for academic scoring and 0.694 for MMI scoring.

DISCUSSION

Analysis of 3 years of admissions data from a single program did not show evidence for gender bias in MMI scores. Although this result is reassuring, it requires continued monitoring and replication.

Medical student selection criteria and junior doctor workplace performance

BACKGROUND

Medical school selection decisions have consequences beyond graduation. With generally low attrition rates, most medical students become junior doctors. Universities are therefore not just selecting students into a medical course; they are choosing the future medical workforce. Understanding the relationship between selection criteria and outcomes beyond the successful completion of a medical degree may inform approaches to student selection.

METHODS

A retrospective data matching study was conducted involving 39 interns employed by a South Australian local health network in 2017 who had originally entered Flinders University's medical school through a graduate pathway. Student selection data were matched with internship workplace performance scores (measured by supervising consultants' reports across five clinical rotations using a standardised assessment). Correlational analyses then examined associations between these two sets of variables.

RESULTS

An overall selection rank (equal thirds of weighted Grade Point Average from a prior degree, a panel interview, and a national selection test) was moderately associated with all performance measures, accounting for up to 25% of variance. Both weighted Grade Point Average and the interview had multiple and mostly moderate correlations with performance. An increasing number of years taken to complete the course was associated with poorer workplace performance across multiple outcome measures (moderate to strong negative associations with 31 to 62% of shared variance), as was age to a lesser extent (7 to 14%). The national selection test contributed a single and small relationship accounting for 5% of variance with one outcome measure.

CONCLUSIONS

Selection into medicine is a critical assessment given that most students become doctors. This study found multiple associations between selection scores and junior doctor workplace performance measures in the internship year, with weighted Grade Point Average from a prior degree and an interview appearing more important than the national selection test. Future collaborative research should map desired workplace performance outcomes to initial student selection and explore the impact of changes to selection which focus on assessment of these domains. The association between slower course progression and poorer workplace performance should also be examined.

The relationship between perfectionism and symptoms of depression in medical school applicants

BACKGROUND

The association between perfectionism and depression in the medical profession can ultimately influence physicians' performance negatively. In medical students, especially maladaptive perfectionism is connected with distress and lower academic performance. The expression of perfectionism and symptoms of depression at the time of medical school application is not known. Therefore, we explored perfectionism and symptoms of depression in participants of multiple mini-interviews for medical school admission and investigated possible differences between applicants who were eventually admitted or rejected.

METHODS

After the multiple mini-interviews admission procedure at Hamburg Medical School in August 2018, 146 applicants filled out a questionnaire including sociodemographic data and the following validated instruments: Multidimensional Perfectionism Scale by Hewitt and Flett (MPS-H), Multidimensional Perfectionism Scale by Frost (MPS-F), Patient Health Questionnaire (PHQ-9), Generalized Anxiety Disorder Scale (GAD-7), and a 10-item version of the Big Five Inventory (BFI-10). The two groups of admitted and rejected applicants were compared and the correlation between symptoms of depression and perfectionism further explored.

RESULTS

The admitted applicants were significantly more extrovert and had lower depression scores compared to the rejected applicants. In both groups, the composite scales of Adaptive Perfectionism (r = .21, p = .011) and Maladaptive Perfectionism (r = .43, p < .001) as well as their components correlated significantly with the PHQ-9 results. Maladaptive Perfectionism accounted for about 18% of variance in the PHQ-9 score.

CONCLUSIONS

Rejected medical school applicants who participated in a multiple mini-interviews admission procedure showed higher levels of depression symptoms than admitted applicants. The degree of depressive symptoms can be partly explained by Maladaptive Perfectionism scores. Since coping in medical school and in postgraduate medical education require robust mental health, perfectionism questionnaires could be an additional tool in medical school selection processes.

An Exploration of the Relationships Between Multiple Mini-Interview Scores and Personality Traits

Objective. To explore the relationship between pharmacy students' social and behavioral competencies assessed using multiple mini-interviews (MMIs) and their personality traits. Methods. A retrospective study was conducted using data from two Doctor of Pharmacy (PharmD) cohorts. Students' self-reported scores from an externally validated tool completed during their first professional year that assessed five personality traits (openness to experiences, conscientiousness, extraversion, agreeableness, and emotionality) were collected. Students' scores on empathy, integrity, adaptability, critical thinking, and collaboration were collected during MMIs conducted during the admissions process. Analysis of variance was used to test the homogeneity of the sample, and correlation analyses were used to determine the strength of relationships between variables. Results. When examining for associations between students' personality traits and their social and behavioral competencies measured during the MMIs, we found a few significant but very weak correlations with extraversion. Conclusion. Our results revealed weak to negligible correlations between scores on competencies measured during the MMIs and personality traits, potentially because the assessments measured different constructs.

Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review

Objectives

Multiple Mini Interviews (MMI) have been conducted across the globe in the student selection process, particularly in health profession education. This paper reported the validity evidence of MMI in various educational settings.

Methods

A literature search was carried out through Scopus, Science Direct, Google Scholar, PubMed, and EBSCOhost databases based on specific search terms. Each article was appraised based on title, abstract, and full text. The selected articles were critically appraised, and relevant information to support the validity of MMI in various educational settings was synthesized. This paper followed the PRISMA guideline to ensure consistency in reporting systematic review results.

Results

A majority of the studies were from Canada, with 41.54%, followed by the United Kingdom (25.39%), the United States (13.85%), and Australia (9.23%). The rest (9.24%) were from Germany, Ireland, the United Arab Emirates, Japan, Pakistan, Taiwan, and Malaysia. Moreover, most MMI stations ranged from seven to 12 with a duration of 10 min per station (including a 2-min gap between stations).

Conclusion

The results suggest that the content, response process, and internal structure of MMI were well supported by evidence; however, the relation and consequences of MMI to important outcome variables were inconsistently supported. The evidence shows that MMI is a non-biased, practical, feasible, reliable, and content-valid admission tool. However, further research on its impact on non-cognitive outcomes is required.

Assessing noncognitive domains of respiratory therapy applicants: Messick's framework appraisal of the multiple mini-interview

Educators who assess incoming applicants into a health professional training program are looking for a wide array of cognitive and noncognitive skills that best predict success in the program and as a future practicing professional. While aptitude tests generally measure cognitive skills, noncognitive constructs are more difficult to measure appropriately. The traditional method of measuring noncognitive constructs has been the panel interview. Panel interviews have been described as inconsistent in measuring noncognitive domains and consistently reported as unreliable and susceptible to bias. An alternate interview method used in many health professions schools is the multiple mini-interview (MMI) that was specifically designed to assess noncognitive domains in health professions education. This paper discusses the purpose of using the MMI, how the MMI is conducted, specific domains of focus for the MMI, and the feasibility of creating an MMI. Finally, the paper uses Messick's framework on validity to guide the consideration of the MMI.

Admissions processes in North American pharmacy schools: To what extent are characteristics of practice success measured?

INTRODUCTION

The goal of preparing competent pharmacy practitioners starts with the admissions process, whereby skills and abilities can be screened to identify those individuals with a strong potential for success. This study sought to describe the admissions processes of top pharmacy schools in North America and identify the extent to which characteristics associated with professional success are measured.

METHODS

An observational survey design was used to collect information to help characterize the admissions processes of 60 pharmacy schools. Online and centralized Pharmacy College Application Service data of admissions requirements were confirmed and further details were collected through a telephone or email cross-sectional survey. Data were analyzed using descriptive statistics.

RESULTS

Online data indicate that every school (n = 60) requires prerequisite courses; 53 (88%) consider grade point average (GPA) and 45 (75%) consider the Pharmacy College Admission Test (PCAT). Most schools utilize interviews during admissions (94%), with 67% using a personal interview, 20% the Multiple Mini Interview (MMI), and 7% a combination of both. Of 60 schools contacted, 42% completed the survey. Most schools (92%) assess critical thinking and professional motivation; the personality traits of agreeableness and conscientiousness were considered by 83%. Personal interview was the most frequently cited method for assessing applicant characteristics.

CONCLUSION

Current admissions practices in the top North American pharmacy schools are varied and continue to rely on traditional components including GPA, PCAT, and interviews. However, there is also a movement toward using standardized and validated measures of non-cognitive aspects potentially more predictive of success in pharmacy school and subsequent practice.

Does the sequence of rotations in Multiple Mini Interview stations influence the candidates' performance?

In Multiple Mini Interviews (MMIs), the candidates start out with different stations depending on the sequence of rotation they are assigned; thus, their perceived difficulty level and nervousness with their first station may differ. We examined whether such differences influenced the candidates' overall performance in the MMI. A 32-item questionnaire was developed and administered to candidates for admission interviews at a medical school regarding their perceptions of the MMI. Candidates' interview scores were also obtained and were compared across groups who differed in the sequence of rotation of MMI stations. Candidates felt nervous when they encountered the first station, which did not differ regardless of which station was their first encounter. Candidates were neutral regarding whether their first station was more difficult than the others and about whether its difficulty level influenced their overall performance in the test. Although candidates' perceived station difficulty differed across stations, there were no differences in their performance in each station between those it was their first station and those it was not. Candidates' total interview scores also did not differ across groups of different sequence of rotations. The MMI is a fair process as it does not give disadvantage to those who encounter a more difficult station than others for their first station.

How good are we at selecting students that meet our mission? Outcomes of the 2011 and 2012 entering classes selected by a locally developed multiple mini interview

BACKGROUND

Can a locally developed multiple mini interview (MMI) process lead to outcomes reflective of local values and mission?

METHODS

In 2017, the authors performed a retrospective analysis of the relationship of MMI with multiple-choice-based outcomes and non-multiple-choice-based outcomes, including clerkship competencies, OSCE, scholarship/service/leadership, academic honor society induction, peer and faculty humanism nominations, and overall performance at graduation for two entering classes with acceptance decisions based exclusively on a locally developed MMI.

RESULTS

There was no association between MMI and performance on multiple-choice-based examinations. For other outcomes, the effect size of MMI for OSCE was small and leadership/service and scholarship did not correlate with MMI score. For clerkship competencies, there was medium effect size for patient care, practice-based learning and improvement, interpersonal and communication skills, and cultural competence. Highest and lowest quartile MMI scorers were no different in academic honor society induction; however, top quartile MMI scorers received more humanism votes versus last quartile and were more likely rated outstanding or excellent graduates.

CONCLUSIONS

Local development of MMI and of admissions processes with sole reliance on MMI for final acceptance decisions will not affect academic preparation/medical school performance in multiple-choice-based assessments but can lead to locally desired attributes in students.

Do Multiple Mini-Interview and Traditional Interview Scores Differ in Their Associations With Acceptance Offers Within and Across Five California Medical Schools?

PURPOSE

In single-school studies, multiple mini-interview (MMI) and traditional interview (TI) scores are associated with acceptance offers. Unexamined is whether scores at one school are associated with acceptance at other schools; such analyses would mitigate single-school design biases and better estimate how well interviews capture desired applicant attributes. Using data from the 5 California Longitudinal Evaluation of Admissions Practices (CA-LEAP) medical schools, the authors examined associations of MMI and TI scores with acceptance offers within and across schools.

METHOD

The analyses included applicants who interviewed at ≥1 CA-LEAP school during the 2011-2013 admissions cycles. Three CA-LEAP schools employed TIs and 2 employed MMIs. Interview scores were standardized (z scores: mean = 0, SD = 1), and associations with acceptance offers were examined within and across schools in analyses stratified by school, adjusting for applicant sociodemographics, academic metrics, year, and total number of interviews.

RESULTS

Of 4,993 applicants interviewed, 428 (8.6%) interviewed at both MMI schools, 681 (13.6%) at ≥2 TI schools, and 1,327 (26.6%) at ≥1 MMI and ≥1 TI school. For each school, acceptance was associated with interview score at that school and also with interview scores at the other 4 schools. Cross-school associations of MMI versus TI scores with acceptance did not differ statistically.

CONCLUSIONS

Interview score at a given school was associated with acceptance at the other 4 schools, with no significant differences in associations for MMIs versus TIs. The findings suggest both MMIs and TIs captured attributes valued by admissions teams across CA-LEAP schools.

Admission testing for higher education: A multi-cohort study on the validity of high-fidelity curriculum-sampling tests

We investigated the validity of curriculum-sampling tests for admission to higher education in two studies. Curriculum-sampling tests mimic representative parts of an academic program to predict future academic achievement. In the first study, we investigated the predictive validity of a curriculum-sampling test for first year academic achievement across three cohorts of undergraduate psychology applicants and for academic achievement after three years in one cohort. We also studied the relationship between the test scores and enrollment decisions. In the second study, we examined the cognitive and noncognitive construct saturation of curriculum-sampling tests in a sample of psychology students. The curriculum-sampling tests showed high predictive validity for first year and third year academic achievement, mostly comparable to the predictive validity of high school GPA. In addition, curriculum-sampling test scores showed incremental validity over high school GPA. Applicants who scored low on the curriculum-sampling tests decided not to enroll in the program more often, indicating that curriculum-sampling admission tests may also promote self-selection. Contrary to expectations, the curriculum-sampling tests scores did not show any relationships with cognitive ability, but there were some indications for noncognitive saturation, mostly for perceived test competence. So, curriculum-sampling tests can serve as efficient admission tests that yield high predictive validity. Furthermore, when self-selection or student-program fit are major objectives of admission procedures, curriculum-sampling test may be preferred over or may be used in addition to high school GPA.

Validity evidence for the Hamburg multiple mini-interview

BACKGROUND

Multiple mini-interviews (MMI) become increasingly popular for the selection of medical students. In this work, we examine the validity evidence for the Hamburg MMI.

METHODS

We conducted three follow-up studies for the 2014 cohort of applicants to medical school over the course of two years. We calculated Spearman's rank correlation (ρ) between MMI results and (1) emotional intelligence measured by the Trait Emotional Intelligence Questionnaire (TEIQue-SF) and the Situational Test of Emotion Management (STEM), (2) supervisors' and practice team members' evaluations of psychosocial competencies and suitability for the medical profession after a one-week 1:1 teaching in a general practice (GP) and (3) objective structured clinical examination (OSCE) scores.

RESULTS

There were no significant correlations between MMI results and the TEIQue-SF (ρ = .07, p > .05) or the STEM (ρ = .05, p > .05). MMI results could significantly predict GP evaluations of psychosocial competencies (ρ = .32, p < .05) and suitability for the medical profession (ρ = .42, p < .01) as well as OSCE scores (ρ = .23, p < .05). The MMI remained a significant predictor of these outcomes in a robust regression model including gender and age as control variables.

CONCLUSIONS

Our findings suggest that MMIs can measure competencies that are relevant in a practical context. However, these competencies do not seem to be related to emotional intelligence as measured by self-report or situational judgement test.

Using the multiple mini interview as an assessment strategy within the first year of a health professions curriculum

BACKGROUND

The multiple mini-interview (MMI) is a common assessment strategy used in student selection. The MMI as an assessment strategy within a health professions curriculum, however, has not been previously studied. This study describes the integration of a 5-station MMI as part of an end-of-year capstone following the first year of a health professions curriculum. The goal of the capstone MMI was to assess professional competencies of students and to offer formative feedback to prepare students for their upcoming clinical practice experiences. The purpose of this study was to evaluate the psychometric properties of an MMI integrated into a health professions curriculum.

METHODS

Five capstone MMI stations were designed to each evaluate a single construct assessed by one rater. A principal component analysis (PCA) was used to evaluate the structure of the model and its ability to distinguish 5 separate constructs. A Multifaceted Rasch Measurement (MFRM) model assessed student performance and estimated the sources of measurement error attributed to 3 facets: student ability, rater stringency, and station difficulty. At the conclusion, students were surveyed about the capstone MMI experience.

RESULTS

The PCA confirmed the MMI reliably assessed 5 unique constructs and performance on each station was not strongly correlated with one another. The 3-facet MFRM analysis explained 58.79% of the total variance in student scores. Specifically, 29.98% of the variance reflected student ability, 20.25% reflected rater stringency, and 8.56% reflected station difficulty. Overall, the data demonstrated an acceptable fit to the MFRM model. The majority of students agreed the MMI allowed them to effectively demonstrate their communication (80.82%), critical thinking (78.77%), and collaboration skills (70.55%).

CONCLUSIONS

The MMI can be a valuable assessment strategy of professional competence within a health professions curriculum. These findings suggest the MMI is well-received by students and can produce reliable results. Future research should explore the impact of using the MMI as a strategy to monitor longitudinal competency development and inform feedback approaches.

Personal domains assessed in multiple mini interviews (MMIs) for healthcare student selection: A narrative synthesis systematic review

OBJECTIVES

To examine the personal domains multiple mini interviews (MMIs) are being designed to assess, explore how they were determined and contextualise such domains in current and future healthcare student selection processes DESIGN: A systematic review of empirical research reporting on MMI model design was conducted from database inception to November 2017.

DATA SOURCES

Twelve electronic bibliographic databases.

REVIEW METHODS

Evidence was extracted from original studies, and integrated in a narrative synthesis guided by the PRISMA statement for reporting systematic reviews. Personal domains were clustered into themes using a modified Delphi technique.

RESULTS

A total of 584 articles were screened. 65 unique studies (80 articles) matched our inclusion criteria of which seven were conducted within nursing/midwifery faculties. Six in 10 studies featured applicants to medical school. Across selection processes, we identified 32 personal domains assessed by MMIs, the most frequent being: communication skills (84%), teamwork/collaboration (70%), and ethical/moral judgement (65%). Domains capturing ability to cope with stressful situations (14%), make decisions (14%), and resolve conflict in the workplace (13%) featured in fewer than ten studies overall. Intra- and inter-disciplinary inconsistencies in domain profiles were noted, as well as differences by entry level. MMIs deployed in nursing and midwifery assessed compassion and decision-making more frequently than in all other disciplines. Own programme philosophy and professional body guidance were most frequently cited (~50%) as sources for personal domains; a blueprinting process was reported in only 8% of studies.

CONCLUSIONS

Nursing, midwifery and allied healthcare professionals should develop their theoretical frameworks for MMIs to ensure they are evidence-based and fit-for-purpose. We suggest a re-evaluation of domain priorities to ensure that students who are selected, not only have the capacity to offer the highest standards of care provision, but are able to maintain these standards when facing clinical practice and organisational pressures.

Qualitative analysis of MMI raters' scorings of medical school candidates: A matter of taste?

Recent years have seen leading medical educationalists repeatedly call for a paradigm shift in the way we view, value and use subjectivity in assessment. The argument is that subjective expert raters generally bring desired quality, not just noise, to performance evaluations. While several reviews document the psychometric qualities of the Multiple Mini-Interview (MMI), we currently lack qualitative studies examining what we can learn from MMI raters' subjectivity. The present qualitative study therefore investigates rater subjectivity or taste in MMI selection interview. Taste (Bourdieu 1984) is a practical sense, which makes it possible at a pre-reflective level to apply 'invisible' or 'tacit' categories of perception for distinguishing between good and bad. The study draws on data from explorative in-depth interviews with 12 purposefully selected MMI raters. We find that MMI raters spontaneously applied subjective criteria-their taste-enabling them to assess the candidates' interpersonal attributes and to predict the candidates' potential. In addition, MMI raters seemed to share a taste for certain qualities in the candidates (e.g. reflectivity, resilience, empathy, contact, alikeness, 'the good colleague'); hence, taste may be the result of an ongoing enculturation in medical education and healthcare systems. This study suggests that taste is an inevitable condition in the assessment of students' performance. The MMI set-up should therefore make room for MMI raters' taste and their connoisseurship, i.e. their ability to taste, to improve the quality of their assessment of medical school candidates.

Multiple mini interviews (MMI): A perspective on the first year of MMI at Brighton and Sussex Medical School

Since it was developed at McMaster University School of Medicine in 2002, multiple medical interviews (MMI) have become a popular and accepted method of assessing candidates applying for medical training. Unlike a traditional panel interview it resembles the objective structured clinical examination (OSCE) with numerous 8-10 min stations assessing different topics or scenarios pertinent to medical training and healthcare. This was the first year of implementing MMI at Brighton and Sussex Medical School and my first year of interviewing. I found the process engaging and thought provoking.

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

BACKGROUND

Selection into specialty training is a high-stakes and resource-intensive process. While substantial literature exists on selection into medical schools, and there are individual studies in postgraduate settings, there seems to be paucity of evidence concerning selection systems and the utility of selection tools in postgraduate training environments.

AIM

To explore, analyze and synthesize the evidence related to selection into postgraduate medical specialty training.

METHOD

Core bibliographic databases including PubMed; Ovid Medline; Embase, CINAHL; ERIC and PsycINFO were searched, and a total of 2640 abstracts were retrieved. After removing duplicates and screening against the inclusion criteria, 202 full papers were coded, of which 116 were included.

RESULTS

Gaps in underlying selection frameworks were illuminated. Frameworks defined by locally derived selection criteria, and heavily weighed on academic parameters seem to be giving way to the evidencing of competency-based selection approaches in some settings. Regarding selection tools, we found favorable psychometric evidence for multiple mini-interviews, situational judgment tests and clinical problem-solving tests, although the bulk of evidence was mostly limited to the United Kingdom. The evidence around the robustness of curriculum vitae, letters of recommendation and personal statements was equivocal. The findings on the predictors of past performance were limited to academic criteria with paucity of long-term evaluations. The evidence around nonacademic criteria was inadequate to make an informed judgment.

CONCLUSIONS

While much has been gained in understanding the utility of individual selection methods, though the evidence around many of them is equivocal, the underlying theoretical and conceptual frameworks for designing holistic and equitable selection systems are yet to be developed.

Reliability of Multiple Mini-Interviews and traditional interviews within and between institutions: a study of five California medical schools

BACKGROUND

Many medical schools use admissions Multiple Mini-Interviews (MMIs) rather than traditional interviews (TIs), partly because MMIs are thought to be more reliable. Yet prior studies examined single-school samples of candidates completing either an MMI or TI (not both). Using data from five California public medical schools, the authors examined the within- and between-school reliabilities of TIs and MMIs.

METHODS

The analyses included applicants interviewing at ≥1 of the five schools during 2011-2013. Three schools employed TIs (TI1, TI2, TI3) and two employed MMIs (MMI1, MMI2). Mixed linear models accounting for nesting of observations within applicants examined standardized TI and MMI scores (mean = 0, SD = 1), adjusting for applicant socio-demographics, academic metrics, year, number of interviews, and interview date.

RESULTS

A total of 4993 individuals (completing 7516 interviews [TI = 4137, MMI = 3379]) interviewed at ≥1 school; 428 (14.5%) interviewed at both MMI schools and 687 (20.2%) at more than one TI school. Within schools, inter-interviewer consistency was generally qualitatively lower for TI1, TI2, and TI3 (Pearson's r 0.07, 0.13, and 0.29, and Cronbach's α, 0.40, 0.44, and 0.61, respectively) than for MMI1 and MMI 2 (Cronbach's α 0.68 and 0.60, respectively). Between schools, the adjusted intraclass correlation coefficient was 0.27 (95% CI 0.20-0.35) for TIs and 0.47 (95% CI 0.41-0.54) for MMIs.

CONCLUSIONS

Within and between-school reliability was qualitatively higher for MMIs than for TIs. Nonetheless, TI reliabilities were higher than anticipated from prior literature, suggesting TIs may not need to be abandoned on reliability grounds if other factors favor their use.

Medical doctors' job specification analysis: A qualitative inquiry

Purpose: A qualitative inquiry was conducted to investigate the qualification requirements of medical doctors in different professional fields and from different perspectives. The inquiry was part of an empirical workplace analysis. Methods: Seventy-four structured interviews were conducted and analyzed to examine critical incidents and behaviors of medical doctors working in different professional fields (clinical theory, clinical practice, practitioner) and disciplines, and from three different perspectives (medical doctors, non-medical staff, and patients). In addition, the National Competency-based Catalogue of Learning Objectives for Medical Education (Nationaler Kompetenzbasierter Lernzielkatalog Medizin / NKLM) was used. Results: The results revealed eleven relevant competencies, which could be categorized into three superordinate competence clusters: interpersonal, work-related, and self-related. The perspectives of medical doctors and non-medical staff included all eleven competencies. However, the perspective of patients did not include one interpersonal and two self-related competencies. Nearly all of the critical behaviors mentioned are included in the NKLM. However, the NKLM also includes behaviors that were not mentioned in the interviews. Conclusions: The behavior-oriented interviews resulted in a requirement profile that is very similar in structure to other competency models in occupational contexts. Comparisons of the different perspectives predominantly revealed similarities. However, the patient perspective also revealed interesting differences compared to the perspectives of medical doctors and non-medical staff. The behavior-related results of the interviews can be directly used for the development of exercises in selection and personnel development contexts and for potential appraisals specific to different medical disciplines. In future steps, the results of this initial qualitative step are to be replicated and extended using quantitative studies and a representative sample. The main overall objective is the definition of relevant competencies both for the selection and development of medical students and for the design of potential appraisals as part of personnel development programs in different medical disciplines.

First experience with multiple mini interview for medical school admission in Brazil: Does it work in a different cultural scenario?

PURPOSE

Evaluation of non-cognitive skills never has been used in Brazil. This study aims to evaluate Multiple Mini Interviews (MMI) in the admission process of a School of Medicine in São Paulo, Brazil.

METHODS

The population of the study comprised 240 applicants summoned for the interviews, and 96 raters. MMI contributed to 25% of the applicants' final grade. Eight scenarios were created with the aim of evaluating different non-cognitive skills, each one had two raters. At the end of the interviews, the applicants and raters described their impressions about MMI. The reliability of the MMI was analyzed using the Theory of Generalization and Many-Facet Rasch Model (MFRM).

RESULTS

The G-study showed that the general reliability of the process was satisfactory (coefficient G = 0.743). The MMI grades were not affected by the raters' profile, time of interview (p = 0.715), and randomization group (p = 0.353). The Rasch analysis showed that there was no misfitting effects or inconsistent stations or raters. A significant majority of the applicants (98%) and all the raters believed MMIs were important in selecting students with a more adequate profile to study medicine.

CONCLUSIONS

The general reliability of the selection process was excellent, and it was fully accepted by the applicants and raters.

Student preparedness characteristics important for clinical learning: perspectives of supervisors from medicine, pharmacy and nursing

BACKGROUND

Student perspectives of clinical preparedness have been studied in the literature, but the viewpoint of supervisors is limited. Hence, the aim was to examine the perspective of supervisors on the characteristics of health professional students important for preparedness for clinical learning.

METHODS

This was a descriptive, questionnaire-based, cross-sectional study conducted at three higher education institutions in Malaysia. A previously published questionnaire with 62 characteristics was adopted with modifications after pre-testing. Descriptive analysis was completed for the demographic data. The sample was grouped based on health profession, clinical practice experience and teaching experience for further analysis. Non-parametric Kruskal-Wallis test was selected to evaluate differences in mean ranks to assess the null hypothesis that the medians are equal across the groups. Kruskal-Wallis post-hoc pair wise comparison was performed on samples with significant differences across samples.

RESULTS

The sample was comprised of 173 supervisors from medicine (55, 32%), pharmacy (84, 48%) and nursing (34, 20%). The majority (63%) of the supervisors were currently in professional practice. A high percentage (40%) of supervisors had less than 4 years of teaching experience. The highest theme ratings were for willingness (6.00) and professionalism (5.90). There was a significant difference (p < 0.05) in the medians, among medicine, pharmacy and nursing professional speciality for willingness (5.70, 6.00 and 6.00), professionalism (5.70, 5.90 and 6.15), communication and interaction (5.42, 5.67 and 6.00), personal attributes (5.42, 5.71 and 6.02) and the professional and interpersonal skills (5.50, 5.63 and 6.00) themes. Post-hoc analysis showed a significant difference (p < 0.05) between medicine and nursing groups in the willingness (5.70 and 6.00), professionalism (5.70 and 6.15) and personal attributes (5.42 and 6.02) themes. Supervisors who are currently in practice had given high ratings compared to other groups. There were no significant differences observed within groups with different level of teaching experiences.

CONCLUSIONS

All supervisors rated professionalism and willingness as the most important characteristics followed by personal attributes. Further strengthening learning opportunities related to these characteristics in the curriculum may improve the students' preparedness in clinical learning.

Reliability and validity of the multiple mini interview (MMI) for admissions to an occupational therapy professional program

Introduction

Admissions decisions are among the most consequential assessment points in educating occupational therapists. Suboptimal measures of academic achievement and personal characteristics are often used to select candidates. The purpose of the study is to investigate reliability and validity evidence supporting the use of the multiple mini interview for selecting candidates to an occupational therapy master’s level program.

Method

The study used a prospective cohort design. Data sources included both archival data and prospective candidate-generated data (multiple mini interview scores, applicant file review scores and academic performance), analyzed using psychometric and correlational approaches.

Results

Seventy-seven of 80 candidates participated in the prospective component of the study. Overall reliability of a 10-station multiple mini interview was .72. Inter-rater reliability was reasonable. Weak interrelationships were found between multiple mini interview scores and other application components.

Conclusion

Multiple mini interview scores may be probing different attributes than the other components of the application package. Our study findings support the use of the multiple mini interview for selecting applicants to a professional master’s program in occupational therapy; however, future work should continue to investigate the predictive validity of the multiple mini interview.

The effect of personality traits on undergraduate dental students' performance in multiple mini interviews

BACKGROUND

Multiple mini interviews (MMI) is a structured student selection method where applicants rotate around a series of stations that do not require previously learned knowledge. The Big Five Inventory (BFI) is a tool often used to measure personality traits commonly used in applicant selection.

OBJECTIVE

The aim of the current study was to assess the effect of personality traits using the BFI on undergraduate dental students' performance in MMI.

METHODS

This research was conducted at The Faculty of Dentistry, King Abdulaziz University, Jeddah, Saudi Arabia. All undergraduates applying to the dental program at King Abdulaziz University, Faculty of Dentistry (KAUFD) in two academic years [(2014-2015) and (2015-2016)] undertook the MMI held at KAUFD and completed the BFI inventory with a total number of 350 students. The MMI consisted of five stations while the BFI contained forty-four items. Data were analysed by SPSS version 20, using independent-samples t-test, Mann-Whitney U test, Spearman's rho correlation coefficient and Chi-square test.

RESULTS

female students scored higher than their male counterparts but scores were only significant in the empathy station (p<0.001) in the first year, and in both empathetic skills and team work abilities (p<0.05) in the second. BFI results indicate that male students scored higher on agreeableness (p<0.003) and lower on neuroticism (p<0.001) in the first year while female students scored higher on agreeableness and conscientiousness (p<0.001) in the second year. Students of private schools had higher total MMI scores compared to those of public schools (p<0.05).

CONCLUSION

MMI combined with the Big Five Inventory can be a useful tool in the admission process.

Are Female Applicants Rated Higher Than Males on the Multiple Mini-Interview? Findings From the University of Calgary

PURPOSE

The multiple mini-interview (MMI) improves reliability and validity of medical school interviews, and many schools have introduced this in an attempt to select individuals more skilled in communication, critical thinking, and ethical decision making. But every change in the admissions process may produce unintended consequences, such as changing intake demographics. In this article, two studies exploring gender differences in MMI ratings are reported.

METHOD

Cumulative meta-analysis was used to compare MMI ratings for female and male applicants to the University of Calgary Cumming School of Medicine between 2010 and 2014. Multiple linear regression was then performed to explore gender differences in MMI ratings after adjusting for other variables, followed by a sensitivity analysis of the impact of varying the weight given to MMI ratings on the odds of females being ranked in the top 150 applicants for 2014.

RESULTS

Females were rated higher than male applicants (standardized mean difference 0.21, 95% CI [0.11, 0.30], P < .001). After adjusting for other explanatory variables, there was a positive association between female applicant and MMI rating (regression coefficient 0.23 [0.14, 0.33], P < .001). Increasing weight assigned to MMI ratings was associated with increased odds of females being ranked in the top 150 applicants.

CONCLUSIONS

In this single-center study, females were rated higher than males on the MMI, and the odds of a female applicant being offered a position increased as more weight was given to MMI ratings. Further studies are needed to confirm and explain gender differences in MMI ratings.

The relationship between extracurricular activities assessed during selection and during medical school and performance

Several medical schools include candidates' extracurricular activities in their selection procedure, with promising results regarding their predictive value for achievement during the clinical years of medical school. This study aims to reveal whether the better achievement in clinical training of students selected on the basis of their extracurricular activities could be explained by persistent participation in extracurricular activities during medical school (msECAs). Lottery-admitted and selected student admission groups were compared on their participation in three types of msECAs: (1) research master, (2) important board positions or (3) additional degree programme. Logistic regression was used to measure the effect of admission group on participation in any msECA, adjusted for pre-university GPA. Two-way ANCOVA was used to examine the inter-relationships between admission group, participation in msECAs and clerkship grade, with pre-university GPA as covariate. Significantly more selected students compared to lottery-admitted students participated in any msECA. Participation in msECAs was associated with a higher pre-university GPA for lottery-admitted students only, whereas participation in msECAs was associated with higher clerkship grades for selected students only. These results suggest that persistent participation in extracurricular activities of selected students favours better clinical achievement, supporting the inclusion of ECAs in the selection procedure. More insight in the rationale behind participation in extracurricular activities during medical school may explain differences found between lottery-admitted and selected students.

Hidden item variance in multiple mini-interview scores

The extant literature has largely ignored a potentially significant source of variance in multiple mini-interview (MMI) scores by "hiding" the variance attributable to the sample of attributes used on an evaluation form. This potential source of hidden variance can be defined as rating items, which typically comprise an MMI evaluation form. Due to its multi-faceted, repeated measures format, reliability for the MMI has been primarily evaluated using generalizability (G) theory. A key assumption of G theory is that G studies model the most important sources of variance to which a researcher plans to generalize. Because G studies can only attribute variance to the facets that are modeled in a G study, failure to model potentially substantial sources of variation in MMI scores can result in biased estimates of variance components. This study demonstrates the implications of hiding the item facet in MMI studies when true item-level effects exist. An extensive Monte Carlo simulation study was conducted to examine whether a commonly used hidden item, person-by-station (p × s|i) G study design results in biased estimated variance components. Estimates from this hidden item model were compared with estimates from a more complete person-by-station-by-item (p × s × i) model. Results suggest that when true item-level effects exist, the hidden item model (p × s|i) will result in biased variance components which can bias reliability estimates; therefore, researchers should consider using the more complete person-by-station-by-item model (p × s × i) when evaluating generalizability of MMI scores.

Multiple mini-interview predictive validity for performance on a pharmacy licensing examination

CONTEXT

Predictive validity studies on the use of the multiple mini-interview (MMI) have been primarily in medicine.

OBJECTIVES

This study sought to determine the predictive validity of the MMI for performance within a pharmacy programme and on the Pharmacy Examining Board of Canada (PEBC) Qualifying Examination for licensure, and to compare the predictive validity of the MMI with that of pre-pharmacy grade point average (GPA) and Pharmacy College Admission Test (PCAT) score.

METHODS

Admissions data for 223 graduates of the pharmacy programme at the University of Toronto were matched to programme and licensure outcome measures. Multiple linear regression assessed the predictive ability of the MMI, pre-pharmacy GPA, PCAT and covariates for performance in final-year experiential rotations, cumulative GPA (cGPA) and PEBC-MCQ (multiple-choice question examination) and PEBC-OSCE (objective structured clinical examination) overall and subcomponent scores.

RESULTS

The PCAT, pre-pharmacy GPA and age significantly predicted the PEBC-MCQ overall score. The MMI was the only significant predictor of overall score on the PEBC-OSCE (β = 0.17, p = 0.02); it also predicted communication and performance subscores. Scores on the PCAT and female gender predicted the communication subscore. Pre-pharmacy GPA, age and female gender significantly predicted cGPA. The MMI was the only significant predictor of institutional/ambulatory rotation score (β = 0.26, p = 0.00).

CONCLUSIONS

The MMI, designed to measure non-academic attributes including communication, motivation and problem-solving skills, was the only admissions tool with significant predictive validity for performance on the PEBC-OSCE national pharmacy certification examination and in an institutional/ambulatory rotation. These findings, from a single cohort of undergraduates, provide the first report of the predictive validity of the MMI for performance on pharmacy licensure examinations and thereby strengthen the evidence for its use in health professions selection. Prior university academic performance significantly predicted cGPA and performance on the PEBC-MCQ. Performance on the PCAT also predicted PEBC-MCQ results.

Improving Otolaryngology Residency Selection Using Principles from Personnel Psychology

There has been a heightened focus on improving the resident selection process, particularly within highly competitive specialties. Previous research, however, has generally lacked a theoretical background, leading to inconsistent and biased results. Our recently published systematic review examining applicant characteristics and performance in residency can provide historical insight into the predictors (ie, constructs) and outcomes (ie, criteria) previously deemed pertinent by the otolaryngology community. Personnel psychology uses evidence-based practices to identify the most qualified candidates for employment using a variety of selection methods. Extensive research in this discipline has shown that integrity tests, structured interviews, work samples, and conscientiousness offer the greatest increase in validity when combined with general cognitive ability. Blending past research knowledge with the principles of personnel selection can provide the necessary foundation with which to engage in theory-driven, longitudinal studies on otolaryngology resident selection moving forward.

Reliability and acceptability of six station multiple mini-interviews: past-behavioural versus situational questions in postgraduate medical admission

BACKGROUND

The multiple mini-interview (MMI) is increasingly used for postgraduate medical admissions and in undergraduate settings. MMIs use mostly Situational Questions (SQs) rather than Past-Behavioural Questions (PBQs). A previous study of MMIs in this setting, where PBQs and SQs were asked in the same order, reported that the reliability of PBQs was non-inferior to SQs and that SQs were more acceptable to candidates. The order in which the questions are asked may affect reliability and acceptability of an MMI. This study investigated the reliability of an MMI using both PBQs and SQs, minimising question order bias. Acceptability of PBQs and SQs was also assessed.

METHODS

Forty candidates applying for a postgraduate medical admission for 2016-2017 were included; 24 examiners were used. The MMI consisted of six stations with one examiner per station; a PBQ and a SQ were asked at every station, and the order of questions was alternated between stations. Reliability was analysed for scores obtained for PBQs or SQs separately, and for both questions. A post-MMI survey was used to assess the acceptability of PBQs and SQs.

RESULTS

The generalisability (G) coefficients for PBQs only, SQs only, and both questions were 0.87, 0.96, and 0.80, respectively. Decision studies suggested that a four-station MMI would also be sufficiently reliable (G-coefficients 0.82 and 0.94 for PBQs and SQs, respectively). In total, 83% of participants were satisfied with the MMI. In terms of face validity, PBQs were more acceptable than SQs for candidates (p = 0.01), but equally acceptable for examiners (88% vs. 83% positive responses for PBQs vs. SQs; p = 0.377). Candidates preferred PBQs to SQs when asked to choose one, though this difference was not significant (p = 0.081); examiners showed a clear preference for PBQs (p = 0.007).

CONCLUSIONS

Reliability and acceptability of six-station MMI were good among 40 postgraduate candidates; modelling suggested that four stations would also be reliable. SQs were more reliable than PBQs. Candidates found PBQs more acceptable than SQs and examiners preferred PBQs when they had to choose between the two. Our findings suggest that it is better to ask both PBQs and SQs during an MMI to maximise acceptability.

The utility of multiple mini-interviews: experience of a medical school

PURPOSE

This paper aims to introduce the design of multiple mini-interviews (MMIs) as a tool to assess medical school applicants' attributes in alignment with the school's educational goals and to evaluate its utility.

METHODS

In this MMI, candidates rotated through six stations (10 minutes per station), in which specific interview topics were drawn by mapping the school's educational goals with the core competencies for entering medical students. We conducted post-MMI surveys of all of the interviewers and candidates to investigate their experiences of MMIs. The G-coefficient and interclass correlation were analyzed to investigate the reliability of this test. Additionally, the candidates' MMI scores were compared across different backgrounds and a univariate analysis was used to estimate correlations between their MMI scores and prior academic achievements.

RESULTS

A total of 164 candidates (a 98.8% response rate) and 19 interviewers (a 100% response rate) completed the surveys in the years 2014 and 2015. Both candidates and assessors showed positive responses to MMIs. The G-coefficient of MMI scores was 0.88 and the interclass correlation coefficients ranged from 0.58 to 0.75. The participants' total MMI scores did not differ across genders or undergraduate backgrounds and were not associated with age, undergraduate graduate point averages, nor the Korean medical school admission test (Medical Education Eligibility Test) scores.

CONCLUSION

Our study illustrates the utility of MMIs that utilize the institution's educational goals to identify attributes to be pursued in the admission interviews in alignment with the institution's core values. Future research is warranted of the predictive validity of this MMI.

Student characteristics, professional preferences, and admission to medical school

Objectives: A potential new avenue to address the shortage of country doctors is to change the rules for admission to medical school. We therefore study the link between high-school grade point average and prospective physicians' choice to work in rural areas. To further inform the discussion about rules for admission, we also study the effects of other predictors: a measure of students' attitudes towards risk; whether they waited for their place of study (Wartesemester); whether their parents worked as medical doctors; and whether they have some practical experience in the medical sector. Methods: We conducted two internet surveys in 2012 and 2014. In the first survey, the sample comprised 701 students and in the second, 474 students. In both surveys, we asked students for their regional preferences; in the 2014 survey, we additionally asked students for their first, second, and third preferences among a comprehensive set of specializations, including becoming a general practitioner. In both surveys, we asked students for basic demographic information (age and gender), their parents' occupation, a measure of subjective income expectations, a measure of risk attitudes, and their high-school grade point average (Abiturnote), and First National Boards Examination grade (Physikum). In 2014, we additionally asked for waiting periods (Wartesemester) as well as for prior professional experience in the health-care sector. Results: We find that three factors increase the probability of having a preference for working in a rural area significantly, holding constant all other influences: having a medical doctor among the parents, having worse grades in the high-school grade point average, and being more risk averse. Moreover, we find that those willing to work in the countryside have significantly more experience in the medical sector before admission to medical school. Discussion: Our results suggest that a change in the selection process for medical school may increase the supply of country doctors. Instead of focusing on the high-school grade point average, universities could even more intensely screen for study motivation through interviews or by taking into account students' background, extracurricular activities, or waiting periods.

Exploring the usefulness of interviewers' training before and after Multiple Mini Interviews (MMI) for undergraduate medical students' selection: Was it really helpful?

Objective:

To compare the change in interviewers’ perception of Multiple Mini Interviews (MMI) after MMI training and after actual MMI experience.

Methods:

Six sessions were conducted during two weeks (October 26, 2015- to November 6, 2015) to a total of 87 faculty members. The evaluation dealt with 13 items questionnaire for representation of assessors’ perception on 5 point rating scale. Assessors rated their perceptions to complete an anonymised questionnaire about rationale behind MMI, the process of MMI, and the use of scoring criteria (rubrics). In addition, assessors were also asked to rate their level of satisfaction on MMI process after training and after interviews. Wilcoxon Signed Ranks Test (two-tailed) was used to compare participant’s pre- and post-interview ratings.

Results:

With 81.6% response rate, the positive views of assessors about the MMI selection process and the use of scoring criteria (Rubric) to assess the candidate are not altered after experiencing a MMI selection day (p> 0.001). Assessors (87%) would prefer to be involved in the process of MMI in future.

Conclusion:

The outstanding consistency of assessors’ ratings before and after interview concluded that MMI training sessions were helpful in improving knowledge and skills about MMI process and candidates’ assessment criteria (rubrics).

Candidate Evaluation Using Targeted Construct Assessment in the Multiple Mini-Interview: A Multifaceted Rasch Model Analysis

Construct: A 7-station multiple mini-interview (MMI) circuit was implemented and assessed for 214 candidates rated by 37 interviewers (N = 1,498 ratings). The MMI stations were designed to assess 6 specific constructs (adaptability, empathy, integrity, critical thinking, teamwork [receiving instruction], teamwork [giving instruction]) and one open station about the candidate's interest in the school.

BACKGROUND

Despite the apparent benefits of the MMI, construct-irrelevant variance continues to be a topic of study. Refining the MMI to more effectively measure candidate ability is critical to improving our ability to identify and select candidates that are equipped for success within health professions education and the workforce.

APPROACH

Each station assessed a single construct and was rated by a single interviewer who was provided only the name of the candidate and no additional information about the candidate's background, application, or prior academic performance. All interviewers received online and in-person training in the fall prior to the MMI and the morning of the MMI. A 3-facet multifaceted Rasch measurement analysis was completed to determine interviewer severity, candidate ability, and MMI station difficulty and examine how the model performed overall (e.g., rating scale).

RESULTS

Altogether, the Rasch measures explained 62.84% of the variance in the ratings. Differences in candidate ability explained 45.28% of the variance in the data, whereas differences in interviewer severity explained 16.09% of the variance in the data. None of the interviewers had Infit or Outfit mean-square scores greater than 1.7, and only 2 (5.4%) had mean-square scores less than 0.5.

CONCLUSIONS

The data demonstrated acceptable fit to the multifaceted Rasch measurement model. This work is the first of its kind in pharmacy and provides insight into the development of an MMI that provides useful and meaningful candidate assessment ratings for institutional decision making.

A Hybrid Interview Model for Medical School Interviews: Combining Traditional and Multisampling Formats

PROBLEM

Most medical schools have either retained a traditional admissions interview or fully adopted an innovative, multisampling format (e.g., the multiple mini-interview) despite there being advantages and disadvantages associated with each format.

APPROACH

The University of Michigan Medical School (UMMS) sought to maximize the strengths associated with both interview formats after recognizing that combining the two approaches had the potential to capture additional, unique information about an applicant. In September 2014, the UMMS implemented a hybrid interview model with six, 6-minute short-form interviews-highly structured scenario-based encounters-and two, 30-minute semistructured long-form interviews. Five core skills were assessed across both interview formats.

OUTCOMES

Overall, applicants and admissions committee members reported favorable reactions to the hybrid model, supporting continued use of the model. The generalizability coefficients for the six-station short-form and the two-interview long-form formats were estimated to be 0.470 and 0.176, respectively. Different skills were more reliably assessed by different interview formats. Scores from each format seemed to be operating independently as evidenced through moderate to low correlations (r = 0.100-0.403) for the same skills measured across different interview formats; however, after correcting for attenuation, these correlations were much higher.

NEXT STEPS

This hybrid model will be revised and optimized to capture the skills most reliably assessed by each format. Future analysis will examine validity by determining whether short-form and long-form interview scores accurately measure the skills intended to be assessed. Additionally, data collected from both formats will be used to establish baselines for entering students' competencies.

Recruiting for values in healthcare: a preliminary review of the evidence

Displaying compassion, benevolence and respect, and preserving the dignity of patients are important for any healthcare professional to ensure the provision of high quality care and patient outcomes. This paper presents a structured search and thematic review of the research evidence relating to values-based recruitment within healthcare. Several different databases, journals and government reports were searched to retrieve studies relating to values-based recruitment published between 1998 and 2013, both in healthcare settings and other occupational contexts. There is limited published research related to values-based recruitment directly, so the available theoretical context of values is explored alongside an analysis of the impact of value congruence. The implications for the design of selection methods to measure values is explored beyond the scope of the initial literature search. Research suggests some selection methods may be appropriate for values-based recruitment, such as situational judgment tests (SJTs), structured interviews and multiple-mini interviews (MMIs). Personality tests were also identified as having the potential to compliment other methods (e.g. structured interviews), as part of a values-based recruitment agenda. Methods including personal statements, references and unstructured/'traditional' interviews were identified as inappropriate for values-based recruitment. Practical implications are discussed in the context of values-based recruitment in the healthcare context. Theoretical implications of our findings imply that prosocial implicit trait policies, which could be measured by selection tools such as SJTs and MMIs, may be linked to individuals' values via the behaviours individuals consider to be effective in given situations. Further research is required to state this conclusively however, and methods for values-based recruitment represent an exciting and relatively unchartered territory for further research.

The Multiple Mini-Interview as an Admission Tool for a PharmD Program Satellite Campus

Objective. To assess the multiple mini-interview (MMI) as an admission tool for a satellite campus. Methods. In 2013, the MMI was implemented as part of a new admissions model at the UNC Eshelman School of Pharmacy. From fall 2013 to spring 2015, 73 candidates were interviewed by 15 raters on the satellite campus in Asheville, North Carolina. A many-facet Rasch measurement (MFRM) with three facets was used to determine the variance in candidate ratings attributable to rater severity, candidate ability, and station difficulty. Candidates were surveyed to explore their perceptions of the MMI. Results. Rasch measures accounted for 48.3% of total variance in candidate scores. Rater severity accounted for 9.1% of the variance, and candidate ability accounted for 36.2% of the variance. Eighty percent of survey respondents (strongly) agreed that interviewers got to know them based on questions they answered.Conclusion. This study suggests that the MMI is a useful and valid tool for candidate selection at a satellite campus.

An Examination of Correlations between MMI scores and Pharmacy School GPA

Objective. To investigate the correlation of mean admission multiple mini-interview (MMI) scores with cumulative and overall GPA across didactic years 1-3 in the doctor of pharmacy (PharmD) curriculum. Methods. The mean admission MMI score and cumulative and overall GPA for first year (P1), second year (P2), and third year (P3) students in the PharmD curriculum was used to conduct a multiple regression analysis and to calculate a Pearson correlation coefficient. Statistical analysis was performed using SPSS, v20. Results. A negative correlation between mean admission MMI and overall GPA was observed for P1 students and P2 students. A significant positive correlation was observed between mean admission MMI and overall GPA for P3 students. Conclusion. A weak association between mean admission MMI score and GPA was observed, and the direction of the association between MMI and GPA was mixed across cohorts of students in the PharmD curriculum. Further research is needed that includes measurement of noncognitive outcomes and continued validation of the MMI for use in pharmacy school admissions.

Mini Surgical Simulation, Role Play, and Group and Behavioral Interviews in Resident Selection

BACKGROUND

A robust selection process is critical to residents' "cultural fit" and success in their program. Traditional selection methods have shortcomings.

OBJECTIVE

We describe a novel residency interview process for obstetrics-gynecology residents that incorporates behavioral, group, and surgical simulation multiple mini interviews (MMIs).

METHODS

In 2010, the Cedars-Sinai Medical Center obstetrics-gynecology residency program developed surgical simulation, role play, ethics group interview, and Accreditation Council for Graduate Medical Education competency-based behavioral interview stations.

RESULTS

From 2010 to 2012, a total of 199 applicants were interviewed, 62 ranked in the top 20, and 18 matched into the program. The MMI scores for interview stations were used in compiling our rank list and were found to adequately differentiate candidates. The MMI mean scores for role play, ethics interview, surgical simulation, and the behavioral interview for the top 20 ranked candidates were statistically significantly higher than those for other applicants. Standardized tests minimally correlated with various interview modalities. Applicants found the interview process acceptable. Implementing these MMI stations increased the total applicant interview time for the day by 15% (from 5.5 to 6.5 hours) and increased the face-to-face interview time from 2 to 4 hours. Approximately 42 hours of coordinator time was required for the yearly interview cycle.

CONCLUSIONS

A multifaceted interview process utilizing MMI, group interview, and surgical simulation MMI is feasible and acceptable. The approach may decrease subjectivity and reliance on traditional interview methods and facilitate the selection of "compatible" residents into the program.

[Predictive validity of selection process in medical school, a systematic review of the literature]

CONTEXT

The number of applicants to medical school exceeds the number of places available, a situation that questions the effectiveness and relevance of medical student assessment and selection.

OBJECTIVES

The objective of this review of the literature was to analyze in a systematic way the studies looking for an association between results of an admission procedure and a later evaluation of the characteristics of the students or the doctors that they became.

DOCUMENTARY SOURCES (KEY WORDS AND LANGUAGE)

MedLine, Web of Science and Cochrane Library. The articles written in English were selected. The keywords were: "medical school"; "student admissions"; "student selection"; "predictive validity"; "student performance"; "interview"; "MCAT" (Medical College Admission Test); "MMI" (Mini Mental Interview).

RESULTS

Of the 1116 listed publications, 22 were analyzed. Criteria assessed at the admission (standardized written tests, evaluation of personality and social skills, academic evaluation, demographic characteristics) were confronted with the criteria measured during the follow-up (academic or clinical evaluations). The previous academic results and the standardized written tests seem associated to academic success in medical school.

CONCLUSION

The actual selection methods used in our French medical universities are very different from those described in international literature. It is difficult to clearly define factors associated with success during student selection as there is a great diversity of cultures and methods involved.

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37

BACKGROUND

In the 11 years since its development at McMaster University Medical School, the multiple mini-interview (MMI) has become a popular selection tool. We aimed to systematically explore, analyze and synthesize the evidence regarding MMIs for selection to undergraduate health programs.

METHODS

The review protocol was peer-reviewed and prospectively registered with the Best Evidence Medical Education (BEME) collaboration. Thirteen databases were searched through 34 terms and their Boolean combinations. Seven key journals were hand-searched since 2004. The reference sections of all included studies were screened. Studies meeting the inclusion criteria were coded independently by two reviewers using a modified BEME coding sheet. Extracted data were synthesized through narrative synthesis.

RESULTS

A total of 4338 citations were identified and screened, resulting in 41 papers that met inclusion criteria. Thirty-two studies report data for selection to medicine, six for dentistry, three for veterinary medicine, one for pharmacy, one for nursing, one for rehabilitation, and one for health science. Five studies investigated selection to more than one profession. MMIs used for selection to undergraduate health programs appear to have reasonable feasibility, acceptability, validity, and reliability. Reliability is optimized by including 7-12 stations, each with one examiner. The evidence is stronger for face validity, with more research needed to explore content validity and predictive validity. In published studies, MMIs do not appear biased against applicants on the basis of age, gender, or socio-economic status. However, applicants of certain ethnic and social backgrounds did less well in a very small number of published studies. Performance on MMIs does not correlate strongly with other measures of noncognitive attributes, such as personality inventories and measures of emotional intelligence.

DISCUSSION

MMI does not automatically mean a more reliable selection process but it can do, if carefully designed. Effective MMIs require careful identification of the noncognitive attributes sought by the program and institution. Attention needs to be given to the number of stations, the blueprint and examiner training.

CONCLUSION

More work is required on MMIs as they may disadvantage groups of certain ethnic or social backgrounds. There is a compelling argument for multi-institutional studies to investigate areas such as the relationship of MMI content to curriculum domains, graduate outcomes, and social missions; relationships of applicants' performance on different MMIs; bias in selecting applicants of minority groups; and the long-term outcomes appropriate for studies of predictive validity.

What does the multiple mini interview have to offer over the panel interview?

INTRODUCTION

This paper compares the panel interview (PI) performance with the multiple mini interview (MMI) performance and indication of behavioural concerns of a sample of medical school applicants. The acceptability of the MMI was also assessed.

MATERIALS AND METHODS

All applicants shortlisted for a PI were invited to an MMI. Applicants attended a 30-min PI with two faculty interviewers followed by an MMI consisting of ten 8-min stations. Applicants were assessed on their performance at each MMI station by one faculty. The interviewer also indicated if they perceived the applicant to be a concern. Finally, applicants completed an acceptability questionnaire.

RESULTS

From the analysis of 133 (75.1%) completed MMI scoresheets, the MMI scores correlated statistically significantly with the PI scores (r=0.438, p=0.001). Both were not statistically associated with sex, age, race, or pre-university academic ability to any significance. Applicants assessed as a concern at two or more stations performed statistically significantly less well at the MMI when compared with those who were assessed as a concern at one station or none at all. However, there was no association with PI performance. Acceptability scores were generally high, and comparison of mean scores for each of the acceptability questionnaire items did not show statistically significant differences between sex and race categories.

CONCLUSIONS

Although PI and MMI performances are correlated, the MMI may have the added advantage of more objectively generating multiple impressions of the applicant's interpersonal skill, thoughtfulness, and general demeanour. Results of the present study indicated that the MMI is acceptable in a multicultural context.