An exploration of the relationship between emotional intelligence (EI) and the Multiple Mini-Interview (MMI)

Interactional competencies in medical student admission- what makes a "good medical doctor"?

Doctors' interactional competencies play a crucial role in patient satisfaction, well-being, and compliance. Accordingly, it is in medical schools' interest to select candidates with strong interactional abilities. While Multiple Mini Interviews (MMIs) provide a useful context to assess such abilities, the evaluation of candidate performance during MMIs is not always based on a solid theoretical framework. The newly developed selection procedure "Interactional Competencies - Medical Doctors (IC-MD)" uses an MMI circuit with five simulation patient scenarios and is rated based on the theoretically and empirically grounded construct of emotional availability. A first validation study with N = 70 first-semester medical students took place in 2021. In terms of convergent validity, IC-MD ratings showed strong correlations with simulation patients' satisfaction with the encounter (r =.57) but no association with emotional intelligence measures. IC-MD ratings were not related to high school performance or a cognitive student aptitude test, indicating divergent validity. Inter-rater reliability (ICC = 0.63) and generalizability (Eρ2 = 0.64) were satisfactory. The IC-MD proved to be fair regarding participants' age and gender. Participants with prior work experience in healthcare outperformed those without such experience. Participant acceptance of the procedure were good. The IC-MD is a promising selection procedure capable of assessing interactional competencies relevant to the medical setting. Measures of interactional competencies can complement the use of cognitive selection criteria in medical student admission. The predictive validity of the IC-MD needs to be addressed in future studies.

Evaluating non-cognitive skills in medical school applicants

Medical school candidates must have both the cognitive and professional competencies required to become good physicians. In this commentary, we outline the evidence and outcomes associated with the implementation of these selection methodologies and evaluate their ability to assess non-cognitive skills.

Does Emotional Intelligence at medical school admission predict future licensing examination performance?

BACKGROUND

Medical school admissions committees are seeking alternatives to traditional academic measures when selecting students; one potential measure being emotional intelligence (EI). If EI is to be used as an admissions criterion, it should predict future performance. The purpose of this study is to determine if EI scores at admissions predicts performance on a medical licensure examination.

METHODS

All medical school applicants to the University of Ottawa in 2006 and 2007 were invited to complete the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT v2.0) after their interview. Students were tracked through medical school into licensure and EI scores were correlated to their scores on the Medical Council of Canada Qualifying Examination (MCCQE) attempted between 2010 and 2014.

RESULTS

The correlation between the MSCEIT and the MCCQE Part I was r (200) = .01 p =. 90 The covariates of age and gender accounted for a significant amount of variance in MCCQE Part I scores (R ² = .10, p<.001, n=202) but the addition of the MSCEIT scores was not statistically significant (R ² change = .002, p=.56). The correlation between the MSCEIT and the MCCQE Part II was r(197) = .06, p = .41. The covariates of age and gender accounted for some variance in MCCQE Part II scores (R ² = .05, p = .007, n=199) but the addition of the MSCEIT did not (R ² change = .002 p =.55).

CONCLUSION

The low correlations between EI and licensure scores replicates other studies that have found weak correlations between EI scores and tests administered at admissions and during medical school. These results suggest caution if one were to use EI as part of their admissions process.

Construct validity of multiple mini interviews - Investigating the role of stations, skills, and raters using Bayesian G-theory

Background: One popular procedure in the medical student selection process are multiple mini-interviews (MMIs), which are designed to assess social skills (e.g., empathy) by means of brief interview and role-play stations. However, it remains unclear whether MMIs reliably measure desired social skills or rather general performance differences that do not depend on specific social skills. Here, we provide a detailed investigation into the construct validity of MMIs, including the identification and quantification of performance facets (social skill-specific performance, station-specific performance, general performance) and their relations with other selection measures.Methods: We used data from three MMI samples (N = 376 applicants, 144 raters) that included six interview and role-play stations and multiple assessed social skills.Results: Bayesian generalizability analyses show that, the largest amount of reliable MMI variance was accounted for by station-specific and general performance differences between applicants. Furthermore, there were low or no correlations with other selection measures.Discussion: Our findings suggest that MMI ratings are less social skill-specific than originally conceptualized and are due more to general performance differences (across and within-stations). Future research should focus on the development of skill-specific MMI stations and on behavioral analyses on the extents to which performance differences are based on desirable skills versus undesired aspects.

Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review

Objectives

Multiple Mini Interviews (MMI) have been conducted across the globe in the student selection process, particularly in health profession education. This paper reported the validity evidence of MMI in various educational settings.

Methods

A literature search was carried out through Scopus, Science Direct, Google Scholar, PubMed, and EBSCOhost databases based on specific search terms. Each article was appraised based on title, abstract, and full text. The selected articles were critically appraised, and relevant information to support the validity of MMI in various educational settings was synthesized. This paper followed the PRISMA guideline to ensure consistency in reporting systematic review results.

Results

A majority of the studies were from Canada, with 41.54%, followed by the United Kingdom (25.39%), the United States (13.85%), and Australia (9.23%). The rest (9.24%) were from Germany, Ireland, the United Arab Emirates, Japan, Pakistan, Taiwan, and Malaysia. Moreover, most MMI stations ranged from seven to 12 with a duration of 10 min per station (including a 2-min gap between stations).

Conclusion

The results suggest that the content, response process, and internal structure of MMI were well supported by evidence; however, the relation and consequences of MMI to important outcome variables were inconsistently supported. The evidence shows that MMI is a non-biased, practical, feasible, reliable, and content-valid admission tool. However, further research on its impact on non-cognitive outcomes is required.

Validity evidence for the Hamburg multiple mini-interview

BACKGROUND

Multiple mini-interviews (MMI) become increasingly popular for the selection of medical students. In this work, we examine the validity evidence for the Hamburg MMI.

METHODS

We conducted three follow-up studies for the 2014 cohort of applicants to medical school over the course of two years. We calculated Spearman's rank correlation (ρ) between MMI results and (1) emotional intelligence measured by the Trait Emotional Intelligence Questionnaire (TEIQue-SF) and the Situational Test of Emotion Management (STEM), (2) supervisors' and practice team members' evaluations of psychosocial competencies and suitability for the medical profession after a one-week 1:1 teaching in a general practice (GP) and (3) objective structured clinical examination (OSCE) scores.

RESULTS

There were no significant correlations between MMI results and the TEIQue-SF (ρ = .07, p > .05) or the STEM (ρ = .05, p > .05). MMI results could significantly predict GP evaluations of psychosocial competencies (ρ = .32, p < .05) and suitability for the medical profession (ρ = .42, p < .01) as well as OSCE scores (ρ = .23, p < .05). The MMI remained a significant predictor of these outcomes in a robust regression model including gender and age as control variables.

CONCLUSIONS

Our findings suggest that MMIs can measure competencies that are relevant in a practical context. However, these competencies do not seem to be related to emotional intelligence as measured by self-report or situational judgement test.

The effect of personality traits on undergraduate dental students' performance in multiple mini interviews

BACKGROUND

Multiple mini interviews (MMI) is a structured student selection method where applicants rotate around a series of stations that do not require previously learned knowledge. The Big Five Inventory (BFI) is a tool often used to measure personality traits commonly used in applicant selection.

OBJECTIVE

The aim of the current study was to assess the effect of personality traits using the BFI on undergraduate dental students' performance in MMI.

METHODS

This research was conducted at The Faculty of Dentistry, King Abdulaziz University, Jeddah, Saudi Arabia. All undergraduates applying to the dental program at King Abdulaziz University, Faculty of Dentistry (KAUFD) in two academic years [(2014-2015) and (2015-2016)] undertook the MMI held at KAUFD and completed the BFI inventory with a total number of 350 students. The MMI consisted of five stations while the BFI contained forty-four items. Data were analysed by SPSS version 20, using independent-samples t-test, Mann-Whitney U test, Spearman's rho correlation coefficient and Chi-square test.

RESULTS

female students scored higher than their male counterparts but scores were only significant in the empathy station (p<0.001) in the first year, and in both empathetic skills and team work abilities (p<0.05) in the second. BFI results indicate that male students scored higher on agreeableness (p<0.003) and lower on neuroticism (p<0.001) in the first year while female students scored higher on agreeableness and conscientiousness (p<0.001) in the second year. Students of private schools had higher total MMI scores compared to those of public schools (p<0.05).

CONCLUSION

MMI combined with the Big Five Inventory can be a useful tool in the admission process.

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37

BACKGROUND

In the 11 years since its development at McMaster University Medical School, the multiple mini-interview (MMI) has become a popular selection tool. We aimed to systematically explore, analyze and synthesize the evidence regarding MMIs for selection to undergraduate health programs.

METHODS

The review protocol was peer-reviewed and prospectively registered with the Best Evidence Medical Education (BEME) collaboration. Thirteen databases were searched through 34 terms and their Boolean combinations. Seven key journals were hand-searched since 2004. The reference sections of all included studies were screened. Studies meeting the inclusion criteria were coded independently by two reviewers using a modified BEME coding sheet. Extracted data were synthesized through narrative synthesis.

RESULTS

A total of 4338 citations were identified and screened, resulting in 41 papers that met inclusion criteria. Thirty-two studies report data for selection to medicine, six for dentistry, three for veterinary medicine, one for pharmacy, one for nursing, one for rehabilitation, and one for health science. Five studies investigated selection to more than one profession. MMIs used for selection to undergraduate health programs appear to have reasonable feasibility, acceptability, validity, and reliability. Reliability is optimized by including 7-12 stations, each with one examiner. The evidence is stronger for face validity, with more research needed to explore content validity and predictive validity. In published studies, MMIs do not appear biased against applicants on the basis of age, gender, or socio-economic status. However, applicants of certain ethnic and social backgrounds did less well in a very small number of published studies. Performance on MMIs does not correlate strongly with other measures of noncognitive attributes, such as personality inventories and measures of emotional intelligence.

DISCUSSION

MMI does not automatically mean a more reliable selection process but it can do, if carefully designed. Effective MMIs require careful identification of the noncognitive attributes sought by the program and institution. Attention needs to be given to the number of stations, the blueprint and examiner training.

CONCLUSION

More work is required on MMIs as they may disadvantage groups of certain ethnic or social backgrounds. There is a compelling argument for multi-institutional studies to investigate areas such as the relationship of MMI content to curriculum domains, graduate outcomes, and social missions; relationships of applicants' performance on different MMIs; bias in selecting applicants of minority groups; and the long-term outcomes appropriate for studies of predictive validity.

Multiple mini-interviews: same concept, different approaches

OBJECTIVES

Increasing numbers of educational institutions in the medical field choose to replace their conventional admissions interviews with a multiple mini-interview (MMI) format because the latter has superior reliability values and reduces interviewer bias. As the MMI format can be adapted to the conditions of each institution, the question of under which circumstances an MMI is most expedient remains unresolved. This article systematically reviews the existing MMI literature to identify the aspects of MMI design that have impact on the reliability, validity and cost-efficiency of the format.

METHODS

Three electronic databases (OVID, PubMed, Web of Science) were searched for any publications in which MMIs and related approaches were discussed. Sixty-six publications were included in the analysis.

RESULTS

Forty studies reported reliability values. Generally, raising the number of stations has more impact on reliability than raising the number of raters per station. Other factors with positive influence include the exclusion of stations that are too easy, and the use of normative anchored rating scales or skills-based rater training. Data on criterion-related validities and analyses of dimensionality were found in 31 studies. Irrespective of design differences, the relationship between MMI results and academic measures is small to zero. The McMaster University MMI predicts in-programme and licensing examination performance. Construct validity analyses are mostly exploratory and their results are inconclusive. Seven publications gave information on required resources or provided suggestions on how to save costs. The most relevant cost factors that are additional to those of conventional interviews are the costs of station development and actor payments.

CONCLUSIONS

The MMI literature provides useful recommendations for reliable and cost-efficient MMI designs, but some important aspects have not yet been fully explored. More theory-driven research is needed concerning dimensionality and construct validity, the predictive validity of MMIs other than those of McMaster University, the comparison of station types, and a cost-efficient station development process.

The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty training

BACKGROUND

Entry into specialty training was determined by a National Assessment Centre (NAC) approach using a combination of a behavioural Multiple-Mini-Interview (MMI) and a written Situational Judgement Test (SJT). We wanted to know if interviewers could make reliable and valid decisions about the non-cognitive characteristics of candidates with the purpose of selecting them into general practice specialty training using the MMI. Second, we explored the concurrent validity of the MMI with the SJT.

METHODS

A variance components analysis estimated the reliability and sources of measurement error. Further modelling estimated the optimal configurations for future MMI iterations. We calculated the relationship of the MMI with the SJT.

RESULTS

Data were available from 1382 candidates, 254 interviewers, six MMI questions, five alternate forms of a 50-item SJT, and 11 assessment centres. For a single MMI question and one assessor, 28% of the variance between scores was due to candidate-to-candidate variation. Interviewer subjectivity, in particular the varying views that interviewer had for particular candidates accounted for 40% of the variance in scores. The generalisability co-efficient for a six question MMI was 0.7; to achieve 0.8 would require ten questions. A disattenuated correlation with the SJT (r = 0.35), and in particular a raw score correlation with the subdomain related to clinical knowledge (r = 0.25) demonstrated evidence for construct and concurrent validity. Less than two per cent of candidates would have failed the MMI.

CONCLUSION

The MMI is a moderately reliable method of assessment in the context of a National Assessment Centre approach. The largest source of error relates to aspects of interviewer subjectivity, suggesting enhanced interviewer training would be beneficial. MMIs need to be sufficiently long for precise comparison for ranking purposes. In order to justify long term sustainable use of the MMI in a postgraduate assessment centre approach, more theoretical work is required to understand how written and performance based test of non-cognitive attributes can be combined, in a way that achieves acceptable generalizability, and has validity.

The Multiple Mini-Interview (MMI) for student selection in health professions training - a systematic review

BACKGROUND

The Multiple Mini-Interview (MMI) has been used increasingly for selection of students to health professions programmes.

OBJECTIVES

This paper reports on the evidence base for the feasibility, acceptability, reliability and validity of the MMI.

DATA SOURCES

CINAHL and MEDLINE STUDY ELIGIBILITY CRITERIA: All studies testing the MMI on applicants to health professions training.

STUDY APPRAISAL AND SYNTHESIS METHODS

Each paper was appraised by two reviewers. Narrative summary findings on feasibility, acceptability, reliability and validity are presented.

RESULTS

Of the 64 citations identified, 30 were selected for review. The modal MMI consisted of 10 stations, each lasting eight minutes and assessed by one interviewer. The MMI was feasible, i.e. did not require more examiners, did not cost more, and interviews were completed over a short period of time. It was acceptable, i.e. fair, transparent, free from gender, cultural and socio-economic bias, and did not favour applicants with previous coaching. Its reliability was reported to be moderate to high, with Cronbach's alpha = 0.69-0.98 and G = 0.55-0.72. MMI scores did not correlate to traditional admission tools scores, were not associated with pre-entry academic qualifications, were the best predictor for OSCE performance and statistically predictive of subsequent performance at medical council examinations.

CONCLUSIONS

The MMI is reliable, acceptable and feasible. The evidence base for its validity against future medical council exams is growing with reports from longitudinal investigations. However, further research is needed for its acceptability in different cultural context and validity against future clinical behaviours.

Internet-based multiple mini-interviews for candidate selection for graduate entry programmes

CONTEXT

Multiple mini-interviews (MMIs) have been used by The University of Sydney graduate medical and dental programmes since 2006. In 2011, interviews with international candidates were conducted using Skype (iMMI), whereas interviews with local candidates were conducted in person. We determined whether the MMI scores derived from both methods were comparable. We describe the feasibility, acceptability and cost-effectiveness of the iMMI.

METHODS

We compared 2011 international student internet-based iMMI results with data from 2009 international student MMIs and 2011 local student MMIs. Analyses of variance (anovas) were used to investigate equivalence of the two formats by exploring whether the medium of interviewing resulted in significantly different mean scores and variance for the in-person MMI and the iMMI. Acceptability of the process was informed by feedback surveys from interviewers and candidates, and cost savings were estimated.

RESULTS

No significant difference was found between the 2011 iMMI scores for international candidates and MMI scores in 2009 (p > 0.05). There was no significant difference between the MMI scores for local and international candidates in 2011 (p > 0.05); the MMI scores for international candidates had greater variation (p < 0.01). Using generalisability theory, the reliability of the nine-question iMMI was 0.76 and for the MMI was 0.70. Delivery of the iMMI occurred smoothly and candidates and interviewers gave positive feedback on its format and delivery. Cost savings have been estimated to be over AU$50 000, representing an 84% saving.

CONCLUSIONS

We believe this is the first study reporting an internet-based MMI for a high stakes interview. We have shown that interviewers were able to make valid and reliable decisions about candidates through the iMMI in a process that was acceptable to participants, producing comparable results to the in-person MMI with a saving of resources. The slightly wider variance in iMMI scores warrants further investigation.