Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review

Investigating the internal structure of multiple mini interviews-A perspective from Pakistan

BACKGROUND

Healthcare professionals require many personal attributes in addition to cognitive abilities and psychomotor skills for competent practice. Multiple Mini- Interviews are being employed globally to assess personality attributes of candidates for selection in health professions education at all level of entry; these attributes are namely, communication skills, critical thinking, honesty, responsibility, health advocacy, empathy and sanctity of life. Considering the high stakes involved for students, faculty, institutions and the society, rigorous quality assurance mechanisms similar to those used for student assessment must be employed for student selection, throughout the continuum of medical education. It is a difficult undertaking as these psychological constructs are difficult to define and measure. Though considered to yield reliable and valid scores, studies providing multiple evidences of internal structure especially dimensionality of Multiple Mini-Interviews are sparse giving rise to questions if they are measuring a single or multiple constructs and even if they are measuring what they are purported to be measuring.

OBJECTIVE

The main objective is to provide statistical support of the multi-dimensional nature of our Multiple Mini Interviews, hypothesized a-priori, through CFA. Another objective is to provide multiple evidences for the internal structure. Our study highlights the link between content and internal structure evidences of the constructs, thus establishing that our Multiple Mini Interviews measure what they were intended to measure.

METHOD

After securing permission from the Institutional review board, an a-priori seven factor-model was hypothesized based on the attributes considered most essential for the graduating student of the institution. After operationally defining the attributes through extensive literature search, scenarios were constructed to assess them. A 5-point rating scale was used to rate each item on the station. A total 259 students participated in the multiple mini interviews over a period of three days. A training workshop had been arranged for the participating faculty.

RESULTS

The reliability coefficient using Cronbach's alpha were calculated (range from 0.73 to 0.94), Standard Error of Measurement (ranged from 0.80 to1.64), and item to station-total correlation ranged from 0.43-0.50 to 0.75-0.83. Inter-station correlation was also determined. Confirmatory factor analysis endorsed the results of Exploratory factor analysis in the study revealing a seven model fit with multiple indices of Goodness-of-fit statistics such as Root mean square error of approximation (RMSEA) value 0.05, Standardized root mean square residual (SRMR) value with less than 0.08. All these indices showed that model fit is good. The Confirmatory factor analysis confirmed the multi-dimensional nature of our MMIs and also confirmed that our stations measured the attributes that they were supposed to measure.

CONCLUSION

This study adds to the validity evidence of Multiple Mini-Interviews, in selection of candidates, with required personality traits for healthcare profession. It provides the evidence for the multi-dimensional structure of Multiple Mini interviews administered with multiple evidences for its internal structure and demonstrates the independence of different constructs being measured.

Medical students' perception of their 'distance travelled' in medical school applications

INTRODUCTION

Within medical school's holistic review of applicants includes a review of their distance travelled to get to this point in their education. The AAMC defines distance travelled (DT) as, 'any obstacles or hardships you've overcome to get to this point in your education or any life challenges you've faced and conquered'. What medical students consider as their distance travelled has not been explored. The authors sought to identify the factors medical students perceive are important for medical school admissions to consider when assessing someone's 'distance travelled' by asking current medical students to share their DT experiences along with the barriers and facilitators they encountered on their medical school journey.

METHODS

The authors conducted semi-structured interviews with US medical students through purposeful sampling methods. The social-ecological model framework was used to develop questions to elicit participants' experiences that contributed to their distance travelled. Interviews were conducted in 2021 and ranged from 60-75 minutes. Transcribed interviews were qualitatively analysed using interpretive description.

RESULTS

A total of 31 medical students from seven medical schools were included in the study. Overall, participants defined distance travelled as an applicant's hardships (e.g. being the primary caregiver for a family member) and privileges (e.g. having physician parents) they experienced. Three major themes were identified: (1) individual-level characteristics and factors, (2) interpersonal relationships and (3) aspects of the participants' community and society.

DISCUSSION

Our findings show that medical school applicants considered DT to be a valuable component of a holistic medical school admission process. Participants' experiences of DT were varied and complex. Our research suggests that admissions teams for medical schools should incorporate more comprehensive recruitment practices and inclusive methodological frameworks to accurately capture the diversity of identities and experiences of medical school applicants and to consider the factors that shape their journey to medical schools.

The influence of occupational therapy students' preferred language on academic and clinical performance in a Canadian university program

The Occupational Therapy Program at this Canadian university is a French program however, students must be bilingual to function in English or French clinical fieldwork settings. An understanding of the role of language in successful completion of program requirements was needed to effectively support students' education. The study objectives were to identify the role of linguistic factors in students' academic and clinical performance and to provide recommendations for strategies to address areas of learning difficulty. A multimethod approach used 4 data sources: (1) Multiple Mini Interview (MMI) informal language assessment scores, (2) grade point average (GPA), (3) fieldwork evaluation reports, and (4) an online survey of program graduates. The GPA on admission and MMI scores of 140 students predicted respectively only 20% and 2% of the variation in GPA on program completion. The areas of poorest performance in failed clinical fieldwork reports were in clinical reasoning and communication competencies. Among survey respondents (n = 47), 44.5% reported that a clinical placement in their second language with related charting (51.6%) and client communication (40.9%) were the most significant difficulties encountered in the program. Clients with mental health issues (45.4%) were the most challenging population to work with, attributed to communication barriers in the students' second language. Strategies are proposed to support occupational therapy students' academic and clinical language proficiency, including conversational training courses, problem-based learning activities in students' second language, focussed teaching on the clinical reasoning process and reflective skills, and language coaching to address early signs of difficulty in clinical fieldwork.

Cross-sectional evaluation of an asynchronous multiple mini-interview (MMI) in selection to health professions training programmes with 10 principles for fairness built-in

OBJECTIVES

We aimed to explore the psychometric properties of the first known online asynchronous multiple mini-interview (MMI) designed for fairness with subgroup analyses by key characteristics, usability and acceptability.

DESIGN

Cross-discipline multimethod evaluation.

SETTING

One UK University.

PARTICIPANTS

Applicants to nursing, midwifery and paramedic science undergraduate programmes during 2021-2022.

PRIMARY, SECONDARY OUTCOME MEASURES

Psychometric properties (internal consistency, construct validity, dimensionality) were assessed using Cronbach's alpha (α), parallel analysis (PA), Schmid-Leiman transformation and ordinal confirmatory factor analysis (CFA). Usability and acceptability were evaluated using descriptive statistics and conventional content analysis.

METHODS

The system was configured in a seven question 4 min MMI. Applicants' videorecorded their answers which were later assessed by interviewers and scores summed. Applicants and interviewers completed online evaluation questionnaires.

RESULTS

Performance data from 712 applicants determined good-excellent reliability for the asynchronous MMI (mean α 0.72) with similar results across subgroups (gender, age, disability/support needs, UK/non-UK). PA and factor analysis results suggested there were seven factors relating to the MMI questions with an underlying general factor that explained the variance in observed candidate responses. A CFA testing a seven-factor hierarchical model showed an excellent fit to the data (Confirmatory Fit Index=0.99), Tucker Lewis Index=0.99, root mean square error (RMSE) =0.034). Applicants (n=210) viewed the flexibility, relaxed environment and cost savings advantageous. Interviewers (n=65) reported the system to be intuitive, flexible with >70% time saved compared with face-to-face interviews. Reduced personal communication was cited as the principal disadvantage.

CONCLUSIONS

We found that the asynchronous MMI was reliable, time-efficient, fair and acceptable and building fairness in was lost-cost. These novel, insights are applicable across health professions selection internationally informing the future configuration of online interviews to ensure workforces represent the societies they serve.

Multiple mini-interviews is a predictor of students' academic achievements in early undergraduate medical years: a retrospective study

BACKGROUND

Our study determined Multiple Mini-Interview (MMI) effectiveness in evaluating specific skill sets based on medical students' performances during the undergraduate years and compared the academic performances of medical students who appeared for onsite/online MMI.

METHODS

A retrospective study of 140 undergraduate medical students between 2016 and 2020 included data on age, gender, pre-university results, MMI scores, and examination results. Appropriate non-parametric tests were applied to compare the students' MMI and academic performances.

RESULTS

Ninety-eight students from cohorts 12 to 15 had an overall MMI score of 69.0(IQR: 65.0-73.2)/100 and an overall Cumulative Grade Point Average(GPA) of 3.64 (3.42-3.78)/5.0. Spearman's correlation revealed a significantly positive relationship between MMI and cGPA (rho = 0.23) and GPA from the first 2 semesters (GPA1 rho = 0.25, GPA2 rho = 0.27). This observation was similar to that for station A in the first year (cGPA rho = 0.28, GPA1 rho = 0.34, GPA2 rho = 0.24), and in station B (GPA4 rho = 0.25) and D (GPA3 rho = 0.28, GPA4 rho = 0.24) in the second year. Of twenty-nine cohort16 students, 17(58.6%) underwent online and 12(41.4%) offline modes of MMI assessment, respectively. The overall median MMI score was 66.6(IQR: 58.6-71.6)/100, and the overall median cGPA was 3.45 (3.23-3.58)/5.0. When comparing the median marks of cohort16 groups, the online group scored significantly higher marks for station D than the offline group (p = 0.040).

CONCLUSION

Correspondence between MMI scores and cGPA predicted MMI scoring during student selection and entry process might ensure the success of their academic performance in medical school.

Evaluating a Global Assessment Measure Created by Standardized Patients for the Multiple Mini Interview in Medical School Admissions: Mixed Methods Study

BACKGROUND

Standardized patients (SPs) are essential stakeholders in the multiple mini interviews (MMIs) that are increasingly used to assess medical school applicants' interpersonal skills. However, there is little evidence for their inclusion in the development of instruments.

OBJECTIVE

This study aimed to describe the process and evaluate the impact of having SPs co-design and cocreate a global measurement question that assesses medical school applicants' readiness for medical school and acceptance status.

METHODS

This study used an exploratory, sequential, and mixed methods study design. First, we evaluated the initial MMI program and determined the next quality improvement steps. Second, we held a collaborative workshop with SPs to codevelop the assessment question and response options. Third, we evaluated the created question and the additional MMI rubric items through statistical tests based on 1084 applicants' data from 3 cohorts of applicants starting in the 2018-2019 academic year. The internal reliability of the MMI was measured using a Cronbach α test, and its prediction of admission status was tested using a forward stepwise binary logistic regression.

RESULTS

Program evaluation indicated the need for an additional quantitative question to assess applicant readiness for medical school. In total, 3 simulation specialists, 2 researchers, and 21 SPs participated in a workshop leading to a final global assessment question and responses. The Cronbach α's were >0.8 overall and in each cohort year. The final stepwise logistic model for all cohorts combined was statistically significant (P<.001), explained 9.2% (R2) of the variance in acceptance status, and correctly classified 65.5% (637/972) of cases. The final model consisted of 3 variables: empathy, rank of readiness, and opening the encounter.

CONCLUSIONS

The collaborative nature of this project between stakeholders, including nonacademics and researchers, was vital for the success of this project. The SP-created question had a significant impact on the final model predicting acceptance to medical school. This finding indicates that SPs bring a critical perspective that can improve the process of evaluating medical school applicants.

The predictive validity of admission criteria for the results of clinical competency assessment with an emphasis on family medicine in the fifth year of medical education: an observational study

BACKGROUND

In many countries, the number of applicants to medical schools exceeds the number of available places. This offers the need, as well as the opportunity to medical schools to select those applicants most suitable for later work as a doctor. However, there is no generally accepted definition of a 'good doctor'. Clinical competencies may serve as surrogates. The aim of this study was to compare medical students in Germany selected based either on their pre-university grade point average alone or based on the result of a university-specific selection procedure regarding their clinical competencies with an emphasis on family medicine in the later years of training.

METHODS

We used the 'Allgemeinarztbarometer Ausbildung' (Undergraduate Family Medicine Barometer), an instrument developed to assess clinical competencies with an emphasis on family medicine, to compare students in the pre-university grade point average admission-quota and the university-specific selection procedure admission-quota in the fifth year of training. Students were judged by their supervising general practitioners after a two-week practical course. Competencies were rated on a five-point Likert-scale (1 = 'totally agree' i.e. the student is very competent to 5 = 'totally disagree' i.e. the student is not competent at all).

RESULTS

We included 94 students (66% female). Students in the university-specific selection procedure quota (n = 80) showed better mean scores in every item of the Undergraduate Family Medicine Barometer. We found a statistically significant difference between the two groups for the item assessing communication skills (M [university-specific selection procedure quota] = 1.81, SD = 0.84 vs. M [pu-GPA quota] = 2.38, SD = 0.96; t[91] = -2.23, p = .03; medium effect size). Logistic regression revealed no statistically significant age or gender contribution.

CONCLUSIONS

Despite the small sample-size, our results indicate, that students selected via an university-specific selection procedure show better communicative competencies in the later years of training.

The validity of the assignment book-based structured interview in predicting academic performance in medical schools: a retrospective cohort study

PURPOSE

Structured interviews have become essential in the medical schools admission selection because structured interviews predict academic achievement after admission. The purpose of this study was to determine validity and fairness of the new structural interview technique, assignment book-based structured interview (ABSI), in predicting future academic achievement of the medical students.

METHODS

The validity of this new interview technique and academic achievement was evaluated based on the data of all the applicants and successful applicants who applied for on-time admission between the year 2011 and 2014.

RESULTS

The ABSI technique showed a significant correlation and predictive validity for academic achievement in the medical school. The retention group received significantly lower T-scores of ABSI compared with the superior student group.

CONCLUSION

The results indicate that ABSI is a feasible, reliable, fair and valid admission selection tool. The ABSI may be meaningful and fair method for predicting academic achievements, and it could be incorporated as one of the contents in the multiple mini-interview.

Comparison between Personality, Emotional, and Educational Outcomes of Multiple Mini Interviews and Personal Interview

Objectives

This study investigated the outcomes of multiple mini interviews and personal interview on personality traits, emotional intelligence, perceived educational environment, and stressors.

Methods

This is a comparative cross-sectional study on two cohorts of pre-clinical medical students who were selected by multiple mini interviews and personal interview, respectively. Their personality traits, emotional intelligence, perceived educational environment, and perceived stressors were measured using different measurement tools.

Results

Multiple mini interviews and personal interview demonstrated a similar ability to recruit medical students with a high level of emotional intelligence. The main advantage of personal interviews over multiple mini interviews in terms of personality traits is that it recruited candidates who had a higher level of conscientiousness trait. The main advantage of multiple mini interviews over personal interview on the educational environment is that medical students chosen by multiple mini interviews had a higher level of satisfaction with social aspects of medical training. Regardless of admission processes, the medical students were equally vulnerable to psychological distress due to various stressful events throughout medical training particularly related to academic loads.

Conclusions

This study provided evidence to support the outcomes that multiple mini interviews and personal interview have on medical students’ emotional intelligence, personality traits, perceived educational environment, and perceived stressors during the pre-clinical medical training. Interestingly, personal interview had a better outcome on conscientiousness while multiple mini interviews had a better outcome on the social aspect.

Reliability and acceptability of the multiple mini-interview for selection of residents in cardiology

INTRODUCTION

The multiple mini-interview (MMI) model can be useful to evaluate non-cognitive domains and guide the selection process in medical residency programs. The aim of this study was to evaluate the reliability and acceptability of the MMI model for the selection of residents in a cardiology residency program.

METHODS

We conducted an observational and prospective study. It was performed in a tertiary-care center specialized in cardiology and included candidates for the cardiology residency program in March 2018. Ten stations were developed to evaluate different non-cognitive domains. Reliability was evaluated by the generalizability G coefficient. Candidates and interviewers were surveyed to evaluate the acceptability of the MMI model.

RESULTS

Nine faculty members were trained and 22 candidates were evaluated. The G study showed a relative G coefficient between 0.56 and 0.73, according to the design. 91% of the candidates stated that they preferred MMI over other types of interviews as a selection method for admission to the residency program, and all the interviewers considered they had enough time to evaluate the candidates and their strengths as future residents.

CONCLUSION

The MMI is a reliable model to evaluate candidates for a residency program in cardiology with high acceptability among residents and observers.