Multiple mini-interviews: same concept, different approaches

Examining interviewer bias in medical school admissions: The interplay between applicant and interviewer gender and its effects on interview outcomes

Selection interviews have long been integral to medical school admissions, yet their limited predictive validity and susceptibility to bias raise concerns. This study delves into potential interviewer bias within the dynamics of interviewee and interviewer gender. We analyze a dataset of 5,200 applicants and over 370 selection committees engaged in semi-structured interviews from 2006 to 2019 at a large German medical school with multiple linear and non-linear regression analyses. Our findings reveal that all-female committees tended to award male candidates, on average, one point more than their female counterparts, significantly enhancing the chances of submission for male applicants despite lower academic grades, which constituted 51% of the selection process points. All-male and mixed-gender committees exhibited similar ratings for both genders. The role of valuing voluntary services emerged prominently: all-male and mixed committees acknowledged women's volunteer work but not men's, while all-female committees demonstrated the opposite pattern. Our results attribute variations in interview outcomes to the absence of standardization, such as insufficient interviewer training, divergent rating strategies, variations in interviewer experience, and imbalances in candidate allocation to selection committees, rather than to a "gender bias", for example by favoritism of males because of their gender.

Interactional competencies in medical student admission- what makes a "good medical doctor"?

Doctors' interactional competencies play a crucial role in patient satisfaction, well-being, and compliance. Accordingly, it is in medical schools' interest to select candidates with strong interactional abilities. While Multiple Mini Interviews (MMIs) provide a useful context to assess such abilities, the evaluation of candidate performance during MMIs is not always based on a solid theoretical framework. The newly developed selection procedure "Interactional Competencies - Medical Doctors (IC-MD)" uses an MMI circuit with five simulation patient scenarios and is rated based on the theoretically and empirically grounded construct of emotional availability. A first validation study with N = 70 first-semester medical students took place in 2021. In terms of convergent validity, IC-MD ratings showed strong correlations with simulation patients' satisfaction with the encounter (r =.57) but no association with emotional intelligence measures. IC-MD ratings were not related to high school performance or a cognitive student aptitude test, indicating divergent validity. Inter-rater reliability (ICC = 0.63) and generalizability (Eρ2 = 0.64) were satisfactory. The IC-MD proved to be fair regarding participants' age and gender. Participants with prior work experience in healthcare outperformed those without such experience. Participant acceptance of the procedure were good. The IC-MD is a promising selection procedure capable of assessing interactional competencies relevant to the medical setting. Measures of interactional competencies can complement the use of cognitive selection criteria in medical student admission. The predictive validity of the IC-MD needs to be addressed in future studies.

Investigating the internal structure of multiple mini interviews-A perspective from Pakistan

BACKGROUND

Healthcare professionals require many personal attributes in addition to cognitive abilities and psychomotor skills for competent practice. Multiple Mini- Interviews are being employed globally to assess personality attributes of candidates for selection in health professions education at all level of entry; these attributes are namely, communication skills, critical thinking, honesty, responsibility, health advocacy, empathy and sanctity of life. Considering the high stakes involved for students, faculty, institutions and the society, rigorous quality assurance mechanisms similar to those used for student assessment must be employed for student selection, throughout the continuum of medical education. It is a difficult undertaking as these psychological constructs are difficult to define and measure. Though considered to yield reliable and valid scores, studies providing multiple evidences of internal structure especially dimensionality of Multiple Mini-Interviews are sparse giving rise to questions if they are measuring a single or multiple constructs and even if they are measuring what they are purported to be measuring.

OBJECTIVE

The main objective is to provide statistical support of the multi-dimensional nature of our Multiple Mini Interviews, hypothesized a-priori, through CFA. Another objective is to provide multiple evidences for the internal structure. Our study highlights the link between content and internal structure evidences of the constructs, thus establishing that our Multiple Mini Interviews measure what they were intended to measure.

METHOD

After securing permission from the Institutional review board, an a-priori seven factor-model was hypothesized based on the attributes considered most essential for the graduating student of the institution. After operationally defining the attributes through extensive literature search, scenarios were constructed to assess them. A 5-point rating scale was used to rate each item on the station. A total 259 students participated in the multiple mini interviews over a period of three days. A training workshop had been arranged for the participating faculty.

RESULTS

The reliability coefficient using Cronbach's alpha were calculated (range from 0.73 to 0.94), Standard Error of Measurement (ranged from 0.80 to1.64), and item to station-total correlation ranged from 0.43-0.50 to 0.75-0.83. Inter-station correlation was also determined. Confirmatory factor analysis endorsed the results of Exploratory factor analysis in the study revealing a seven model fit with multiple indices of Goodness-of-fit statistics such as Root mean square error of approximation (RMSEA) value 0.05, Standardized root mean square residual (SRMR) value with less than 0.08. All these indices showed that model fit is good. The Confirmatory factor analysis confirmed the multi-dimensional nature of our MMIs and also confirmed that our stations measured the attributes that they were supposed to measure.

CONCLUSION

This study adds to the validity evidence of Multiple Mini-Interviews, in selection of candidates, with required personality traits for healthcare profession. It provides the evidence for the multi-dimensional structure of Multiple Mini interviews administered with multiple evidences for its internal structure and demonstrates the independence of different constructs being measured.

Has a national recruitment scheme created a positive intervention for Black, Asian or other Minority Ethnic pharmacy trainees?

OBJECTIVES

A National Recruitment Scheme (NRS) for Pharmacy trainees was introduced in England and Wales in 2017, standardising recruitment processes on behalf of employers and with the aim of reducing bias for candidates applying to training posts within the National Health Service (NHS). This research attempted to identify whether the introduction of the NRS had an impact on the recruitment of Black, Asian, or other Minority Ethnic applicants into the most sought-after posts within the Scheme (hospital posts).

METHODS

An observational study was undertaken. Anonymised pharmacist trainee recruitment data between the cohort intakes of 2015-16 and 2020-21 was obtained from the pharmacy regulator the General Pharmaceutical Council and a comparison of proportional representations of ethnicities was undertaken, to ascertain whether a greater proportion of applicants from minority backgrounds attained the most sought-after posts in the NHS after the NRS was introduced. A robust generalised linear model was then used to analyse the data using binomial as the variance function and logit as a link function, where the proportion of hospital recruitment was an outcome with a two-way interaction between intervention and ethnicity after adjusting for overall proportion.

KEY FINDINGS

The statistical analysis of 18 283 pharmacy trainees in total, of whom 4446 were in hospital, shows a significant overall impact of intervention, with a significant positive change in the proportions of Asian-Pakistani applicants (P-value < 0.001) and Black-African applicants (P-value < 0.001) recruited to hospital posts.

CONCLUSIONS

Since the introduction of the NRS there has been a statistically significant impact on the correlation between the overall number of Black, Asian or other Minority Ethnic applicants and their proportion in hospital. That is, not only is the makeup of the hospital cohort increasingly reflecting the diversity of the overall cohort, but also a larger percentage of each ethnic cohort is attaining hospital training places.

Multiple mini-interviews is a predictor of students' academic achievements in early undergraduate medical years: a retrospective study

BACKGROUND

Our study determined Multiple Mini-Interview (MMI) effectiveness in evaluating specific skill sets based on medical students' performances during the undergraduate years and compared the academic performances of medical students who appeared for onsite/online MMI.

METHODS

A retrospective study of 140 undergraduate medical students between 2016 and 2020 included data on age, gender, pre-university results, MMI scores, and examination results. Appropriate non-parametric tests were applied to compare the students' MMI and academic performances.

RESULTS

Ninety-eight students from cohorts 12 to 15 had an overall MMI score of 69.0(IQR: 65.0-73.2)/100 and an overall Cumulative Grade Point Average(GPA) of 3.64 (3.42-3.78)/5.0. Spearman's correlation revealed a significantly positive relationship between MMI and cGPA (rho = 0.23) and GPA from the first 2 semesters (GPA1 rho = 0.25, GPA2 rho = 0.27). This observation was similar to that for station A in the first year (cGPA rho = 0.28, GPA1 rho = 0.34, GPA2 rho = 0.24), and in station B (GPA4 rho = 0.25) and D (GPA3 rho = 0.28, GPA4 rho = 0.24) in the second year. Of twenty-nine cohort16 students, 17(58.6%) underwent online and 12(41.4%) offline modes of MMI assessment, respectively. The overall median MMI score was 66.6(IQR: 58.6-71.6)/100, and the overall median cGPA was 3.45 (3.23-3.58)/5.0. When comparing the median marks of cohort16 groups, the online group scored significantly higher marks for station D than the offline group (p = 0.040).

CONCLUSION

Correspondence between MMI scores and cGPA predicted MMI scoring during student selection and entry process might ensure the success of their academic performance in medical school.

A mixed methods evaluation of Multiple Mini Interviews for entry into the Bachelor of Midwifery

BACKGROUND

Multiple-mini interviews (MMI) are increasingly used as part of the admission process into healthcare degrees. Evaluations have found MMIs to be a fair assessment tool in terms of reliability and validity and viewed positively by those involved in the MMI process. The use of MMIs in midwifery is novel and evaluation is lacking.

AIM

To evaluate the use of MMIs as part of the admission process for the Bachelor of Midwifery in one Australian university.

METHODS

A basic convergent mixed methods study design was utilised. Data included linked data sets, Likert scale responses to survey questions, focus groups and open-ended survey questions. Integration took place at the interpretation and reporting stage.

FINDINGS

Participants viewed the MMI experience positively. The study confirmed the reliability of the MMIs as an assessment tool. Most variance in MMI scores was attributed to the candidate at 31.4 % with the interviewer and the interview station having less influence on the MMI score at 11 % and 6.4 % variance. Older applicants on average achieved higher MMI scores, and those who spoke a language other than English at home or were first in family to attend university had lower on average MMI scores. Being born overseas did not impact an applicant's MMI score. The overall experience was seen as fair, offering further opportunity to gain entry into the Bachelor of Midwifery.

CONCLUSION

MMIs were viewed positively and findings support the use of MMIs as part of an admissions process for the Bachelor of Midwifery.

Can Different Admissions to Medical School Predict Performance of Non-Technical Skill Performance in Simulated Clinical Settings?

Non-technical skills (NTS) in medical care are essential to ensure patient safety. Focussing on applicants' NTS during medical school admission could be a promising approach to ensure that future physicians master NTS at a high level. Next to pre-university educational attainment, many selection tests have been developed worldwide to facilitate and standardise the selection process of medical students. The predictive validity of these tests regarding NTS performance in clinical settings has not been investigated (yet). Therefore, we explored the predictive validities and prognosis of the Hamburg MMI (HAM-Int), HAM-Nat, PEA, and waiting as well as other quota (as example) designated by the Federal Armed Forces) for NTS performance in clinical emergency medicine training of medical students. During 2017 and 2020, N = 729 second, third, and fourth year students were enrolled within the study. The mean age of participants was 26.68 years (SD 3.96) and 49% were female students. NTS of these students were assessed during simulation scenarios of emergency training with a validated rating tool. Students admitted via waiting quota and designated by the Armed Forces performed significantly better than students admitted by excellent PEA (p = 0.026). Non-EU students performed significantly inferior (p = 0.003). Our findings provide further insight to explain how and if admission to medical school could predict NTS performance of further physicians.

Best practices for interviewing applicants for medical school admissions: a systematic review

INTRODUCTION

Interviews are commonly used to select applicants for medical school, residency, and fellowship. However, interview techniques vary in acceptability, feasibility, reliability, and validity. This systematic review investigated the effectiveness of different interview methods in selecting the best qualified applicants for admission to medical school and developed a logic model to implement best practices for interviewing.

METHODS

Five electronic literature databases were searched for comparative studies related to interviewing in medical schools from inception through February 1, 2021. Inclusion criteria included publications in English that compared different methods of conducting a selection interview in medical schools with a controlled trial design. General study characteristics, measurement methodologies, and outcomes were reviewed. Quality appraisal was performed using the Medical Education Research Study Quality Instrument (MERSQI) and the Oxford Risk of Bias Scale. Based on these findings, a logic model was constructed using content analysis.

RESULTS

Thirteen studies were included. The multiple mini-interview (MMI) was reliable, unbiased, and predicted clinical and academic performance; the virtual MMI increased reliability and lowered costs. For unstructured interviews, blinding interviewers to academic scores reduced bias towards higher scorers; student and faculty interviewers rated applicants similarly. Applicants preferred structured over unstructured interviews. Study quality was above average per the MERSQI, risk of bias was high per the Oxford scale, and between-study heterogeneity was substantial.

DISCUSSION

There were few high-quality studies on interviewing applicants for admission to medical school; the MMI appears to offer a reliable method of interviewing. A logic model can provide a conceptual framework for conducting evidence-based admissions interviews.

Effect of station format on the psychometric properties of Multiple Mini Interviews

BACKGROUND

Given the widespread use of Multiple Mini Interviews (MMIs), their impact on the selection of candidates and the considerable resources invested in preparing and administering them, it is essential to ensure their quality. Given the variety of station formats used and the degree to which that factor resides in the control of training programmes that we know so little about, format's effect on MMI quality is a considerable oversight. This study assessed the effect of two popular station formats (interview vs. role-play) on the psychometric properties of MMIs.

METHODS

We analysed candidate data from the first 8 years of the Integrated French MMIs (IF-MMI) (2010-2017, n = 11 761 applicants), an MMI organised yearly by three francophone universities and administered at four testing sites located in two Canadian provinces. There were 84 role-play and 96 interview stations administered, totalling 180 stations. Mixed design analyses of variance (ANOVAs) were used to test the effect of station format on candidates' scores and stations' discrimination. Cronbach's alpha coefficients for interview and role-play stations were also compared. Predictive validity of both station formats was estimated with a mixed multiple linear regression model testing the relation between interview and role-play scores with average clerkship performance for those who gained entry to medical school (n = 462).

RESULTS

Role-play stations (M = 20.67, standard deviation [SD] = 3.38) had a slightly lower mean score than interview stations (M = 21.36, SD = 3.08), p < 0.01, Cohen's d = 0.2. The correlation between role-play and interview stations scores was r = 0.5 (p < 0.01). Discrimination coefficients, Cronbach's alpha and predictive validity statistics did not vary by station format.

CONCLUSION

Interview and role-play stations have comparable psychometric properties, suggesting format to be interchangeable. Programmes should select station format based on match to the personal qualities for which they are trying to select.

Can selection interviews predict OSCE performance? Evidence from Hannover Medical School

OBJECTIVES

We analyze whether the student selection process at Hannover Medical School (MHH), which combined a semi-structured interview with school leaving grades, can predict performances in an Objective Structured Clinical Examination (OSCE). We also check whether there are differences between assessments of clinical knowledge, practical skills, and communication abilities.

METHODS

We use data from 525 medical students who were admitted after a successful selection process and who completed the OSCE in the years 2015-2019. We employ multivariate regressions and a mediation analysis approach to learn whether study success after admission and prior to the OSCE mediates the outcome of the latter.

RESULTS

A better performance in the MHH's selection interview is unrelated to success in the OSCE. However, there is a small but significant influence of school grades on OSCE results in each part except for the assessment of communication skills. The impact of the school grade is partially mediated by performances in written and oral exams preceding the OSCE.

DISCUSSION

School grades matter for the OSCE outcome, albeit to different degrees for more learning-based vs. practical parts of the examination. The interview at MHH was purely informative and unrelated to study success, also in the assessment of communication skills. Better structured interview tools may yield better results.

CONCLUSION

Students' cognitive abilities predict study success in an undergraduate OSCE. Performances in a semi-structured selection interview have no impact, not even the assessment of communication skills.

Are different station formats assessing different dimensions in multiple mini-interviews? Findings from the Canadian integrated French multiple mini-interviews

BACKGROUND

Multiple mini-interviews (MMI) are used to assess non-academic attributes for selection in medicine and other healthcare professions. It remains unclear if different MMI station formats (discussions, role-plays, collaboration) assess different dimensions.

METHODS

Based on station formats of the 2018 and 2019 Integrated French MMI (IFMMI), which comprised five discussions, three role-plays and two collaboration stations, the authors performed confirmatory factor analysis (CFA) using the lavaan 0.6-5 R package and compared a one-factor solution to a three-factor solution for scores of the 2018 (n = 1438) and 2019 (n = 1440) cohorts of the IFMMI across three medical schools in Quebec, Canada.

RESULTS

The three-factor solution was retained, with discussions, role-plays and collaboration stations all loading adequately with their scores. Furthermore, all three factors had moderate-to-high covariance (range 0.44 to 0.64). The model fit was also excellent with a Comparative fit index (CFI) of 0.983 (good if > 0.9), a Tucker Lewis index of 0.976 (good if > 0.95), a Standardized Root Mean Square Residual of 0.021 (good if < .08) and a Root Mean Square Error of 0.023 (good if < 0.08) for 2018 and similar results for 2019. In comparison, the single factor solution presented a lower fit (CFI = 0.819, TLI = 0.767, SRMR = 0.049 and RMSEA = 0.070).

CONCLUSIONS

The IFMMI assessed three dimensions that were related to stations formats, a finding that was consistent across two cohorts. This suggests that different station formats may be assessing different skills, and has implications for the choice of appropriate reliability metrics and the interpretation of scores. Further studies should try to characterize the underlying constructs associated with each station format and look for differential predictive validity according to these formats.

Predicting Actual Social Skill Expression from Personality and Skill Self-Concepts

Social skills are of key importance in everyday and work life. However, the way in which they are typically assessed via self-report questionnaires has one potential downside; self-reports assess individuals' global self-concepts, which do not necessarily reflect individuals' actual social behaviors. In this research, we aimed to investigate how self-concepts assessed via questionnaires relate to skill expression assessed via behavioral observations after short interpersonal simulations. For this, we used an alternative behavior-based skill assessment approach designed to capture expressions of predefined social skills. Self- and observer ratings were collected to assess three different social skills: agency (i.e., getting ahead in social situations), communion (i.e., getting along in social situations), and interpersonal resilience (i.e., staying calm in social situations). We explored how these skills were related to self-concepts by differentiating between a classic personality measure (i.e., Big Five Inventory 2; BFI-2) and a novel skill questionnaire (i.e., Behavioral, Emotional, and Social Skills Inventory; BESSI). The results (N = 137) showed that both personality and skill self-concepts predicted self-rated skill expression, with the BESSI showing incremental validity. For both personality and skills self-concepts, the relationships with observer-rated skill expression were significant for agency but not for communion or interpersonal resilience. We discuss these results and highlight the theoretical and practical importance of differentiating between skill self-concepts and actual skill expression.

LEGO®-based communication assessment in virtual general surgery residency interviews

Purpose

Effective communication skills are a critical quality and skill that is highly sought after for surgeons which largely impacts patient outcomes. Residency programs design their interview processes to select the best candidates. LEGO®-based activities have been frequently used to enhance communication skills and team building. This study investigates the effectiveness and reliability of a novel LEGO®-based communication assessment in interviews for surgical residencies and the feasibility of implementing it in a virtual setting.

Methods

This study conducted a retrospective analysis of a LEGO®-based communication assessment at the program's 2020/2021 residency interviews. Each applicant was assessed on a different model. The total scores were analyzed for consistency among raters and correlated to faculty interviews. Furthermore, the impact of the assessment structure, scoring criteria, and range of models' difficulties on the total scores were explored.

Results

A total of 54 categorical and 55 preliminary applicants interviewed on 2 days. The assessment on different models and had no impact on applicants' total scores for either categorical and preliminary groups (p = 0.791 and 0.709, respectively). The communication components of the assessment showed high consistency between the raters. The two applicant groups displayed a statistically significant difference (p = 0.004) in the communication evaluation and model accuracy components. Total scores did not correlate with the faculty interviews of standardized questions in either group.

Conclusion

This novel LEGO®-based communication assessment showed high reliability and promising results as a tool to assess communication and problem solving for residency interviews that can be readily implemented in a virtual setting.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-022-00021-4.

Assessing distinguishable social skills in medical admission: does construct-driven development solve validity issues of situational judgment tests?

BACKGROUND

Social skills are important for future physicians and are therefore increasingly considered in selection processes. One economic assessment method from which different social skills can be inferred are Situational Judgment Tests (SJTs) in which applicants are asked to rate behavioral responses in context-relevant situations. However, traditional SJTs have so far failed to distinctively measure specified constructs. To address this shortcoming in the medical admission context, we applied a construct-driven approach of SJT development in which test development was deductively guided by agency and communion as target constructs.

METHOD

The final version of the construct-driven SJT includes 15 items per construct with three behavioral responses. Medical school applicants (N = 1527) completed the construct-driven SJT, a traditional SJT, and an aptitude test under high-stakes condition as part of their application. A subsample (N = 575) participated in a subsequent voluntary online study with self-report measures of personality and past behavior.

RESULTS

The proposed two-factor structure and internal consistency of the construct-driven SJT was confirmed. Communal SJT scores were positively associated with self-reported communal personality and communal behavior, yet effects were smaller than expected. Findings for agentic SJT scores were mixed with positive small associations to self-reported agentic personality scores and agentic behavior but unexpected negative relations to communal self-reported measures.

CONCLUSIONS

Results suggest that construct-driven SJTs might overcome validity limitations of traditional SJTs, although their implementation is challenging. Despite first indicators of validity, future research needs to address practical points of application in high-stakes settings, inclusion of other constructs, and especially prediction of actual behavior before the application of construct-driven SJTs for selection purposes in medical admission can be recommended.

Black students applying and admitted to medicine in the province of Quebec, Canada: what do we know so far?

To address the underrepresentation of Black students in medical schools in Canada and identify barriers in selection processes, we compare data from the latest Canadian census to that of an exit-survey conducted after a situational judgment test (Casper) among medical school applicants and from questionnaires done after selection interviews in Quebec, Canada. The proportion of Black people aged 15-34 years old in Quebec in 2016 was 5.3% province-wide and 8.2% in the Montreal metropolitan area. The proportion in the applicant pool for 2020 in Quebec was estimated to be 4.5% based on Casper exit-survey data. Comparatively, it is estimated that Black people represented 1.8% of applicants invited to admission interviews and 1.2% of admitted students in Quebec in 2019. Although data from different cohorts and data sources do not allow for direct comparisons, these numbers suggest that Black students applying to medical school are disproportionately rejected at the first step compared to non-Black students. Longitudinal data collection among medical school applicants will be necessary to monitor the situation. Further studies are required to pinpoint the factors contributing to this underrepresentation, to keep improving the equity of our selection processes.

A pilot project to introduce the multiple mini interview (MMI) at a Borneo medical school: The universiti Malaysia Sabah experience- a cross-sectional study

Introduction

The Multiple Mini Interview (MMI) demonstrates efficacy and superiority over traditional medical interviews in assessing non-cognitive domains during the recruitment of medical undergraduates. At Universiti Malaysia Sabah (UMS), a five-station MMI was piloted in 2019, featuring a mix of three examiner-driven stations (assessing professionalism, ethics, and motivation to study medicine), and two roleplayer-driven stations (assessing empathy and science communication specifically, and communication skills in general).

Methods

260 candidates were grouped into two separate geographical groups – urban and suburban/rural. Descriptive analysis, skewness and kurtosis were performed for normality assessment, whereas Cronbach's alpha, McDonald's omega, and Greatest lower bound assessed internal consistency. For validity measures, correlations were calculated between scores for separate stations, overall scores, urban and suburban/rural status. Also, exploratory factor analysis was performed on the five stations as validity measures. Difficulty and discrimination indices were calculated as quality measures. Qualitative analysis was performed on “red flag” comments detailing grossly unsuitable candidates.

Results

Roleplayer-driven stations yielded more red flags than examiner-driven stations. The three examiner-driven stations were significantly and moderately correlated (rho between 0.602 and 0.609, p < 0.001). The Empathy roleplayer-driven station was not correlated with two examiner-driven stations and only weakly correlated with the Ethics examiner-driven and the Science Communication roleplayer-driven station. Factor analysis suggests a three-factor model. The two roleplayer-driven stations stood as independent factors, and the three examiner-driven stations coalescing as one factor provided the best explanatory model. Quality measures suggest all five stations had suitable discriminatory properties (all >0.530), whereas the stations were distributed equally in difficulty index.

Conclusion

The UMS MMI has identified specific skillsets that may be in short supply in our incoming medical students. Also, it illustrates the yawning gap between academic knowledge and ‘translational’ scientific knowledge and communication skills.

•

The Multiple Mini Interview demonstrates superiority over traditional interviews during the recruitment of medical undergraduates.

•

At Universiti Malaysia Sabah, a five-station MMI was piloted in 2019.

•

The MMI featured a mix of three examiner-driven stations and two roleplayer-driven stations.

•

The UMS MMI has identified specific skillsets that may be in short supply in the incoming medical students.

•

It also illustrates the yawning gap between academic knowledge, “translational” scientific knowledge and communication skills.

Association of professional pre-qualifications, study success in medical school and the eligibility for becoming a physician: A scoping review

BACKGROUND

Literature, individual experiences and common considerations suggest that prior professional qualification can be an advantage for later career development. For instance, in Germany, professional pre-qualification has been honored by medical faculties in selection procedures for several years. However, a systematic evaluation of this relationship lacks. This scoping review summarizes existing literature and addresses the role of prior professional pre-qualifications on objective or subjective study success and the choice of a specialization.

METHODS

The scoping review was performed oriented on the PRISMA guidelines. PsycINFO and PubMed databases were searched for relevant studies that included data of medical students with and without professional pre-qualifications. To answer the underlying research questions, this scoping review also includes studies that examine professional pre-qualifications in association with non-cognitive "soft" criteria.

RESULTS AND FURTHER DIRECTIONS

1055 items were identified and reviewed by two independent reviewers with final 11 studies were included for this scoping review. The results of identified studies that report possible effects of prior pre-professional qualifications are inconclusive but suggest that prior professional qualifications tend not to have rather an advantage on study success. Medical school success for students with prior professional qualifications tended to be below average in the preclinical setting, and there were no differences in the clinical setting compared with students without prior professional qualifications. The influence of professional pre-qualifications has not yet been adequately studied without the moderator variable "waiting time" and "A-levels grade". The scoping review indicates insufficient number of articles stating a co-relation of prior pre-qualifications and subjective data. Again, the results found are not sufficient to state a clear relationship between professional pre-qualifications and the choice of a specific speciality preference. However, professional pre-qualifications, both in medicine and as "practical experience in rural areas", tend to be beneficial for the choice of becoming a rural physician. Large-scale cross-sectional and longitudinal studies are needed to investigate the influence of professional pre-qualifications on different study trajectory parameters.

Are different medical school admission tests associated with the outcomes of a simulation-based OSCE?

BACKGROUND

Medical school admission procedures have the common goal to select applicants with the greatest potential of becoming successful physicians. Hamburg Medical Faculty selects medical students by grade point average (GPA) and employs a two-step selection process of a natural sciences test (HAM-Nat), in some cases followed by multiple mini-interviews (HAM-Int). Multiple mini-interviews can predict non-cognitive outcomes, while GPA has predictive validity for cognitive outcomes. The aim of our study was to explore communication skills and clinical knowledge of advanced medical students according to their respective admission procedure.

METHODS

In July 2019, 146 students grouped according to their admission procedure into GPA-only (19.2 %), HAM-Nat (33.6 %), HAM-Int (30.8 %), and Waiting List (16.4 %) participated in four OSCE stations which equally assessed students' communication skills (OSCE part 1) and clinical knowledge (OSCE part 2) in simulated patient encounters, rated by physicians with checklists. Additionally, psychosocial assessors ranked communication skills with a global rating scale (GR). The students also participated in a multiple choice (MC) exam testing clinical knowledge. Kruskal-Wallis analyses of variance of test performance and Spearman correlation of instruments were calculated.

RESULTS

Students from the Waiting List group performed significantly worse on the MC exam compared to GPA-only and HAM-Int (adjusted p = .029 and 0.018, respectively). No significant differences were found between the admission groups with respect to communication skills. Global Rating and OSCE part 1 (communication) correlated significantly (ρ = 0.228, p = .006) as did OSCE part 2 (clinical knowledge) and MC exam (ρ = 0.242, p = .003), indicating criterion validity. Constructs did not overlap, indicating divergent validity.

CONCLUSIONS

Advanced medical students selected for undergraduate studies by multiple mini-interviews assessing psychosocial skills showed similar communication skills compared to students admitted to medical school by other entryways. It is unclear whether these similarities are due to an effective undergraduate longitudinal communication curriculum. Assessing baseline communication skills of all medical students at entry-level may aid with this question.

Accuracy of rating scale interval values used in multiple mini-interviews: a mixed methods study

When determining the score given to candidates in multiple mini-interview (MMI) stations, raters have to translate a narrative judgment to an ordinal rating scale. When adding individual scores to calculate final ranking, it is generally presumed that the values of possible scores on the evaluation grid are separated by constant intervals, following a linear function, although this assumption is seldom validated with raters themselves. Inaccurate interval values could lead to systemic bias that could potentially distort candidates' final cumulative scores. The aim of this study was to establish rating scale values based on rater's intent, to validate these with an independent quantitative method, to explore their impact on final score, and to appraise their meaning according to experienced MMI interviewers. A 4-round consensus-group exercise was independently conducted with 42 MMI interviewers who were asked to determine relative values for the 6-point rating scale (from A to F) used in the Canadian integrated French MMI (IFMMI). In parallel, relative values were also calculated for each option of the scale by comparing the average scores concurrently given to the same individual in other stations every time that option was selected during three consecutive IFMMI years. Data from the same three cohorts was used to simulate the impact of using new score values on final rankings. Comments from the consensus group exercise were reviewed independently by two authors to explore raters' rationale for choosing specific values. Relative to the maximum (A = 100%) and minimum (F = 0%), experienced raters concluded to values of 86.7% (95% CI 86.3-87.1), 69.5% (68.9-70.1), 51.2% (50.6-51.8), and 29.3% (28.1-30.5), for scores of B, C, D and E respectively. The concurrent score approach was based on 43,412 IFMMI stations performed by 4345 medical school applicants. It provided quasi-identical values of 87.1% (82.4-91.5), 70.4% (66.1-74.7), 51.2% (47.1-55.3) and 31.8% (27.9-35.7), respectively. Qualitative analysis explained that while high scores are usually based on minor details of relatively low importance, low scores are usually attributed for more serious offenses and were assumed by the raters to carry more weight in the final score. Individual drop or increase in final MMI ranking with the use of new scale values ranged from - 21 to + 5 percentiles, with the average candidate changing by ± 1.4 percentiles. Consulting with experienced interviewers is a simple and effective approach to establish rating scale values that truly reflects raters' intent in MMI, thus improving the accuracy of the instrument and contributing to the general fairness of the process.

Benefits and Feasibility of Using Videos to Assess Medical School Applicants' Empathetic Abilities in Multiple Mini Interviews

PURPOSE

We sought to evaluate the feasibility and benefits of using video-based scenarios in Multiple Mini Interviews (MMIs) to assess candidate's empathic abilities by investigating candidate perceptions and the acceptability, fairness, reliability, and validity of the test.

METHODS

The study sample was candidates for admission interviews held in the MMI format at a medical school in South Korea. In this six-station MMI, one station included a 2-min video clip of a patient-doctor communication scenario to assess candidate emphatic abilities, whereas paper-based scenarios were used in the other stations. Candidate's perceptions and acceptability of using the video-based scenario in the empathy station were examined using a 41-item post-MMI questionnaire. Fairness of the test was assessed by means of differences in candidate perceptions and performance across different demographics or backgrounds. Construct validity was assessed by examining the relationship of candidate performances in the empathy station with those in other stations. The G-coefficient was analyzed to estimate the reliability of the test.

RESULTS

Eighty-two questionnaires were returned, a 97.6% response rate. Candidates showed overall positive perceptions of the video-based scenario and they found it authentic and interesting. The test was fair as there were no differences in candidates' perceptions of the patient-doctor relationship presented in the video clip and neither in their performance nor in their perceived difficulty of the station across demographics or backgrounds. Construct validity was established as candidate performance in the empathy station was not associated with that of any other stations. The G-coefficient was 0.74.

CONCLUSIONS

The present study demonstrates that the video-based scenario is a feasible tool to assess candidate's empathy in the MMI.

Prioritizing the Interview in Selecting Resident Applicants: Behavioral Interviews to Determine Goodness of Fit

From our initial screening of applications, we assess that the 10% to 15% of applicants whom we will interview are all academically qualified to complete our residency training program. This initial screening to select applicants to interview includes a personality assessment provided by the personal statement, Dean's letter, and letters of recommendation that, taken together, begin our evaluation of the applicant's cultural fit for our program. While the numerical scoring ranks applicants preinterview, the final ranking into best fit categories is determined solely on the interview day at a consensus conference by faculty and residents. We analyzed data of 819 applicants from 2005 to 2017. Most candidates were US medical graduates (62.5%) with 23.7% international medical graduates, 11.7% Doctors of Osteopathic Medicine (DO), and 2.1% Caribbean medical graduates. Given that personality assessment began with application review, there was excellent correlation between the preinterview composite score and the final categorical ranking in all 4 categories. For most comparisons, higher scores and categorical rankings were associated with applicants subsequently working in academia versus private practice. We found no problem in using our 3-step process employing virtual interviews during the COVID pandemic.

Adaptation of internet-based multiple mini-interviews in a limited-resource medical school during the coronavirus disease 2019 pandemic

The coronavirus disease 2019 pandemic has affected many aspects of life, including medical education. The selection of students using multiple mini-interviews (MMI) should be adapted promptly as internet-based MMI (iMMI) in the current setting. The current transition from MMI to iMMI is supported by experience in conducting conventional MMI in previous years; hence, the station and assessor preparation could be completed without significant issues. The greater challenges to the use of iMMI are due to technical issues, including poor internet connections and adaptations necessary for assessing candidates' nonverbal expressions. This paper aims to describe the experience of conducting iMMI in the current pandemic situation, with further highlights on contextual challenges in a limited-resource setting.

More Than Their Test Scores: Redefining Success with Multiple Mini-Interviews

INTRODUCTION

Medical schools are shifting to a holistic approach for admissions. As non-cognitive abilities become more valued in the medical field and admissions become more competitive, multiple mini-interviews (MMI) are becoming increasingly common in selection criteria. The purpose of this article is to examine the relationship among admissions criteria to create a predictive model of acceptance to medical school. This article also aims to examine, among matriculated students, the relationship between MMI and traditional academic success, and the relationship between MMI scores and clinical competence and academic accolades.

METHODS

This longitudinal study followed a cohort of students from the MMI process through the first two pre-clinical years at a medical school in the southeastern USA. Multiple logistic regression with backward elimination variable selection was used to examine the relationship between admission criteria and acceptance. A multivariate multinomial ordered probit model was used to assess the relationship between MMI and traditional academic success among matriculated students. Simple linear regression models were used to assess relationships between MMI and Objective Structured Clinical Examination (OSCE) scores and honors nomination.

RESULTS

MMI are among eight significant predictors of acceptance. Among matriculated students, there were weak negative associations between MMI and traditional academic success; however, all but one relationship was statistically non-significant. There was not a significant association between MMI and OSCE scores or academic accolades.

CONCLUSIONS

While MMI can identify students with non-cognitive skills needed to be a good physician, continued assessment needs to be incorporated in their education.

Opening the black box of selection

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

Need for cognitive closure, tolerance for ambiguity, and perfectionism in medical school applicants

BACKGROUND

Physicians have to deal with uncertainty on a daily basis, which requires high tolerance for ambiguity. When medical decisions have to be made in ambiguous situations, low levels of need for cognitive closure and high levels of adaptive perfectionism are beneficial. It might be useful to measure such personality traits during medical school selection processes. In our study, we explored the expression of need for cognitive closure, tolerance for ambiguity, and perfectionism in medical school applicants who participated in a multiple mini-interview selection process with respect to the final decision of admission or rejection.

METHODS

After participating in the multiple mini-interview procedure (HAM-Int) at Hamburg Medical School in August 2019, 189 medical school applicants filled out a questionnaire including the Multidimensional Perfectionism Scale by Hewitt and Flett (MPS-H), the Multidimensional Perfectionism Scale by Frost (MPS-F), the Tolerance for Ambiguity Scale (TAS), the 16-Need for Cognitive Closure Scale (16-NCCS), and sociodemographic data. After the final admission decision, the scores of need for cognitive closure, tolerance for ambiguity, and perfectionism of admitted and rejected applicants were compared. We also assessed the predictive power of need for cognitive closure and age for the admission decision in a binary logistic regression model.

RESULTS

Compared to the admitted applicants, the rejected applicants showed a significantly higher need for cognitive closure (p = .009). A high need for cognitive closure correlated significantly positively with maladaptive perfectionism (p < .001) and significantly negatively with tolerance for ambiguity (p < .001). Low need for cognitive closure and older age were associated with a positive admission decision.

CONCLUSIONS

Regarding the personality traits need for cognitive closure, tolerance for ambiguity, and perfectionism we identified interesting differences and correlations of relevance for physicians' daily work in medical school applicants who were admitted or rejected after participating in a multiple mini-interview selection procedure. Further studies are needed to investigate these characteristics and their development longitudinally in medical students and to correlate them with students' medical performance.

A Practical Guide for Implementing a Holistic Admissions Review

BACKGROUND

Various minorities in terms of race, ethnicity, gender, sexual identity, and socioeconomic status experience poorer health outcomes than those in the majority. Evidence indicates that provider diversity is associated with improved access to care for minorities, increased patient satisfaction, improved outcomes, and better educational experiences for health professions students.

METHOD

The holistic admissions review is a flexible, individualized way of assessing an applicant's capabilities through consideration of experiences, attributes, and metrics (EAMs) that can serve as an important tool in diversifying the nursing workforce.

RESULTS

This article discusses (a) critical elements that must be addressed during the development and implementation of a holistic admissions review and (b) the creation and use of a scoring rubric based on the EAMs identified.

CONCLUSION

The implementation of a rubric was found to be a helpful tool to create a more thoughtful review of noncognitive factors and greater diversity in the student body. [J Nurs Educ. 2019;58(11):669-673.].

Predictive validity of multiple mini interview scores for future academic and clinical placement performance in physiotherapy, occupational therapy and speech and language therapy programmes

Background/aims

Limited research exists on the predictive validity of multiple mini interviews when used during the selection process for physiotherapy, occupational therapy and speech and language therapy programmes. Traditional interviews were replaced by multiple mini interviews in the selection process for these three undergraduate programmes in one UK university. The purpose of secondary data analysis was to determine whether multiple mini interviews can predict academic and clinical placement performance during the first and second year of study.

Methods

Secondary data analysis was performed using the admissions data from 169 students. Data analysed comprised predictors (entry tariff, age and multiple mini interview scores) and outcomes (academic and placement achievement in years 1 and 2 over a period of 2 years from 2014 to 2016).

Results

Multiple standard regression analyses revealed that, overall, multiple mini interview score was not a significant predictor of academic achievement or clinical placement performance in years 1 and 2 for any of the three professions. Pearson's correlations indicated that age and entry tariff were frequently and significantly correlated with academic achievements for all cohorts.

Conclusions

Performance in the multiple mini interview is not a predictor of academic achievement or clinical placement performance in this specific cohort of students. Entry tariff showed a significant correlation with academic achievement. Additional studies with larger cohorts are recommended.

The Multiple Mini Interview as a Dental School Admission Tool: Can It Predict Noncognitive Traits Associated with Professional Behaviors?

The aim of this study was to investigate the reliability and validity of the Multiple Mini Interview (MMI) process and to assess its use as an admission tool to predict noncognitive traits associated with professional behavior during patient care in one cohort of dental students at a single U.S. dental school. Data were analyzed for the 95 candidates who matriculated and graduated as part of the 2017 graduating cohort at the University of Michigan School of Dentistry. All MMI interviewees for one cycle of admissions rotated through ten stations: two traditional interview question stations and eight scenario stations measuring domains that included four questions scored on a five-point Likert scale. Generalizability theory analysis showed the MMI to have good reliability (G Coefficient of 0.74). Station reliabilities (Cronbach's alpha) ranged from 0.88 to 0.92. MMI scores showed a positive significant correlation with students' scores on the Dental Admission Test and Perceptual Ability Test, D1 cumulative GPA, and D4 Patient Management grade. MMI scores positively correlated with professional behaviors relating to three domains-openness, conscientiousness, and emotional stability-thus demonstrating good predictive validity for measuring noncognitive traits associated with professionalism. This study found that the MMI was a reliable and valid tool that predicted key behavioral traits associated with professionalism in dental students.

Multiple Speed Assessments

Abstract. This paper presents Multiple Speed Assessments as an umbrella term to encompass a variety of approaches that include multiple (e.g., 20), short (e.g., 3 min), and often integrated interpersonal simulations to elicit overt behavior in a standardized way across participants. Multiple Speed Assessments can be used to get insight into the behavioral repertoire of a target person in situations sampled from a predefined target domain and their intraindividual variability across these situations. This paper outlines the characteristics and theoretical basis of Multiple Speed Assessments. We also discuss various already existing examples of Multiple Speed Assessments (Objective Structured Clinical Examinations, Multiple Mini-Interviews, and constructed response multimedia tests) and provide an overview of design variations. Finally, we present current research evidence and future research directions related to Multiple Speed Assessments. Although we present Multiple Speed Assessments in the context of personnel selection, it can also be used for assessment in the educational, personality, or clinical psychology field.

Construct validity of multiple mini interviews - Investigating the role of stations, skills, and raters using Bayesian G-theory

Background: One popular procedure in the medical student selection process are multiple mini-interviews (MMIs), which are designed to assess social skills (e.g., empathy) by means of brief interview and role-play stations. However, it remains unclear whether MMIs reliably measure desired social skills or rather general performance differences that do not depend on specific social skills. Here, we provide a detailed investigation into the construct validity of MMIs, including the identification and quantification of performance facets (social skill-specific performance, station-specific performance, general performance) and their relations with other selection measures.Methods: We used data from three MMI samples (N = 376 applicants, 144 raters) that included six interview and role-play stations and multiple assessed social skills.Results: Bayesian generalizability analyses show that, the largest amount of reliable MMI variance was accounted for by station-specific and general performance differences between applicants. Furthermore, there were low or no correlations with other selection measures.Discussion: Our findings suggest that MMI ratings are less social skill-specific than originally conceptualized and are due more to general performance differences (across and within-stations). Future research should focus on the development of skill-specific MMI stations and on behavioral analyses on the extents to which performance differences are based on desirable skills versus undesired aspects.

The relationship between perfectionism and symptoms of depression in medical school applicants

BACKGROUND

The association between perfectionism and depression in the medical profession can ultimately influence physicians' performance negatively. In medical students, especially maladaptive perfectionism is connected with distress and lower academic performance. The expression of perfectionism and symptoms of depression at the time of medical school application is not known. Therefore, we explored perfectionism and symptoms of depression in participants of multiple mini-interviews for medical school admission and investigated possible differences between applicants who were eventually admitted or rejected.

METHODS

After the multiple mini-interviews admission procedure at Hamburg Medical School in August 2018, 146 applicants filled out a questionnaire including sociodemographic data and the following validated instruments: Multidimensional Perfectionism Scale by Hewitt and Flett (MPS-H), Multidimensional Perfectionism Scale by Frost (MPS-F), Patient Health Questionnaire (PHQ-9), Generalized Anxiety Disorder Scale (GAD-7), and a 10-item version of the Big Five Inventory (BFI-10). The two groups of admitted and rejected applicants were compared and the correlation between symptoms of depression and perfectionism further explored.

RESULTS

The admitted applicants were significantly more extrovert and had lower depression scores compared to the rejected applicants. In both groups, the composite scales of Adaptive Perfectionism (r = .21, p = .011) and Maladaptive Perfectionism (r = .43, p < .001) as well as their components correlated significantly with the PHQ-9 results. Maladaptive Perfectionism accounted for about 18% of variance in the PHQ-9 score.

CONCLUSIONS

Rejected medical school applicants who participated in a multiple mini-interviews admission procedure showed higher levels of depression symptoms than admitted applicants. The degree of depressive symptoms can be partly explained by Maladaptive Perfectionism scores. Since coping in medical school and in postgraduate medical education require robust mental health, perfectionism questionnaires could be an additional tool in medical school selection processes.

The Hamburg integrated medical degree program iMED

The integrated medical degree program (iMED) was established in winter semester 2012/2013 at the Faculty of Medicine of Universität Hamburg with the aim of improving medical education. The main features of the iMED medical degree program include the close integration of theoretical knowledge and practical skills, scientific orientation and the teaching of psychosocial and communication skills. All these features are commonly found in the modular compulsory core curriculum and elective courses ("2nd Tracks"): The compulsory core curriculum comprises 19 modules which are arranged thematically in seven module groups and cover three stages of a "learning spiral". By comprehensively coordinating the teaching content and the learning objectives of the participating theoretical and clinical subjects, theoretical content is taught on the basis of real patient's medical histories from the first stage of the learning spiral. The elective courses enable students to learn and apply scientific work in a structured curriculum according to their own interests. Relevant practical skills for students future professional routines are taught in the longitudinal training course "Clinical Examination Methods plus Communication" (KUMplusKOM), which runs through the entire curriculum up to the final practical year. Accompanying, extra-curricular projects such as crash courses in the natural sciences or using the iMED Textbook as an online learning platform increase the attractiveness of the iMED degree program. Results of the evaluation show that the introduction and the accompanying optimization of iMED were very successful.

Tracking Indigenous Applicants Through the Admissions Process of a Socially Accountable Medical School

PURPOSE

To describe the admissions process and outcomes for Indigenous applicants to the Northern Ontario School of Medicine (NOSM), a Canadian medical school with the mandate to recruit students whose demographics reflect the service region's population.

METHOD

The authors examined 10-year trends (2006-2015) for self-identified Indigenous applicants through major admission stages. Demographics (age, sex, northern and rural backgrounds) and admission scores (grade point average [GPA], preinterview, multiple mini-interview [MMI], final), along with score-based ranks, of Indigenous and non-Indigenous applicants were compared using Pearson chi-square and Mann-Whitney tests. Binary logistic regression was used to assess the relationship between Indigenous status and likelihood of admission outcomes (interviewed, received offer, admitted).

RESULTS

Indigenous qualified applicants (338/17,060; 2.0%) were more likely to be female, mature (25 or older), or of northern or rural background than non-Indigenous applicants. They had lower GPA-based ranks than non-Indigenous applicants (P < .001) but had comparable preinterview-, MMI-, and final-score-based ranks across all admission stages. Indigenous applicants were 2.4 times more likely to be interviewed and 2.5 times more likely to receive an admission offer, but 3 times less likely to accept an offer than non-Indigenous applicants. Overall, 41/338 (12.1%) Indigenous qualified applicants were admitted compared with 569/16,722 (3.4%) non-Indigenous qualified applicants.

CONCLUSIONS

Increased representation of Indigenous peoples among applicants admitted to medical school can be achieved through the use of socially accountable admissions. Further tracking of Indigenous students through medical education and practice may help assess the effectiveness of NOSM's social accountability admissions process.

Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI

BACKGROUND

Sociodemographic subgroup differences in multiple mini-interview (MMI) performance have been extensively studied within the MMI research literature, but heterogeneous findings demand a closer look at how specific aspects of MMI design (such as station type) affect these differences. So far, it has not been investigated whether sociodemographic subgroup differences imply that an MMI is biased, particularly in terms of its predictive validity.

METHODS

Between 2010 and 2017, the University Medical Centre Hamburg-Eppendorf (UKE) tested 1438 candidates in an MMI who also provided sociodemographic data and agreed to participate in this study. Out of these, 400 candidates were admitted and underwent a first objective structured clinical examination (OSCE) after one and a half years, including one station assessing communication skills. First, we analyzed the relationship between gender, age, native language and medical family background and MMI station performance including interaction terms with MMI station type (simulation, interview, and group) in a hierarchical linear model. Second, we tested whether the prediction of OSCE overall and communication station performance in particular differed depending on sociodemographic background by adding interaction terms between MMI performance and gender, age and medical family background in a linear regression model.

RESULTS

Young female candidates performed better than young male candidates both at interview and simulation stations. The gender difference was smaller (simulation) or non-significant (interview) in older candidates. There were no gender or age effects in MMI group station performance. All effects were very small, with the overall model explaining only 0.6% of the variance. MMI performance was not related to OSCE overall performance but significantly predicted OSCE communication station performance with no differences in the prediction for sociodemographic subgroups.

CONCLUSIONS

The Hamburg MMI is fair in its prediction of OSCE communication scores. Differences in MMI station performance for gender and age and their interaction with MMI station type can be related to the dimensions assessed at different station types and thus support the validity of the MMI. Rather than being threats to fairness, these differences could be useful for decisions relating to the design and use of an MMI.

Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review

Objectives

Multiple Mini Interviews (MMI) have been conducted across the globe in the student selection process, particularly in health profession education. This paper reported the validity evidence of MMI in various educational settings.

Methods

A literature search was carried out through Scopus, Science Direct, Google Scholar, PubMed, and EBSCOhost databases based on specific search terms. Each article was appraised based on title, abstract, and full text. The selected articles were critically appraised, and relevant information to support the validity of MMI in various educational settings was synthesized. This paper followed the PRISMA guideline to ensure consistency in reporting systematic review results.

Results

A majority of the studies were from Canada, with 41.54%, followed by the United Kingdom (25.39%), the United States (13.85%), and Australia (9.23%). The rest (9.24%) were from Germany, Ireland, the United Arab Emirates, Japan, Pakistan, Taiwan, and Malaysia. Moreover, most MMI stations ranged from seven to 12 with a duration of 10 min per station (including a 2-min gap between stations).

Conclusion

The results suggest that the content, response process, and internal structure of MMI were well supported by evidence; however, the relation and consequences of MMI to important outcome variables were inconsistently supported. The evidence shows that MMI is a non-biased, practical, feasible, reliable, and content-valid admission tool. However, further research on its impact on non-cognitive outcomes is required.

Twelve tips for constructing a multiple mini-interview

Health professions the world over value various competencies in their practitioners that are not easily captured by academic measures of performance. As a result, many programs have begun using multiple mini-interviews (MMIs) to facilitate the selection of candidates who are most likely to demonstrate and further develop such qualities. In this twelve-tips article, the authors offer evidence- and experience-based advice regarding how to construct an MMI that is fit for purpose. The tips are provided chronologically, offering guidance regarding how one might conceptualize their goals for creating an MMI, how to establish a database of stations that are context appropriate, and how to prepare both candidates and examiners for their task. While MMIs have been shown to have utility in many instances, the authors urge caution against over-generalization by stressing the importance of post-MMI considerations including data monitoring and integration between one's admissions philosophy and one's curricular efforts.

[Selection of medical students : Measurement of cognitive abilities and psychosocial competencies]

The German Constitutional Court is currently reviewing whether the actual study admission process in medicine is compatible with the constitutional right of freedom of profession, since applicants without an excellent GPA usually have to wait for seven years. If the admission system is changed, politicians would like to increase the influence of psychosocial criteria on selection as specified by the Masterplan Medizinstudium 2020.What experiences have been made with the actual selection procedures? How could Situational Judgement Tests contribute to the validity of future selection procedures to German medical schools?High school GPA is the best predictor of study performance, but is more and more under discussion due to the lack of comparability between states and schools and the growing number of applicants with top grades. Aptitude and knowledge tests, especially in the natural sciences, show incremental validity in predicting study performance. The measurement of psychosocial competencies with traditional interviews shows rather low reliability and validity. The more reliable multiple mini-interviews are superior in predicting practical study performance. Situational judgement tests (SJTs) used abroad are regarded as reliable and valid; the correlation of a German SJT piloted in Hamburg with the multiple mini-interview is cautiously encouraging.A model proposed by the Medizinischer Fakultätentag and the Bundesvertretung der Medizinstudierenden considers these results. Student selection is proposed to be based on a combination of high school GPA (40%) and a cognitive test (40%) as well as an SJT (10%) and job experience (10%). Furthermore, the faculties still have the option to carry out specific selection procedures.

Selection and academic success of medical students in Hamburg, Germany

BACKGROUND

Student selection at Hamburg medical school is based on the combination of a natural science knowledge test (HAM-Nat) and pre-university educational attainment.

METHOD

Of the 1565 medical students enrolled in Hamburg from 2012 to 2015 about half were admitted by an entrance test, half by quotas. First, we analysed sociodemographic determinants of entrance test performance. Then, we used regression analysis to describe the interplay of variables in the prediction of study outcome, the role of sociodemographic factors, and differences in the calibration of educational attainment specific to German federal states.

RESULTS

Better performance in the entrance test was associated with age over 21, male gender, German nationality, first language German and both parents holding an academic degree - effect sizes were small. No differences were found for the birthplaces of parents (a proxy for migration background). Study outcome differed considerably among admission paths: Students admitted by entrance test or the quota for excellent pre-university educational attainment performed markedly better during the first 3 terms than students admitted by the waiting list quota and the quota for foreign students. Gender differences in study outcome were slight with better performance by males. The relation of pre-university educational attainment to study outcome was moderated by the federal state in which secondary schooling took place. Methods for the equating of state-specific grades are explored. The predictive validity of the HAM-Nat after correction for range restriction was r = .31. The relatively low value of this coefficient may be attributed to 3 factors: 1. self-selection of applicants which leads to a validity-enhancing effect that is not revealed by the predictor-outcome correlation, 2. reduction of variance due to a high selection ratio, and 3. high test difficulty, exceeding the demands of the medical curriculum.

CONCLUSION

The HAM-Nat achieves a small amount of incremental validity over pre-university educational attainment. This effect, obtained from correlational analysis, underestimates the validity of the test, because it does not reflect the role of self-selection and other validity enhancing features of the selection process.

Does the sequence of rotations in Multiple Mini Interview stations influence the candidates' performance?

In Multiple Mini Interviews (MMIs), the candidates start out with different stations depending on the sequence of rotation they are assigned; thus, their perceived difficulty level and nervousness with their first station may differ. We examined whether such differences influenced the candidates' overall performance in the MMI. A 32-item questionnaire was developed and administered to candidates for admission interviews at a medical school regarding their perceptions of the MMI. Candidates' interview scores were also obtained and were compared across groups who differed in the sequence of rotation of MMI stations. Candidates felt nervous when they encountered the first station, which did not differ regardless of which station was their first encounter. Candidates were neutral regarding whether their first station was more difficult than the others and about whether its difficulty level influenced their overall performance in the test. Although candidates' perceived station difficulty differed across stations, there were no differences in their performance in each station between those it was their first station and those it was not. Candidates' total interview scores also did not differ across groups of different sequence of rotations. The MMI is a fair process as it does not give disadvantage to those who encounter a more difficult station than others for their first station.

2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions

Selection and recruitment into healthcare education and practice is a key area of interest for educators with significant developments in research, policy, and practice in recent years. This updated consensus statement, developed through a multi-stage process, examines future opportunities and challenges in selection and recruitment. There is both a gap in the literature around and a compelling case for further theoretical and empirical literature to underpin the development of overall selection philosophes and policies and their enactment. More consistent evidence has emerged regarding the quality of different selection methods. Approaches to selection are context-dependent, requiring the consideration of an institution's philosophy regarding what they are trying to achieve, the communities it purports to serve, along with the system within which they are used. Diversity and globalization issues continue to be critically important topics. Further research is required to explore differential attainment and explain why there are substantial differences in culturally acceptable ways of approaching diversity and widening access. More sophisticated evaluation approaches using multi-disciplinary theoretical frameworks are required to address the issues. Following a discussion of these areas, 10 recommendations are presented to guide future research and practice and to encourage debate between colleagues across the globe.

Multiple mini interviews: revealing similarities across institutions

BACKGROUND

Across the globe multiple mini interviews (MMIs) have rapidly replaced the use of panel interviews in the selection of medical students and other health professionals. MMIs typically demonstrate better reliability and validity than panel interviews but there is limited research on whether these different types of interview process measure the same or different constructs. Our research aims to ascertain if MMIs are multidimensional or unidimensional, and whether MMIs conducted at different institutions assess the same or different constructs to each other or to panel interviews.

METHODS

Participants were applicants to medical degrees who were shortlisted for interviews at three different institutions in 2013 (n = 165) and 2014 (n = 128). Two institutions used a bespoke MMI developed independently from each other and the third used a panel interview. Stations scores and overall (mean) interview scores were examined.

RESULTS

Exploratory principal components analysis and confirmatory factor analysis showed similar results in both years' data, supporting a unidimensional model. The two overall MMI scores were more strongly correlated to each other (r = .56 and .64 in 2013 and 2014 respectively) than either were to the panel interview scores (r = .07 and .15 in 2013; .39 and .48 in 2014).

CONCLUSIONS

It appears that both MMIs panel interviews tap a single latent construct, but not the same construct. We suggest that the MMI methodology might allow the measurement of an emergent construct such as adaptability.

Do Multiple Mini-Interview and Traditional Interview Scores Differ in Their Associations With Acceptance Offers Within and Across Five California Medical Schools?

PURPOSE

In single-school studies, multiple mini-interview (MMI) and traditional interview (TI) scores are associated with acceptance offers. Unexamined is whether scores at one school are associated with acceptance at other schools; such analyses would mitigate single-school design biases and better estimate how well interviews capture desired applicant attributes. Using data from the 5 California Longitudinal Evaluation of Admissions Practices (CA-LEAP) medical schools, the authors examined associations of MMI and TI scores with acceptance offers within and across schools.

METHOD

The analyses included applicants who interviewed at ≥1 CA-LEAP school during the 2011-2013 admissions cycles. Three CA-LEAP schools employed TIs and 2 employed MMIs. Interview scores were standardized (z scores: mean = 0, SD = 1), and associations with acceptance offers were examined within and across schools in analyses stratified by school, adjusting for applicant sociodemographics, academic metrics, year, and total number of interviews.

RESULTS

Of 4,993 applicants interviewed, 428 (8.6%) interviewed at both MMI schools, 681 (13.6%) at ≥2 TI schools, and 1,327 (26.6%) at ≥1 MMI and ≥1 TI school. For each school, acceptance was associated with interview score at that school and also with interview scores at the other 4 schools. Cross-school associations of MMI versus TI scores with acceptance did not differ statistically.

CONCLUSIONS

Interview score at a given school was associated with acceptance at the other 4 schools, with no significant differences in associations for MMIs versus TIs. The findings suggest both MMIs and TIs captured attributes valued by admissions teams across CA-LEAP schools.

Vive la Différence: The Freedom and Inherent Responsibilities When Designing and Implementing Multiple Mini-Interviews

The literature on multiple mini-interviews (MMIs) is replete with heterogeneous study results related to the constructs measured, correlations with other measures, and demographic relationships. Rather than view these results as contradictory, the authors ask, What if all of the results are correct? They point out that the MMI is not an assessment tool but, rather, an assessment method. Design and implementation of locally conducted MMIs in medical school admissions processes should reflect local needs. As with other local assessments, MMIs should be considered separate from nationally conducted assessments that reflect more universal competencies. With the freedom to exercise unique values in locally constructed MMIs, individual institutions, or small bands of like-minded institutions, in parallel carry the responsibility to ensure local assessment tool validity.

Medical School Applicant Characteristics Associated With Performance in Multiple Mini-Interviews Versus Traditional Interviews: A Multi-Institutional Study

PURPOSE

To examine applicant characteristics associated with multiple mini-interview (MMI) or traditional interview (TI) performance at five California medical schools.

METHOD

Of the five California Longitudinal Evaluation of Admission Practices consortium schools, three used TIs and two used MMIs. Schools provided retrospective data on 2011-2013 admissions cycle interviewees: age, gender, race/ethnicity (underrepresented in medicine [UIM] or not), disadvantaged (DA) status, undergraduate GPA, Medical College Admission Test (MCAT) score, and interview score (standardized as z score; mean = 0; SD = 1). Adjusted linear regression analyses, stratified by interview type, examined associations with interview performance.

RESULTS

The 4,993 applicants who completed 7,516 interviews included 931 (18.6%) UIM and 962 (19.3%) DA individuals; 3,226 (64.6%) had only 1 interview. Mean age was 24.4 (SD = 2.7); mean GPA and MCAT score were 3.72 (SD = 0.22) and 33.6 (SD = 3.7), respectively. Older age, female gender, and number of prior interviews were associated with better performance on both MMIs and TIs. Higher GPA was associated with lower MMI scores (z score, per unit GPA = -0.26; 95% CI = -0.45, -0.06) but unrelated to TI scores. DA applicants had higher TI scores (z score = 0.17; 95% CI = 0.07, 0.28) but lower MMI scores (z score = -0.18; 95% CI = -0.28, -0.08) than non-DA applicants. Neither UIM status nor MCAT score was associated with interview performance.

CONCLUSIONS

These findings have potentially important workforce implications, particularly regarding MMI performance of DA applicants, and illustrate the need for other multi-institutional studies.

Validity evidence for the Hamburg multiple mini-interview

BACKGROUND

Multiple mini-interviews (MMI) become increasingly popular for the selection of medical students. In this work, we examine the validity evidence for the Hamburg MMI.

METHODS

We conducted three follow-up studies for the 2014 cohort of applicants to medical school over the course of two years. We calculated Spearman's rank correlation (ρ) between MMI results and (1) emotional intelligence measured by the Trait Emotional Intelligence Questionnaire (TEIQue-SF) and the Situational Test of Emotion Management (STEM), (2) supervisors' and practice team members' evaluations of psychosocial competencies and suitability for the medical profession after a one-week 1:1 teaching in a general practice (GP) and (3) objective structured clinical examination (OSCE) scores.

RESULTS

There were no significant correlations between MMI results and the TEIQue-SF (ρ = .07, p > .05) or the STEM (ρ = .05, p > .05). MMI results could significantly predict GP evaluations of psychosocial competencies (ρ = .32, p < .05) and suitability for the medical profession (ρ = .42, p < .01) as well as OSCE scores (ρ = .23, p < .05). The MMI remained a significant predictor of these outcomes in a robust regression model including gender and age as control variables.

CONCLUSIONS

Our findings suggest that MMIs can measure competencies that are relevant in a practical context. However, these competencies do not seem to be related to emotional intelligence as measured by self-report or situational judgement test.

Personal domains assessed in multiple mini interviews (MMIs) for healthcare student selection: A narrative synthesis systematic review

OBJECTIVES

To examine the personal domains multiple mini interviews (MMIs) are being designed to assess, explore how they were determined and contextualise such domains in current and future healthcare student selection processes DESIGN: A systematic review of empirical research reporting on MMI model design was conducted from database inception to November 2017.

DATA SOURCES

Twelve electronic bibliographic databases.

REVIEW METHODS

Evidence was extracted from original studies, and integrated in a narrative synthesis guided by the PRISMA statement for reporting systematic reviews. Personal domains were clustered into themes using a modified Delphi technique.

RESULTS

A total of 584 articles were screened. 65 unique studies (80 articles) matched our inclusion criteria of which seven were conducted within nursing/midwifery faculties. Six in 10 studies featured applicants to medical school. Across selection processes, we identified 32 personal domains assessed by MMIs, the most frequent being: communication skills (84%), teamwork/collaboration (70%), and ethical/moral judgement (65%). Domains capturing ability to cope with stressful situations (14%), make decisions (14%), and resolve conflict in the workplace (13%) featured in fewer than ten studies overall. Intra- and inter-disciplinary inconsistencies in domain profiles were noted, as well as differences by entry level. MMIs deployed in nursing and midwifery assessed compassion and decision-making more frequently than in all other disciplines. Own programme philosophy and professional body guidance were most frequently cited (~50%) as sources for personal domains; a blueprinting process was reported in only 8% of studies.

CONCLUSIONS

Nursing, midwifery and allied healthcare professionals should develop their theoretical frameworks for MMIs to ensure they are evidence-based and fit-for-purpose. We suggest a re-evaluation of domain priorities to ensure that students who are selected, not only have the capacity to offer the highest standards of care provision, but are able to maintain these standards when facing clinical practice and organisational pressures.

Qualitative analysis of MMI raters' scorings of medical school candidates: A matter of taste?

Recent years have seen leading medical educationalists repeatedly call for a paradigm shift in the way we view, value and use subjectivity in assessment. The argument is that subjective expert raters generally bring desired quality, not just noise, to performance evaluations. While several reviews document the psychometric qualities of the Multiple Mini-Interview (MMI), we currently lack qualitative studies examining what we can learn from MMI raters' subjectivity. The present qualitative study therefore investigates rater subjectivity or taste in MMI selection interview. Taste (Bourdieu 1984) is a practical sense, which makes it possible at a pre-reflective level to apply 'invisible' or 'tacit' categories of perception for distinguishing between good and bad. The study draws on data from explorative in-depth interviews with 12 purposefully selected MMI raters. We find that MMI raters spontaneously applied subjective criteria-their taste-enabling them to assess the candidates' interpersonal attributes and to predict the candidates' potential. In addition, MMI raters seemed to share a taste for certain qualities in the candidates (e.g. reflectivity, resilience, empathy, contact, alikeness, 'the good colleague'); hence, taste may be the result of an ongoing enculturation in medical education and healthcare systems. This study suggests that taste is an inevitable condition in the assessment of students' performance. The MMI set-up should therefore make room for MMI raters' taste and their connoisseurship, i.e. their ability to taste, to improve the quality of their assessment of medical school candidates.