Cutting costs of multiple mini-interviews - changes in reliability and efficiency of the Hamburg medical school admission test between two applications

Multiple mini-interviews as admission process: A study on perception of health science students in Universiti Brunei Darussalam

BACKGROUND

As multiple mini-interviews (MMI) have grown in popularity in selecting applicants to health science programmes, it is essential to comprehend students' views towards MMI and its usefulness in the admissions process. The study aims to explore students' perceptions on the usefulness and satisfaction of the MMI as an admission process into the health science programmes offered in Universiti Brunei Darussalam (UBD).

METHODS

A cross-sectional study using a self-administered questionnaire was distributed to all Year 1 and 2 health sciences students in the university. For qualitative data, the responses obtained from the answers to the open-ended questions were analyzed manually using thematic analysis. Open coding was performed initially to identify words with similar meanings, recurring patterns and ideas. Focus coding was performed to group codes which sounded and felt similar. Themes were formulated and then reviewed. Descriptive statistics and univariate analysis were applied.

RESULTS

127 students participated in this study (53.4% response rate). 38.6% underwent the online MMI and 61.4% had the face-to-face MMI. 81% of participants agreed that the MMI was a fair assessment tool (81%), with adequate time allocated (91%), provided opportunity to demonstrate understanding of the profession (81%), and to express themselves (81%) but only 27% agreed that MMI was an enjoyable experience. No statistically significant differences were detected by gender. Those who did online MMI (89%) reported higher opportunity to express themselves, where 80% agreed it was an effective tool to assess selection of students in the health science profession. Univariate analysis revealed that male participants (42%) and those who did the online MMI (40%) were slightly more satisfied than those who did it onsite.

CONCLUSION

Student perception is important for MMIs because it provides valuable insights into the effectiveness and fairness of the interview process. Study participants' perceptions of the MMI were positive, with objective reports on its fairness, timing, and feasibility but felt unprepared for the interview process and less enjoyable. Online MMI is favoured more by participants who rated it higher effectiveness and adequate timing with a better understanding of stations.

Interactional competencies in medical student admission- what makes a "good medical doctor"?

Doctors' interactional competencies play a crucial role in patient satisfaction, well-being, and compliance. Accordingly, it is in medical schools' interest to select candidates with strong interactional abilities. While Multiple Mini Interviews (MMIs) provide a useful context to assess such abilities, the evaluation of candidate performance during MMIs is not always based on a solid theoretical framework. The newly developed selection procedure "Interactional Competencies - Medical Doctors (IC-MD)" uses an MMI circuit with five simulation patient scenarios and is rated based on the theoretically and empirically grounded construct of emotional availability. A first validation study with N = 70 first-semester medical students took place in 2021. In terms of convergent validity, IC-MD ratings showed strong correlations with simulation patients' satisfaction with the encounter (r =.57) but no association with emotional intelligence measures. IC-MD ratings were not related to high school performance or a cognitive student aptitude test, indicating divergent validity. Inter-rater reliability (ICC = 0.63) and generalizability (Eρ2 = 0.64) were satisfactory. The IC-MD proved to be fair regarding participants' age and gender. Participants with prior work experience in healthcare outperformed those without such experience. Participant acceptance of the procedure were good. The IC-MD is a promising selection procedure capable of assessing interactional competencies relevant to the medical setting. Measures of interactional competencies can complement the use of cognitive selection criteria in medical student admission. The predictive validity of the IC-MD needs to be addressed in future studies.

Measuring personal characteristics in applicants to German medical schools: Piloting an online Situational Judgement Test with an open-ended response format

Objectives

Situational Judgement Tests (SJT) are a cost-efficient method for the assessment of personal characteristics (e.g., empathy, professionalism, ethical thinking) in medical school admission. Recently, complex open-ended response format SJTs have become more feasible to conduct. However, research on their applicability to a German context is missing. This pilot study tests the acceptability, reliability, subgroup differences, and validity of an online SJT with open-ended response format developed in Canada ("Casper").

Methods

German medical school applicants and students from Hamburg were invited to take Casper in 2020 and 2021. The test consisted of 12 video- and text-based scenarios, each followed by three open-ended questions. Participants subsequently evaluated their test experience in an online survey. Data on sociodemographic characteristics, other admission criteria (Abitur, TMS, HAM-Nat, HAM-SJT) and study success (OSCE) was available in a central research database (stav).

Results

The full sample consisted of 582 participants. Test-takers' global perception of Casper was positive. Internal consistency was satisfactory in both years (α=0.73; 0.82) while interrater agreement was moderate (ICC(1,2)=0.54). Participants who were female (d=0.37) or did not have a migration background (d=0.40) received higher scores. Casper scores correlated with HAM-SJT (r=.18) but not with OSCE communication stations performance. The test was also related to Abitur grades (r=-.15), the TMS (r=.18), and HAM-Nat logical reasoning scores (r=.23).

Conclusion

This study provides positive evidence for the acceptability, internal consistency, and convergent validity of Casper. The selection and training of raters as well as the scenario content require further observation and adjustments to a German context to improve interrater reliability and predictive validity.

Best practices for interviewing applicants for medical school admissions: a systematic review

INTRODUCTION

Interviews are commonly used to select applicants for medical school, residency, and fellowship. However, interview techniques vary in acceptability, feasibility, reliability, and validity. This systematic review investigated the effectiveness of different interview methods in selecting the best qualified applicants for admission to medical school and developed a logic model to implement best practices for interviewing.

METHODS

Five electronic literature databases were searched for comparative studies related to interviewing in medical schools from inception through February 1, 2021. Inclusion criteria included publications in English that compared different methods of conducting a selection interview in medical schools with a controlled trial design. General study characteristics, measurement methodologies, and outcomes were reviewed. Quality appraisal was performed using the Medical Education Research Study Quality Instrument (MERSQI) and the Oxford Risk of Bias Scale. Based on these findings, a logic model was constructed using content analysis.

RESULTS

Thirteen studies were included. The multiple mini-interview (MMI) was reliable, unbiased, and predicted clinical and academic performance; the virtual MMI increased reliability and lowered costs. For unstructured interviews, blinding interviewers to academic scores reduced bias towards higher scorers; student and faculty interviewers rated applicants similarly. Applicants preferred structured over unstructured interviews. Study quality was above average per the MERSQI, risk of bias was high per the Oxford scale, and between-study heterogeneity was substantial.

DISCUSSION

There were few high-quality studies on interviewing applicants for admission to medical school; the MMI appears to offer a reliable method of interviewing. A logic model can provide a conceptual framework for conducting evidence-based admissions interviews.

Associations between communication OSCE performance and admission interviews in medical education

OBJECTIVE

To investigate associations between medical students' communication and other clinical skills assessed in OSCEs and MMIs performance upon admission by developing an assessment strategy based on an internationally acknowledged curriculum framework.

METHODS

Between July 2019 and March 2020, 365 medical students in the 4th and 5th semester took two OSCEs containing 10 5-minute stations examining communication and other clinical skills. We used a European conceptual framework to determine the content validity of the communication score and calculated scores of communication and other clinical skills. We assessed students' OSCE performance and estimated multiple regressions to predict its association with MMIs.

RESULTS

The multiple linear regressions showed that students' MMI performance upon admission is significantly associated with the communication score (b = 0.32, p = 0.006) but not the clinical score (b = 0.19, p = 0.121), when controlling for gender and cognitive criteria.

CONCLUSIONS

Our assessment strategy designed to distinguish between different areas of competence provides a more thorough description of the positive relationship between OSCE performance and MMIs.

PRACTICE IMPLICATIONS

We developed a communication skills assessment strategy that can be easily applied by medical schools that use OSCEs as a training or assessment method.

Reliability and validity of new online selection tests for midwifery students

BACKGROUND AND OBJECTIVE

Newly created midwifery bachelor programmes attract an increasing number of applicants. Hence, educators have moved away from traditional non-standardised selection methods and started developing new tools for the admission of students. Researchers have expressed concerns around the validity of such selection measures and their ability to identify the appropriate candidates. The objective of this study was to develop and implement selection procedures for a new midwifery Bachelor of Science degree in Hamburg, Germany, and assess their psychometric properties.

DESIGN

This is a mixed-methods longitudinal study on the reliability, validity and acceptability of newly developed selection criteria.

SETTING

The newly established midwifery bachelor of science in Hamburg, Germany.

PARTICIPANTS

Upon completion of their online application, all midwifery applicants were invited by email to participate in our research project and thereby informed of its purpose and voluntary nature of their participation. The total number of candidates who took the test was 366, of which 309 agreed to take part in this study. Sixty-five applicants were eventually admitted to the midwifery Bachelor programme, of which 59 were included in the study.

MEASUREMENTS AND FINDINGS

We developed two tests for cognitive ability (HAM-Mid I and II) and adapted one test for personal and professional characteristics (Casper - Computer-based Assessment for Sampling Personal Characteristics). Due to the Covid pandemic, we implemented the tests online instead of on site as initially planned. HAM-Mid I had the lowest (Cronbach's Alpha = 0.38) whereas Casper had the highest internal consistency (Cronbach's Alpha = 0.77) of all three tests. Age, previous academic achievement and native tongue were significantly associated with applicants' performance on HAM-Mid II and Casper admission tests. HAM-Mid II was associated with students' performance in the first year midwifery exam (r = 0.31, p < 0.05). Overall evaluation of HAM-Mid II and Casper was good while HAM-Mid I received a more negative feedback.

KEY CONCLUSIONS

Evidence on the reliability and predictive validity of the newly developed tests suggests that the applied admission criteria are appropriate for an objective selection of bachelor midwifery students.

IMPLICATIONS FOR PRACTICE

The interests of university and clinical stakeholders of a midwifery bachelor degree can be represented through the development of tests for the assessment of cognitive ability and personal and professional characteristics.

Are different medical school admission tests associated with the outcomes of a simulation-based OSCE?

BACKGROUND

Medical school admission procedures have the common goal to select applicants with the greatest potential of becoming successful physicians. Hamburg Medical Faculty selects medical students by grade point average (GPA) and employs a two-step selection process of a natural sciences test (HAM-Nat), in some cases followed by multiple mini-interviews (HAM-Int). Multiple mini-interviews can predict non-cognitive outcomes, while GPA has predictive validity for cognitive outcomes. The aim of our study was to explore communication skills and clinical knowledge of advanced medical students according to their respective admission procedure.

METHODS

In July 2019, 146 students grouped according to their admission procedure into GPA-only (19.2 %), HAM-Nat (33.6 %), HAM-Int (30.8 %), and Waiting List (16.4 %) participated in four OSCE stations which equally assessed students' communication skills (OSCE part 1) and clinical knowledge (OSCE part 2) in simulated patient encounters, rated by physicians with checklists. Additionally, psychosocial assessors ranked communication skills with a global rating scale (GR). The students also participated in a multiple choice (MC) exam testing clinical knowledge. Kruskal-Wallis analyses of variance of test performance and Spearman correlation of instruments were calculated.

RESULTS

Students from the Waiting List group performed significantly worse on the MC exam compared to GPA-only and HAM-Int (adjusted p = .029 and 0.018, respectively). No significant differences were found between the admission groups with respect to communication skills. Global Rating and OSCE part 1 (communication) correlated significantly (ρ = 0.228, p = .006) as did OSCE part 2 (clinical knowledge) and MC exam (ρ = 0.242, p = .003), indicating criterion validity. Constructs did not overlap, indicating divergent validity.

CONCLUSIONS

Advanced medical students selected for undergraduate studies by multiple mini-interviews assessing psychosocial skills showed similar communication skills compared to students admitted to medical school by other entryways. It is unclear whether these similarities are due to an effective undergraduate longitudinal communication curriculum. Assessing baseline communication skills of all medical students at entry-level may aid with this question.

"You can't always get what you want…": economic thinking, constrained optimization and health professions education

Every choice we make in health professions education has a cost, whether it be financial or otherwise; by choosing one action (e.g., integrating more simulation, studying more for a summative examination) we lose the opportunity to take an alternative action (e.g., freeing up time for other teaching, leisure time). Economics significantly shapes the way we behave and think as educators and learners and so there is increasing interest in using economic ways of thinking and approaches to examine and understand how choices are made, the influence of constraints and boundaries in educational decision making, and how costs are felt. Thus, in this article, we provide a brief historical overview of modern economics, to illustrate how the core concepts of economics-scarcity (and desirability), rationality, and optimization-developed over time. We explain the important concept of bounded rationality, which explains how individual, meso-factors and contextual factors influence decision making. We then consider the opportunities that these concepts afford for health professions education and research. We conclude by proposing that embracing economic thinking opens up new questions and new ways of approaching old questions which can add knowledge about how choice is enacted in contemporary health professions education.

COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape

Introduction: The COVID-19 pandemic presented numerous, significant challenges for medical schools, including how to select the best candidates from a pool of applicants when social distancing and other measures prevented "business as usual" admissions processes. However, selection into medical school is the gateway to medicine in many countries, and it is critical to use processes which are evidence-based, valid and reliable even under challenging circumstances. Our challenge was to plan and conduct a multiple-mini interview (MMI) in a dynamic and stringent safe distancing context.Methods: This paper reports a case study of how to plan, re-plan and conduct MMIs in an environment where substantially tighter safe distancing measures were introduced just before the MMI was due to be delivered.Results: We report on how to design and implement a fully remote, online MMI which ensured the safety of candidates and assessors.Discussion: We discuss the challenges of this approach and also reflect on broader issues associated with selection into medical school during a pandemic. The aim of the paper is to provide broadly generalizable guidance to other medical schools faced with the challenge of selecting future students under difficult conditions.

Need for cognitive closure, tolerance for ambiguity, and perfectionism in medical school applicants

BACKGROUND

Physicians have to deal with uncertainty on a daily basis, which requires high tolerance for ambiguity. When medical decisions have to be made in ambiguous situations, low levels of need for cognitive closure and high levels of adaptive perfectionism are beneficial. It might be useful to measure such personality traits during medical school selection processes. In our study, we explored the expression of need for cognitive closure, tolerance for ambiguity, and perfectionism in medical school applicants who participated in a multiple mini-interview selection process with respect to the final decision of admission or rejection.

METHODS

After participating in the multiple mini-interview procedure (HAM-Int) at Hamburg Medical School in August 2019, 189 medical school applicants filled out a questionnaire including the Multidimensional Perfectionism Scale by Hewitt and Flett (MPS-H), the Multidimensional Perfectionism Scale by Frost (MPS-F), the Tolerance for Ambiguity Scale (TAS), the 16-Need for Cognitive Closure Scale (16-NCCS), and sociodemographic data. After the final admission decision, the scores of need for cognitive closure, tolerance for ambiguity, and perfectionism of admitted and rejected applicants were compared. We also assessed the predictive power of need for cognitive closure and age for the admission decision in a binary logistic regression model.

RESULTS

Compared to the admitted applicants, the rejected applicants showed a significantly higher need for cognitive closure (p = .009). A high need for cognitive closure correlated significantly positively with maladaptive perfectionism (p < .001) and significantly negatively with tolerance for ambiguity (p < .001). Low need for cognitive closure and older age were associated with a positive admission decision.

CONCLUSIONS

Regarding the personality traits need for cognitive closure, tolerance for ambiguity, and perfectionism we identified interesting differences and correlations of relevance for physicians' daily work in medical school applicants who were admitted or rejected after participating in a multiple mini-interview selection procedure. Further studies are needed to investigate these characteristics and their development longitudinally in medical students and to correlate them with students' medical performance.

Multiple Mini-Interviews: Current Perspectives on Utility and Limitations

The growing role of healthcare professionals urged admissions committees to restructure their selection process and assess key personal attributes rather than academic achievements only. Multiple mini interviews (MMIs) were designed in 2002 to assess such domains in prospective healthcare professions. Being a high-stake assessment, the utility and limitations of MMI need to be explored. The purpose of this article is to review the available evidence to establish its utility. The claim of the reliability is verified by the studies assessing the effect of number of stations, duration of stations, format and scoring systems of stations and number of raters assessing the applicants. Similarly, by gathering evidence concerning its content validity, convergent/divergent correlation and predictive ability, validity is ensured. Finally, its acceptability and feasibility along with limitations is discussed. This article concludes by providing recommendations for further work required to deal with the limitations and enhance its utility.

The relationship between perfectionism and symptoms of depression in medical school applicants

BACKGROUND

The association between perfectionism and depression in the medical profession can ultimately influence physicians' performance negatively. In medical students, especially maladaptive perfectionism is connected with distress and lower academic performance. The expression of perfectionism and symptoms of depression at the time of medical school application is not known. Therefore, we explored perfectionism and symptoms of depression in participants of multiple mini-interviews for medical school admission and investigated possible differences between applicants who were eventually admitted or rejected.

METHODS

After the multiple mini-interviews admission procedure at Hamburg Medical School in August 2018, 146 applicants filled out a questionnaire including sociodemographic data and the following validated instruments: Multidimensional Perfectionism Scale by Hewitt and Flett (MPS-H), Multidimensional Perfectionism Scale by Frost (MPS-F), Patient Health Questionnaire (PHQ-9), Generalized Anxiety Disorder Scale (GAD-7), and a 10-item version of the Big Five Inventory (BFI-10). The two groups of admitted and rejected applicants were compared and the correlation between symptoms of depression and perfectionism further explored.

RESULTS

The admitted applicants were significantly more extrovert and had lower depression scores compared to the rejected applicants. In both groups, the composite scales of Adaptive Perfectionism (r = .21, p = .011) and Maladaptive Perfectionism (r = .43, p < .001) as well as their components correlated significantly with the PHQ-9 results. Maladaptive Perfectionism accounted for about 18% of variance in the PHQ-9 score.

CONCLUSIONS

Rejected medical school applicants who participated in a multiple mini-interviews admission procedure showed higher levels of depression symptoms than admitted applicants. The degree of depressive symptoms can be partly explained by Maladaptive Perfectionism scores. Since coping in medical school and in postgraduate medical education require robust mental health, perfectionism questionnaires could be an additional tool in medical school selection processes.

Longitudinal development of different dimensions of perfectionism in undergraduate medical students with respect to their medical school admission procedure

Objective: The concept of perfectionism comprises high standards of performance as needed in medicine, but also concerns about making mistakes and dealing with social reactions about not being perfect. Perfectionism is associated with motivation and deep learning strategies but high expression of perfectionism has been found to be associated with symptoms of stress and anxiety in students. We aim to gain insights into the longitudinal development of different dimensions of perfectionism in medical students with respect to their way of medical school admission. Methods: At the Medical Faculty of Hamburg University, 167 undergraduate medical students completed validated questionnaires (MPS-H and MPS-F) of different dimensions of perfectionism and sociodemographic data including medical school admission procedures, personality traits (BSI-10 and GSE), and symptoms of depression and anxiety (PHQ-9 and GAD-7) at the start of their first year and at half term of their second year. Results: On average, after controlling for baseline and age, a significant decrease (p≤0.05) in Self-Oriented Perfectionism was found during the first two years in students who were admitted after a waiting period (M: -12.57; 95% CI: [-21.94 - -3.35]), by other ways of medical school entrance (M: -6.36; 95% CI: [-12.71 - -0.02]), by multiple mini-interviews (HAM-Int) (M: -5.52; 95% CI: [-9.90 - -1.14]), and by a natural science test (HAM-Nat) (M: -3.41; 95% CI: [-6.71 - -0.11]. Waiting period students also showed a significant longitudinal decline in the scale Personal Standards (M: -4.62; 95% CI: [-8.04 - -1.21]. Conclusions: Since medical students from all admission groups except from the high school degree group showed a significant longitudinal decrease in Self-Oriented Perfectionism, high levels of aspects of perfectionism associated with intrinsic motivation or deep learning strategies could be included medical school admission processes. Additionally, particular attention needs to be paid not to induce a loss of intrinsic motivation or deep learning strategies during undergraduate medical education.

Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI

BACKGROUND

Sociodemographic subgroup differences in multiple mini-interview (MMI) performance have been extensively studied within the MMI research literature, but heterogeneous findings demand a closer look at how specific aspects of MMI design (such as station type) affect these differences. So far, it has not been investigated whether sociodemographic subgroup differences imply that an MMI is biased, particularly in terms of its predictive validity.

METHODS

Between 2010 and 2017, the University Medical Centre Hamburg-Eppendorf (UKE) tested 1438 candidates in an MMI who also provided sociodemographic data and agreed to participate in this study. Out of these, 400 candidates were admitted and underwent a first objective structured clinical examination (OSCE) after one and a half years, including one station assessing communication skills. First, we analyzed the relationship between gender, age, native language and medical family background and MMI station performance including interaction terms with MMI station type (simulation, interview, and group) in a hierarchical linear model. Second, we tested whether the prediction of OSCE overall and communication station performance in particular differed depending on sociodemographic background by adding interaction terms between MMI performance and gender, age and medical family background in a linear regression model.

RESULTS

Young female candidates performed better than young male candidates both at interview and simulation stations. The gender difference was smaller (simulation) or non-significant (interview) in older candidates. There were no gender or age effects in MMI group station performance. All effects were very small, with the overall model explaining only 0.6% of the variance. MMI performance was not related to OSCE overall performance but significantly predicted OSCE communication station performance with no differences in the prediction for sociodemographic subgroups.

CONCLUSIONS

The Hamburg MMI is fair in its prediction of OSCE communication scores. Differences in MMI station performance for gender and age and their interaction with MMI station type can be related to the dimensions assessed at different station types and thus support the validity of the MMI. Rather than being threats to fairness, these differences could be useful for decisions relating to the design and use of an MMI.

Multiple Mini Interview as an admission tool in higher education: Insights from a systematic review

Objectives

Multiple Mini Interviews (MMI) have been conducted across the globe in the student selection process, particularly in health profession education. This paper reported the validity evidence of MMI in various educational settings.

Methods

A literature search was carried out through Scopus, Science Direct, Google Scholar, PubMed, and EBSCOhost databases based on specific search terms. Each article was appraised based on title, abstract, and full text. The selected articles were critically appraised, and relevant information to support the validity of MMI in various educational settings was synthesized. This paper followed the PRISMA guideline to ensure consistency in reporting systematic review results.

Results

A majority of the studies were from Canada, with 41.54%, followed by the United Kingdom (25.39%), the United States (13.85%), and Australia (9.23%). The rest (9.24%) were from Germany, Ireland, the United Arab Emirates, Japan, Pakistan, Taiwan, and Malaysia. Moreover, most MMI stations ranged from seven to 12 with a duration of 10 min per station (including a 2-min gap between stations).

Conclusion

The results suggest that the content, response process, and internal structure of MMI were well supported by evidence; however, the relation and consequences of MMI to important outcome variables were inconsistently supported. The evidence shows that MMI is a non-biased, practical, feasible, reliable, and content-valid admission tool. However, further research on its impact on non-cognitive outcomes is required.

[Model study programs in medicine : Innovations in medical education in Hamburg and Berlin]

Medical science is constantly evolving. Teaching and training must keep pace with these innovations and react in a flexible fashion to new requirements. Model medical education programs, which are governed by the provisions of Sect. 41 of the Regulations for the Licensing of Medical Practitioners (ÄAppO), permit the piloting of innovative teaching programs, which support the continuous development of medical training through incorporation into the standard curricula. This paper reports on the model study programs at the University Medical Centre Hamburg-Eppendorf (iMED) and Charité - University Medicine Berlin (MSM). It describes the reform objectives, the curricula and selected projects accompanying both models and concludes by exploring the significance of various training concepts for the development of medical education.

[Selection of medical students : Measurement of cognitive abilities and psychosocial competencies]

The German Constitutional Court is currently reviewing whether the actual study admission process in medicine is compatible with the constitutional right of freedom of profession, since applicants without an excellent GPA usually have to wait for seven years. If the admission system is changed, politicians would like to increase the influence of psychosocial criteria on selection as specified by the Masterplan Medizinstudium 2020.What experiences have been made with the actual selection procedures? How could Situational Judgement Tests contribute to the validity of future selection procedures to German medical schools?High school GPA is the best predictor of study performance, but is more and more under discussion due to the lack of comparability between states and schools and the growing number of applicants with top grades. Aptitude and knowledge tests, especially in the natural sciences, show incremental validity in predicting study performance. The measurement of psychosocial competencies with traditional interviews shows rather low reliability and validity. The more reliable multiple mini-interviews are superior in predicting practical study performance. Situational judgement tests (SJTs) used abroad are regarded as reliable and valid; the correlation of a German SJT piloted in Hamburg with the multiple mini-interview is cautiously encouraging.A model proposed by the Medizinischer Fakultätentag and the Bundesvertretung der Medizinstudierenden considers these results. Student selection is proposed to be based on a combination of high school GPA (40%) and a cognitive test (40%) as well as an SJT (10%) and job experience (10%). Furthermore, the faculties still have the option to carry out specific selection procedures.

Assessing noncognitive domains of respiratory therapy applicants: Messick's framework appraisal of the multiple mini-interview

Educators who assess incoming applicants into a health professional training program are looking for a wide array of cognitive and noncognitive skills that best predict success in the program and as a future practicing professional. While aptitude tests generally measure cognitive skills, noncognitive constructs are more difficult to measure appropriately. The traditional method of measuring noncognitive constructs has been the panel interview. Panel interviews have been described as inconsistent in measuring noncognitive domains and consistently reported as unreliable and susceptible to bias. An alternate interview method used in many health professions schools is the multiple mini-interview (MMI) that was specifically designed to assess noncognitive domains in health professions education. This paper discusses the purpose of using the MMI, how the MMI is conducted, specific domains of focus for the MMI, and the feasibility of creating an MMI. Finally, the paper uses Messick's framework on validity to guide the consideration of the MMI.

Selection and academic success of medical students in Hamburg, Germany

BACKGROUND

Student selection at Hamburg medical school is based on the combination of a natural science knowledge test (HAM-Nat) and pre-university educational attainment.

METHOD

Of the 1565 medical students enrolled in Hamburg from 2012 to 2015 about half were admitted by an entrance test, half by quotas. First, we analysed sociodemographic determinants of entrance test performance. Then, we used regression analysis to describe the interplay of variables in the prediction of study outcome, the role of sociodemographic factors, and differences in the calibration of educational attainment specific to German federal states.

RESULTS

Better performance in the entrance test was associated with age over 21, male gender, German nationality, first language German and both parents holding an academic degree - effect sizes were small. No differences were found for the birthplaces of parents (a proxy for migration background). Study outcome differed considerably among admission paths: Students admitted by entrance test or the quota for excellent pre-university educational attainment performed markedly better during the first 3 terms than students admitted by the waiting list quota and the quota for foreign students. Gender differences in study outcome were slight with better performance by males. The relation of pre-university educational attainment to study outcome was moderated by the federal state in which secondary schooling took place. Methods for the equating of state-specific grades are explored. The predictive validity of the HAM-Nat after correction for range restriction was r = .31. The relatively low value of this coefficient may be attributed to 3 factors: 1. self-selection of applicants which leads to a validity-enhancing effect that is not revealed by the predictor-outcome correlation, 2. reduction of variance due to a high selection ratio, and 3. high test difficulty, exceeding the demands of the medical curriculum.

CONCLUSION

The HAM-Nat achieves a small amount of incremental validity over pre-university educational attainment. This effect, obtained from correlational analysis, underestimates the validity of the test, because it does not reflect the role of self-selection and other validity enhancing features of the selection process.

Does selection pay off? A cost-benefit comparison of medical school selection and lottery systems

CONTEXT

Resources for medical education are becoming more constrained, whereas accountability in medical education is increasing. In this constrictive environment, medical schools need to consider and justify their selection procedures in terms of costs and benefits. To date, there have been no studies focusing on this aspect of selection.

OBJECTIVES

We aimed to examine and compare the costs and benefits of two different approaches to admission into medical school: a tailored, multimethod selection process versus a lottery procedure. Our goal was to assess the relative effectiveness of each approach and to compare these in terms of benefits and costs from the perspective of the medical school.

METHODS

The study was conducted at Maastricht University Medical School, at which the selection process and a weighted lottery procedure ran in parallel for 3 years (2011-2013). The costs and benefits of the selection process were compared with those of the lottery procedure over three student cohorts throughout the Bachelor's programme. The extra costs of selection represented the monetary investment of the medical school in conducting the selection procedure; the benefits were derived from the increase in income generated by the prevention of dropout and the reductions in extra costs facilitated by decreases in the repetition of blocks and objective structured clinical examinations.

RESULTS

The tailor-made selection procedure cost about €139 000 when extrapolated to a full cohort of students (n = 286). The lottery procedure came with negligible costs for the medical school. However, the average benefits of selection compared with the lottery system added up to almost €207 000.

CONCLUSIONS

This study not only shows that conducting a cost-benefit comparison is feasible in the context of selection for medical school, but also that an 'expensive' selection process can be cost-beneficial in comparison with an 'inexpensive' lottery system. We encourage other medical schools to examine the cost-effectiveness of their own selection processes in relation to student outcomes in order to extend knowledge on this important topic.

Validity evidence for the Hamburg multiple mini-interview

BACKGROUND

Multiple mini-interviews (MMI) become increasingly popular for the selection of medical students. In this work, we examine the validity evidence for the Hamburg MMI.

METHODS

We conducted three follow-up studies for the 2014 cohort of applicants to medical school over the course of two years. We calculated Spearman's rank correlation (ρ) between MMI results and (1) emotional intelligence measured by the Trait Emotional Intelligence Questionnaire (TEIQue-SF) and the Situational Test of Emotion Management (STEM), (2) supervisors' and practice team members' evaluations of psychosocial competencies and suitability for the medical profession after a one-week 1:1 teaching in a general practice (GP) and (3) objective structured clinical examination (OSCE) scores.

RESULTS

There were no significant correlations between MMI results and the TEIQue-SF (ρ = .07, p > .05) or the STEM (ρ = .05, p > .05). MMI results could significantly predict GP evaluations of psychosocial competencies (ρ = .32, p < .05) and suitability for the medical profession (ρ = .42, p < .01) as well as OSCE scores (ρ = .23, p < .05). The MMI remained a significant predictor of these outcomes in a robust regression model including gender and age as control variables.

CONCLUSIONS

Our findings suggest that MMIs can measure competencies that are relevant in a practical context. However, these competencies do not seem to be related to emotional intelligence as measured by self-report or situational judgement test.

Personal domains assessed in multiple mini interviews (MMIs) for healthcare student selection: A narrative synthesis systematic review

OBJECTIVES

To examine the personal domains multiple mini interviews (MMIs) are being designed to assess, explore how they were determined and contextualise such domains in current and future healthcare student selection processes DESIGN: A systematic review of empirical research reporting on MMI model design was conducted from database inception to November 2017.

DATA SOURCES

Twelve electronic bibliographic databases.

REVIEW METHODS

Evidence was extracted from original studies, and integrated in a narrative synthesis guided by the PRISMA statement for reporting systematic reviews. Personal domains were clustered into themes using a modified Delphi technique.

RESULTS

A total of 584 articles were screened. 65 unique studies (80 articles) matched our inclusion criteria of which seven were conducted within nursing/midwifery faculties. Six in 10 studies featured applicants to medical school. Across selection processes, we identified 32 personal domains assessed by MMIs, the most frequent being: communication skills (84%), teamwork/collaboration (70%), and ethical/moral judgement (65%). Domains capturing ability to cope with stressful situations (14%), make decisions (14%), and resolve conflict in the workplace (13%) featured in fewer than ten studies overall. Intra- and inter-disciplinary inconsistencies in domain profiles were noted, as well as differences by entry level. MMIs deployed in nursing and midwifery assessed compassion and decision-making more frequently than in all other disciplines. Own programme philosophy and professional body guidance were most frequently cited (~50%) as sources for personal domains; a blueprinting process was reported in only 8% of studies.

CONCLUSIONS

Nursing, midwifery and allied healthcare professionals should develop their theoretical frameworks for MMIs to ensure they are evidence-based and fit-for-purpose. We suggest a re-evaluation of domain priorities to ensure that students who are selected, not only have the capacity to offer the highest standards of care provision, but are able to maintain these standards when facing clinical practice and organisational pressures.

Comparison of open and closed book test for admission in medical school

This article was migrated. The article was marked as recommended. Introduction: Students' learning methods are highly influenced by the type of evaluation. Consequently, evaluation is a deciding step in the learning process. Multiple Choice Format Tests (MCQs) are almost exclusive in the admission to Health studies. PluriPASS suggests an educational reform of Health studies. This research aims at analyzing docimological features of Open-book Multiple Choice Format tests (OBT) compared to other usual tests used during the first year in Health studies curriculum (CC). Methods: This educational research took place during PluriPASS year in Health studies curriculum at Angers University, during the academic period 2015-2016. The optional course « Disability and Health » (DH) was partly assessed by open-book tests with complex wording, requiring careful thought from students. Results: Out of the 1161 students enrolled in first year curriculum, 190 have chosen the DH. For the students who followed the DH, the CC and OBT distributions are respectively a Skewness score at -0.11 and -0.12 and a Kurtosis score at -0.9 and -0.22. Bland Altman test or Deming method demonstrate a concordance between both method. Conclusion: Distribution characteristics of OBT are satisfactory and allow to consider introducing this method with the objective of promoting reflection and depth learning.

Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45

BACKGROUND

Selection into specialty training is a high-stakes and resource-intensive process. While substantial literature exists on selection into medical schools, and there are individual studies in postgraduate settings, there seems to be paucity of evidence concerning selection systems and the utility of selection tools in postgraduate training environments.

AIM

To explore, analyze and synthesize the evidence related to selection into postgraduate medical specialty training.

METHOD

Core bibliographic databases including PubMed; Ovid Medline; Embase, CINAHL; ERIC and PsycINFO were searched, and a total of 2640 abstracts were retrieved. After removing duplicates and screening against the inclusion criteria, 202 full papers were coded, of which 116 were included.

RESULTS

Gaps in underlying selection frameworks were illuminated. Frameworks defined by locally derived selection criteria, and heavily weighed on academic parameters seem to be giving way to the evidencing of competency-based selection approaches in some settings. Regarding selection tools, we found favorable psychometric evidence for multiple mini-interviews, situational judgment tests and clinical problem-solving tests, although the bulk of evidence was mostly limited to the United Kingdom. The evidence around the robustness of curriculum vitae, letters of recommendation and personal statements was equivocal. The findings on the predictors of past performance were limited to academic criteria with paucity of long-term evaluations. The evidence around nonacademic criteria was inadequate to make an informed judgment.

CONCLUSIONS

While much has been gained in understanding the utility of individual selection methods, though the evidence around many of them is equivocal, the underlying theoretical and conceptual frameworks for designing holistic and equitable selection systems are yet to be developed.

Correcting the predictive validity of a selection test for the effect of indirect range restriction

BACKGROUND

The validity of selection tests is underestimated if it is determined by simply calculating the predictor-outcome correlation found in the admitted group. This correlation is usually attenuated by two factors: (1) the combination of selection variables which can compensate for each other and (2) range restriction in predictor and outcome due to the absence of outcome measures for rejected applicants.

METHODS

Here we demonstrate the logic of these artifacts in a situation typical for student selection tests and compare four different methods for their correction: two formulas for the correction of direct and indirect range restriction, expectation maximization algorithm (EM) and multiple imputation by chained equations (MICE). First we show with simulated data how a realistic estimation of predictive validity could be achieved; second we apply the same methods to empirical data from one medical school.

RESULTS

The results of the four methods are very similar except for the direct range restriction formula which underestimated validity.

CONCLUSION

For practical purposes Thorndike's case C formula is a relatively straightforward solution to the range restriction problem, provided distributional assumptions are met. With EM and MICE more precision is obtained when distributional requirements are not met, but access to a sophisticated statistical package such as R is needed. The use of true score correlation has its own problems and does not seem to provide a better correction than other methods.

Is perfect good? - Dimensions of perfectionism in newly admitted medical students

BACKGROUND

Society expects physicians to perform perfectly but high levels of perfectionism are associated with symptoms of distress in medical students. This study investigated whether medical students admitted to medical school by different selection criteria differ in the occurrence of perfectionism.

METHODS

Newly enrolled undergraduate medical students (n = 358) filled out the following instruments: Multidimensional Perfectionism Scale (MPS-H), Multidimensional Perfectionism Scale (MPS-F), Big Five Inventory (BFI-10), General Self-Efficacy Scale (GSE), Patient Health Questionnaire 9 (PHQ-9), and Generalized Anxiety Disorder 7 (GAD-7). Sociodemographic data such as age, gender, high school degrees, and the way of admission to medical school were also included in the questionnaire.

RESULTS

The 298 participating students had significantly lower scores in Socially-Prescribed Perfectionism than the general population independently of their way of admission to medical school. Students who were selected for medical school by their high school degree showed the highest score for Adaptive Perfectionism. Maladaptive Perfectionism was the strongest predictor for the occurrence symptoms of depression and anxiety regardless of the way of admission.

CONCLUSIONS

Students from all admission groups should be observed longitudinally for performance and to assess whether perfectionism questionnaires might be an additional useful instrument for medical school admission processes.

Reliability of Multiple Mini-Interviews and traditional interviews within and between institutions: a study of five California medical schools

BACKGROUND

Many medical schools use admissions Multiple Mini-Interviews (MMIs) rather than traditional interviews (TIs), partly because MMIs are thought to be more reliable. Yet prior studies examined single-school samples of candidates completing either an MMI or TI (not both). Using data from five California public medical schools, the authors examined the within- and between-school reliabilities of TIs and MMIs.

METHODS

The analyses included applicants interviewing at ≥1 of the five schools during 2011-2013. Three schools employed TIs (TI1, TI2, TI3) and two employed MMIs (MMI1, MMI2). Mixed linear models accounting for nesting of observations within applicants examined standardized TI and MMI scores (mean = 0, SD = 1), adjusting for applicant socio-demographics, academic metrics, year, number of interviews, and interview date.

RESULTS

A total of 4993 individuals (completing 7516 interviews [TI = 4137, MMI = 3379]) interviewed at ≥1 school; 428 (14.5%) interviewed at both MMI schools and 687 (20.2%) at more than one TI school. Within schools, inter-interviewer consistency was generally qualitatively lower for TI1, TI2, and TI3 (Pearson's r 0.07, 0.13, and 0.29, and Cronbach's α, 0.40, 0.44, and 0.61, respectively) than for MMI1 and MMI 2 (Cronbach's α 0.68 and 0.60, respectively). Between schools, the adjusted intraclass correlation coefficient was 0.27 (95% CI 0.20-0.35) for TIs and 0.47 (95% CI 0.41-0.54) for MMIs.

CONCLUSIONS

Within and between-school reliability was qualitatively higher for MMIs than for TIs. Nonetheless, TI reliabilities were higher than anticipated from prior literature, suggesting TIs may not need to be abandoned on reliability grounds if other factors favor their use.

Multiple mini-interviews as a predictor of academic achievements during the first 2 years of medical school

Background

Recently, conventional interviews have been replaced with the multiple mini-interviews (MMI) for medical student selection in Korea. We first introduced the MMI as a new admissions tool in Korea. The aim of this study is to determine whether the MMI accurately predicts academic achievement on both written and performance-based examinations during the first 2 years of medical school.

Methods

The original scores of each station were standardized to T-scores in the candidates group. Three cohorts of students were included depending upon the year they entered medical school. Pearson’s correlations were calculated to estimate the correlations between MMI scores and academic achievements. Additional correlated factors were run through multiple stepwise linear regression analysis to estimate predictive validity.

Results

There were no differences between T-scores or grade point averages (GPA) among the cohorts. The correlation coefficients between total MMI scores and academic achievement in Year 1 and the Year 2 performance-based examinations ranged from 0.17 to 0.43. Station 1 significantly predicted academic achievement over the second year. Station 3 significantly predicted only performance-based examination performance over the second year.

Conclusion

MMI is a useful tool to assist with medical student selection. In particular, critical thinking, professionalism, and presentation and communication skills may be meaningful topics for predicting academic achievements, especially in performance-based subjects.

How effective are selection methods in medical education? A systematic review

CONTEXT

Selection methods used by medical schools should reliably identify whether candidates are likely to be successful in medical training and ultimately become competent clinicians. However, there is little consensus regarding methods that reliably evaluate non-academic attributes, and longitudinal studies examining predictors of success after qualification are insufficient. This systematic review synthesises the extant research evidence on the relative strengths of various selection methods. We offer a research agenda and identify key considerations to inform policy and practice in the next 50 years.

METHODS

A formalised literature search was conducted for studies published between 1997 and 2015. A total of 194 articles met the inclusion criteria and were appraised in relation to: (i) selection method used; (ii) research question(s) addressed, and (iii) type of study design.

RESULTS

Eight selection methods were identified: (i) aptitude tests; (ii) academic records; (iii) personal statements; (iv) references; (v) situational judgement tests (SJTs); (vi) personality and emotional intelligence assessments; (vii) interviews and multiple mini-interviews (MMIs), and (viii) selection centres (SCs). The evidence relating to each method was reviewed against four evaluation criteria: effectiveness (reliability and validity); procedural issues; acceptability, and cost-effectiveness.

CONCLUSIONS

Evidence shows clearly that academic records, MMIs, aptitude tests, SJTs and SCs are more effective selection methods and are generally fairer than traditional interviews, references and personal statements. However, achievement in different selection methods may differentially predict performance at the various stages of medical education and clinical practice. Research into selection has been over-reliant on cross-sectional study designs and has tended to focus on reliability estimates rather than validity as an indicator of quality. A comprehensive framework of outcome criteria should be developed to allow researchers to interpret empirical evidence and compare selection methods fairly. This review highlights gaps in evidence for the combination of selection tools that is most effective and the weighting to be given to each tool.

Multiple mini-interviews: same concept, different approaches

OBJECTIVES

Increasing numbers of educational institutions in the medical field choose to replace their conventional admissions interviews with a multiple mini-interview (MMI) format because the latter has superior reliability values and reduces interviewer bias. As the MMI format can be adapted to the conditions of each institution, the question of under which circumstances an MMI is most expedient remains unresolved. This article systematically reviews the existing MMI literature to identify the aspects of MMI design that have impact on the reliability, validity and cost-efficiency of the format.

METHODS

Three electronic databases (OVID, PubMed, Web of Science) were searched for any publications in which MMIs and related approaches were discussed. Sixty-six publications were included in the analysis.

RESULTS

Forty studies reported reliability values. Generally, raising the number of stations has more impact on reliability than raising the number of raters per station. Other factors with positive influence include the exclusion of stations that are too easy, and the use of normative anchored rating scales or skills-based rater training. Data on criterion-related validities and analyses of dimensionality were found in 31 studies. Irrespective of design differences, the relationship between MMI results and academic measures is small to zero. The McMaster University MMI predicts in-programme and licensing examination performance. Construct validity analyses are mostly exploratory and their results are inconclusive. Seven publications gave information on required resources or provided suggestions on how to save costs. The most relevant cost factors that are additional to those of conventional interviews are the costs of station development and actor payments.

CONCLUSIONS

The MMI literature provides useful recommendations for reliable and cost-efficient MMI designs, but some important aspects have not yet been fully explored. More theory-driven research is needed concerning dimensionality and construct validity, the predictive validity of MMIs other than those of McMaster University, the comparison of station types, and a cost-efficient station development process.