Medical School Applicant Characteristics Associated With Performance in Multiple Mini-Interviews Versus Traditional Interviews: A Multi-Institutional Study

Examining interviewer bias in medical school admissions: The interplay between applicant and interviewer gender and its effects on interview outcomes

Selection interviews have long been integral to medical school admissions, yet their limited predictive validity and susceptibility to bias raise concerns. This study delves into potential interviewer bias within the dynamics of interviewee and interviewer gender. We analyze a dataset of 5,200 applicants and over 370 selection committees engaged in semi-structured interviews from 2006 to 2019 at a large German medical school with multiple linear and non-linear regression analyses. Our findings reveal that all-female committees tended to award male candidates, on average, one point more than their female counterparts, significantly enhancing the chances of submission for male applicants despite lower academic grades, which constituted 51% of the selection process points. All-male and mixed-gender committees exhibited similar ratings for both genders. The role of valuing voluntary services emerged prominently: all-male and mixed committees acknowledged women's volunteer work but not men's, while all-female committees demonstrated the opposite pattern. Our results attribute variations in interview outcomes to the absence of standardization, such as insufficient interviewer training, divergent rating strategies, variations in interviewer experience, and imbalances in candidate allocation to selection committees, rather than to a "gender bias", for example by favoritism of males because of their gender.

Adoption of Internal Medicine Milestone Ratings and Changes in Bias Against Black, Latino, and Asian Internal Medicine Residents

BACKGROUND

The 2014 adoption of the Milestone ratings system may have affected evaluation bias against minoritized groups.

OBJECTIVE

To assess bias in internal medicine (IM) residency knowledge ratings against Black or Latino residents-who are underrepresented in medicine (URiM)-and Asian residents before versus after Milestone adoption in 2014.

DESIGN

Cross-sectional and interrupted time-series comparisons.

SETTING

U.S. IM residencies.

PARTICIPANTS

59 835 IM residents completing residencies during 2008 to 2013 and 2015 to 2020.

INTERVENTION

Adoption of the Milestone ratings system.

MEASUREMENTS

Pre-Milestone (2008 to 2013) and post-Milestone (2015 to 2020) bias was estimated as differences in standardized knowledge ratings between U.S.-born and non-U.S.-born minoritized groups versus non-Latino U.S.-born White (NLW) residents, with adjustment for performance on the American Board of Internal Medicine IM certification examination and other physician characteristics. Interrupted time-series analysis measured deviations from pre-Milestone linear bias trends.

RESULTS

During the pre-Milestone period, ratings biases against minoritized groups were large (-0.40 SDs [95% CI, -0.48 to -0.31 SDs; P < 0.001] for URiM residents, -0.24 SDs [CI, -0.30 to -0.18 SDs; P < 0.001] for U.S.-born Asian residents, and -0.36 SDs [CI, -0.45 to -0.27 SDs; P < 0.001] for non-U.S.-born Asian residents). These estimates decreased to less than -0.15 SDs after adoption of Milestone ratings for all groups except U.S.-born Black residents, among whom substantial (though lower) bias persisted (-0.26 SDs [CI, -0.36 to -0.17 SDs; P < 0.001]). Substantial deviations from pre-Milestone linear bias trends coincident with adoption of Milestone ratings were also observed.

LIMITATIONS

Unobserved variables correlated with ratings bias and Milestone ratings adoption, changes in identification of race/ethnicity, and generalizability to Milestones 2.0.

CONCLUSION

Knowledge ratings bias against URiM and Asian residents was ameliorated with the adoption of the Milestone ratings system. However, substantial ratings bias against U.S.-born Black residents persisted.

PRIMARY FUNDING SOURCE

None.

Interviewer bias when using multiple mini-interviews in selecting student nurses in a Chinese setting

BACKGROUND

Interviewer effects may cause unfairness in assessments in multiple mini-interviews (MMIs). Due to cultural differences, the bias factors of interviewers may vary between the East and the West. MMIs are a relatively new type of assessment setting in China and few studies have been conducted to examine the interviewer effects of MMIs in this context.

OBJECTIVES

We adopted a multi-faceted Rasch measurement (MFRM) to measure interviewer effects in assessments in Hong Kong.

METHODS

Data were collected from a nursing school in Hong Kong. There were 431 candidates and 12 interviewers engaged in a six-station MMI setting. The scores collected from the interviews were analyzed in terms of 1) interviewer stringency/leniency, 2) candidate gender, 3) interview time, and 4) rating category in the station. The Student's t-statistic values were calculated to investigate the marking tendencies of individual interviewers.

RESULTS

The research findings suggest that interviewers differ in their degree of stringency/leniency, but the number of candidates examined by each interviewer does not affect interviewer stringency/leniency in terms of the interviewer's assessment. There is not sufficient evidence indicating that candidate gender and interview time are bias factors affecting assessment score in this study. Among the six rating categories examined, honesty/integrity is the most stringent category, while self-awareness is the most lenient category. Interview bias from individuals was identified. When we consider the interview scores given by individual interviewers, it is evident that some interviewers may have been biased toward a certain gender or rating categories.

CONCLUSIONS

MMIs are useful when selecting student nurses in a Chinese setting. However, interviewer bias may exist. We used an MFRM to better understand interviewer bias across various dimensions. The present study contributes to the development and use of MMIs in non-Western countries and can be used as a reference to extend this research to other locations.

Bias, Fairness, and Validity in Graduate-School Admissions: A Psychometric Perspective

As many schools and departments are considering the removal of the Graduate Record Examination (GRE) from their graduate-school admission processes to enhance equity and diversity in higher education, controversies arise. From a psychometric perspective, we see a critical need for clarifying the meanings of measurement "bias" and "fairness" to create common ground for constructive discussions within the field of psychology, higher education, and beyond. We critically evaluate six major sources of information that are widely used to help inform graduate-school admissions decisions: grade point average, personal statements, resumes/curriculum vitae, letters of recommendation, interviews, and GRE. We review empirical research evidence available to date on the validity, bias, and fairness issues associated with each of these admission measures and identify potential issues that have been overlooked in the literature. We conclude by suggesting several directions for practical steps to improve the current admissions decisions and highlighting areas in which future research would be beneficial.

Strategies For Enhancing Equity, Diversity, and Inclusion in Medical School Admissions–A Canadian Medical School's Journey

Background

Medical schools aim to select and train future physicians representative of and able to serve their diverse population needs. Enhancing equity, diversity, and inclusion (EDI) in admissions processes includes identifying and mitigating barriers for those underrepresented in medicine (URM).

Summary of Innovations

In 2017, Schulich School of Medicine and Dentistry (Western University, Ontario, Canada) critically reviewed its general Admissions pathways for the Doctor of Medicine (MD) program. Till that time, interview invitations were primarily based on academic metrics rather than a holistic review as for its Indigenous MD Admissions pathway. To help diversify the Canadian physician workforce, Schulich Medicine utilized a multipronged approach with five key changes implemented over 2 years into the general MD Admissions pathways: 1. A voluntary applicant diversity survey (race, socioeconomic status, and community size) to examine potential barriers within the Admissions process; 2. Diversification of the admissions committee and evaluator pool with the inclusion of an Equity Representative on the admissions committee; 3. A biosketch for applicants' life experiences; 4. Implicit bias awareness training for Committee members, file reviewers and interviewers; and 5. A specific pathway for applicants with financial, sociocultural, and medical barriers (termed ACCESS pathway). Diversity data before (Class of 2022) vs. after (Class of 2024) these initiatives and of the applicant pool vs. admitted class were examined.

Conclusion

For the Class of 2024, the percentage of admitted racialized students (55.2%), those with socioeconomic challenges (32.3%), and those from remote/rural/small town communities (18.6%) reflected applicant pool demographics (52.8, 29.9, and 17.2%, respectively). Additionally, 5.3% (vs. 5.6% applicant pool) of admitted students had applied through ACCESS. These data suggest that barriers within the admissions process for these URM populations were potentially mitigated by these initiatives. The initiatives broadly improved representation of racialized students, LGBTQ2S+, and those with disability with statistically significant increases in representation of those with socioeconomic challenges (32.3 vs. 19.3%, p = 0.04), and those with language diversity (42.1 vs. 35.0%, p = 0.04). Thus, these changes within the general MD admissions pathways will help diversify the future Canadian physician workforce and inform future initiatives to address health equity and social accountability within Canada.

Gender Associations with Selection into Australian Orthopaedic Surgical Training: 2007-2019

BACKGROUND

Communities have better health outcomes when their clinicians reflect the diversity of the communities they serve. More than 50% of Australian medical school graduates are female, yet women represent less than 5% of Australian orthopaedic surgeons. Selection into orthopaedic surgical training in Australia is an annual, nation-wide process, based on curriculum vitae (CV), referee reports and performance in multiple mini-interviews (MMI). The influence of applicant gender on these selection scores was examined.

METHODS

The CV, referee reports and MMI scores used for selection for each year from 2007 to 2019 were analysed from the perspective of the applicant's gender.

RESULTS

Over the years of the study, male applicants had higher CV scores and referee report scores, which determined the gender proportions invited to interview. By contrast, the interview process and selection from interview did not demonstrate a gender association.

CONCLUSION

We describe the impact of selection tools, utilized over the past 13 years, on the gender diversity of trainees commencing orthopaedic surgery training in Australia. Leaders in postgraduate training should examine commonly used selection procedures to identify and reduce the unconscious biases that may affect their performance and value.

Implementation of Virtual Multiple Mini-Interviews for Fellowship Recruitment

Introduction

The SARS-CoV Disease (COVID-19) pandemic has upended health care systems and one of the casualties has been the trainee recruitment process since social distancing and travel restrictions make an in-person experience improbable. At the University of California, Davis (UCD), our Pain Division transitioned our internally validated multiple mini-interview (MMI) process to a virtual environment

Methods

Applicants signed a confidentiality agreement prior to their interviews and were invited to watch a series of videos orienting them to the process and to the program itself. All faculty raters interviewed candidates using a total of 6 non-medical MMI scenarios with corresponding questions and scoring rubrics through the Zoom platform. Applicants were then welcomed to voluntary informal conversations with the current fellow trainees and faculty. An optional survey was sent to the applicants post-interview to assess their overall satisfaction with the virtual process.

Results

The survey analyzed the following using a 5 point Likert scale: Overall Satisfaction, Video Overview Program, Interview Day Details, Video Tour, Web-Based Interviews, Process was Fair, and finally a question on Informed Decision regarding receiving sufficient information to formulate their rank list. All respondents (80% response rate) reported being either satisfied or very satisfied with each of the aspects of the interview process detailed above.

Conclusions

While technical difficulties and confidentiality issues are of concern when offering an entirely web based recruitment, our group was able to transition traditional in-person MMI to a virtual platform using a similar structure which was well-received by applicants

Admissions experiences of aspiring physicians from low-income backgrounds

INTRODUCTION

Students from low-income backgrounds (LIB) have been under-represented in Canadian medical schools for over fifty years. Despite our awareness of this problem, little is known about the experiences of aspiring physicians from LIB in Canada who are working towards medical school admission. Consequently, we have little insight into the barriers and facilitators that may be used to increase the representation of students from LIB in Canadian medical schools.

METHODS

This paper describes a qualitative description interview study aimed at understanding the experiences of aspiring physicians from LIB as they attempt to gain entry to medical school. We conducted semi-structured interviews with 21 participants at different stages of their undergraduate, master's, and non-medical professional education, and used the theories of intersectionality and identity capital as a theoretical framework for identifying barriers and facilitators to a career in medicine.

RESULTS

Participants experienced social, identity-related, economic, structural and informational barriers to a career in medicine. Intrinsic facilitators included motivation, self-confidence, attitude, strategy, information-seeking and sorting, and financial literacy and increasing income. Extrinsic facilitators were social, informational, financial and institutional in nature.

CONCLUSION

This study fills existing knowledge gaps in the literature by identifying the pre-admissions barriers and facilitators encountered by aspiring physicians from LIB in Canada. The barriers and facilitators outlined in this study offer a framework for identifying target areas in developing support for admitting medical students from LIB. Given that medical students from LIB are more likely to serve underserved populations, our study is relevant to Canadian medical schools' social accountability commitment to producing physicians that meet the health needs of marginalised and vulnerable patients.

Blackface in White Space: Using Admissions to Address Racism in Medical Education

Given the long history and pervasive nature of racism in medical culture, this essay argues that diversifying efforts alone cannot address systemic racism in medical education. Positive affirmation of anti-racist values and racial consciousness in the admissions process is necessary to create a truly inclusive culture in medical education and begin to undo centuries of racial prejudice in medicine. Drawing from historic examples, scholarship on the sociology of racialized space, recent research on race and medical education, and personal experience, we propose that medical educational institutions make a more concerted effort to consider racial attitudes and awareness as part of the admissions process as well as curricular reform efforts. We also provide examples of potential ways to practically implement this proposal in the admissions process.

More Than Their Test Scores: Redefining Success with Multiple Mini-Interviews

INTRODUCTION

Medical schools are shifting to a holistic approach for admissions. As non-cognitive abilities become more valued in the medical field and admissions become more competitive, multiple mini-interviews (MMI) are becoming increasingly common in selection criteria. The purpose of this article is to examine the relationship among admissions criteria to create a predictive model of acceptance to medical school. This article also aims to examine, among matriculated students, the relationship between MMI and traditional academic success, and the relationship between MMI scores and clinical competence and academic accolades.

METHODS

This longitudinal study followed a cohort of students from the MMI process through the first two pre-clinical years at a medical school in the southeastern USA. Multiple logistic regression with backward elimination variable selection was used to examine the relationship between admission criteria and acceptance. A multivariate multinomial ordered probit model was used to assess the relationship between MMI and traditional academic success among matriculated students. Simple linear regression models were used to assess relationships between MMI and Objective Structured Clinical Examination (OSCE) scores and honors nomination.

RESULTS

MMI are among eight significant predictors of acceptance. Among matriculated students, there were weak negative associations between MMI and traditional academic success; however, all but one relationship was statistically non-significant. There was not a significant association between MMI and OSCE scores or academic accolades.

CONCLUSIONS

While MMI can identify students with non-cognitive skills needed to be a good physician, continued assessment needs to be incorporated in their education.

Transition to multiple mini interview (MMI) interviewing for medical school admissions

INTRODUCTION

The multiple mini interview (MMI) has been incorporated into the holistic review process in the selection of students to US medical schools. The MMI has been used to evaluate interpersonal and intrapersonal attributes which are deemed as necessary for future physicians. We hypothesized that there would be little difference in overall MMI evaluation data compared with traditional interview ratings.

METHODS

The University of North Carolina School of Medicine developed an interview process that included a traditional interview and MMI format during the 2019 admissions cycle. Evaluation data along with key demographic variables for 608 MD program applicants were analyzed using descriptive and inferential statistical analyses.

RESULTS

The MMI format slightly favored female over male applicants (p = 0.002) but did not select for or against applicants based on age, race/ethnicity, underserved/rural area upbringing, or indicators of disadvantage. Out of 608 applicants, 356 (59%) completed a post-interview survey in which the experience was positively rated.

DISCUSSION

Based on our experience, the use of a hybrid model of traditional interviews complemented with MMI stations provided greater details in the assessment of medical school applicants while obtaining equivalent data and acceptability amongst applicants.

Do Professionalism Lapses in Medical School Predict Problems in Residency and Clinical Practice?

PURPOSE

Recognizing that physicians must exhibit high levels of professionalism, researchers have attempted to identify the precursors of clinicians' professionalism difficulties, typically using retrospective designs that trace sanctioned physicians back to medical school. To better establish relative risk for professionalism lapses in practice, however, this relationship must also be studied prospectively. Therefore, this study investigated the sequelae of medical school professionalism lapses by following students with medical school professionalism problems into residency and practice.

METHOD

Beginning in 2014, 108 graduates from Harvard Medical School and Case Western Reserve University School of Medicine who appeared before their schools' review boards between 1993 and 2007 for professionalism-related reasons were identified, as well as 216 controls matched by sex, minority status, and graduation year. Prematriculation information and medical school performance data were collected for both groups. Outcomes for the groups were studied at 2 points in time: ratings by residency directors, and state medical board sanctions and malpractice suits during clinical practice.

RESULTS

Compared with controls, students who appeared before their schools' review boards were over 5 times more likely to undergo disciplinary review during residency (16% vs 3%, respectively) and almost 4 times more likely to require remediation or counseling (35% vs 9%, respectively). During clinical practice, 10% of those who had made review board appearances were sued or sanctioned vs 5% of controls. Logistic regression for these outcomes indicated, however, that professional lapses in medical school were not the only, or even the most important, predictor of problems in practice.

CONCLUSIONS

Students with professionalism lapses in medical school are significantly more likely to experience professionalism-related problems during residency and practice, although other factors may also play an important predictive role.

Medical Student Involvement and Perceptions of the Admissions Process

Minimal attention has been given to student involvement in medical school admissions practices. This study explores the role of medical students and perceptions of their involvement on admissions committees. Survey responses from US medical schools were varied regarding student role, service on the committee, and voting privileges. Medical student admissions committee members surveyed at our institution felt they were able to offer different perspectives to applicant evaluation than faculty. Findings suggest that medical students may be able to contribute to the admissions process in a variety of ways depending on institution-specific missions and goals.

Addressing the Diversity-Validity Dilemma Using Situational Judgment Tests

PURPOSE

To examine the magnitudes of score differences across different demographic groups for three academic (grade point average [GPA], old Medical College Admission Test [MCAT], and MCAT 2015) and one nonacademic (situational judgment test [SJT]) screening measures and one nonacademic (multiple mini-interview [MMI]) interview measure (analysis 1), and the demographic implications of including an SJT in the screening stage for the pool of applicants who are invited to interview (analysis 2).

METHOD

The authors ran the analyses using data from New York Medical College School of Medicine applicants from the 2015-2016 admissions cycle. For analysis 1, effect sizes (Cohen d) were calculated for GPA, old MCAT, MCAT 2015, CASPer (an online SJT), and MMI. Comparisons were made across gender, race, ethnicity (African American, Hispanic/Latino), and socioeconomic status (SES). For analysis 2, a series of simulations were conducted to estimate the number of underrepresented in medicine (UIM) applicants who would have been invited to interview with different weightings of GPA, MCAT, and CASPer scores.

RESULTS

A total of 9,096 applicants were included in analysis 1. Group differences were significantly smaller or reversed for CASPer and MMI compared with the academic assessments (MCAT, GPA) across nearly all demographic variables/indicators. The simulations suggested that a higher weighting of CASPer may help increase gender, racial, and ethnic diversity in the interview pool; results for low-SES applicants were mixed.

CONCLUSIONS

The inclusion of an SJT in the admissions process has the potential to widen access to medical education for a number of UIM groups.

Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI

BACKGROUND

Sociodemographic subgroup differences in multiple mini-interview (MMI) performance have been extensively studied within the MMI research literature, but heterogeneous findings demand a closer look at how specific aspects of MMI design (such as station type) affect these differences. So far, it has not been investigated whether sociodemographic subgroup differences imply that an MMI is biased, particularly in terms of its predictive validity.

METHODS

Between 2010 and 2017, the University Medical Centre Hamburg-Eppendorf (UKE) tested 1438 candidates in an MMI who also provided sociodemographic data and agreed to participate in this study. Out of these, 400 candidates were admitted and underwent a first objective structured clinical examination (OSCE) after one and a half years, including one station assessing communication skills. First, we analyzed the relationship between gender, age, native language and medical family background and MMI station performance including interaction terms with MMI station type (simulation, interview, and group) in a hierarchical linear model. Second, we tested whether the prediction of OSCE overall and communication station performance in particular differed depending on sociodemographic background by adding interaction terms between MMI performance and gender, age and medical family background in a linear regression model.

RESULTS

Young female candidates performed better than young male candidates both at interview and simulation stations. The gender difference was smaller (simulation) or non-significant (interview) in older candidates. There were no gender or age effects in MMI group station performance. All effects were very small, with the overall model explaining only 0.6% of the variance. MMI performance was not related to OSCE overall performance but significantly predicted OSCE communication station performance with no differences in the prediction for sociodemographic subgroups.

CONCLUSIONS

The Hamburg MMI is fair in its prediction of OSCE communication scores. Differences in MMI station performance for gender and age and their interaction with MMI station type can be related to the dimensions assessed at different station types and thus support the validity of the MMI. Rather than being threats to fairness, these differences could be useful for decisions relating to the design and use of an MMI.

Do Admissions Multiple Mini-Interview and Traditional Interview Scores Predict Subsequent Academic Performance? A Study of Five California Medical Schools

PURPOSE

To compare the predictive validities of medical school admissions multiple mini-interviews (MMIs) and traditional interviews (TIs).

METHOD

This longitudinal observational study of 2011-2013 matriculants to five California public medical schools examined the associations of MMI scores (two schools) and TI scores (three schools) with subsequent academic performance. Regression models adjusted for sociodemographics and undergraduate academic metrics examined associations of standardized mean MMI and TI scores with United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge (CK) scores and, for required clerkships, with mean National Board of Medical Examiners Clinical Science subject (shelf) exam score and number of honors grades.

RESULTS

Of the 1,460 medical students, 746 (51.1%) interviewed at more than one study school; 579 (39.7%) completed at least one MMI and at least one TI. Neither interview type was associated with Step 1 scores. Higher MMI scores were associated with more clerkship honors grades (adjusted incidence rate ratio [AIRR] 1.28 more [95% CI 1.18, 1.39; P < .01] per SD increase) and higher shelf exam and Step 2 CK scores (adjusted mean 0.73 points higher [95% CI 0.28, 1.18; P < .01] and 1.25 points higher [95% CI 0.09, 2.41; P = .035], respectively, per SD increase). Higher TI scores were associated only with more honors grades (AIRR 1.11 more [95% CI 1.01, 1.20; P = .03] per SD increase).

CONCLUSIONS

MMI scores were more strongly associated with subsequent academic performance measures than were TI scores.

Do Multiple Mini-Interview and Traditional Interview Scores Differ in Their Associations With Acceptance Offers Within and Across Five California Medical Schools?

PURPOSE

In single-school studies, multiple mini-interview (MMI) and traditional interview (TI) scores are associated with acceptance offers. Unexamined is whether scores at one school are associated with acceptance at other schools; such analyses would mitigate single-school design biases and better estimate how well interviews capture desired applicant attributes. Using data from the 5 California Longitudinal Evaluation of Admissions Practices (CA-LEAP) medical schools, the authors examined associations of MMI and TI scores with acceptance offers within and across schools.

METHOD

The analyses included applicants who interviewed at ≥1 CA-LEAP school during the 2011-2013 admissions cycles. Three CA-LEAP schools employed TIs and 2 employed MMIs. Interview scores were standardized (z scores: mean = 0, SD = 1), and associations with acceptance offers were examined within and across schools in analyses stratified by school, adjusting for applicant sociodemographics, academic metrics, year, and total number of interviews.

RESULTS

Of 4,993 applicants interviewed, 428 (8.6%) interviewed at both MMI schools, 681 (13.6%) at ≥2 TI schools, and 1,327 (26.6%) at ≥1 MMI and ≥1 TI school. For each school, acceptance was associated with interview score at that school and also with interview scores at the other 4 schools. Cross-school associations of MMI versus TI scores with acceptance did not differ statistically.

CONCLUSIONS

Interview score at a given school was associated with acceptance at the other 4 schools, with no significant differences in associations for MMIs versus TIs. The findings suggest both MMIs and TIs captured attributes valued by admissions teams across CA-LEAP schools.

Vive la Différence: The Freedom and Inherent Responsibilities When Designing and Implementing Multiple Mini-Interviews

The literature on multiple mini-interviews (MMIs) is replete with heterogeneous study results related to the constructs measured, correlations with other measures, and demographic relationships. Rather than view these results as contradictory, the authors ask, What if all of the results are correct? They point out that the MMI is not an assessment tool but, rather, an assessment method. Design and implementation of locally conducted MMIs in medical school admissions processes should reflect local needs. As with other local assessments, MMIs should be considered separate from nationally conducted assessments that reflect more universal competencies. With the freedom to exercise unique values in locally constructed MMIs, individual institutions, or small bands of like-minded institutions, in parallel carry the responsibility to ensure local assessment tool validity.