An admissions OSCE: the multiple mini-interview

Behavioral assessment of soft skill development in a highly structured pre-health biology course for undergraduates

In this study, we assessed a highly structured, yearlong, case-based course designed for undergraduate pre-health students. We incorporated both content learning assessments and developed a novel method called Multiple Mini Exams for assessing course impact on the development of skills that professional schools often seek in pre-health students, focusing on students' abilities to collaborate with others, display bedside manners, synthesize patient case details, appropriately use scientific and medical language, and effectively attain patients' medical histories. This novel method utilized a rubric based on desired medical student skills to score videotaped behaviors and interactions of students role playing as doctors in a hypothetical patient case study scenario. Overall, our findings demonstrate that a highly structured course, incorporating weekly student performance and presentation of patient cases encompassing history taking, diagnosis, and treatment, can result in content learning, as well as improve desired skills specific for success in medical fields.

Investigating the internal structure of multiple mini interviews-A perspective from Pakistan

BACKGROUND

Healthcare professionals require many personal attributes in addition to cognitive abilities and psychomotor skills for competent practice. Multiple Mini- Interviews are being employed globally to assess personality attributes of candidates for selection in health professions education at all level of entry; these attributes are namely, communication skills, critical thinking, honesty, responsibility, health advocacy, empathy and sanctity of life. Considering the high stakes involved for students, faculty, institutions and the society, rigorous quality assurance mechanisms similar to those used for student assessment must be employed for student selection, throughout the continuum of medical education. It is a difficult undertaking as these psychological constructs are difficult to define and measure. Though considered to yield reliable and valid scores, studies providing multiple evidences of internal structure especially dimensionality of Multiple Mini-Interviews are sparse giving rise to questions if they are measuring a single or multiple constructs and even if they are measuring what they are purported to be measuring.

OBJECTIVE

The main objective is to provide statistical support of the multi-dimensional nature of our Multiple Mini Interviews, hypothesized a-priori, through CFA. Another objective is to provide multiple evidences for the internal structure. Our study highlights the link between content and internal structure evidences of the constructs, thus establishing that our Multiple Mini Interviews measure what they were intended to measure.

METHOD

After securing permission from the Institutional review board, an a-priori seven factor-model was hypothesized based on the attributes considered most essential for the graduating student of the institution. After operationally defining the attributes through extensive literature search, scenarios were constructed to assess them. A 5-point rating scale was used to rate each item on the station. A total 259 students participated in the multiple mini interviews over a period of three days. A training workshop had been arranged for the participating faculty.

RESULTS

The reliability coefficient using Cronbach's alpha were calculated (range from 0.73 to 0.94), Standard Error of Measurement (ranged from 0.80 to1.64), and item to station-total correlation ranged from 0.43-0.50 to 0.75-0.83. Inter-station correlation was also determined. Confirmatory factor analysis endorsed the results of Exploratory factor analysis in the study revealing a seven model fit with multiple indices of Goodness-of-fit statistics such as Root mean square error of approximation (RMSEA) value 0.05, Standardized root mean square residual (SRMR) value with less than 0.08. All these indices showed that model fit is good. The Confirmatory factor analysis confirmed the multi-dimensional nature of our MMIs and also confirmed that our stations measured the attributes that they were supposed to measure.

CONCLUSION

This study adds to the validity evidence of Multiple Mini-Interviews, in selection of candidates, with required personality traits for healthcare profession. It provides the evidence for the multi-dimensional structure of Multiple Mini interviews administered with multiple evidences for its internal structure and demonstrates the independence of different constructs being measured.

How professional and academic pre-qualifications relate to success in medical education: Results of a multicentre study in Germany

OBJECTIVE

Every year, many applicants want to study medicine. Appropriate selection procedures are needed to identify suitable candidates for the demanding curriculum. Although research on medical school admissions has shown good predictive validity for cognitive selection methods (undergraduate GPA, aptitude tests), the literature on applicants with professional and/or academic experience prior to entering medical school remains slim. In our study, we therefore aimed to examine the association between academic success in medical school and having previously completed vocational training in the medical field, voluntary service (≥11 months) or an academic degree.

METHODS

Data were collected in a multicentre, cross-sectional study at five medical schools in Germany (Baden-Wuerttemberg) from students during medical school (i.e. 3rd-, 6th-, and 10th-semester and final-year students). Academic success was assessed according to scores on the first and second state examinations, the total number of examinations repeated and the number of semesters beyond the standard period of study. For the analysis we calculated ordinal logistic regression models for each outcome variable of academic success.

RESULTS

A total of N = 2,370 participants (response rate: RR = 47%) participated in the study. Having completed vocational training was associated with a higher amount of repeated examinations (small effect), while having an academic degree was associated with worse scores on the second state examination (medium effect). No significant association emerged between voluntary service and academic success.

CONCLUSION

The results indicate that professional and academic pre-qualifications pose no advantage for academic success. Possible associations with the financing of study and living conditions of students with pre-qualifications were analysed and discussed in an exploratory manner. However, the operationalisation of academic success from objective and cognitive data should be critically discussed, as the benefits of prior experience may be captured by personal qualities rather than examination results.

Demographic disparities in dental school selection: An analysis of current UK practices

INTRODUCTION

Specific social groups remain under-represented within dentistry. While the University Clinical Aptitude Test (UCAT) aims to widen participation in under-represented social groups, there is no evidence in dental education that this aim is being met.

MATERIALS AND METHODS

Data over two admission cycles (2012 and 2013), including 3246 applicants to 10 UK dental schools, were analysed. Applicant and selected pools were compared to the UK population. Multiple logistic regression was used to investigate the association between demographic variables and UCAT and receiving an offer of a place at dental school.

RESULTS

Over-representation of Female, Asian, least deprived and grammar school groups were found in applicant and selected pools compared with the UK population. White ethnic applicants were significantly more selected than Black (OR 0.25), Asian (OR 0.57) and Mixed (OR 0.80) ethnicities, while least deprived applicants were significantly more selected than most deprived (OR 0.59). Grammar school education increased odds of selection by 1.8 when compared to state school. The addition of UCAT to the model for applicants reduced ethnic disparities but led to disparities between other groups.

CONCLUSION

Current widening participation practices focus on attracting applicants from lower socio-economic groups. However, this study showed that ethnicity, sex and educational background biases also affect demographic diversity in dentistry. The UCAT shows promise in levelling the playing field; however, widening access measures will only succeed if selection committees radically change selection processes to address the systemic biases, enabling the dentists of tomorrow to represent the society they serve.

The art of reinvention: The remarkable longevity of the OSCE

This commentary delves into the theatre and performances of OSCEs. Join us in exploring their 50‐year journey of reinvention and how they continue to hold the stage in medical assessments.

Evaluating non-cognitive skills in medical school applicants

Medical school candidates must have both the cognitive and professional competencies required to become good physicians. In this commentary, we outline the evidence and outcomes associated with the implementation of these selection methodologies and evaluate their ability to assess non-cognitive skills.

'Click, I Guess I'm Done': Applicants' and Assessors' Experiences Transitioning to a Virtual Multiple Mini Interview Format

Introduction

During the COVID-19 pandemic, medical schools were forced to suspend in-person interviews and transition to a virtual Multiple Mini Interview (vMMI) format. MMIs typically comprise multiple short assessments overseen by assessors, with the aim of measuring a wide range of non-cognitive competencies. The adaptation to vMMI required medical schools to make swift changes to their MMI structure and delivery. In this paper, we focus on two specific groups greatly impacted by the decision to transition to vMMIs: medical school applicants and MMI assessors.

Methods

We conducted an interpretive qualitative study to explore medical school applicants' and assessors' experiences transitioning to an asynchronous vMMI format. Ten assessors and five medical students from one Canadian medical school participated in semi-structured interviews. Data was analyzed using a thematic analysis framework.

Results

Both applicants and assessors shared a mutual feeling of longing and nostalgia for an interview experience that, due to the pandemic, was understandably adapted. The most obvious forms of loss experienced - albeit in different ways - were: 1) human connection and 2) missed opportunity. Applicants and assessors described several factors that amplified their grief/loss response. These were: 1) resource availability, 2) technological concerns, and 3) the virtual interview environment.

Discussion

While virtual interviewing has obvious advantages, we cannot overlook that asynchronous vMMIs do not lend themselves to the same caliber of interaction and camaraderie as experienced in in-person interviews. We outline several recommendations medical schools can implement to enhance the vMMI experience for applicants and assessors.

Front row seat: The role MMI assessors play in widening access to medical school

BACKGROUND

While many medical schools utilize the Multiple Mini-Interview (MMI) to help select a diverse student body, we know little about MMI assessors' roles. Do MMI assessors carry unique insights on widening access (WA) to medical school? Herein we discuss the hidden expertise and insights that assessors contribute to the conversation around WA.

METHODS

Ten MMI assessors (1-10 years' experience) participated in semi-structured interviews exploring factors influencing equitable medical school recruitment. Given their thoughtfulness during initial interviews, we invited them for follow-up interviews to gain further insight into their perceived role in WA. Fourteen interviews were conducted and analyzed using a thematic analysis approach.

RESULTS

Assessors expressed concerns with diversity in medicine; dissatisfaction with the status quo fueled their contributions to the selection process. Assessors advocated for greater diversity among the assessor pool, citing benefits for all students, not only those from underrepresented groups. They noted that good intentions were not enough and that medical schools can do more to include underrepresented groups' perspectives in the admissions process.

CONCLUSION

Our analysis reveals that MMI assessors are committed to WA and make thoughtful contributions to the selection process. A medical school selection process, inclusive of assessors' expertise is an important step in WA.

The influence of occupational therapy students' preferred language on academic and clinical performance in a Canadian university program

The Occupational Therapy Program at this Canadian university is a French program however, students must be bilingual to function in English or French clinical fieldwork settings. An understanding of the role of language in successful completion of program requirements was needed to effectively support students' education. The study objectives were to identify the role of linguistic factors in students' academic and clinical performance and to provide recommendations for strategies to address areas of learning difficulty. A multimethod approach used 4 data sources: (1) Multiple Mini Interview (MMI) informal language assessment scores, (2) grade point average (GPA), (3) fieldwork evaluation reports, and (4) an online survey of program graduates. The GPA on admission and MMI scores of 140 students predicted respectively only 20% and 2% of the variation in GPA on program completion. The areas of poorest performance in failed clinical fieldwork reports were in clinical reasoning and communication competencies. Among survey respondents (n = 47), 44.5% reported that a clinical placement in their second language with related charting (51.6%) and client communication (40.9%) were the most significant difficulties encountered in the program. Clients with mental health issues (45.4%) were the most challenging population to work with, attributed to communication barriers in the students' second language. Strategies are proposed to support occupational therapy students' academic and clinical language proficiency, including conversational training courses, problem-based learning activities in students' second language, focussed teaching on the clinical reasoning process and reflective skills, and language coaching to address early signs of difficulty in clinical fieldwork.

A geographic-location-based medical school admissions process does not influence pre-clerkship and licensing examination academic performance

Background

Students are selected for admission to the Northern Ontario School of Medicine University (NOSM U) MD degree program using criteria aiming to maximize access of persons thought most likely to practice in the region, including use of a geographic context score (GCS) which ranks those with lived experience in northern Ontario and/or rurality most highly. This study investigates the effect of this admissions process upon medical school academic performance.

Methods

We used a retrospective cohort design combined with multiple linear regression analysis to investigate the relationship between admission scores and performance on pre-clerkship courses, and the Medical Council of Canada Qualifying Exam Part 1 (MCCQE1).The GCS did not significantly explain performance variance on any pre-clerkship course, nor on the MCCQE1, while the undergraduate Grade Point Average correlated with most assessment scores. The number of prior undergraduate biomedical courses predicted science and clinical skills performance, particularly in Year 1, but not with MCCQE1 scores. Performance on Year 2 courses, particularly foundational sciences and clinical skills, significantly predicted MCCQE1 scores.

Results

Our data suggest that admission geographic context scoring is unrelated to future academic performance. Further, students with fewer prior undergraduate biomedical courses may benefit from increased support and/or a modified program during the early years.

Cross-sectional evaluation of an asynchronous multiple mini-interview (MMI) in selection to health professions training programmes with 10 principles for fairness built-in

OBJECTIVES

We aimed to explore the psychometric properties of the first known online asynchronous multiple mini-interview (MMI) designed for fairness with subgroup analyses by key characteristics, usability and acceptability.

DESIGN

Cross-discipline multimethod evaluation.

SETTING

One UK University.

PARTICIPANTS

Applicants to nursing, midwifery and paramedic science undergraduate programmes during 2021-2022.

PRIMARY, SECONDARY OUTCOME MEASURES

Psychometric properties (internal consistency, construct validity, dimensionality) were assessed using Cronbach's alpha (α), parallel analysis (PA), Schmid-Leiman transformation and ordinal confirmatory factor analysis (CFA). Usability and acceptability were evaluated using descriptive statistics and conventional content analysis.

METHODS

The system was configured in a seven question 4 min MMI. Applicants' videorecorded their answers which were later assessed by interviewers and scores summed. Applicants and interviewers completed online evaluation questionnaires.

RESULTS

Performance data from 712 applicants determined good-excellent reliability for the asynchronous MMI (mean α 0.72) with similar results across subgroups (gender, age, disability/support needs, UK/non-UK). PA and factor analysis results suggested there were seven factors relating to the MMI questions with an underlying general factor that explained the variance in observed candidate responses. A CFA testing a seven-factor hierarchical model showed an excellent fit to the data (Confirmatory Fit Index=0.99), Tucker Lewis Index=0.99, root mean square error (RMSE) =0.034). Applicants (n=210) viewed the flexibility, relaxed environment and cost savings advantageous. Interviewers (n=65) reported the system to be intuitive, flexible with >70% time saved compared with face-to-face interviews. Reduced personal communication was cited as the principal disadvantage.

CONCLUSIONS

We found that the asynchronous MMI was reliable, time-efficient, fair and acceptable and building fairness in was lost-cost. These novel, insights are applicable across health professions selection internationally informing the future configuration of online interviews to ensure workforces represent the societies they serve.

Validity evidence for a virtual multiple mini interview at a pharmacy program

BACKGROUND

Numerous health professions schools have transitioned to virtual admissions interviews in recent years. While some research suggests that virtual multiple mini-interviews (vMMIs) are feasible, acceptable, and more affordable, there is a paucity of research concerning the validity of this approach. The purpose of this study was to examine the validity and reliability of vMMIs and explore differences in performance between vMMI and in-person MMIs.

METHODS

Data were collected for two years of in-person MMIs and two years of vMMIs at a pharmacy program/school in the United States. An exploratory factor analysis (principal components analysis) with varimax rotation and Kaiser rule (i.e. retaining factors with eigenvalue > 1.0) was used to explore the construct validity of the vMMI data. Pearson correlation was used to examine correlations between vMMI stations and Cronbach alpha was used to determine the internal consistency of each station. Independent t-tests were used to examine differences between in-person MMI and vMMI scores. Cohen's d was used to determine effect sizes.

RESULTS

Four hundred and thirty-eight (42.69%) candidates completed an in-person MMI and 588 (57.31%) completed a vMMI. Factor analysis indicated that each vMMI station formed a single factor with loads ranging from 0.86 to 0.96. The vMMI stations accounted for most of the total variance, demonstrated weak to negligible intercorrelations, and high internal consistency. Significant differences between in-person and vMMI scores were found for the teamwork-giving, teamwork-receiving, and integrity stations. Medium effect sizes were found for teamwork-giving and teamwork-receiving and a small effect size was found for integrity.

CONCLUSIONS

Initial evidence suggests that the vMMI is a valid and reliable alternative to in-person MMIs. Additional research is needed to examine sources of differences in rating patterns between the two approaches and identify strategies that align with institutional priorities for recruitment and admissions.

Comparison of the Multiple Mini-Interview and the Traditional Interview in Medical School Admissions: Lessons Learned Using a Hybrid Model at One Institution

PURPOSE

Medical school admissions interviews are a critical form of assessment; however, the most effective interview strategy is debated. This study compares the traditional interview (TI) and multiple mini-interview (MMI) within a hybrid TI-MMI model at one medical school to determine whether the interview approaches reveal different information about applicants and whether a hybrid model results in a more diversified applicant pool.

METHOD

Admissions data from 3 application cycles at the Donald and Barbara Zucker School of Medicine at Hofstra/Northwell were used. The TI was used in 2017-2018 and the hybrid TI-MMI model in 2018-2019 and 2019-2020. Applicants were scored on a 5-point scale and referred to a voting committee for acceptance consideration if interview scores met threshold criteria. Changes in the number of students referred to the committee using the TI vs the TI-MMI score criteria were analyzed.

RESULTS

In 2017-2018 (TI only), 683 applicants were interviewed; in 2018-2019 (TI-MMI), 844 applicants were interviewed; and in 2019-2020 (TI-MMI), 805 applicants were interviewed. Medium correlations were found between total MMI and TI scores in 2018-2019 ( ρ = 0.37, P < .001) and 2019-2020 ( ρ = 0.33, P < .001). No differences were found in TI scores between 2017-2018 and 2018-2019 ( P = .30), but TI scores were significantly lower in 2019-2020 vs 2017-2018 ( P < .001) and 2018-2019 ( P = .002). Overall, a 10% to 18% increase was found in the number of applicants referred to the voting committee when using hybrid criteria, with a 19% to 27% increase in underrepresented in medicine applicants.

CONCLUSIONS

The TI-MMI model may allow for a more holistic interview approach and an expanded pool of applicants, particularly underrepresented in medicine applicants, considered for acceptance.

Exploring the relationships between first impressions and MMI ratings: a pilot study

The phenomenon of first impression is well researched in social psychology, but less so in the study of OSCEs and the multiple mini interview (MMI). To explore its bearing on the MMI method we included a rating of first impression in the MMI for student selection executed 2012 at the University Medical Center Hamburg-Eppendorf, Germany (196 applicants, 26 pairs of raters) and analyzed how it was related to MMI performance ratings made by (a) the same rater, and (b) a different rater. First impression was assessed immediately after an applicant entered the test room. Each MMI-task took 5 min and was rated subsequently. Internal consistency was α = .71 for first impression and α = .69 for MMI performance. First impression and MMI performance correlated by r = .49. Both measures weakly predicted performance in two OSCEs for communication skills, assessed 18 months later. MMI performance did not increment prediction above the contribution of first impression and vice versa. Prediction was independent of whether or not the rater who rated first impression also rated MMI performance. The correlation between first impression and MMI-performance is in line with the results of corresponding social psychological studies, showing that judgements based on minimal information moderately predict behavioral measures. It is also in accordance with the notion that raters often blend their specific assessment task outlined in MMI-instructions with the self-imposed question of whether a candidate would fit the role of a medical doctor.

[Multimodal selection of medical students: The predictive power of individual process components in the two-stage selection process at Witten/Herdecke University (UW/H)]

OBJECTIVE

According to the legislator's ideas, the selection of medical students in Germany should no longer be based on the Abitur grade alone. This approach has already been implemented in the two-stage selection process at Witten/Herdecke University (UW/H) using several criteria. On the one hand, the present study aims to determine the prognostic value of the procedural components for the overall performance on the selection day. On the other hand, the different strategies of the applicants in processing the written task (phase 1) will be examined with regard to their application success.

METHODOLOGY

Data on applications for the summer semester 2020 (N = 819 phase 1; N = 233 phase 2) were available retrospectively. A stepwise regression analysis was conducted to determine the predictive power of each procedural component. Using a summary content analysis, the four essays from the applicants' motivation letters were structured and categories were identified, and an extreme group comparison (Group 1: Not invited; Group 2: University acceptance; N = 60 essays) was conducted.

RESULTS

As the stepwise regression analysis shows, the individual biographical interview emerged as the strongest predictor in terms of overall performance, followed by lecture, group interview, and multiple mini interviews. Content analysis extracted content and scaling categories for the individual essays, as well as an additional meta-category (Impression Management, IM). Successful applicants demonstrated, among other things, better judgment skills, more sophisticated reasoning skills, and an internalized role model as a physician. In addition, they used defensive IM strategies, e.g., subjectification and self-deprecation, more frequently.

CONCLUSION

Biographical interview is considered the strongest predictor of overall performance. The dimensions of impression management, reasoning quality and judgment proved to be reliable predictors of successful performance in the selection process. In addition, role image as a physician and professional commitment had a favorable effect on the selection decision.

Understanding the Urology Program Directors Perspective on the Current Resident Selection Process: The Society of Academic Urologists National Survey of Urology Program Directors

OBJECTIVE

The traditional residency selection process was altered dramatically by the SARS CoV-2 (COVID-19) pandemic. For the 2020-2021 application cycle in-person interviews were transitioned to the virtual format. What was thought to be a temporary transition has now become the new standard with continued endorsement from the Association of American Medical Colleges (AAMC) and the Society of Academic Urologists (SAU) for virtual interviews (VI). We sought to assess the perceived efficacy and satisfaction of the VI format from the urology residency program director's (PDs) perspective.

DESIGN

A designated SAU Taskforce on "Optimizing the Applicant Experience in the Virtual Interview Era" developed and refined a survey composed of 69 questions on VI and was distributed to all urology program directors (PD) of member institutions of the SAU. The survey focused on candidate selection, faculty preparation, and interview day logistics. PDs were also asked to reflect on the impact of VI on their match results, recruitment of underrepresented minorities and female gender, and what their preference would be for future applications cycles.

PARTICIPANTS

Urology residency PDs (84.7% response rate) between January 13, 2022 - February 10, 2022 were included in the study.

RESULTS

Most programs interviewed a total of 36 to 50 applicants (80%), with an average of 10 to 20 applicants per interview day. The top 3 ranked criteria for interview selection reported by urology PDs surveyed included letters of recommendation, clerkship grades, and USMLE Step 1 score. The most common areas of formal training for faculty interviewers were diversity, equity and inclusion (55%), implicit bias (66%), and review of the SAU guidelines on illegal questions (83%). Over half (61.4%) of PDs believed that they were able to accurately represent their training program through the virtual platform, while 51% felt that VI did not afford similar assessments of applicant as in-person interviews. Two-thirds of PDs believed the VI platform improve access for all applicants to attend interviews. Focusing on the impact of the VI platform for recruitment of underrepresented minorities (URM) and female gender applicants, 15% and 24% reported improved visibility respectively for their program, and 24% and 11% reported increased ability to interview URM and female gender applicants respectively. Overall, in-person interviews were reported to be preferred by 42%, and 51% of PDs desired VIs to be included in future years.

CONCLUSIONS

PDs opinion and role of the VIs into the future is variable. Despite uniform agreement of cost savings and belief that VI platform improves access for all, only half of PDs expressed interest of the VI format being continued in some form. PDs note limitation of VI in the ability to comprehensively assess applicants as well as the in-person format. Many programs have begun to incorporate vital training in the areas of diversity equity and inclusion bias, and illegal questions. There is a role for continued development and research on ways to optimize virtual interviews.

A mixed methods evaluation of Multiple Mini Interviews for entry into the Bachelor of Midwifery

BACKGROUND

Multiple-mini interviews (MMI) are increasingly used as part of the admission process into healthcare degrees. Evaluations have found MMIs to be a fair assessment tool in terms of reliability and validity and viewed positively by those involved in the MMI process. The use of MMIs in midwifery is novel and evaluation is lacking.

AIM

To evaluate the use of MMIs as part of the admission process for the Bachelor of Midwifery in one Australian university.

METHODS

A basic convergent mixed methods study design was utilised. Data included linked data sets, Likert scale responses to survey questions, focus groups and open-ended survey questions. Integration took place at the interpretation and reporting stage.

FINDINGS

Participants viewed the MMI experience positively. The study confirmed the reliability of the MMIs as an assessment tool. Most variance in MMI scores was attributed to the candidate at 31.4 % with the interviewer and the interview station having less influence on the MMI score at 11 % and 6.4 % variance. Older applicants on average achieved higher MMI scores, and those who spoke a language other than English at home or were first in family to attend university had lower on average MMI scores. Being born overseas did not impact an applicant's MMI score. The overall experience was seen as fair, offering further opportunity to gain entry into the Bachelor of Midwifery.

CONCLUSION

MMIs were viewed positively and findings support the use of MMIs as part of an admissions process for the Bachelor of Midwifery.

Interviewer bias when using multiple mini-interviews in selecting student nurses in a Chinese setting

BACKGROUND

Interviewer effects may cause unfairness in assessments in multiple mini-interviews (MMIs). Due to cultural differences, the bias factors of interviewers may vary between the East and the West. MMIs are a relatively new type of assessment setting in China and few studies have been conducted to examine the interviewer effects of MMIs in this context.

OBJECTIVES

We adopted a multi-faceted Rasch measurement (MFRM) to measure interviewer effects in assessments in Hong Kong.

METHODS

Data were collected from a nursing school in Hong Kong. There were 431 candidates and 12 interviewers engaged in a six-station MMI setting. The scores collected from the interviews were analyzed in terms of 1) interviewer stringency/leniency, 2) candidate gender, 3) interview time, and 4) rating category in the station. The Student's t-statistic values were calculated to investigate the marking tendencies of individual interviewers.

RESULTS

The research findings suggest that interviewers differ in their degree of stringency/leniency, but the number of candidates examined by each interviewer does not affect interviewer stringency/leniency in terms of the interviewer's assessment. There is not sufficient evidence indicating that candidate gender and interview time are bias factors affecting assessment score in this study. Among the six rating categories examined, honesty/integrity is the most stringent category, while self-awareness is the most lenient category. Interview bias from individuals was identified. When we consider the interview scores given by individual interviewers, it is evident that some interviewers may have been biased toward a certain gender or rating categories.

CONCLUSIONS

MMIs are useful when selecting student nurses in a Chinese setting. However, interviewer bias may exist. We used an MFRM to better understand interviewer bias across various dimensions. The present study contributes to the development and use of MMIs in non-Western countries and can be used as a reference to extend this research to other locations.

Selection and recruitment of pre-registration occupational therapy students in the United Kingdom: Exploring entry criteria across education providers

Introduction:

Since widening access to higher education became a United Kingdom (UK) governmental priority in the 1990’s, occupational therapy has made little progress in diversifying student and workforce populations that mirror increasingly diverse service user populations. This research aims to, for the first time, map entry criteria across UK pre-registration programmes, while considering fair access and exploring who might be missing at the point of enquiry and entry.

Method:

A cross-sectional quantitative content analysis was conducted of all UK university websites, identifying programme type, academic, professional and alternative entry criteria for 2021/2022.

Findings:

Five entry routes via undergraduate and post-graduate programmes ( n = 73) offer limited part-time opportunities (n = 11). Visible academic entry criteria appear weighted towards ‘traditional’ qualifications, while assessment of professional skills at application and selection is explicit for over 75% of programmes. 86% (n = 63) utilise interviews at selection, with 33% (n = 24) not publicly acknowledging alternative access routes into the profession.

Conclusion:

If the profession is to avoid continued stagnation in diversity amongst student populations and successfully reflect service user diversity in the workforce, it is essential UK universities increase parity across academic entry criteria, ensure the visibility of acceptable skills for alternative access and substantially improve flexibility for part-time study.

[Association between students' clinical performance and their success in the computerized national ranking tests: A single-center retrospective cohort study]

INTRODUCTION

Before attending residency, 6th-year French medical students must validate a final examination including a practical clinical test in their faculty. However, the national ranking that determines their future specialty and region solely relies on a computerized knowledge test. Our goal was to investigate the association between the final faculty test and the national ranking test.

METHODS

In our faculty, the final examination includes a computerized theoretical test (similar to the national one) and a practical test: a standardized evaluation of semiology skills at the bedside and a standardized assessment of relational skills with role plays. The agreements between the national test and faculty computerized and practical tests were analyzed by intraclass correlation coefficients (ICC).

RESULTS

Data from 1806 students who underwent the three examinations from 2017 to 2021 were analyzed. There was a good agreement between the ranks in the faculty and national computerized tests: ICC 0.83 (95% CI 0.81-0.85). By contrast, the agreement between the ranks in the faculty practical test and the national computerized test was poor: ICC 0.13 (95% CI 0.08-0.17). Results were stable over the years.

CONCLUSION

The agreement between the ranking of the current national test and the clinical skills assessed by a specific faculty test is poor. This could relate to a true independence or to different levels of motivation to perform well. Indeed, the result of the national test is the most important one as it determines their career. Incorporating a clinical assessment into the national ranking test will motivate students to acquire clinical skills and value those who perform well this practical dimension.

Part 2: Assessment of a virtual vs. onsite interview experience from the interviewer perspective

INTRODUCTION

The University of Texas at Austin College of Pharmacy transitioned from onsite interviews to virtual interviews in fall 2020. There is limited literature on whether the virtual format impacts an interviewer's assessment of a candidate. This study examined interviewer ability to assess candidates and barriers to participation.

METHODS

During the virtual interview process, interviewers utilized a modified multiple mini interview (mMMI) format to evaluate prospective college of pharmacy students. An 18-item survey was emailed to 62 interviewers from the 2020-2021 cycle. Virtual mMMI scores were compared to the previous year's onsite MMI scores. Descriptive statistics and thematic analysis were used to assess the data.

RESULTS

The response rate to the survey was 53% (33/62), and 59% of interviewers preferred virtual interviews to in-person. Interviewers stated that there were fewer barriers to participation, increased comfort, and more time with applicants during virtual interviews. For six of the nine attributes evaluated, ≥ 90% of interviewers reported that they were able to assess applicants as well as they did in person. When comparing virtual and onsite MMI scores, seven of nine attributes were statistically significantly higher in the virtual cohort than onsite.

CONCLUSIONS

From the perspective of interviewers, the virtual interview decreased barriers to participation and still allowed the ability to assess the candidates. While offering a choice of interview setting to interviewers could increase accessibility, the statistically significant difference in MMI scores between virtual and onsite formats suggests that additional standardization is required to offer both formats simultaneously.

A Semistructured Interview for Surgical Residency Targeting Nontechnical Skills

OBJECTIVE

We review the development, implementation, and initial outcomes of a semistructured interview process to assess the nontechnical skills of surgical residency applicants.

DESIGN

In 2018, we restructured our residency selection interview process. Through semistructured faculty interviews, we sought to evaluate candidates along seven nontechnical skills (grit, ownership, rigor, teamwork, presence, impact, and organizational alignment). We plotted each candidate's scores on a radar plot for graphical representation and calculated the plot area of each candidate. We retrospectively evaluated 3 years of data, comparing the nontechnical skill scores of matriculants into the training program to those of nonmatriculants.

SETTING

Tenwek Hospital is a 361-bed tertiary teaching and referral hospital in rural western Kenya with a 5-year general surgery residency program.

PARTICIPANTS

Thirty-one applicants were interviewed over 3 years. Thirteen matriculated into the program.

RESULTS

Scores for grit, (4.8 vs 3.9; p = 0.0004), impact (4.2 vs 3.5; p = 0.014), ownership (4.2 vs 3.6; p = 0.01), and organizational alignment (4.3 vs 3.8; p = 0.008) were significantly higher in matriculants.

CONCLUSIONS

This semistructured interview process provides a robust and beneficial mechanism for assessing applicants' nontechnical skills, which may allow for the matriculation of more well-rounded candidates into surgical residency and, ultimately, surgical practice.

The Purpose, Design, and Promise of Medical Education Research Labs

Medical education researchers are often subject to challenges that include lack of funding, collaborators, study subjects, and departmental support. The construct of a research lab provides a framework that can be employed to overcome these challenges and effectively support the work of medical education researchers; however, labs are relatively uncommon in the medical education field. Using case examples, the authors describe the organization and mission of medical education research labs contrasted with those of larger research team configurations, such as research centers, collaboratives, and networks. They discuss several key elements of education research labs: the importance of lab identity, the signaling effect of a lab designation, required infrastructure, and the training mission of a lab. The need for medical education researchers to be visionary and strategic when designing their labs is emphasized, start-up considerations and the likelihood of support for medical education labs is considered, and the degree to which department leaders should support such labs is questioned.

Are different station formats assessing different dimensions in multiple mini-interviews? Findings from the Canadian integrated French multiple mini-interviews

BACKGROUND

Multiple mini-interviews (MMI) are used to assess non-academic attributes for selection in medicine and other healthcare professions. It remains unclear if different MMI station formats (discussions, role-plays, collaboration) assess different dimensions.

METHODS

Based on station formats of the 2018 and 2019 Integrated French MMI (IFMMI), which comprised five discussions, three role-plays and two collaboration stations, the authors performed confirmatory factor analysis (CFA) using the lavaan 0.6-5 R package and compared a one-factor solution to a three-factor solution for scores of the 2018 (n = 1438) and 2019 (n = 1440) cohorts of the IFMMI across three medical schools in Quebec, Canada.

RESULTS

The three-factor solution was retained, with discussions, role-plays and collaboration stations all loading adequately with their scores. Furthermore, all three factors had moderate-to-high covariance (range 0.44 to 0.64). The model fit was also excellent with a Comparative fit index (CFI) of 0.983 (good if > 0.9), a Tucker Lewis index of 0.976 (good if > 0.95), a Standardized Root Mean Square Residual of 0.021 (good if < .08) and a Root Mean Square Error of 0.023 (good if < 0.08) for 2018 and similar results for 2019. In comparison, the single factor solution presented a lower fit (CFI = 0.819, TLI = 0.767, SRMR = 0.049 and RMSEA = 0.070).

CONCLUSIONS

The IFMMI assessed three dimensions that were related to stations formats, a finding that was consistent across two cohorts. This suggests that different station formats may be assessing different skills, and has implications for the choice of appropriate reliability metrics and the interpretation of scores. Further studies should try to characterize the underlying constructs associated with each station format and look for differential predictive validity according to these formats.

Associations between communication OSCE performance and admission interviews in medical education

OBJECTIVE

To investigate associations between medical students' communication and other clinical skills assessed in OSCEs and MMIs performance upon admission by developing an assessment strategy based on an internationally acknowledged curriculum framework.

METHODS

Between July 2019 and March 2020, 365 medical students in the 4th and 5th semester took two OSCEs containing 10 5-minute stations examining communication and other clinical skills. We used a European conceptual framework to determine the content validity of the communication score and calculated scores of communication and other clinical skills. We assessed students' OSCE performance and estimated multiple regressions to predict its association with MMIs.

RESULTS

The multiple linear regressions showed that students' MMI performance upon admission is significantly associated with the communication score (b = 0.32, p = 0.006) but not the clinical score (b = 0.19, p = 0.121), when controlling for gender and cognitive criteria.

CONCLUSIONS

Our assessment strategy designed to distinguish between different areas of competence provides a more thorough description of the positive relationship between OSCE performance and MMIs.

PRACTICE IMPLICATIONS

We developed a communication skills assessment strategy that can be easily applied by medical schools that use OSCEs as a training or assessment method.

LEGO®-based communication assessment in virtual general surgery residency interviews

Purpose

Effective communication skills are a critical quality and skill that is highly sought after for surgeons which largely impacts patient outcomes. Residency programs design their interview processes to select the best candidates. LEGO®-based activities have been frequently used to enhance communication skills and team building. This study investigates the effectiveness and reliability of a novel LEGO®-based communication assessment in interviews for surgical residencies and the feasibility of implementing it in a virtual setting.

Methods

This study conducted a retrospective analysis of a LEGO®-based communication assessment at the program's 2020/2021 residency interviews. Each applicant was assessed on a different model. The total scores were analyzed for consistency among raters and correlated to faculty interviews. Furthermore, the impact of the assessment structure, scoring criteria, and range of models' difficulties on the total scores were explored.

Results

A total of 54 categorical and 55 preliminary applicants interviewed on 2 days. The assessment on different models and had no impact on applicants' total scores for either categorical and preliminary groups (p = 0.791 and 0.709, respectively). The communication components of the assessment showed high consistency between the raters. The two applicant groups displayed a statistically significant difference (p = 0.004) in the communication evaluation and model accuracy components. Total scores did not correlate with the faculty interviews of standardized questions in either group.

Conclusion

This novel LEGO®-based communication assessment showed high reliability and promising results as a tool to assess communication and problem solving for residency interviews that can be readily implemented in a virtual setting.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-022-00021-4.

Associations between admissions factors and the need for remediation

This study examines the way in which student characteristics and pre-admissions measures are statistically associated with the likelihood a student will require remediation for academic and professionalism offenses. We anchor our inquiry within Irby and Hamstra's (2016) conceptual framework of constructs of professionalism. Data from five graduating cohorts (2014-2018) from McMaster University (Hamilton, Canada) (N = 1,021) were retroactively collected and analyzed using traditional and multinominal logistic regression analyses. The relationship among student characteristics, pre-admissions variables, and referral for potential remediation both by occurrence (yes/no) as well as type (academic/professional/no referral) were examined separately. Findings indicate that gender (OR = 0.519, 95% CI 0.326-0.827, p < 0.01) and undergraduate grade point average (GPA) (OR = 0.245, 95% CI 0.070-0.855, p < 0.05) were significantly associated with instances of referral for potential professionalism and academic remediation, respectively. Women were less likely than men to require remediation for professionalism (OR = 0.332, 95% CI 0.174-0.602, p < 0.001). Undergraduate GPAs (OR = 0.826, 95% CI 0.021-0.539, p < 0.01) were significantly associated with remediation for academic reasons. Lower undergraduate GPAs were associated with a higher likelihood of remediation. These findings point to the admissions variables that are associated with instances that prompt referral for potential remediation. Where associations are not significant, we consider the application of different conceptualizations of professionalism across periods of admissions and training. We encourage those involved in applicant selection and student remediation to emphasize the importance of the interactions that occur between personal and contextual factors to influence learner behaviour and professional identity formation.

The validity of the assignment book-based structured interview in predicting academic performance in medical schools: a retrospective cohort study

PURPOSE

Structured interviews have become essential in the medical schools admission selection because structured interviews predict academic achievement after admission. The purpose of this study was to determine validity and fairness of the new structural interview technique, assignment book-based structured interview (ABSI), in predicting future academic achievement of the medical students.

METHODS

The validity of this new interview technique and academic achievement was evaluated based on the data of all the applicants and successful applicants who applied for on-time admission between the year 2011 and 2014.

RESULTS

The ABSI technique showed a significant correlation and predictive validity for academic achievement in the medical school. The retention group received significantly lower T-scores of ABSI compared with the superior student group.

CONCLUSION

The results indicate that ABSI is a feasible, reliable, fair and valid admission selection tool. The ABSI may be meaningful and fair method for predicting academic achievements, and it could be incorporated as one of the contents in the multiple mini-interview.

Feasibility of an automated interview grounded in multiple mini interview (MMI) methodology for selection into the health professions: an international multimethod evaluation

OBJECTIVES

Global, COVID-driven restrictions around face-to-face interviews for healthcare student selection have forced admission staff to rapidly adopt adapted online systems before supporting evidence is available. We have developed, what we believe is, the first automated interview grounded in multiple mini-interview (MMI) methodology. This study aimed to explore test-retest reliability, acceptability and usability of the system.

DESIGN, SETTING AND PARTICIPANTS

Multimethod feasibility study in Physician Associate programmes from two UK and one US university during 2019-2020.

PRIMARY, SECONDARY OUTCOMES

Feasibility measures (test-retest reliability, acceptability and usability) were assessed using intraclass correlation (ICC), descriptive statistics, thematic and content analysis.

METHODS

Volunteers took (T1), then repeated (T2), the automated MMI, with a 7-day interval (±2) then completed an evaluation questionnaire. Admission staff participated in focus group discussions.

RESULTS

Sixty-two students and seven admission staff participated; 34 students and 4 staff from UK and 28 students and 3 staff from US universities. Good-excellent test-retest reliability was observed at two sites (US and UK2) with T1 and T2 ICC between 0.65 and 0.81 (p<0.001) when assessed by individual total scores (range 80.6-119), station total scores 0.6-0.91, p<0.005 and individual site (≥0.79 p<0.001). Mean test re-test ICC across all three sites was 0.82 p<0.001 (95% CI 0.7 to 0.9). Admission staff reported potential to reduce resource costs and bias through a more objective screening tool for preselection or to replace some MMI stations in a 'hybrid model'. Maintaining human interaction through 'touch points' was considered essential. Users positively evaluated the system, stating it was intuitive with an accessible interface. Concepts chosen for dynamic probing needed to be appropriately tailored.

CONCLUSION

These preliminary findings suggest that the system is reliable, generating consistent scores for candidates and is acceptable to end users provided human touchpoints are maintained. Thus, there is evidence for the potential of such an automated system to augment healthcare student selection.

Weighting admission scores to balance predictiveness-diversity: The Pareto-optimization approach

CONTEXT

Although many medical schools seek to improve diversity, they grapple with the challenge of how to weight the scores of different admission methods to achieve a balance between obtaining high predictiveness and ensuring diversity in the selected student pool. Yet, in large-scale employment settings, substantial progress has been made on this front: Pareto-optimization has been introduced as an elegant statistical tool to assist decision makers in determining the weights assigned to selection methods in advance (before the selection has taken place) so that a selection system is designed to achieve an optimal balance as reflected by the trade-off that one outcome (e.g., predictiveness) cannot be improved without harm to the other outcome (e.g., diversity).

AIMS

This paper reviews the theory and research evidence about Pareto-optimization and explains how Pareto-optimization permits medical schools to better balance predictiveness and diversity in medical admission systems.

METHODS

After reviewing common weighting schemes (unit, regression-based and ad hoc weighting) and their drawbacks, we introduce the theory and logic of Pareto-optimization for better balancing predictiveness and diversity. To this end, we also offer an illustrative example. Next, we review the mathematical basis and available research evidence regarding Pareto-optimization. Finally, we discuss potential criticisms (i.e., complexity and legal concerns).

CONCLUSIONS

Compared to traditional unit weighting, regression-based weighting and ad hoc weighting, Pareto-optimization leads to substantial increases in diversity intake (up to three times more), while keeping the predictiveness of the selection methods at the same level. Moreover, the Pareto-optimization is robust to sampling variability and variability of the input selection parameters.

Conducting a synchronous virtual multiple mini-interview using Webex for medical school admissions

COVID-19 pandemic restrictions abruptly changed the way interviews for medical school admissions have been conducted. This study is unique as it highlights the first successful virtual synchronous multiple mini interview (MMI) in Canada. Our low technical incident rate, troubleshooting strategies and approach may reassure other medical schools considering conducting a virtual MMI. Success was achieved with collaboration, a strong organizational and communication strategy, learning along the way and a priori contingency plans. Virtual interviewing in academic medicine is likely here to stay, and future work to highlight the impact on applicants will help to build on the diversity mission in undergraduate medicine admissions.

Conducting multiple mini-interviews in the midst of COVID-19 pandemic

Multiple mini-interview (MMI) is a 'multiple sample-based' approach comprising multiple focused encounters intended to access and assess a range of attributes in order to gain more objectively multiple impressions of an applicant's interpersonal skills, thoughtfulness and general demeanour. It is designed to focus on four domains that are not considered to be comprehensive, but are considered to be vital for a successful career in the health sciences: critical thinking, ethical decision making, communication and knowledge of the healthcare system. Traditionally, the MMI is conducted face-to-face, but with COVID-19 pandemic and the implementation of social distancing measures, no onsite or campus teaching, banning of mass gatherings and cancellation of face-to-face interviews, Pengiran Anak Puteri Rashidah Sa'adatul Bolkiah Institute of Health Sciences at Universiti Brunei Darussalam explored the feasibility of conducting MMI through virtual means. This report provides an account of our experience in conducting internet-MMI for the selection of new applicants into the August 2020 cohort of the Medicine programme. We also aimed to determine whether the scores derived from internet-MMI were reliable and equivalent to the scores derived from traditional MMI.

A pilot project to introduce the multiple mini interview (MMI) at a Borneo medical school: The universiti Malaysia Sabah experience- a cross-sectional study

Introduction

The Multiple Mini Interview (MMI) demonstrates efficacy and superiority over traditional medical interviews in assessing non-cognitive domains during the recruitment of medical undergraduates. At Universiti Malaysia Sabah (UMS), a five-station MMI was piloted in 2019, featuring a mix of three examiner-driven stations (assessing professionalism, ethics, and motivation to study medicine), and two roleplayer-driven stations (assessing empathy and science communication specifically, and communication skills in general).

Methods

260 candidates were grouped into two separate geographical groups – urban and suburban/rural. Descriptive analysis, skewness and kurtosis were performed for normality assessment, whereas Cronbach's alpha, McDonald's omega, and Greatest lower bound assessed internal consistency. For validity measures, correlations were calculated between scores for separate stations, overall scores, urban and suburban/rural status. Also, exploratory factor analysis was performed on the five stations as validity measures. Difficulty and discrimination indices were calculated as quality measures. Qualitative analysis was performed on “red flag” comments detailing grossly unsuitable candidates.

Results

Roleplayer-driven stations yielded more red flags than examiner-driven stations. The three examiner-driven stations were significantly and moderately correlated (rho between 0.602 and 0.609, p < 0.001). The Empathy roleplayer-driven station was not correlated with two examiner-driven stations and only weakly correlated with the Ethics examiner-driven and the Science Communication roleplayer-driven station. Factor analysis suggests a three-factor model. The two roleplayer-driven stations stood as independent factors, and the three examiner-driven stations coalescing as one factor provided the best explanatory model. Quality measures suggest all five stations had suitable discriminatory properties (all >0.530), whereas the stations were distributed equally in difficulty index.

Conclusion

The UMS MMI has identified specific skillsets that may be in short supply in our incoming medical students. Also, it illustrates the yawning gap between academic knowledge and ‘translational’ scientific knowledge and communication skills.

•

The Multiple Mini Interview demonstrates superiority over traditional interviews during the recruitment of medical undergraduates.

•

At Universiti Malaysia Sabah, a five-station MMI was piloted in 2019.

•

The MMI featured a mix of three examiner-driven stations and two roleplayer-driven stations.

•

The UMS MMI has identified specific skillsets that may be in short supply in the incoming medical students.

•

It also illustrates the yawning gap between academic knowledge, “translational” scientific knowledge and communication skills.

An investigation of the predictive validity of selection tools on performance in physiotherapy training in Australia

OBJECTIVES

Despite a large body of research on selection in medical education, very little is conducted in other health professions. This study investigated the predictive validity of multiple selection tools on academic and clinical performance outcomes of undergraduate physiotherapy students.

DESIGN

A retrospective observational study.

SETTING

Undergraduate physiotherapy program in Australia.

PARTICIPANTS

497 undergraduate physiotherapy students across seven entry cohorts. Including students directly from secondary school (n=381) and with prior tertiary study (n=116).

MAIN OUTCOME MEASURES

Academic performance as measured by written examinations. Clinical performance, measured by Objective Structured Clinical Examinations (OSCEs) during on-campus units and the Assessment of Physiotherapy Practice (APP) for off-campus clinical placements. Predictor variables included selection tools (academic achievement, interview, aptitude test) and demographic variables (age, gender).

RESULTS

Selection interview was a positive predictor of OSCEs and final year clinical performance in direct school leaver participants. Academic achievement scores from selection positively predicted written examinations scores.

CONCLUSION

Clinical and academic performance were predicted by tools measuring different domains at selection. Assessing broadly across academic and non-academic domains at selection can be valuable in identifying applicants who will be able to meet the range of outcomes for course completion and subsequent registration in the physiotherapy profession.

Conducting Virtual Simulated Skills Multiple Mini-Interviews for General Surgery Residency Interviews

OBJECTIVE

We describe the feasibility of transitioning simulated skills assessments during general surgery interviews from an in-person to virtual format.

DESIGN/SETTING

Technical and nontechnical skill multiple mini-interviews (MMIs) were performed virtually and assessed for 109 applicants during virtual general surgery interviews over 2 days at a tertiary academic medical center.

RESULTS

We demonstrate the feasibility of virtually assessing general surgery residency applicants' technical and non-technical skills. Using a virtual MMI format during general surgery interviews, we assessed communication, emotional intelligence, anatomical knowledge, interpretation of medical tests, knot tying, and suturing. Four tasks (communication, emotional intelligence , anatomical knowledge, and interpretation of tests) were assessed synchronously by trained general surgery interns. Applicants submitted a recording of themselves performing knot tying and suturing tasks, which were asynchronously assessed after the interview day. Applicants rated the MMI experience highly (4.3/5) via postinterview day survey and the majority of applicants felt that station objectives were met in the virtual format.

CONCLUSIONS

We report a successful experience implementing technical and nontechnical virtual MMIs with capacity for 120 applicants during general surgery residency interviews. In the midst of a COVID-19 pandemic, the ability to assess surgical leaners virtually is essential. Virtual skills assessments may provide a more comprehensive picture of applicants and enable residency programs to better assess residents when gathering in person is not feasible.

Gender Associations with Selection into Australian Orthopaedic Surgical Training: 2007-2019

BACKGROUND

Communities have better health outcomes when their clinicians reflect the diversity of the communities they serve. More than 50% of Australian medical school graduates are female, yet women represent less than 5% of Australian orthopaedic surgeons. Selection into orthopaedic surgical training in Australia is an annual, nation-wide process, based on curriculum vitae (CV), referee reports and performance in multiple mini-interviews (MMI). The influence of applicant gender on these selection scores was examined.

METHODS

The CV, referee reports and MMI scores used for selection for each year from 2007 to 2019 were analysed from the perspective of the applicant's gender.

RESULTS

Over the years of the study, male applicants had higher CV scores and referee report scores, which determined the gender proportions invited to interview. By contrast, the interview process and selection from interview did not demonstrate a gender association.

CONCLUSION

We describe the impact of selection tools, utilized over the past 13 years, on the gender diversity of trainees commencing orthopaedic surgery training in Australia. Leaders in postgraduate training should examine commonly used selection procedures to identify and reduce the unconscious biases that may affect their performance and value.

In Lebanon "It Never Rains But It Pours"-How the American University of Beirut faced dangers and seized opportunities: Transforming medical education through multiple crises

Higher education and healthcare are in the eye of the health, economic, and political storms induced by the COVID-19 pandemic. Many of the questions that have been raised had been lingering incipiently, but others are legitimately new and specific to the very natures of the healing professions and their educational components. Forced to adapt to the pandemic and the collapse of the Lebanese economy and socio-political structure, the American University of Beirut Faculty of Medicine is evolving in order to survive while enhancing undergraduate and graduate medical education.

Difference in demographics and motivation to study medicine with respect to medical students' channel of admission: A national study

INTRODUCTION

Medical schools employ various tools to select suitable medical students (MS). This study investigated whether MS who were admitted through multiple mini-interviews (MMI) and MS who were admitted through Taiwan's Joint College Entrance Written Test (JCEWT) differed in their characteristics.

METHODS AND SUBJECTS

First-year MS from seven medical schools completed a semi-structured questionnaire that inquired into their channel of admission (MMI or JCEWT), gender, location (metropolitan or rural), high school type (public or private), parents' socioeconomic status (SES), and motivations to study medicine.

RESULTS

In total, 513 MS participated, 493 (96%) returned valid questionnaires, and 397 were enrolled in the study, (MMI group: 205 MS; JCEWT group: 192 MS). Irrespective of channel of admission, most MS came from metropolitan areas (80%-86%), belonged to high-SES families (73%-76%), and had mixed motivations (51%-96%). Female applicants, private school leavers, and those who were less motivated by the physician's SES were more likely to be selected through the MMI channel than the JCEWT channel.

CONCLUSION

Irrespective of the channels of entry, MS had similar demographics and motivations for studying medicine. MS selected through MMI had different characteristics than those selected through a JCEWT.

The impact of Multiple Mini Interviews on the attrition and academic outcomes of midwifery students

BACKGROUND

Admission to the Bachelor of Midwifery (BMid) in Australia has traditionally been based on academic ranking. The BMid is a high demand course offered to a limited number of students and therefore choosing applicants who complete the degree is important. Multiple Mini Interviews (MMIs) are used to assess non-cognitive skills and select students into healthcare degrees. One university in Australia has introduced MMIs as part of the application process for the BMid.

AIM

Compare attrition rates and Grade Point Average (GPA) scores between students admitted into the BMid using both academic ranking and MMIs, to those admitted on academic ranking alone.

METHODS

A basic convergent mixed methods design, using an explanatory unidirectional framework to integrate data. Attrition rates, GPA, and multiple mini interview scores (2013-2019), were linked and compared for before and after the use of MMI's. Focus groups with students, interviewers, and hospital-based educators, explored stakeholder experiences. Open-ended questions from an applicant survey were added to the qualitative data set, which was analysed thematically.

FINDINGS

Students who enrolled via the MMI's had significantly lower attrition rates than those enrolled before MMI's were introduced. GPA scores were significantly higher for students who enrolled via the MMI's. Integration of data found MMI's identified students passionate to undertake midwifery, and that success at the interviews increased students' confidence to successfully complete their studies.

CONCLUSION

MMI's as part of the entry process into the BMid enabled identification of applicants more likely to remain in the course and succeed in their studies.

'Why didn't they see my scars?' Critical thematic analysis of simulated participants' perceived tensions surrounding objective structured clinical examinations

Background

Simulated participants (SPs) play an important role in simulated assessments of clinical encounters between medical students and patients, most notably in objective structured clinical examinations (OSCEs). SP contributions to OSCEs are invaluable, taking the role of a patient or carer. While SPs in some settings/contexts may rate students, their role has been problematized in the literature for their lack of agency within a standardised format of OSCEs that promotes reliability, objectivity and accountability. In this study, we explored SP experiences for tensions that result from simulated assessments and their potential implications for education.

Methods

Semi-structured interviews were conducted with seven SPs who were also tasked with providing a global mark for students. They were purposively selected to include women and men of different ages, occupation, education and experience as an SP. The interviews were analysed using a critical thematic analysis using a phenomenological approach.

Results

SP experiences directly addressed tensions and contradictions around OSCEs. SP participants described their experiences under four themes: industrialising, reducing, performativity and patient safety. OSCEs were compared to an industrial process that promoted efficiency but which bore no resemblance to real-life doctor-patient encounters. They were perceived to have a power and agency that reduced SPs to verbalising scripts to ensure that students were exposed to a standardised simulated experience that also underlined the performative role of SPs as props. These performative and reductionist experiences extended to students, for whom the mark sheet acted as a checklist, promoting standardised responses that lacked genuineness. All of this created a tension for SPs in promoting patient safety by ensuring that those medical students who passed were clinically competent.

Conclusions

OSCEs often form part of high-stakes exams. As such, they are governed by processes of industrialisation, accountability and standardisation. OSCEs possess a power and agency that can have unintended negative consequences. These include ‘conditioning’ students to adopt behaviours that are not suited to real-life clinical encounters and are not person-centred.

Health professions education scholarship: The emergence, current status, and future of a discipline in its own right

Medical education, as a domain of scholarly pursuit, has enjoyed a remarkably rapid development in the past 70 years and is now more commonly known as health professions education (HPE) scholarship. Evidenced by a solid increase of publications, numbers of specialized journals, professional associations, national and international conferences, academies for medical educators, masters and doctoral courses, and the establishment of many units of HPE scholarship, the domain of HPE education scholarship has matured into a scholarly discipline in its own right. In this contribution, the author reviews the developments of the field from Boyer's four criteria that determine scholarship: discovery, integration, application, and teaching. Born mid-20th century, and in the first decades developed in the predominant area of physician education, HPE scholarship has matured, with increasing breadth, depth, and volume of scholars, publications, conferences, and dedicated centers for research and development. The author concludes that, given the infrastructure that has emerged, HPE can arguably be considered a discipline in its own right. This academic question may not matter hugely for practices of scholarly work in this domain, and any stance in this academic debate inevitably reflects a personal view, but the author would support the view of health professions scholarship as being a unique niche, with inherent dependence on both medical and other health professional sciences, on the one hand, and social sciences, including educational sciences, on the other hand.

Multiple Mini-Interview Utilization in United States Physician Assistant Program Admission Processes

PURPOSE

Health care program admission processes utilize multi-faceted approaches to evaluate cognitive and noncognitive attributes of applicants. The multiple mini-interview (MMI) was developed in response to the need for a reliable and validated tool to assess noncognitive factors and has been increasingly incorporated into the admissions process by physician assistant (PA) programs. The study's purpose was to explore the current implementation and utilization of the MMI within PA programs.

METHODS

The study used a mixed-methods exploratory approach including a telephone survey and semi-structured interview of 11 PA programs using the MMI in their admissions process. Quantitative data collected included demographic information, MMI implementation characteristics, station structure, scoring, feasibility, satisfaction with MMI utilization, and MMI evaluation methods.

RESULTS

During the 2015-2016 admissions cycle, the participating programs used from 5 to 10 stations, averaging 7 minutes per station, requiring 8 faculty, 2 staff, and 7 students per interview session. Despite variation in program size, number of applicants, and years of MMI utilization, all participating programs reported that they were satisfied with the format and would continue to utilize the MMI in the admissions process.

CONCLUSIONS

While there is substantial literature describing the use of the MMI within health care programs globally, this study represents the first characterization of its use within PA programs on a national level. Although there was variation among PA program implementation of the MMI, our results are comparable to studies within other health care professions. Additional studies are necessary to further describe the MMI and its correlation with PA program educational outcomes and the impact on diversity.

Are different medical school admission tests associated with the outcomes of a simulation-based OSCE?

BACKGROUND

Medical school admission procedures have the common goal to select applicants with the greatest potential of becoming successful physicians. Hamburg Medical Faculty selects medical students by grade point average (GPA) and employs a two-step selection process of a natural sciences test (HAM-Nat), in some cases followed by multiple mini-interviews (HAM-Int). Multiple mini-interviews can predict non-cognitive outcomes, while GPA has predictive validity for cognitive outcomes. The aim of our study was to explore communication skills and clinical knowledge of advanced medical students according to their respective admission procedure.

METHODS

In July 2019, 146 students grouped according to their admission procedure into GPA-only (19.2 %), HAM-Nat (33.6 %), HAM-Int (30.8 %), and Waiting List (16.4 %) participated in four OSCE stations which equally assessed students' communication skills (OSCE part 1) and clinical knowledge (OSCE part 2) in simulated patient encounters, rated by physicians with checklists. Additionally, psychosocial assessors ranked communication skills with a global rating scale (GR). The students also participated in a multiple choice (MC) exam testing clinical knowledge. Kruskal-Wallis analyses of variance of test performance and Spearman correlation of instruments were calculated.

RESULTS

Students from the Waiting List group performed significantly worse on the MC exam compared to GPA-only and HAM-Int (adjusted p = .029 and 0.018, respectively). No significant differences were found between the admission groups with respect to communication skills. Global Rating and OSCE part 1 (communication) correlated significantly (ρ = 0.228, p = .006) as did OSCE part 2 (clinical knowledge) and MC exam (ρ = 0.242, p = .003), indicating criterion validity. Constructs did not overlap, indicating divergent validity.

CONCLUSIONS

Advanced medical students selected for undergraduate studies by multiple mini-interviews assessing psychosocial skills showed similar communication skills compared to students admitted to medical school by other entryways. It is unclear whether these similarities are due to an effective undergraduate longitudinal communication curriculum. Assessing baseline communication skills of all medical students at entry-level may aid with this question.

Reliability of simulation-based assessment for practicing physicians: performance is context-specific

INTRODUCTION

Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.

METHODS

Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.

RESULTS

The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.

CONCLUSIONS

The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Perspectives on Transitioning Step 1 of the United States Medical Licensing Examination to a Pass/Fail Scoring Model: Defining New Frameworks for Medical Students Applying for Residency

For years, the USMLE Step 1 has acted as an unofficial "concours" for medical students applying to residency positions in the United States. The three-digit numeric score has been used to rank thousands of applicants without any evidence of validity. The USMLE will soon change score reporting to a pass/fail outcome. The main reason given was to address the concerns about its effects on the well-being of the students and medical education. It is argued that time for change has come. The authors discuss the various viewpoints of the stakeholders and the effects of this change on applicants and potential changes on the undergraduate medical curriculum. Furthermore, this article discusses several metrics that can be utilized in the application process in lieu of the USMLE Step 1. Additionally, some novel key metrics in the application process are identified, and their unique dynamic and adaptive characteristics are deliberated. Finally, the benefits of a transparent and holistic process are strongly advocated.