Outcomes-Based Selection Into Medical School: Predicting Excellence in Multiple Competencies During the Clinical Years

Robust, defensible, and fair: The AMEE guide to selection into medical school: AMEE Guide No. 153

Selection is the first assessment of medical education and training. Medical schools must select from a pool of academically successful applicants and ensure that the way in which they choose future clinicians is robust, defensible, fair to all who apply and cost-effective. However, there is no comprehensive and evidence-informed guide to help those tasked with setting up or rejuvenating their local selection process. To address this gap, our guide draws on the latest research, international case studies and consideration of common dilemmas to provide practical guidance for designing, implementing and evaluating an effective medical school selection system. We draw on a model from the field of instructional design to frame the many different activities involved in doing so: the ADDIE model. ADDIE provides a systematic framework of Analysis (of the outcomes to be achieved by the selection process, and the barriers and facilitators to achieving these), Design (what tools and content are needed so the goals of selection are achieved), Development (what materials and resources are needed and available), Implementation (plan [including piloting], do study and adjust) and Evaluation (quality assurance is embedded throughout but the last step involves extensive evaluation of the entire process and its outcomes).HIGHLIGHTSRobust, defensible and fair selection into medical school is essential. This guide systematically covers the processes required to achieve this, from needs analysis through design, development and implementation, to evaluation of the success of a selection process.

Selection tools and student diversity in health professions education: a multi-site study

Student diversity in health professions education (HPE) can be affected by selection procedures. Little is known about how different selection tools impact student diversity across programs using different combinations of traditional and broadened selection criteria. The present multi-site study examined the chances in selection of subgroups of applicants to HPE undergraduate programs with distinctive selection procedures, and their performance on corresponding selection tools. Probability of selection of subgroups (based on gender, migration background, prior education, parental education) of applicants (N = 1935) to five selection procedures of corresponding Dutch HPE undergraduate programs was estimated using multilevel logistic regression. Multilevel linear regression was used to analyze performance on four tools: prior-education grade point average (pe-GPA), biomedical knowledge test, curriculum-sampling test, and curriculum vitae (CV). First-generation Western immigrants and applicants with a foreign education background were significantly less likely to be selected than applicants without a migration background and with pre-university education. These effects did not vary across programs. More variability in effects was found between different selection tools. Compared to women, men performed significantly poorer on CVs, while they had higher scores on biomedical knowledge tests. Applicants with a non-Western migration background scored lower on curriculum-sampling tests. First-generation Western immigrants had lower CV-scores. First-generation university applicants had significantly lower pe-GPAs. There was a variety in effects for applicants with different alternative forms of prior education. For curriculum-sampling tests and CVs, effects varied across programs. Our findings highlight the need for continuous evaluation, identifying best practices within existing tools, and applying alternative tools.

Does medical education reform change who is selected? A national cross-sectional survey from China

OBJECTIVES

Approximately 10 years ago, China introduced an education plan to improve the overall quality of medical education and to better serve the population's health needs. Many medical schools were then recognised and financed by China's Ministry of Education to develop and operationalise new pilot programmes (PPs) aligned with this plan. These ran in parallel with the traditional programmes (TPs). One way to achieve the plan's first aim, improving the quality of medical education, is to select academically stronger candidates. We, thus, examined and compared who were selected into PPs and TPs.

DESIGN

Cross-sectional study.

SETTING

Data were collected from 123 medical schools across China via the 2021 China Medical Student Survey.

PARTICIPANTS

Participants were undergraduate clinical medicine students across all year groups.

PRIMARY AND SECONDARY OUTCOME MEASURES

Medical school selection was via the National College Entrance Examination (NCEE). Medical students' NCEE performance and their sociodemographics were used as the primary and secondary outcome measures. Mann-Whitney or χ2 tests were used to compare the means between educational programmes (PPs vs TPs) and various selection outcomes. Multilevel mixed-effects regressions were employed to account for school idiosyncratic selection results.

RESULTS

Of the 204 817 respondents, 194 163 (94.8%) were in a TP and 10 654 (5.2%) a PP. PP respondents (median=75.2, IQR=69.5-78.8) had significantly higher NCEE scores than their TP counterparts (median=73.9, IQR=68.5-78.7). Holding constant their NCEE score, PP respondents were significantly more likely to come from urban areas, not be first-generation college students, and have parents with higher occupational status and income.

CONCLUSIONS

Assuming quality can be indicated by prior academic achievement at the point of selection, PPs achieved this mission. However, doing so limited medical students' diversity. This may be unhelpful in achieving the Education Plan's goal to better serve China's health needs.

The association between how medical students were selected and their perceived stress levels in Year-1 of medical school

BACKGROUND

The prevalence of medical students' mental distress is high. While schools apply various methods to select a well-performing and diverse student population, little is known about the association between different selection methods and the well-being of these students during medical school. The present retrospective multi-cohort study assessed whether students selected by high grades, assessment, or weighted lottery showed different stress perception levels in Year-1 of medical school.

METHODS

Of 1144 Dutch Year-1 medical students, 650 (57%) of the cohorts 2013, 2014, and 2018 who were selected by high grades, assessment, or weighted lottery completed a stress perception questionnaire (PSS-14). A multilevel regression analysis assessed the association between selection method (independent variable) and stress perception levels (dependent variable) while controlling for gender and cohort. In a post-hoc analysis, academic performance (optimal vs. non-optimal) was included in the multilevel model.

RESULTS

Students selected by assessment (B = 2.25, p < .01, effect size (ES) = small) or weighted lottery (B = 3.95, p < .01, ES = medium) had higher stress perception levels than students selected by high grades. Extending the regression model with optimal academic performance (B=-4.38, p < .001, ES = medium), eliminated the statistically significant difference in stress perception between assessment and high grades and reduced the difference between weighted lottery and high grades from 3.95 to 2.45 (B = 2.45, p < .05, ES = small).

CONCLUSIONS

Selection methods intended to create a diverse student population - assessment and lottery - are associated with higher stress perception levels in Year-1 of medical school. These findings offer medical schools insights into fulfilling their responsibility to take care of their students' well-being.

A framework for residents' pursuit of excellence based upon non-cognitive and cognitive attributes

PURPOSE

Excellence, although variably conceptualized, is commonly used in medicine and the resident excellence literature is limited. Both cognitive attributes (CAs) and non-cognitive attributes (NCAs) are essential for academic and clinical performance; however, the latter are difficult to evaluate. Undergirded by an inclusive and non-competitive approach and utilizing CAs and NCAs, we propose a criterion-referenced behavioral framework of resident excellence.

METHODS

Perceptions of multiple stakeholders (educational administrators, faculty, and residents), gathered by survey (n = 218), document analysis (n = 52), and focus group (n = 23), were analyzed. Inductive thematic analysis was followed by deductive interpretation and categorization using sensitizing concepts for excellence, NCAs, and CAs. Chi-squared tests were used to determine stakeholder perception differences.

RESULTS

All stakeholders had similar perceptions (P > .05) and 13 behavioral attributes in 6 themes undergirded by insight and conscientiousness were identified. The NCAs included: interpersonal skills (works with others, available, humble), professional (compassionate, trustworthy), commitment to profession (visible, volunteers), commitment to learn (proactively seeks feedback, creates learning opportunities), and work-life balance/integration (calm demeanor, inspirational). The CA (medical knowledge and intellect) included: applies knowledge to gain expertise and improves program's caliber.

CONCLUSION

Resident excellence is posited as a pursuit. The attributes are non-competitive, inclusionary, potentially achievable by all, and do not negatively affect freedom of choice. However, contextual and cultural differences are likely and these need validation across societal equity segments. There are implications for learners (adaptive reflection and learning goal orientation), faculty (reduced bias and whole-person feedback), and system leaders (enhancing culture and learning environments) to foster excellence.

The CASPER preparation program innovation: increasing self-perceived competence and confidence of underrepresented applicants on the novel CASPER Snapshot and CanMEDS roles

BACKGROUND

Underrepresented Minorities in Medicine (URMMs) may face financial and social limitations when matriculating into medical schools. Performance on situational judgment tests such as Computer-based Assessment for Sampling Personal Characteristics (CASPER) can be enhanced by coaching and mentorship. The CASPER Preparation Program (CPP) coaches URMMs to prepare for the CASPER test. During the coronavirus 2019 pandemic (COVID-19), CPP implemented novel curricula on the CASPER Snapshot and CanMEDS roles.

METHODS

Pre and post-program questionnaires were completed by the students, which assessed their: 1) confidence in understanding the CanMEDS roles, and 2) perceived confidence in performing well and their familiarity and preparedness with the CASPER Snapshot. With a second post-program questionnaire, participants' scores on the CASPER test as well as medical school application outcome were also assessed.

RESULTS

Participants reported a significant increase in the URMMs' knowledge, self-perceived competency to complete the CASPER Snapshot, and their anxiety significantly decreased. The level of confidence in understanding CanMEDS roles for a career in healthcare increased as well. The majority (91%) agreed that the feedback received from tutors was adequate and the virtual component of the program was beneficial during COVID-19. 51% of students scored in the highest quartile on the CASPER test and 35% received an offer of admission from CASPER-requiring medical schools.

CONCLUSION

Pathway coaching programs have the potential to increase confidence and familiarity amongst URMMs for the CASPER tests and CanMEDS roles. Similar programs should be developed with the aim to increase the chances of URMMs matriculating into medical schools.

BMAT's predictive validity for medical school performance: A retrospective cohort study

BACKGROUND

Although used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools.

METHODS

This was a retrospective cohort study of four ICSM (1188 students, entering 2010-2013) and three LKCMedicine cohorts (222 students, 2013-2015). We investigated associations between BMAT Section 1 ('Thinking Skills'), Section 2 ('Scientific Knowledge and Applications') and Section 3a ('Writing Task') scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A-levels) in a subset of ICSM students.

RESULTS

When BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08-0.19; LKCMedicine: 0.22-0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09-0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A-level grades.

DISCUSSION

Schools who wish to assess scientific knowledge independently of A-levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.

Quality Assurance of Undergraduate Medical Education in Israel by Continuous Monitoring and Prioritization of the Accreditation Standards

External accreditation reviews of undergraduate medical curricula play an important role in their quality assurance. However, these reviews occur only at 4-10-year intervals and are not optimal for the immediate identification of problems related to teaching. Therefore, the Standards of Medical Education in Israel require medical schools to engage in continuous, ongoing monitoring of their teaching programs for compliance with accreditation standards. In this paper, we propose the following: (1) this monitoring be assigned to independent medical education units (MEUs), rather than to an infrastructure of the dean's office, and such MEUs to be part of the school governance and draw their authority from university institutions; and (2) the differences in the importance of the accreditation standards be addressed by discerning between the "most important" standards that have been shown to improve student well-being and/or patient health outcomes; "important" standards associated with student learning and/or performance; "possibly important" standards with face validity or conflicting evidence for validity; and "least important" standards that may lead to undesirable consequences. According to this proposal, MEUs will evolve into entities dedicated to ongoing monitoring of the education program for compliance with accreditation standards, with an authority to implement interventions. Hopefully, this will provide MEUs and faculty with the common purpose of meeting accreditation requirements, and an agreed-upon prioritization of accreditation standards will improve their communication and recommendations to faculty.

The predictive validity of admission criteria for the results of clinical competency assessment with an emphasis on family medicine in the fifth year of medical education: an observational study

BACKGROUND

In many countries, the number of applicants to medical schools exceeds the number of available places. This offers the need, as well as the opportunity to medical schools to select those applicants most suitable for later work as a doctor. However, there is no generally accepted definition of a 'good doctor'. Clinical competencies may serve as surrogates. The aim of this study was to compare medical students in Germany selected based either on their pre-university grade point average alone or based on the result of a university-specific selection procedure regarding their clinical competencies with an emphasis on family medicine in the later years of training.

METHODS

We used the 'Allgemeinarztbarometer Ausbildung' (Undergraduate Family Medicine Barometer), an instrument developed to assess clinical competencies with an emphasis on family medicine, to compare students in the pre-university grade point average admission-quota and the university-specific selection procedure admission-quota in the fifth year of training. Students were judged by their supervising general practitioners after a two-week practical course. Competencies were rated on a five-point Likert-scale (1 = 'totally agree' i.e. the student is very competent to 5 = 'totally disagree' i.e. the student is not competent at all).

RESULTS

We included 94 students (66% female). Students in the university-specific selection procedure quota (n = 80) showed better mean scores in every item of the Undergraduate Family Medicine Barometer. We found a statistically significant difference between the two groups for the item assessing communication skills (M [university-specific selection procedure quota] = 1.81, SD = 0.84 vs. M [pu-GPA quota] = 2.38, SD = 0.96; t[91] = -2.23, p = .03; medium effect size). Logistic regression revealed no statistically significant age or gender contribution.

CONCLUSIONS

Despite the small sample-size, our results indicate, that students selected via an university-specific selection procedure show better communicative competencies in the later years of training.

COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape

Introduction: The COVID-19 pandemic presented numerous, significant challenges for medical schools, including how to select the best candidates from a pool of applicants when social distancing and other measures prevented "business as usual" admissions processes. However, selection into medical school is the gateway to medicine in many countries, and it is critical to use processes which are evidence-based, valid and reliable even under challenging circumstances. Our challenge was to plan and conduct a multiple-mini interview (MMI) in a dynamic and stringent safe distancing context.Methods: This paper reports a case study of how to plan, re-plan and conduct MMIs in an environment where substantially tighter safe distancing measures were introduced just before the MMI was due to be delivered.Results: We report on how to design and implement a fully remote, online MMI which ensured the safety of candidates and assessors.Discussion: We discuss the challenges of this approach and also reflect on broader issues associated with selection into medical school during a pandemic. The aim of the paper is to provide broadly generalizable guidance to other medical schools faced with the challenge of selecting future students under difficult conditions.