1
|
Cleland J, Blitz J, Cleutjens KBJM, Oude Egbrink MGA, Schreurs S, Patterson F. Robust, defensible, and fair: The AMEE guide to selection into medical school: AMEE Guide No. 153. MEDICAL TEACHER 2023; 45:1071-1084. [PMID: 36708606 DOI: 10.1080/0142159x.2023.2168529] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/05/2023]
Abstract
Selection is the first assessment of medical education and training. Medical schools must select from a pool of academically successful applicants and ensure that the way in which they choose future clinicians is robust, defensible, fair to all who apply and cost-effective. However, there is no comprehensive and evidence-informed guide to help those tasked with setting up or rejuvenating their local selection process. To address this gap, our guide draws on the latest research, international case studies and consideration of common dilemmas to provide practical guidance for designing, implementing and evaluating an effective medical school selection system. We draw on a model from the field of instructional design to frame the many different activities involved in doing so: the ADDIE model. ADDIE provides a systematic framework of Analysis (of the outcomes to be achieved by the selection process, and the barriers and facilitators to achieving these), Design (what tools and content are needed so the goals of selection are achieved), Development (what materials and resources are needed and available), Implementation (plan [including piloting], do study and adjust) and Evaluation (quality assurance is embedded throughout but the last step involves extensive evaluation of the entire process and its outcomes).HIGHLIGHTSRobust, defensible and fair selection into medical school is essential. This guide systematically covers the processes required to achieve this, from needs analysis through design, development and implementation, to evaluation of the success of a selection process.
Collapse
Affiliation(s)
- J Cleland
- Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore
| | - J Blitz
- Centre for Health Professions Education, Faculty of Medicine and Health Sciences, Stellenbosch University, Stellenbosch, South Africa
| | - K B J M Cleutjens
- School of Health Professions Education, Maastricht University, the Netherlands
| | - M G A Oude Egbrink
- School of Health Professions Education, Maastricht University, the Netherlands
| | - S Schreurs
- School of Health Professions Education, Maastricht University, the Netherlands
- Centrum for Evidence Based Education, University of Utrecht, the Netherlands
| | | |
Collapse
|
2
|
Broks VMA, Stegers-Jager KM, Fikrat-Wevers S, Van den Broek WW, Woltman AM. The association between how medical students were selected and their perceived stress levels in Year-1 of medical school. BMC MEDICAL EDUCATION 2023; 23:443. [PMID: 37328850 PMCID: PMC10276376 DOI: 10.1186/s12909-023-04411-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Accepted: 05/30/2023] [Indexed: 06/18/2023]
Abstract
BACKGROUND The prevalence of medical students' mental distress is high. While schools apply various methods to select a well-performing and diverse student population, little is known about the association between different selection methods and the well-being of these students during medical school. The present retrospective multi-cohort study assessed whether students selected by high grades, assessment, or weighted lottery showed different stress perception levels in Year-1 of medical school. METHODS Of 1144 Dutch Year-1 medical students, 650 (57%) of the cohorts 2013, 2014, and 2018 who were selected by high grades, assessment, or weighted lottery completed a stress perception questionnaire (PSS-14). A multilevel regression analysis assessed the association between selection method (independent variable) and stress perception levels (dependent variable) while controlling for gender and cohort. In a post-hoc analysis, academic performance (optimal vs. non-optimal) was included in the multilevel model. RESULTS Students selected by assessment (B = 2.25, p < .01, effect size (ES) = small) or weighted lottery (B = 3.95, p < .01, ES = medium) had higher stress perception levels than students selected by high grades. Extending the regression model with optimal academic performance (B=-4.38, p < .001, ES = medium), eliminated the statistically significant difference in stress perception between assessment and high grades and reduced the difference between weighted lottery and high grades from 3.95 to 2.45 (B = 2.45, p < .05, ES = small). CONCLUSIONS Selection methods intended to create a diverse student population - assessment and lottery - are associated with higher stress perception levels in Year-1 of medical school. These findings offer medical schools insights into fulfilling their responsibility to take care of their students' well-being.
Collapse
Affiliation(s)
- Vera M A Broks
- Institute of Medical Education Research Rotterdam, Erasmus MC University Medical Centre Rotterdam, Rotterdam, the Netherlands
| | - Karen M Stegers-Jager
- Institute of Medical Education Research Rotterdam, Erasmus MC University Medical Centre Rotterdam, Rotterdam, the Netherlands
| | - Suzanne Fikrat-Wevers
- Institute of Medical Education Research Rotterdam, Erasmus MC University Medical Centre Rotterdam, Rotterdam, the Netherlands
| | - Walter W Van den Broek
- Institute of Medical Education Research Rotterdam, Erasmus MC University Medical Centre Rotterdam, Rotterdam, the Netherlands
| | - Andrea M Woltman
- Institute of Medical Education Research Rotterdam, Erasmus MC University Medical Centre Rotterdam, Rotterdam, the Netherlands.
| |
Collapse
|
3
|
Davies DJ, Sam AH, Murphy KG, Khan SA, Choe R, Cleland J. BMAT's predictive validity for medical school performance: A retrospective cohort study. MEDICAL EDUCATION 2022; 56:936-948. [PMID: 35514145 PMCID: PMC9545404 DOI: 10.1111/medu.14819] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/30/2022] [Revised: 04/24/2022] [Accepted: 04/30/2022] [Indexed: 05/09/2023]
Abstract
BACKGROUND Although used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools. METHODS This was a retrospective cohort study of four ICSM (1188 students, entering 2010-2013) and three LKCMedicine cohorts (222 students, 2013-2015). We investigated associations between BMAT Section 1 ('Thinking Skills'), Section 2 ('Scientific Knowledge and Applications') and Section 3a ('Writing Task') scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A-levels) in a subset of ICSM students. RESULTS When BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08-0.19; LKCMedicine: 0.22-0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09-0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A-level grades. DISCUSSION Schools who wish to assess scientific knowledge independently of A-levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.
Collapse
Affiliation(s)
- Daniel J. Davies
- Imperial College School of MedicineImperial College LondonLondonUK
| | - Amir H. Sam
- Imperial College School of MedicineImperial College LondonLondonUK
| | - Kevin G. Murphy
- Imperial College School of MedicineImperial College LondonLondonUK
| | - Shahid A. Khan
- Imperial College School of MedicineImperial College LondonLondonUK
| | - Ruth Choe
- Lee Kong Chian School of MedicineNanyang Technological University SingaporeSingapore
| | - Jennifer Cleland
- Lee Kong Chian School of MedicineNanyang Technological University SingaporeSingapore
| |
Collapse
|
4
|
Foo J, Cook DA, Tolsgaard M, Rivers G, Cleland J, Walsh K, Abdalla ME, You Y, Ilic D, Golub R, Levin H, Maloney S. How to conduct cost and value analyses in health professions education: AMEE Guide No. 139. MEDICAL TEACHER 2021; 43:984-998. [PMID: 33280483 DOI: 10.1080/0142159x.2020.1838466] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Growing demand for accountability, transparency, and efficiency in health professions education is expected to drive increased demand for, and use of, cost and value analyses. In this AMEE Guide, we introduce key concepts, methods, and literature that will enable novices in economics to conduct simple cost and value analyses, hold informed discussions with economic specialists, and undertake further learning on more advanced economic topics. The practical structure for conducting analyses provided in this guide will enable researchers to produce robust results that are meaningful and useful for improving educational practice. Key steps include defining the economic research question, identifying an appropriate economic study design, carefully identifying cost ingredients, quantifying, and pricing the ingredients consumed, and conducting sensitivity analyses to explore uncertainties in the results.
Collapse
Affiliation(s)
- Jonathan Foo
- School of Primary and Allied Health Care, Monash University, Melbourne, Australia
| | - David A Cook
- Division of General Internal Medicine, Mayo Clinic College of Medicine and Science, Rochester, NY, USA
| | - Martin Tolsgaard
- Centre for Clinical Education, University of Copenhagen, Copenhagen, Denmark
| | - George Rivers
- Department of Economics, Monash Business School, Monash University, Melbourne, Australia
| | - Jennifer Cleland
- Medical Education Research and Scholarship Unit, Nanyang Technological University, Singapore
| | - Kieran Walsh
- BMJ Learning and Quality, BMJ, London, United Kingdom
| | | | - You You
- National Center for Health Professions Education Development, Institute of Medical Education, and Institute of Economics of Education, Peking University, Beijing, China
| | - Dragan Ilic
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Robert Golub
- Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Henry Levin
- Teachers College, Columbia University, New York, NY, USA
| | - Stephen Maloney
- School of Primary and Allied Health Care, Monash University, Melbourne, Australia
| |
Collapse
|
5
|
Sheehan A, Thomson R, Pierce H, Arundell F. The impact of Multiple Mini Interviews on the attrition and academic outcomes of midwifery students. Women Birth 2021; 35:e318-e327. [PMID: 34452869 DOI: 10.1016/j.wombi.2021.08.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2021] [Revised: 07/16/2021] [Accepted: 08/02/2021] [Indexed: 10/20/2022]
Abstract
BACKGROUND Admission to the Bachelor of Midwifery (BMid) in Australia has traditionally been based on academic ranking. The BMid is a high demand course offered to a limited number of students and therefore choosing applicants who complete the degree is important. Multiple Mini Interviews (MMIs) are used to assess non-cognitive skills and select students into healthcare degrees. One university in Australia has introduced MMIs as part of the application process for the BMid. AIM Compare attrition rates and Grade Point Average (GPA) scores between students admitted into the BMid using both academic ranking and MMIs, to those admitted on academic ranking alone. METHODS A basic convergent mixed methods design, using an explanatory unidirectional framework to integrate data. Attrition rates, GPA, and multiple mini interview scores (2013-2019), were linked and compared for before and after the use of MMI's. Focus groups with students, interviewers, and hospital-based educators, explored stakeholder experiences. Open-ended questions from an applicant survey were added to the qualitative data set, which was analysed thematically. FINDINGS Students who enrolled via the MMI's had significantly lower attrition rates than those enrolled before MMI's were introduced. GPA scores were significantly higher for students who enrolled via the MMI's. Integration of data found MMI's identified students passionate to undertake midwifery, and that success at the interviews increased students' confidence to successfully complete their studies. CONCLUSION MMI's as part of the entry process into the BMid enabled identification of applicants more likely to remain in the course and succeed in their studies.
Collapse
Affiliation(s)
- A Sheehan
- School of Nursing and Midwifery, Parramatta South Campus, Western Sydney University, NSW, Australia.
| | - R Thomson
- Graduate Research School and the Centre for Research in Mathematics and Data Science, Western Sydney University, NSW, Australia.
| | - H Pierce
- School of Nursing and Midwifery, Parramatta South Campus, Western Sydney University, NSW, Australia.
| | - F Arundell
- School of Nursing and Midwifery, Parramatta South Campus, Western Sydney University, NSW, Australia.
| |
Collapse
|
6
|
Maloney S, Pather N, Foo J, Lazarus MD. Spending Wisely: The Role of Cost and Value Research in the Pursuit of Advancing Anatomical Sciences Education. ANATOMICAL SCIENCES EDUCATION 2021; 14:263-269. [PMID: 33068329 DOI: 10.1002/ase.2027] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/20/2020] [Revised: 10/07/2020] [Accepted: 10/13/2020] [Indexed: 06/11/2023]
Abstract
Studies of "cost and value" in anatomical sciences education examine not only what works, but at what cost, thus evaluating the inputs and outputs of education. This research provides insights into how to use available resources (e.g., academic time, budgets, infrastructure) as a mechanism to obtaining the maximum outcomes available. The purpose of this viewpoint article is to expand on the application of cost and value concepts to anatomical sciences education, contextualizing these concepts through a deeper dive into the more costly educational approaches of human donor dissection. In doing so, both questions and opportunities are raised for the discipline of anatomical sciences going forward. Educational decisions, inclusive of cost and value appraisals, consider the range of outcomes for which the activity is designed to achieve, and the activity's integration with the philosophy of the educational program it is contributing to; these decisions, thus, evaluate more than just cost alone. Healthcare students' engagement with human donor dissection pedagogy offers an array of reported non-economic benefits, including non-traditional discipline-independent skill (NDIS) development (e.g., professionalism, teamwork skills). These skills are often harder to measure, but are no less important to the final pedagogical decision-making process. The goal of cost and value research is to create an evidence-base toward education that delivers maximum value for a given spend. Anatomy educators, researchers, and decision makers who embrace cost and value dialogue, and interpret and apply findings from studies of educational costs, are best positioned to improve the educational value for their learners and provide effective outputs for all stakeholders.
Collapse
Affiliation(s)
- Stephen Maloney
- School of Primary and Allied Health Care, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
| | - Nalini Pather
- Department of Anatomy, School of Medical Sciences, Faculty of Medicine, University of New South Wales, Sydney, New South Wales, Australia
| | - Jonathan Foo
- School of Primary and Allied Health Care, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
| | - Michelle D Lazarus
- Centre for Human Anatomy Education, Department of Anatomy and Developmental Biology, Faculty of Medicine Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
- Monash Centre for Scholarship in Health Education, Faculty of Medicine Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
| |
Collapse
|
7
|
Cleland JA, Foo J, Ilic D, Maloney S, You Y. "You can't always get what you want…": economic thinking, constrained optimization and health professions education. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2020; 25:1163-1175. [PMID: 33141344 PMCID: PMC7606851 DOI: 10.1007/s10459-020-10007-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/30/2020] [Accepted: 10/21/2020] [Indexed: 05/24/2023]
Abstract
Every choice we make in health professions education has a cost, whether it be financial or otherwise; by choosing one action (e.g., integrating more simulation, studying more for a summative examination) we lose the opportunity to take an alternative action (e.g., freeing up time for other teaching, leisure time). Economics significantly shapes the way we behave and think as educators and learners and so there is increasing interest in using economic ways of thinking and approaches to examine and understand how choices are made, the influence of constraints and boundaries in educational decision making, and how costs are felt. Thus, in this article, we provide a brief historical overview of modern economics, to illustrate how the core concepts of economics-scarcity (and desirability), rationality, and optimization-developed over time. We explain the important concept of bounded rationality, which explains how individual, meso-factors and contextual factors influence decision making. We then consider the opportunities that these concepts afford for health professions education and research. We conclude by proposing that embracing economic thinking opens up new questions and new ways of approaching old questions which can add knowledge about how choice is enacted in contemporary health professions education.
Collapse
Affiliation(s)
- J A Cleland
- Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, 308232, Singapore.
| | - J Foo
- School of Primary and Allied Health Care, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Australia
| | - D Ilic
- Medical Education Research and Quality (MERQ) Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - S Maloney
- School of Primary and Allied Health Care, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Australia
| | - Y You
- Health Science Centre, Peking University, Beijing, China
| |
Collapse
|
8
|
Schreurs S, Cleutjens KB, Cleland J, oude Egbrink MG. Outcomes-Based Selection Into Medical School: Predicting Excellence in Multiple Competencies During the Clinical Years. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:1411-1420. [PMID: 32134790 PMCID: PMC7447174 DOI: 10.1097/acm.0000000000003279] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
PURPOSE Medical school selection committees aim to identify the best possible students and, ultimately, the best future doctors from a large, well-qualified, generally homogeneous pool of applicants. Constructive alignment of medical school selection, curricula, and assessment with the ultimate outcomes (e.g., CanMEDS roles) has been proposed as means to attain this goal. Whether this approach is effective has not yet been established. The authors addressed this gap by assessing the relationship between performance in an outcomes-based selection procedure and performance during the clinical years of medical school. METHOD Two groups of students were compared: (1) those admitted into Maastricht University Medical School via an outcomes-based selection procedure and (2) those rejected through this procedure who were admitted into the program through a national, grade-point-average-based lottery. The authors compared performance scores of students from the 2 groups on all 7 CanMEDS roles, using assessment data gathered during clinical rotations. The authors examined data from 3 cohorts (2011-2013). RESULTS Students admitted through the local, outcomes-based selection procedure significantly outperformed the initially rejected but lottery-admitted students in all years, and the differences between groups increased over time. The selected students performed significantly better in the CanMEDS roles of Communicator, Collaborator, and Professional in the first year of clinical rotations; in these 3 roles-plus Organizer-in the second year; and in 2 additional roles (Advocate and Scholar-all except Medical Expert) at the end of their clinical training. CONCLUSIONS A constructively aligned selection procedure has increasing predictive value across the clinical years of medical school compared with a GPA-based lottery procedure. The data reported here suggest that constructive alignment of selection, curricula, and assessment to ultimate outcomes is effective in creating a selection procedure predictive of clinical performance.
Collapse
Affiliation(s)
- Sanne Schreurs
- S. Schreurs is teacher/educational advisor, Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-0233-9775
| | - Kitty B.J.M. Cleutjens
- K.B.J.M. Cleutjens is associate professor, Department of Pathology, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-7870-1670
| | - Jennifer Cleland
- J. Cleland is full professor and John Simpson Chair of Medical Education Research, Centre for Healthcare Education Research and Innovation (CHERI), University of Aberdeen, Aberdeen, Scotland; ORCID: http://orcid.org/0000-0003-1433-9323
| | - Mirjam G.A. oude Egbrink
- M.G.A. oude Egbrink is full professor, Implementation of Educational Innovations, Department of Physiology, School of Health Professions Education, and scientific director, Institute for Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-5530-6598
| |
Collapse
|
9
|
Foo J, Rivers G, Allen L, Ilic D, Maloney S, Hay M. The economic costs of selecting medical students: An Australian case study. MEDICAL EDUCATION 2020; 54:643-651. [PMID: 32119155 DOI: 10.1111/medu.14145] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/23/2019] [Accepted: 02/26/2020] [Indexed: 06/10/2023]
Abstract
CONTEXT The design of selection methods must balance, amongst a range of factors, the desire to select the best possible future doctors with the reality of our resource-constrained environment. Examining the cost of selection processes enables us to identify areas in which efficiencies may be gained. METHODS A cost description study was conducted based on selection for 2018 entry into medical school directly from secondary school. The perspectives of applicants, volunteer interviewers and the admitting institution were considered. Costs were modelled based on the Monash University (Australia) selection process, which uses a combination of secondary school matriculation score, aptitude test score (Undergraduate Medicine and Health Sciences Admission Test) and multiple mini-interview score. A variety of data sources were utilised, including bespoke surveys, audit data and existing literature. All costs are expressed in 2018 Australian dollars (AU$). Applicant behaviours in preparing for selection tests were also evaluated. RESULTS A total of 381 of 383 applicants returned the survey. Over 70% of applicants had utilised commercial preparation materials. The median total cost to applicants was AU$2586 (interquartile range [IQR] AU$1574-3999), including costs to both prepare for and attend selection tests. Of 217 volunteer interviewers, 108 returned the survey. These were primarily health professional clinicians at a mid-career stage. The median total cost to interviewers was AU$452 (IQR AU$252-715) for participation in a half-day interview session, largely due to the loss of income. The cost to the admitting institution was AU$269 per applicant, accounted for by the costs of equipment and consumables (52%), personnel (34%) and facilities (14%). CONCLUSIONS The costs of student selection for medical school are substantial. Understanding costs facilitates achievement of the objective of selecting the desired future medical workforce within the constraints of the resources available. Opportunities for change may arise from changes in applicant preparation behaviours, opportunities for economies of scale, and efficiencies driven by technological solutions.
Collapse
Affiliation(s)
- Jonathan Foo
- Department of Physiotherapy, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
| | - George Rivers
- Department of Economics, Faculty of Business and Economics, Monash University, Melbourne, Victoria, Australia
| | - Louise Allen
- Monash Institute for Health and Clinical Education, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
| | - Dragan Ilic
- Medical Education Research and Quality Unit, Monash University, Melbourne, Victoria, Australia
| | - Stephen Maloney
- Department of Physiotherapy, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
| | - Margaret Hay
- Monash Institute for Health and Clinical Education, Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Victoria, Australia
| |
Collapse
|
10
|
Cleland J, Chu J, Lim S, Low J, Low-Beer N, Kwek TK. COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape. MEDICAL TEACHER 2020; 42:776-780. [PMID: 32412815 DOI: 10.1080/0142159x.2020.1762851] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Introduction: The COVID-19 pandemic presented numerous, significant challenges for medical schools, including how to select the best candidates from a pool of applicants when social distancing and other measures prevented "business as usual" admissions processes. However, selection into medical school is the gateway to medicine in many countries, and it is critical to use processes which are evidence-based, valid and reliable even under challenging circumstances. Our challenge was to plan and conduct a multiple-mini interview (MMI) in a dynamic and stringent safe distancing context.Methods: This paper reports a case study of how to plan, re-plan and conduct MMIs in an environment where substantially tighter safe distancing measures were introduced just before the MMI was due to be delivered.Results: We report on how to design and implement a fully remote, online MMI which ensured the safety of candidates and assessors.Discussion: We discuss the challenges of this approach and also reflect on broader issues associated with selection into medical school during a pandemic. The aim of the paper is to provide broadly generalizable guidance to other medical schools faced with the challenge of selecting future students under difficult conditions.
Collapse
Affiliation(s)
- Jennifer Cleland
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Jowe Chu
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Samuel Lim
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Jamie Low
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Naomi Low-Beer
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Tong Kiat Kwek
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| |
Collapse
|
11
|
Mommert A, Wagner J, Jünger J, Westermann J. Exam performance of different admission quotas in the first part of the state examination in medicine: a cross-sectional study. BMC MEDICAL EDUCATION 2020; 20:169. [PMID: 32450862 PMCID: PMC7249435 DOI: 10.1186/s12909-020-02069-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/11/2019] [Accepted: 05/11/2020] [Indexed: 05/06/2023]
Abstract
BACKGROUND Most medical students in Germany are admitted via selection procedures, which are adjusted to the demands of the universities. At Lübeck medical school, scores from interviews that measure non-academic skills and pre-university GPAs are summed to arrive at an admission decision. This article seeks to illuminate the effectiveness of this selection procedure in comparison to other non-selected student groups. METHODS Quota information and exam results from the first federal exam were linked for students admitted to Lübeck medical school between 2012 and 2015 (N = 655). Five different student groups (university-specific selection quota, pre-university GPA quota, waiting time quota, ex-ante quota and foreign students) were compared regarding exam attempts, written and oral grades, temporal continuity and examination success in the standard study period. RESULTS While the pre-university GPA quota outperformed all other quotas regarding written and oral grades, it did not differ from the selection quota regarding exam attempts, temporal continuity and examination success in the standard study period. Students in the waiting time and ex-ante quotas performed inferior by comparison. The results of foreign students were the most problematic. CONCLUSION Students selected by the university show high temporal continuity and examination success. These results, and possible advantages in physician eligibility, argue for the utilisation of non-academic skills for admission.
Collapse
Affiliation(s)
- Alex Mommert
- Division of Study and Teaching, Faculty of Medicine, University of Lübeck, Ratzeburger Allee 160, House 2, 23562, Lübeck, Germany.
| | - Josefin Wagner
- Division of Study and Teaching, Faculty of Medicine, University of Lübeck, Ratzeburger Allee 160, House 2, 23562, Lübeck, Germany
| | - Jana Jünger
- The German National Institute for state examinations in Medicine, Pharmacy and Psychotherapy, Mainz, Germany
| | - Jürgen Westermann
- Division of Study and Teaching, Faculty of Medicine, University of Lübeck, Ratzeburger Allee 160, House 2, 23562, Lübeck, Germany
- Institute of Anatomy, University of Lübeck, Lübeck, Germany
| |
Collapse
|
12
|
Schreurs S, Cleutjens K, Collares CF, Cleland J, Oude Egbrink MGA. Opening the black box of selection. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2020; 25:363-382. [PMID: 31598883 PMCID: PMC7210244 DOI: 10.1007/s10459-019-09925-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/18/2019] [Accepted: 10/01/2019] [Indexed: 05/21/2023]
Abstract
Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.
Collapse
Affiliation(s)
- Sanne Schreurs
- Department of Educational Development and Research, School of Health Professions Education (SHE), Faculty of Health, Medicine and Life Sciences, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands.
| | - Kitty Cleutjens
- Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Carlos F Collares
- Department of Educational Development and Research, School of Health Professions Education (SHE), Faculty of Health, Medicine and Life Sciences, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands
| | - Jennifer Cleland
- Centre for Healthcare Education Research and Innovation (CHERI), University of Aberdeen, Foresterhill, Aberdeen, UK
| | - Mirjam G A Oude Egbrink
- Department of Physiology, School of Health Professions Education (SHE), Institute for Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
13
|
Schreurs S, Cleutjens K, oude Egbrink MGA. Increasing value in research: cost evaluations in health professions education. MEDICAL EDUCATION 2019; 53:1171-1173. [PMID: 31750571 PMCID: PMC6899714 DOI: 10.1111/medu.14007] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
The authors use Foo et al.'s discussion of the value of economic evaluations to consider how such techniques might advance the practice of selection for medical school.
Collapse
Affiliation(s)
- Sanne Schreurs
- Department of Educational Development and ResearchSchool of Health Professions EducationFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| | - Kitty Cleutjens
- Department of PathologyFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| | - Mirjam G A oude Egbrink
- Department of PhysiologyInstitute for Education and School of Health Professions EducationFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| |
Collapse
|
14
|
Vos CMP, Wouters A, Jonker M, de Haan M, Westerhof MA, Croiset G, Kusurkar RA. Bachelor completion and dropout rates of selected, rejected and lottery-admitted medical students in the Netherlands. BMC MEDICAL EDUCATION 2019; 19:80. [PMID: 30866918 PMCID: PMC6417017 DOI: 10.1186/s12909-019-1511-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/09/2018] [Accepted: 03/03/2019] [Indexed: 06/09/2023]
Abstract
BACKGROUND Evidence for the effectiveness of the selection of medical students is weak. This study aimed to examine the added value of a two-step selection procedure (first step non-academic, second step academic tests) to a pre-university GPA-based lottery procedure. Because previous research has suggested that participation in selection (regardless of the outcome) is a predictor of study success, this study is the first to include students who initially applied for selection, then refrained from (actively) participating in selection and were eventually admitted through lottery. METHODS Bachelor completion and dropout rates of selected (n = 416) and lottery-admitted students from four cohorts (2006-2009) were compared using logistic regression analysis. Four groups of lottery-admitted students were distinguished: students who were rejected after step 2 (n = 57), were rejected after step 1 (n = 169), withdrew during selection step 1 (n = 42) and students who only applied for lottery (n = 366). Covariates included gender, age, pre-university GPA and cohort. RESULTS There was a significant association between admission group and obtaining a bachelor degree in three years. Selected students were more likely to obtain a bachelor degree within three years (64.2% versus 51.6%; OR = 1.7) or four years (81.5% versus 74.3%; OR = 1.6) than students who only applied to a lottery (p < 0.05); selected students also seemed more likely to obtain all Year-1 course credits than students who withdrew during step 1 (40.4% versus 21.4%; OR = 2.3; p < 0.05). We found no significant association between dropout and admission groups. Students rejected at step 1 or 2 did not perform significantly different from selected students on any of the outcome measures. CONCLUSIONS The findings indicated that students at risk for study delay in the preclinical phase in our context were more likely to refrain from applying to a demanding selection procedure when a less demanding alternative was available. We found no significant associations between the non-academic and academic selection steps and bachelor completion and dropout rates. These findings suggest that the presence of the selection was more important than these specific selection components. In follow-up research, we plan to investigate the associations between the admission groups and outcome measures in the clinical phase.
Collapse
Affiliation(s)
- Catharina M. P. Vos
- Department of Research in Education, Amsterdam UMC, Vrije Universiteit Amsterdam, VUmc School of Medical Sciences, Amsterdam, the Netherlands
- LEARN! research institute for learning and education. Faculty of Psychology and Education, VU University, Amsterdam, the Netherlands
- Department of Policy, Innovation and Quality, Amsterdam UMC, Vrije Universiteit Amsterdam, VUmc School of Medical Sciences , Amsterdam, the Netherlands
| | - Anouk Wouters
- Department of Research in Education, Amsterdam UMC, Vrije Universiteit Amsterdam, VUmc School of Medical Sciences, Amsterdam, the Netherlands
- LEARN! research institute for learning and education. Faculty of Psychology and Education, VU University, Amsterdam, the Netherlands
- Department of Policy, Innovation and Quality, Amsterdam UMC, Vrije Universiteit Amsterdam, VUmc School of Medical Sciences , Amsterdam, the Netherlands
| | - Marianne Jonker
- At the time of the study: Department of Epidemiology & Biostatistics, VU University Medical Center, Amsterdam, the Netherlands
- Present address:Department of Health Evidence, Radboud University Nijmegen, Medical Center, Nijmegen, the Netherlands
| | - Marian de Haan
- Department of Policy, Innovation and Quality, Amsterdam UMC, Vrije Universiteit Amsterdam, VUmc School of Medical Sciences , Amsterdam, the Netherlands
| | - Marleen A. Westerhof
- Department of Policy, Innovation and Quality, Amsterdam UMC, Vrije Universiteit Amsterdam, VUmc School of Medical Sciences , Amsterdam, the Netherlands
| | - Gerda Croiset
- Department of Research in Education, Amsterdam UMC, Vrije Universiteit Amsterdam, VUmc School of Medical Sciences, Amsterdam, the Netherlands
- LEARN! research institute for learning and education. Faculty of Psychology and Education, VU University, Amsterdam, the Netherlands
- UMC Groningen, Groningen, the Netherlands
| | - Rashmi A. Kusurkar
- Department of Research in Education, Amsterdam UMC, Vrije Universiteit Amsterdam, VUmc School of Medical Sciences, Amsterdam, the Netherlands
- LEARN! research institute for learning and education. Faculty of Psychology and Education, VU University, Amsterdam, the Netherlands
| |
Collapse
|
15
|
Wouters A, Croiset G, Kusurkar RA. Selection and lottery in medical school admissions: who gains and who loses? MEDEDPUBLISH 2018; 7:271. [PMID: 38089206 PMCID: PMC10711990 DOI: 10.15694/mep.2018.0000271.1] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/28/2024] Open
Abstract
This article was migrated. The article was marked as recommended. Concerns related to fairness of medical school admissions through selection have led some scholars to consider selection as an expensive lottery and suggest that lottery may be fairer. This paper considers the issue of selection versus lottery from the perspectives of three groups of stakeholders: 1) applicants, 2) medical schools, and 3) society. This paper contributes to the discussion by addressing advantages and disadvantages of the use of selection and lottery for these stakeholder groups, grounded in the findings from research. Themes that are discussed are reliability and validity issues, perceived influence on selection outcomes and student uptake, effects on student diversity, financial costs, impact on rejected applicants, transparency, and strategic behaviour. For each stakeholder group both lottery and selection yield a combination of advantages and disadvantages, which implies that none of the currently available admissions strategies completely fulfils stakeholders' needs. Research indicates that selection yields only small gains compared to a lottery procedure, while the student diversity, necessary for serving the increasingly diverse patient population, may be compromised. We argue that society's needs should drive admissions policies rather than institutional gains, which means that until a selection procedure is developed that does not disadvantage certain types of students, a lottery procedure should be preferred.
Collapse
|
16
|
Schreurs S, Cleutjens KB, Muijtjens AMM, Cleland J, Oude Egbrink MGA. Selection into medicine: the predictive validity of an outcome-based procedure. BMC MEDICAL EDUCATION 2018; 18:214. [PMID: 30223816 PMCID: PMC6142422 DOI: 10.1186/s12909-018-1316-x] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/26/2018] [Accepted: 08/23/2018] [Indexed: 05/15/2023]
Abstract
BACKGROUND Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a 'backward chaining' approach to selection, based on the competences of a 'good doctor'. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program. METHODS We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students. RESULTS Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs. CONCLUSIONS The use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.
Collapse
Affiliation(s)
- Sanne Schreurs
- Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, P.O. Box 616, 6229 ER, Maastricht, the Netherlands.
| | - Kitty B Cleutjens
- Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Arno M M Muijtjens
- Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, P.O. Box 616, 6229 ER, Maastricht, the Netherlands
| | - Jennifer Cleland
- Institute of Education for Medical and Dental Sciences, University of Aberdeen, Foresterhill, Aberdeen, UK
| | - Mirjam G A Oude Egbrink
- Department of Physiology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| |
Collapse
|