1
|
Post JJ. Does basic physician trainee selection predict successful progression in the Royal Australasian College of Physicians adult medicine training programme? A retrospective cohort study. Intern Med J 2024. [PMID: 39212372 DOI: 10.1111/imj.16507] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2024] [Accepted: 08/02/2024] [Indexed: 09/04/2024]
Abstract
BACKGROUND The optimal selection process for basic physician trainees (BPTs) is unclear. AIMS To assess an adult BPT selection process, including a standardised recruitment interview assessing clinical decision-making and situational judgement. METHODS A retrospective multi-year cohort study of applicants who were selected for interview during the annual recruitment for 2011-2015 clinical years in an adult basic physician training network in NSW. The predictive capacity of the final recruitment assessment score was compared with success in the Royal Australasian College of Physicians (RACP) adult medicine clinical examination within 2 years. RESULTS Four hundred thirty-six applicants were selected for interview (84-91 in each year). Summary mean post-interview scores ranged from 1.25 to 10, with the median ranging between 7 and 8. Median scores were higher in those who passed the examination than those who did not (7.75 vs 6.43, P < 0.0001). Those who did not sit for the examination in the relevant year or never sat for the examination had similar median scores (7 vs 6.75, P = 0.11). Those not deemed eligible for recruitment were less likely to pass the clinical examination within 2 years than those on the eligibility list (odds ratio (OR) = 0.38, 95% confidence interval (CI) = 0.25-0.57). Applicants with mean scores lower than 8 were less likely to pass than those in the top band (8.1-10); scores between 6.1 and 8 OR 0.42 (95% CI = 0.26-0.68), 4.1-6 OR 0.23 (95% CI = 0.13-0.41) and 2.1-4 OR 0.24 (95% CI = 0.10-0.60). CONCLUSION(S) The predictive capacity of the selection process for success in the RACP clinical examination within the minimum time period is high.
Collapse
Affiliation(s)
- Jeffrey J Post
- Department of Infectious Diseases, Prince of Wales Hospital, Sydney, New South Wales, Australia
- School of Clinical Medicine, UNSW Sydney, Sydney, New South Wales, Australia
| |
Collapse
|
2
|
Caretta-Weyer HA, Eva KW, Schumacher DJ, Yarris LM, Teunissen PW. Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:S98-S107. [PMID: 37983402 DOI: 10.1097/acm.0000000000005365] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/11/2023]
Abstract
PURPOSE The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda. METHOD Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion. RESULTS Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process. CONCLUSIONS This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.
Collapse
Affiliation(s)
- Holly A Caretta-Weyer
- H.A. Caretta-Weyer is associate professor and associate residency director, Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California; ORCID: https://orcid.org/0000-0002-9783-5797
| | - Kevin W Eva
- K.W. Eva is associate director and senior scientist, Centre for Health Education Scholarship, and professor and director, educational research and scholarship, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada; ORCID: http://orcid.org/0000-0002-8672-2500
| | - Daniel J Schumacher
- D.J. Schumacher is professor of pediatrics, Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: http://orcid.org/0000-0001-5507-8452
| | - Lalena M Yarris
- L.M. Yarris is professor and vice chair of faculty development, Department of Emergency Medicine, Oregon Health & Science University, Portland, Oregon
| | - Pim W Teunissen
- P.W. Teunissen is professor of workplace learning in health care, School of Health Professions Education, and gynecologist, Department of Obstetrics & Gynecology, Maastricht University and Maastricht University Medical Center, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-0930-0048
| |
Collapse
|
3
|
Jiang C, O'Neill B, Bennett H, Yazdabadi A. Relationship between selection criteria and trainee performance in medical specialty training: A retrospective longitudinal study. Australas J Dermatol 2023; 64:58-66. [PMID: 36658664 DOI: 10.1111/ajd.13979] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2022] [Revised: 01/01/2023] [Accepted: 01/04/2023] [Indexed: 01/21/2023]
Abstract
BACKGROUND Investigation of tools used for candidate selection as predictors of future performance in medical education has been primarily undertaken within the undergraduate setting, but little is known about the selection to medical specialist programs. This retrospective longitudinal study aims to explore correlations between selection tools and the performance of trainees enrolled in the Australasian College of Dermatologists (ACD), the accredited specialist medical college for training in dermatology in Australia. METHODS Data were collected from consecutive cohorts of ACD trainees commencing from 2007 to 2015. Predictive variables were trainee demographics and selection tools (prior academic qualification; research experience; clinical experience; curriculum vitae [CV] rating; interview scores). The outcome variables were exam performance. First Year ([Pharmacology and Clinical Sciences] and Fellowship Exam [Written, Clinical, and Overall Score]). Statistical analyses included: descriptive statistics summaries; bivariate correlation of selection criteria and demographic data with exam performance; and linear regression analysis to identify predictors of exam performance. RESULTS Demographic analysis (N = 172) showed that 64% of trainees were female, the average age was 30.5 years (± 3.47) and trainees living in high socioeconomic status (SES) areas were over-represented. Using Pearson correlation analysis, interview scores were significantly positively correlated with First Year Pharmacology results (p = 0.018), Fellowship Written results (p = 0.002), and Fellowship Overall Scores (p = 0.006). First Year Pharmacology Exam performance was most highly correlated with Fellowship Exam performance (p = 0.000). No association was identified between exam performance and gender, SES, prior academic qualification, research experience or CV rating. Linear regression analysis showed that interview score (p = 0.012), entry age (p = 0.026) and First Year Pharmacology score (p = 0.002) were predictors of Fellowship exam performance. CONCLUSION These results suggest that the comprehensive selection tools used by ACD are effective, with interviews being a highly valid tool linked to exam performance. These findings have important practical implications for assessing the selection process in specialist dermatology training.
Collapse
Affiliation(s)
- Chali Jiang
- The Australasian College of Dermatologists, Rhodes, New South Wales, Australia
| | - Brett O'Neill
- The Australasian College of Dermatologists, Rhodes, New South Wales, Australia
| | - Haley Bennett
- The Australasian College of Dermatologists, Rhodes, New South Wales, Australia
| | - Anosha Yazdabadi
- Department of Medical Education, School of Medicine University of Melbourne, Melbourne, Victoria, Australia.,Department of Dermatology, School of Medicine University of Melbourne, Melbourne, Victoria, Australia.,Department of Dermatology, Eastern Health, Monash University, Melbourne, Victoria, Australia
| |
Collapse
|
4
|
Cunningham C, Kiezebrink K. Insights on selection of undergraduate dental students. EUROPEAN JOURNAL OF DENTAL EDUCATION : OFFICIAL JOURNAL OF THE ASSOCIATION FOR DENTAL EDUCATION IN EUROPE 2022. [PMID: 35771180 DOI: 10.1111/eje.12834] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/07/2020] [Revised: 04/13/2022] [Accepted: 05/15/2022] [Indexed: 06/15/2023]
Abstract
INTRODUCTION The goal of selection to dental education was to find those with the greatest aptitude for dentistry. Recently, schools have introduced a variety of tools; however, these have often been adopted without appropriate evaluation regarding existing evidence for fairness, reliability or validity. We explore dental admissions staff beliefs about the quality of different selection tools, with the objective of exploring their decision-making in implementing selection practices. METHODS This qualitative study is underpinned by a social constructionist epistemology, in which our principal concern is "explicating the processes by which people come to describe, explain or otherwise account for the world (including themselves) in which they live." We conducted individual interviews with 15 of the 16 UK dental admission leads to elicit their views around admissions processes and aims. Data coding and analysis were initially inductive, using thematic analysis. After the themes emerged, we applied a deductive framework of affordances to group themes and then examined these for heuristics. RESULTS AND DISCUSSION We identified three main themes; "Selection Tool Use," "Widening Participation Practices," and "Professionalising the Admission Lead Role." Admission leads spoke favourably of tools that allowed a "holistic" view of the applicants "potential". Selection tools were favoured if they enabled "Gut feeling". Leads spoke of evaluating candidates, making sure they were "rounded", and "know what dentistry is all about." In justifying the use of elements of their procedure, the use of heuristics was prominent. CONCLUSION In order to minimise the potential consequences of poor selection decisions, it is important to acknowledge that dental admission leads are at risk of depending on sub-optimal heuristics to make judgements about effective selection (shaped by previous practices) rather than using more rational decision-making processes based on the extant evidence (regarding the quality of different selection tools). Future research may be usefully informed by the knowledge translation literature to offer solutions for improving selection practices in dental education.
Collapse
Affiliation(s)
- Claudia Cunningham
- Aberdeen Institute of Dentistry, University of Aberdeen, Aberdeen, Scotland, UK
| | - Kirsty Kiezebrink
- School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, Scotland, UK
| |
Collapse
|
5
|
Thomas A, Bussières A. Leveraging knowledge translation and implementation science in the pursuit of evidence informed health professions education. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:1157-1171. [PMID: 33651210 DOI: 10.1007/s10459-020-10021-y] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/08/2020] [Accepted: 12/17/2020] [Indexed: 06/12/2023]
Abstract
Evidence informed approaches to health professions education can ensure accountability to learners and society in providing meaningful and effective education and helping resource strained systems via streamlined and cost-efficient practices. Knowledge translation and implementation science are two areas of study originally developed in clinical medicine in response to concerns that health care practices were incongruent with the scientific evidence. Two decades of research have led to important advances in our understanding of the nature and magnitude of research-practice gaps, the factors that support or impede adoption of evidence in clinical decision-making, and in the design and evaluation of theory driven interventions to reduce gaps. This paper borrows concepts from knowledge translation and implementation science to further our thinking about how health professions education can 'truly' be evidence informed. The article is organised in four sections: a discussion of the impetus for the evidence informed health professions education movement; a description of the origins of knowledge translation and implementation science; a discussion on how knowledge translation and implementation science can be leveraged to advance the evidence informed health professions education agenda; and suggestions for future discussion and research. An example is used to illustrate the application of the underpinning principles of knowledge translation and implementation science. The authors suggest a theory driven, staged and systematic approach that integrates knowledge translation principles and processes and involves key stakeholders interested in promoting the application of educational research of evidence.
Collapse
Affiliation(s)
- Aliki Thomas
- School of Physical and Occupational Therapy, Faculty of Medicine and Health Sciences, McGill University, Montreal, Canada.
- Institute of Health Sciences Education, Faculty of Medicine and Health Sciences, McGill University, Montreal, Canada.
- Centre for Interdisciplinary Research in Rehabilitation, Montreal, Canada.
| | - André Bussières
- School of Physical and Occupational Therapy, Faculty of Medicine and Health Sciences, McGill University, Montreal, Canada
- Centre for Interdisciplinary Research in Rehabilitation, Montreal, Canada
- Département Chiropratique, Université du Québec à Trois-Rivières, Trois-Rivières, Canada
| |
Collapse
|
6
|
Foo J, Hay M. Designing for value in medical residency selection. MEDICAL EDUCATION 2020; 54:977-978. [PMID: 32629532 DOI: 10.1111/medu.14290] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/16/2020] [Accepted: 06/29/2020] [Indexed: 06/11/2023]
Affiliation(s)
- Jonathan Foo
- School of Primary and Allied Health Care, Monash University, Melbourne, Vic., Australia
| | - Margaret Hay
- Monash Centre for Professional Development and Monash Online Education, Monash University, Clayton, Vic., Australia
| |
Collapse
|
7
|
Bowe SN. "Fit" Has a Broader Meaning: Recognizing the Utility of Person-Environment Fit Theory in Residency Recruitment and Selection. J Grad Med Educ 2020; 12:637-638. [PMID: 33149839 PMCID: PMC7594772 DOI: 10.4300/jgme-d-20-00886.1] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Affiliation(s)
- Sarah N Bowe
- Associate Professor, Department of Otolaryngology-Head & Neck Surgery, San Antonio Uniformed Services Health Education Consortium
| |
Collapse
|
8
|
Schreurs S, Cleutjens KB, Cleland J, oude Egbrink MG. Outcomes-Based Selection Into Medical School: Predicting Excellence in Multiple Competencies During the Clinical Years. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2020; 95:1411-1420. [PMID: 32134790 PMCID: PMC7447174 DOI: 10.1097/acm.0000000000003279] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
PURPOSE Medical school selection committees aim to identify the best possible students and, ultimately, the best future doctors from a large, well-qualified, generally homogeneous pool of applicants. Constructive alignment of medical school selection, curricula, and assessment with the ultimate outcomes (e.g., CanMEDS roles) has been proposed as means to attain this goal. Whether this approach is effective has not yet been established. The authors addressed this gap by assessing the relationship between performance in an outcomes-based selection procedure and performance during the clinical years of medical school. METHOD Two groups of students were compared: (1) those admitted into Maastricht University Medical School via an outcomes-based selection procedure and (2) those rejected through this procedure who were admitted into the program through a national, grade-point-average-based lottery. The authors compared performance scores of students from the 2 groups on all 7 CanMEDS roles, using assessment data gathered during clinical rotations. The authors examined data from 3 cohorts (2011-2013). RESULTS Students admitted through the local, outcomes-based selection procedure significantly outperformed the initially rejected but lottery-admitted students in all years, and the differences between groups increased over time. The selected students performed significantly better in the CanMEDS roles of Communicator, Collaborator, and Professional in the first year of clinical rotations; in these 3 roles-plus Organizer-in the second year; and in 2 additional roles (Advocate and Scholar-all except Medical Expert) at the end of their clinical training. CONCLUSIONS A constructively aligned selection procedure has increasing predictive value across the clinical years of medical school compared with a GPA-based lottery procedure. The data reported here suggest that constructive alignment of selection, curricula, and assessment to ultimate outcomes is effective in creating a selection procedure predictive of clinical performance.
Collapse
Affiliation(s)
- Sanne Schreurs
- S. Schreurs is teacher/educational advisor, Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-0233-9775
| | - Kitty B.J.M. Cleutjens
- K.B.J.M. Cleutjens is associate professor, Department of Pathology, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-7870-1670
| | - Jennifer Cleland
- J. Cleland is full professor and John Simpson Chair of Medical Education Research, Centre for Healthcare Education Research and Innovation (CHERI), University of Aberdeen, Aberdeen, Scotland; ORCID: http://orcid.org/0000-0003-1433-9323
| | - Mirjam G.A. oude Egbrink
- M.G.A. oude Egbrink is full professor, Implementation of Educational Innovations, Department of Physiology, School of Health Professions Education, and scientific director, Institute for Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-5530-6598
| |
Collapse
|
9
|
Cleland J, Chu J, Lim S, Low J, Low-Beer N, Kwek TK. COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape. MEDICAL TEACHER 2020; 42:776-780. [PMID: 32412815 DOI: 10.1080/0142159x.2020.1762851] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Introduction: The COVID-19 pandemic presented numerous, significant challenges for medical schools, including how to select the best candidates from a pool of applicants when social distancing and other measures prevented "business as usual" admissions processes. However, selection into medical school is the gateway to medicine in many countries, and it is critical to use processes which are evidence-based, valid and reliable even under challenging circumstances. Our challenge was to plan and conduct a multiple-mini interview (MMI) in a dynamic and stringent safe distancing context.Methods: This paper reports a case study of how to plan, re-plan and conduct MMIs in an environment where substantially tighter safe distancing measures were introduced just before the MMI was due to be delivered.Results: We report on how to design and implement a fully remote, online MMI which ensured the safety of candidates and assessors.Discussion: We discuss the challenges of this approach and also reflect on broader issues associated with selection into medical school during a pandemic. The aim of the paper is to provide broadly generalizable guidance to other medical schools faced with the challenge of selecting future students under difficult conditions.
Collapse
Affiliation(s)
- Jennifer Cleland
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Jowe Chu
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Samuel Lim
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Jamie Low
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Naomi Low-Beer
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Tong Kiat Kwek
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| |
Collapse
|
10
|
Schreurs S, Cleutjens K, Collares CF, Cleland J, Oude Egbrink MGA. Opening the black box of selection. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2020; 25:363-382. [PMID: 31598883 PMCID: PMC7210244 DOI: 10.1007/s10459-019-09925-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/18/2019] [Accepted: 10/01/2019] [Indexed: 05/21/2023]
Abstract
Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.
Collapse
Affiliation(s)
- Sanne Schreurs
- Department of Educational Development and Research, School of Health Professions Education (SHE), Faculty of Health, Medicine and Life Sciences, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands.
| | - Kitty Cleutjens
- Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Carlos F Collares
- Department of Educational Development and Research, School of Health Professions Education (SHE), Faculty of Health, Medicine and Life Sciences, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands
| | - Jennifer Cleland
- Centre for Healthcare Education Research and Innovation (CHERI), University of Aberdeen, Foresterhill, Aberdeen, UK
| | - Mirjam G A Oude Egbrink
- Department of Physiology, School of Health Professions Education (SHE), Institute for Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
11
|
Patterson F, Galbraith K, Flaxman C, Kirkpatrick CM. Evaluation of a Situational Judgement Test to Develop Non-Academic Skills in Pharmacy Students. AMERICAN JOURNAL OF PHARMACEUTICAL EDUCATION 2019; 83:7074. [PMID: 32001871 PMCID: PMC6983889 DOI: 10.5688/ajpe7074] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/21/2018] [Accepted: 04/19/2019] [Indexed: 05/22/2023]
Abstract
Objective. To design, implement, and psychometrically evaluate a situational judgement test (SJT) to use as a formative assessment of pharmacy students' non-academic skills in an Australian-based university. Methods. An SJT was developed using a previously validated design process including involvement of subject matter experts. The first phase included design of a blueprint through stakeholder consultation and the development of bespoke attribute definitions and a tool specification. Following on from this, SJT items were developed through subject matter expert interviews and in-depth review process. Results. Students (702) from four different cohorts (first through fourth years) of a Bachelor of Pharmacy degree program completed the situational judgment test. Data from 648 students was eligible for inclusion in the analysis. The SJT demonstrated good reliability, appropriateness for use (difficulty and quality), fairness, and face validity. The variability in students' scores suggested that the SJT may be a useful metric to identify students most in need of additional support. Conclusion. Evaluation of the SJT demonstrated that the tool was valid, reliable, fair and appropriate to use as a formative assessment. Through implementing an SJT such as this, pharmacy students are provided the opportunity to receive feedback on their non-academic skills and consider how to approach challenging or unfamiliar situations before entering the profession.
Collapse
|
12
|
Benbassat J. Assessments of Non-academic Attributes in Applicants for Undergraduate Medical Education: an Overview of Advantages and Limitations. MEDICAL SCIENCE EDUCATOR 2019; 29:1129-1134. [PMID: 34457592 PMCID: PMC8368911 DOI: 10.1007/s40670-019-00791-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Affiliation(s)
- Jochanan Benbassat
- Smokler Center for Health Policy Research, Myers-JDC-Brookdale Institute, PO Box 3886, 91037 Jerusalem, Israel
| |
Collapse
|
13
|
Abstract
Mylène Lagarde, Luis Huicho, and Irene Papanicolas discuss different strategies policy makers can use to motivate health providers in order to improve quality of care
Collapse
Affiliation(s)
- Mylène Lagarde
- Department of Health Policy, London School of Economics, London, UK
| | - Luis Huicho
- Centro de Investigación en Salud Materna e Infantil and Centro de Investigación para el Desarrollo Integral y Sostenible, Universidad Peruana Cayetano Heredia, Lima, Peru
| | - Irene Papanicolas
- Department of Health Policy, London School of Economics, London, UK
- Department of Health Policy and Management, Harvard T H Chan School of Public Health, Boston, MA, USA
| |
Collapse
|
14
|
Boulet JR, Durning SJ. What we measure … and what we should measure in medical education. MEDICAL EDUCATION 2019; 53:86-94. [PMID: 30216508 DOI: 10.1111/medu.13652] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/04/2018] [Revised: 03/06/2018] [Accepted: 05/31/2018] [Indexed: 05/20/2023]
Abstract
CONTEXT As the practice of medicine evolves, the knowledge, skills and attitudes required to provide patient care will continue to change. These competency-based changes will necessitate the restructuring of assessment systems. High-quality assessment programmes are needed to fulfil health professions education's contract with society. OBJECTIVES We discuss several issues that are important to consider when developing assessments in health professions education. We organise the discussion along the continuum of medical education, outlining the tension between what has been deemed important to measure and what should be measured. We also attempt to alleviate some of the apprehension associated with measuring evolving competencies by discussing how emerging technologies, including simulation and artificial intelligence, can play a role. METHODS We focus our thoughts on the assessment of competencies that, at least historically, have been difficult to measure. We highlight several assessment challenges, discuss some of the important issues concerning the validity of assessment scores, and argue that medical educators must do a better job of justifying their use of specific assessment strategies. DISCUSSION As in most professions, there are clear tensions in medicine in relation to what should be assessed, who should be responsible for administering assessment content, and how much evidence should be gathered to support the evaluation process. Although there have been advances in assessment practices, there is still room for improvement. From the student's, resident's and practising physician's perspectives, assessments need to be relevant. Knowledge is certainly required, but there are other qualities and attributes that are important, and perhaps far more important. Research efforts spent now on delineating what makes a good physician, and on aligning new and upcoming assessment tools with the relevant competencies, will ensure that assessment practices, whether aimed at establishing competence or at fostering learning, are effective with respect to their primary goal: to produce qualified physicians.
Collapse
Affiliation(s)
- John R Boulet
- Foundation for Advancement of International Medical Education and Research (FAIMER), Philadelphia, Pennsylvania, USA
| | - Steven J Durning
- Department of Medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland, USA
| |
Collapse
|
15
|
Kreiter C, O’Shea M, Bruen C, Murphy P, Pawlikowska T. A meta-analytic perspective on the valid use of subjective human judgement to make medical school admission decisions. MEDICAL EDUCATION ONLINE 2018; 23:1522225. [PMID: 30286694 PMCID: PMC6179055 DOI: 10.1080/10872981.2018.1522225] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/03/2018] [Revised: 07/06/2018] [Accepted: 09/05/2018] [Indexed: 06/08/2023]
Abstract
While medical educators appear to believe that admission to the medical school should be governed, at least in part, by human judgement, there has been no systematic presentation of evidence suggesting it improves selection. From a fair testing perspective, legal, ethical, and psychometric considerations, all dictate that the scientific evidence regarding human judgement in selection should be given consideration. To investigate the validity of using human judgements in admissions, multi-disciplinary meta-analytic research evidence from the wider literature is combined with studies from within medical education to provide evidence regarding the fairness and validity of using interviews and holistic review in medical school admissions. Fourteen studies, 6 of which are meta-analytic studies that summarized 292 individual studies, were included in the final review. Within these studies, a total of 33 studies evaluated the reliability of the traditional interview. These studies reveal that the interview has low to moderate reliability (~.42) which significantly limits its validity. This is confirmed by over 100 studies examining interview validity which collectively show interview scores to be moderately correlated with important outcome variables (corrected value ~.29). Meta-analyses of over 150 studies demonstrate that mechanical/formula-based selection decisions produce better results than decisions made with holistic/clinical methods (human judgement). Three conclusions regarding the use of interviews and holistic review are provided by these meta-analyses. First, it is clear that the traditional interview has low reliability and that this significantly limits its validity. Second, the reliable variance from interview scores appears moderately predictive of outcomes that are relevant to consider in medical school admission. And third, the use of holistic review as a method of incorporating human judgement is not a valid alternative to mechanical/statistical approaches as the evidence clearly indicates that mechanistic methods are more predictive, reliable, cost efficient, and transparent.
Collapse
Affiliation(s)
- Clare Kreiter
- Department of Family Medicine, Office of Consultation and Research in Medical Education, University of Iowa Carver College of Medicine, Iowa City, IA, USA
| | - Marie O’Shea
- Health Professions Education Center, Royal College of Surgeons, Dublin, Ireland
| | - Catherine Bruen
- Technology Enhanced Learning Manager, Royal College of Surgeons, Dublin, Ireland
| | - Paul Murphy
- Information Specialist, Royal College of Surgeons, Dublin, Ireland
| | - Teresa Pawlikowska
- Director of Health Professions Education Centre, Royal College of Surgeons, Dublin, Ireland
| |
Collapse
|
16
|
Schreurs S, Cleland J, Muijtjens AMM, oude Egbrink MGA, Cleutjens K. Does selection pay off? A cost-benefit comparison of medical school selection and lottery systems. MEDICAL EDUCATION 2018; 52:1240-1248. [PMID: 30324680 PMCID: PMC6282742 DOI: 10.1111/medu.13698] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2018] [Revised: 06/18/2018] [Accepted: 07/23/2018] [Indexed: 05/15/2023]
Abstract
CONTEXT Resources for medical education are becoming more constrained, whereas accountability in medical education is increasing. In this constrictive environment, medical schools need to consider and justify their selection procedures in terms of costs and benefits. To date, there have been no studies focusing on this aspect of selection. OBJECTIVES We aimed to examine and compare the costs and benefits of two different approaches to admission into medical school: a tailored, multimethod selection process versus a lottery procedure. Our goal was to assess the relative effectiveness of each approach and to compare these in terms of benefits and costs from the perspective of the medical school. METHODS The study was conducted at Maastricht University Medical School, at which the selection process and a weighted lottery procedure ran in parallel for 3 years (2011-2013). The costs and benefits of the selection process were compared with those of the lottery procedure over three student cohorts throughout the Bachelor's programme. The extra costs of selection represented the monetary investment of the medical school in conducting the selection procedure; the benefits were derived from the increase in income generated by the prevention of dropout and the reductions in extra costs facilitated by decreases in the repetition of blocks and objective structured clinical examinations. RESULTS The tailor-made selection procedure cost about €139 000 when extrapolated to a full cohort of students (n = 286). The lottery procedure came with negligible costs for the medical school. However, the average benefits of selection compared with the lottery system added up to almost €207 000. CONCLUSIONS This study not only shows that conducting a cost-benefit comparison is feasible in the context of selection for medical school, but also that an 'expensive' selection process can be cost-beneficial in comparison with an 'inexpensive' lottery system. We encourage other medical schools to examine the cost-effectiveness of their own selection processes in relation to student outcomes in order to extend knowledge on this important topic.
Collapse
Affiliation(s)
- Sanne Schreurs
- Department of Educational Development and ResearchInstitute for Education and School of Health Professions EducationFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| | - Jennifer Cleland
- Centre for Healthcare Education Research and Innovation (CHERI)Institute of Education for Medical and Dental SciencesUniversity of AberdeenAberdeenUK
| | - Arno M M Muijtjens
- Department of Educational Development and ResearchInstitute for Education and School of Health Professions EducationFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| | - Mirjam G A oude Egbrink
- Department of PhysiologyInstitute for EducationFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| | - Kitty Cleutjens
- Department of PathologyFaculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtthe Netherlands
| |
Collapse
|
17
|
Patterson F, Roberts C, Hanson MD, Hampe W, Eva K, Ponnamperuma G, Magzoub M, Tekian A, Cleland J. 2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions. MEDICAL TEACHER 2018; 40:1091-1101. [PMID: 30251906 DOI: 10.1080/0142159x.2018.1498589] [Citation(s) in RCA: 55] [Impact Index Per Article: 9.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/15/2023]
Abstract
Selection and recruitment into healthcare education and practice is a key area of interest for educators with significant developments in research, policy, and practice in recent years. This updated consensus statement, developed through a multi-stage process, examines future opportunities and challenges in selection and recruitment. There is both a gap in the literature around and a compelling case for further theoretical and empirical literature to underpin the development of overall selection philosophes and policies and their enactment. More consistent evidence has emerged regarding the quality of different selection methods. Approaches to selection are context-dependent, requiring the consideration of an institution's philosophy regarding what they are trying to achieve, the communities it purports to serve, along with the system within which they are used. Diversity and globalization issues continue to be critically important topics. Further research is required to explore differential attainment and explain why there are substantial differences in culturally acceptable ways of approaching diversity and widening access. More sophisticated evaluation approaches using multi-disciplinary theoretical frameworks are required to address the issues. Following a discussion of these areas, 10 recommendations are presented to guide future research and practice and to encourage debate between colleagues across the globe.
Collapse
Affiliation(s)
- F Patterson
- a Work Psychology Group, Derby United Kingdom of Great Britain and Northern Ireland, UK
| | - C Roberts
- b Northern Clinical School, University of Sydney, Sydney, New South Wales, Australia
| | - M D Hanson
- c Department of Psychiatry, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - W Hampe
- d Department of Biochemistry and Molecular Cell Biology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - K Eva
- e Centre for Health Education Scholarship and Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| | - G Ponnamperuma
- f Centre for Medical Education, Yong Loo Lin School of Medicine, Singapore
| | - M Magzoub
- g Department of Medical Education, College of Medicine, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia
| | - A Tekian
- h Department of Medical Education, University of Illinois at Chicago, Chicago, Illinois, USA
| | - J Cleland
- i Centre for Healthcare Research and Innovation (CHERI), University of Aberdeen, UK
| |
Collapse
|
18
|
Schreurs S, Cleutjens KB, Muijtjens AMM, Cleland J, Oude Egbrink MGA. Selection into medicine: the predictive validity of an outcome-based procedure. BMC MEDICAL EDUCATION 2018; 18:214. [PMID: 30223816 PMCID: PMC6142422 DOI: 10.1186/s12909-018-1316-x] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/26/2018] [Accepted: 08/23/2018] [Indexed: 05/15/2023]
Abstract
BACKGROUND Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a 'backward chaining' approach to selection, based on the competences of a 'good doctor'. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program. METHODS We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students. RESULTS Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs. CONCLUSIONS The use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.
Collapse
Affiliation(s)
- Sanne Schreurs
- Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, P.O. Box 616, 6229 ER, Maastricht, the Netherlands.
| | - Kitty B Cleutjens
- Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Arno M M Muijtjens
- Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, P.O. Box 616, 6229 ER, Maastricht, the Netherlands
| | - Jennifer Cleland
- Institute of Education for Medical and Dental Sciences, University of Aberdeen, Foresterhill, Aberdeen, UK
| | - Mirjam G A Oude Egbrink
- Department of Physiology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| |
Collapse
|
19
|
Kumar K, Roberts C, Bartle E, Eley DS. Testing for medical school selection: What are prospective doctors' experiences and perceptions of the GAMSAT and what are the consequences of testing? ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2018; 23:533-546. [PMID: 29388087 DOI: 10.1007/s10459-018-9811-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/08/2017] [Accepted: 01/22/2018] [Indexed: 06/07/2023]
Abstract
Written tests for selection into medicine have demonstrated reliability and there is accumulating evidence regarding their validity, but we know little about the broader impacts or consequences of medical school selection tests from the perspectives of key stakeholders. In this first Australian study of its kind, we use consequential validity as a theoretical lens to examine how medical school students and applicants view and experience the Graduate Medical Schools Admission Test (GAMSAT), and the consequences of testing. Participants (n = 447) were recruited from five graduate-entry medical schools across Australia and a publicly available online test preparation forum. An online survey was used to gather demographic information, and quantitative and qualitative data. Quantitative data were analysed via descriptive statistics and qualitative data were thematically analysed. The findings showed there was a considerable financial burden associated with preparing for and sitting the GAMSAT and moderate agreement regarding the GAMSAT as a fair selection method. The main unintended consequences of using the GAMSAT as a selection tool included barriers related to test affordability and language, and socialisation into the hidden curriculum of medicine. Selection tools such as the GAMSAT have some limitations when the goals are to support equitable participation in medicine and professional identity development. Our study highlights the value interpretive and theoretically-informed research in contributing to the evidence base on medical school selection.
Collapse
Affiliation(s)
- K Kumar
- Prideaux Centre for Research in Health Professions Education, College of Medicine and Public Health, Flinders University, Adelaide, Australia.
| | - C Roberts
- Sydney Medical School - Northern, The University of Sydney, Sydney, Australia
| | - E Bartle
- School of Dentistry, University of Queensland, Brisbane, Australia
| | - D S Eley
- Office of Medical Education, Faculty of Medicine, University of Queensland, Brisbane, Australia
| |
Collapse
|
20
|
Zibarras L, Patterson F, Driver R. A future research agenda for selection into healthcare. EUROPEAN JOURNAL OF DENTAL EDUCATION : OFFICIAL JOURNAL OF THE ASSOCIATION FOR DENTAL EDUCATION IN EUROPE 2018; 22:e300-e302. [PMID: 28653402 DOI: 10.1111/eje.12279] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 05/30/2017] [Indexed: 06/07/2023]
Affiliation(s)
- L Zibarras
- Work Psychology Group, Derby, UK
- Department of Psychology, University of London, London, UK
| | - F Patterson
- Work Psychology Group, Derby, UK
- University of Cambridge, Cambridge, UK
| | - R Driver
- Work Psychology Group, Derby, UK
| |
Collapse
|