1
|
Schumacher DJ, Michelson C, Winn AS, Turner DA, Martini A, Kinnear B. A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees. MEDICAL EDUCATION 2024; 58:812-824. [PMID: 38088227 DOI: 10.1111/medu.15296] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/11/2023] [Revised: 11/03/2023] [Accepted: 11/21/2023] [Indexed: 06/04/2024]
Abstract
INTRODUCTION The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why? METHODS Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs. RESULTS PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes. DISCUSSION PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.
Collapse
Affiliation(s)
- Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Catherine Michelson
- Ann and Robert H. Lurie Children's Hospital of Chicago/Northwestern University, Chicago, Illinois, USA
| | - Ariel S Winn
- Boston Children's Hospital/Harvard Medical School, Boston, Massachusetts, USA
| | - David A Turner
- American Board of Pediatrics, Chapel Hill, North Carolina, USA
| | - Abigail Martini
- Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
| | - Benjamin Kinnear
- Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
2
|
Alharbi NS. Evaluating competency-based medical education: a systematized review of current practices. BMC MEDICAL EDUCATION 2024; 24:612. [PMID: 38831271 PMCID: PMC11149276 DOI: 10.1186/s12909-024-05609-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/13/2023] [Accepted: 05/27/2024] [Indexed: 06/05/2024]
Abstract
BACKGROUND Few published articles provide a comprehensive overview of the available evidence on the topic of evaluating competency-based medical education (CBME) curricula. The purpose of this review is therefore to synthesize the available evidence on the evaluation practices for competency-based curricula employed in schools and programs for undergraduate and postgraduate health professionals. METHOD This systematized review was conducted following the systematic reviews approach with minor modifications to synthesize the findings of published studies that examined the evaluation of CBME undergraduate and postgraduate programs for health professionals. RESULTS Thirty-eight articles met the inclusion criteria and reported evaluation practices in CBME curricula from various countries and regions worldwide, such as Canada, China, Turkey, and West Africa. 57% of the evaluated programs were at the postgraduate level, and 71% were in the field of medicine. The results revealed variation in reporting evaluation practices, with numerous studies failing to clarify evaluations' objectives, approaches, tools, and standards as well as how evaluations were reported and communicated. It was noted that questionnaires were the primary tool employed for evaluating programs, often combined with interviews or focus groups. Furthermore, the utilized evaluation standards considered the well-known competencies framework, specialized association guidelines, and accreditation criteria. CONCLUSION This review calls attention to the importance of ensuring that reports of evaluation experiences include certain essential elements of evaluation to better inform theory and practice.
Collapse
Affiliation(s)
- Nouf Sulaiman Alharbi
- Department of Medical Education, College of Medicine, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia.
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia.
- Ministry of the National Guard - Health Affairs, Riyadh, Saudi Arabia.
| |
Collapse
|
3
|
Kitto S, Fantaye AW, Zevin B, Fowler A, Sachdeva AK, Raiche I. A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs. JOURNAL OF SURGICAL EDUCATION 2024; 81:823-840. [PMID: 38679495 DOI: 10.1016/j.jsurg.2024.02.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 02/20/2024] [Indexed: 05/01/2024]
Abstract
OBJECTIVE Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties. DESIGN Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace. PARTICIPANTS A total of 42 empirical and nonempirical articles were included. RESULTS Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education. CONCLUSIONS This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.
Collapse
Affiliation(s)
- Simon Kitto
- Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore
| | - Arone W Fantaye
- Office of Continuing Professional Development, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | - Boris Zevin
- Department of Surgery, Queen's University, Kingston, Canada
| | - Amanda Fowler
- Department of Surgery, Memorial University of Newfoundland, St. John's, Newfoundland, Canada
| | - Ajit K Sachdeva
- Division of Education, American College of Surgeons, Chicago, Illinios
| | - Isabelle Raiche
- Department of Surgery, University of Ottawa, Ottawa, Canada.
| |
Collapse
|
4
|
Pinilla S, Lerch S, Lüdi R, Neubauer F, Feller S, Stricker D, Berendonk C, Huwendiek S. Entrustment versus performance scale in high-stakes OSCEs: Rater insights and psychometric properties. MEDICAL TEACHER 2023; 45:885-892. [PMID: 36919450 DOI: 10.1080/0142159x.2023.2187683] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
BACKGROUND Although entrustment scales are increasingly applied in workplace-based assessments, their role in OSCEs remains unclear. We investigated raters' perceptions using an entrustment scale and psychometric analyses. METHOD A mixed-methods design was used. OSCE raters' (n = 162) perceptions were explored via questionnaire and four focus groups (n = 14). Psychometric OSCE properties were analyzed statistically. RESULTS Raters (n = 53, response rate = 41%) considered the entrustment scale comprehensible (89%) and applicable (60%). A total of 43% preferred the entrustment scale, 21% preferred the global performance scale, and 36% were undecided. Raters' written comments indicated that while they appreciated the authenticity of entrustment levels, they considered them subjective. The focus groups highlighted three main themes: (1) recollections of the clinical workplace as a cognitive reference triggered by entrustment scales; (2) factors influencing entrustment decisions; and (3) cognitive load is reduced at the perceived cost of objectivity. Psychometric analyses (n = 480 students) revealed improvements in some OSCE metrics when entrustment and global performance scales were combined. CONCLUSION Entrustment scales are beneficial for high-stakes OSCEs and have greater clinical relevance from the raters' perspective. Our findings support the use of entrustment and global performance scales in combination.
Collapse
Affiliation(s)
- Severin Pinilla
- University Hospital of Old Age Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland
| | - Seraina Lerch
- University Hospital of Old Age Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
- Institute of Medical Psychology, Heidelberg University Hospital, Ruprecht-Karls University Heidelberg, Heidelberg, Germany
| | - Raphaela Lüdi
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland
| | - Florian Neubauer
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland
| | - Sabine Feller
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland
| | - Daniel Stricker
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland
| | - Christoph Berendonk
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland
| | - Sören Huwendiek
- Department for Assessment and Evaluation, Institute for Medical Education, University of Bern, Bern, Switzerland
| |
Collapse
|
5
|
Acai A. The role of groups in assessing learners with specific learning difficulties. MEDICAL EDUCATION 2023; 57:498-501. [PMID: 36851815 DOI: 10.1111/medu.15068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/20/2023] [Accepted: 02/24/2023] [Indexed: 05/12/2023]
Affiliation(s)
- Anita Acai
- Department of Psychiatry and Behavioural Neurosciences, McMaster University, Hamilton, Ontario, Canada
| |
Collapse
|
6
|
Maimone C, Dolan BM, Green MM, Sanguino SM, Garcia PM, O’Brien CL. Utilizing Natural Language Processing of Narrative Feedback to Develop a Predictive Model of Pre-Clerkship Performance: Lessons Learned. PERSPECTIVES ON MEDICAL EDUCATION 2023; 12:141-148. [PMID: 37151853 PMCID: PMC10162355 DOI: 10.5334/pme.40] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/20/2022] [Accepted: 04/19/2023] [Indexed: 05/09/2023]
Abstract
Background Natural language processing is a promising technique that can be used to create efficiencies in the review of narrative feedback to learners. The Feinberg School of Medicine has implemented formal review of pre-clerkship narrative feedback since 2014 through its portfolio assessment system but this process requires considerable time and effort. This article describes how natural language processing was used to build a predictive model of pre-clerkship student performance that can be utilized to assist competency committee reviews. Approach The authors took an iterative and inductive approach to the analysis, which allowed them to identify characteristics of narrative feedback that are both predictive of performance and useful to faculty reviewers. Words and phrases were manually grouped into topics that represented concepts illustrating student performance. Topics were reviewed by experienced reviewers, tested for consistency across time, and checked to ensure they did not demonstrate bias. Outcomes Sixteen topic groups of words and phrases were found to be predictive of performance. The best-fitting model used a combination of topic groups, word counts, and categorical ratings. The model had an AUC value of 0.92 on the training data and 0.88 on the test data. Reflection A thoughtful, careful approach to using natural language processing was essential. Given the idiosyncrasies of narrative feedback in medical education, standard natural language processing packages were not adequate for predicting student outcomes. Rather, employing qualitative techniques including repeated member checking and iterative revision resulted in a useful and salient predictive model.
Collapse
Affiliation(s)
- Christina Maimone
- Associate director of research data services, Northwestern IT Research Computing Services, Northwestern University, Evanston, Illinois, USA
| | - Brigid M. Dolan
- Associate professor of medicine and medical education and director of assessment, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | - Marianne M. Green
- Raymond H. Curry, MD Professor of Medical Education, professor of medicine, and vice dean for education, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | - Sandra M. Sanguino
- Associate professor of pediatrics and senior associate dean of medical education, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | - Patricia M. Garcia
- Professor of obstetrics and gynecology and medical education, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| | - Celia Laird O’Brien
- Assistant professor of medical education and assistant dean of program evaluation and accreditation, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
| |
Collapse
|
7
|
Maimone C, Dolan BM, Green MM, Sanguino SM, Garcia PM, O'Brien CL. Automated Assessment of Medical Students' Competency-Based Performance Using Natural Language Processing (NLP). ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:S165. [PMID: 37838894 DOI: 10.1097/acm.0000000000004813] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/20/2023]
Affiliation(s)
- Christina Maimone
- Author affiliations: C. Maimone, Northwestern University; B.M. Dolan, M.M. Green, S.M. Sanguino, P.M. Garcia, C. Laird O'Brien, Northwestern University Feinberg School of Medicine
| | | | | | | | | | | |
Collapse
|
8
|
Cheung WJ, Wagner N, Frank JR, Oswald A, Van Melle E, Skutovich A, Dalseg TR, Cooke LJ, Hall AK. Implementation of competence committees during the transition to CBME in Canada: A national fidelity-focused evaluation. MEDICAL TEACHER 2022; 44:781-789. [PMID: 35199617 DOI: 10.1080/0142159x.2022.2041191] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
PURPOSE This study evaluated the fidelity of competence committee (CC) implementation in Canadian postgraduate specialist training programs during the transition to competency-based medical education (CBME). METHODS A national survey of CC chairs was distributed to all CBME training programs in November 2019. Survey questions were derived from guiding documents published by the Royal College of Physicians and Surgeons of Canada reflecting intended processes and design. RESULTS Response rate was 39% (113/293) with representation from all eligible disciplines. Committee size ranged from 3 to 20 members, 42% of programs included external members, and 20% included a resident representative. Most programs (72%) reported that a primary review and synthesis of resident assessment data occurs prior to the meeting, with some data reviewed collectively during meetings. When determining entrustable professional activity (EPA) achievement, most programs followed the national specialty guidelines closely with some exceptions (53%). Documented concerns about professionalism, EPA narrative comments, and EPA entrustment scores were most highly weighted when determining resident progress decisions. CONCLUSIONS Heterogeneity in CC implementation likely reflects local adaptations, but may also explain some of the variable challenges faced by programs during the transition to CBME. Our results offer educational leaders important fidelity data that can help inform the larger evaluation and transformation of CBME.
Collapse
Affiliation(s)
- Warren J Cheung
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
| | - Natalie Wagner
- Office of Professional Development & Educational Scholarship and Department of Biomedical & Molecular Sciences, Queen's University, Kingston, Canada
| | - Jason R Frank
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
| | - Anna Oswald
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Medicine, University of Alberta, Edmonton, Canada
| | - Elaine Van Melle
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Family Medicine, Queen's University, Kingston, Canada
| | | | - Timothy R Dalseg
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Medicine, Division of Emergency Medicine, University of Toronto, Toronto, Canada
| | - Lara J Cooke
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
- Department of Clinical Neurosciences, Cumming School of Medicine, University of Calgary, Calgary, Canada
| | - Andrew K Hall
- Department of Emergency Medicine, University of Ottawa, Ottawa, Canada
- Royal College of Physicians and Surgeons of Canada, Ottawa, Canada
| |
Collapse
|