1
|
Costich M, Friedman S, Robinson V, Catallozzi M. Implementation and faculty perception of outpatient medical student workplace-based assessments. CLINICAL TEACHER 2024; 21:e13751. [PMID: 38433555 DOI: 10.1111/tct.13751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Accepted: 02/06/2024] [Indexed: 03/05/2024]
Abstract
BACKGROUND There is growing interest in use of entrustable professional activity (EPA)-grounded workplace-based assessments (WBAs) to assess medical students through direct observation in the clinical setting. However, there has been very little reflection on how these tools are received by the faculty using them to deliver feedback. Faculty acceptance of WBAs is fundamentally important to sustained utilisation in the clinical setting, and understanding faculty perceptions of the WBA as an adjunct for giving targeted feedback is necessary to guide future faculty development in this area. APPROACH Use of a formative EPA-grounded WBA was implemented in the ambulatory setting during the paediatrics clerkship following performance-driven training and frame-of-reference training with faculty. Surveys and semi-structured interviews with faculty members explored how faculty perceived the tool and its impact on feedback delivery. EVALUATION Faculty reported providing more specific, task-oriented feedback following implementation of the WBA, as well as greater timeliness of feedback and greater satisfaction with opportunities to provide feedback, although these later two findings did not reach significance. Themes from the interviews reflected the benefits of WBAs, persistent barriers to the provision of feedback and suggestions for improvement of the WBA. IMPLICATIONS EPA-grounded WBAs are feasible to implement in the outpatient primary care setting and improve feedback delivery around core EPAs. The WBAs positively impacted the way faculty conceptualise feedback and provide learners with more actionable, behaviour-based feedback. Findings will inform modifications to the WBA and future faculty development and training to allow for sustainable WBA utilisation in the core clerkship.
Collapse
Affiliation(s)
- Marguerite Costich
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
| | - Suzanne Friedman
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
| | - Victoria Robinson
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
| | - Marina Catallozzi
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Population and Family Health, Mailman School of Public Health, Columbia University Irving Medical Center, New York, New York, USA
| |
Collapse
|
2
|
Cotta RMM, de Souza Ferreira E, de Aguiar Franco F, da Costa Souza Barros G, Januário JPT, Moreira TR, Martín RL. The effectiveness of faculty development programs for training university professors in the health area: a systematic review and meta-analysis. BMC MEDICAL EDUCATION 2024; 24:768. [PMID: 39014385 PMCID: PMC11253325 DOI: 10.1186/s12909-024-05735-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Accepted: 07/02/2024] [Indexed: 07/18/2024]
Abstract
BACKGROUND The growing discussion on teacher development focuses on diversified educational skills that promote knowledge and innovation in the teaching, learning and assessment process. With the Covid-19 scenario, this picture of necessary changes has become more evident, demonstrating the need for professional preparation to work in teacher development. The aim of the study was to analyze the effectiveness of teacher development programs for the training of university teachers in the health area, through a systematic review and meta-analysis. METHODS The systematic review and meta-analysis were carried out according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and involved searching five databases - PubMed-Medline, Education Resource Information Center (ERIC), SCOPUS, Embase and Web of Science. The review included randomized clinical trials and cohort studies that addressed the effectiveness of teaching professionalization in the health area for university professors. The quality of the selected studies was assessed based on the evaluation criteria of the Joanna Briggs Institute tool. The random effects meta-analysis method was used to explain the distribution of effects between the studies, using Stata® software (version 11.0) and publication bias was examined by visual inspection of the graphs and Egger's test. RESULTS We included 12 studies in the systematic review and 8 in the meta-analysis. These studies were published between 1984 and 2022 in 14 countries. Significant changes were reported in teachers' behavior to stimulate and encourage students, improvement in the quality of teaching and teaching staff, as well as improvement in skills such as leadership and self-evaluation. Furthermore, the result of the meta-analysis showed that there is evidence of the effectiveness of the positive effects of teacher development programs after their implementation, with this effect being 1.70% and an increase of 4.75 in the effect of these teacher development programs. CONCLUSION Our study shows that development programs have been implemented in different countries and contexts, all of which have proven to be effective in the short, medium and long term. We recommend that future research focus specifically on the different competencies that have been acquired following the implementation of these programs.
Collapse
Affiliation(s)
| | | | | | | | | | - Tiago Ricardo Moreira
- Department of Medicine and Nursing, Federal University of Viçosa, Viçosa, MG, Brazil
| | - Ramón López Martín
- Facultad de Filosofía y Ciencias de La Educación, Universidad de Valencia, Valencia, VA, Spain
| |
Collapse
|
3
|
Langhan ML, Boyer DL, Hsu D, Moffatt ME, Pitts SA, Atlas MP, Aye T, Chess P, Curran ML, Czaja AS, Dammann CEL, Fussell J, George RP, Herman BE, High P, James SH, Kamin DS, Karnik R, Kesselheim J, Lopez MA, Mahan JD, McFadden V, McGann KA, Mehta JJ, Rama J, Robinson BW, Sauer C, Stafford DEJ, Turner DA, Weiss P, Yussman SM, Schwartz A, Mink R. Implementing Entrustable Professional Activities in Pediatric Fellowships: Facilitating the Process. Pediatrics 2024; 153:e2023065024. [PMID: 38757175 DOI: 10.1542/peds.2023-065024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Revised: 02/02/2024] [Accepted: 02/20/2024] [Indexed: 05/18/2024] Open
Abstract
BACKGROUND AND OBJECTIVES Entrustable professional activities (EPAs) will be used for initial certification by the American Board of Pediatrics by 2028. Less than half of pediatric fellowships currently use EPAs for assessment, yet all will need to adopt them. Our objectives were to identify facilitators and barriers to the implementation of EPAs to assess pediatric fellows and to determine fellowship program directors' (FPD) perceptions of EPAs and Milestones. METHODS We conducted a survey of FPDs from 15 pediatric subspecialties. EPA users were asked about their implementation of EPAs, barriers encountered, and perceptions of EPAs. Nonusers were queried about deterrents to using EPAs. Both groups were asked about potential facilitators of implementation and their perceptions of Milestones. RESULTS The response rate was 65% (575/883). Of these, 344 (59.8%) were EPA users and 231 (40.2%) were nonusers. Both groups indicated work burden as a barrier to implementation. Nonusers reported more barriers than users (mean [SD]: 7 [3.8] vs 5.8 [3.4], P < .001). Both groups identified training materials and premade assessment forms as facilitators to implementation. Users felt that EPAs were easier to understand than Milestones (89%) and better reflected what it meant to be a practicing subspecialty physician (90%). In contrast, nonusers felt that Milestones were easy to understand (57%) and reflected what it meant to be a practicing subspecialist (58%). CONCLUSIONS Implementing EPA-based assessment will require a substantial investment by FPDs, facilitated by guidance and easily accessible resources provided by multiple organizations. Perceived barriers to be addressed include FPD time constraints, a need for additional assessment tools, and outcomes data.
Collapse
Affiliation(s)
| | - Donald L Boyer
- Children's Hospital of Philadelphia, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania
| | - Deborah Hsu
- Stanford University School of Medicine, Stanford, California
| | - Mary E Moffatt
- University of Missouri Kansas City School of Medicine, Kansas City, Missouri
| | - Sarah A Pitts
- Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts
| | - Mark P Atlas
- Zucker School of Medicine at Hofstra-Northwell, Cohen Children's Medical Center, Queens, New York
| | - Tandy Aye
- Stanford University School of Medicine, Stanford, California
| | - Patricia Chess
- University of Rochester Medical Center, Rochester, New York
| | - Megan L Curran
- University of Colorado School of Medicine, Aurora, Colorado
| | - Angela S Czaja
- University of Colorado School of Medicine, Aurora, Colorado
| | | | - Jill Fussell
- University of Arkansas for Medical Sciences, Little Rock, Arkansas
| | - Roshan P George
- Emory University and Children's Healthcare of Atlanta, Atlanta, Georgia
| | - Bruce E Herman
- University of Utah School of Medicine, Salt Lake City, Utah
| | - Pamela High
- Warren Alpert Medical School of Brown University, Providence, Rhode Island
| | - Scott H James
- University of Alabama at Birmingham, Birmingham, Alabama
| | - Daniel S Kamin
- Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts
| | - Ruchika Karnik
- Yale University School of Medicine, New Haven, Connecticut
| | - Jennifer Kesselheim
- Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts
| | - Michelle A Lopez
- Baylor College of Medicine, Texas Children's Hospital, Houston, Texas
| | - John D Mahan
- The Ohio State University College of Medicine, Columbus, Ohio
| | | | | | - Jay J Mehta
- Children's Hospital of Philadelphia, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania
| | - Jennifer Rama
- Baylor College of Medicine, Texas Children's Hospital, Houston, Texas
| | - Bradley W Robinson
- Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, Pennsylvania
| | - Cary Sauer
- Emory University and Children's Healthcare of Atlanta, Atlanta, Georgia
| | | | - David A Turner
- American Board of Pediatrics, Chapel Hill, North Carolina
| | - Pnina Weiss
- Yale University School of Medicine, New Haven, Connecticut
| | | | - Alan Schwartz
- University of Illinois College of Medicine at Chicago, Chicago, Illinois
| | - Richard Mink
- Harbor-UCLA Medical Center and The Lundquist Institute for Biomedical Innovation, Torrance, California
| |
Collapse
|
4
|
Birman NA, Vashdi DR, Miller-Mor Atias R, Riskin A, Zangen S, Litmanovitz I, Sagi D. Unveiling the paradoxes of implementing post graduate competency based medical education programs. MEDICAL TEACHER 2024:1-8. [PMID: 38803298 DOI: 10.1080/0142159x.2024.2356826] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/05/2023] [Accepted: 05/14/2024] [Indexed: 05/29/2024]
Abstract
PURPOSE Competency-based medical education (CBME) has gained prominence as an innovative model for post-graduate medical education, yet its implementation poses significant challenges, especially with regard to its sustainability. Drawing on paradox theory, we suggest that revealing the paradoxes underlying these challenges may contribute to our understanding of post graduate competency-based medical education (PGCBME) implementation processes and serve as a first-step in enhancing better implementation. Thus, the purpose of the current study is to identify the paradoxes associated with PGCBME implementation. METHOD A qualitative study was conducted, as part of a larger action research, using in-depth semi-structured interviews with fellows and educators in eight Neonatal wards. RESULTS Analysis revealed that the PGCBME program examined in this study involves three different levels of standardization, each serving as one side of paradoxical tensions; (1) a paradox between the need for standardized assessment tools and for free-flow flexible assessment tools, (2) a paradox between the need for a standardized implementation process across all wards and the need for unique implementation protocols in each ward; and 3) a paradox between the need for a standardized meaning of competency proficiency and the need for flexible and personal competency achievement indicators. CONCLUSIONS Implementing PGCBME programs involves many challenges, some of which are paradoxical, i.e. two contradictory challenges in which solving one challenge exacerbates another. Revealing these paradoxes is important in navigating them successfully.
Collapse
Affiliation(s)
- Noa A Birman
- University of Haifa, The Herta and Paul Amir Faculty of Social Sciences, School of Political Science, Department of Public Administration, Haifa, Israel
| | - Dana R Vashdi
- University of Haifa, The Herta and Paul Amir Faculty of Social Sciences, School of Political Science, Department of Public Administration, Haifa, Israel
| | - Rotem Miller-Mor Atias
- University of Haifa, The Herta and Paul Amir Faculty of Social Sciences, School of Political Science, Department of Public Administration, Haifa, Israel
| | - Arieh Riskin
- Technion Israel Institute of Technology, The Ruth and Bruce Rappaport Faculty of Medicine, Haifa, Israel
| | - Shmuel Zangen
- Ben- Gurion University of the Negev, Faculty of Health Sciences, Be'er-Sheva, Israel
| | - Ita Litmanovitz
- Tel Aviv University, Faculty of Medicine & Health Sciences, Tel-Aviv, Israel
| | - Doron Sagi
- The Israel Center for Medical Simulation, Sheba Medical Center, Tel-Hashomer, Ramat-Gan, Israel
| |
Collapse
|
5
|
Andreou V, Peters S, Eggermont J, Schoenmakers B. Co-designing Entrustable Professional Activities in General Practitioner's training: a participatory research study. BMC MEDICAL EDUCATION 2024; 24:549. [PMID: 38760773 PMCID: PMC11100052 DOI: 10.1186/s12909-024-05530-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/25/2023] [Accepted: 05/07/2024] [Indexed: 05/19/2024]
Abstract
BACKGROUND In medical education, Entrustable Professional Activities (EPAs) have been gaining momentum for the last decade. Such novel educational interventions necessitate accommodating competing needs, those of curriculum designers, and those of users in practice, in order to be successfully implemented. METHODS We employed a participatory research design, engaging diverse stakeholders in designing an EPA framework. This iterative approach allowed for continuous refinement, shaping a comprehensive blueprint comprising 60 EPAs. Our approach involved two iterative cycles. In the first cycle, we utilized a modified-Delphi methodology with clinical competence committee (CCC) members, asking them whether each EPA should be included. In the second cycle, we used semi-structured interviews with General Practitioner (GP) trainers and trainees to explore their perceptions about the framework and refine it accordingly. RESULTS During the first cycle, 14 CCC members agreed that all the 60 EPAs should be included in the framework. Regarding the formulation of each EPAs, 20 comments were given and 16 adaptations were made to enhance clarity. In the second cycle, the semi-structured interviews with trainers and trainees echoed the same findings, emphasizing the need of the EPA framework for improving workplace-based assessment, and its relevance to real-world clinical scenarios. However, trainees and trainers expressed concerns regarding implementation challenges, such as the large number of EPAs to be assessed, and perception of EPAs as potentially high-stakes. CONCLUSION Accommodating competing stakeholders' needs during the design process can significantly enhance the EPA implementation. Recognizing users as experts in their own experiences empowers them, enabling a priori identification of implementation barriers and potential pitfalls. By embracing a collaborative approach, wherein diverse stakeholders contribute their unique viewpoints, we can only create effective educational interventions to complex assessment challenges.
Collapse
Affiliation(s)
- Vasiliki Andreou
- Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium.
- Department of Public Health and Primary Care, KU Leuven, Box 7001, Kapucijnenvoer 7, Leuven, 3000, Belgium.
| | - Sanne Peters
- Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium
- School of Health Sciences, Faculty of Medicine, Dentistry and Health Sciences, The University of Melbourne, Melbourne, Australia
| | - Jan Eggermont
- Department of Cellular and Molecular Medicine, KU Leuven, Leuven, Belgium
| | - Birgitte Schoenmakers
- Academic Centre for General Practice, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium
| |
Collapse
|
6
|
Elster MJ, Parsons AS, Collins S, Gusic ME, Hauer KE. 'We're like Spider-Man; with great power comes great responsibility': Coaches' experiences supporting struggling medical students. MEDICAL TEACHER 2024:1-9. [PMID: 38588710 DOI: 10.1080/0142159x.2024.2337250] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/29/2023] [Accepted: 03/26/2024] [Indexed: 04/10/2024]
Abstract
BACKGROUND Medical students can experience a range of academic and non-academic struggles. Coaching is a valuable strategy to support learners, but coaches describe working with struggling learners as taxing. Transformative learning theory (TLT) provides insights into how educators grow from challenging experiences to build resilience. This study explores how coaches evolve as educators through supporting struggling students. METHODS This qualitative study grounded in an interpretivist paradigm used interviews of longitudinal medical student coaches at two academic institutions. Interviews, using TLT as a sensitizing concept, explored coaches' experience coaching struggling learners. We performed thematic analysis. RESULTS We interviewed 15 coaches. Coaches described supporting students through multi-faceted struggles which often surprised the coach. Three themes characterized coaches' experiences: personal responsibility, emotional response, and personal learning. Coaches shouldered high personal responsibility for learners' success. For some, this burden felt emotional, raised parental instincts and questions about maintaining boundaries with learners. Coaches evolved their coaching approach, challenged biases, and built skills. Coaches learned to better appreciate the learner point of view and employ resources to support students. DISCUSSION Through navigating learner struggles, educators can gain self-efficacy, learn to understand learners' perspectives, and evolve their coaching approach to lessen their personal emotional burden through time.
Collapse
Affiliation(s)
- Martha J Elster
- University of California San Francisco School of Medicine, San Francisco, California, USA
| | - Andrew S Parsons
- University of Virginia School of Medicine, Charlottesville, Virginia, USA
| | - Sally Collins
- University of California San Francisco School of Medicine, San Francisco, California, USA
| | | | - Karen E Hauer
- University of California San Francisco School of Medicine, San Francisco, California, USA
| |
Collapse
|
7
|
Bentley H, Darras KE, Forster BB, Probyn L, Sedlic A, Hague CJ. Knowledge and Perceptions of Competency-Based Medical Education in Diagnostic Radiology Post-Graduate Medical Education: Identifying Priorities and Developing a Framework for Professional Development Activities. Can Assoc Radiol J 2023; 74:487-496. [PMID: 36384331 DOI: 10.1177/08465371221137087] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/23/2023] Open
Abstract
Introduction: We evaluated knowledge and perceptions of an established Competency-Based Medical Education (CBME) model developed by the Royal College of Physicians and Surgeons of Canada, Competence by Design (CBD), and identified evidence-informed priorities for professional development activities (PDAs). Materials and Methods: Teaching faculty and residents at a single, large diagnostic radiology post-graduate medical education (PGME) program were eligible to participate in this cross-sectional, survey-based study. Knowledge of CBD was evaluated through multiple choice questions (MCQs), which assessed participants' understanding of major principles and terms associated with CBD. Participants' perceptions of the anticipated impact of CBD on resident education and patient care were evaluated and priorities for PDAs were identified, which informed a framework for CBD PDAs. Results: Fifty faculty and residents participated. The faculty and resident response rates were 11.6% (n = 29/249) and 55.3% (n = 21/38), respectively. The mean ± standard deviation overall score on MCQs was 39.0% ± 20.4%. The majority of participants perceived the impact of CBD on resident education to be equivocal and to not impact patient care. Knowledge of CBD was not statistically significantly associated with participants' perceptions of the impact of CBD on either resident education or patient care (P > .05). Delivery of high-quality feedback was the greatest priority identified for PDAs. Discussion: Our results and proposed CBD PDAs framework may help to guide diagnostic radiology PGME programs in designing evidence-informed PDAs, which may meaningfully contribute to the successful implementation of CBD in diagnostic radiology PGME. As diagnostic radiology PGME programs throughout the world increasingly implement CBME models, evidence-informed PDAs will become of increasing importance.
Collapse
Affiliation(s)
- Helena Bentley
- Department of Radiology, Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada
| | - Kathryn E Darras
- Department of Radiology, Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada
- Department of Radiology, Vancouver General Hospital, Vancouver, BC, Canada
| | - Bruce B Forster
- Department of Radiology, Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada
- Department of Radiology, Vancouver General Hospital, Vancouver, BC, Canada
| | - Linda Probyn
- Department of Medical Imaging, Faculty of Medicine, University of Toronto, Toronto, ON, Canada
| | - Anto Sedlic
- Department of Radiology, Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada
- Department of Radiology, Vancouver General Hospital, Vancouver, BC, Canada
| | - Cameron J Hague
- Department of Radiology, Faculty of Medicine, University of British Columbia, Vancouver, BC, Canada
- Department of Radiology, St Paul's Hospital, Vancouver, BC, Canada
| |
Collapse
|
8
|
Ehlinger C, Fernandez N, Strub M. Entrustable professional activities in dental education: a scoping review. Br Dent J 2023; 234:171-176. [PMID: 36765231 DOI: 10.1038/s41415-023-5503-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2022] [Revised: 11/28/2022] [Accepted: 12/09/2022] [Indexed: 02/12/2023]
Abstract
Background Competency-based education has led to the introduction of entrustable professional activities (EPAs) in health sciences education. EPAs are assessment tools that serve to certify a trainee's preparedness to conduct a given clinical activity. Objective Given its modest introduction into dental education, we sought to examine the current situation of EPAs in our field and identify possible barriers and facilitators. This review aimed to summarise evidence about EPA introduction in dental education programmes.Data sources A review of articles published between January 2005 and December 2021 was conducted using the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews methodology in three databases (PubMed, Cochrane and Embase).Data selection Keywords were 'entrustable professional activity' OR 'entrustable professional activities' AND 'dentistry' OR 'dental education'.Data extraction Eligibility criteria targeted papers published in English describing dental education EPA.Data synthesis Ten publications were selected: six reported on new EPAs, three discussed their relevance for dental education and one article described a process for developing validated EPAs. EPAs focused on clinical examination, health promotion, treatment planning, pain and/or anxiety control and topics related to a specific discipline or care procedures.Conclusion Topics covered by EPAs described in our study adequately reflect the specificities of today's dental professional. The benefits of EPAs will ensure tomorrow's dentists consistently meet societal expectations.
Collapse
Affiliation(s)
- Claire Ehlinger
- Université de Strasbourg, Faculté de Chirurgie Dentaire, 8 Rue Sainte Elisabeth, 67000 Strasbourg, France; Hôpitaux Universitaires de Strasbourg, Pôle de Médecine et Chirurgie Bucco-Dentaires, 1 Place de l'Hôpital, 67000 Strasbourg, France
| | - Nicolas Fernandez
- Département de Médecine de Famille et Médecine D'urgence, Faculty of Medicine, Université de Montréal, 2900 Boulevard Édouard-Montpetit, Montréal, Québec, H3C 1J7, Canada
| | - Marion Strub
- Université de Strasbourg, Faculté de Chirurgie Dentaire, 8 Rue Sainte Elisabeth, 67000 Strasbourg, France; Hôpitaux Universitaires de Strasbourg, Pôle de Médecine et Chirurgie Bucco-Dentaires, 1 Place de l'Hôpital, 67000 Strasbourg, France.
| |
Collapse
|
9
|
Keeley MG, Bray MJ, Bradley EB, Peterson CM, Waggoner-Fountain LA, Gusic ME. Fidelity to Best Practices in EPA Implementation: Outcomes Supporting Use of the Core Components Framework From the University of Virginia Entrustable Professional Activity Program. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1637-1642. [PMID: 35976718 DOI: 10.1097/acm.0000000000004944] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
PROBLEM The rapid expansion of entrustable professional activity (EPA) assessment programs has led to calls to ensure fidelity in implementation and integrity in meeting the goals of competency-based medical education. Initiated in July 2017, in advance of the articulated core components of EPA implementation, this article describes the structure and outcomes of the University of Virginia (UVA) EPA Program and provides support for the identified essential components. APPROACH The UVA EPA Program includes workplace assessments by residents/fellows, attending faculty, and master assessors (MAs), experienced clinicians who assess students across disciplines and clinical settings. All assessors participate in formal professional development and provide verbal and written comments to support their supervision ratings. The Entrustment Committee, composed of 12 MAs, uses a shared mental model and aggregates all assessor data to make a high-stakes summative entrustment decision about students' readiness to assume the role of an acting intern. OUTCOMES Since 2017, over 2,000 assessors have completed 56,969 EPA assessments for 1,479 students. Ninety-four percent of assessments have been done during the clerkship phase. Residents/fellows have completed a mean of 18 assessments, attending faculty a mean of 27, and MAs a mean of 882. Seventy-four percent of observed encounters involved patients with acute concerns with or without a co-morbid condition. Fifty percent of assessments occurred in inpatient and 32% in ambulatory settings. Eighty-seven percent of assessments contained narrative comments with more than 100 characters. NEXT STEPS Planned next steps will include earlier identification of students who require individualized learning to promote the development of skills related to EPAs, expansion of the remediation program to enable more students to engage in a clinical performance mastery elective, and creation of targeted professional development for assessors to reinforce the tenets of the EPA program.
Collapse
Affiliation(s)
- Meg G Keeley
- M.G. Keeley is senior associate dean for education and professor, Office of Medical Education and Department of Pediatrics, University of Virginia School of Medicine, Charlottesville, Virginia
| | - Megan J Bray
- M.J. Bray is associate dean for curriculum and associate professor, Office of Medical Education and Department of Obstetrics and Gynecology, University of Virginia School of Medicine, Charlottesville, Virginia
| | - Elizabeth B Bradley
- E.B. Bradley is director of evaluation and associate professor, Office of Medical Education and Center for Medical Education Research and Scholarly Innovation, University of Virginia School of Medicine, Charlottesville, Virginia
| | - Christine M Peterson
- C.M. Peterson is assistant dean for student affairs and associate professor, Office of Medical Education, University of Virginia School of Medicine, Charlottesville, Virginia
| | - Linda A Waggoner-Fountain
- L.A. Waggoner-Fountain is professor and associate program director, Department of Pediatrics, University of Virginia School of Medicine, Charlottesville, Virginia
| | - Maryellen E Gusic
- M.E. Gusic is currently senior associate dean for education and professor, Lewis Katz School of Medicine, Temple University, Philadelphia, Pennsylvania. She was formerly senior advisor, educational affairs, Office of Medical Education, University of Virginia School of Medicine, Charlottesville, Virginia
| |
Collapse
|
10
|
Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments. J Gen Intern Med 2022; 37:2200-2207. [PMID: 35710663 PMCID: PMC9296736 DOI: 10.1007/s11606-022-07509-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/08/2021] [Accepted: 03/24/2022] [Indexed: 10/18/2022]
Abstract
BACKGROUND Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding. OBJECTIVE In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks. DESIGN Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel. PARTICIPANTS A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor. MAIN MEASURES Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors. KEY RESULTS IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty. CONCLUSIONS To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.
Collapse
|
11
|
Soukoulis V, Martindale J, Bray MJ, Bradley E, Gusic ME. The use of EPA assessments in decision-making: Do supervision ratings correlate with other measures of clinical performance? MEDICAL TEACHER 2021; 43:1323-1329. [PMID: 34242113 DOI: 10.1080/0142159x.2021.1947480] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
BACKGROUND Entrustable professional activities (EPAs) have been introduced as a framework for teaching and assessment in competency-based educational programs. With growing use, has come a call to examine the validity of EPA assessments. We sought to explore the correlation of EPA assessments with other clinical performance measures to support use of supervision ratings in decisions about medical students' curricular progression. METHODS Spearman rank coefficients were used to determine correlation of supervision ratings from EPA assessments with scores on clerkship evaluations and performance on an end-of-clerkship-year Objective Structured Clinical Examination (CPX). RESULTS Both overall clinical evaluation items score (rho 0.40; n = 166) and CPX patient encounter domain score (rho 0.31; n = 149) showed significant correlation with students' overall mean EPA supervision rating during the clerkship year. There was significant correlation between mean supervision rating for EPA assessments of history, exam, note, and oral presentation skills with scores for these skills on clerkship evaluations; less so on the CPX. CONCLUSIONS Correlation of EPA supervision ratings with commonly used clinical performance measures offers support for their use in undergraduate medical education. Data supporting the validity of EPA assessments promotes stakeholders' acceptance of their use in summative decisions about students' readiness for increased patient care responsibility.
Collapse
Affiliation(s)
- Victor Soukoulis
- Division of Cardiovascular Medicine, University of Virginia School of Medicine, Charlottesville, VA, USA
| | - James Martindale
- Center for Medical Education Research and Scholarly Innovation, University of Virginia School of Medicine, Charlottesville, VA, USA
| | - Megan J Bray
- Center for Medical Education Research and Scholarly Innovation and Department of Obstetrics and Gynecology, University of Virginia School of Medicine, Charlottesville, VA, USA
| | - Elizabeth Bradley
- Center for Medical Education Research and Scholarly Innovation, University of Virginia School of Medicine, Charlottesville, VA, USA
| | - Maryellen E Gusic
- Center for Medical Education Research and Scholarly Innovation and Department of Pediatrics, University of Virginia School of Medicine, Charlottesville, VA, USA
| |
Collapse
|
12
|
Bradley EB, Waselewski EA, Gusic ME. How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment. MEDICAL SCIENCE EDUCATOR 2021; 31:1419-1428. [PMID: 34457983 PMCID: PMC8368261 DOI: 10.1007/s40670-021-01327-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 05/20/2021] [Indexed: 06/13/2023]
Abstract
INTRODUCTION The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated. METHODS In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding. RESULTS Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout. CONCLUSIONS Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.
Collapse
Affiliation(s)
- Elizabeth B. Bradley
- Center for Medical Education Research and Scholarly Innovation, Office of Medical Education, University of Virginia School of Medicine, VA Charlottesville, USA
| | - Eric A. Waselewski
- Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan USA
| | - Maryellen E. Gusic
- Center for Medical Education Research and Scholarly Innovation, Office of Medical Education, University of Virginia School of Medicine, VA Charlottesville, USA
| |
Collapse
|