1
|
Ryan MS, Gielissen KA, Shin D, Perera RA, Gusic M, Ferenchick G, Ownby A, Cutrer WB, Obeso V, Santen SA. How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study. MEDICAL EDUCATION 2024; 58:825-837. [PMID: 38167833 DOI: 10.1111/medu.15291] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 10/27/2023] [Accepted: 11/16/2023] [Indexed: 01/05/2024]
Abstract
BACKGROUND Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.
Collapse
Affiliation(s)
- Michael S Ryan
- Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA
- School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Katherine A Gielissen
- Departments of Medicine and Pediatrics, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Dongho Shin
- Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
| | - Robert A Perera
- Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
| | - Maryellen Gusic
- Departments of Pediatrics, Biomedical Education and Data Science, Lewis Katz School of Medicine, Philadelphia, Pennsylvania, USA
| | - Gary Ferenchick
- Department of Medicine, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA
| | - Allison Ownby
- McGovern Medical School at UTHealth Houston, Houston, Texas, USA
| | - William B Cutrer
- Department of Pediatrics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA
| | - Vivian Obeso
- Department of Medical Education, University of Miami Miller School of Medicine, Miami, Florida, USA
| | - Sally A Santen
- Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
- Emergency Medicine and Medical Education at University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
2
|
Schumacher DJ, Michelson C, Winn AS, Turner DA, Martini A, Kinnear B. A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees. MEDICAL EDUCATION 2024; 58:812-824. [PMID: 38088227 DOI: 10.1111/medu.15296] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/11/2023] [Revised: 11/03/2023] [Accepted: 11/21/2023] [Indexed: 06/04/2024]
Abstract
INTRODUCTION The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why? METHODS Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs. RESULTS PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes. DISCUSSION PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.
Collapse
Affiliation(s)
- Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Catherine Michelson
- Ann and Robert H. Lurie Children's Hospital of Chicago/Northwestern University, Chicago, Illinois, USA
| | - Ariel S Winn
- Boston Children's Hospital/Harvard Medical School, Boston, Massachusetts, USA
| | - David A Turner
- American Board of Pediatrics, Chapel Hill, North Carolina, USA
| | - Abigail Martini
- Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
| | - Benjamin Kinnear
- Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
3
|
Guldner G, Wells J, Ayutyanont N, Iyengar R, Sprenger S, Siegel JT, Kashyap R. COVID-19 related disruptions to medical education and perceived clinical capability of new resident physicians: a nationwide study of over 1200 first-year residents. MEDICAL EDUCATION ONLINE 2023; 28:2143307. [PMID: 36369921 PMCID: PMC9665094 DOI: 10.1080/10872981.2022.2143307] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 07/06/2022] [Revised: 10/27/2022] [Accepted: 10/31/2022] [Indexed: 06/16/2023]
Abstract
The COVID-19 pandemic transformed the final year of undergraduate medical education for thousands of medical students across the globe. Out of concern for spreading SARS-CoV-2 and conserving personal protective equipment, many students experienced declines in bedside clinical exposures. The perceived competency of this class within the context of the pandemic is unclear. We designed and distributed a survey to measure the degree to which recent medical school graduates from the USA felt clinically prepared on 13 core clinical skills. Of the 1283 graduates who matched at HCA Healthcare facilities, 90% (1156) completed the survey. In this national survey, most participants felt they were competent in their clinical skills. However, approximately one out of four soon-to-be residents felt they were clinically below where they should be with regard to calling consultations, performing procedures, and performing pelvic and rectal exams. One in five felt they were below where they should be with regard to safely transitioning care. These perceived deficits in important skill sets suggest the need for evaluation and revised educational approaches in these areas, especially when traditional in-person practical skills teaching and practice are disrupted.
Collapse
Affiliation(s)
- Gregory Guldner
- Graduate Medical Education, HCA Healthcare, Brentwood, TN, USA
| | - Jessica Wells
- Graduate Medical Education, HCA Healthcare, Brentwood, TN, USA
| | | | - Rahul Iyengar
- Southern Hills Medical Center, TriStar Division, HCA Healthcare, Nashville, TN, USA
| | - Steven Sprenger
- Tristar Centennial Medical Center, HCA Healthcare, Nashville, TN, USA
| | - Jason T. Siegel
- Department of Behavioral and Organizational Science, Claremont Graduate University, Claremont, CA, USA
| | - Rahul Kashyap
- Tristar Centennial Medical Center, HCA Healthcare, Nashville, TN, USA
| |
Collapse
|
4
|
Parsons AS, Greenfield J, Bradley E, Waggoner-Fountain LA, Norwood V, Weis A, Kulkarni S, Bray MJ, Keeley M, Ryan MS. Summative Entrustment Decisions in UME: Outcomes of a Binding Entrustment Committee Decision. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:S186-S187. [PMID: 37983431 DOI: 10.1097/acm.0000000000005391] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/27/2023]
Affiliation(s)
- Andrew S Parsons
- Author affiliations: A.S. Parsons, J. Greenfield, E. Bradley, L.A. Waggoner-Fountain, V. Norwood, A. Weis, S. Kulkarni, M.J. Bray, M. Keeley, M.S. Ryan, University of Virginia School of Medicine
| | | | | | | | | | | | | | | | | | | |
Collapse
|
5
|
Ten Cate O, Jarrett JB. Would I Trust or Will I Trust? The Gap between Entrustment Determinations and Entrustment Decisions for Trainees in Pharmacy and Other Health Professions. PHARMACY 2023; 11:107. [PMID: 37368433 DOI: 10.3390/pharmacy11030107] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2023] [Revised: 06/16/2023] [Accepted: 06/16/2023] [Indexed: 06/28/2023] Open
Abstract
Entrustable Professional Activities (EPAs) and entrustment decision making are rapidly becoming mainstream in competency-based education in the health professions. EPAs are the units of professional practice to entrust graduates with once they have developed the required competencies. They were conceived to enable a gradual increase in professional autonomy during training, by allowing trainees to practice activities in which they have demonstrated they have mastered well, with decreasing supervision. However, practicing health care unsupervised generally requires licensure. The question for pharmacy education, as well as for undergraduate medical education, is can students be given any autonomy in practice, even when they have fully mastered an EPA yet remain unlicensed? While entrustment decisions for licensed practitioners have autonomy consequences, some educators in undergraduate programs speak of 'entrustment determinations', to avoid decisions about students that affect patient care, in other words saying, we would trust you, rather than we will trust you. However, graduating learners without the experience of responsibility and reasonable autonomy creates a gap with full practice responsibilities, which may jeopardize patient safety after training. What can programs do to retain the power of using EPAs while at the same time guarding patient safety?
Collapse
Affiliation(s)
- Olle Ten Cate
- Center for Research and Development of Health Professions Education, University Medical Center Utrecht, 3584 CX Utrecht, The Netherlands
| | - Jennie B Jarrett
- Department of Pharmacy Practice, University of Illinois Chicago College of Pharmacy, Chicago, IL 60612, USA
| |
Collapse
|
6
|
Hanna K, Gupta S, Hurst R, McKeon BA, DeWaay D. Specialty-Specific Entrustable Professional Activities: A Bridge to Internship. Cureus 2023; 15:e35547. [PMID: 37007399 PMCID: PMC10057663 DOI: 10.7759/cureus.35547] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2023] [Accepted: 02/27/2023] [Indexed: 03/03/2023] Open
Abstract
Background Undergraduate medical education aims to prepare learners to become capable residents. New interns are expected to perform clinical tasks with distant supervision reliant on having acquired a medical degree. However, there is limited data to discuss what entrustment residency programs grant versus what the medical schools believe they have trained their graduates to perform. At our institution, we sought to foster an alliance between undergraduate medical education (UME) and graduate medical education (GME) toward specialty-specific entrustable professional activities (SSEPAs). These SSEPAs create a bridge to residency and help students structure the final year of medical school while striving for entrustability for day one of residency. This paper describes the SSEPA curriculum development process and student self-assessment of competence. Methodology We piloted an SSEPA program with the departments of Family Medicine, Internal Medicine, Neurology, and Obstetrics & Gynecology. Utilizing Kern's curriculum development framework, each specialty designed a longitudinal curriculum with a post-match capstone course. Students participated in pre-course and post-course self-assessments utilizing the Chen scale for each entrustable professional activity (EPA). Results A total of 42 students successfully completed the SSEPA curriculum in these four specialties. Students' self-assessed competence levels rose from 2.61 to 3.65 in Internal Medicine; 3.23 to 4.12 in Obstetrics and Gynecology; 3.62 to 4.13 in Neurology; and 3.65 to 3.79 in Family Medicine. Students across all specialties noted an increase in confidence from 3.45 to 4.38 in Internal Medicine; 3.3 to 4.6 in Obstetrics and Gynecology; 3.25 to 4.25 in Neurology; and 4.33 to 4.67 in Family Medicine. Conclusions A specialty-specific curriculum utilizing a competency-based framework for learners traversing the UME to GME journey in the final year of medical school improves learner confidence in their clinical abilities and may lead to an improved educational handoff between UME and GME.
Collapse
|