1
|
Gielissen KA, Foust A, Weinstein AR, Duca N, Jenkins MO, Kisielewski M, Misra S, Dunne D. How Internal Medicine Clerkship Directors Are Using Entrustable Professional Activities: A National Survey Study. J Gen Intern Med 2025; 40:46-53. [PMID: 39147939 PMCID: PMC11780069 DOI: 10.1007/s11606-024-08991-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Academic Contribution Register] [Received: 02/06/2024] [Accepted: 08/02/2024] [Indexed: 08/17/2024]
Abstract
BACKGROUND In the present assessment environment in undergraduate medical education at U.S. medical schools, the prevalence and implementation of Entrustable Professional Activities (EPAs) in internal medicine (IM) clerkships are not well understood. OBJECTIVE To describe the prevalence and approach to EPA use in U.S. IM clerkships. DESIGN Cross-sectional, nationally representative survey of core IM clerkship directors. PARTICIPANTS One-hundred forty IM clerkship directors at Liaison Committee on Medical Education-accredited U.S./U.S. territory-based allopathic medical schools with membership in the Clerkship Directors in Internal Medicine (CDIM) as of December 2022. MAIN MEASURES Use of EPAs in IM core clerkships, including use for grading, types of EPAs, use of supportive measures for assessment, and current validity frameworks. KEY RESULTS The survey response was 80% (112/140); two additional respondents completed the section on EPA use (n = 114). Approximately half of respondents (47%) reported their IM clerkship used EPAs. Among schools accredited after 1977, a higher percentage was associated with having incorporated EPAs (p = 0.03). The Association of American Medical Colleges Core EPAs for Entering Residency (CEPAER) was the most common framework used by Clerkship Directors (CDs) for developing EPAs (55%). Most CDs (56%) used EPAs for both formative and summative assessments, and approximately half of CDs (48%) used EPAs for a portion of the final grade determination. CDs who used EPAs were no more likely to report efforts to ensure the validity of assessment, the use of faculty development, or that written assessments were a valid measure of students' performance compared to those who did not use EPAs. CONCLUSIONS Although EPAs have experienced substantial uptake in the IM clerkship and contribute to formative and summative assessment of learners, their use does not appear to be associated with enhanced efforts to obtain validity information.
Collapse
Affiliation(s)
| | - Anna Foust
- University of South Alabama College of Medicine, Mobile, AL, USA
| | - Amy R Weinstein
- Harvard Medical School and Beth Israel Deaconess Medical Center, Boston, MA, USA
| | | | | | | | - Savita Misra
- Alliance for Academic Internal Medicine, Alexandria, VA, USA
| | - Dana Dunne
- Yale University School of Medicine, New Haven, CT, USA
| |
Collapse
|
2
|
Shrivastava SR, Gade S, Vagha S. Roadmap for the Development and Implementation of Entrustable Professional Activities to Enable Effective Clinical Training. Contemp Clin Dent 2024; 15:289-291. [PMID: 39845628 PMCID: PMC11749045 DOI: 10.4103/ccd.ccd_148_24] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Academic Contribution Register] [Received: 04/05/2024] [Revised: 07/22/2024] [Accepted: 08/05/2024] [Indexed: 01/24/2025] Open
Abstract
In the field of clinical training, entrustable professional activities (EPAs) generally deal with those activities that health-care professionals will be encountering in their routine practice, and thus exposure to these activities prepares them for their future clinical practice. Workplace-based assessment tools have been used to assess the progress of medical students across these defined EPAs. The act of development of EPAs essentially requires the adoption of a systematic process to identify and define the core tasks and responsibilities that medical students must be able to perform independently in a specific specialty. The process of integration of EPAs into the medical school curriculum essentially requires systematic planning and implementation to ensure that students get an adequate number of opportunities to develop and demonstrate the desired competencies. In conclusion, EPAs provide an excellent framework for transforming medical education by preparing medical students to be ready for the complexities of health-care delivery.
Collapse
Affiliation(s)
- Saurabh RamBihariLal Shrivastava
- Deputy Director (Research and Development), Off Campus, Datta Meghe Institute of Higher Education and Research, Wanadongri, Nagpur, Maharashtra, India
- Department of Community Medicine, Datta Meghe Medical College, Datta Meghe Institute of Higher Education and Research, Wanadongri, Nagpur, Maharashtra, India
| | - Shubhada Gade
- Department of Physiology, Datta Meghe Medical College, Datta Meghe Institute of Higher Education and Research, Wanadongri, Nagpur, Maharashtra, India
| | - Sunita Vagha
- Department of Pathology, Jawaharlal Nehru Medical College, Datta Meghe Institute of Higher Education and Research, Sawangi (M), Wardha, Maharashtra, India
| |
Collapse
|
3
|
Costich M, Friedman S, Robinson V, Catallozzi M. Implementation and faculty perception of outpatient medical student workplace-based assessments. CLINICAL TEACHER 2024; 21:e13751. [PMID: 38433555 DOI: 10.1111/tct.13751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Academic Contribution Register] [Received: 07/10/2023] [Accepted: 02/06/2024] [Indexed: 03/05/2024]
Abstract
BACKGROUND There is growing interest in use of entrustable professional activity (EPA)-grounded workplace-based assessments (WBAs) to assess medical students through direct observation in the clinical setting. However, there has been very little reflection on how these tools are received by the faculty using them to deliver feedback. Faculty acceptance of WBAs is fundamentally important to sustained utilisation in the clinical setting, and understanding faculty perceptions of the WBA as an adjunct for giving targeted feedback is necessary to guide future faculty development in this area. APPROACH Use of a formative EPA-grounded WBA was implemented in the ambulatory setting during the paediatrics clerkship following performance-driven training and frame-of-reference training with faculty. Surveys and semi-structured interviews with faculty members explored how faculty perceived the tool and its impact on feedback delivery. EVALUATION Faculty reported providing more specific, task-oriented feedback following implementation of the WBA, as well as greater timeliness of feedback and greater satisfaction with opportunities to provide feedback, although these later two findings did not reach significance. Themes from the interviews reflected the benefits of WBAs, persistent barriers to the provision of feedback and suggestions for improvement of the WBA. IMPLICATIONS EPA-grounded WBAs are feasible to implement in the outpatient primary care setting and improve feedback delivery around core EPAs. The WBAs positively impacted the way faculty conceptualise feedback and provide learners with more actionable, behaviour-based feedback. Findings will inform modifications to the WBA and future faculty development and training to allow for sustainable WBA utilisation in the core clerkship.
Collapse
Affiliation(s)
- Marguerite Costich
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
| | - Suzanne Friedman
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
| | - Victoria Robinson
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
| | - Marina Catallozzi
- Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Pediatrics, Columbia University Vagelos College of Physicians and Surgeons and NewYork-Presbyterian, New York, New York, USA
- Department of Population and Family Health, Mailman School of Public Health, Columbia University Irving Medical Center, New York, New York, USA
| |
Collapse
|
4
|
Ryan MS, Gielissen KA, Shin D, Perera RA, Gusic M, Ferenchick G, Ownby A, Cutrer WB, Obeso V, Santen SA. How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study. MEDICAL EDUCATION 2024; 58:825-837. [PMID: 38167833 DOI: 10.1111/medu.15291] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Academic Contribution Register] [Received: 08/25/2023] [Revised: 10/27/2023] [Accepted: 11/16/2023] [Indexed: 01/05/2024]
Abstract
BACKGROUND Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.
Collapse
Affiliation(s)
- Michael S Ryan
- Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA
- School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Katherine A Gielissen
- Departments of Medicine and Pediatrics, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Dongho Shin
- Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
| | - Robert A Perera
- Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
| | - Maryellen Gusic
- Departments of Pediatrics, Biomedical Education and Data Science, Lewis Katz School of Medicine, Philadelphia, Pennsylvania, USA
| | - Gary Ferenchick
- Department of Medicine, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA
| | - Allison Ownby
- McGovern Medical School at UTHealth Houston, Houston, Texas, USA
| | - William B Cutrer
- Department of Pediatrics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA
| | - Vivian Obeso
- Department of Medical Education, University of Miami Miller School of Medicine, Miami, Florida, USA
| | - Sally A Santen
- Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
- Emergency Medicine and Medical Education at University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
5
|
Abraham C. Transition From Normative to Criterion-Based Grading in the Obstetrics and Gynecology Clerkship. JOURNAL OF MEDICAL EDUCATION AND CURRICULAR DEVELOPMENT 2024; 11:23821205241239201. [PMID: 38500498 PMCID: PMC10946066 DOI: 10.1177/23821205241239201] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Academic Contribution Register] [Received: 10/25/2023] [Accepted: 02/27/2024] [Indexed: 03/20/2024]
Abstract
OBJECTIVES To compare grades, National Board of Medical Examiners (NBME) Shelf Exam scores, and student satisfaction with the Obstetrics and Gynecology (OB/GYN) clerkship after transitioning from normative to criterion-based grading. METHODS Between July 2021 and July 2022, the Icahn School of Medicine at Mount Sinai (ISMMS) adhered to a normative grading scheme in which ∼60% of students achieved a grade of Honors, 30% achieved a grade of High Pass and 10% achieved a grade of Pass for the OB/GYN clerkship. In July 2022, ISMMS transitioned to a criterion-based scheme. In this scheme, 6 competencies were created. Criteria were determined for each competency, delineating achieving a score of "Pass" versus "Honors" for the specific objective. Students needed to meet the criteria for Honors for 4 out of 6 of the competencies in order to ultimately receive a grade of Honors for the clerkship. The number of students achieving Honors, NBME shelf exam scores, and student clerkship satisfaction ratings between the normative and criterion-based schemes were compared. RESULTS The number of students studying in academic year (AY) 2021-2022 and AY 2022-2023 were 134 and 137, respectively. A significantly lower percentage of students received Honors in AY 2021-2022 than in AY 2022-2023 (66% vs. 96%, P < .01). Mean exam scores were significantly higher for those receiving Honors in AY 2021-2022 than in AY 2022-2023 (P < .05); scores for AY 2021-2022 and AY 2022-2023 were 78.9, 95% CI [77.6, 80.1] and 76.7, 95% CI [75.6, 77.8], respectively. Mean exam scores for all students were not significantly different between the 2 academic cohorts (77.8 vs. 76.2, P = .06). Clerkship satisfaction rating was significantly higher in AY 2022-2023 than in AY 2021-2022 (4.1 vs. 3.7, P < .05). CONCLUSIONS These findings support a paradigm that compares learner performance to predefined measures as opposed to peer performance.
Collapse
Affiliation(s)
- Cynthia Abraham
- Department of Obstetrics, Gynecology and Reproductive Science, Department of Medical Education at the Icahn School of Medicine at Mount Sinai, Mount Sinai Health System, New York, NY, USA
| |
Collapse
|
6
|
Abraham C. Call for Action: Undergraduate Medical Education Assessment-Advocating for a Transition From Tiered to Competency-Based Grading. JOURNAL OF MEDICAL EDUCATION AND CURRICULAR DEVELOPMENT 2023; 10:23821205231207696. [PMID: 37854278 PMCID: PMC10580707 DOI: 10.1177/23821205231207696] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Subscribe] [Academic Contribution Register] [Received: 09/07/2022] [Accepted: 09/27/2023] [Indexed: 10/20/2023]
Abstract
Traditionally, undergraduate medical education (UME) grading has been based on a tiered system. Tier-based grading can cause anxiety as medical students are compared to their peers. Students then become overly driven by the pursuit of creating favorable impressions by supervisors as well as by high grades. Additionally, the emphasis on normative parameters appears to misalign with the goal of UME which is to not sort learners into different residency programs but to train future doctors to meet the needs of society. This commentary is a call for action to shift from utilizing a normative-based grading paradigm in UME to implementing one in which learners are being assessed on their ability to attain specific competencies. It is important that UME transitions to competency-based assessments as the graduate medical education (GME) realm has already adopted this framework.
Collapse
Affiliation(s)
- Cynthia Abraham
- Icahn School of Medicine, Mount Sinai Health System, New York, NY, USA
| |
Collapse
|
7
|
Miller DT, Gibb W, Caretta‐Weyer H, Ng K, Sebok‐Syer SS, Gisondi MA. Filling the Core EPA 10 assessment void: A framework for individual assessment of Core Entrustable professional activity 10 competencies in medical students. AEM EDUCATION AND TRAINING 2022; 6:e10787. [PMID: 36389650 PMCID: PMC9646936 DOI: 10.1002/aet2.10787] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Academic Contribution Register] [Received: 03/17/2022] [Revised: 06/28/2022] [Accepted: 06/29/2022] [Indexed: 06/16/2023]
Abstract
Objectives The goal of this study was to develop and evaluate a novel curriculum and assessment tool for Core Entrustable Professional Activity (EPA) 10 competencies and entrustment scoring in a cohort of medical students in their emergency medicine (EM) clerkship using a framework of individualized, ad hoc, formative assessment. Core EPA 10 is an observable workplace-based activity for graduating medical students to recognize a patient requiring urgent or emergent care and initiate evaluation and management. Methods This is a prospective, pretest-posttest study of medical students during their EM clerkship. Using the Thomas and Kern framework, we created a curriculum of simulation cases about chest pain/cardiac arrest and respiratory distress, which included novel assessment checklists, and instructional videos about recognizing and managing emergencies. Students were individually pretested on EPA 10 competencies using the simulation cases. Two raters scored students using standardized checklists. Students then watched instructional videos, underwent a posttest with the simulation cases, and were scored again by the two raters using the checklists. Differences between pretest and posttest scores were analyzed using paired t-tests and Wilcoxon signed-rank tests. Results Seventy-three out of 85 (86%) students completed the curriculum. Mean scores from pretest to final posttest in the chest pain/cardiac arrest and respiratory distress cases significantly improved from 14.8/19 (SD 1.91), to 17.1/19 (SD = 1.00), t(68) = 10.56, p < 0.001, and 8.5/13 (SD 1.79), to 11.1/13(SD 0.89), t(67) = 11.15, p < 0.001, respectively. The kappa coefficients were 0.909 (n = 2698, p < 0.001) and 0.931 (n = 1872, p < 0.001). Median modified Chen entrustment scores improved from 1b (i.e., "Watch me do this") to 2b (i.e., "I'll watch you") for the chest pain/cardiac arrest case (p < 0.001) and 1b/2a (i.e., "Watch me do this"/ "Let's do this together") to 3a (i.e. "You go ahead, and I'll double-check all of your findings") for the respiratory distress case (p < 0.001). Conclusion A new directed curriculum of standardized simulation cases and asynchronous instructional videos improved medical student performance in EPA 10 competencies and entrustment scores. This study provides a curricular framework to support formative individualized assessments for EPA 10.
Collapse
Affiliation(s)
- Danielle T. Miller
- Department of Emergency MedicineUniversity of Colorado School of MedicineDenverColoradoUSA
| | - William Gibb
- Department of Emergency MedicineUniversity of Colorado School of MedicineDenverColoradoUSA
| | - Holly Caretta‐Weyer
- Department of Emergency MedicineStanford University School of MedicinePalo AltoCaliforniaUSA
| | - Kristen Ng
- Department of Emergency MedicineStanford University School of MedicinePalo AltoCaliforniaUSA
| | - Stefanie S. Sebok‐Syer
- Department of Emergency MedicineStanford University School of MedicinePalo AltoCaliforniaUSA
| | - Michael A. Gisondi
- Department of Emergency MedicineStanford University School of MedicinePalo AltoCaliforniaUSA
| |
Collapse
|
8
|
Ryan MS, Khamishon R, Richards A, Perera R, Garber A, Santen SA. A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:552-561. [PMID: 34074896 DOI: 10.1097/acm.0000000000004189] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Academic Contribution Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. METHOD For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies. RESULTS Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. CONCLUSIONS Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is associate professor and assistant dean for clinical medical education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Rebecca Khamishon
- R. Khamishon is a fourth-year medical student, Virginia Commonwealth University, Richmond, Virginia
| | - Alicia Richards
- A. Richards is a graduate student, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Robert Perera
- R. Perera is associate professor, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Adam Garber
- A. Garber is associate professor, Department of Internal Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-7296-2896
| | - Sally A Santen
- S.A. Santen is professor and senior associate dean of assessment, evaluation, and scholarship, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8327-8002
| |
Collapse
|
9
|
Shrivastava S, Shrivastava P. Employing clinical work sampling tool for monitoring the clinical competence among medical students. MEDICAL JOURNAL OF DR. D.Y. PATIL VIDYAPEETH 2022. [DOI: 10.4103/mjdrdypu.mjdrdypu_583_20] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Academic Contribution Register] [Indexed: 11/04/2022] Open
|