1
|
Stumbar SE, Eliason S, Toonkel R, Amie K, Bonnin R, Gralnik L, Kantor J, Minor S. Next steps for workplace-based assessments of entrustable professional activities. CLINICAL TEACHER 2024; 21:e13739. [PMID: 38311985 DOI: 10.1111/tct.13739] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2023] [Accepted: 01/05/2024] [Indexed: 02/06/2024]
Abstract
BACKGROUND Florida International University Herbert Wertheim College of Medicine (FIU-HWCOM) participated in the AAMC Core Entrustable Professional Activities (EPA) implementation pilot. Entrustment decision processes based on data from workplace-based assessments (WBAs) were piloted. Outcomes illustrated challenges including variability across EPAs with regards to learner level alignment and feasibility of data collection in the form of WBAs. In addition, students reported discomfort requesting WBA completion by preceptors and dissatisfaction with associated feedback. APPROACH To guide future directions, we conducted a survey of third-year students to better understand their experience with and perceptions of WBAs used to evaluate EPAs at FIU-HWCOM. EVALUATION Survey response was 96% (n = 107/112). Most (84%) reported that WBAs were not valuable to their development and that preceptors often did not complete WBAs in a timely fashion. Many (47%) reported not receiving verbal feedback. Most students (78%) used language in written responses demonstrating confusion between the EPAs and the WBAs used to assess them. IMPLICATIONS The use of WBAs to assess EPAs did not have its intended impact at FIU-HWCOM. For future classes, WBA forms will consist of paper cards with questions directly assessing performance of skills aligned with EPAs 1, 5 and 6 only. To continue to promote feedback, students will be required to collect WBAs on all clerkships, but the number of required WBAs will be less than prior and no entrustment decisions will be made.
Collapse
Affiliation(s)
- Sarah E Stumbar
- Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA
| | - Sarah Eliason
- Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA
| | - Rebecca Toonkel
- Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA
| | - Katrina Amie
- Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA
| | - Rodolfo Bonnin
- Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA
| | - Leonard Gralnik
- Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA
| | - Julie Kantor
- Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA
| | - Suzanne Minor
- Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA
| |
Collapse
|
2
|
Ryan MS, Gielissen KA, Shin D, Perera RA, Gusic M, Ferenchick G, Ownby A, Cutrer WB, Obeso V, Santen SA. How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study. MEDICAL EDUCATION 2024; 58:825-837. [PMID: 38167833 DOI: 10.1111/medu.15291] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 10/27/2023] [Accepted: 11/16/2023] [Indexed: 01/05/2024]
Abstract
BACKGROUND Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.
Collapse
Affiliation(s)
- Michael S Ryan
- Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA
- School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Katherine A Gielissen
- Departments of Medicine and Pediatrics, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Dongho Shin
- Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
| | - Robert A Perera
- Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
| | - Maryellen Gusic
- Departments of Pediatrics, Biomedical Education and Data Science, Lewis Katz School of Medicine, Philadelphia, Pennsylvania, USA
| | - Gary Ferenchick
- Department of Medicine, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA
| | - Allison Ownby
- McGovern Medical School at UTHealth Houston, Houston, Texas, USA
| | - William B Cutrer
- Department of Pediatrics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA
| | - Vivian Obeso
- Department of Medical Education, University of Miami Miller School of Medicine, Miami, Florida, USA
| | - Sally A Santen
- Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
- Emergency Medicine and Medical Education at University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
3
|
Caretta-Weyer HA, Smirnova A, Barone MA, Frank JR, Hernandez-Boussard T, Levinson D, Lombarts KMJMH, Lomis KD, Martini A, Schumacher DJ, Turner DA, Schuh A. The Next Era of Assessment: Building a Trustworthy Assessment System. PERSPECTIVES ON MEDICAL EDUCATION 2024; 13:12-23. [PMID: 38274558 PMCID: PMC10809864 DOI: 10.5334/pme.1110] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/16/2023] [Accepted: 12/18/2023] [Indexed: 01/27/2024]
Abstract
Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.
Collapse
Affiliation(s)
- Holly A. Caretta-Weyer
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California, USA
| | - Alina Smirnova
- Department of Family Medicine, University of Calgary, Calgary, Alberta, Canada
- Kern Institute for the Transformation of Medical Education, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Michael A. Barone
- NBME, Philadelphia, Pennsylvania, USA
- Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| | - Jason R. Frank
- Department of Emergency Medicine, University of Ottawa, Ottawa, Ontario, CA
| | | | - Dana Levinson
- Josiah Macy Jr Foundation, Philadelphia, Pennsylvania, USA
| | - Kiki M. J. M. H. Lombarts
- Department of Medical Psychology, Amsterdam University Medical Centers, University of Amsterdam, NL
- Amsterdam Public Health research institute, Amsterdam, NL
| | - Kimberly D. Lomis
- Undergraduate Medical Education Innovations, American Medical Association, Chicago, Illinois, USA
| | - Abigail Martini
- Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, USA
| | - Daniel J. Schumacher
- Division of Emergency Medicine, Cincinnati Children’s Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - David A. Turner
- American Board of Pediatrics, Chapel Hill, North Carolina, USA
| | - Abigail Schuh
- Division of Emergency Medicine, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| |
Collapse
|
4
|
Encandela JA, Shaull L, Jayas A, Amiel JM, Brown DR, Obeso VT, Ryan MS, Andriole DA. Entrustable professional activities as a training and assessment framework in undergraduate medical education: A case study of a multi-institutional pilot. MEDICAL EDUCATION ONLINE 2023; 28:2175405. [PMID: 36794397 PMCID: PMC9937016 DOI: 10.1080/10872981.2023.2175405] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/12/2022] [Revised: 10/16/2022] [Accepted: 01/27/2023] [Indexed: 06/18/2023]
Abstract
In 2014, the Association of American Medical Colleges (AAMC) published 13 Core Entrustable Professional Activities (EPAs) that graduating students should be able to perform with indirect supervision when entering residency. A ten-school multi-year pilot was commissioned to test feasibility of implementing training and assessment of the AAMC's 13 Core EPAs. In 2020-21, a case study was employed to describe pilot schools' implementation experiences. Teams from nine of ten schools were interviewed to identify means and contexts of implementing EPAs and lessons learned. Audiotapes were transcribed then coded by investigators using conventional content analysis and a constant comparative method. Coded passages were organized in a database and analyzed for themes. Consensus among school teams regarding facilitators of EPA implementation included team commitment to piloting EPAs; agreement that: proximal EPA adoption with curriculum reform facilitates EPA implementation; EPAs 'naturally fit' in clerkships and provided opportunity for schools to reflect on and adjust curricula and assessments; and inter-school collaboration bolstered individual school progress. Schools did not make high-stakes decisions about student progress (e.g., promotion, graduation), yet EPA assessment results complemented other forms of assessment in providing students with robust formative feedback about their progress. Teams had varied perceptions of school capability to implement an EPA framework, influenced by various levels of dean involvement, willingness, and capability of schools to invest in data systems and provide other resources, strategic deployment of EPAs and assessments, and faculty buy-in. These factors affected varied pace of implementation. Teams agreed on the worthiness of piloting the Core EPAs, but substantial work is still needed to fully employ an EPA framework at the scale of entire classes of students with enough assessments per EPA and with required data validity/reliability. Recommendations stemming from findings may help inform further implementation efforts across other schools adopting or considering an EPA framework.
Collapse
Affiliation(s)
- John A. Encandela
- Associate Proessor of Psychiatry and Executive Director of the Teaching & Learning Center at the Yale School of Medicine, New Haven, Connecticut
| | - Lynn Shaull
- Senior Research Specialist at the Association of American Medical Colleges, Washington, DC, USA
| | - Amy Jayas
- Research Analyst at the Association of American Medical Colleges, Washington, DC, USA
| | - Jonathan M. Amiel
- Professor of Psychiatry and Senior Associate Dean at Columbia University Vagelos College of Physicians and Surgeons, New York, New York, USA
| | - David R. Brown
- Professor and Chief of Family and Community Medicine at Florida International University Herbert Wertheim College of Medicine, Miami, FL, USA
| | - Vivian T. Obeso
- Associate Professor of Medicine and Associate Dean for Curriculum and Medical Education at Florida International University Herbert Wertheim College of Medicine, Miami, FL, USA
| | - Michael S. Ryan
- , Professor of Pediatrics and Associate Dean for Assessment, Evaluation, Research, Scholarly Innovation in Medical Education at the University of Virginia School of Medicine, Charlottesville, VA, USA
| | - Dorothy A. Andriole
- Senior Director of Medical Research at the Association of American Medical Colleges, Washington, DC, USA
| |
Collapse
|
5
|
Gummesson C, Alm S, Cederborg A, Ekstedt M, Hellman J, Hjelmqvist H, Hultin M, Jood K, Leanderson C, Lindahl B, Möller R, Rosengren B, Själander A, Svensson PJ, Särnblad S, Tejera A. Entrustable professional activities (EPAs) for undergraduate medical education - development and exploration of social validity. BMC MEDICAL EDUCATION 2023; 23:635. [PMID: 37667366 PMCID: PMC10478490 DOI: 10.1186/s12909-023-04621-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Accepted: 08/28/2023] [Indexed: 09/06/2023]
Abstract
BACKGROUND The development of entrustable professional activities (EPAs) as a framework for work-based training and assessment in undergraduate medical education has become popular. EPAs are defined as units of a professional activity requiring adequate knowledge, skills, and attitudes, with a recognized output of professional labor, independently executable within a time frame, observable and measurable in its process and outcome, and reflecting one or more competencies. Before a new framework is implemented in a specific context, it is valuable to explore social validity, that is, the acceptability by relevant stakeholders. AIM The aim of our work was to define Core EPAs for undergraduate medical education and further explore the social validity of the constructs. METHOD AND MATERIAL In a nationwide collaboration, EPAs were developed using a modified Delphi procedure and validated according to EQual by a group consisting of teachers nominated from each of the seven Swedish medical schools, two student representatives, and an educational developer (n = 16). In the next step, social validity was explored in a nationwide survey. The survey introduced the suggested EPAs. For each EPA, the importance of the EPA was rated, as was the rater's perception of the present graduates' required level of supervision when performing the activity. Free-text comments were also included and analyzed. RESULTS Ten Core EPAs were defined and validated. The validation scores for EQual ranged from 4.1 to 4.9. The nationwide survey had 473 responders. All activities were rated as "important" by most responders, ranging from 54 to 96%. When asked how independent current graduates were in performing the ten activities, 6 to 35% reported "independent". The three themes of the free text comments were: 'relevant target areas and content'; 'definition of the activities'; and 'clinical practice and learning'. CONCLUSION Ten Core EPAs were defined and assessed as relevant for Swedish undergraduate medical education. There was a consistent gap between the perceived importance and the certainty that the students could perform these professional activities independently at the time of graduation. These results indicate that the ten EPAs may have a role in undergraduate education by creating clarity for all stakeholders.
Collapse
Affiliation(s)
- Christina Gummesson
- Faculty of Medicine, Faculty of Odontology, Lund University, Malmö University, Malmö, Sweden.
| | - Stina Alm
- Department of Clinical Sciences, Futurum - the Academy for Health and Care, Region Jönköping County, Umeå University, Paediatrics, Umeå, Sweden
| | - Anna Cederborg
- Institute of Medicine, Sahlgrenska Academy, Department of Medicine, University of Gothenburg, Sahlgrenska University Hospital, Gothenburg, Sweden
| | - Mattias Ekstedt
- Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, 581 83, Sweden
| | - Jarl Hellman
- Department of Medical Sciences, Uppsala University, Uppsala, Sweden
| | - Hans Hjelmqvist
- Department of Anesthesiology and Intensive Care, Faculty of Medicine and Health, Örebro University, Örebro, Sweden
| | - Magnus Hultin
- Department of Surgical and Perioperative Sciences, Anesthesiology and Intensive Care Medicine, Umeå University, Umeå, Sweden
| | - Katarina Jood
- Department of Clinical Neuroscience, Institute of Neuroscience and Physiology, Sahlgrenska Academy, Department of Neurology, University of Gothenburg, Sahlgrenska University Hospital, Gothenburg, Sweden
| | - Charlotte Leanderson
- Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden
| | - Bertil Lindahl
- Department of Medical Sciences, Uppsala University, Uppsala, Sweden
| | - Riitta Möller
- Department of Clinical Science, Intervention and Technology, Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Karolinska Institutet, Stockholm, Sweden
| | - Björn Rosengren
- Clinical and Molecular Osteoporosis Research Unit, Department of Orthopedics and Clinical Sciences, Skåne University Hospital Malmö, Lund University, Malmö, Sweden
| | - Anders Själander
- Department of Public Health and Clinical Medicine, Umeå University, Umeå, Sweden
| | - Peter J Svensson
- Department of Clinical Sciences Lund University, Skåne University Hospital, Malmö, Sweden
| | - Stefan Särnblad
- Department of Pediatrics, Faculty of Medicine and Health, Örebro University, Örebro, Sweden
| | - Alexander Tejera
- Division of Translational Cancer Research, Department of Laboratory Medicine, Skåne University Hospital, Lund University, Lund University, Lund, Sweden
| |
Collapse
|
6
|
Looi JC, Bonner D, Maguire PA, Brazel M, Keightley P, Reay RE, Tedeschi M. Commentary on alignment of medical student assessment and vocational training in psychiatry and addiction medicine. Australas Psychiatry 2023; 31:220-223. [PMID: 36710072 PMCID: PMC10088331 DOI: 10.1177/10398562231153010] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/31/2023]
Abstract
OBJECTIVE To comment upon the potential for alignment of medical student assessment and vocational specialist training through the RANZCP-CanMEDS model of Entrustable Professional Activities (EPAs) and Workplace-Based Assessments (WBAs). We discuss a specific post hoc example of such an alignment in an Australian graduate medical school in Psychiatry and Addiction Medicine. CONCLUSIONS Vocational training models of assessment, such as the RANZCP specialist training program for psychiatrists, can potentially be mapped to medical student education in formative and summative assessment through CanMEDs-based EPAs and WBAs, to assist in transition to specialist training.
Collapse
Affiliation(s)
- Jeffrey Cl Looi
- Academic Unit of Psychiatry and Addiction Medicine, 104822The Australian National University School of Medicine and Psychology, Canberra Hospital, Canberra, ACT, Australia
| | - Daniel Bonner
- Academic Unit of Psychiatry and Addiction Medicine, 104822The Australian National University School of Medicine and Psychology, Canberra Hospital, Canberra, ACT, Australia
| | - Paul A Maguire
- Academic Unit of Psychiatry and Addiction Medicine, 104822The Australian National University School of Medicine and Psychology, Canberra Hospital, Canberra, ACT, Australia
| | - Matthew Brazel
- Academic Unit of Psychiatry and Addiction Medicine, 104822The Australian National University School of Medicine and Psychology, Canberra Hospital, Canberra, ACT, Australia
| | - Philip Keightley
- Academic Unit of Psychiatry and Addiction Medicine, 104822The Australian National University School of Medicine and Psychology, Canberra Hospital, Canberra, ACT, Australia
| | - Rebecca E Reay
- Academic Unit of Psychiatry and Addiction Medicine, 104822The Australian National University School of Medicine and Psychology, Canberra Hospital, Canberra, ACT, Australia
| | - Michael Tedeschi
- Academic Unit of Psychiatry and Addiction Medicine, 104822The Australian National University School of Medicine and Psychology, Canberra Hospital, Canberra, ACT, Australia
| |
Collapse
|
7
|
Oral examination of fourth-year medical students on surgery rotations allows faculty to assess Core Entrustable Professional Activities for entering Residency (CEPAR): Proof-of-concept and analysis of student submissions. Am J Surg 2023; 225:841-846. [PMID: 36764899 DOI: 10.1016/j.amjsurg.2023.01.020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2022] [Revised: 01/10/2023] [Accepted: 01/20/2023] [Indexed: 01/29/2023]
Abstract
BACKGROUND As a community-based medical school which recruited faculty preceptors new to teaching, we sought to create objective assessments for fourth-year surgery experiences via administration of an oral exam. Students provided three authentic cases, which faculty used as a springboard to ascertain student proficiency in five entrustable professional activities: 1-oral presentation, 2-recognition of urgency/instability, 3-calling consults, 4-transitions of care, 5-informed consent. We present proof-of-concept and analysis of student case submissions. METHODS Twenty-seven student submissions (79 cases in total) were evaluated for case complexity, level-appropriateness, and an estimation of the ability to conduct a quality exam based on the information provided (subjective measures). Objective metrics included word count, instruction adherence, inclusion of figures/captions. A resident-in-training rated cases via the same metrics. In-examination data was separately culled. RESULTS The average word count was 281.70 (SD 140.23; range 40-743). Figures were included in 26.1% of cases. Faculty raters scored 29.0% as low-complexity, 37.7% medium-complexity, and 33.3% high-complexity. Raters felt 62.3% of cases provided enough information to conduct a quality exam. The majority of cases submitted (65.2%) were level-appropriate or higher. The resident rater scored cases more favorably than surgeons (Cohen's kappa of -0.5), suggesting low inter-rater agreement between those of differing experience levels. CONCLUSION Student's case submissions lessened faculty burden and provided assessors with adequate information to deliver a quality exam to assess proficiency in clinical skills essential for residency. Cases demonstrated sufficient complexity and level-appropriateness. The request to correlate case rating with exam performance is under review by our institution's assessment office. Near-peer tutoring by resident alumni is a program under development.
Collapse
|
8
|
Kuehl SE, Spicer JO. Using entrustable professional activities to better prepare students for their postgraduate medical training: A medical student's perspective. PERSPECTIVES ON MEDICAL EDUCATION 2022; 11:359-364. [PMID: 36441351 PMCID: PMC9743878 DOI: 10.1007/s40037-022-00731-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 10/12/2022] [Accepted: 10/14/2022] [Indexed: 06/16/2023]
Abstract
THE PROBLEM Medical students graduate underprepared for postgraduate medical training despite years of classroom and clinical training. In this article, a medical student shares her personal perspectives on three factors contributing to this problem in undergraduate medical education: students' peripheral roles in the clinical environment impede learning, students receive inadequate feedback, and assessments do not measure desired learning outcomes. A SOLUTION The authors describe how using entrustable professional activities (EPAs) could address these issues and promote students' clinical engagement by clarifying their roles, providing them with frequent and actionable feedback, and aligning their assessments with authentic work. These factors combined with grading schemes rewarding improvement could contribute to a growth mindset that reprioritizes clinical skill acquisition. The authors explore how medical schools have begun implementing the EPA framework, highlight insights from these efforts, and describe barriers that must be addressed. THE FUTURE Incorporating EPAs into medical school curricula could better prepare students for postgraduate training while also alleviating issues that contribute to student burnout by defining students' roles, improving feedback, and aligning assessments with desired learning outcomes.
Collapse
Affiliation(s)
- Sarah E Kuehl
- Emory University School of Medicine and Goizueta Business School, Atlanta, GA, USA.
| | - Jennifer O Spicer
- J. Willis Hurst Internal Medicine Residency Program, Division of Infectious Diseases, Department of Medicine, Emory University School of Medicine, Atlanta, GA, USA
| |
Collapse
|
9
|
Dunne D, Gielissen K, Slade M, Park YS, Green M. WBAs in UME-How Many Are Needed? A Reliability Analysis of 5 AAMC Core EPAs Implemented in the Internal Medicine Clerkship. J Gen Intern Med 2022; 37:2684-2690. [PMID: 34561828 PMCID: PMC9411433 DOI: 10.1007/s11606-021-07151-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/19/2021] [Accepted: 09/09/2021] [Indexed: 01/07/2023]
Abstract
BACKGROUND Reliable assessments of clinical skills are important for undergraduate medical education, trustworthy handoffs to graduate medical programs, and safe, effective patient care. Entrustable professional activities (EPAs) for entering residency have been developed; research is needed to assess reliability of such assessments in authentic clinical workspaces. DESIGN A student-driven mobile assessment platform was developed and used for clinical supervisors to record ad hoc entrustment decisions using the modified Ottawa scale on 5 core EPAs in an 8-week internal medicine (IM) clerkship. After a 12-month period, generalizability (G) theory analysis was performed to estimate the reliability of entrustment scores and determine the proportion of variance attributable to the student and the other facets, including particular EPA, evaluator type (attending versus resident), or case complexity. Decision (D) theory analysis determined the expected reliability based on the number of hypothetical observations. A g-coefficient of 0.7 was used as a generally agreed upon minimum reliability threshold. KEY RESULTS A total of 1368 ratings over the 5 EPAs were completed on 94 students. Variance attributed to person (true variance) was high for all EPAs; EPA-5 had the lowest person variance (9.8% across cases and four blocks). Across cases, reliability ranged from 0.02 to 0.60. Applying this to the Decision study, the estimated number of observations needed to reach a reliability index of 0.7 ranged between 9 and 11 for all EPAs except EPA5 which was sensitive to case complexity. CONCLUSIONS Work place-based clinical skills in IM clerkship students were assessed and logged using a convenient mobile platform. Our analysis suggests that 9-11 observations are needed for these EPA workplace-based assessments (WBAs) to achieve a reliability index of 0.7. Note writing was very sensitive to case complexity. Further reliability analyses of core EPAs are needed before US medical schools consider wider adoption into summative entrustment processes and GME handoffs.
Collapse
Affiliation(s)
- Dana Dunne
- Department of Internal Medicine, Section of Infectious Diseases, Yale School of Medicine, 15 York Street LMP 1074, New Haven, CT, 065111, USA.
| | - Katherine Gielissen
- Department of Internal Medicine, Section of General Medicine, Yale School of Medicine, New Haven, CT, 06511, USA
| | - Martin Slade
- Occupational Medicine, Yale School of Medicine, New Haven, CT, 06511, USA
| | | | - Michael Green
- Department of Internal Medicine, Section of General Medicine, Yale School of Medicine, New Haven, CT, 06511, USA
| |
Collapse
|
10
|
Forrest LL, Geraghty JR. Student-Led Initiatives and Advocacy in Academic Medicine: Empowering the Leaders of Tomorrow. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:781-785. [PMID: 35234719 DOI: 10.1097/acm.0000000000004644] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Trainees' value as effective leaders within academic medicine has been increasingly recognized. From their perspective as adult learners who enter medical education from diverse backgrounds, trainees offer significant value to the teaching, learning, and practice of medicine. As such, trainees have developed and led various successful initiatives throughout academic medicine. In this Invited Commentary, 2 medical students with national leadership roles provide their perspectives on how student-led initiatives and advocacy can help push academic medicine forward. The authors first provide an overview of the success of student-led initiatives throughout medical education as evidenced by the Trainee-Authored Letters to the Editor in this issue, highlighting the unique contributions and perspectives of trainees in the development and implementation of new initiatives or ways of thinking. Although trainees add value to many areas in academic medicine, here the authors highlight 4 current areas that align with Association of American Medical Colleges priorities: (1) public health emergencies including the COVID-19 pandemic; (2) equity, inclusion, and diversity; (3) wellness and resilience amongst trainees and health care providers; and (4) recent changes to the United States Medical Licensing Examination and the transition to residency. By sharing their experiences with student-led initiatives within each of these domains, the authors provide lessons learned and discuss successes and obstacles encountered along the way. Overall, there is a critical need for increased engagement of trainees in medical education. Empowering trainees now ensures the academic medicine leaders of tomorrow are prepared to face the challenges that await them.
Collapse
Affiliation(s)
- Lala L Forrest
- L.L. Forrest is a third-year medical student at the Frank H. Netter MD School of Medicine at Quinnipiac University, North Haven, Connecticut
| | - Joseph R Geraghty
- J.R. Geraghty is a third-year medical student in the Medical Scientist Training Program (MSTP) at the University of Illinois College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-6828-4893
| |
Collapse
|
11
|
Klapheke M, Abrams MP, Cubero M, Zhu X. Aligning Medical Student Workplace-Based Assessments with Entrustable Professional Activities and the RIME Model in a Psychiatry Clerkship. ACADEMIC PSYCHIATRY : THE JOURNAL OF THE AMERICAN ASSOCIATION OF DIRECTORS OF PSYCHIATRIC RESIDENCY TRAINING AND THE ASSOCIATION FOR ACADEMIC PSYCHIATRY 2022; 46:283-288. [PMID: 35288865 DOI: 10.1007/s40596-022-01614-3] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/13/2021] [Accepted: 02/18/2022] [Indexed: 06/14/2023]
Abstract
OBJECTIVE The authors piloted use of workplace-based assessments of students during the psychiatry clerkship utilizing both entrustable professional activities (EPAs) and the reporter, interpreter, manager, and educator (RIME) model. METHODS After supervising clinicians conducted assessments of medical students (N=109) during the psychiatry clerkship using a supervisory scale aligned with both EPA and RIME models, each student received individualized formative feedback. Students were then surveyed on the usefulness of this feedback, and participating faculty/residents were surveyed on the ease of completion of the supervisory scale. RESULTS Students' mean skill profile suggested they no longer needed direct supervision on EPA1 and EPA6. Mean scores on other studied EPAs suggested students were well on their way toward performing these EPAs without direct supervision. Students had mean RIME scores that exceeded the suggested levels identified for a Reporter to start clerkships, for an Interpreter to start clerkships, and for a Manager to transition to the fourth year. Close to half of the students found the feedback helpful in their development as a clinician but most felt their performance should not be shared with residency program directors, either before or after the Match. Almost all responding preceptors felt the supervisory ratings were easy to complete. CONCLUSIONS This pilot RIME/EPA framework served as a successful step toward a more competency-based medical education in the psychiatry clerkship with relatively little additional faculty time commitment by using workplace-based assessments already in place and a supervisory scale based on EPAs and RIME.
Collapse
Affiliation(s)
| | | | | | - Xiang Zhu
- University of Central Florida, Orlando, FL, USA
| |
Collapse
|
12
|
Ten Cate O, Schumacher DJ. Entrustable professional activities versus competencies and skills: Exploring why different concepts are often conflated. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2022; 27:491-499. [PMID: 35226240 PMCID: PMC9117349 DOI: 10.1007/s10459-022-10098-7] [Citation(s) in RCA: 33] [Impact Index Per Article: 16.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/14/2021] [Accepted: 01/23/2022] [Indexed: 05/25/2023]
Abstract
Despite explanations in the literature, a returning question in the use of entrustable professional activities (EPAs) is how to distinguish them from competencies and skills. In this article, we attempt to analyze the causes of the frequent confusion and conflation of EPAs with competencies and skills, and argue why the distinction is important for education, qualification and patient safety. 'Tracheotomy', 'lumbar puncture', 'interprofessional collaboration' for example are colloquially called 'skills', but its is a person's ability to perform these activities that is the actual skill; the EPA is simply the activity itself. We identify two possible causes for the confusion. One is a tendency to frame all educational objectives as EPAs. Many objectives of medical training can be conceptualized as EPAs, if 'the ability to do X' is the corresponding competency; but that does not work for all. We offer ways to deal with objectives of training that are not usefully conceptualized as EPAs. A more fundamental cause relates to entrustment decisions. The permission to contribute to health care reflects entrustment. Entrustment decisions are the links or pivots between a person's readiness for the task and the actual task execution. However, if entrustment decisions do not lead to increased autonomy in the practice of health care, but only serve to decide upon the advancement to a next stage of training, EPAs can become the tick boxes learners feel they need to collect to 'pass'. Gradually, then, EPAs can loose their original meaning of units of practice for which one becomes qualified.
Collapse
Affiliation(s)
- Olle Ten Cate
- Utrecht Center for Research and Development of Health Professions Education, University Medical Center Utrecht, P.O. Box # 85500, 3508 GA, Utrecht, The Netherlands.
| | - Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
13
|
Weissenbacher A, Bolz R, Stehr SN, Hempel G. Development and consensus of entrustable professional activities for final-year medical students in anaesthesiology. BMC Anesthesiol 2022; 22:128. [PMID: 35488205 PMCID: PMC9052481 DOI: 10.1186/s12871-022-01668-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2021] [Accepted: 04/20/2022] [Indexed: 11/18/2022] Open
Abstract
Background The transfer of classic concepts of competency-based medical education into clinical practice has been proven to be difficult in the past, being described as partially fragmented, misleading and inadequate. At the beginning of training, novice doctors commonly feel overwhelmed, overloaded and exposed to extreme time pressure. The discrepancy between expected and actual clinical competence of doctors at the start of their speciality training jeopardizes patient safety. The framework of Entrustable Professional Activities (EPAs) is a promising instrument to effectively integrate competency-based training into clinical practice and may help to close this gap and consequently to improve patient safety. Methods For anaesthesiology, we developed 5 EPAs for final-year medical students. The EPAs comprised the following seven categories: 1. Title, 2. Specifications, 3. Limitations, 4. Competency domains, 5. Knowledge, abilities and skills, professional attitudes, 6. Assessment and 7. Entrustment. Based on a modified, online-based Delphi study, we further developed and refined these EPAs. Education experts were recruited from the alumni network of the Master of Medical Education (MME) degree course from the University of Heidelberg, Germany. Results 28 data sets were evaluated in three Delphi rounds. 82% of study participants had previous experience with EPAs. Qualitative and quantitative data formed the basis during the iterative process and resulted in complete descriptions of 5 EPAs for final-year medical students in anaesthesiology. Conclusions Our study including the associated description of 5 EPAs represent a further step and starting point for EPA-based curricula in medical training in Germany linking undergraduate training, to residency training and continuous medical education. Supplementary Information The online version contains supplementary material available at 10.1186/s12871-022-01668-8.
Collapse
Affiliation(s)
- Andreas Weissenbacher
- Department of Anaesthesiology and Intensive Care, University of Leipzig Medical Centre, Liebigstrasse 20, 04103, Leipzig, Germany
| | - Robert Bolz
- Department of Anaesthesiology and Intensive Care, University of Leipzig Medical Centre, Liebigstrasse 20, 04103, Leipzig, Germany
| | - Sebastian N Stehr
- Department of Anaesthesiology and Intensive Care, University of Leipzig Medical Centre, Liebigstrasse 20, 04103, Leipzig, Germany
| | - Gunther Hempel
- Department of Anaesthesiology and Intensive Care, University of Leipzig Medical Centre, Liebigstrasse 20, 04103, Leipzig, Germany.
| |
Collapse
|
14
|
Ryan MS, Khamishon R, Richards A, Perera R, Garber A, Santen SA. A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:552-561. [PMID: 34074896 DOI: 10.1097/acm.0000000000004189] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. METHOD For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies. RESULTS Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. CONCLUSIONS Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is associate professor and assistant dean for clinical medical education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Rebecca Khamishon
- R. Khamishon is a fourth-year medical student, Virginia Commonwealth University, Richmond, Virginia
| | - Alicia Richards
- A. Richards is a graduate student, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Robert Perera
- R. Perera is associate professor, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Adam Garber
- A. Garber is associate professor, Department of Internal Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-7296-2896
| | - Sally A Santen
- S.A. Santen is professor and senior associate dean of assessment, evaluation, and scholarship, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8327-8002
| |
Collapse
|
15
|
Brown DR, Moeller JJ, Grbic D, Biskobing DM, Crowe R, Cutrer WB, Green ML, Obeso VT, Wagner DP, Warren JB, Yingling SL, Andriole DA. Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:536-543. [PMID: 34261864 DOI: 10.1097/acm.0000000000004242] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
PURPOSE In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions. METHOD For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables. RESULTS The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05). CONCLUSIONS TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.
Collapse
Affiliation(s)
- David R Brown
- D.R. Brown is professor, chief, Division of Family and Community Medicine, and interim chair, Department of Humanities, Health, and Society, Florida International University Herbert Wertheim College of Medicine, Miami, Florida; ORCID: http://orcid.org/0000-0002-5361-6664
| | - Jeremy J Moeller
- J.J. Moeller is associate professor and residency program director, Department of Neurology, Yale University School of Medicine, New Haven, Connecticut; ORCID: https://orcid.org/0000-0002-6135-5572
| | - Douglas Grbic
- D. Grbic is lead research analyst, Medical Education Research, Association of American Medical Colleges, Washington, DC
| | - Diane M Biskobing
- D.M. Biskobing is professor of medicine and associate dean of medical education, Virginia Commonwealth University School of Medicine, Richmond, Virginia
| | - Ruth Crowe
- R. Crowe is director of integrated clinical skills, director of practice of medicine, Office of Medical Education, and associate professor of medicine, New York University Grossman School of Medicine, New York, New York
| | - William B Cutrer
- W.B. Cutrer is associate dean for undergraduate medical education and associate professor of pediatrics (critical care medicine), Vanderbilt University School of Medicine, Nashville, Tennessee; ORCID: https://orcid.org/0000-0003-1538-9779
| | - Michael L Green
- M.L. Green is professor of medicine and director of student assessment, Teaching and Learning Center, Yale University School of Medicine, New Haven, Connecticut
| | - Vivian T Obeso
- V.T. Obeso is associate dean for curriculum and medical education and associate professor, Division of Internal Medicine, Department of Translational Medicine, Florida International University Herbert Wertheim College of Medicine, Miami, Florida
| | - Dianne P Wagner
- D.P. Wagner is associate dean for undergraduate medical education and professor of medicine, Michigan State University College of Human Medicine, East Lansing, Michigan
| | - Jamie B Warren
- J.B. Warren is associate professor, Division of Neonatology, and clinical vice chair, Department of Pediatrics, Oregon Health & Science University, Portland, Oregon; ORCID: https://orcid.org/0000-0003-4422-1502
| | - Sandra L Yingling
- S.L. Yingling is associate dean for educational planning and quality improvement, University of Illinois College of Medicine (Chicago, Peoria, Rockford, and Urbana), Chicago, Illinois
| | - Dorothy A Andriole
- D.A. Andriole is senior director, Medical Education Research, Association of American Medical Colleges, Washington, DC; ORCID: https://orcid.org/0000-0001-8902-1227
| |
Collapse
|
16
|
Peng CR, Schertzer KA, Caretta-Weyer HA, Sebok-Syer SS, Lu W, Tansomboon C, Gisondi MA. Assessment of Entrustable Professional Activities Using a Web-Based Simulation Platform During Transition to Emergency Medicine Residency: Mixed Methods Pilot Study. JMIR MEDICAL EDUCATION 2021; 7:e32356. [PMID: 34787582 PMCID: PMC8663509 DOI: 10.2196/32356] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/23/2021] [Revised: 09/30/2021] [Accepted: 10/03/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND The 13 core entrustable professional activities (EPAs) are key competency-based learning outcomes in the transition from undergraduate to graduate medical education in the United States. Five of these EPAs (EPA2: prioritizing differentials, EPA3: recommending and interpreting tests, EPA4: entering orders and prescriptions, EPA5: documenting clinical encounters, and EPA10: recognizing urgent and emergent conditions) are uniquely suited for web-based assessment. OBJECTIVE In this pilot study, we created cases on a web-based simulation platform for the diagnostic assessment of these EPAs and examined the feasibility and acceptability of the platform. METHODS Four simulation cases underwent 3 rounds of consensus panels and pilot testing. Incoming emergency medicine interns (N=15) completed all cases. A maximum of 4 "look for" statements, which encompassed specific EPAs, were generated for each participant: (1) performing harmful or missing actions, (2) narrowing differential or wrong final diagnosis, (3) errors in documentation, and (4) lack of recognition and stabilization of urgent diagnoses. Finally, we interviewed a sample of interns (n=5) and residency leadership (n=5) and analyzed the responses using thematic analysis. RESULTS All participants had at least one missing critical action, and 40% (6/15) of the participants performed at least one harmful action across all 4 cases. The final diagnosis was not included in the differential diagnosis in more than half of the assessments (8/15, 54%). Other errors included selecting incorrect documentation passages (6/15, 40%) and indiscriminately applying oxygen (9/15, 60%). The interview themes included psychological safety of the interface, ability to assess learning, and fidelity of cases. The most valuable feature cited was the ability to place orders in a realistic electronic medical record interface. CONCLUSIONS This study demonstrates the feasibility and acceptability of a web-based platform for diagnostic assessment of specific EPAs. The approach rapidly identifies potential areas of concern for incoming interns using an asynchronous format, provides feedback in a manner appreciated by residency leadership, and informs individualized learning plans.
Collapse
Affiliation(s)
- Cynthia R Peng
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Kimberly A Schertzer
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Holly A Caretta-Weyer
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Stefanie S Sebok-Syer
- Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - William Lu
- Cornell University College of Engineering, Ithaca, NY, United States
| | | | | |
Collapse
|
17
|
Bradley EB, Waselewski EA, Gusic ME. How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment. MEDICAL SCIENCE EDUCATOR 2021; 31:1419-1428. [PMID: 34457983 PMCID: PMC8368261 DOI: 10.1007/s40670-021-01327-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 05/20/2021] [Indexed: 06/13/2023]
Abstract
INTRODUCTION The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated. METHODS In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding. RESULTS Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout. CONCLUSIONS Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.
Collapse
Affiliation(s)
- Elizabeth B. Bradley
- Center for Medical Education Research and Scholarly Innovation, Office of Medical Education, University of Virginia School of Medicine, VA Charlottesville, USA
| | - Eric A. Waselewski
- Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan USA
| | - Maryellen E. Gusic
- Center for Medical Education Research and Scholarly Innovation, Office of Medical Education, University of Virginia School of Medicine, VA Charlottesville, USA
| |
Collapse
|
18
|
Amiel JM, Andriole DA, Biskobing DM, Brown DR, Cutrer WB, Emery MT, Mejicano GC, Ryan MS, Swails JL, Wagner DP. Revisiting the Core Entrustable Professional Activities for Entering Residency. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S14-S21. [PMID: 34183597 DOI: 10.1097/acm.0000000000004088] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
The Core EPAs for Entering Residency Pilot project aimed to test the feasibility of implementing 13 entrustable professional activities (EPAs) at 10 U.S. medical schools and to gauge whether the use of the Core EPAs could improve graduates' performance early in residency. In this manuscript, the authors (members of the pilot institutions and Association of American Medical Colleges staff supporting the project evaluation) describe the schools' capacity to collect multimodal evidence about their students' performance in each of the Core EPAs and the ability of faculty committees to use those data to make decisions regarding learners' readiness for entrustment. In reviewing data for each of the Core EPAs, the authors reflected on how each activity performed as an EPA informed by how well it could be assessed and entrusted. For EPAs that did not perform well, the authors examined whether there are underlying practical and/or theoretical issues limiting its utility as a measure of student performance in medical school.
Collapse
Affiliation(s)
- Jonathan M Amiel
- J.M. Amiel is associate professor of psychiatry, senior associate dean for curricular affairs, and interim co-vice dean for education, Columbia University Vagelos College of Physicians & Surgeons, New York, New York; ORCID: https://orcid.org/0000-0003-4027-6397
| | - Dorothy A Andriole
- D.A. Andriole is senior director, Medical Education Research, Association of American Medical Colleges, Washington, DC; ORCID: https://orcid.org/0000-0001-8902-1227
| | - Diane M Biskobing
- D.M. Biskobing is professor of medicine and associate dean for pre-clinical medical education, Virginia Commonwealth University School of Medicine, Richmond, Virginia
| | - David R Brown
- D.R. Brown is associate professor, chief, Family and Community Medicine, and interim chair, Humanities, Health, and Society, Florida International University Herbert Wertheim College of Medicine, Miami, Florida; ORCID: https://orcid.org/0000-0002-5361-6664
| | - William B Cutrer
- W.B. Cutrer is associate dean for undergraduate medical education and associate professor of pediatrics, Critical Care Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee; ORCID: https://orcid.org/0000-0003-1538-9779
| | - Matthew T Emery
- M.T. Emery is associate professor of emergency medicine and medical director for simulation, Michigan State University College of Human Medicine, East Lansing, Michigan
| | - George C Mejicano
- G.C. Mejicano is professor of medicine and senior associate dean for Education, Oregon Health & Science University School of Medicine, Portland, Oregon; ORCID: https://orcid.org/0000-0002-6087-3730
| | - Michael S Ryan
- M.S. Ryan is associate professor of pediatrics and assistant dean for clinical medical education, Virginia Commonwealth University School of Medicine, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Jennifer L Swails
- J.L. Swails is associate professor of internal medicine, McGovern Medical School, University of Texas Health Science Center, Houston, Texas; ORCID: https://orcid.org/0000-0002-6102-831X
| | - Dianne P Wagner
- D.P. Wagner is professor of medicine, associate dean for undergraduate medical education, and interim senior associate dean for academic affairs, Michigan State University College of Human Medicine, East Lansing, Michigan
| |
Collapse
|
19
|
Ten Cate O, Balmer DF, Caretta-Weyer H, Hatala R, Hennus MP, West DC. Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S96-S104. [PMID: 34183610 DOI: 10.1097/acm.0000000000004106] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.
Collapse
Affiliation(s)
- Olle Ten Cate
- O. ten Cate is professor of medical education and senior scientist, Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, the Netherlands; ORCID: https://orcid.org/0000-0002-6379-8780
| | - Dorene F Balmer
- D.F. Balmer is associate professor, Department of Pediatrics, The Children's Hospital of Philadelphia and University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: http://orcid.org/0000-0001-6805-4062
| | - Holly Caretta-Weyer
- H. Caretta-Weyer is assistant professor, Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California; ORCID: https://orcid.org/0000-0002-9783-5797
| | - Rose Hatala
- R. Hatala is professor, Department of Medicine, University of British Columbia, Vancouver, Canada; ORCID: https://orcid.org/0000-0003-0521-2590
| | - Marije P Hennus
- M.P. Hennus is a pediatric intensivist and program director, pediatric intensive care fellowship, University Medical Center Utrecht, Utrecht, the Netherlands; ORCID: https://orcid.org/0000-0003-1508-0456
| | - Daniel C West
- D.C. West is professor and senior director of medical education, Department of Pediatrics, Children's Hospital of Philadelphia and The Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0002-0909-4213
| |
Collapse
|
20
|
Roberts LW. Emerging Issues in Assessment in Medical Education: A Collection. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:159-160. [PMID: 33492817 DOI: 10.1097/acm.0000000000003855] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
|