1
|
Woodworth GE, Goldstein ZT, Ambardekar AP, Arthur ME, Bailey CF, Booth GJ, Carney PA, Chen F, Duncan MJ, Fromer IR, Hallman MR, Hoang T, Isaak R, Klesius LL, Ladlie BL, Mitchell SA, Miller Juve AK, Mitchell JD, McGrath BJ, Shepler JA, Sims CR, Spofford CM, Tanaka PP, Maniker RB. Development and Pilot Testing of a Programmatic System for Competency Assessment in US Anesthesiology Residency Training. Anesth Analg 2024; 138:1081-1093. [PMID: 37801598 DOI: 10.1213/ane.0000000000006667] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/08/2023]
Abstract
BACKGROUND In 2018, a set of entrustable professional activities (EPAs) and procedural skills assessments were developed for anesthesiology training, but they did not assess all the Accreditation Council for Graduate Medical Education (ACGME) milestones. The aims of this study were to (1) remap the 2018 EPA and procedural skills assessments to the revised ACGME Anesthesiology Milestones 2.0, (2) develop new assessments that combined with the original assessments to create a system of assessment that addresses all level 1 to 4 milestones, and (3) provide evidence for the validity of the assessments. METHODS Using a modified Delphi process, a panel of anesthesiology education experts remapped the original assessments developed in 2018 to the Anesthesiology Milestones 2.0 and developed new assessments to create a system that assessed all level 1 through 4 milestones. Following a 24-month pilot at 7 institutions, the number of EPA and procedural skill assessments and mean scores were computed at the end of the academic year. Milestone achievement and subcompetency data for assessments from a single institution were compared to scores assigned by the institution's clinical competency committee (CCC). RESULTS New assessment development, 2 months of testing and feedback, and revisions resulted in 5 new EPAs, 11 nontechnical skills assessments (NTSAs), and 6 objective structured clinical examinations (OSCEs). Combined with the original 20 EPAs and procedural skills assessments, the new system of assessment addresses 99% of level 1 to 4 Anesthesiology Milestones 2.0. During the 24-month pilot, aggregate mean EPA and procedural skill scores significantly increased with year in training. System subcompetency scores correlated significantly with 15 of 23 (65.2%) corresponding CCC scores at a single institution, but 8 correlations (36.4%) were <30.0, illustrating poor correlation. CONCLUSIONS A panel of experts developed a set of EPAs, procedural skill assessment, NTSAs, and OSCEs to form a programmatic system of assessment for anesthesiology residency training in the United States. The method used to develop and pilot test the assessments, the progression of assessment scores with time in training, and the correlation of assessment scores with CCC scoring of milestone achievement provide evidence for the validity of the assessments.
Collapse
Affiliation(s)
- Glenn E Woodworth
- From the Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon
| | - Zachary T Goldstein
- Department of Anesthesiology, Cedars Sinai Medical Center, Los Angeles, California
| | - Aditee P Ambardekar
- Department of Anesthesiology and Pain Management, University of Texas, Southwestern Medical Center, Dallas, Texas
| | - Mary E Arthur
- Department of Anesthesiology and Perioperative Medicine, Medical College of Georgia at Augusta University, Augusta, Georgia
| | - Caryl F Bailey
- Department of Anesthesiology and Perioperative Medicine, Medical College of Georgia at Augusta University, Augusta, Georgia
| | - Gregory J Booth
- Uniformed Services University of the Health Sciences, Department of Anesthesiology and Pain Medicine, Naval Medical Center Portsmouth, Portsmouth, Virginia
| | - Patricia A Carney
- Division of Hospital Medicine, Department of Family Medicine and Internal Medicine, Oregon Health & Science University, Portland, Oregon
| | - Fei Chen
- Department of Anesthesiology, University of North Carolina at Chapel Hill School of Medicine, Chapel Hill, North Carolina
| | - Michael J Duncan
- Department of Anesthesiology, University of Missouri-Kansas City, Kansas City, Missouri
| | - Ilana R Fromer
- Department of Anesthesiology, University of Minnesota, Minneapolis, Minnesota
| | - Matthew R Hallman
- Department of Anesthesiology and Pain Medicine, University of Washington, Seattle, Washington
| | - Thomas Hoang
- From the Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon
| | - Robert Isaak
- Department of Anesthesiology, University of North Carolina, Chapel Hill, North Carolina
| | - Lisa L Klesius
- Department of Anesthesiology, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin
| | - Beth L Ladlie
- Department of Anesthesiology, Mayo Clinic, Rochester, Minnesota
| | | | - Amy K Miller Juve
- From the Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon
| | - John D Mitchell
- Department of Anesthesiology, Critical Care, and Perioperative Medicine, Henry Ford Health, Detroit, Michigan
| | - Brian J McGrath
- Department of Anesthesiology, University of Florida College of Medicine-Jacksonville, Jacksonville, Florida
| | - John A Shepler
- Department of Anesthesiology, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin
| | - Charles R Sims
- Department of Anesthesiology & Perioperative Medicine, Mayo Clinic, Rochester, Minnesota
| | - Christina M Spofford
- Department of Anesthesiology, Medical College of Wisconsin, Milwaukee, Wisconsin
| | - Pedro P Tanaka
- Department of Anesthesiology, Stanford University, Stanford, California
| | - Robert B Maniker
- Department of Anesthesiology, Columbia University, New York, New York
| |
Collapse
|
2
|
Fuller SL, Ambardekar AP, Diachun CAB, Kearney MD, Long TR, Miller Juve AK, Mitchell JD, Woodworth GE. Competency-Based Time-Variable Anesthesiology Residency Training: Identification of Problems and Solutions. Anesth Analg 2024; 138:848-855. [PMID: 37450642 DOI: 10.1213/ane.0000000000006625] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/18/2023]
Abstract
BACKGROUND Global medical education is gradually moving toward more comprehensive implementations of a competency-based education (CBE) model. Elimination of standard time-based training and adoption of time-variable training (competency-based time-variable training [CB-TVT]) is one of the final stages of implementation of CBE. While CB-TVT has been implemented in some programs outside the United States, residency programs in the United States are still exploring this approach to training. The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) are encouraging member boards and residency review committees to consider innovative ways programs could implement CB-TVT. The goals of this study were to (1) identify potential problems with the implementation of CB-TVT in anesthesiology residency training, (2) rank the importance of the problems and the perceived difficulty of solving them, and (3) develop proposed solutions to the identified problems. METHODS Study participants were recruited from key stakeholder groups in anesthesiology education, including current or former program directors, department chairs, residents, fellows, American Board of Anesthesiology (ABA) board members, ACGME residency review committee members or ACGME leaders, designated institutional officials, residency program coordinators, clinical operations directors, and leaders of large anesthesiology community practice groups. This study was conducted in 2 phases. In phase 1, survey questionnaires were iteratively distributed to participants to identify problems with the implementation of CB-TVT. Participants were also asked to rank the perceived importance and difficulty of each problem and to identify relevant stakeholder groups that would be responsible for solving each problem. In phase 2, surveys focused on identifying potential solutions for problems identified in phase 1. RESULTS A total of 36 stakeholders identified 39 potential problems, grouped into 7 major categories, with the implementation of CB-TVT in anesthesiology residency training. Of the 39 problems, 19 (48.7%) were marked as important or very important on a 5-point scale and 12 of 19 (63.2%) of the important problems were marked as difficult or very difficult to solve on a 5-point scale. Stakeholders proposed 165 total solutions to the identified problems. CONCLUSIONS CB-TVT is a promising educational model for anesthesiology residency, which potentially results in learner flexibility, individualization of curricula, and utilization of competencies to determine learner advancement. Because of the potential problems with the implementation of CB-TVT, it is important for future pilot implementations of CB-TVT to document realized problems, efficacy of solutions, and effects on educational outcomes to justify the burden of implementing CB-TVT.
Collapse
Affiliation(s)
- Skylar L Fuller
- From the Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon
| | - Aditee P Ambardekar
- Department of Anesthesiology and Pain Management, University of Texas, Southwestern Medical Center, Dallas, Texas
| | - Carol Ann B Diachun
- Department of Anesthesiology, University of Florida-Jacksonville, Jacksonville, Florida
| | - Matthew D Kearney
- Department of Family Medicine and Community Health, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania
| | - Timothy R Long
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, Minnesota
| | - Amy K Miller Juve
- From the Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon
| | - John D Mitchell
- Department of Anesthesiology, Critical Care, and Perioperative Medicine, Henry Ford Health, Detroit, Michigan
| | - Glenn E Woodworth
- From the Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon
| |
Collapse
|
3
|
Marty AP, Linsenmeyer M, George B, Young JQ, Breckwoldt J, Ten Cate O. Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154. MEDICAL TEACHER 2023; 45:1203-1213. [PMID: 36706225 DOI: 10.1080/0142159x.2023.2168527] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.
Collapse
Affiliation(s)
| | - Machelle Linsenmeyer
- West Virginia School of Osteopathic Medicine, Lewisburg, WV, United States of America
| | - Brian George
- Surgery and Learning Health Sciences, University of Michigan, Ann Arbor, Michigan, United States of America
| | - John Q Young
- Department of Psychiatry, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell &, Zucker Hillside Hospital, NY, United States of America
| | - Jan Breckwoldt
- Institute of Anesthesia at the University Hospital Zurich, Switzerland
| | - Olle Ten Cate
- Utrecht Center for Research and Development of Health Professions Education at UMC Utrecht, The Netherlands
| |
Collapse
|
4
|
Elmes AT, Schwartz A, Tekian A, Jarrett JB. Evaluating the Quality of the Core Entrustable Professional Activities for New Pharmacy Graduates. PHARMACY 2023; 11:126. [PMID: 37624081 PMCID: PMC10458068 DOI: 10.3390/pharmacy11040126] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2023] [Revised: 07/19/2023] [Accepted: 08/07/2023] [Indexed: 08/26/2023] Open
Abstract
This study aimed to evaluate the quality of the American Association of Colleges of Pharmacy Core Entrustable Professional Activities (Core EPAs) for New Pharmacy Graduates according to standards outlined in competency-based education literature utilizing the Queen's EPA Quality (EQual) rubric. A cohort of pharmacists with EPA expertise rated Core EPA quality with the EQual rubric and provided recommendations for revisions. A generalizability study determined the reliability of the EQual ratings with pharmacist users. Nine pharmacists responded (4.4%). Most EPAs (9/15) did not reach the overall cut-off score, indicating low quality. EPAs 1 through 5 and EPA 14 (fulfill a medication order) were deemed high quality. EPA 12 (use evidence-based information to advance patient care) scored the lowest at 3.47 (SEM 0.29). EPA 14 scored the highest at 4.60 (SEM 0.14). EPA 15 (create a written plan for continuous professional development) was the only EPA to fail to reach the cut-off across all EQual domains. EPAs in the Patient Care Provider Domain received significantly higher ratings than other EPAs. On average, three respondents recommended revision for each. Most comments aligned with the EPA's EQual rubric performance. The generalizability study analysis revealed excellent reliability (G = 0.80). Determining EPA quality utilizing objective measurement tools should drive EPA development and revisions to more accurately reflect the roles, responsibilities, and expectations of pharmacists on the healthcare team.
Collapse
Affiliation(s)
- Abigail T. Elmes
- Department of Pharmacy Practice, University of Illinois Chicago College of Pharmacy, Chicago, IL 60612, USA;
| | - Alan Schwartz
- Department of Medical Education, University of Illinois Chicago College of Medicine, Chicago, IL 60612, USA; (A.S.); (A.T.)
| | - Ara Tekian
- Department of Medical Education, University of Illinois Chicago College of Medicine, Chicago, IL 60612, USA; (A.S.); (A.T.)
| | - Jennie B. Jarrett
- Department of Pharmacy Practice, University of Illinois Chicago College of Pharmacy, Chicago, IL 60612, USA;
- American Medical Association, Chicago, IL 60611, USA
| |
Collapse
|
5
|
Berger S, Stalmeijer RE, Marty AP, Berendonk C. Exploring the Impact of Entrustable Professional Activities on Feedback Culture: A Qualitative Study of Anesthesiology Residents and Attendings. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:836-843. [PMID: 36812061 DOI: 10.1097/acm.0000000000005188] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
PURPOSE Entrustable professional activities (EPAs) were introduced as a potential way to optimize workplace-based assessments. Yet, recent studies suggest that EPAs have not yet overcome all of the challenges to implementing meaningful feedback. The aim of this study was to explore the extent to which the introduction of EPAs via mobile app impacts feedback culture as experienced by anesthesiology residents and attending physicians. METHOD Using a constructivist grounded theory approach, the authors interviewed a purposive and theoretical sample of residents (n = 11) and attendings (n = 11) at the Institute of Anaesthesiology, University Hospital of Zurich, where EPAs had recently been implemented. Interviews took place between February and December 2021. Data collection and analysis were conducted iteratively. The authors used open, axial, and selective coding to gain knowledge and understanding on the interplay of EPAs and feedback culture. RESULTS Participants reflected on a number of changes in their day-to-day experience of feedback culture with the implementation of EPAs. Three main mechanisms were instrumental in this process: lowering the feedback threshold, change in feedback focus, and gamification. Participants felt a lower threshold to feedback seeking and giving and that the frequency of feedback conversations increased and tended to be more focused on a specific topic and shorter, while feedback content tended to focus more on technical skills and more attention was given to average performances. Residents indicated that the app-based approach fostered a game-like motivation to "climb levels," while attendings did not perceive a game-like experience. CONCLUSIONS EPAs may offer a solution to problems of infrequent occurrence of feedback and invite attention to average performances and technical competencies, but may come at the expense of feedback on nontechnical skills. This study suggests that feedback culture and feedback instruments have a mutually interacting influence on each other.
Collapse
Affiliation(s)
- Sabine Berger
- S. Berger is a third-year medical resident, Internal Medicine Training Program, St. Claraspital, Basel, Switzerland
| | - Renee E Stalmeijer
- R.E. Stalmeijer is associate professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Adrian P Marty
- A.P. Marty is currently senior attending physician and team lead for education, Institute of Anaesthesiology, Intensive Care and Pain Medicine, Orthopedic University Hospital Balgrist, Zurich, Switzerland. At the time of writing, he was attending physician, Institute of Anaesthesiology, University of Zurich, University Hospital of Zurich, Zurich, Switzerland
| | - Christoph Berendonk
- C. Berendonk is senior lecturer in medical education, Institute for Medical Education, University of Bern, Bern, Switzerland
| |
Collapse
|
6
|
Porter S, Prendiville E, Allen BFS, Booth G, Boublik J, Burnett GW, Elkassabany N, Hausman J, Klesius L, Le-Wendling L, Machi AT, Maniker R, Parra M, Rosenquist R, Spofford CM, Suresh S, Tedore T, Wilson EH, Zhou JY, Woodworth G. Development of entrustable professional activities for regional anesthesia and pain medicine fellowship training. Reg Anesth Pain Med 2022; 47:rapm-2022-103854. [PMID: 35878963 DOI: 10.1136/rapm-2022-103854] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Accepted: 07/14/2022] [Indexed: 11/04/2022]
Abstract
INTRODUCTION The Accreditation Council for Graduate Medical Education (ACGME) offers descriptions of competencies and milestones but does not provide standardized assessments to track trainee competency. Entrustable professional activities (EPAs) and special assessments (SAs) are emerging methods to assess the level of competency obtained by regional anesthesiology and acute pain medicine (RAAPM) fellows. METHODS A panel of RAAPM physicians with experience in education and competency assessment and one medical student were recruited to participate in a modified Delphi method with iterative rounds to reach consensus on: a list of EPAs, SAs, and procedural skills; detailed definitions for each EPA and SA; a mapping of the EPAs and SAs to the ACGME milestones; and a target level of entrustment for graduating US RAAPM fellows for each EPA and procedural skill. A gap analysis was performed and a heat map was created to cross-check the EPAs and SAs to the ACGME milestones. RESULTS Participants in EPA and SA development included 19 physicians and 1 medical student from 18 different programs. The Delphi rounds yielded a final list of 23 EPAs, a defined entrustment scale, mapping of the EPAs to ACGME milestones, and graduation targets. A list of 73 procedural skills and 7 SAs were similarly developed. DISCUSSION A list of 23 RAAPM EPAs, 73 procedural skills, and 7 SAs were created using a rigorous methodology to reach consensus. This framework can be utilized to help assess RAAPM fellows in the USA for competency and allow for meaningful performance feedback.
Collapse
Affiliation(s)
- Steven Porter
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic Florida, Jacksonville, Florida, USA
| | - Elaine Prendiville
- Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon, USA
| | | | - Gregory Booth
- Department of Anesthesiology and Pain Medicine, Naval Medical Center Portsmouth Department of Anesthesiology and Pain Medicine, Portsmouth, Virginia, USA
| | - Jan Boublik
- Anesthesiology, Stanford Hospital and Clinics, Stanford, California, USA
| | - Garrett W Burnett
- Anesthesiology, Perioperative & Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Nabil Elkassabany
- Department of Anesthesiology and Critical Care, University of Pennsylvania Health System, Philadelphia, Pennsylvania, USA
| | - Jonathan Hausman
- Department of Anesthesiology, Cedars-Sinai Medical Center, Los Angeles, California, USA
| | - Lisa Klesius
- Department of Anesthesiology, University of Wisconsin System, Madison, Wisconsin, USA
| | | | - Anthony T Machi
- Anesthesiology and Pain Management, University of Texas Southwestern Medical Center at Dallas, Dallas, Texas, USA
| | - Robert Maniker
- Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, California, USA
- Department of Anesthesiology, Columbia University Medical Center, New York, New York, USA
| | | | | | - Christina M Spofford
- Department of Anesthesiology, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Santhanam Suresh
- Pediatric Anesthesiology, Northwestern Medicine, Chicago, Illinois, USA
| | - Tiffany Tedore
- Anesthesiology, NewYork-Presbyterian Hospital/Weill Cornell Medical Center, New York, New York, USA
| | - Elizabeth H Wilson
- Department of Anesthesiology, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, USA
| | - Jon Yan Zhou
- Anesthesiology and Pain Medicine, University of California Davis Health System, Sacramento, California, USA
| | - Glenn Woodworth
- Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon, USA
| |
Collapse
|